2024-03-28T17:23:13.078 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-28T17:23:13.078 DEBUG:teuthology.run:Teuthology command: teuthology --name mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi --verbose --owner mchangir --archive /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625370 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} -- /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625370/orig.config.yaml 2024-03-28T17:23:13.133 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T17:23:13.228 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625370 branch: wip-mchangir-testing-main-20240318.032620 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} email: mchangir@redhat.com first_in_suite: false job_id: '7625370' kernel: client: branch: testing flavor: default kdb: true sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-testing-main-20240318.032620 ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - slow metadata IO - SLOW_OPS - slow request sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mchangir-testing-main-20240318.032620 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 owner: mchangir priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3645 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 sleep_before_teardown: 0 subset: 132/256 suite: fs suite_branch: wip-mchangir-testing-main-20240318.032620 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi045.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF7qGpnpDqWuSSv7MpCt5sNTHh/zsudVZ1RQWQLy14SwvJL8ynieuMZhJiHahZu+cAWoNIvj5RUjJV9FcSF3O9U= smithi071.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHsY3T/zbMT0/oz5MhsTO96HeLQ9CXsevuBN4CshHRe+OS/aq53yFMIzFF7r35/j+J6giLVewgklnApq/PRgt6E= smithi151.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKRV0++JyoPwr8Z+TdUdhcJHQHddAJIFJXFP9EQNPoL8YajJ9ahX8Wbyw2dZIEIcUoqTY1qSmqvg37NBITsxhek= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-27_03:45:23 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-28T17:23:13.229 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa; will attempt to use it 2024-03-28T17:23:13.229 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks 2024-03-28T17:23:13.230 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-28T17:23:13.231 INFO:teuthology.task.internal:Checking packages... 2024-03-28T17:23:13.258 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3be9a5c9bc793e11e2800a8c0c696e8b46742033' 2024-03-28T17:23:13.258 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-28T17:23:13.258 INFO:teuthology.packaging:ref: None 2024-03-28T17:23:13.258 INFO:teuthology.packaging:tag: None 2024-03-28T17:23:13.258 INFO:teuthology.packaging:branch: wip-mchangir-testing-main-20240318.032620 2024-03-28T17:23:13.258 INFO:teuthology.packaging:sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:23:13.259 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-testing-main-20240318.032620 2024-03-28T17:23:13.404 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2145.g3be9a5c9 2024-03-28T17:23:13.405 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-28T17:23:13.417 INFO:teuthology.task.internal:no buildpackages task found 2024-03-28T17:23:13.417 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-28T17:23:13.452 INFO:teuthology.task.internal:Saving configuration 2024-03-28T17:23:13.472 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-28T17:23:13.483 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T17:23:13.509 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625370', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 17:17:49.690447', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF7qGpnpDqWuSSv7MpCt5sNTHh/zsudVZ1RQWQLy14SwvJL8ynieuMZhJiHahZu+cAWoNIvj5RUjJV9FcSF3O9U='} 2024-03-28T17:23:13.527 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625370', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 17:17:49.687565', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHsY3T/zbMT0/oz5MhsTO96HeLQ9CXsevuBN4CshHRe+OS/aq53yFMIzFF7r35/j+J6giLVewgklnApq/PRgt6E='} 2024-03-28T17:23:13.546 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625370', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 17:17:49.689105', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKRV0++JyoPwr8Z+TdUdhcJHQHddAJIFJXFP9EQNPoL8YajJ9ahX8Wbyw2dZIEIcUoqTY1qSmqvg37NBITsxhek='} 2024-03-28T17:23:13.546 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-28T17:23:13.557 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-28T17:23:13.557 INFO:teuthology.task.internal:roles: ubuntu@smithi071.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-28T17:23:13.557 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-28T17:23:13.557 INFO:teuthology.run_tasks:Running task console_log... 2024-03-28T17:23:13.678 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7d763c30d0>, signals=[15]) 2024-03-28T17:23:13.679 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-28T17:23:13.688 INFO:teuthology.task.internal:Opening connections... 2024-03-28T17:23:13.689 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi045.front.sepia.ceph.com 2024-03-28T17:23:13.690 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T17:23:13.794 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi071.front.sepia.ceph.com 2024-03-28T17:23:13.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T17:23:13.879 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi151.front.sepia.ceph.com 2024-03-28T17:23:13.880 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T17:23:13.966 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-28T17:23:13.975 DEBUG:teuthology.orchestra.run.smithi045:> uname -m 2024-03-28T17:23:13.994 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2024-03-28T17:23:13.994 DEBUG:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2024-03-28T17:23:14.052 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="CentOS Stream" 2024-03-28T17:23:14.052 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="9" 2024-03-28T17:23:14.052 INFO:teuthology.orchestra.run.smithi045.stdout:ID="centos" 2024-03-28T17:23:14.052 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE="rhel fedora" 2024-03-28T17:23:14.052 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="9" 2024-03-28T17:23:14.052 INFO:teuthology.orchestra.run.smithi045.stdout:PLATFORM_ID="platform:el9" 2024-03-28T17:23:14.052 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T17:23:14.052 INFO:teuthology.orchestra.run.smithi045.stdout:ANSI_COLOR="0;31" 2024-03-28T17:23:14.052 INFO:teuthology.orchestra.run.smithi045.stdout:LOGO="fedora-logo-icon" 2024-03-28T17:23:14.053 INFO:teuthology.orchestra.run.smithi045.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T17:23:14.053 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://centos.org/" 2024-03-28T17:23:14.053 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T17:23:14.053 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T17:23:14.053 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T17:23:14.053 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2024-03-28T17:23:14.078 DEBUG:teuthology.orchestra.run.smithi071:> uname -m 2024-03-28T17:23:14.096 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2024-03-28T17:23:14.096 DEBUG:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2024-03-28T17:23:14.153 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="CentOS Stream" 2024-03-28T17:23:14.153 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="9" 2024-03-28T17:23:14.153 INFO:teuthology.orchestra.run.smithi071.stdout:ID="centos" 2024-03-28T17:23:14.153 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE="rhel fedora" 2024-03-28T17:23:14.154 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="9" 2024-03-28T17:23:14.154 INFO:teuthology.orchestra.run.smithi071.stdout:PLATFORM_ID="platform:el9" 2024-03-28T17:23:14.154 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T17:23:14.154 INFO:teuthology.orchestra.run.smithi071.stdout:ANSI_COLOR="0;31" 2024-03-28T17:23:14.154 INFO:teuthology.orchestra.run.smithi071.stdout:LOGO="fedora-logo-icon" 2024-03-28T17:23:14.154 INFO:teuthology.orchestra.run.smithi071.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T17:23:14.154 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://centos.org/" 2024-03-28T17:23:14.154 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T17:23:14.154 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T17:23:14.154 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T17:23:14.154 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2024-03-28T17:23:14.186 DEBUG:teuthology.orchestra.run.smithi151:> uname -m 2024-03-28T17:23:14.204 INFO:teuthology.orchestra.run.smithi151.stdout:x86_64 2024-03-28T17:23:14.205 DEBUG:teuthology.orchestra.run.smithi151:> cat /etc/os-release 2024-03-28T17:23:14.262 INFO:teuthology.orchestra.run.smithi151.stdout:NAME="CentOS Stream" 2024-03-28T17:23:14.262 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION="9" 2024-03-28T17:23:14.262 INFO:teuthology.orchestra.run.smithi151.stdout:ID="centos" 2024-03-28T17:23:14.262 INFO:teuthology.orchestra.run.smithi151.stdout:ID_LIKE="rhel fedora" 2024-03-28T17:23:14.262 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_ID="9" 2024-03-28T17:23:14.263 INFO:teuthology.orchestra.run.smithi151.stdout:PLATFORM_ID="platform:el9" 2024-03-28T17:23:14.263 INFO:teuthology.orchestra.run.smithi151.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T17:23:14.263 INFO:teuthology.orchestra.run.smithi151.stdout:ANSI_COLOR="0;31" 2024-03-28T17:23:14.263 INFO:teuthology.orchestra.run.smithi151.stdout:LOGO="fedora-logo-icon" 2024-03-28T17:23:14.263 INFO:teuthology.orchestra.run.smithi151.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T17:23:14.263 INFO:teuthology.orchestra.run.smithi151.stdout:HOME_URL="https://centos.org/" 2024-03-28T17:23:14.263 INFO:teuthology.orchestra.run.smithi151.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T17:23:14.263 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T17:23:14.263 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T17:23:14.263 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2024-03-28T17:23:14.287 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-28T17:23:14.299 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-28T17:23:14.321 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-28T17:23:14.321 DEBUG:teuthology.orchestra.run.smithi045:> test '!' -e /home/ubuntu/cephtest 2024-03-28T17:23:14.323 DEBUG:teuthology.orchestra.run.smithi071:> test '!' -e /home/ubuntu/cephtest 2024-03-28T17:23:14.326 DEBUG:teuthology.orchestra.run.smithi151:> test '!' -e /home/ubuntu/cephtest 2024-03-28T17:23:14.342 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-28T17:23:14.352 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-28T17:23:14.352 DEBUG:teuthology.orchestra.run.smithi045:> test -z $(ls -A /var/lib/ceph) 2024-03-28T17:23:14.380 DEBUG:teuthology.orchestra.run.smithi071:> test -z $(ls -A /var/lib/ceph) 2024-03-28T17:23:14.383 DEBUG:teuthology.orchestra.run.smithi151:> test -z $(ls -A /var/lib/ceph) 2024-03-28T17:23:14.456 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-28T17:23:14.532 INFO:teuthology.run_tasks:Running task kernel... 2024-03-28T17:23:14.550 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'} 2024-03-28T17:23:14.551 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-28T17:23:14.551 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-28T17:23:14.551 DEBUG:teuthology.orchestra.run.smithi045:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T17:23:14.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T17:23:14.568 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-28T17:23:14.970 DEBUG:teuthology.task.kernel:sha1 for client.0 is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T17:23:14.970 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-03-28T17:23:14.970 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-03-28T17:23:14.988 INFO:teuthology.orchestra.run.smithi045.stdout:5.14.0-361.el9.x86_64 2024-03-28T17:23:14.988 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T17:23:14.989 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-28T17:23:14.989 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum upgrade -y linux-firmware 2024-03-28T17:23:17.235 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-28T17:23:18.794 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - BaseOS 4.8 MB/s | 2.0 MB 00:00 2024-03-28T17:23:23.776 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 1.7 MB/s | 8.0 MB 00:04 2024-03-28T17:23:26.201 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-03-28T17:23:30.568 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 8.0 MB/s | 6.1 MB 00:00 2024-03-28T17:23:32.911 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-03-28T17:23:34.698 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-28T17:23:39.904 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-28T17:23:40.086 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:01 ago on Thu 28 Mar 2024 05:23:39 PM UTC. 2024-03-28T17:23:42.264 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-28T17:23:42.275 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-28T17:23:42.276 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-28T17:23:42.276 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-28T17:23:42.276 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrading: 2024-03-28T17:23:42.276 INFO:teuthology.orchestra.run.smithi045.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-28T17:23:42.276 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:23:42.276 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-28T17:23:42.276 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-28T17:23:42.276 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrade 1 Package 2024-03-28T17:23:42.276 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:23:42.276 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 375 M 2024-03-28T17:23:42.276 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-28T17:23:49.513 INFO:teuthology.orchestra.run.smithi045.stdout:linux-firmware-20240219-143.el9.noarch.rpm 52 MB/s | 375 MB 00:07 2024-03-28T17:23:49.514 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-28T17:23:49.514 INFO:teuthology.orchestra.run.smithi045.stdout:Total 52 MB/s | 375 MB 00:07 2024-03-28T17:23:49.515 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-28T17:23:49.620 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-28T17:23:49.620 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-28T17:23:52.125 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-28T17:23:52.126 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-28T17:23:58.685 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-28T17:23:58.927 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-28T17:23:59.041 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T17:24:00.483 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T17:24:00.483 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-28T17:24:00.984 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T17:24:00.984 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:24:00.984 INFO:teuthology.orchestra.run.smithi045.stdout:Upgraded: 2024-03-28T17:24:00.984 INFO:teuthology.orchestra.run.smithi045.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-28T17:24:00.984 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:24:00.984 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-28T17:24:01.260 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.0... 2024-03-28T17:24:01.261 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T17:24:01.408 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/ 2024-03-28T17:24:01.408 DEBUG:teuthology.orchestra.run.smithi045:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-28T17:24:01.411 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi045.front.sepia.ceph.com to complete... 2024-03-28T17:24:13.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28 17:24:13 URL:https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm [336317861/336317861] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-28T17:24:13.354 INFO:teuthology.orchestra.run.smithi045.stderr:FINISHED --2024-03-28 17:24:13-- 2024-03-28T17:24:13.354 INFO:teuthology.orchestra.run.smithi045.stderr:Total wall clock time: 12s 2024-03-28T17:24:13.354 INFO:teuthology.orchestra.run.smithi045.stderr:Downloaded: 1 files, 321M in 12s (27.8 MB/s) 2024-03-28T17:24:13.357 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.0... 2024-03-28T17:24:13.358 DEBUG:teuthology.orchestra.run.smithi045:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-28T17:24:15.385 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying... ######################################## 2024-03-28T17:24:15.442 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing... ######################################## 2024-03-28T17:24:15.527 INFO:teuthology.orchestra.run.smithi045.stdout:Updating / installing... 2024-03-28T17:24:23.442 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-6.8.0_g1df0c7e8a235-1 ######################################## 2024-03-28T17:24:39.039 INFO:teuthology.orchestra.run.smithi045.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-28T17:25:02.314 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi045.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-28T17:25:02.314 DEBUG:teuthology.orchestra.run.smithi045:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-28T17:25:02.393 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235 2024-03-28T17:25:02.394 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/System.map 2024-03-28T17:25:02.394 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/config 2024-03-28T17:25:02.394 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel 2024-03-28T17:25:02.394 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch 2024-03-28T17:25:02.394 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86 2024-03-28T17:25:02.394 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto 2024-03-28T17:25:02.394 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-28T17:25:02.394 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-28T17:25:02.394 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-28T17:25:02.394 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-28T17:25:02.394 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-28T17:25:02.395 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-28T17:25:02.395 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-28T17:25:02.395 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-28T17:25:02.395 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-28T17:25:02.395 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-28T17:25:02.395 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-28T17:25:02.395 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-28T17:25:02.395 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-28T17:25:02.395 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-28T17:25:02.395 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-28T17:25:02.395 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-28T17:25:02.395 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-28T17:25:02.395 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-28T17:25:02.395 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-28T17:25:02.396 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-28T17:25:02.396 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel 2024-03-28T17:25:02.396 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu 2024-03-28T17:25:02.396 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce 2024-03-28T17:25:02.396 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-28T17:25:02.396 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm 2024-03-28T17:25:02.396 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-28T17:25:02.396 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-28T17:25:02.396 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-03-28T17:25:02.396 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto 2024-03-28T17:25:02.396 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-03-28T17:25:02.396 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-03-28T17:25:02.396 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-03-28T17:25:02.396 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx 2024-03-28T17:25:02.397 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-03-28T17:25:02.397 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-03-28T17:25:02.397 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-28T17:25:02.397 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-03-28T17:25:02.397 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-03-28T17:25:02.397 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-03-28T17:25:02.397 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-03-28T17:25:02.397 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-03-28T17:25:02.397 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-03-28T17:25:02.397 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-03-28T17:25:02.397 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-03-28T17:25:02.397 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-03-28T17:25:02.397 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-03-28T17:25:02.398 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-03-28T17:25:02.398 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-03-28T17:25:02.398 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-03-28T17:25:02.398 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-03-28T17:25:02.398 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-03-28T17:25:02.398 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-03-28T17:25:02.398 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-03-28T17:25:02.398 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-03-28T17:25:02.398 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-03-28T17:25:02.398 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-03-28T17:25:02.398 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-03-28T17:25:02.398 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-03-28T17:25:02.398 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-03-28T17:25:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-03-28T17:25:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-03-28T17:25:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-03-28T17:25:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-03-28T17:25:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-03-28T17:25:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-03-28T17:25:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-03-28T17:25:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-03-28T17:25:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-03-28T17:25:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-03-28T17:25:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-03-28T17:25:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-03-28T17:25:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-03-28T17:25:02.400 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-03-28T17:25:02.400 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-03-28T17:25:02.400 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-03-28T17:25:02.400 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-03-28T17:25:02.400 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-03-28T17:25:02.400 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-03-28T17:25:02.400 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers 2024-03-28T17:25:02.400 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi 2024-03-28T17:25:02.400 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-28T17:25:02.400 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-03-28T17:25:02.400 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/custom_method.ko 2024-03-28T17:25:02.400 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-03-28T17:25:02.400 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-03-28T17:25:02.400 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-03-28T17:25:02.401 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-03-28T17:25:02.401 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-03-28T17:25:02.401 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata 2024-03-28T17:25:02.401 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-03-28T17:25:02.401 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-03-28T17:25:02.401 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ata_generic.ko 2024-03-28T17:25:02.401 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-03-28T17:25:02.401 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-03-28T17:25:02.401 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-03-28T17:25:02.401 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-03-28T17:25:02.401 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-03-28T17:25:02.401 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-03-28T17:25:02.401 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-03-28T17:25:02.401 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-03-28T17:25:02.401 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-03-28T17:25:02.401 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-03-28T17:25:02.402 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-03-28T17:25:02.402 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-03-28T17:25:02.402 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-03-28T17:25:02.402 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-28T17:25:02.402 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-28T17:25:02.402 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-03-28T17:25:02.402 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-03-28T17:25:02.402 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-03-28T17:25:02.402 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-03-28T17:25:02.402 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-03-28T17:25:02.402 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-03-28T17:25:02.402 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-03-28T17:25:02.402 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-03-28T17:25:02.402 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-03-28T17:25:02.403 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-03-28T17:25:02.403 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-03-28T17:25:02.403 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-03-28T17:25:02.403 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-03-28T17:25:02.403 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-28T17:25:02.403 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-28T17:25:02.403 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-03-28T17:25:02.403 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-03-28T17:25:02.403 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-03-28T17:25:02.403 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-03-28T17:25:02.403 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-03-28T17:25:02.403 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sis.ko 2024-03-28T17:25:02.403 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-03-28T17:25:02.403 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-03-28T17:25:02.403 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-03-28T17:25:02.404 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-03-28T17:25:02.404 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-03-28T17:25:02.404 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-03-28T17:25:02.404 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-03-28T17:25:02.404 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-03-28T17:25:02.404 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-03-28T17:25:02.404 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-03-28T17:25:02.404 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-03-28T17:25:02.404 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-03-28T17:25:02.404 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-03-28T17:25:02.404 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-03-28T17:25:02.404 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-03-28T17:25:02.404 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-03-28T17:25:02.404 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-03-28T17:25:02.404 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay 2024-03-28T17:25:02.405 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-03-28T17:25:02.405 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base 2024-03-28T17:25:02.405 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap 2024-03-28T17:25:02.405 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-28T17:25:02.405 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-28T17:25:02.405 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma 2024-03-28T17:25:02.405 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-03-28T17:25:02.405 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block 2024-03-28T17:25:02.405 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe 2024-03-28T17:25:02.405 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-03-28T17:25:02.405 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/brd.ko 2024-03-28T17:25:02.405 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd 2024-03-28T17:25:02.405 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-03-28T17:25:02.405 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-03-28T17:25:02.405 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/loop.ko 2024-03-28T17:25:02.405 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx 2024-03-28T17:25:02.406 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-28T17:25:02.406 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-03-28T17:25:02.406 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk 2024-03-28T17:25:02.406 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-03-28T17:25:02.406 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-03-28T17:25:02.406 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-03-28T17:25:02.406 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/virtio_blk.ko 2024-03-28T17:25:02.406 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback 2024-03-28T17:25:02.406 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-28T17:25:02.406 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkfront.ko 2024-03-28T17:25:02.406 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram 2024-03-28T17:25:02.406 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-03-28T17:25:02.406 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char 2024-03-28T17:25:02.406 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-03-28T17:25:02.407 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random 2024-03-28T17:25:02.407 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-28T17:25:02.407 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-28T17:25:02.407 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-28T17:25:02.407 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-03-28T17:25:02.407 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-28T17:25:02.407 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi 2024-03-28T17:25:02.407 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-28T17:25:02.407 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-28T17:25:02.407 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-28T17:25:02.407 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-28T17:25:02.407 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-28T17:25:02.407 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-28T17:25:02.407 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-03-28T17:25:02.407 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave 2024-03-28T17:25:02.408 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-03-28T17:25:02.408 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-03-28T17:25:02.408 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-03-28T17:25:02.408 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm 2024-03-28T17:25:02.408 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm.ko 2024-03-28T17:25:02.408 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-28T17:25:02.408 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-28T17:25:02.408 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-28T17:25:02.408 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-28T17:25:02.408 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-28T17:25:02.408 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/uv_mmtimer.ko 2024-03-28T17:25:02.408 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/virtio_console.ko 2024-03-28T17:25:02.408 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq 2024-03-28T17:25:02.408 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-28T17:25:02.408 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-28T17:25:02.409 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-28T17:25:02.409 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-28T17:25:02.409 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-28T17:25:02.409 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-28T17:25:02.409 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto 2024-03-28T17:25:02.409 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp 2024-03-28T17:25:02.409 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-28T17:25:02.409 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-03-28T17:25:02.409 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel 2024-03-28T17:25:02.409 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat 2024-03-28T17:25:02.409 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common 2024-03-28T17:25:02.409 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-28T17:25:02.409 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-28T17:25:02.409 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-28T17:25:02.409 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-03-28T17:25:02.410 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-03-28T17:25:02.410 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio 2024-03-28T17:25:02.410 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-28T17:25:02.410 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca 2024-03-28T17:25:02.410 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-03-28T17:25:02.410 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq 2024-03-28T17:25:02.410 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-28T17:25:02.410 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma 2024-03-28T17:25:02.410 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw 2024-03-28T17:25:02.410 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-28T17:25:02.410 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat 2024-03-28T17:25:02.410 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-28T17:25:02.410 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac 2024-03-28T17:25:02.410 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-03-28T17:25:02.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-03-28T17:25:02.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-03-28T17:25:02.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-03-28T17:25:02.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-03-28T17:25:02.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-03-28T17:25:02.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-03-28T17:25:02.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-03-28T17:25:02.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-03-28T17:25:02.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-03-28T17:25:02.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-03-28T17:25:02.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-03-28T17:25:02.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-03-28T17:25:02.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware 2024-03-28T17:25:02.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/edd.ko 2024-03-28T17:25:02.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-28T17:25:02.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio 2024-03-28T17:25:02.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-03-28T17:25:02.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-28T17:25:02.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid 2024-03-28T17:25:02.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-03-28T17:25:02.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-03-28T17:25:02.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-03-28T17:25:02.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-03-28T17:25:02.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-03-28T17:25:02.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-03-28T17:25:02.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-03-28T17:25:02.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-03-28T17:25:02.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-03-28T17:25:02.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-03-28T17:25:02.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-28T17:25:02.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-28T17:25:02.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-03-28T17:25:02.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-03-28T17:25:02.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-03-28T17:25:02.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-03-28T17:25:02.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-03-28T17:25:02.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-03-28T17:25:02.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-03-28T17:25:02.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-03-28T17:25:02.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-28T17:25:02.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-28T17:25:02.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-03-28T17:25:02.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-03-28T17:25:02.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-03-28T17:25:02.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-03-28T17:25:02.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-03-28T17:25:02.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-03-28T17:25:02.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-03-28T17:25:02.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-03-28T17:25:02.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-03-28T17:25:02.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-28T17:25:02.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-03-28T17:25:02.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-28T17:25:02.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-28T17:25:02.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-28T17:25:02.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-28T17:25:02.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-28T17:25:02.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-28T17:25:02.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-28T17:25:02.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-28T17:25:02.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-28T17:25:02.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-03-28T17:25:02.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-03-28T17:25:02.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-03-28T17:25:02.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-28T17:25:02.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-03-28T17:25:02.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-03-28T17:25:02.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-03-28T17:25:02.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-03-28T17:25:02.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-03-28T17:25:02.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-28T17:25:02.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-03-28T17:25:02.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-03-28T17:25:02.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-03-28T17:25:02.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-03-28T17:25:02.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-03-28T17:25:02.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-03-28T17:25:02.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-03-28T17:25:02.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-03-28T17:25:02.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-03-28T17:25:02.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-03-28T17:25:02.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-03-28T17:25:02.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-03-28T17:25:02.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv 2024-03-28T17:25:02.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-03-28T17:25:02.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-03-28T17:25:02.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-03-28T17:25:02.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon 2024-03-28T17:25:02.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-03-28T17:25:02.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-03-28T17:25:02.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-28T17:25:02.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-03-28T17:25:02.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-03-28T17:25:02.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-03-28T17:25:02.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-03-28T17:25:02.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-03-28T17:25:02.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-03-28T17:25:02.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-03-28T17:25:02.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-03-28T17:25:02.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-03-28T17:25:02.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-03-28T17:25:02.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-03-28T17:25:02.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-03-28T17:25:02.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-03-28T17:25:02.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-03-28T17:25:02.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-03-28T17:25:02.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-03-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-03-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-03-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-03-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-03-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-03-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-03-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-03-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-03-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-03-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-03-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-03-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-03-28T17:25:02.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-03-28T17:25:02.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-03-28T17:25:02.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-03-28T17:25:02.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-03-28T17:25:02.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-03-28T17:25:02.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-03-28T17:25:02.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-03-28T17:25:02.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-28T17:25:02.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-03-28T17:25:02.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-03-28T17:25:02.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-03-28T17:25:02.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-03-28T17:25:02.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-03-28T17:25:02.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-03-28T17:25:02.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-03-28T17:25:02.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-03-28T17:25:02.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-03-28T17:25:02.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-03-28T17:25:02.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-03-28T17:25:02.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-03-28T17:25:02.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-03-28T17:25:02.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-03-28T17:25:02.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-03-28T17:25:02.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-03-28T17:25:02.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-03-28T17:25:02.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-03-28T17:25:02.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-03-28T17:25:02.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-03-28T17:25:02.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-03-28T17:25:02.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-03-28T17:25:02.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-03-28T17:25:02.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-03-28T17:25:02.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-03-28T17:25:02.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-03-28T17:25:02.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-03-28T17:25:02.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-03-28T17:25:02.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-03-28T17:25:02.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-03-28T17:25:02.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-03-28T17:25:02.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-03-28T17:25:02.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-03-28T17:25:02.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-03-28T17:25:02.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-03-28T17:25:02.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-03-28T17:25:02.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-03-28T17:25:02.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-03-28T17:25:02.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-03-28T17:25:02.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-03-28T17:25:02.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-03-28T17:25:02.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-03-28T17:25:02.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-03-28T17:25:02.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-03-28T17:25:02.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-03-28T17:25:02.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-03-28T17:25:02.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-03-28T17:25:02.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus 2024-03-28T17:25:02.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-28T17:25:02.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-28T17:25:02.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-28T17:25:02.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-28T17:25:02.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-28T17:25:02.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-28T17:25:02.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-28T17:25:02.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-28T17:25:02.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-28T17:25:02.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-28T17:25:02.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-28T17:25:02.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-28T17:25:02.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-03-28T17:25:02.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-03-28T17:25:02.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-03-28T17:25:02.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-28T17:25:02.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-03-28T17:25:02.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-03-28T17:25:02.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-03-28T17:25:02.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-03-28T17:25:02.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-03-28T17:25:02.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-03-28T17:25:02.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-03-28T17:25:02.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-03-28T17:25:02.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-03-28T17:25:02.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-03-28T17:25:02.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-03-28T17:25:02.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-03-28T17:25:02.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-03-28T17:25:02.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-03-28T17:25:02.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-03-28T17:25:02.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-03-28T17:25:02.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-03-28T17:25:02.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-03-28T17:25:02.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-03-28T17:25:02.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-03-28T17:25:02.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-03-28T17:25:02.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-03-28T17:25:02.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-03-28T17:25:02.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-03-28T17:25:02.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-03-28T17:25:02.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c 2024-03-28T17:25:02.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos 2024-03-28T17:25:02.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-28T17:25:02.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-28T17:25:02.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses 2024-03-28T17:25:02.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-28T17:25:02.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-28T17:25:02.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-28T17:25:02.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-28T17:25:02.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-28T17:25:02.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-28T17:25:02.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-28T17:25:02.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-28T17:25:02.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-28T17:25:02.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-28T17:25:02.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-28T17:25:02.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-28T17:25:02.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-28T17:25:02.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-28T17:25:02.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-28T17:25:02.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-28T17:25:02.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-28T17:25:02.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-28T17:25:02.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-28T17:25:02.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-28T17:25:02.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-28T17:25:02.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-28T17:25:02.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-28T17:25:02.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-dev.ko 2024-03-28T17:25:02.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-03-28T17:25:02.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-28T17:25:02.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-03-28T17:25:02.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-03-28T17:25:02.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband 2024-03-28T17:25:02.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core 2024-03-28T17:25:02.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-28T17:25:02.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-03-28T17:25:02.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-28T17:25:02.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-28T17:25:02.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-28T17:25:02.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-28T17:25:02.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-28T17:25:02.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw 2024-03-28T17:25:02.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4 2024-03-28T17:25:02.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-28T17:25:02.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4 2024-03-28T17:25:02.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-28T17:25:02.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5 2024-03-28T17:25:02.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-28T17:25:02.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca 2024-03-28T17:25:02.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-28T17:25:02.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp 2024-03-28T17:25:02.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib 2024-03-28T17:25:02.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-28T17:25:02.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser 2024-03-28T17:25:02.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-28T17:25:02.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert 2024-03-28T17:25:02.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-28T17:25:02.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp 2024-03-28T17:25:02.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-28T17:25:02.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt 2024-03-28T17:25:02.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-28T17:25:02.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input 2024-03-28T17:25:02.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport 2024-03-28T17:25:02.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-28T17:25:02.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-28T17:25:02.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-03-28T17:25:02.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-03-28T17:25:02.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard 2024-03-28T17:25:02.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-28T17:25:02.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse 2024-03-28T17:25:02.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-03-28T17:25:02.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-03-28T17:25:02.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-03-28T17:25:02.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-28T17:25:02.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-03-28T17:25:02.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-28T17:25:02.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-28T17:25:02.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-28T17:25:02.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4 2024-03-28T17:25:02.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-28T17:25:02.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio 2024-03-28T17:25:02.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-03-28T17:25:02.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-03-28T17:25:02.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-28T17:25:02.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-03-28T17:25:02.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-03-28T17:25:02.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds 2024-03-28T17:25:02.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-03-28T17:25:02.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-03-28T17:25:02.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-03-28T17:25:02.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-03-28T17:25:02.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger 2024-03-28T17:25:02.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-28T17:25:02.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-28T17:25:02.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-28T17:25:02.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-28T17:25:02.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-28T17:25:02.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-28T17:25:02.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-28T17:25:02.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-28T17:25:02.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-28T17:25:02.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md 2024-03-28T17:25:02.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache 2024-03-28T17:25:02.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-03-28T17:25:02.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-03-28T17:25:02.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-03-28T17:25:02.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-03-28T17:25:02.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-03-28T17:25:02.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-03-28T17:25:02.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-03-28T17:25:02.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-03-28T17:25:02.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-03-28T17:25:02.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-03-28T17:25:02.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-03-28T17:25:02.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-03-28T17:25:02.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-03-28T17:25:02.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-03-28T17:25:02.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-03-28T17:25:02.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-03-28T17:25:02.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data 2024-03-28T17:25:02.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-28T17:25:02.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-03-28T17:25:02.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-03-28T17:25:02.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-03-28T17:25:02.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-03-28T17:25:02.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message 2024-03-28T17:25:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion 2024-03-28T17:25:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-03-28T17:25:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-03-28T17:25:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-03-28T17:25:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-03-28T17:25:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-03-28T17:25:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-03-28T17:25:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-03-28T17:25:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd 2024-03-28T17:25:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-03-28T17:25:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-03-28T17:25:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mfd-core.ko 2024-03-28T17:25:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-03-28T17:25:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-03-28T17:25:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-03-28T17:25:02.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-03-28T17:25:02.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc 2024-03-28T17:25:02.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl 2024-03-28T17:25:02.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-28T17:25:02.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-03-28T17:25:02.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-03-28T17:25:02.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-03-28T17:25:02.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710 2024-03-28T17:25:02.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-03-28T17:25:02.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom 2024-03-28T17:25:02.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-03-28T17:25:02.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-28T17:25:02.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-03-28T17:25:02.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-03-28T17:25:02.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-03-28T17:25:02.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-03-28T17:25:02.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-03-28T17:25:02.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d 2024-03-28T17:25:02.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-28T17:25:02.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-28T17:25:02.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei 2024-03-28T17:25:02.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-03-28T17:25:02.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-03-28T17:25:02.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-03-28T17:25:02.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru 2024-03-28T17:25:02.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-28T17:25:02.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp 2024-03-28T17:25:02.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-28T17:25:02.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-28T17:25:02.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-28T17:25:02.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-03-28T17:25:02.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-03-28T17:25:02.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-03-28T17:25:02.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-03-28T17:25:02.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci 2024-03-28T17:25:02.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-28T17:25:02.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net 2024-03-28T17:25:02.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding 2024-03-28T17:25:02.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-03-28T17:25:02.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa 2024-03-28T17:25:02.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53 2024-03-28T17:25:02.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-28T17:25:02.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-28T17:25:02.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-28T17:25:02.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx 2024-03-28T17:25:02.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-28T17:25:02.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-03-28T17:25:02.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-03-28T17:25:02.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet 2024-03-28T17:25:02.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com 2024-03-28T17:25:02.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-28T17:25:02.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-28T17:25:02.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-28T17:25:02.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-28T17:25:02.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390 2024-03-28T17:25:02.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-28T17:25:02.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-28T17:25:02.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-28T17:25:02.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-28T17:25:02.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec 2024-03-28T17:25:02.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-28T17:25:02.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere 2024-03-28T17:25:02.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-28T17:25:02.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon 2024-03-28T17:25:02.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-28T17:25:02.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera 2024-03-28T17:25:02.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-28T17:25:02.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd 2024-03-28T17:25:02.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-28T17:25:02.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-28T17:25:02.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-28T17:25:02.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros 2024-03-28T17:25:02.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx 2024-03-28T17:25:02.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-28T17:25:02.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-28T17:25:02.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-28T17:25:02.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-28T17:25:02.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-28T17:25:02.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx 2024-03-28T17:25:02.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-28T17:25:02.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-28T17:25:02.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom 2024-03-28T17:25:02.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-28T17:25:02.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-28T17:25:02.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-28T17:25:02.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-28T17:25:02.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-28T17:25:02.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet 2024-03-28T17:25:02.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-28T17:25:02.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-28T17:25:02.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade 2024-03-28T17:25:02.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna 2024-03-28T17:25:02.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-28T17:25:02.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio 2024-03-28T17:25:02.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-28T17:25:02.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-28T17:25:02.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-28T17:25:02.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-28T17:25:02.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-28T17:25:02.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-28T17:25:02.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-28T17:25:02.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-28T17:25:02.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-28T17:25:02.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-28T17:25:02.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco 2024-03-28T17:25:02.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic 2024-03-28T17:25:02.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-28T17:25:02.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec 2024-03-28T17:25:02.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip 2024-03-28T17:25:02.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-28T17:25:02.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-28T17:25:02.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-28T17:25:02.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-28T17:25:02.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-28T17:25:02.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-28T17:25:02.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink 2024-03-28T17:25:02.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-28T17:25:02.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-28T17:25:02.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-03-28T17:25:02.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex 2024-03-28T17:25:02.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet 2024-03-28T17:25:02.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-28T17:25:02.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-03-28T17:25:02.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-03-28T17:25:02.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel 2024-03-28T17:25:02.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-28T17:25:02.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000 2024-03-28T17:25:02.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-28T17:25:02.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e 2024-03-28T17:25:02.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-28T17:25:02.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k 2024-03-28T17:25:02.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-28T17:25:02.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e 2024-03-28T17:25:02.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-28T17:25:02.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf 2024-03-28T17:25:02.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-28T17:25:02.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb 2024-03-28T17:25:02.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-28T17:25:02.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf 2024-03-28T17:25:02.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-28T17:25:02.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-28T17:25:02.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-28T17:25:02.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-28T17:25:02.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-28T17:25:02.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-03-28T17:25:02.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell 2024-03-28T17:25:02.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-28T17:25:02.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-28T17:25:02.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-28T17:25:02.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox 2024-03-28T17:25:02.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-28T17:25:02.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-28T17:25:02.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-28T17:25:02.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-28T17:25:02.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-28T17:25:02.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-28T17:25:02.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel 2024-03-28T17:25:02.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-28T17:25:02.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom 2024-03-28T17:25:02.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-28T17:25:02.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-28T17:25:02.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi 2024-03-28T17:25:02.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-28T17:25:02.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-28T17:25:02.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion 2024-03-28T17:25:02.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-28T17:25:02.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia 2024-03-28T17:25:02.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-28T17:25:02.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines 2024-03-28T17:25:02.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-28T17:25:02.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-28T17:25:02.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic 2024-03-28T17:25:02.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-28T17:25:02.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-28T17:25:02.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-28T17:25:02.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-28T17:25:02.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-28T17:25:02.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc 2024-03-28T17:25:02.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-28T17:25:02.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek 2024-03-28T17:25:02.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-28T17:25:02.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-28T17:25:02.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-28T17:25:02.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-28T17:25:02.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker 2024-03-28T17:25:02.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-28T17:25:02.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc 2024-03-28T17:25:02.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-28T17:25:02.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan 2024-03-28T17:25:02.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-28T17:25:02.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis 2024-03-28T17:25:02.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-28T17:25:02.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-28T17:25:02.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc 2024-03-28T17:25:02.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-28T17:25:02.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-28T17:25:02.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-28T17:25:02.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-28T17:25:02.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro 2024-03-28T17:25:02.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-28T17:25:02.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-28T17:25:02.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-28T17:25:02.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun 2024-03-28T17:25:02.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-28T17:25:02.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-28T17:25:02.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-28T17:25:02.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-28T17:25:02.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti 2024-03-28T17:25:02.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-28T17:25:02.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti 2024-03-28T17:25:02.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-28T17:25:02.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via 2024-03-28T17:25:02.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-28T17:25:02.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-28T17:25:02.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet 2024-03-28T17:25:02.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-28T17:25:02.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-28T17:25:02.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom 2024-03-28T17:25:02.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-28T17:25:02.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-03-28T17:25:02.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv 2024-03-28T17:25:02.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-28T17:25:02.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154 2024-03-28T17:25:02.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-28T17:25:02.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-03-28T17:25:02.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan 2024-03-28T17:25:02.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-28T17:25:02.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-03-28T17:25:02.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-03-28T17:25:02.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio 2024-03-28T17:25:02.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-03-28T17:25:02.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-28T17:25:02.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-28T17:25:02.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-28T17:25:02.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-03-28T17:25:02.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/net_failover.ko 2024-03-28T17:25:02.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-03-28T17:25:02.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-03-28T17:25:02.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs 2024-03-28T17:25:02.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-28T17:25:02.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-28T17:25:02.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy 2024-03-28T17:25:02.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-03-28T17:25:02.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-28T17:25:02.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-28T17:25:02.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-28T17:25:02.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-03-28T17:25:02.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-03-28T17:25:02.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-03-28T17:25:02.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-03-28T17:25:02.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/dp83640.ko 2024-03-28T17:25:02.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-03-28T17:25:02.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-03-28T17:25:02.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-03-28T17:25:02.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-03-28T17:25:02.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-03-28T17:25:02.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-03-28T17:25:02.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-03-28T17:25:02.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-03-28T17:25:02.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-03-28T17:25:02.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-03-28T17:25:02.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-03-28T17:25:02.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-03-28T17:25:02.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp 2024-03-28T17:25:02.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-28T17:25:02.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-03-28T17:25:02.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-28T17:25:02.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-28T17:25:02.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-28T17:25:02.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-28T17:25:02.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-03-28T17:25:02.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-03-28T17:25:02.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-03-28T17:25:02.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip 2024-03-28T17:25:02.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slhc.ko 2024-03-28T17:25:02.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-03-28T17:25:02.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-03-28T17:25:02.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-03-28T17:25:02.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team 2024-03-28T17:25:02.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-03-28T17:25:02.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-28T17:25:02.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-28T17:25:02.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-28T17:25:02.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-03-28T17:25:02.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-28T17:25:02.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tun.ko 2024-03-28T17:25:02.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-03-28T17:25:02.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/virtio_net.ko 2024-03-28T17:25:02.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3 2024-03-28T17:25:02.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-28T17:25:02.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan 2024-03-28T17:25:02.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-03-28T17:25:02.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback 2024-03-28T17:25:02.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-28T17:25:02.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netfront.ko 2024-03-28T17:25:02.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb 2024-03-28T17:25:02.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-03-28T17:25:02.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme 2024-03-28T17:25:02.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host 2024-03-28T17:25:02.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-03-28T17:25:02.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-03-28T17:25:02.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport 2024-03-28T17:25:02.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-03-28T17:25:02.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-03-28T17:25:02.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-03-28T17:25:02.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-03-28T17:25:02.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci 2024-03-28T17:25:02.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug 2024-03-28T17:25:02.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-28T17:25:02.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie 2024-03-28T17:25:02.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-28T17:25:02.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-03-28T17:25:02.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia 2024-03-28T17:25:02.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-03-28T17:25:02.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-03-28T17:25:02.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-28T17:25:02.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl 2024-03-28T17:25:02.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel 2024-03-28T17:25:02.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-28T17:25:02.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform 2024-03-28T17:25:02.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86 2024-03-28T17:25:02.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-28T17:25:02.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-03-28T17:25:02.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-28T17:25:02.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-28T17:25:02.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-28T17:25:02.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-28T17:25:02.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-28T17:25:02.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-28T17:25:02.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-28T17:25:02.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-28T17:25:02.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-28T17:25:02.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-03-28T17:25:02.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel 2024-03-28T17:25:02.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-28T17:25:02.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-03-28T17:25:02.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-28T17:25:02.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-28T17:25:02.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-28T17:25:02.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-28T17:25:02.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-28T17:25:02.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-28T17:25:02.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-28T17:25:02.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-28T17:25:02.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-28T17:25:02.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-03-28T17:25:02.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap 2024-03-28T17:25:02.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-28T17:25:02.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-28T17:25:02.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps 2024-03-28T17:25:02.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients 2024-03-28T17:25:02.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-28T17:25:02.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-28T17:25:02.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-03-28T17:25:02.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-03-28T17:25:02.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp 2024-03-28T17:25:02.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-03-28T17:25:02.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-03-28T17:25:02.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc 2024-03-28T17:25:02.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-28T17:25:02.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-28T17:25:02.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-28T17:25:02.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-28T17:25:02.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-28T17:25:02.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-28T17:25:02.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-28T17:25:02.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-28T17:25:02.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-28T17:25:02.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-28T17:25:02.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-03-28T17:25:02.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-28T17:25:02.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-28T17:25:02.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-28T17:25:02.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-28T17:25:02.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-28T17:25:02.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-28T17:25:02.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-03-28T17:25:02.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-28T17:25:02.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-28T17:25:02.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-28T17:25:02.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-28T17:25:02.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-28T17:25:02.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-28T17:25:02.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-28T17:25:02.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-28T17:25:02.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-28T17:25:02.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-28T17:25:02.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-28T17:25:02.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-28T17:25:02.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-03-28T17:25:02.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi 2024-03-28T17:25:02.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-03-28T17:25:02.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-03-28T17:25:02.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-03-28T17:25:02.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-03-28T17:25:02.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-03-28T17:25:02.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid 2024-03-28T17:25:02.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-28T17:25:02.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-03-28T17:25:02.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx 2024-03-28T17:25:02.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-28T17:25:02.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-28T17:25:02.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-03-28T17:25:02.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr 2024-03-28T17:25:02.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-28T17:25:02.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-03-28T17:25:02.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi 2024-03-28T17:25:02.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-28T17:25:02.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa 2024-03-28T17:25:02.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-03-28T17:25:02.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc 2024-03-28T17:25:02.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-28T17:25:02.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i 2024-03-28T17:25:02.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-28T17:25:02.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-03-28T17:25:02.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor 2024-03-28T17:25:02.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-28T17:25:02.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi 2024-03-28T17:25:02.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-28T17:25:02.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-28T17:25:02.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-28T17:25:02.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-28T17:25:02.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-28T17:25:02.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-03-28T17:25:02.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler 2024-03-28T17:25:02.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-28T17:25:02.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-28T17:25:02.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-28T17:25:02.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-28T17:25:02.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-03-28T17:25:02.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r 2024-03-28T17:25:02.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-28T17:25:02.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-03-28T17:25:02.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe 2024-03-28T17:25:02.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-28T17:25:02.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-28T17:25:02.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic 2024-03-28T17:25:02.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-03-28T17:25:02.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-03-28T17:25:02.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-03-28T17:25:02.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-03-28T17:25:02.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-03-28T17:25:02.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-03-28T17:25:02.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-03-28T17:25:02.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci 2024-03-28T17:25:02.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-03-28T17:25:02.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-28T17:25:02.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-28T17:25:02.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc 2024-03-28T17:25:02.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-03-28T17:25:02.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-03-28T17:25:02.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-28T17:25:02.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas 2024-03-28T17:25:02.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-03-28T17:25:02.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc 2024-03-28T17:25:02.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-28T17:25:02.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid 2024-03-28T17:25:02.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-03-28T17:25:02.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-28T17:25:02.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-28T17:25:02.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-28T17:25:02.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas 2024-03-28T17:25:02.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-28T17:25:02.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas 2024-03-28T17:25:02.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-28T17:25:02.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-03-28T17:25:02.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001 2024-03-28T17:25:02.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-28T17:25:02.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-03-28T17:25:02.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-03-28T17:25:02.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx 2024-03-28T17:25:02.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-28T17:25:02.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-28T17:25:02.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx 2024-03-28T17:25:02.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-28T17:25:02.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-03-28T17:25:02.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-03-28T17:25:02.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-28T17:25:02.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-28T17:25:02.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-28T17:25:02.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-28T17:25:02.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-28T17:25:02.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-03-28T17:25:02.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-03-28T17:25:02.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-03-28T17:25:02.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2 2024-03-28T17:25:02.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-28T17:25:02.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-03-28T17:25:02.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-28T17:25:02.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-03-28T17:25:02.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb 2024-03-28T17:25:02.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-03-28T17:25:02.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target 2024-03-28T17:25:02.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi 2024-03-28T17:25:02.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-28T17:25:02.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback 2024-03-28T17:25:02.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-28T17:25:02.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-03-28T17:25:02.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-03-28T17:25:02.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-03-28T17:25:02.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-03-28T17:25:02.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-03-28T17:25:02.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc 2024-03-28T17:25:02.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-28T17:25:02.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal 2024-03-28T17:25:02.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel 2024-03-28T17:25:02.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal 2024-03-28T17:25:02.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-28T17:25:02.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-28T17:25:02.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-28T17:25:02.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-28T17:25:02.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-28T17:25:02.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-28T17:25:02.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-28T17:25:02.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-28T17:25:02.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-28T17:25:02.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-28T17:25:02.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-28T17:25:02.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-28T17:25:02.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-28T17:25:02.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-28T17:25:02.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-28T17:25:02.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-28T17:25:02.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-28T17:25:02.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-28T17:25:02.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty 2024-03-28T17:25:02.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless 2024-03-28T17:25:02.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-28T17:25:02.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-03-28T17:25:02.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-03-28T17:25:02.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-03-28T17:25:02.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial 2024-03-28T17:25:02.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250 2024-03-28T17:25:02.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-28T17:25:02.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-03-28T17:25:02.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm 2024-03-28T17:25:02.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-28T17:25:02.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs 2024-03-28T17:25:02.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core 2024-03-28T17:25:02.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-28T17:25:02.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host 2024-03-28T17:25:02.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-28T17:25:02.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio 2024-03-28T17:25:02.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-03-28T17:25:02.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-03-28T17:25:02.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-03-28T17:25:02.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-03-28T17:25:02.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-03-28T17:25:02.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb 2024-03-28T17:25:02.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class 2024-03-28T17:25:02.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-03-28T17:25:02.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-28T17:25:02.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-03-28T17:25:02.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-03-28T17:25:02.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host 2024-03-28T17:25:02.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-28T17:25:02.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image 2024-03-28T17:25:02.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-03-28T17:25:02.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-03-28T17:25:02.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc 2024-03-28T17:25:02.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-03-28T17:25:02.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-03-28T17:25:02.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-03-28T17:25:02.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-03-28T17:25:02.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-03-28T17:25:02.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-03-28T17:25:02.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-03-28T17:25:02.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-28T17:25:02.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-03-28T17:25:02.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-03-28T17:25:02.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga 2024-03-28T17:25:02.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-28T17:25:02.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-28T17:25:02.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-03-28T17:25:02.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-03-28T17:25:02.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-28T17:25:02.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-03-28T17:25:02.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-03-28T17:25:02.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy 2024-03-28T17:25:02.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-03-28T17:25:02.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial 2024-03-28T17:25:02.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-03-28T17:25:02.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-03-28T17:25:02.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-28T17:25:02.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-03-28T17:25:02.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-03-28T17:25:02.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-03-28T17:25:02.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-28T17:25:02.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-28T17:25:02.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-03-28T17:25:02.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-28T17:25:02.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-28T17:25:02.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-28T17:25:02.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-03-28T17:25:02.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-03-28T17:25:02.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-03-28T17:25:02.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-03-28T17:25:02.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-28T17:25:02.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-03-28T17:25:02.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-28T17:25:02.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-28T17:25:02.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-28T17:25:02.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-03-28T17:25:02.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-03-28T17:25:02.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-03-28T17:25:02.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-03-28T17:25:02.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-03-28T17:25:02.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-03-28T17:25:02.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-03-28T17:25:02.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-03-28T17:25:02.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-03-28T17:25:02.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-03-28T17:25:02.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-03-28T17:25:02.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-03-28T17:25:02.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-03-28T17:25:02.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-03-28T17:25:02.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-28T17:25:02.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-03-28T17:25:02.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-03-28T17:25:02.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-28T17:25:02.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-28T17:25:02.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-03-28T17:25:02.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-28T17:25:02.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-03-28T17:25:02.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-03-28T17:25:02.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-28T17:25:02.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage 2024-03-28T17:25:02.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-03-28T17:25:02.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-28T17:25:02.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-28T17:25:02.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-28T17:25:02.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-28T17:25:02.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-28T17:25:02.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-28T17:25:02.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-28T17:25:02.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-03-28T17:25:02.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-28T17:25:02.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-28T17:25:02.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-28T17:25:02.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-28T17:25:02.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-28T17:25:02.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-03-28T17:25:02.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip 2024-03-28T17:25:02.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-28T17:25:02.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-28T17:25:02.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-28T17:25:02.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio 2024-03-28T17:25:02.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci 2024-03-28T17:25:02.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-28T17:25:02.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-28T17:25:02.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-03-28T17:25:02.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-28T17:25:02.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost 2024-03-28T17:25:02.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-03-28T17:25:02.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-28T17:25:02.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-03-28T17:25:02.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-03-28T17:25:02.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video 2024-03-28T17:25:02.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight 2024-03-28T17:25:02.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-03-28T17:25:02.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-03-28T17:25:02.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-28T17:25:02.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio 2024-03-28T17:25:02.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio.ko 2024-03-28T17:25:02.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_balloon.ko 2024-03-28T17:25:02.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_mmio.ko 2024-03-28T17:25:02.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci.ko 2024-03-28T17:25:02.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-28T17:25:02.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-28T17:25:02.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_ring.ko 2024-03-28T17:25:02.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1 2024-03-28T17:25:02.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters 2024-03-28T17:25:02.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-03-28T17:25:02.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-03-28T17:25:02.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves 2024-03-28T17:25:02.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-28T17:25:02.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-28T17:25:02.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-28T17:25:02.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-28T17:25:02.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-28T17:25:02.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-28T17:25:02.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-28T17:25:02.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-28T17:25:02.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-28T17:25:02.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-28T17:25:02.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-28T17:25:02.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-03-28T17:25:02.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog 2024-03-28T17:25:02.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-28T17:25:02.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-28T17:25:02.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-28T17:25:02.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-03-28T17:25:02.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-03-28T17:25:02.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-28T17:25:02.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-28T17:25:02.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-03-28T17:25:02.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-03-28T17:25:02.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-28T17:25:02.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-28T17:25:02.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-03-28T17:25:02.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-03-28T17:25:02.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-03-28T17:25:02.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-28T17:25:02.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-28T17:25:02.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-28T17:25:02.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-28T17:25:02.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-03-28T17:25:02.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-28T17:25:02.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-03-28T17:25:02.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-28T17:25:02.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-28T17:25:02.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-28T17:25:02.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-03-28T17:25:02.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-03-28T17:25:02.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen 2024-03-28T17:25:02.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-28T17:25:02.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-03-28T17:25:02.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-03-28T17:25:02.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-03-28T17:25:02.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback 2024-03-28T17:25:02.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-28T17:25:02.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-03-28T17:25:02.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs 2024-03-28T17:25:02.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-28T17:25:02.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs 2024-03-28T17:25:02.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p 2024-03-28T17:25:02.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-03-28T17:25:02.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs 2024-03-28T17:25:02.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-03-28T17:25:02.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs 2024-03-28T17:25:02.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-03-28T17:25:02.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-03-28T17:25:02.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs 2024-03-28T17:25:02.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-03-28T17:25:02.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles 2024-03-28T17:25:02.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-03-28T17:25:02.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph 2024-03-28T17:25:02.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-03-28T17:25:02.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda 2024-03-28T17:25:02.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-03-28T17:25:02.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs 2024-03-28T17:25:02.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-03-28T17:25:02.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm 2024-03-28T17:25:02.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-03-28T17:25:02.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs 2024-03-28T17:25:02.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-28T17:25:02.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs 2024-03-28T17:25:02.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-03-28T17:25:02.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat 2024-03-28T17:25:02.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/fat.ko 2024-03-28T17:25:02.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-03-28T17:25:02.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/vfat.ko 2024-03-28T17:25:02.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse 2024-03-28T17:25:02.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-03-28T17:25:02.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/fuse.ko 2024-03-28T17:25:02.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2 2024-03-28T17:25:02.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-03-28T17:25:02.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs 2024-03-28T17:25:02.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-03-28T17:25:02.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus 2024-03-28T17:25:02.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-03-28T17:25:02.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs 2024-03-28T17:25:02.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-03-28T17:25:02.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs 2024-03-28T17:25:02.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-03-28T17:25:02.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd 2024-03-28T17:25:02.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-03-28T17:25:02.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix 2024-03-28T17:25:02.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-03-28T17:25:02.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs 2024-03-28T17:25:02.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-03-28T17:25:02.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs 2024-03-28T17:25:02.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout 2024-03-28T17:25:02.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-28T17:25:02.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout 2024-03-28T17:25:02.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-28T17:25:02.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout 2024-03-28T17:25:02.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-28T17:25:02.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-03-28T17:25:02.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-03-28T17:25:02.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-03-28T17:25:02.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common 2024-03-28T17:25:02.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-03-28T17:25:02.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-03-28T17:25:02.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd 2024-03-28T17:25:02.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-03-28T17:25:02.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2 2024-03-28T17:25:02.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-03-28T17:25:02.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls 2024-03-28T17:25:02.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-03-28T17:25:02.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-03-28T17:25:02.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-03-28T17:25:02.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-03-28T17:25:02.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-03-28T17:25:02.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-03-28T17:25:02.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-03-28T17:25:02.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-03-28T17:25:02.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-03-28T17:25:02.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-03-28T17:25:02.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-03-28T17:25:02.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-03-28T17:25:02.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-03-28T17:25:02.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-03-28T17:25:02.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-03-28T17:25:02.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-03-28T17:25:02.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-03-28T17:25:02.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-03-28T17:25:02.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-03-28T17:25:02.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-03-28T17:25:02.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-03-28T17:25:02.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-03-28T17:25:02.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-03-28T17:25:02.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-03-28T17:25:02.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-03-28T17:25:02.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-03-28T17:25:02.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-03-28T17:25:02.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-03-28T17:25:02.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-03-28T17:25:02.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-03-28T17:25:02.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-03-28T17:25:02.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-03-28T17:25:02.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-03-28T17:25:02.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-03-28T17:25:02.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-03-28T17:25:02.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-03-28T17:25:02.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-03-28T17:25:02.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-03-28T17:25:02.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-03-28T17:25:02.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-03-28T17:25:02.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-03-28T17:25:02.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-03-28T17:25:02.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-03-28T17:25:02.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-03-28T17:25:02.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-03-28T17:25:02.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-03-28T17:25:02.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-03-28T17:25:02.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-03-28T17:25:02.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-28T17:25:02.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-03-28T17:25:02.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2 2024-03-28T17:25:02.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster 2024-03-28T17:25:02.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-28T17:25:02.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm 2024-03-28T17:25:02.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-28T17:25:02.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs 2024-03-28T17:25:02.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-28T17:25:02.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-03-28T17:25:02.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-28T17:25:02.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-28T17:25:02.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-28T17:25:02.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs 2024-03-28T17:25:02.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-03-28T17:25:02.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore 2024-03-28T17:25:02.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-03-28T17:25:02.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs 2024-03-28T17:25:02.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-03-28T17:25:02.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs 2024-03-28T17:25:02.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-03-28T17:25:02.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb 2024-03-28T17:25:02.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client 2024-03-28T17:25:02.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-03-28T17:25:02.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common 2024-03-28T17:25:02.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-03-28T17:25:02.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-03-28T17:25:02.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs 2024-03-28T17:25:02.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-03-28T17:25:02.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv 2024-03-28T17:25:02.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-03-28T17:25:02.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf 2024-03-28T17:25:02.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-03-28T17:25:02.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs 2024-03-28T17:25:02.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-03-28T17:25:02.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs 2024-03-28T17:25:02.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-03-28T17:25:02.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel 2024-03-28T17:25:02.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu 2024-03-28T17:25:02.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu/rcutorture.ko 2024-03-28T17:25:02.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-03-28T17:25:02.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace 2024-03-28T17:25:02.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-28T17:25:02.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib 2024-03-28T17:25:02.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/asn1_encoder.ko 2024-03-28T17:25:02.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-03-28T17:25:02.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-03-28T17:25:02.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto 2024-03-28T17:25:02.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-03-28T17:25:02.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-03-28T17:25:02.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-03-28T17:25:02.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-03-28T17:25:02.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4 2024-03-28T17:25:02.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-03-28T17:25:02.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-03-28T17:25:02.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math 2024-03-28T17:25:02.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-03-28T17:25:02.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6 2024-03-28T17:25:02.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-03-28T17:25:02.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon 2024-03-28T17:25:02.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-28T17:25:02.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-03-28T17:25:02.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-03-28T17:25:02.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-03-28T17:25:02.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd 2024-03-28T17:25:02.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd/zstd_compress.ko 2024-03-28T17:25:02.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm 2024-03-28T17:25:02.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-03-28T17:25:02.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net 2024-03-28T17:25:02.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan 2024-03-28T17:25:02.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-03-28T17:25:02.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-03-28T17:25:02.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-03-28T17:25:02.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-03-28T17:25:02.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-28T17:25:02.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-03-28T17:25:02.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-03-28T17:25:02.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-03-28T17:25:02.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802 2024-03-28T17:25:02.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-03-28T17:25:02.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-03-28T17:25:02.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-03-28T17:25:02.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-03-28T17:25:02.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-03-28T17:25:02.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q 2024-03-28T17:25:02.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-03-28T17:25:02.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p 2024-03-28T17:25:02.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-03-28T17:25:02.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-03-28T17:25:02.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-03-28T17:25:02.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-03-28T17:25:02.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv 2024-03-28T17:25:02.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-03-28T17:25:02.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge 2024-03-28T17:25:02.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-03-28T17:25:02.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-03-28T17:25:02.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter 2024-03-28T17:25:02.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-28T17:25:02.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-28T17:25:02.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-28T17:25:02.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-28T17:25:02.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-28T17:25:02.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-28T17:25:02.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-28T17:25:02.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-28T17:25:02.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-28T17:25:02.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-28T17:25:02.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-28T17:25:02.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-28T17:25:02.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-28T17:25:02.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-28T17:25:02.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-28T17:25:02.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-28T17:25:02.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-28T17:25:02.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-28T17:25:02.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-28T17:25:02.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-28T17:25:02.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-03-28T17:25:02.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-28T17:25:02.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-28T17:25:02.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-28T17:25:02.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph 2024-03-28T17:25:02.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-03-28T17:25:02.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core 2024-03-28T17:25:02.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/failover.ko 2024-03-28T17:25:02.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-03-28T17:25:02.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp 2024-03-28T17:25:02.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-03-28T17:25:02.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-03-28T17:25:02.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-03-28T17:25:02.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-03-28T17:25:02.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver 2024-03-28T17:25:02.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver/dns_resolver.ko 2024-03-28T17:25:02.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa 2024-03-28T17:25:02.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-03-28T17:25:02.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-03-28T17:25:02.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-03-28T17:25:02.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-03-28T17:25:02.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-03-28T17:25:02.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154 2024-03-28T17:25:02.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan 2024-03-28T17:25:02.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-28T17:25:02.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-03-28T17:25:02.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-28T17:25:02.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4 2024-03-28T17:25:02.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-03-28T17:25:02.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-03-28T17:25:02.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-03-28T17:25:02.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-03-28T17:25:02.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-03-28T17:25:02.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-03-28T17:25:02.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-03-28T17:25:02.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-03-28T17:25:02.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-03-28T17:25:02.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-03-28T17:25:02.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter 2024-03-28T17:25:02.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-28T17:25:02.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-28T17:25:02.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-28T17:25:02.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-28T17:25:02.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-28T17:25:02.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-28T17:25:02.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-28T17:25:02.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-28T17:25:02.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-28T17:25:02.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-28T17:25:02.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-28T17:25:02.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-28T17:25:02.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-28T17:25:02.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-28T17:25:02.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-28T17:25:02.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-28T17:25:02.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-28T17:25:02.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-28T17:25:02.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-28T17:25:02.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-28T17:25:02.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-28T17:25:02.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-03-28T17:25:02.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-03-28T17:25:02.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-03-28T17:25:02.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-03-28T17:25:02.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-03-28T17:25:02.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-03-28T17:25:02.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-03-28T17:25:02.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-03-28T17:25:02.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-03-28T17:25:02.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-03-28T17:25:02.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-03-28T17:25:02.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-03-28T17:25:02.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-03-28T17:25:02.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-03-28T17:25:02.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-03-28T17:25:02.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-03-28T17:25:02.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-28T17:25:02.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6 2024-03-28T17:25:02.504 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-03-28T17:25:02.504 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-03-28T17:25:02.504 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-03-28T17:25:02.504 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-03-28T17:25:02.504 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-28T17:25:02.504 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-03-28T17:25:02.504 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-03-28T17:25:02.504 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter 2024-03-28T17:25:02.504 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-28T17:25:02.504 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-28T17:25:02.504 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-28T17:25:02.504 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-28T17:25:02.504 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-28T17:25:02.504 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-28T17:25:02.504 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-28T17:25:02.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-28T17:25:02.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-28T17:25:02.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-28T17:25:02.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-28T17:25:02.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-28T17:25:02.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-28T17:25:02.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-28T17:25:02.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-28T17:25:02.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-28T17:25:02.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-28T17:25:02.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-28T17:25:02.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-28T17:25:02.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-28T17:25:02.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-28T17:25:02.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-28T17:25:02.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-28T17:25:02.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-28T17:25:02.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-03-28T17:25:02.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-03-28T17:25:02.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-28T17:25:02.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key 2024-03-28T17:25:02.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-03-28T17:25:02.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp 2024-03-28T17:25:02.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-03-28T17:25:02.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-28T17:25:02.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-03-28T17:25:02.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-03-28T17:25:02.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-03-28T17:25:02.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-03-28T17:25:02.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-03-28T17:25:02.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc 2024-03-28T17:25:02.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-03-28T17:25:02.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154 2024-03-28T17:25:02.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-03-28T17:25:02.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls 2024-03-28T17:25:02.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-03-28T17:25:02.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter 2024-03-28T17:25:02.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset 2024-03-28T17:25:02.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-03-28T17:25:02.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-28T17:25:02.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-28T17:25:02.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-28T17:25:02.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-28T17:25:02.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-28T17:25:02.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-28T17:25:02.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-28T17:25:02.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-28T17:25:02.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-28T17:25:02.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-28T17:25:02.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-28T17:25:02.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-28T17:25:02.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-28T17:25:02.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-28T17:25:02.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-28T17:25:02.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs 2024-03-28T17:25:02.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-28T17:25:02.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-28T17:25:02.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-28T17:25:02.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-28T17:25:02.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-28T17:25:02.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-28T17:25:02.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-28T17:25:02.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-28T17:25:02.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-28T17:25:02.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-28T17:25:02.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-28T17:25:02.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-28T17:25:02.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-28T17:25:02.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-28T17:25:02.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-03-28T17:25:02.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-03-28T17:25:02.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-28T17:25:02.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-28T17:25:02.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-28T17:25:02.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-28T17:25:02.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-28T17:25:02.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-28T17:25:02.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-28T17:25:02.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-28T17:25:02.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-28T17:25:02.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-28T17:25:02.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-28T17:25:02.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-28T17:25:02.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-28T17:25:02.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-03-28T17:25:02.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-28T17:25:02.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-03-28T17:25:02.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-03-28T17:25:02.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-28T17:25:02.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-28T17:25:02.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-03-28T17:25:02.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-03-28T17:25:02.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-28T17:25:02.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-28T17:25:02.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-03-28T17:25:02.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-03-28T17:25:02.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-28T17:25:02.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-03-28T17:25:02.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-28T17:25:02.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-28T17:25:02.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-03-28T17:25:02.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-03-28T17:25:02.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-03-28T17:25:02.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-28T17:25:02.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-03-28T17:25:02.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-28T17:25:02.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-03-28T17:25:02.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-03-28T17:25:02.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-03-28T17:25:02.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-03-28T17:25:02.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-03-28T17:25:02.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-03-28T17:25:02.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-03-28T17:25:02.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_quota.ko 2024-03-28T17:25:02.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-03-28T17:25:02.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-03-28T17:25:02.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-03-28T17:25:02.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-03-28T17:25:02.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tunnel.ko 2024-03-28T17:25:02.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-03-28T17:25:02.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-28T17:25:02.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-28T17:25:02.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-28T17:25:02.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-03-28T17:25:02.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-03-28T17:25:02.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-03-28T17:25:02.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-03-28T17:25:02.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-28T17:25:02.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-03-28T17:25:02.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-03-28T17:25:02.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-28T17:25:02.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-03-28T17:25:02.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-03-28T17:25:02.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-28T17:25:02.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-03-28T17:25:02.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-28T17:25:02.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-03-28T17:25:02.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-28T17:25:02.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-28T17:25:02.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-03-28T17:25:02.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-03-28T17:25:02.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-03-28T17:25:02.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-03-28T17:25:02.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-03-28T17:25:02.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-03-28T17:25:02.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-03-28T17:25:02.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-03-28T17:25:02.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-03-28T17:25:02.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-03-28T17:25:02.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-03-28T17:25:02.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-03-28T17:25:02.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-03-28T17:25:02.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-03-28T17:25:02.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-03-28T17:25:02.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-03-28T17:25:02.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-03-28T17:25:02.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-03-28T17:25:02.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-03-28T17:25:02.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-03-28T17:25:02.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-03-28T17:25:02.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-03-28T17:25:02.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-03-28T17:25:02.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-03-28T17:25:02.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-03-28T17:25:02.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-03-28T17:25:02.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-03-28T17:25:02.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-03-28T17:25:02.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-03-28T17:25:02.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-03-28T17:25:02.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-03-28T17:25:02.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-03-28T17:25:02.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-03-28T17:25:02.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-03-28T17:25:02.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-03-28T17:25:02.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-03-28T17:25:02.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-03-28T17:25:02.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-03-28T17:25:02.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-03-28T17:25:02.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-03-28T17:25:02.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-03-28T17:25:02.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-03-28T17:25:02.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-03-28T17:25:02.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-03-28T17:25:02.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-03-28T17:25:02.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-03-28T17:25:02.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-03-28T17:25:02.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-03-28T17:25:02.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-03-28T17:25:02.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-03-28T17:25:02.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-03-28T17:25:02.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink 2024-03-28T17:25:02.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-03-28T17:25:02.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh 2024-03-28T17:25:02.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-03-28T17:25:02.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch 2024-03-28T17:25:02.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-03-28T17:25:02.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-03-28T17:25:02.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-03-28T17:25:02.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-03-28T17:25:02.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet 2024-03-28T17:25:02.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-03-28T17:25:02.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds 2024-03-28T17:25:02.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-03-28T17:25:02.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-03-28T17:25:02.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-03-28T17:25:02.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched 2024-03-28T17:25:02.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-03-28T17:25:02.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-03-28T17:25:02.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-03-28T17:25:02.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-03-28T17:25:02.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-03-28T17:25:02.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-03-28T17:25:02.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-03-28T17:25:02.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-03-28T17:25:02.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-03-28T17:25:02.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-03-28T17:25:02.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-03-28T17:25:02.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-03-28T17:25:02.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-03-28T17:25:02.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-03-28T17:25:02.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-03-28T17:25:02.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-03-28T17:25:02.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-03-28T17:25:02.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-03-28T17:25:02.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-03-28T17:25:02.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-03-28T17:25:02.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-03-28T17:25:02.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-03-28T17:25:02.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-03-28T17:25:02.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-03-28T17:25:02.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-03-28T17:25:02.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-03-28T17:25:02.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-03-28T17:25:02.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-03-28T17:25:02.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-03-28T17:25:02.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-03-28T17:25:02.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-03-28T17:25:02.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-03-28T17:25:02.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-03-28T17:25:02.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-03-28T17:25:02.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-03-28T17:25:02.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-03-28T17:25:02.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-03-28T17:25:02.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-03-28T17:25:02.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-03-28T17:25:02.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-03-28T17:25:02.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-03-28T17:25:02.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-03-28T17:25:02.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-03-28T17:25:02.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp 2024-03-28T17:25:02.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-03-28T17:25:02.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-03-28T17:25:02.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc 2024-03-28T17:25:02.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss 2024-03-28T17:25:02.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-28T17:25:02.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-28T17:25:02.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-03-28T17:25:02.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma 2024-03-28T17:25:02.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-28T17:25:02.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc 2024-03-28T17:25:02.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-03-28T17:25:02.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-03-28T17:25:02.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix 2024-03-28T17:25:02.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-03-28T17:25:02.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock 2024-03-28T17:25:02.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-28T17:25:02.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-28T17:25:02.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-03-28T17:25:02.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-28T17:25:02.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-28T17:25:02.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm 2024-03-28T17:25:02.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-28T17:25:02.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security 2024-03-28T17:25:02.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys 2024-03-28T17:25:02.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys 2024-03-28T17:25:02.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-28T17:25:02.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys 2024-03-28T17:25:02.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys/trusted.ko 2024-03-28T17:25:02.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt 2024-03-28T17:25:02.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib 2024-03-28T17:25:02.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-03-28T17:25:02.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-03-28T17:25:02.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-03-28T17:25:02.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-03-28T17:25:02.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.alias.bin 2024-03-28T17:25:02.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-03-28T17:25:02.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-03-28T17:25:02.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-03-28T17:25:02.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-03-28T17:25:02.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-03-28T17:25:02.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-03-28T17:25:02.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-03-28T17:25:02.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-03-28T17:25:02.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-03-28T17:25:02.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-28T17:25:02.528 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-03-28T17:25:02.528 DEBUG:teuthology.orchestra.run.smithi045:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-28T17:25:02.552 INFO:teuthology.orchestra.run.smithi045.stdout:postinstall scriptlet (using /bin/sh): 2024-03-28T17:25:02.552 INFO:teuthology.orchestra.run.smithi045.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-28T17:25:02.552 INFO:teuthology.orchestra.run.smithi045.stdout: /usr/bin/kernel-install add 6.8.0-g1df0c7e8a235 /lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-28T17:25:02.552 INFO:teuthology.orchestra.run.smithi045.stdout:fi 2024-03-28T17:25:02.552 INFO:teuthology.orchestra.run.smithi045.stdout:for file in vmlinuz System.map config; do 2024-03-28T17:25:02.552 INFO:teuthology.orchestra.run.smithi045.stdout: if ! cmp --silent "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235"; then 2024-03-28T17:25:02.552 INFO:teuthology.orchestra.run.smithi045.stdout: cp "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235" 2024-03-28T17:25:02.553 INFO:teuthology.orchestra.run.smithi045.stdout: fi 2024-03-28T17:25:02.553 INFO:teuthology.orchestra.run.smithi045.stdout:done 2024-03-28T17:25:02.553 INFO:teuthology.orchestra.run.smithi045.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-28T17:25:02.553 INFO:teuthology.orchestra.run.smithi045.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-28T17:25:02.553 INFO:teuthology.orchestra.run.smithi045.stdout:new-kernel-pkg --remove 6.8.0-g1df0c7e8a235 --rminitrd --initrdfile=/boot/initramfs-6.8.0-g1df0c7e8a235.img 2024-03-28T17:25:02.553 INFO:teuthology.orchestra.run.smithi045.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-28T17:25:02.553 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-install remove 6.8.0-g1df0c7e8a235 2024-03-28T17:25:02.553 INFO:teuthology.orchestra.run.smithi045.stdout:fi 2024-03-28T17:25:02.553 INFO:teuthology.orchestra.run.smithi045.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-28T17:25:02.553 INFO:teuthology.orchestra.run.smithi045.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-28T17:25:02.553 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/update-bootloader --remove 6.8.0-g1df0c7e8a235 2024-03-28T17:25:02.553 INFO:teuthology.orchestra.run.smithi045.stdout:fi 2024-03-28T17:25:02.554 DEBUG:teuthology.orchestra.run.smithi045:> sudo rpm -qi grub2-tools 2024-03-28T17:25:02.643 INFO:teuthology.orchestra.run.smithi045.stdout:Name : grub2-tools 2024-03-28T17:25:02.643 INFO:teuthology.orchestra.run.smithi045.stdout:Epoch : 1 2024-03-28T17:25:02.643 INFO:teuthology.orchestra.run.smithi045.stdout:Version : 2.06 2024-03-28T17:25:02.643 INFO:teuthology.orchestra.run.smithi045.stdout:Release : 68.el9 2024-03-28T17:25:02.643 INFO:teuthology.orchestra.run.smithi045.stdout:Architecture: x86_64 2024-03-28T17:25:02.643 INFO:teuthology.orchestra.run.smithi045.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-28T17:25:02.643 INFO:teuthology.orchestra.run.smithi045.stdout:Group : Unspecified 2024-03-28T17:25:02.643 INFO:teuthology.orchestra.run.smithi045.stdout:Size : 8264254 2024-03-28T17:25:02.644 INFO:teuthology.orchestra.run.smithi045.stdout:License : GPLv3+ 2024-03-28T17:25:02.644 INFO:teuthology.orchestra.run.smithi045.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-28T17:25:02.644 INFO:teuthology.orchestra.run.smithi045.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-28T17:25:02.644 INFO:teuthology.orchestra.run.smithi045.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-28T17:25:02.644 INFO:teuthology.orchestra.run.smithi045.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-28T17:25:02.644 INFO:teuthology.orchestra.run.smithi045.stdout:Packager : builder@centos.org 2024-03-28T17:25:02.644 INFO:teuthology.orchestra.run.smithi045.stdout:Vendor : CentOS 2024-03-28T17:25:02.644 INFO:teuthology.orchestra.run.smithi045.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-28T17:25:02.644 INFO:teuthology.orchestra.run.smithi045.stdout:Summary : Support tools for GRUB. 2024-03-28T17:25:02.644 INFO:teuthology.orchestra.run.smithi045.stdout:Description : 2024-03-28T17:25:02.644 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:25:02.644 INFO:teuthology.orchestra.run.smithi045.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-28T17:25:02.644 INFO:teuthology.orchestra.run.smithi045.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-28T17:25:02.644 INFO:teuthology.orchestra.run.smithi045.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-28T17:25:02.644 INFO:teuthology.orchestra.run.smithi045.stdout:hardware devices. 2024-03-28T17:25:02.645 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:25:02.645 INFO:teuthology.orchestra.run.smithi045.stdout:This subpackage provides tools for support of all platforms. 2024-03-28T17:25:02.645 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-28T17:25:02.645 INFO:teuthology.task.kernel:Updating grub on smithi045 to boot 6.8.0-g1df0c7e8a235 2024-03-28T17:25:02.646 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-28T17:25:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:Generating grub configuration file ... 2024-03-28T17:25:04.690 INFO:teuthology.orchestra.run.smithi045.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-28T17:25:04.718 INFO:teuthology.orchestra.run.smithi045.stderr:done 2024-03-28T17:25:04.721 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-03-28T17:25:04.736 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.212Y63M15X 2024-03-28T17:25:04.736 DEBUG:teuthology.orchestra.run.smithi045:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.212Y63M15X 2024-03-28T17:25:04.824 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.212Y63M15X 2024-03-28T17:25:04.927 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.212Y63M15X is 6KB 2024-03-28T17:25:04.948 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.212Y63M15X 2024-03-28T17:25:04.962 DEBUG:teuthology.orchestra.run.smithi045:> sudo /bin/ls /boot/loader/entries || true 2024-03-28T17:25:05.034 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-28T17:25:05.034 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-28T17:25:05.034 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-28T17:25:05.034 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g1df0c7e8a235.conf 2024-03-28T17:25:05.035 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g1df0c7e8a235 2024-03-28T17:25:05.209 DEBUG:teuthology.orchestra.run.smithi045:> sudo shutdown -r now 2024-03-28T17:25:35.212 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-28T17:25:35.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-28T17:25:35.213 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T17:25:53.570 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.45 2024-03-28T17:26:02.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-28T17:26:02.572 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T17:27:02.574 DEBUG:teuthology.orchestra.remote:timed out 2024-03-28T17:27:14.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-28T17:27:14.576 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T17:27:15.063 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-03-28T17:27:15.771 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-28T17:27:15.771 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-g1df0c7e8a235"... 2024-03-28T17:27:15.772 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-03-28T17:27:15.829 INFO:teuthology.orchestra.run.smithi045.stdout:6.8.0-g1df0c7e8a235 2024-03-28T17:27:15.829 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-03-28T17:27:15.829 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T17:27:15.829 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-28T17:27:16.830 DEBUG:teuthology.parallel:result is None 2024-03-28T17:27:16.830 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-28T17:27:16.838 INFO:teuthology.task.internal:Creating test directory... 2024-03-28T17:27:16.838 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T17:27:16.846 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T17:27:16.849 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T17:27:16.898 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-28T17:27:16.904 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-28T17:27:16.931 INFO:teuthology.task.internal:Creating archive directory... 2024-03-28T17:27:16.931 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T17:27:16.939 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T17:27:16.941 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T17:27:17.239 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-28T17:27:17.246 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-28T17:27:17.246 DEBUG:teuthology.orchestra.run.smithi071:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T17:27:17.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T17:27:17.262 DEBUG:teuthology.orchestra.run.smithi151:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T17:27:17.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T17:27:17.279 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T17:27:17.286 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T17:27:17.304 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T17:27:17.343 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T17:27:17.362 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T17:27:17.374 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T17:27:17.395 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T17:27:17.431 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T17:27:17.505 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T17:27:17.510 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-28T17:27:17.517 INFO:teuthology.task.internal:Configuring sudo... 2024-03-28T17:27:17.518 DEBUG:teuthology.orchestra.run.smithi045:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T17:27:17.525 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T17:27:17.527 DEBUG:teuthology.orchestra.run.smithi151:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T17:27:17.622 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-28T17:27:17.630 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-28T17:27:17.630 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T17:27:17.637 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T17:27:17.640 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T17:27:17.687 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T17:27:17.769 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T17:27:17.889 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T17:27:17.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T17:27:18.081 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-28T17:27:18.081 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T17:27:18.184 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T17:27:18.226 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T17:27:18.286 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T17:27:18.361 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T17:27:18.415 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-28T17:27:18.415 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T17:27:18.486 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T17:27:18.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T17:27:18.592 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T17:27:18.628 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T17:27:18.679 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-28T17:27:18.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T17:27:18.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo service rsyslog restart 2024-03-28T17:27:18.756 DEBUG:teuthology.orchestra.run.smithi071:> sudo service rsyslog restart 2024-03-28T17:27:18.759 DEBUG:teuthology.orchestra.run.smithi151:> sudo service rsyslog restart 2024-03-28T17:27:18.800 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T17:27:18.825 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T17:27:18.888 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T17:27:19.248 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-28T17:27:19.256 INFO:teuthology.task.internal:Starting timer... 2024-03-28T17:27:19.256 INFO:teuthology.run_tasks:Running task pcp... 2024-03-28T17:27:19.325 INFO:teuthology.run_tasks:Running task selinux... 2024-03-28T17:27:19.376 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-28T17:27:19.377 DEBUG:teuthology.orchestra.run.smithi045:> sudo service auditd rotate 2024-03-28T17:27:19.497 INFO:teuthology.orchestra.run.smithi045.stdout:Rotating logs: 2024-03-28T17:27:19.502 DEBUG:teuthology.orchestra.run.smithi071:> sudo service auditd rotate 2024-03-28T17:27:19.553 INFO:teuthology.orchestra.run.smithi071.stdout:Rotating logs: 2024-03-28T17:27:19.555 DEBUG:teuthology.orchestra.run.smithi151:> sudo service auditd rotate 2024-03-28T17:27:19.599 INFO:teuthology.orchestra.run.smithi151.stdout:Rotating logs: 2024-03-28T17:27:19.600 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-28T17:27:19.601 DEBUG:teuthology.orchestra.run.smithi045:> /usr/sbin/getenforce 2024-03-28T17:27:19.657 INFO:teuthology.orchestra.run.smithi045.stdout:Permissive 2024-03-28T17:27:19.657 DEBUG:teuthology.orchestra.run.smithi071:> /usr/sbin/getenforce 2024-03-28T17:27:19.680 INFO:teuthology.orchestra.run.smithi071.stdout:Permissive 2024-03-28T17:27:19.681 DEBUG:teuthology.orchestra.run.smithi151:> /usr/sbin/getenforce 2024-03-28T17:27:19.703 INFO:teuthology.orchestra.run.smithi151.stdout:Permissive 2024-03-28T17:27:19.703 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi045.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi071.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi151.front.sepia.ceph.com': 'permissive'} 2024-03-28T17:27:19.703 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T17:27:19.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T17:27:19.784 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T17:27:19.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T17:27:19.813 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T17:27:19.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T17:27:19.841 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-28T17:27:19.841 DEBUG:teuthology.orchestra.run.smithi045:> sudo /usr/sbin/setenforce permissive 2024-03-28T17:27:19.936 DEBUG:teuthology.orchestra.run.smithi071:> sudo /usr/sbin/setenforce permissive 2024-03-28T17:27:19.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo /usr/sbin/setenforce permissive 2024-03-28T17:27:20.003 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-28T17:27:20.012 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-28T17:27:20.029 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-28T17:27:20.030 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi045.front.sepia.ceph.com,smithi071.front.sepia.ceph.com,smithi151.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-28T17:33:42.010 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi045.front.sepia.ceph.com'), Remote(name='ubuntu@smithi071.front.sepia.ceph.com'), Remote(name='ubuntu@smithi151.front.sepia.ceph.com')] 2024-03-28T17:33:42.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-28T17:33:42.034 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T17:33:42.184 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-03-28T17:33:42.340 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-28T17:33:42.340 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-28T17:33:42.341 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T17:33:42.429 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-03-28T17:33:42.456 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-28T17:33:42.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-28T17:33:42.457 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T17:33:42.547 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-03-28T17:33:42.623 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-28T17:33:42.623 INFO:teuthology.run_tasks:Running task clock... 2024-03-28T17:33:42.671 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-28T17:33:42.672 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T17:33:42.672 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T17:33:42.684 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T17:33:42.684 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T17:33:42.691 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T17:33:42.692 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T17:33:42.723 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T17:33:42.735 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T17:33:42.740 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T17:33:42.754 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T17:33:42.771 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: ntpd: command not found 2024-03-28T17:33:42.784 INFO:teuthology.orchestra.run.smithi071.stdout:506 Cannot talk to daemon 2024-03-28T17:33:42.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: ntpd: command not found 2024-03-28T17:33:42.788 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T17:33:42.800 INFO:teuthology.orchestra.run.smithi151.stdout:506 Cannot talk to daemon 2024-03-28T17:33:42.802 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T17:33:42.829 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T17:33:42.830 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T17:33:42.834 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T17:33:42.843 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T17:33:42.861 INFO:teuthology.orchestra.run.smithi071.stderr:bash: line 1: ntpq: command not found 2024-03-28T17:33:42.865 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T17:33:42.865 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2024-03-28T17:33:42.865 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:33:42.865 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:33:42.865 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:33:42.865 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:33:42.877 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: ntpq: command not found 2024-03-28T17:33:42.880 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T17:33:42.880 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2024-03-28T17:33:42.880 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:33:42.880 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:33:42.880 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:33:42.880 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:33:42.939 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: ntpd: command not found 2024-03-28T17:33:42.980 INFO:teuthology.orchestra.run.smithi045.stdout:506 Cannot talk to daemon 2024-03-28T17:33:43.035 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T17:33:43.092 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T17:33:43.226 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2024-03-28T17:33:43.235 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T17:33:43.235 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2024-03-28T17:33:43.235 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:33:43.235 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:33:43.235 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:33:43.235 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T17:33:43.237 INFO:teuthology.run_tasks:Running task install... 2024-03-28T17:33:43.245 DEBUG:teuthology.task.install:project ceph 2024-03-28T17:33:43.246 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}} 2024-03-28T17:33:43.246 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'} 2024-03-28T17:33:43.246 INFO:teuthology.task.install:Using flavor: default 2024-03-28T17:33:43.276 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-28T17:33:43.277 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-28T17:33:43.277 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T17:33:43.277 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:33:43.279 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T17:33:43.279 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:33:43.280 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T17:33:43.280 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:33:43.431 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T17:33:43.431 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T17:33:43.434 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T17:33:43.434 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T17:33:43.494 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T17:33:43.494 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T17:33:43.543 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T17:33:43.543 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-28T17:33:43.543 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T17:33:43.573 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T17:33:43.573 DEBUG:teuthology.orchestra.run.smithi071:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T17:33:43.603 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T17:33:43.603 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-28T17:33:43.603 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T17:33:43.605 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T17:33:43.605 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-28T17:33:43.605 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T17:33:43.636 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T17:33:43.636 DEBUG:teuthology.orchestra.run.smithi151:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T17:33:43.642 DEBUG:teuthology.orchestra.run.smithi071:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T17:33:43.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T17:33:43.711 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T17:33:43.711 DEBUG:teuthology.orchestra.run.smithi045:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T17:33:43.723 DEBUG:teuthology.orchestra.run.smithi071:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T17:33:43.787 DEBUG:teuthology.orchestra.run.smithi151:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T17:33:43.789 INFO:teuthology.orchestra.run.smithi071.stdout:check_obsoletes = 1 2024-03-28T17:33:43.791 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum clean all 2024-03-28T17:33:43.854 INFO:teuthology.orchestra.run.smithi151.stdout:check_obsoletes = 1 2024-03-28T17:33:43.855 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum clean all 2024-03-28T17:33:43.857 DEBUG:teuthology.orchestra.run.smithi045:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T17:33:43.995 DEBUG:teuthology.orchestra.run.smithi045:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T17:33:44.024 INFO:teuthology.orchestra.run.smithi071.stdout:74 files removed 2024-03-28T17:33:44.050 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T17:33:44.090 INFO:teuthology.orchestra.run.smithi151.stdout:74 files removed 2024-03-28T17:33:44.093 INFO:teuthology.orchestra.run.smithi045.stdout:check_obsoletes = 1 2024-03-28T17:33:44.098 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum clean all 2024-03-28T17:33:44.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T17:33:44.464 INFO:teuthology.orchestra.run.smithi045.stdout:74 files removed 2024-03-28T17:33:44.504 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T17:33:44.806 INFO:teuthology.orchestra.run.smithi071.stdout:ceph packages for x86_64 178 kB/s | 83 kB 00:00 2024-03-28T17:33:44.879 INFO:teuthology.orchestra.run.smithi151.stdout:ceph packages for x86_64 176 kB/s | 83 kB 00:00 2024-03-28T17:33:45.165 INFO:teuthology.orchestra.run.smithi071.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-28T17:33:45.296 INFO:teuthology.orchestra.run.smithi151.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-28T17:33:45.499 INFO:teuthology.orchestra.run.smithi071.stdout:ceph source packages 6.2 kB/s | 1.9 kB 00:00 2024-03-28T17:33:45.656 INFO:teuthology.orchestra.run.smithi045.stdout:ceph packages for x86_64 133 kB/s | 83 kB 00:00 2024-03-28T17:33:45.714 INFO:teuthology.orchestra.run.smithi151.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-03-28T17:33:45.954 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-28T17:33:46.157 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-28T17:33:46.185 INFO:teuthology.orchestra.run.smithi045.stdout:ceph noarch packages 23 kB/s | 10 kB 00:00 2024-03-28T17:33:46.760 INFO:teuthology.orchestra.run.smithi045.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-03-28T17:33:47.422 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-28T17:33:47.527 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - BaseOS 4.3 MB/s | 2.0 MB 00:00 2024-03-28T17:33:47.557 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-28T17:33:49.323 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-03-28T17:33:50.392 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - BaseOS 3.2 MB/s | 8.0 MB 00:02 2024-03-28T17:33:52.573 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - BaseOS 1.7 MB/s | 8.0 MB 00:04 2024-03-28T17:33:52.737 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-03-28T17:33:54.390 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-03-28T17:33:57.402 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - CRB 5.1 MB/s | 6.1 MB 00:01 2024-03-28T17:33:58.744 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - Extras packages 78 kB/s | 16 kB 00:00 2024-03-28T17:33:58.751 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 913 kB/s | 8.0 MB 00:09 2024-03-28T17:33:59.824 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 6.1 MB 00:01 2024-03-28T17:34:00.685 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-03-28T17:34:01.079 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 9.3 MB/s | 21 MB 00:02 2024-03-28T17:34:01.492 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-28T17:34:02.412 INFO:teuthology.orchestra.run.smithi151.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-03-28T17:34:05.592 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 5.8 MB/s | 6.1 MB 00:01 2024-03-28T17:34:06.004 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-28T17:34:07.402 INFO:teuthology.orchestra.run.smithi151.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-28T17:34:08.212 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-28T17:34:08.370 INFO:teuthology.orchestra.run.smithi071.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T17:34:08.371 INFO:teuthology.orchestra.run.smithi071.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T17:34:08.385 INFO:teuthology.orchestra.run.smithi071.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T17:34:08.386 INFO:teuthology.orchestra.run.smithi071.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T17:34:08.387 INFO:teuthology.orchestra.run.smithi071.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T17:34:08.388 INFO:teuthology.orchestra.run.smithi071.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T17:34:08.390 INFO:teuthology.orchestra.run.smithi071.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T17:34:08.393 INFO:teuthology.orchestra.run.smithi071.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T17:34:08.395 INFO:teuthology.orchestra.run.smithi071.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T17:34:08.478 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-28T17:34:08.499 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2024-03-28T17:34:08.499 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-28T17:34:08.499 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2024-03-28T17:34:08.499 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-28T17:34:08.499 INFO:teuthology.orchestra.run.smithi071.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T17:34:08.499 INFO:teuthology.orchestra.run.smithi071.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T17:34:08.499 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T17:34:08.499 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T17:34:08.499 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T17:34:08.499 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T17:34:08.499 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T17:34:08.499 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T17:34:08.500 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T17:34:08.500 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T17:34:08.500 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T17:34:08.500 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T17:34:08.500 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T17:34:08.500 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T17:34:08.500 INFO:teuthology.orchestra.run.smithi071.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T17:34:08.500 INFO:teuthology.orchestra.run.smithi071.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T17:34:08.500 INFO:teuthology.orchestra.run.smithi071.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T17:34:08.500 INFO:teuthology.orchestra.run.smithi071.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T17:34:08.500 INFO:teuthology.orchestra.run.smithi071.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T17:34:08.500 INFO:teuthology.orchestra.run.smithi071.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T17:34:08.500 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T17:34:08.500 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T17:34:08.500 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T17:34:08.501 INFO:teuthology.orchestra.run.smithi071.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T17:34:08.501 INFO:teuthology.orchestra.run.smithi071.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T17:34:08.501 INFO:teuthology.orchestra.run.smithi071.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T17:34:08.501 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T17:34:08.501 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T17:34:08.501 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T17:34:08.501 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T17:34:08.501 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T17:34:08.501 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T17:34:08.501 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T17:34:08.501 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T17:34:08.501 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T17:34:08.501 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T17:34:08.501 INFO:teuthology.orchestra.run.smithi071.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T17:34:08.501 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrading: 2024-03-28T17:34:08.502 INFO:teuthology.orchestra.run.smithi071.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T17:34:08.502 INFO:teuthology.orchestra.run.smithi071.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T17:34:08.502 INFO:teuthology.orchestra.run.smithi071.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T17:34:08.502 INFO:teuthology.orchestra.run.smithi071.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T17:34:08.502 INFO:teuthology.orchestra.run.smithi071.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T17:34:08.502 INFO:teuthology.orchestra.run.smithi071.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T17:34:08.502 INFO:teuthology.orchestra.run.smithi071.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T17:34:08.502 INFO:teuthology.orchestra.run.smithi071.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T17:34:08.502 INFO:teuthology.orchestra.run.smithi071.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T17:34:08.502 INFO:teuthology.orchestra.run.smithi071.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T17:34:08.502 INFO:teuthology.orchestra.run.smithi071.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T17:34:08.502 INFO:teuthology.orchestra.run.smithi071.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T17:34:08.502 INFO:teuthology.orchestra.run.smithi071.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T17:34:08.502 INFO:teuthology.orchestra.run.smithi071.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T17:34:08.502 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T17:34:08.503 INFO:teuthology.orchestra.run.smithi071.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T17:34:08.503 INFO:teuthology.orchestra.run.smithi071.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T17:34:08.503 INFO:teuthology.orchestra.run.smithi071.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T17:34:08.503 INFO:teuthology.orchestra.run.smithi071.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T17:34:08.503 INFO:teuthology.orchestra.run.smithi071.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T17:34:08.503 INFO:teuthology.orchestra.run.smithi071.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T17:34:08.503 INFO:teuthology.orchestra.run.smithi071.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T17:34:08.503 INFO:teuthology.orchestra.run.smithi071.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T17:34:08.503 INFO:teuthology.orchestra.run.smithi071.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T17:34:08.503 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-28T17:34:08.503 INFO:teuthology.orchestra.run.smithi071.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T17:34:08.503 INFO:teuthology.orchestra.run.smithi071.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T17:34:08.503 INFO:teuthology.orchestra.run.smithi071.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T17:34:08.503 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T17:34:08.503 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T17:34:08.504 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T17:34:08.504 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T17:34:08.504 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T17:34:08.504 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T17:34:08.504 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T17:34:08.504 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T17:34:08.504 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T17:34:08.504 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T17:34:08.504 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T17:34:08.504 INFO:teuthology.orchestra.run.smithi071.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T17:34:08.504 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T17:34:08.504 INFO:teuthology.orchestra.run.smithi071.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T17:34:08.504 INFO:teuthology.orchestra.run.smithi071.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T17:34:08.504 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T17:34:08.504 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T17:34:08.505 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T17:34:08.505 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T17:34:08.505 INFO:teuthology.orchestra.run.smithi071.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T17:34:08.505 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T17:34:08.505 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T17:34:08.505 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T17:34:08.505 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T17:34:08.505 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T17:34:08.505 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T17:34:08.505 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T17:34:08.505 INFO:teuthology.orchestra.run.smithi071.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T17:34:08.505 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T17:34:08.505 INFO:teuthology.orchestra.run.smithi071.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T17:34:08.505 INFO:teuthology.orchestra.run.smithi071.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T17:34:08.506 INFO:teuthology.orchestra.run.smithi071.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T17:34:08.506 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T17:34:08.506 INFO:teuthology.orchestra.run.smithi071.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T17:34:08.506 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T17:34:08.506 INFO:teuthology.orchestra.run.smithi071.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T17:34:08.506 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T17:34:08.506 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T17:34:08.506 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T17:34:08.506 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T17:34:08.506 INFO:teuthology.orchestra.run.smithi071.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T17:34:08.506 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T17:34:08.506 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T17:34:08.506 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T17:34:08.507 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T17:34:08.507 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T17:34:08.507 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T17:34:08.507 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T17:34:08.507 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T17:34:08.507 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T17:34:08.507 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T17:34:08.507 INFO:teuthology.orchestra.run.smithi071.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T17:34:08.507 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T17:34:08.507 INFO:teuthology.orchestra.run.smithi071.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T17:34:08.507 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T17:34:08.507 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T17:34:08.507 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T17:34:08.507 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T17:34:08.507 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T17:34:08.507 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T17:34:08.508 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T17:34:08.508 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T17:34:08.508 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T17:34:08.508 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T17:34:08.508 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T17:34:08.508 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T17:34:08.508 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T17:34:08.508 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T17:34:08.508 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T17:34:08.508 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T17:34:08.508 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T17:34:08.508 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T17:34:08.508 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T17:34:08.508 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T17:34:08.508 INFO:teuthology.orchestra.run.smithi071.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T17:34:08.508 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T17:34:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T17:34:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T17:34:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T17:34:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T17:34:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T17:34:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T17:34:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T17:34:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T17:34:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T17:34:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T17:34:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T17:34:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T17:34:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T17:34:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T17:34:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T17:34:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T17:34:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T17:34:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T17:34:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T17:34:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T17:34:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T17:34:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T17:34:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T17:34:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T17:34:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T17:34:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T17:34:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T17:34:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T17:34:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T17:34:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T17:34:08.511 INFO:teuthology.orchestra.run.smithi071.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T17:34:08.511 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T17:34:08.511 INFO:teuthology.orchestra.run.smithi071.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T17:34:08.511 INFO:teuthology.orchestra.run.smithi071.stdout:Installing weak dependencies: 2024-03-28T17:34:08.511 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T17:34:08.511 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-28T17:34:08.511 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-28T17:34:08.511 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2024-03-28T17:34:08.511 INFO:teuthology.orchestra.run.smithi071.stdout:Install 150 Packages 2024-03-28T17:34:08.511 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrade 24 Packages 2024-03-28T17:34:08.511 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-28T17:34:08.512 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 225 M 2024-03-28T17:34:08.512 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-28T17:34:09.271 INFO:teuthology.orchestra.run.smithi071.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 40 kB/s | 6.6 kB 00:00 2024-03-28T17:34:09.496 INFO:teuthology.orchestra.run.smithi071.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 3.9 MB/s | 890 kB 00:00 2024-03-28T17:34:09.600 INFO:teuthology.orchestra.run.smithi071.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 1.4 MB/s | 145 kB 00:00 2024-03-28T17:34:09.752 INFO:teuthology.orchestra.run.smithi151.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T17:34:09.753 INFO:teuthology.orchestra.run.smithi151.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T17:34:09.767 INFO:teuthology.orchestra.run.smithi151.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T17:34:09.768 INFO:teuthology.orchestra.run.smithi151.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T17:34:09.769 INFO:teuthology.orchestra.run.smithi151.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T17:34:09.770 INFO:teuthology.orchestra.run.smithi151.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T17:34:09.772 INFO:teuthology.orchestra.run.smithi151.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T17:34:09.775 INFO:teuthology.orchestra.run.smithi151.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T17:34:09.776 INFO:teuthology.orchestra.run.smithi151.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T17:34:09.856 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-28T17:34:09.875 INFO:teuthology.orchestra.run.smithi071.stdout:(4/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 7.0 MB/s | 5.4 MB 00:00 2024-03-28T17:34:09.877 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================================ 2024-03-28T17:34:09.877 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-28T17:34:09.877 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================================ 2024-03-28T17:34:09.877 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-28T17:34:09.877 INFO:teuthology.orchestra.run.smithi151.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T17:34:09.877 INFO:teuthology.orchestra.run.smithi151.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T17:34:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T17:34:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T17:34:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T17:34:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T17:34:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T17:34:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T17:34:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T17:34:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T17:34:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T17:34:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T17:34:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T17:34:09.879 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T17:34:09.879 INFO:teuthology.orchestra.run.smithi151.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T17:34:09.879 INFO:teuthology.orchestra.run.smithi151.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T17:34:09.879 INFO:teuthology.orchestra.run.smithi151.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T17:34:09.879 INFO:teuthology.orchestra.run.smithi151.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T17:34:09.879 INFO:teuthology.orchestra.run.smithi151.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T17:34:09.879 INFO:teuthology.orchestra.run.smithi151.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T17:34:09.879 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T17:34:09.879 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T17:34:09.879 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T17:34:09.879 INFO:teuthology.orchestra.run.smithi151.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T17:34:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T17:34:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T17:34:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T17:34:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T17:34:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T17:34:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T17:34:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T17:34:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T17:34:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T17:34:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T17:34:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T17:34:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T17:34:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T17:34:09.881 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrading: 2024-03-28T17:34:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T17:34:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T17:34:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T17:34:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T17:34:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T17:34:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T17:34:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T17:34:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T17:34:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T17:34:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T17:34:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T17:34:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T17:34:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T17:34:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T17:34:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T17:34:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T17:34:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T17:34:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T17:34:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T17:34:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T17:34:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T17:34:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T17:34:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T17:34:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T17:34:09.883 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-28T17:34:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T17:34:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T17:34:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T17:34:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T17:34:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T17:34:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T17:34:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T17:34:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T17:34:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T17:34:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T17:34:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T17:34:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T17:34:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T17:34:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T17:34:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T17:34:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T17:34:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T17:34:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T17:34:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T17:34:09.885 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T17:34:09.885 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T17:34:09.885 INFO:teuthology.orchestra.run.smithi151.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T17:34:09.885 INFO:teuthology.orchestra.run.smithi151.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T17:34:09.885 INFO:teuthology.orchestra.run.smithi151.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T17:34:09.885 INFO:teuthology.orchestra.run.smithi151.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T17:34:09.885 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T17:34:09.885 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T17:34:09.885 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T17:34:09.885 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T17:34:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T17:34:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T17:34:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T17:34:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T17:34:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T17:34:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T17:34:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T17:34:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T17:34:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T17:34:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T17:34:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T17:34:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T17:34:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T17:34:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T17:34:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T17:34:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T17:34:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T17:34:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T17:34:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T17:34:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T17:34:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T17:34:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T17:34:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T17:34:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T17:34:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T17:34:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T17:34:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T17:34:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T17:34:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T17:34:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T17:34:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T17:34:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T17:34:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T17:34:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T17:34:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T17:34:09.889 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T17:34:09.889 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T17:34:09.889 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T17:34:09.889 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T17:34:09.889 INFO:teuthology.orchestra.run.smithi151.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T17:34:09.889 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T17:34:09.889 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T17:34:09.889 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T17:34:09.889 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T17:34:09.889 INFO:teuthology.orchestra.run.smithi151.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T17:34:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T17:34:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T17:34:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T17:34:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T17:34:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T17:34:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T17:34:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T17:34:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T17:34:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T17:34:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T17:34:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T17:34:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T17:34:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T17:34:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T17:34:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T17:34:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T17:34:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T17:34:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T17:34:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T17:34:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T17:34:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T17:34:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T17:34:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T17:34:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T17:34:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T17:34:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T17:34:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T17:34:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T17:34:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T17:34:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T17:34:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T17:34:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T17:34:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T17:34:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T17:34:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T17:34:09.893 INFO:teuthology.orchestra.run.smithi151.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T17:34:09.893 INFO:teuthology.orchestra.run.smithi151.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T17:34:09.893 INFO:teuthology.orchestra.run.smithi151.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T17:34:09.893 INFO:teuthology.orchestra.run.smithi151.stdout:Installing weak dependencies: 2024-03-28T17:34:09.893 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T17:34:09.893 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-28T17:34:09.893 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-28T17:34:09.893 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================================ 2024-03-28T17:34:09.893 INFO:teuthology.orchestra.run.smithi151.stdout:Install 150 Packages 2024-03-28T17:34:09.893 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrade 24 Packages 2024-03-28T17:34:09.894 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-28T17:34:09.894 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 225 M 2024-03-28T17:34:09.894 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-28T17:34:10.022 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-28T17:34:10.092 INFO:teuthology.orchestra.run.smithi071.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 6.9 MB/s | 1.5 MB 00:00 2024-03-28T17:34:10.143 INFO:teuthology.orchestra.run.smithi071.stdout:(6/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 4.3 MB/s | 2.3 MB 00:00 2024-03-28T17:34:10.635 INFO:teuthology.orchestra.run.smithi071.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 8.7 MB/s | 4.7 MB 00:00 2024-03-28T17:34:10.694 INFO:teuthology.orchestra.run.smithi151.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-03-28T17:34:10.851 INFO:teuthology.orchestra.run.smithi071.stdout:(8/174): ceph-common-19.0.0-2145.g3be9a5c9.el9. 12 MB/s | 20 MB 00:01 2024-03-28T17:34:10.952 INFO:teuthology.orchestra.run.smithi071.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 252 kB/s | 25 kB 00:00 2024-03-28T17:34:11.161 INFO:teuthology.orchestra.run.smithi151.stdout:(2/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 7.0 MB/s | 5.4 MB 00:00 2024-03-28T17:34:11.244 INFO:teuthology.orchestra.run.smithi151.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 1.7 MB/s | 145 kB 00:00 2024-03-28T17:34:11.645 INFO:teuthology.orchestra.run.smithi151.stdout:(4/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 5.8 MB/s | 2.3 MB 00:00 2024-03-28T17:34:11.670 INFO:teuthology.orchestra.run.smithi151.stdout:(5/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 913 kB/s | 890 kB 00:00 2024-03-28T17:34:11.895 INFO:teuthology.orchestra.run.smithi151.stdout:(6/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 6.0 MB/s | 1.5 MB 00:00 2024-03-28T17:34:12.435 INFO:teuthology.orchestra.run.smithi071.stdout:(10/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el 5.5 MB/s | 9.9 MB 00:01 2024-03-28T17:34:12.535 INFO:teuthology.orchestra.run.smithi071.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 2.2 MB/s | 220 kB 00:00 2024-03-28T17:34:12.619 INFO:teuthology.orchestra.run.smithi071.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 325 kB/s | 27 kB 00:00 2024-03-28T17:34:12.711 INFO:teuthology.orchestra.run.smithi071.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 341 kB/s | 31 kB 00:00 2024-03-28T17:34:12.894 INFO:teuthology.orchestra.run.smithi071.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 4.0 MB/s | 746 kB 00:00 2024-03-28T17:34:12.995 INFO:teuthology.orchestra.run.smithi071.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.6 MB/s | 163 kB 00:00 2024-03-28T17:34:13.120 INFO:teuthology.orchestra.run.smithi071.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.0 MB/s | 127 kB 00:00 2024-03-28T17:34:13.745 INFO:teuthology.orchestra.run.smithi071.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 802 kB/s | 501 kB 00:00 2024-03-28T17:34:13.953 INFO:teuthology.orchestra.run.smithi151.stdout:(7/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x86 8.2 MB/s | 17 MB 00:02 2024-03-28T17:34:14.229 INFO:teuthology.orchestra.run.smithi151.stdout:(8/174): ceph-common-19.0.0-2145.g3be9a5c9.el9. 5.3 MB/s | 20 MB 00:03 2024-03-28T17:34:14.320 INFO:teuthology.orchestra.run.smithi151.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 276 kB/s | 25 kB 00:00 2024-03-28T17:34:14.579 INFO:teuthology.orchestra.run.smithi071.stdout:(18/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 6.2 MB/s | 5.1 MB 00:00 2024-03-28T17:34:14.654 INFO:teuthology.orchestra.run.smithi071.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 603 kB/s | 45 kB 00:00 2024-03-28T17:34:14.737 INFO:teuthology.orchestra.run.smithi071.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.5 MB/s | 131 kB 00:00 2024-03-28T17:34:14.830 INFO:teuthology.orchestra.run.smithi071.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 1.7 MB/s | 162 kB 00:00 2024-03-28T17:34:14.921 INFO:teuthology.orchestra.run.smithi071.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 3.4 MB/s | 321 kB 00:00 2024-03-28T17:34:15.022 INFO:teuthology.orchestra.run.smithi071.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 2.9 MB/s | 298 kB 00:00 2024-03-28T17:34:15.105 INFO:teuthology.orchestra.run.smithi071.stdout:(24/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.2 MB/s | 99 kB 00:00 2024-03-28T17:34:15.189 INFO:teuthology.orchestra.run.smithi071.stdout:(25/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.0 MB/s | 85 kB 00:00 2024-03-28T17:34:15.412 INFO:teuthology.orchestra.run.smithi151.stdout:(10/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el 6.8 MB/s | 9.9 MB 00:01 2024-03-28T17:34:15.479 INFO:teuthology.orchestra.run.smithi151.stdout:(11/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x8 1.2 MB/s | 4.7 MB 00:03 2024-03-28T17:34:15.504 INFO:teuthology.orchestra.run.smithi151.stdout:(12/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 2.4 MB/s | 220 kB 00:00 2024-03-28T17:34:15.580 INFO:teuthology.orchestra.run.smithi151.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 417 kB/s | 31 kB 00:00 2024-03-28T17:34:15.638 INFO:teuthology.orchestra.run.smithi151.stdout:(14/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 170 kB/s | 27 kB 00:00 2024-03-28T17:34:15.673 INFO:teuthology.orchestra.run.smithi071.stdout:(26/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 6.4 MB/s | 3.1 MB 00:00 2024-03-28T17:34:15.730 INFO:teuthology.orchestra.run.smithi151.stdout:(15/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 4.8 MB/s | 746 kB 00:00 2024-03-28T17:34:15.765 INFO:teuthology.orchestra.run.smithi071.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-28T17:34:15.798 INFO:teuthology.orchestra.run.smithi151.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.9 MB/s | 127 kB 00:00 2024-03-28T17:34:15.822 INFO:teuthology.orchestra.run.smithi151.stdout:(17/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 887 kB/s | 163 kB 00:00 2024-03-28T17:34:15.831 INFO:teuthology.orchestra.run.smithi071.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 509 kB/s | 34 kB 00:00 2024-03-28T17:34:15.923 INFO:teuthology.orchestra.run.smithi151.stdout:(18/174): libradosstriper1-19.0.0-2145.g3be9a5c 3.9 MB/s | 501 kB 00:00 2024-03-28T17:34:15.955 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-03-28T17:34:15.990 INFO:teuthology.orchestra.run.smithi151.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 678 kB/s | 45 kB 00:00 2024-03-28T17:34:16.057 INFO:teuthology.orchestra.run.smithi151.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.9 MB/s | 131 kB 00:00 2024-03-28T17:34:16.057 INFO:teuthology.orchestra.run.smithi071.stdout:(29/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 2.8 MB/s | 17 MB 00:05 2024-03-28T17:34:16.082 INFO:teuthology.orchestra.run.smithi071.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 554 kB/s | 139 kB 00:00 2024-03-28T17:34:16.115 INFO:teuthology.orchestra.run.smithi151.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 2.7 MB/s | 162 kB 00:00 2024-03-28T17:34:16.207 INFO:teuthology.orchestra.run.smithi151.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 3.4 MB/s | 321 kB 00:00 2024-03-28T17:34:16.299 INFO:teuthology.orchestra.run.smithi151.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 3.2 MB/s | 298 kB 00:00 2024-03-28T17:34:16.382 INFO:teuthology.orchestra.run.smithi151.stdout:(24/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.2 MB/s | 99 kB 00:00 2024-03-28T17:34:16.449 INFO:teuthology.orchestra.run.smithi151.stdout:(25/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.3 MB/s | 85 kB 00:00 2024-03-28T17:34:16.816 INFO:teuthology.orchestra.run.smithi151.stdout:(26/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 8.4 MB/s | 3.1 MB 00:00 2024-03-28T17:34:16.866 INFO:teuthology.orchestra.run.smithi071.stdout:(31/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 8.1 MB/s | 48 MB 00:05 2024-03-28T17:34:16.892 INFO:teuthology.orchestra.run.smithi151.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 2.2 MB/s | 170 kB 00:00 2024-03-28T17:34:16.916 INFO:teuthology.orchestra.run.smithi071.stdout:(32/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 2.1 MB/s | 1.8 MB 00:00 2024-03-28T17:34:16.950 INFO:teuthology.orchestra.run.smithi071.stdout:(33/174): ceph-mgr-modules-core-19.0.0-2145.g3b 2.9 MB/s | 248 kB 00:00 2024-03-28T17:34:16.950 INFO:teuthology.orchestra.run.smithi151.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 583 kB/s | 34 kB 00:00 2024-03-28T17:34:16.975 INFO:teuthology.orchestra.run.smithi071.stdout:(34/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 844 kB/s | 49 kB 00:00 2024-03-28T17:34:17.017 INFO:teuthology.orchestra.run.smithi071.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2145.g3 246 kB/s | 16 kB 00:00 2024-03-28T17:34:17.025 INFO:teuthology.orchestra.run.smithi151.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 1.8 MB/s | 139 kB 00:00 2024-03-28T17:34:17.076 INFO:teuthology.orchestra.run.smithi071.stdout:(36/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 2.6 MB/s | 264 kB 00:00 2024-03-28T17:34:17.159 INFO:teuthology.orchestra.run.smithi071.stdout:(37/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 5.2 MB/s | 748 kB 00:00 2024-03-28T17:34:17.184 INFO:teuthology.orchestra.run.smithi071.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 168 kB/s | 18 kB 00:00 2024-03-28T17:34:17.259 INFO:teuthology.orchestra.run.smithi151.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 7.7 MB/s | 1.8 MB 00:00 2024-03-28T17:34:17.268 INFO:teuthology.orchestra.run.smithi071.stdout:(39/174): ceph-mgr-diskprediction-local-19.0.0- 6.2 MB/s | 7.4 MB 00:01 2024-03-28T17:34:17.427 INFO:teuthology.orchestra.run.smithi071.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 987 kB/s | 106 kB 00:00 2024-03-28T17:34:17.460 INFO:teuthology.orchestra.run.smithi071.stdout:(41/174): automake-1.16.2-8.el9.noarch.rpm 2.2 MB/s | 662 kB 00:00 2024-03-28T17:34:17.494 INFO:teuthology.orchestra.run.smithi071.stdout:(42/174): bison-3.7.4-5.el9.x86_64.rpm 2.9 MB/s | 920 kB 00:00 2024-03-28T17:34:17.519 INFO:teuthology.orchestra.run.smithi071.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 3.3 MB/s | 307 kB 00:00 2024-03-28T17:34:17.544 INFO:teuthology.orchestra.run.smithi071.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 638 kB/s | 32 kB 00:00 2024-03-28T17:34:17.611 INFO:teuthology.orchestra.run.smithi071.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-03-28T17:34:17.645 INFO:teuthology.orchestra.run.smithi071.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 138 kB/s | 17 kB 00:00 2024-03-28T17:34:17.703 INFO:teuthology.orchestra.run.smithi071.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 64 kB/s | 10 kB 00:00 2024-03-28T17:34:17.729 INFO:teuthology.orchestra.run.smithi071.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-03-28T17:34:17.762 INFO:teuthology.orchestra.run.smithi071.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 756 kB/s | 44 kB 00:00 2024-03-28T17:34:17.788 INFO:teuthology.orchestra.run.smithi071.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-28T17:34:17.821 INFO:teuthology.orchestra.run.smithi071.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 6.2 MB/s | 578 kB 00:00 2024-03-28T17:34:17.847 INFO:teuthology.orchestra.run.smithi071.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 803 kB/s | 46 kB 00:00 2024-03-28T17:34:17.851 INFO:teuthology.orchestra.run.smithi151.stdout:(31/174): ceph-mgr-diskprediction-local-19.0.0- 13 MB/s | 7.4 MB 00:00 2024-03-28T17:34:17.881 INFO:teuthology.orchestra.run.smithi071.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-28T17:34:17.906 INFO:teuthology.orchestra.run.smithi071.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 187 kB 00:00 2024-03-28T17:34:17.910 INFO:teuthology.orchestra.run.smithi151.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 4.2 MB/s | 248 kB 00:00 2024-03-28T17:34:17.931 INFO:teuthology.orchestra.run.smithi071.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 683 kB/s | 34 kB 00:00 2024-03-28T17:34:17.957 INFO:teuthology.orchestra.run.smithi071.stdout:(56/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-28T17:34:17.960 INFO:teuthology.orchestra.run.smithi151.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 991 kB/s | 49 kB 00:00 2024-03-28T17:34:18.040 INFO:teuthology.orchestra.run.smithi151.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 207 kB/s | 16 kB 00:00 2024-03-28T17:34:18.041 INFO:teuthology.orchestra.run.smithi071.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 35 MB/s | 4.7 MB 00:00 2024-03-28T17:34:18.074 INFO:teuthology.orchestra.run.smithi071.stdout:(58/174): protobuf-3.14.0-13.el9.x86_64.rpm 7.3 MB/s | 1.0 MB 00:00 2024-03-28T17:34:18.099 INFO:teuthology.orchestra.run.smithi071.stdout:(59/174): python3-attrs-20.3.0-7.el9.noarch.rpm 558 kB/s | 78 kB 00:00 2024-03-28T17:34:18.102 INFO:teuthology.orchestra.run.smithi151.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 4.2 MB/s | 264 kB 00:00 2024-03-28T17:34:18.202 INFO:teuthology.orchestra.run.smithi151.stdout:(36/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 7.3 MB/s | 748 kB 00:00 2024-03-28T17:34:18.209 INFO:teuthology.orchestra.run.smithi071.stdout:(60/174): python3-babel-2.9.1-2.el9.noarch.rpm 35 MB/s | 5.8 MB 00:00 2024-03-28T17:34:18.233 INFO:teuthology.orchestra.run.smithi071.stdout:(61/174): python3-jsonpatch-1.21-16.el9.noarch. 186 kB/s | 24 kB 00:00 2024-03-28T17:34:18.258 INFO:teuthology.orchestra.run.smithi071.stdout:(62/174): python3-jsonpointer-2.0-4.el9.noarch. 116 kB/s | 18 kB 00:00 2024-03-28T17:34:18.269 INFO:teuthology.orchestra.run.smithi151.stdout:(37/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 2.1 MB/s | 5.1 MB 00:02 2024-03-28T17:34:18.283 INFO:teuthology.orchestra.run.smithi071.stdout:(63/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-28T17:34:18.294 INFO:teuthology.orchestra.run.smithi151.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 198 kB/s | 18 kB 00:00 2024-03-28T17:34:18.309 INFO:teuthology.orchestra.run.smithi071.stdout:(64/174): python3-markupsafe-1.1.1-12.el9.x86_6 631 kB/s | 32 kB 00:00 2024-03-28T17:34:18.401 INFO:teuthology.orchestra.run.smithi071.stdout:(65/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-28T17:34:18.426 INFO:teuthology.orchestra.run.smithi071.stdout:(66/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-03-28T17:34:18.451 INFO:teuthology.orchestra.run.smithi071.stdout:(67/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-03-28T17:34:18.453 INFO:teuthology.orchestra.run.smithi151.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.5 MB/s | 662 kB 00:00 2024-03-28T17:34:18.477 INFO:teuthology.orchestra.run.smithi071.stdout:(68/174): python3-packaging-20.9-5.el9.noarch.r 918 kB/s | 69 kB 00:00 2024-03-28T17:34:18.495 INFO:teuthology.orchestra.run.smithi151.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.5 MB/s | 920 kB 00:00 2024-03-28T17:34:18.502 INFO:teuthology.orchestra.run.smithi071.stdout:(69/174): python3-prettytable-0.7.2-27.el9.noar 822 kB/s | 41 kB 00:00 2024-03-28T17:34:18.520 INFO:teuthology.orchestra.run.smithi151.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-28T17:34:18.527 INFO:teuthology.orchestra.run.smithi071.stdout:(70/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-28T17:34:18.545 INFO:teuthology.orchestra.run.smithi151.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-28T17:34:18.552 INFO:teuthology.orchestra.run.smithi071.stdout:(71/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-28T17:34:18.570 INFO:teuthology.orchestra.run.smithi151.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 637 kB/s | 32 kB 00:00 2024-03-28T17:34:18.577 INFO:teuthology.orchestra.run.smithi071.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 949 kB/s | 47 kB 00:00 2024-03-28T17:34:18.603 INFO:teuthology.orchestra.run.smithi071.stdout:(73/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-28T17:34:18.628 INFO:teuthology.orchestra.run.smithi071.stdout:(74/174): python3-requests-oauthlib-1.3.0-12.el 567 kB/s | 43 kB 00:00 2024-03-28T17:34:18.661 INFO:teuthology.orchestra.run.smithi045.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T17:34:18.663 INFO:teuthology.orchestra.run.smithi151.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-03-28T17:34:18.663 INFO:teuthology.orchestra.run.smithi045.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T17:34:18.684 INFO:teuthology.orchestra.run.smithi045.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T17:34:18.685 INFO:teuthology.orchestra.run.smithi045.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T17:34:18.687 INFO:teuthology.orchestra.run.smithi045.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T17:34:18.688 INFO:teuthology.orchestra.run.smithi045.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T17:34:18.688 INFO:teuthology.orchestra.run.smithi151.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 148 kB/s | 17 kB 00:00 2024-03-28T17:34:18.691 INFO:teuthology.orchestra.run.smithi045.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T17:34:18.694 INFO:teuthology.orchestra.run.smithi045.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T17:34:18.696 INFO:teuthology.orchestra.run.smithi045.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T17:34:18.713 INFO:teuthology.orchestra.run.smithi151.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 200 kB/s | 10 kB 00:00 2024-03-28T17:34:18.747 INFO:teuthology.orchestra.run.smithi151.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-28T17:34:18.780 INFO:teuthology.orchestra.run.smithi151.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 673 kB/s | 44 kB 00:00 2024-03-28T17:34:18.791 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-28T17:34:18.795 INFO:teuthology.orchestra.run.smithi071.stdout:(75/174): python3-toml-0.10.2-6.el9.noarch.rpm 235 kB/s | 39 kB 00:00 2024-03-28T17:34:18.805 INFO:teuthology.orchestra.run.smithi151.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-28T17:34:18.820 INFO:teuthology.orchestra.run.smithi071.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 216 kB/s | 41 kB 00:00 2024-03-28T17:34:18.826 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2024-03-28T17:34:18.826 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-28T17:34:18.826 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2024-03-28T17:34:18.827 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-28T17:34:18.827 INFO:teuthology.orchestra.run.smithi045.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T17:34:18.827 INFO:teuthology.orchestra.run.smithi045.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T17:34:18.827 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T17:34:18.827 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T17:34:18.827 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T17:34:18.827 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T17:34:18.827 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T17:34:18.827 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T17:34:18.827 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T17:34:18.827 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T17:34:18.828 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T17:34:18.828 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T17:34:18.828 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T17:34:18.828 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T17:34:18.828 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T17:34:18.828 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T17:34:18.828 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T17:34:18.828 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T17:34:18.828 INFO:teuthology.orchestra.run.smithi045.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T17:34:18.828 INFO:teuthology.orchestra.run.smithi045.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T17:34:18.828 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T17:34:18.828 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T17:34:18.828 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T17:34:18.828 INFO:teuthology.orchestra.run.smithi045.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T17:34:18.829 INFO:teuthology.orchestra.run.smithi045.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T17:34:18.829 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T17:34:18.829 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T17:34:18.829 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T17:34:18.829 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T17:34:18.829 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T17:34:18.829 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T17:34:18.829 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T17:34:18.829 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T17:34:18.829 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T17:34:18.829 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T17:34:18.829 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T17:34:18.829 INFO:teuthology.orchestra.run.smithi045.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T17:34:18.829 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrading: 2024-03-28T17:34:18.830 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T17:34:18.830 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T17:34:18.830 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T17:34:18.830 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T17:34:18.830 INFO:teuthology.orchestra.run.smithi045.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T17:34:18.830 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T17:34:18.830 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T17:34:18.830 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T17:34:18.830 INFO:teuthology.orchestra.run.smithi045.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T17:34:18.830 INFO:teuthology.orchestra.run.smithi045.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T17:34:18.830 INFO:teuthology.orchestra.run.smithi045.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T17:34:18.830 INFO:teuthology.orchestra.run.smithi045.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T17:34:18.830 INFO:teuthology.orchestra.run.smithi045.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T17:34:18.830 INFO:teuthology.orchestra.run.smithi045.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T17:34:18.831 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T17:34:18.831 INFO:teuthology.orchestra.run.smithi045.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T17:34:18.831 INFO:teuthology.orchestra.run.smithi045.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T17:34:18.831 INFO:teuthology.orchestra.run.smithi045.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T17:34:18.831 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T17:34:18.831 INFO:teuthology.orchestra.run.smithi045.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T17:34:18.831 INFO:teuthology.orchestra.run.smithi045.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T17:34:18.831 INFO:teuthology.orchestra.run.smithi045.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T17:34:18.831 INFO:teuthology.orchestra.run.smithi045.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T17:34:18.831 INFO:teuthology.orchestra.run.smithi045.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T17:34:18.831 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-28T17:34:18.831 INFO:teuthology.orchestra.run.smithi045.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T17:34:18.831 INFO:teuthology.orchestra.run.smithi045.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T17:34:18.831 INFO:teuthology.orchestra.run.smithi045.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T17:34:18.831 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T17:34:18.832 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T17:34:18.832 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T17:34:18.832 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T17:34:18.832 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T17:34:18.832 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T17:34:18.832 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T17:34:18.832 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T17:34:18.832 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T17:34:18.832 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T17:34:18.832 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T17:34:18.832 INFO:teuthology.orchestra.run.smithi045.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T17:34:18.832 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T17:34:18.832 INFO:teuthology.orchestra.run.smithi045.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T17:34:18.832 INFO:teuthology.orchestra.run.smithi045.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T17:34:18.832 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T17:34:18.833 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T17:34:18.833 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T17:34:18.833 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T17:34:18.833 INFO:teuthology.orchestra.run.smithi045.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T17:34:18.833 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T17:34:18.833 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T17:34:18.833 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T17:34:18.833 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T17:34:18.833 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T17:34:18.833 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T17:34:18.833 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T17:34:18.833 INFO:teuthology.orchestra.run.smithi045.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T17:34:18.833 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T17:34:18.833 INFO:teuthology.orchestra.run.smithi045.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T17:34:18.833 INFO:teuthology.orchestra.run.smithi045.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T17:34:18.834 INFO:teuthology.orchestra.run.smithi045.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T17:34:18.834 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T17:34:18.834 INFO:teuthology.orchestra.run.smithi045.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T17:34:18.834 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T17:34:18.834 INFO:teuthology.orchestra.run.smithi045.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T17:34:18.834 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T17:34:18.834 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T17:34:18.834 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T17:34:18.834 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T17:34:18.834 INFO:teuthology.orchestra.run.smithi045.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T17:34:18.834 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T17:34:18.834 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T17:34:18.834 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T17:34:18.835 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T17:34:18.835 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T17:34:18.835 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T17:34:18.835 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T17:34:18.835 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T17:34:18.835 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T17:34:18.835 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T17:34:18.835 INFO:teuthology.orchestra.run.smithi045.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T17:34:18.835 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T17:34:18.835 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T17:34:18.835 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T17:34:18.835 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T17:34:18.835 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T17:34:18.835 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T17:34:18.835 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T17:34:18.836 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T17:34:18.836 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T17:34:18.836 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T17:34:18.836 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T17:34:18.836 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T17:34:18.836 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T17:34:18.836 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T17:34:18.836 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T17:34:18.836 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T17:34:18.836 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T17:34:18.836 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T17:34:18.836 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T17:34:18.836 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T17:34:18.836 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T17:34:18.836 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T17:34:18.837 INFO:teuthology.orchestra.run.smithi045.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T17:34:18.837 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T17:34:18.837 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T17:34:18.837 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T17:34:18.837 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T17:34:18.837 INFO:teuthology.orchestra.run.smithi045.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T17:34:18.837 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T17:34:18.837 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T17:34:18.837 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T17:34:18.837 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T17:34:18.837 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T17:34:18.837 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T17:34:18.837 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T17:34:18.837 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T17:34:18.837 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T17:34:18.838 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T17:34:18.838 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T17:34:18.838 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T17:34:18.838 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T17:34:18.838 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T17:34:18.838 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T17:34:18.838 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T17:34:18.839 INFO:teuthology.orchestra.run.smithi045.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T17:34:18.839 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T17:34:18.839 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T17:34:18.839 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T17:34:18.839 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T17:34:18.839 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T17:34:18.839 INFO:teuthology.orchestra.run.smithi045.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T17:34:18.840 INFO:teuthology.orchestra.run.smithi045.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T17:34:18.840 INFO:teuthology.orchestra.run.smithi045.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T17:34:18.840 INFO:teuthology.orchestra.run.smithi045.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T17:34:18.840 INFO:teuthology.orchestra.run.smithi045.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T17:34:18.840 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T17:34:18.840 INFO:teuthology.orchestra.run.smithi045.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T17:34:18.840 INFO:teuthology.orchestra.run.smithi045.stdout:Installing weak dependencies: 2024-03-28T17:34:18.840 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T17:34:18.840 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:34:18.840 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-28T17:34:18.840 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2024-03-28T17:34:18.840 INFO:teuthology.orchestra.run.smithi045.stdout:Install 150 Packages 2024-03-28T17:34:18.840 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrade 24 Packages 2024-03-28T17:34:18.840 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:34:18.841 INFO:teuthology.orchestra.run.smithi151.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-03-28T17:34:18.841 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 225 M 2024-03-28T17:34:18.841 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-28T17:34:18.865 INFO:teuthology.orchestra.run.smithi151.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 791 kB/s | 46 kB 00:00 2024-03-28T17:34:18.890 INFO:teuthology.orchestra.run.smithi151.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-28T17:34:18.915 INFO:teuthology.orchestra.run.smithi151.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2024-03-28T17:34:18.940 INFO:teuthology.orchestra.run.smithi151.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-28T17:34:18.965 INFO:teuthology.orchestra.run.smithi151.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 686 kB/s | 34 kB 00:00 2024-03-28T17:34:19.004 INFO:teuthology.orchestra.run.smithi071.stdout:(77/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-03-28T17:34:19.029 INFO:teuthology.orchestra.run.smithi071.stdout:(78/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 300 kB 00:00 2024-03-28T17:34:19.054 INFO:teuthology.orchestra.run.smithi071.stdout:(79/174): uuid-1.6.2-55.el9.x86_64.rpm 243 kB/s | 56 kB 00:00 2024-03-28T17:34:19.066 INFO:teuthology.orchestra.run.smithi151.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-28T17:34:19.080 INFO:teuthology.orchestra.run.smithi071.stdout:(80/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-28T17:34:19.099 INFO:teuthology.orchestra.run.smithi151.stdout:(57/174): protobuf-3.14.0-13.el9.x86_64.rpm 7.7 MB/s | 1.0 MB 00:00 2024-03-28T17:34:19.105 INFO:teuthology.orchestra.run.smithi071.stdout:(81/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 588 kB/s | 44 kB 00:00 2024-03-28T17:34:19.124 INFO:teuthology.orchestra.run.smithi151.stdout:(58/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-28T17:34:19.130 INFO:teuthology.orchestra.run.smithi071.stdout:(82/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-28T17:34:19.155 INFO:teuthology.orchestra.run.smithi071.stdout:(83/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-28T17:34:19.197 INFO:teuthology.orchestra.run.smithi071.stdout:(84/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-03-28T17:34:19.222 INFO:teuthology.orchestra.run.smithi071.stdout:(85/174): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-28T17:34:19.256 INFO:teuthology.orchestra.run.smithi071.stdout:(86/174): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 124 kB 00:00 2024-03-28T17:34:19.267 INFO:teuthology.orchestra.run.smithi151.stdout:(59/174): python3-babel-2.9.1-2.el9.noarch.rpm 35 MB/s | 5.8 MB 00:00 2024-03-28T17:34:19.281 INFO:teuthology.orchestra.run.smithi071.stdout:(87/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-28T17:34:19.292 INFO:teuthology.orchestra.run.smithi151.stdout:(60/174): python3-jsonpatch-1.21-16.el9.noarch. 149 kB/s | 24 kB 00:00 2024-03-28T17:34:19.309 INFO:teuthology.orchestra.run.smithi151.stdout:(61/174): python3-jsonpointer-2.0-4.el9.noarch. 436 kB/s | 18 kB 00:00 2024-03-28T17:34:19.334 INFO:teuthology.orchestra.run.smithi151.stdout:(62/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-28T17:34:19.351 INFO:teuthology.orchestra.run.smithi151.stdout:(63/174): python3-markupsafe-1.1.1-12.el9.x86_6 758 kB/s | 32 kB 00:00 2024-03-28T17:34:19.398 INFO:teuthology.orchestra.run.smithi071.stdout:(88/174): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-28T17:34:19.424 INFO:teuthology.orchestra.run.smithi071.stdout:(89/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-28T17:34:19.449 INFO:teuthology.orchestra.run.smithi071.stdout:(90/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-28T17:34:19.460 INFO:teuthology.orchestra.run.smithi151.stdout:(64/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-28T17:34:19.485 INFO:teuthology.orchestra.run.smithi151.stdout:(65/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 217 kB 00:00 2024-03-28T17:34:19.491 INFO:teuthology.orchestra.run.smithi071.stdout:(91/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.4 MB/s | 803 kB 00:00 2024-03-28T17:34:19.510 INFO:teuthology.orchestra.run.smithi151.stdout:(66/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-28T17:34:19.535 INFO:teuthology.orchestra.run.smithi151.stdout:(67/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-28T17:34:19.591 INFO:teuthology.orchestra.run.smithi071.stdout:(92/174): elfutils-libelf-devel-0.190-2.el9.x86 129 kB/s | 21 kB 00:00 2024-03-28T17:34:19.617 INFO:teuthology.orchestra.run.smithi071.stdout:(93/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 152 kB/s | 25 kB 00:00 2024-03-28T17:34:19.661 INFO:teuthology.orchestra.run.smithi151.stdout:(68/174): python3-prettytable-0.7.2-27.el9.noar 274 kB/s | 41 kB 00:00 2024-03-28T17:34:19.686 INFO:teuthology.orchestra.run.smithi151.stdout:(69/174): python3-protobuf-3.14.0-13.el9.noarch 1.5 MB/s | 238 kB 00:00 2024-03-28T17:34:19.711 INFO:teuthology.orchestra.run.smithi151.stdout:(70/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-28T17:34:19.737 INFO:teuthology.orchestra.run.smithi151.stdout:(71/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-28T17:34:19.750 INFO:teuthology.orchestra.run.smithi071.stdout:(94/174): postgresql-contrib-13.14-1.el9.x86_64 6.7 MB/s | 911 kB 00:00 2024-03-28T17:34:19.762 INFO:teuthology.orchestra.run.smithi151.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 948 kB/s | 47 kB 00:00 2024-03-28T17:34:19.787 INFO:teuthology.orchestra.run.smithi151.stdout:(73/174): python3-requests-oauthlib-1.3.0-12.el 854 kB/s | 43 kB 00:00 2024-03-28T17:34:19.792 INFO:teuthology.orchestra.run.smithi071.stdout:(95/174): postgresql-13.14-1.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-03-28T17:34:19.817 INFO:teuthology.orchestra.run.smithi071.stdout:(96/174): postgresql-private-libs-13.14-1.el9.x 1.9 MB/s | 133 kB 00:00 2024-03-28T17:34:19.876 INFO:teuthology.orchestra.run.smithi071.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 4.1 MB/s | 245 kB 00:00 2024-03-28T17:34:19.960 INFO:teuthology.orchestra.run.smithi071.stdout:(98/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 8.8 MB/s | 4.1 MB 00:00 2024-03-28T17:34:19.985 INFO:teuthology.orchestra.run.smithi071.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.2 MB/s | 248 kB 00:00 2024-03-28T17:34:20.027 INFO:teuthology.orchestra.run.smithi071.stdout:(100/174): qatlib-service-24.02.0-1.el9.x86_64. 866 kB/s | 36 kB 00:00 2024-03-28T17:34:20.065 INFO:teuthology.orchestra.run.smithi045.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 21 kB/s | 6.6 kB 00:00 2024-03-28T17:34:20.077 INFO:teuthology.orchestra.run.smithi071.stdout:(101/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 949 kB/s | 47 kB 00:00 2024-03-28T17:34:20.110 INFO:teuthology.orchestra.run.smithi071.stdout:(102/174): qatlib-24.02.0-1.el9.x86_64.rpm 1.4 MB/s | 221 kB 00:00 2024-03-28T17:34:20.152 INFO:teuthology.orchestra.run.smithi071.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-03-28T17:34:20.186 INFO:teuthology.orchestra.run.smithi071.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 296 kB/s | 22 kB 00:00 2024-03-28T17:34:20.221 INFO:teuthology.orchestra.run.smithi151.stdout:(74/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 8.1 MB/s | 48 MB 00:05 2024-03-28T17:34:20.286 INFO:teuthology.orchestra.run.smithi071.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 6.3 MB/s | 863 kB 00:00 2024-03-28T17:34:20.346 INFO:teuthology.orchestra.run.smithi151.stdout:(75/174): python3-toml-0.10.2-6.el9.noarch.rpm 70 kB/s | 39 kB 00:00 2024-03-28T17:34:20.390 INFO:teuthology.orchestra.run.smithi045.stdout:(2/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 8.3 MB/s | 5.4 MB 00:00 2024-03-28T17:34:20.432 INFO:teuthology.orchestra.run.smithi045.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 3.5 MB/s | 145 kB 00:00 2024-03-28T17:34:20.505 INFO:teuthology.orchestra.run.smithi151.stdout:(76/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2024-03-28T17:34:20.530 INFO:teuthology.orchestra.run.smithi151.stdout:(77/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 134 kB/s | 41 kB 00:00 2024-03-28T17:34:20.553 INFO:teuthology.orchestra.run.smithi071.stdout:(106/174): postgresql-server-13.14-1.el9.x86_64 7.6 MB/s | 5.8 MB 00:00 2024-03-28T17:34:20.555 INFO:teuthology.orchestra.run.smithi151.stdout:(78/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-03-28T17:34:20.580 INFO:teuthology.orchestra.run.smithi151.stdout:(79/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-28T17:34:20.605 INFO:teuthology.orchestra.run.smithi151.stdout:(80/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 840 kB/s | 63 kB 00:00 2024-03-28T17:34:20.607 INFO:teuthology.orchestra.run.smithi045.stdout:(4/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.6 MB/s | 890 kB 00:00 2024-03-28T17:34:20.630 INFO:teuthology.orchestra.run.smithi151.stdout:(81/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 587 kB/s | 44 kB 00:00 2024-03-28T17:34:20.656 INFO:teuthology.orchestra.run.smithi151.stdout:(82/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-28T17:34:20.681 INFO:teuthology.orchestra.run.smithi151.stdout:(83/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-28T17:34:20.699 INFO:teuthology.orchestra.run.smithi045.stdout:(5/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 8.7 MB/s | 2.3 MB 00:00 2024-03-28T17:34:20.714 INFO:teuthology.orchestra.run.smithi151.stdout:(84/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-28T17:34:20.741 INFO:teuthology.orchestra.run.smithi151.stdout:(85/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-28T17:34:20.765 INFO:teuthology.orchestra.run.smithi151.stdout:(86/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-28T17:34:20.787 INFO:teuthology.orchestra.run.smithi071.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 617 kB/s | 309 kB 00:00 2024-03-28T17:34:20.790 INFO:teuthology.orchestra.run.smithi151.stdout:(87/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-28T17:34:20.812 INFO:teuthology.orchestra.run.smithi071.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-03-28T17:34:20.896 INFO:teuthology.orchestra.run.smithi071.stdout:(109/174): abseil-cpp-20211102.0-3.el9.x86_64.r 773 kB/s | 548 kB 00:00 2024-03-28T17:34:20.921 INFO:teuthology.orchestra.run.smithi071.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 239 kB/s | 26 kB 00:00 2024-03-28T17:34:21.021 INFO:teuthology.orchestra.run.smithi071.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 388 kB/s | 49 kB 00:00 2024-03-28T17:34:21.058 INFO:teuthology.orchestra.run.smithi045.stdout:(6/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 3.3 MB/s | 1.5 MB 00:00 2024-03-28T17:34:21.115 INFO:teuthology.orchestra.run.smithi151.stdout:(88/174): libquadmath-11.4.1-3.el9.x86_64.rpm 602 kB/s | 195 kB 00:00 2024-03-28T17:34:21.121 INFO:teuthology.orchestra.run.smithi071.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 337 kB/s | 67 kB 00:00 2024-03-28T17:34:21.147 INFO:teuthology.orchestra.run.smithi071.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-03-28T17:34:21.166 INFO:teuthology.orchestra.run.smithi151.stdout:(89/174): jq-1.6-16.el9.x86_64.rpm 468 kB/s | 187 kB 00:00 2024-03-28T17:34:21.166 INFO:teuthology.orchestra.run.smithi045.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 10 MB/s | 4.7 MB 00:00 2024-03-28T17:34:21.264 INFO:teuthology.orchestra.run.smithi071.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 9.3 MB/s | 4.4 MB 00:00 2024-03-28T17:34:21.274 INFO:teuthology.orchestra.run.smithi151.stdout:(90/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-03-28T17:34:21.339 INFO:teuthology.orchestra.run.smithi071.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 2.8 MB/s | 548 kB 00:00 2024-03-28T17:34:21.373 INFO:teuthology.orchestra.run.smithi071.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 400 kB/s | 43 kB 00:00 2024-03-28T17:34:21.402 INFO:teuthology.orchestra.run.smithi151.stdout:(91/174): elfutils-libelf-devel-0.190-2.el9.x86 102 kB/s | 21 kB 00:00 2024-03-28T17:34:21.427 INFO:teuthology.orchestra.run.smithi151.stdout:(92/174): python3-requests-2.25.1-8.el9.noarch. 401 kB/s | 125 kB 00:00 2024-03-28T17:34:21.448 INFO:teuthology.orchestra.run.smithi071.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 296 kB/s | 32 kB 00:00 2024-03-28T17:34:21.469 INFO:teuthology.orchestra.run.smithi151.stdout:(93/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 150 kB/s | 25 kB 00:00 2024-03-28T17:34:21.498 INFO:teuthology.orchestra.run.smithi071.stdout:(118/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.2 MB/s | 838 kB 00:00 2024-03-28T17:34:21.523 INFO:teuthology.orchestra.run.smithi071.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 100 kB/s | 15 kB 00:00 2024-03-28T17:34:21.574 INFO:teuthology.orchestra.run.smithi071.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2024-03-28T17:34:21.649 INFO:teuthology.orchestra.run.smithi071.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 2.3 MB/s | 358 kB 00:00 2024-03-28T17:34:21.670 INFO:teuthology.orchestra.run.smithi151.stdout:(94/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 16 MB/s | 4.1 MB 00:00 2024-03-28T17:34:21.683 INFO:teuthology.orchestra.run.smithi071.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.9 MB/s | 303 kB 00:00 2024-03-28T17:34:21.708 INFO:teuthology.orchestra.run.smithi071.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 264 kB/s | 35 kB 00:00 2024-03-28T17:34:21.711 INFO:teuthology.orchestra.run.smithi151.stdout:(95/174): postgresql-13.14-1.el9.x86_64.rpm 5.6 MB/s | 1.6 MB 00:00 2024-03-28T17:34:21.753 INFO:teuthology.orchestra.run.smithi151.stdout:(96/174): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-03-28T17:34:21.812 INFO:teuthology.orchestra.run.smithi151.stdout:(97/174): postgresql-contrib-13.14-1.el9.x86_64 2.6 MB/s | 911 kB 00:00 2024-03-28T17:34:21.879 INFO:teuthology.orchestra.run.smithi151.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-03-28T17:34:21.884 INFO:teuthology.orchestra.run.smithi071.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 866 kB/s | 203 kB 00:00 2024-03-28T17:34:21.913 INFO:teuthology.orchestra.run.smithi151.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-03-28T17:34:21.950 INFO:teuthology.orchestra.run.smithi071.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 597 kB/s | 144 kB 00:00 2024-03-28T17:34:21.992 INFO:teuthology.orchestra.run.smithi071.stdout:(126/174): python3-grpcio-1.46.7-10.el9.x86_64. 6.6 MB/s | 2.0 MB 00:00 2024-03-28T17:34:21.996 INFO:teuthology.orchestra.run.smithi151.stdout:(100/174): postgresql-server-13.14-1.el9.x86_64 20 MB/s | 5.8 MB 00:00 2024-03-28T17:34:22.018 INFO:teuthology.orchestra.run.smithi071.stdout:(127/174): python3-jaraco-8.2.1-3.el9.noarch.rp 80 kB/s | 11 kB 00:00 2024-03-28T17:34:22.022 INFO:teuthology.orchestra.run.smithi151.stdout:(101/174): qatlib-24.02.0-1.el9.x86_64.rpm 1.5 MB/s | 221 kB 00:00 2024-03-28T17:34:22.035 INFO:teuthology.orchestra.run.smithi071.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 210 kB/s | 18 kB 00:00 2024-03-28T17:34:22.052 INFO:teuthology.orchestra.run.smithi151.stdout:(102/174): qatlib-service-24.02.0-1.el9.x86_64. 258 kB/s | 36 kB 00:00 2024-03-28T17:34:22.076 INFO:teuthology.orchestra.run.smithi071.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 277 kB/s | 23 kB 00:00 2024-03-28T17:34:22.077 INFO:teuthology.orchestra.run.smithi151.stdout:(103/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 588 kB/s | 47 kB 00:00 2024-03-28T17:34:22.102 INFO:teuthology.orchestra.run.smithi071.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 232 kB/s | 19 kB 00:00 2024-03-28T17:34:22.119 INFO:teuthology.orchestra.run.smithi071.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 235 kB/s | 20 kB 00:00 2024-03-28T17:34:22.125 INFO:teuthology.orchestra.run.smithi151.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 911 kB/s | 94 kB 00:00 2024-03-28T17:34:22.183 INFO:teuthology.orchestra.run.smithi045.stdout:(8/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 9.7 MB/s | 9.9 MB 00:01 2024-03-28T17:34:22.210 INFO:teuthology.orchestra.run.smithi071.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 426 kB/s | 46 kB 00:00 2024-03-28T17:34:22.236 INFO:teuthology.orchestra.run.smithi071.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 674 kB/s | 79 kB 00:00 2024-03-28T17:34:22.242 INFO:teuthology.orchestra.run.smithi045.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 440 kB/s | 25 kB 00:00 2024-03-28T17:34:22.269 INFO:teuthology.orchestra.run.smithi071.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 5.3 MB/s | 1.0 MB 00:00 2024-03-28T17:34:22.328 INFO:teuthology.orchestra.run.smithi071.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 493 kB/s | 58 kB 00:00 2024-03-28T17:34:22.354 INFO:teuthology.orchestra.run.smithi151.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 74 kB/s | 22 kB 00:00 2024-03-28T17:34:22.388 INFO:teuthology.orchestra.run.smithi151.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 2.0 MB/s | 548 kB 00:00 2024-03-28T17:34:22.403 INFO:teuthology.orchestra.run.smithi071.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-03-28T17:34:22.421 INFO:teuthology.orchestra.run.smithi151.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 4.5 MB/s | 309 kB 00:00 2024-03-28T17:34:22.428 INFO:teuthology.orchestra.run.smithi071.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 103 kB/s | 16 kB 00:00 2024-03-28T17:34:22.454 INFO:teuthology.orchestra.run.smithi071.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 715 kB/s | 90 kB 00:00 2024-03-28T17:34:22.455 INFO:teuthology.orchestra.run.smithi151.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 291 kB/s | 19 kB 00:00 2024-03-28T17:34:22.492 INFO:teuthology.orchestra.run.smithi045.stdout:(10/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 7.5 MB/s | 20 MB 00:02 2024-03-28T17:34:22.504 INFO:teuthology.orchestra.run.smithi071.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 281 kB/s | 28 kB 00:00 2024-03-28T17:34:22.554 INFO:teuthology.orchestra.run.smithi071.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 245 kB/s | 31 kB 00:00 2024-03-28T17:34:22.555 INFO:teuthology.orchestra.run.smithi151.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 259 kB/s | 26 kB 00:00 2024-03-28T17:34:22.584 INFO:teuthology.orchestra.run.smithi045.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 2.4 MB/s | 220 kB 00:00 2024-03-28T17:34:22.596 INFO:teuthology.orchestra.run.smithi071.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.3 MB/s | 188 kB 00:00 2024-03-28T17:34:22.622 INFO:teuthology.orchestra.run.smithi071.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 505 kB/s | 59 kB 00:00 2024-03-28T17:34:22.647 INFO:teuthology.orchestra.run.smithi071.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 388 kB/s | 36 kB 00:00 2024-03-28T17:34:22.655 INFO:teuthology.orchestra.run.smithi151.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 490 kB/s | 49 kB 00:00 2024-03-28T17:34:22.684 INFO:teuthology.orchestra.run.smithi045.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 273 kB/s | 27 kB 00:00 2024-03-28T17:34:22.739 INFO:teuthology.orchestra.run.smithi151.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 811 kB/s | 67 kB 00:00 2024-03-28T17:34:22.747 INFO:teuthology.orchestra.run.smithi071.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-03-28T17:34:22.772 INFO:teuthology.orchestra.run.smithi071.stdout:(145/174): python3-typing-extensions-4.4.0-2.el 290 kB/s | 51 kB 00:00 2024-03-28T17:34:22.785 INFO:teuthology.orchestra.run.smithi045.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 317 kB/s | 31 kB 00:00 2024-03-28T17:34:22.848 INFO:teuthology.orchestra.run.smithi151.stdout:(112/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.1 MB/s | 863 kB 00:00 2024-03-28T17:34:22.881 INFO:teuthology.orchestra.run.smithi151.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-28T17:34:22.881 INFO:teuthology.orchestra.run.smithi071.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 383 kB/s | 90 kB 00:00 2024-03-28T17:34:22.906 INFO:teuthology.orchestra.run.smithi071.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 149 kB/s | 20 kB 00:00 2024-03-28T17:34:22.948 INFO:teuthology.orchestra.run.smithi071.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.1 MB/s | 427 kB 00:00 2024-03-28T17:34:22.968 INFO:teuthology.orchestra.run.smithi045.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 4.0 MB/s | 746 kB 00:00 2024-03-28T17:34:22.973 INFO:teuthology.orchestra.run.smithi151.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 8.0 MB/s | 4.4 MB 00:00 2024-03-28T17:34:23.007 INFO:teuthology.orchestra.run.smithi071.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-03-28T17:34:23.076 INFO:teuthology.orchestra.run.smithi045.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.5 MB/s | 163 kB 00:00 2024-03-28T17:34:23.107 INFO:teuthology.orchestra.run.smithi151.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 325 kB/s | 43 kB 00:00 2024-03-28T17:34:23.157 INFO:teuthology.orchestra.run.smithi151.stdout:(116/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.6 MB/s | 838 kB 00:00 2024-03-28T17:34:23.174 INFO:teuthology.orchestra.run.smithi071.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 5.9 MB/s | 1.6 MB 00:00 2024-03-28T17:34:23.177 INFO:teuthology.orchestra.run.smithi045.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.3 MB/s | 127 kB 00:00 2024-03-28T17:34:23.216 INFO:teuthology.orchestra.run.smithi071.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-28T17:34:23.216 INFO:teuthology.orchestra.run.smithi151.stdout:(117/174): python3-asyncssh-2.13.2-5.el9.noarch 1.6 MB/s | 548 kB 00:00 2024-03-28T17:34:23.241 INFO:teuthology.orchestra.run.smithi151.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 241 kB/s | 32 kB 00:00 2024-03-28T17:34:23.266 INFO:teuthology.orchestra.run.smithi151.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-03-28T17:34:23.319 INFO:teuthology.orchestra.run.smithi045.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 3.5 MB/s | 501 kB 00:00 2024-03-28T17:34:23.333 INFO:teuthology.orchestra.run.smithi151.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-03-28T17:34:23.408 INFO:teuthology.orchestra.run.smithi071.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 12 MB/s | 2.3 MB 00:00 2024-03-28T17:34:23.409 INFO:teuthology.orchestra.run.smithi151.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 2.1 MB/s | 358 kB 00:00 2024-03-28T17:34:23.451 INFO:teuthology.orchestra.run.smithi151.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.6 MB/s | 303 kB 00:00 2024-03-28T17:34:23.475 INFO:teuthology.orchestra.run.smithi071.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 28 MB/s | 1.8 MB 00:00 2024-03-28T17:34:23.493 INFO:teuthology.orchestra.run.smithi151.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 222 kB/s | 35 kB 00:00 2024-03-28T17:34:23.517 INFO:teuthology.orchestra.run.smithi071.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 1.0 MB/s | 42 kB 00:00 2024-03-28T17:34:23.535 INFO:teuthology.orchestra.run.smithi151.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 1.6 MB/s | 203 kB 00:00 2024-03-28T17:34:23.594 INFO:teuthology.orchestra.run.smithi045.stdout:(18/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 6.7 MB/s | 17 MB 00:02 2024-03-28T17:34:23.594 INFO:teuthology.orchestra.run.smithi071.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 5.8 MB/s | 248 kB 00:00 2024-03-28T17:34:23.600 INFO:teuthology.orchestra.run.smithi071.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.7 MB/s | 71 kB 00:00 2024-03-28T17:34:23.643 INFO:teuthology.orchestra.run.smithi071.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 875 kB/s | 36 kB 00:00 2024-03-28T17:34:23.652 INFO:teuthology.orchestra.run.smithi045.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 690 kB/s | 45 kB 00:00 2024-03-28T17:34:23.677 INFO:teuthology.orchestra.run.smithi151.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 75 kB/s | 11 kB 00:00 2024-03-28T17:34:23.684 INFO:teuthology.orchestra.run.smithi071.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 4.5 MB/s | 192 kB 00:00 2024-03-28T17:34:23.726 INFO:teuthology.orchestra.run.smithi071.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 6.0 MB/s | 256 kB 00:00 2024-03-28T17:34:23.735 INFO:teuthology.orchestra.run.smithi151.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 306 kB/s | 18 kB 00:00 2024-03-28T17:34:23.744 INFO:teuthology.orchestra.run.smithi045.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.4 MB/s | 131 kB 00:00 2024-03-28T17:34:23.761 INFO:teuthology.orchestra.run.smithi151.stdout:(127/174): python3-grpcio-tools-1.46.7-10.el9.x 540 kB/s | 144 kB 00:00 2024-03-28T17:34:23.768 INFO:teuthology.orchestra.run.smithi071.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 555 kB/s | 23 kB 00:00 2024-03-28T17:34:23.810 INFO:teuthology.orchestra.run.smithi071.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.5 MB/s | 108 kB 00:00 2024-03-28T17:34:23.853 INFO:teuthology.orchestra.run.smithi045.stdout:(21/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 9.7 MB/s | 5.1 MB 00:00 2024-03-28T17:34:23.860 INFO:teuthology.orchestra.run.smithi071.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 155 kB 00:00 2024-03-28T17:34:23.886 INFO:teuthology.orchestra.run.smithi045.stdout:(22/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 1.1 MB/s | 162 kB 00:00 2024-03-28T17:34:23.902 INFO:teuthology.orchestra.run.smithi071.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 136 kB 00:00 2024-03-28T17:34:23.919 INFO:teuthology.orchestra.run.smithi151.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 126 kB/s | 23 kB 00:00 2024-03-28T17:34:23.945 INFO:teuthology.orchestra.run.smithi071.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.5 MB/s | 64 kB 00:00 2024-03-28T17:34:23.946 INFO:teuthology.orchestra.run.smithi045.stdout:(23/174): python3-rados-19.0.0-2145.g3be9a5c9.e 3.4 MB/s | 321 kB 00:00 2024-03-28T17:34:23.953 INFO:teuthology.orchestra.run.smithi151.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 102 kB/s | 19 kB 00:00 2024-03-28T17:34:23.987 INFO:teuthology.orchestra.run.smithi071.stdout:(165/174): libuuid-2.37.4-18.el9.x86_64.rpm 690 kB/s | 29 kB 00:00 2024-03-28T17:34:23.987 INFO:teuthology.orchestra.run.smithi045.stdout:(24/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 2.9 MB/s | 298 kB 00:00 2024-03-28T17:34:23.996 INFO:teuthology.orchestra.run.smithi151.stdout:(130/174): python3-grpcio-1.46.7-10.el9.x86_64. 3.7 MB/s | 2.0 MB 00:00 2024-03-28T17:34:24.021 INFO:teuthology.orchestra.run.smithi151.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 296 kB/s | 20 kB 00:00 2024-03-28T17:34:24.045 INFO:teuthology.orchestra.run.smithi071.stdout:(166/174): openssl-3.0.7-27.el9.x86_64.rpm 21 MB/s | 1.2 MB 00:00 2024-03-28T17:34:24.046 INFO:teuthology.orchestra.run.smithi045.stdout:(25/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.0 MB/s | 99 kB 00:00 2024-03-28T17:34:24.071 INFO:teuthology.orchestra.run.smithi045.stdout:(26/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.0 MB/s | 85 kB 00:00 2024-03-28T17:34:24.120 INFO:teuthology.orchestra.run.smithi071.stdout:(167/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 29 MB/s | 2.2 MB 00:00 2024-03-28T17:34:24.154 INFO:teuthology.orchestra.run.smithi151.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 292 kB/s | 46 kB 00:00 2024-03-28T17:34:24.155 INFO:teuthology.orchestra.run.smithi045.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 2.0 MB/s | 170 kB 00:00 2024-03-28T17:34:24.180 INFO:teuthology.orchestra.run.smithi151.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 497 kB/s | 79 kB 00:00 2024-03-28T17:34:24.204 INFO:teuthology.orchestra.run.smithi071.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 27 MB/s | 2.3 MB 00:00 2024-03-28T17:34:24.214 INFO:teuthology.orchestra.run.smithi151.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 3.9 MB/s | 1.0 MB 00:00 2024-03-28T17:34:24.246 INFO:teuthology.orchestra.run.smithi071.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 11 MB/s | 465 kB 00:00 2024-03-28T17:34:24.273 INFO:teuthology.orchestra.run.smithi045.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 292 kB/s | 34 kB 00:00 2024-03-28T17:34:24.280 INFO:teuthology.orchestra.run.smithi071.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 935 kB/s | 31 kB 00:00 2024-03-28T17:34:24.297 INFO:teuthology.orchestra.run.smithi151.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 407 kB/s | 58 kB 00:00 2024-03-28T17:34:24.338 INFO:teuthology.orchestra.run.smithi071.stdout:(171/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-03-28T17:34:24.355 INFO:teuthology.orchestra.run.smithi151.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-03-28T17:34:24.372 INFO:teuthology.orchestra.run.smithi045.stdout:(29/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 9.5 MB/s | 3.1 MB 00:00 2024-03-28T17:34:24.373 INFO:teuthology.orchestra.run.smithi151.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 103 kB/s | 16 kB 00:00 2024-03-28T17:34:24.406 INFO:teuthology.orchestra.run.smithi045.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 1.0 MB/s | 139 kB 00:00 2024-03-28T17:34:24.422 INFO:teuthology.orchestra.run.smithi071.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 892 kB/s | 74 kB 00:00 2024-03-28T17:34:24.440 INFO:teuthology.orchestra.run.smithi151.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 631 kB/s | 90 kB 00:00 2024-03-28T17:34:24.457 INFO:teuthology.orchestra.run.smithi151.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 280 kB/s | 28 kB 00:00 2024-03-28T17:34:24.482 INFO:teuthology.orchestra.run.smithi151.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 282 kB/s | 31 kB 00:00 2024-03-28T17:34:24.516 INFO:teuthology.orchestra.run.smithi151.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-28T17:34:24.541 INFO:teuthology.orchestra.run.smithi151.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 704 kB/s | 59 kB 00:00 2024-03-28T17:34:24.575 INFO:teuthology.orchestra.run.smithi151.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 388 kB/s | 36 kB 00:00 2024-03-28T17:34:24.650 INFO:teuthology.orchestra.run.smithi151.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 381 kB/s | 51 kB 00:00 2024-03-28T17:34:24.730 INFO:teuthology.orchestra.run.smithi071.stdout:(173/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 1.9 MB/s | 3.1 MB 00:01 2024-03-28T17:34:24.734 INFO:teuthology.orchestra.run.smithi151.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.4 MB/s | 230 kB 00:00 2024-03-28T17:34:24.751 INFO:teuthology.orchestra.run.smithi151.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 511 kB/s | 90 kB 00:00 2024-03-28T17:34:24.776 INFO:teuthology.orchestra.run.smithi151.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-03-28T17:34:24.801 INFO:teuthology.orchestra.run.smithi151.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 302 kB/s | 20 kB 00:00 2024-03-28T17:34:24.910 INFO:teuthology.orchestra.run.smithi151.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-03-28T17:34:24.966 INFO:teuthology.orchestra.run.smithi071.stdout:(174/174): librados2-19.0.0-2145.g3be9a5c9.el9. 1.7 MB/s | 3.4 MB 00:02 2024-03-28T17:34:24.970 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-28T17:34:24.970 INFO:teuthology.orchestra.run.smithi071.stdout:Total 14 MB/s | 225 MB 00:16 2024-03-28T17:34:24.989 INFO:teuthology.orchestra.run.smithi045.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 2.9 MB/s | 1.8 MB 00:00 2024-03-28T17:34:25.077 INFO:teuthology.orchestra.run.smithi151.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 5.3 MB/s | 1.6 MB 00:00 2024-03-28T17:34:25.090 INFO:teuthology.orchestra.run.smithi045.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 2.5 MB/s | 248 kB 00:00 2024-03-28T17:34:25.127 INFO:teuthology.orchestra.run.smithi151.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-28T17:34:25.181 INFO:teuthology.orchestra.run.smithi045.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 545 kB/s | 49 kB 00:00 2024-03-28T17:34:25.282 INFO:teuthology.orchestra.run.smithi045.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 165 kB/s | 16 kB 00:00 2024-03-28T17:34:25.416 INFO:teuthology.orchestra.run.smithi045.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 1.9 MB/s | 264 kB 00:00 2024-03-28T17:34:25.549 INFO:teuthology.orchestra.run.smithi045.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 6.5 MB/s | 7.4 MB 00:01 2024-03-28T17:34:25.557 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-28T17:34:25.616 INFO:teuthology.orchestra.run.smithi045.stdout:(37/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 3.7 MB/s | 748 kB 00:00 2024-03-28T17:34:25.628 INFO:teuthology.orchestra.run.smithi151.stdout:(152/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 4.4 MB/s | 3.1 MB 00:00 2024-03-28T17:34:25.641 INFO:teuthology.orchestra.run.smithi045.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 201 kB/s | 18 kB 00:00 2024-03-28T17:34:25.678 INFO:teuthology.orchestra.run.smithi151.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 4.2 MB/s | 2.3 MB 00:00 2024-03-28T17:34:25.830 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-28T17:34:25.830 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-28T17:34:25.978 INFO:teuthology.orchestra.run.smithi151.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 153 kB/s | 42 kB 00:00 2024-03-28T17:34:26.187 INFO:teuthology.orchestra.run.smithi151.stdout:(155/174): NetworkManager-libnm-1.47.2-1.el9.x8 3.3 MB/s | 1.8 MB 00:00 2024-03-28T17:34:26.267 INFO:teuthology.orchestra.run.smithi045.stdout:(39/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 12 MB/s | 48 MB 00:04 2024-03-28T17:34:26.279 INFO:teuthology.orchestra.run.smithi151.stdout:(156/174): librados2-19.0.0-2145.g3be9a5c9.el9. 2.3 MB/s | 3.4 MB 00:01 2024-03-28T17:34:26.405 INFO:teuthology.orchestra.run.smithi151.stdout:(157/174): NetworkManager-tui-1.47.2-1.el9.x86_ 583 kB/s | 248 kB 00:00 2024-03-28T17:34:26.417 INFO:teuthology.orchestra.run.smithi045.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 1.2 MB/s | 920 kB 00:00 2024-03-28T17:34:26.459 INFO:teuthology.orchestra.run.smithi045.stdout:(41/174): automake-1.16.2-8.el9.noarch.rpm 787 kB/s | 662 kB 00:00 2024-03-28T17:34:26.492 INFO:teuthology.orchestra.run.smithi045.stdout:(42/174): boost-program-options-1.75.0-8.el9.x8 480 kB/s | 106 kB 00:00 2024-03-28T17:34:26.505 INFO:teuthology.orchestra.run.smithi151.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 225 kB/s | 71 kB 00:00 2024-03-28T17:34:26.518 INFO:teuthology.orchestra.run.smithi045.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 3.1 MB/s | 307 kB 00:00 2024-03-28T17:34:26.539 INFO:teuthology.orchestra.run.smithi151.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 155 kB/s | 36 kB 00:00 2024-03-28T17:34:26.543 INFO:teuthology.orchestra.run.smithi045.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 399 kB/s | 32 kB 00:00 2024-03-28T17:34:26.677 INFO:teuthology.orchestra.run.smithi045.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 79 kB/s | 10 kB 00:00 2024-03-28T17:34:26.702 INFO:teuthology.orchestra.run.smithi045.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 112 kB/s | 17 kB 00:00 2024-03-28T17:34:26.722 INFO:teuthology.orchestra.run.smithi151.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 606 kB/s | 192 kB 00:00 2024-03-28T17:34:26.747 INFO:teuthology.orchestra.run.smithi151.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 110 kB/s | 23 kB 00:00 2024-03-28T17:34:26.777 INFO:teuthology.orchestra.run.smithi045.stdout:(47/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 11 MB/s | 3.0 MB 00:00 2024-03-28T17:34:26.802 INFO:teuthology.orchestra.run.smithi045.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2024-03-28T17:34:26.827 INFO:teuthology.orchestra.run.smithi045.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 379 kB/s | 44 kB 00:00 2024-03-28T17:34:26.864 INFO:teuthology.orchestra.run.smithi151.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 712 kB/s | 256 kB 00:00 2024-03-28T17:34:26.903 INFO:teuthology.orchestra.run.smithi045.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 661 kB 00:00 2024-03-28T17:34:26.936 INFO:teuthology.orchestra.run.smithi045.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 4.3 MB/s | 578 kB 00:00 2024-03-28T17:34:26.961 INFO:teuthology.orchestra.run.smithi045.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 364 kB/s | 46 kB 00:00 2024-03-28T17:34:26.995 INFO:teuthology.orchestra.run.smithi045.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 282 kB 00:00 2024-03-28T17:34:27.037 INFO:teuthology.orchestra.run.smithi045.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.5 MB/s | 187 kB 00:00 2024-03-28T17:34:27.040 INFO:teuthology.orchestra.run.smithi151.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 342 kB/s | 108 kB 00:00 2024-03-28T17:34:27.063 INFO:teuthology.orchestra.run.smithi045.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.2 MB/s | 217 kB 00:00 2024-03-28T17:34:27.065 INFO:teuthology.orchestra.run.smithi151.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 488 kB/s | 155 kB 00:00 2024-03-28T17:34:27.088 INFO:teuthology.orchestra.run.smithi045.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 388 kB/s | 34 kB 00:00 2024-03-28T17:34:27.157 INFO:teuthology.orchestra.run.smithi151.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 465 kB/s | 136 kB 00:00 2024-03-28T17:34:27.180 INFO:teuthology.orchestra.run.smithi045.stdout:(57/174): protobuf-3.14.0-13.el9.x86_64.rpm 8.8 MB/s | 1.0 MB 00:00 2024-03-28T17:34:27.246 INFO:teuthology.orchestra.run.smithi045.stdout:(58/174): python3-attrs-20.3.0-7.el9.noarch.rpm 515 kB/s | 78 kB 00:00 2024-03-28T17:34:27.274 INFO:teuthology.orchestra.run.smithi151.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 138 kB/s | 29 kB 00:00 2024-03-28T17:34:27.291 INFO:teuthology.orchestra.run.smithi151.stdout:(167/174): libsmartcols-2.37.4-18.el9.x86_64.rp 254 kB/s | 64 kB 00:00 2024-03-28T17:34:27.372 INFO:teuthology.orchestra.run.smithi045.stdout:(59/174): openblas-openmp-0.3.21-2.el9.x86_64.r 15 MB/s | 4.7 MB 00:00 2024-03-28T17:34:27.506 INFO:teuthology.orchestra.run.smithi045.stdout:(60/174): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-28T17:34:27.522 INFO:teuthology.orchestra.run.smithi045.stdout:(61/174): python3-jsonpatch-1.21-16.el9.noarch. 97 kB/s | 24 kB 00:00 2024-03-28T17:34:27.539 INFO:teuthology.orchestra.run.smithi045.stdout:(62/174): python3-jsonpointer-2.0-4.el9.noarch. 119 kB/s | 18 kB 00:00 2024-03-28T17:34:27.557 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-28T17:34:27.557 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-28T17:34:27.572 INFO:teuthology.orchestra.run.smithi045.stdout:(63/174): python3-markupsafe-1.1.1-12.el9.x86_6 1.0 MB/s | 32 kB 00:00 2024-03-28T17:34:27.600 INFO:teuthology.orchestra.run.smithi151.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 2.7 MB/s | 1.2 MB 00:00 2024-03-28T17:34:27.614 INFO:teuthology.orchestra.run.smithi045.stdout:(64/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-28T17:34:27.731 INFO:teuthology.orchestra.run.smithi045.stdout:(65/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2024-03-28T17:34:27.756 INFO:teuthology.orchestra.run.smithi045.stdout:(66/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 217 kB 00:00 2024-03-28T17:34:27.775 INFO:teuthology.orchestra.run.smithi151.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 4.3 MB/s | 2.2 MB 00:00 2024-03-28T17:34:27.782 INFO:teuthology.orchestra.run.smithi045.stdout:(67/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.1 MB/s | 174 kB 00:00 2024-03-28T17:34:27.832 INFO:teuthology.orchestra.run.smithi045.stdout:(68/174): python3-prettytable-0.7.2-27.el9.noar 853 kB/s | 41 kB 00:00 2024-03-28T17:34:27.850 INFO:teuthology.orchestra.run.smithi151.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.1 MB/s | 2.3 MB 00:00 2024-03-28T17:34:27.866 INFO:teuthology.orchestra.run.smithi045.stdout:(69/174): python3-packaging-20.9-5.el9.noarch.r 837 kB/s | 69 kB 00:00 2024-03-28T17:34:27.893 INFO:teuthology.orchestra.run.smithi045.stdout:(70/174): python3-protobuf-3.14.0-13.el9.noarch 2.2 MB/s | 238 kB 00:00 2024-03-28T17:34:27.916 INFO:teuthology.orchestra.run.smithi045.stdout:(71/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 132 kB 00:00 2024-03-28T17:34:27.949 INFO:teuthology.orchestra.run.smithi045.stdout:(72/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.7 MB/s | 211 kB 00:00 2024-03-28T17:34:27.966 INFO:teuthology.orchestra.run.smithi045.stdout:(73/174): python3-pytz-2021.1-5.el9.noarch.rpm 651 kB/s | 47 kB 00:00 2024-03-28T17:34:27.992 INFO:teuthology.orchestra.run.smithi045.stdout:(74/174): python3-requests-oauthlib-1.3.0-12.el 593 kB/s | 43 kB 00:00 2024-03-28T17:34:28.001 INFO:teuthology.orchestra.run.smithi151.stdout:(171/174): util-linux-core-2.37.4-18.el9.x86_64 1.1 MB/s | 465 kB 00:00 2024-03-28T17:34:28.025 INFO:teuthology.orchestra.run.smithi045.stdout:(75/174): python3-toml-0.10.2-6.el9.noarch.rpm 680 kB/s | 39 kB 00:00 2024-03-28T17:34:28.068 INFO:teuthology.orchestra.run.smithi151.stdout:(172/174): util-linux-user-2.37.4-18.el9.x86_64 107 kB/s | 31 kB 00:00 2024-03-28T17:34:28.243 INFO:teuthology.orchestra.run.smithi151.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 308 kB/s | 74 kB 00:00 2024-03-28T17:34:28.368 INFO:teuthology.orchestra.run.smithi151.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 2.2 MB/s | 1.1 MB 00:00 2024-03-28T17:34:28.373 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-28T17:34:28.373 INFO:teuthology.orchestra.run.smithi151.stdout:Total 12 MB/s | 225 MB 00:18 2024-03-28T17:34:28.401 INFO:teuthology.orchestra.run.smithi045.stdout:(76/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-03-28T17:34:28.426 INFO:teuthology.orchestra.run.smithi045.stdout:(77/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 96 kB/s | 41 kB 00:00 2024-03-28T17:34:28.451 INFO:teuthology.orchestra.run.smithi045.stdout:(78/174): socat-1.7.4.1-5.el9.x86_64.rpm 732 kB/s | 300 kB 00:00 2024-03-28T17:34:28.493 INFO:teuthology.orchestra.run.smithi045.stdout:(79/174): uuid-1.6.2-55.el9.x86_64.rpm 1.4 MB/s | 56 kB 00:00 2024-03-28T17:34:28.518 INFO:teuthology.orchestra.run.smithi045.stdout:(80/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 963 kB/s | 63 kB 00:00 2024-03-28T17:34:28.543 INFO:teuthology.orchestra.run.smithi045.stdout:(81/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 489 kB/s | 44 kB 00:00 2024-03-28T17:34:28.568 INFO:teuthology.orchestra.run.smithi045.stdout:(82/174): fuse-2.9.9-15.el9.x86_64.rpm 1.6 MB/s | 79 kB 00:00 2024-03-28T17:34:28.593 INFO:teuthology.orchestra.run.smithi045.stdout:(83/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.2 MB/s | 241 kB 00:00 2024-03-28T17:34:28.644 INFO:teuthology.orchestra.run.smithi045.stdout:(84/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-03-28T17:34:28.669 INFO:teuthology.orchestra.run.smithi045.stdout:(85/174): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 103 kB 00:00 2024-03-28T17:34:28.694 INFO:teuthology.orchestra.run.smithi045.stdout:(86/174): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 124 kB 00:00 2024-03-28T17:34:28.728 INFO:teuthology.orchestra.run.smithi045.stdout:(87/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.3 MB/s | 187 kB 00:00 2024-03-28T17:34:28.887 INFO:teuthology.orchestra.run.smithi045.stdout:(88/174): jq-1.6-16.el9.x86_64.rpm 990 kB/s | 187 kB 00:00 2024-03-28T17:34:28.920 INFO:teuthology.orchestra.run.smithi045.stdout:(89/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-28T17:34:28.948 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-28T17:34:28.962 INFO:teuthology.orchestra.run.smithi045.stdout:(90/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.0 MB/s | 803 kB 00:00 2024-03-28T17:34:28.988 INFO:teuthology.orchestra.run.smithi045.stdout:(91/174): python3-requests-2.25.1-8.el9.noarch. 1.9 MB/s | 125 kB 00:00 2024-03-28T17:34:29.046 INFO:teuthology.orchestra.run.smithi045.stdout:(92/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 308 kB/s | 25 kB 00:00 2024-03-28T17:34:29.063 INFO:teuthology.orchestra.run.smithi045.stdout:(93/174): elfutils-libelf-devel-0.190-2.el9.x86 153 kB/s | 21 kB 00:00 2024-03-28T17:34:29.226 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-28T17:34:29.226 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-28T17:34:29.272 INFO:teuthology.orchestra.run.smithi045.stdout:(94/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 15 MB/s | 4.1 MB 00:00 2024-03-28T17:34:29.414 INFO:teuthology.orchestra.run.smithi045.stdout:(95/174): postgresql-contrib-13.14-1.el9.x86_64 2.5 MB/s | 911 kB 00:00 2024-03-28T17:34:29.456 INFO:teuthology.orchestra.run.smithi045.stdout:(96/174): postgresql-private-libs-13.14-1.el9.x 730 kB/s | 133 kB 00:00 2024-03-28T17:34:29.506 INFO:teuthology.orchestra.run.smithi045.stdout:(97/174): postgresql-13.14-1.el9.x86_64.rpm 3.5 MB/s | 1.6 MB 00:00 2024-03-28T17:34:29.598 INFO:teuthology.orchestra.run.smithi045.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.7 MB/s | 248 kB 00:00 2024-03-28T17:34:29.640 INFO:teuthology.orchestra.run.smithi045.stdout:(99/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-03-28T17:34:29.715 INFO:teuthology.orchestra.run.smithi045.stdout:(100/174): qatlib-service-24.02.0-1.el9.x86_64. 484 kB/s | 36 kB 00:00 2024-03-28T17:34:29.749 INFO:teuthology.orchestra.run.smithi045.stdout:(101/174): qatlib-24.02.0-1.el9.x86_64.rpm 1.4 MB/s | 221 kB 00:00 2024-03-28T17:34:29.874 INFO:teuthology.orchestra.run.smithi045.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 755 kB/s | 94 kB 00:00 2024-03-28T17:34:29.899 INFO:teuthology.orchestra.run.smithi045.stdout:(103/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 259 kB/s | 47 kB 00:00 2024-03-28T17:34:30.041 INFO:teuthology.orchestra.run.smithi045.stdout:(104/174): postgresql-server-13.14-1.el9.x86_64 9.2 MB/s | 5.8 MB 00:00 2024-03-28T17:34:30.098 INFO:teuthology.orchestra.run.smithi045.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 100 kB/s | 22 kB 00:00 2024-03-28T17:34:30.325 INFO:teuthology.orchestra.run.smithi045.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.0 MB/s | 863 kB 00:00 2024-03-28T17:34:30.392 INFO:teuthology.orchestra.run.smithi045.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 302 kB/s | 19 kB 00:00 2024-03-28T17:34:30.425 INFO:teuthology.orchestra.run.smithi045.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.4 MB/s | 548 kB 00:00 2024-03-28T17:34:30.459 INFO:teuthology.orchestra.run.smithi045.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 858 kB/s | 309 kB 00:00 2024-03-28T17:34:30.764 INFO:teuthology.orchestra.run.smithi045.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 78 kB/s | 26 kB 00:00 2024-03-28T17:34:30.793 INFO:teuthology.orchestra.run.smithi045.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 146 kB/s | 49 kB 00:00 2024-03-28T17:34:30.951 INFO:teuthology.orchestra.run.smithi045.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 428 kB/s | 67 kB 00:00 2024-03-28T17:34:30.973 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-28T17:34:30.974 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-28T17:34:30.985 INFO:teuthology.orchestra.run.smithi045.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 792 kB/s | 151 kB 00:00 2024-03-28T17:34:31.277 INFO:teuthology.orchestra.run.smithi045.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 1.9 MB/s | 548 kB 00:00 2024-03-28T17:34:31.335 INFO:teuthology.orchestra.run.smithi045.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.2 MB/s | 838 kB 00:00 2024-03-28T17:34:31.361 INFO:teuthology.orchestra.run.smithi045.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 536 kB/s | 43 kB 00:00 2024-03-28T17:34:31.461 INFO:teuthology.orchestra.run.smithi045.stdout:(117/174): libarrow-9.0.0-10.el9.x86_64.rpm 4.1 MB/s | 4.4 MB 00:01 2024-03-28T17:34:31.486 INFO:teuthology.orchestra.run.smithi045.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 219 kB/s | 32 kB 00:00 2024-03-28T17:34:31.511 INFO:teuthology.orchestra.run.smithi045.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 102 kB/s | 15 kB 00:00 2024-03-28T17:34:31.537 INFO:teuthology.orchestra.run.smithi045.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-03-28T17:34:31.579 INFO:teuthology.orchestra.run.smithi045.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.9 MB/s | 358 kB 00:00 2024-03-28T17:34:31.604 INFO:teuthology.orchestra.run.smithi045.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.4 MB/s | 303 kB 00:00 2024-03-28T17:34:31.629 INFO:teuthology.orchestra.run.smithi045.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 403 kB/s | 35 kB 00:00 2024-03-28T17:34:31.663 INFO:teuthology.orchestra.run.smithi045.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 3.5 MB/s | 203 kB 00:00 2024-03-28T17:34:31.738 INFO:teuthology.orchestra.run.smithi045.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 152 kB/s | 11 kB 00:00 2024-03-28T17:34:31.763 INFO:teuthology.orchestra.run.smithi045.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-03-28T17:34:31.814 INFO:teuthology.orchestra.run.smithi045.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 9.8 MB/s | 2.0 MB 00:00 2024-03-28T17:34:31.839 INFO:teuthology.orchestra.run.smithi045.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 183 kB/s | 18 kB 00:00 2024-03-28T17:34:31.864 INFO:teuthology.orchestra.run.smithi045.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 244 kB/s | 23 kB 00:00 2024-03-28T17:34:31.890 INFO:teuthology.orchestra.run.smithi045.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 398 kB/s | 19 kB 00:00 2024-03-28T17:34:31.906 INFO:teuthology.orchestra.run.smithi045.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 300 kB/s | 20 kB 00:00 2024-03-28T17:34:31.973 INFO:teuthology.orchestra.run.smithi045.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 755 kB/s | 46 kB 00:00 2024-03-28T17:34:32.015 INFO:teuthology.orchestra.run.smithi045.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2024-03-28T17:34:32.040 INFO:teuthology.orchestra.run.smithi045.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 615 kB/s | 79 kB 00:00 2024-03-28T17:34:32.066 INFO:teuthology.orchestra.run.smithi045.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 640 kB/s | 58 kB 00:00 2024-03-28T17:34:32.099 INFO:teuthology.orchestra.run.smithi045.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 4.7 MB/s | 272 kB 00:00 2024-03-28T17:34:32.116 INFO:teuthology.orchestra.run.smithi045.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 223 kB/s | 16 kB 00:00 2024-03-28T17:34:32.141 INFO:teuthology.orchestra.run.smithi045.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-28T17:34:32.191 INFO:teuthology.orchestra.run.smithi045.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 385 kB/s | 28 kB 00:00 2024-03-28T17:34:32.208 INFO:teuthology.orchestra.run.smithi045.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 342 kB/s | 31 kB 00:00 2024-03-28T17:34:32.242 INFO:teuthology.orchestra.run.smithi045.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.9 MB/s | 188 kB 00:00 2024-03-28T17:34:32.267 INFO:teuthology.orchestra.run.smithi045.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-28T17:34:32.293 INFO:teuthology.orchestra.run.smithi045.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 444 kB/s | 36 kB 00:00 2024-03-28T17:34:32.318 INFO:teuthology.orchestra.run.smithi045.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 693 kB/s | 51 kB 00:00 2024-03-28T17:34:32.351 INFO:teuthology.orchestra.run.smithi045.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.9 MB/s | 230 kB 00:00 2024-03-28T17:34:32.376 INFO:teuthology.orchestra.run.smithi045.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-03-28T17:34:32.427 INFO:teuthology.orchestra.run.smithi045.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.0 MB/s | 427 kB 00:00 2024-03-28T17:34:32.452 INFO:teuthology.orchestra.run.smithi045.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 285 kB/s | 20 kB 00:00 2024-03-28T17:34:32.477 INFO:teuthology.orchestra.run.smithi045.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-28T17:34:32.594 INFO:teuthology.orchestra.run.smithi045.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 9.6 MB/s | 1.6 MB 00:00 2024-03-28T17:34:32.669 INFO:teuthology.orchestra.run.smithi045.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 4.2 MB/s | 321 kB 00:00 2024-03-28T17:34:32.845 INFO:teuthology.orchestra.run.smithi045.stdout:(152/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 8.7 MB/s | 3.1 MB 00:00 2024-03-28T17:34:32.962 INFO:teuthology.orchestra.run.smithi045.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 7.9 MB/s | 2.3 MB 00:00 2024-03-28T17:34:33.012 INFO:teuthology.orchestra.run.smithi045.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 855 kB/s | 42 kB 00:00 2024-03-28T17:34:33.128 INFO:teuthology.orchestra.run.smithi045.stdout:(155/174): librados2-19.0.0-2145.g3be9a5c9.el9. 5.3 MB/s | 3.4 MB 00:00 2024-03-28T17:34:33.168 INFO:teuthology.orchestra.run.smithi045.stdout:(156/174): NetworkManager-tui-1.47.2-1.el9.x86_ 1.6 MB/s | 248 kB 00:00 2024-03-28T17:34:33.227 INFO:teuthology.orchestra.run.smithi045.stdout:(157/174): NetworkManager-libnm-1.47.2-1.el9.x8 4.8 MB/s | 1.8 MB 00:00 2024-03-28T17:34:33.252 INFO:teuthology.orchestra.run.smithi045.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 580 kB/s | 71 kB 00:00 2024-03-28T17:34:33.278 INFO:teuthology.orchestra.run.smithi045.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 337 kB/s | 36 kB 00:00 2024-03-28T17:34:33.311 INFO:teuthology.orchestra.run.smithi045.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.3 MB/s | 192 kB 00:00 2024-03-28T17:34:33.336 INFO:teuthology.orchestra.run.smithi045.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 398 kB/s | 23 kB 00:00 2024-03-28T17:34:33.370 INFO:teuthology.orchestra.run.smithi045.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.1 MB/s | 256 kB 00:00 2024-03-28T17:34:33.395 INFO:teuthology.orchestra.run.smithi045.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 108 kB 00:00 2024-03-28T17:34:33.421 INFO:teuthology.orchestra.run.smithi045.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 155 kB 00:00 2024-03-28T17:34:33.446 INFO:teuthology.orchestra.run.smithi045.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 136 kB 00:00 2024-03-28T17:34:33.471 INFO:teuthology.orchestra.run.smithi045.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 856 kB/s | 64 kB 00:00 2024-03-28T17:34:33.496 INFO:teuthology.orchestra.run.smithi045.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 385 kB/s | 29 kB 00:00 2024-03-28T17:34:33.572 INFO:teuthology.orchestra.run.smithi045.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 9.7 MB/s | 1.2 MB 00:00 2024-03-28T17:34:33.655 INFO:teuthology.orchestra.run.smithi045.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 12 MB/s | 2.2 MB 00:00 2024-03-28T17:34:33.697 INFO:teuthology.orchestra.run.smithi045.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 3.7 MB/s | 465 kB 00:00 2024-03-28T17:34:33.714 INFO:teuthology.orchestra.run.smithi045.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 539 kB/s | 31 kB 00:00 2024-03-28T17:34:33.831 INFO:teuthology.orchestra.run.smithi045.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 6.8 MB/s | 2.3 MB 00:00 2024-03-28T17:34:33.881 INFO:teuthology.orchestra.run.smithi045.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.9 MB/s | 1.1 MB 00:00 2024-03-28T17:34:33.906 INFO:teuthology.orchestra.run.smithi045.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 390 kB/s | 74 kB 00:00 2024-03-28T17:34:33.919 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-28T17:34:33.920 INFO:teuthology.orchestra.run.smithi045.stdout:Total 15 MB/s | 225 MB 00:15 2024-03-28T17:34:34.461 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-28T17:34:34.577 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T17:34:34.603 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T17:34:34.642 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T17:34:34.652 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T17:34:34.692 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T17:34:34.745 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T17:34:34.779 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T17:34:34.813 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T17:34:34.872 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T17:34:35.191 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T17:34:35.255 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T17:34:35.360 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T17:34:35.398 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T17:34:35.490 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T17:34:35.528 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T17:34:35.568 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T17:34:35.603 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-28T17:34:35.735 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T17:34:35.791 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T17:34:35.820 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T17:34:35.923 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T17:34:35.946 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T17:34:36.003 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-28T17:34:36.003 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-28T17:34:36.010 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T17:34:36.049 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T17:34:36.072 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T17:34:36.089 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T17:34:36.255 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T17:34:36.286 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T17:34:36.318 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T17:34:36.430 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T17:34:36.537 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T17:34:36.597 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T17:34:36.671 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T17:34:36.748 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T17:34:36.948 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T17:34:37.052 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T17:34:37.149 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T17:34:37.186 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T17:34:37.213 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T17:34:37.248 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T17:34:37.331 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T17:34:37.385 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T17:34:37.423 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T17:34:37.487 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T17:34:37.522 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T17:34:37.566 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T17:34:37.598 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T17:34:37.626 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T17:34:37.696 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T17:34:37.739 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T17:34:37.783 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T17:34:37.811 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T17:34:37.882 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T17:34:37.926 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T17:34:38.000 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T17:34:38.002 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-28T17:34:38.123 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T17:34:38.133 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T17:34:38.157 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T17:34:38.186 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T17:34:38.196 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T17:34:38.206 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T17:34:38.246 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T17:34:38.299 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T17:34:38.334 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T17:34:38.368 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T17:34:38.410 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T17:34:38.447 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-28T17:34:38.448 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-28T17:34:38.712 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T17:34:38.729 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T17:34:38.799 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T17:34:38.803 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T17:34:38.891 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T17:34:38.924 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T17:34:38.935 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T17:34:38.945 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T17:34:38.954 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T17:34:38.975 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T17:34:39.024 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T17:34:39.045 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T17:34:39.069 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T17:34:39.075 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T17:34:39.098 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T17:34:39.112 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T17:34:39.292 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T17:34:39.303 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T17:34:39.349 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T17:34:39.378 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T17:34:39.477 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T17:34:39.481 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T17:34:39.504 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T17:34:39.551 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T17:34:39.568 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T17:34:39.595 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T17:34:39.600 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T17:34:39.623 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T17:34:39.638 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T17:34:39.798 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T17:34:39.829 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T17:34:39.861 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T17:34:39.903 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T17:34:39.964 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T17:34:40.089 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T17:34:40.106 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T17:34:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T17:34:40.161 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T17:34:40.224 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T17:34:40.233 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T17:34:40.285 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T17:34:40.407 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T17:34:40.444 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T17:34:40.491 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T17:34:40.499 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T17:34:40.526 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T17:34:40.569 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T17:34:40.609 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T17:34:40.619 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T17:34:40.713 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T17:34:40.750 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T17:34:40.764 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T17:34:40.776 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T17:34:40.812 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T17:34:40.817 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T17:34:40.855 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T17:34:40.887 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T17:34:40.905 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T17:34:40.940 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T17:34:40.979 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T17:34:40.999 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T17:34:41.038 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T17:34:41.044 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T17:34:41.078 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T17:34:41.078 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T17:34:41.104 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T17:34:41.129 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T17:34:41.131 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T17:34:41.163 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T17:34:41.191 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T17:34:41.200 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T17:34:41.262 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T17:34:41.297 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T17:34:41.329 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T17:34:41.340 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T17:34:41.369 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T17:34:41.430 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T17:34:41.447 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T17:34:41.479 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T17:34:41.492 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T17:34:41.535 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T17:34:41.566 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T17:34:41.590 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T17:34:41.709 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T17:34:41.762 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T17:34:41.775 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T17:34:41.823 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T17:34:41.960 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T17:34:41.997 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T17:34:42.029 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T17:34:42.060 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T17:34:42.088 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T17:34:42.130 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T17:34:42.178 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T17:34:42.210 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T17:34:42.239 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T17:34:42.282 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T17:34:42.316 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T17:34:42.350 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T17:34:42.358 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T17:34:42.383 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T17:34:42.425 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T17:34:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T17:34:42.459 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T17:34:42.471 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T17:34:42.482 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T17:34:42.511 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T17:34:42.549 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T17:34:42.560 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T17:34:42.624 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T17:34:42.630 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T17:34:42.653 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T17:34:42.681 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T17:34:42.713 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T17:34:42.760 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T17:34:42.865 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T17:34:42.886 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T17:34:42.915 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T17:34:43.059 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T17:34:43.117 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T17:34:43.169 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T17:34:43.308 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T17:34:43.356 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T17:34:43.411 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T17:34:43.484 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T17:34:43.525 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T17:34:43.554 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T17:34:43.612 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T17:34:43.637 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T17:34:43.686 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T17:34:43.720 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T17:34:43.721 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T17:34:43.765 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T17:34:43.768 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T17:34:43.807 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T17:34:43.847 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T17:34:43.879 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T17:34:43.940 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T17:34:43.988 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T17:34:44.018 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T17:34:44.057 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T17:34:44.059 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T17:34:44.094 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T17:34:44.152 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T17:34:44.179 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T17:34:44.193 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T17:34:44.234 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T17:34:44.260 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T17:34:44.285 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T17:34:44.356 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T17:34:44.477 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T17:34:44.576 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T17:34:44.620 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T17:34:44.667 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T17:34:44.723 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T17:34:44.820 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T17:34:44.906 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T17:34:44.932 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T17:34:44.950 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T17:34:45.076 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T17:34:45.123 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T17:34:45.164 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T17:34:45.195 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T17:34:45.223 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T17:34:45.257 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T17:34:45.297 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T17:34:45.328 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T17:34:45.359 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T17:34:45.434 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T17:34:45.470 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T17:34:45.503 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T17:34:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T17:34:45.596 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T17:34:45.684 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T17:34:45.775 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T17:34:45.819 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T17:34:45.851 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T17:34:45.898 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T17:34:45.995 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T17:34:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T17:34:46.091 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-28T17:34:46.158 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T17:34:46.180 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T17:34:46.219 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T17:34:46.252 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T17:34:46.259 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T17:34:46.283 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T17:34:46.303 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T17:34:46.322 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T17:34:46.350 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T17:34:46.390 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T17:34:46.654 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T17:34:46.662 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T17:34:46.688 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T17:34:46.716 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T17:34:46.749 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T17:34:46.766 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T17:34:46.787 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T17:34:46.809 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T17:34:46.817 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T17:34:46.844 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T17:34:46.854 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T17:34:46.892 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T17:34:46.946 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T17:34:46.970 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T17:34:47.012 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T17:34:47.188 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T17:34:47.218 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T17:34:47.295 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T17:34:47.329 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T17:34:47.357 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T17:34:47.418 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T17:34:47.428 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T17:34:47.456 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T17:34:47.459 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T17:34:47.639 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T17:34:47.769 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T17:34:47.906 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T17:34:47.924 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T17:34:47.945 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T17:34:48.027 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T17:34:48.040 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T17:34:48.353 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T17:34:48.417 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T17:34:48.441 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T17:34:48.609 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T17:34:48.643 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T17:34:48.742 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T17:34:48.795 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T17:34:48.828 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T17:34:48.893 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T17:34:49.239 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T17:34:49.243 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T17:34:49.304 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T17:34:49.334 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T17:34:49.353 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T17:34:49.366 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T17:34:49.397 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T17:34:49.425 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T17:34:49.678 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T17:34:49.707 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T17:34:49.739 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T17:34:49.745 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T17:34:49.780 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T17:34:49.869 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T17:34:49.871 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T17:34:49.910 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T17:34:49.946 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T17:34:49.950 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T17:34:50.149 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T17:34:50.205 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T17:34:50.275 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T17:34:50.317 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T17:34:50.381 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T17:34:50.381 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T17:34:50.381 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T17:34:50.381 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-28T17:34:50.405 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T17:34:50.409 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T17:34:50.434 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T17:34:50.505 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T17:34:50.536 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T17:34:50.684 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T17:34:51.081 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T17:34:51.216 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T17:34:51.253 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T17:34:51.303 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T17:34:51.368 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T17:34:51.493 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T17:34:51.565 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T17:34:51.621 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T17:34:51.726 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T17:34:51.777 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T17:34:51.842 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T17:34:51.897 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T17:34:51.939 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T17:34:52.064 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T17:34:52.109 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T17:34:52.172 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T17:34:52.208 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T17:34:52.336 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T17:34:52.403 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T17:34:52.505 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T17:34:52.687 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T17:34:52.756 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T17:34:52.794 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T17:34:52.826 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T17:34:53.361 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T17:34:53.405 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T17:34:53.485 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T17:34:53.485 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T17:34:53.486 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T17:34:53.486 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-28T17:34:53.509 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T17:34:53.514 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T17:34:53.616 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T17:34:53.750 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T17:34:53.812 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T17:34:53.843 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T17:34:53.891 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T17:34:53.952 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T17:34:54.005 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T17:34:54.052 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T17:34:54.478 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T17:34:54.749 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T17:34:54.829 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T17:34:54.883 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T17:34:55.084 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T17:34:55.139 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T17:34:55.253 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T17:34:55.348 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T17:34:55.397 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T17:34:55.437 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T17:34:55.507 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T17:34:55.554 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T17:34:55.622 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T17:34:55.715 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T17:34:55.930 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T17:34:55.985 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T17:34:56.024 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T17:34:56.084 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T17:34:56.226 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T17:34:56.271 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T17:34:56.319 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T17:34:56.354 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T17:34:56.387 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T17:34:56.565 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T17:34:56.773 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T17:34:56.917 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T17:34:56.973 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T17:34:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T17:34:57.137 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T17:34:57.403 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T17:34:57.461 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T17:34:57.636 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T17:34:57.694 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T17:34:57.741 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T17:34:57.779 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T17:34:57.830 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T17:34:57.878 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T17:34:57.946 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T17:34:58.002 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T17:34:58.077 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T17:34:58.224 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T17:34:58.270 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T17:34:58.306 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T17:34:58.370 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T17:34:58.416 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T17:34:58.567 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T17:34:58.704 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T17:34:58.762 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T17:34:58.807 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T17:34:58.859 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T17:34:58.996 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T17:34:59.073 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T17:35:00.076 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T17:35:00.130 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T17:35:00.163 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T17:35:00.221 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T17:35:00.337 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T17:35:00.373 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T17:35:00.616 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T17:35:00.652 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T17:35:00.748 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T17:35:01.601 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T17:35:01.738 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T17:35:01.941 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T17:35:01.941 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /sys 2024-03-28T17:35:01.941 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /proc 2024-03-28T17:35:01.941 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /mnt 2024-03-28T17:35:01.941 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /var/tmp 2024-03-28T17:35:01.942 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /home 2024-03-28T17:35:01.942 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /root 2024-03-28T17:35:01.942 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /tmp 2024-03-28T17:35:01.942 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-28T17:35:02.104 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T17:35:02.120 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T17:35:02.120 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:02.120 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T17:35:02.120 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-28T17:35:03.359 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T17:35:03.450 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T17:35:03.487 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T17:35:03.520 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T17:35:03.560 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T17:35:03.591 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T17:35:03.616 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T17:35:03.616 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:03.616 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T17:35:03.616 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T17:35:03.617 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T17:35:03.617 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-28T17:35:03.841 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T17:35:03.864 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T17:35:03.864 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:03.864 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T17:35:03.864 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T17:35:03.864 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T17:35:03.864 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-28T17:35:03.951 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T17:35:03.991 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T17:35:04.114 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T17:35:04.148 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T17:35:04.205 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T17:35:04.308 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T17:35:04.330 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T17:35:04.330 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:04.331 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T17:35:04.331 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T17:35:04.331 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T17:35:04.331 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-28T17:35:04.418 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T17:35:04.772 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T17:35:04.802 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T17:35:04.869 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T17:35:04.900 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T17:35:04.903 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T17:35:04.930 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T17:35:04.931 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T17:35:04.992 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T17:35:05.023 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T17:35:05.256 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T17:35:05.291 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T17:35:05.642 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T17:35:05.642 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /sys 2024-03-28T17:35:05.642 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /proc 2024-03-28T17:35:05.642 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /mnt 2024-03-28T17:35:05.642 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /var/tmp 2024-03-28T17:35:05.642 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /home 2024-03-28T17:35:05.642 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /root 2024-03-28T17:35:05.642 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /tmp 2024-03-28T17:35:05.642 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-28T17:35:05.806 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T17:35:05.822 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T17:35:05.822 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:05.822 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T17:35:05.822 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-28T17:35:06.247 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T17:35:06.282 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T17:35:06.383 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T17:35:06.546 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T17:35:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T17:35:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T17:35:06.600 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:06.600 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T17:35:06.600 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T17:35:06.600 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T17:35:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-28T17:35:06.668 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T17:35:06.680 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T17:35:06.725 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T17:35:07.310 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T17:35:07.336 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T17:35:07.336 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:07.336 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T17:35:07.336 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T17:35:07.336 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T17:35:07.336 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-28T17:35:07.553 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T17:35:07.575 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T17:35:07.575 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:07.575 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T17:35:07.575 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T17:35:07.575 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T17:35:07.575 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-28T17:35:07.599 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T17:35:07.629 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T17:35:07.631 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T17:35:07.652 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T17:35:07.652 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:07.652 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T17:35:07.653 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T17:35:07.653 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T17:35:07.653 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-28T17:35:07.677 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T17:35:07.696 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T17:35:07.718 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T17:35:07.718 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:07.719 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T17:35:07.719 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-28T17:35:07.763 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T17:35:07.786 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T17:35:07.786 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:07.786 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T17:35:07.786 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T17:35:07.786 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T17:35:07.786 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-28T17:35:08.012 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T17:35:08.035 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T17:35:08.035 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:08.035 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T17:35:08.035 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T17:35:08.035 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T17:35:08.035 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-28T17:35:08.089 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T17:35:08.112 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T17:35:08.112 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:08.112 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T17:35:08.112 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T17:35:08.112 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T17:35:08.112 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-28T17:35:08.131 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T17:35:08.346 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T17:35:08.389 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T17:35:08.516 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T17:35:08.547 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T17:35:08.576 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T17:35:08.576 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T17:35:08.576 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T17:35:08.576 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:35:08.579 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T17:35:08.623 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T17:35:08.639 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T17:35:08.670 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T17:35:08.902 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T17:35:08.938 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T17:35:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T17:35:09.922 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T17:35:10.022 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T17:35:10.185 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T17:35:10.208 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T17:35:10.232 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T17:35:10.232 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:10.232 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T17:35:10.232 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T17:35:10.232 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T17:35:10.232 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-28T17:35:10.276 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T17:35:10.287 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T17:35:10.332 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T17:35:11.205 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T17:35:11.229 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T17:35:11.252 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T17:35:11.252 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:11.252 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T17:35:11.252 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T17:35:11.253 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T17:35:11.253 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-28T17:35:11.313 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T17:35:11.334 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T17:35:11.334 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:11.334 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T17:35:11.335 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-28T17:35:11.388 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T17:35:11.411 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T17:35:11.411 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:11.411 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T17:35:11.411 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T17:35:11.411 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T17:35:11.411 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-28T17:35:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T17:35:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T17:35:11.746 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:11.746 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T17:35:11.746 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T17:35:11.746 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T17:35:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-28T17:35:12.545 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T17:35:12.585 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T17:35:12.624 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T17:35:12.660 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T17:35:12.849 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T17:35:12.963 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T17:35:13.008 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T17:35:13.047 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T17:35:13.116 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T17:35:13.159 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T17:35:13.192 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T17:35:13.232 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T17:35:13.262 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T17:35:13.318 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T17:35:13.377 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T17:35:13.411 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T17:35:13.980 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T17:35:14.025 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T17:35:14.045 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T17:35:14.861 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T17:35:14.945 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T17:35:14.985 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T17:35:15.065 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T17:35:15.098 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T17:35:15.123 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T17:35:15.148 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T17:35:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T17:35:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T17:35:15.280 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T17:35:15.309 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T17:35:15.310 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T17:35:15.394 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T17:35:15.420 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T17:35:15.462 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T17:35:15.496 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T17:35:15.520 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T17:35:15.554 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T17:35:15.593 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T17:35:15.625 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T17:35:15.650 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T17:35:15.685 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T17:35:15.709 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T17:35:15.735 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T17:35:15.742 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T17:35:15.742 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T17:35:15.921 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T17:35:15.921 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T17:35:16.003 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T17:35:16.028 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T17:35:16.059 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T17:35:16.088 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T17:35:16.121 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T17:35:16.171 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T17:35:16.185 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T17:35:16.211 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T17:35:16.242 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T17:35:16.270 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T17:35:16.451 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T17:35:16.562 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T17:35:16.602 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T17:35:16.642 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T17:35:16.712 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T17:35:16.747 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T17:35:16.771 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T17:35:16.804 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T17:35:16.833 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T17:35:16.888 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T17:35:16.948 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T17:35:16.983 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T17:35:17.571 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T17:35:17.608 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T17:35:17.628 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T17:35:18.461 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T17:35:18.539 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T17:35:18.580 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T17:35:18.662 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T17:35:18.703 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T17:35:18.728 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T17:35:18.753 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T17:35:18.780 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T17:35:18.780 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T17:35:18.868 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T17:35:18.898 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T17:35:18.898 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T17:35:18.983 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T17:35:19.009 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T17:35:19.042 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T17:35:19.068 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T17:35:19.093 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T17:35:19.126 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T17:35:19.166 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T17:35:19.199 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T17:35:19.233 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T17:35:19.258 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T17:35:19.284 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T17:35:19.309 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T17:35:19.316 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T17:35:19.316 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T17:35:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T17:35:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T17:35:19.604 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T17:35:19.629 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T17:35:19.661 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T17:35:19.688 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T17:35:19.713 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T17:35:19.778 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T17:35:21.264 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T17:35:21.265 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T17:35:21.265 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T17:35:21.265 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T17:35:21.265 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T17:35:21.265 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T17:35:21.265 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T17:35:21.265 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T17:35:21.265 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T17:35:21.265 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T17:35:21.265 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T17:35:21.265 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T17:35:21.266 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T17:35:21.266 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T17:35:21.266 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T17:35:21.266 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T17:35:21.266 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T17:35:21.266 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T17:35:21.266 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T17:35:21.266 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T17:35:21.266 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T17:35:21.266 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T17:35:21.266 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T17:35:21.266 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T17:35:21.266 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T17:35:21.267 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T17:35:21.267 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T17:35:21.267 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T17:35:21.267 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T17:35:21.267 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T17:35:21.267 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T17:35:21.267 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T17:35:21.283 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T17:35:21.283 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T17:35:21.283 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T17:35:21.283 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T17:35:21.283 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T17:35:21.283 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T17:35:21.283 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T17:35:21.283 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T17:35:21.284 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T17:35:21.284 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T17:35:21.284 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T17:35:21.284 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T17:35:21.284 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T17:35:21.284 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T17:35:21.284 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T17:35:21.284 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T17:35:21.284 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T17:35:21.284 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T17:35:21.284 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T17:35:21.284 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T17:35:21.284 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T17:35:21.285 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T17:35:21.285 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T17:35:21.285 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T17:35:21.285 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T17:35:21.285 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T17:35:21.285 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T17:35:21.285 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T17:35:21.285 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T17:35:21.285 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T17:35:21.285 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T17:35:21.285 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T17:35:21.285 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T17:35:21.285 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T17:35:21.286 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T17:35:21.286 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T17:35:21.286 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T17:35:21.286 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T17:35:21.286 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T17:35:21.286 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T17:35:21.286 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T17:35:21.286 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T17:35:21.286 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T17:35:21.286 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T17:35:21.286 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T17:35:21.286 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T17:35:21.287 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T17:35:21.287 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T17:35:21.287 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T17:35:21.287 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T17:35:21.287 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T17:35:21.287 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T17:35:21.287 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T17:35:21.287 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T17:35:21.287 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T17:35:21.287 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T17:35:21.287 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T17:35:21.287 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T17:35:21.287 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T17:35:21.287 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T17:35:21.288 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T17:35:21.288 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T17:35:21.288 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T17:35:21.288 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T17:35:21.288 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T17:35:21.288 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T17:35:21.288 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T17:35:21.288 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T17:35:21.288 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T17:35:21.288 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T17:35:21.288 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T17:35:21.288 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T17:35:21.288 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T17:35:21.288 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T17:35:21.289 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T17:35:21.289 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T17:35:21.289 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T17:35:21.289 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T17:35:21.289 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T17:35:21.289 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T17:35:21.289 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T17:35:21.289 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T17:35:21.289 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T17:35:21.289 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T17:35:21.289 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T17:35:21.289 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T17:35:21.289 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T17:35:21.290 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T17:35:21.290 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T17:35:21.290 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T17:35:21.290 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T17:35:21.290 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T17:35:21.290 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T17:35:21.290 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T17:35:21.290 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T17:35:21.290 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T17:35:21.290 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T17:35:21.290 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T17:35:21.290 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T17:35:21.290 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T17:35:21.291 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T17:35:21.291 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T17:35:21.291 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T17:35:21.291 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T17:35:21.291 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T17:35:21.291 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T17:35:21.291 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T17:35:21.291 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T17:35:21.291 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T17:35:21.291 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T17:35:21.291 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T17:35:21.291 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T17:35:21.291 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T17:35:21.292 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T17:35:21.292 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T17:35:21.292 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T17:35:21.292 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T17:35:21.292 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T17:35:21.292 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T17:35:21.292 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T17:35:21.292 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T17:35:21.292 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T17:35:21.292 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T17:35:21.292 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T17:35:21.292 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T17:35:21.292 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T17:35:21.293 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T17:35:21.293 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T17:35:21.293 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T17:35:21.293 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T17:35:21.293 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T17:35:21.293 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T17:35:21.293 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T17:35:21.293 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T17:35:21.293 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T17:35:21.293 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T17:35:21.293 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T17:35:21.293 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T17:35:21.293 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T17:35:21.294 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T17:35:21.294 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T17:35:21.294 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T17:35:21.294 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T17:35:21.294 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T17:35:21.294 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T17:35:21.294 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T17:35:21.294 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T17:35:21.294 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T17:35:21.294 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T17:35:21.294 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T17:35:21.294 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T17:35:21.295 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T17:35:21.295 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T17:35:21.295 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T17:35:21.295 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T17:35:21.295 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T17:35:21.295 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T17:35:21.295 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T17:35:21.295 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T17:35:21.295 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T17:35:21.295 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T17:35:21.295 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T17:35:21.295 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T17:35:21.295 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T17:35:21.295 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T17:35:21.296 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T17:35:21.972 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T17:35:21.972 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /sys 2024-03-28T17:35:21.972 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /proc 2024-03-28T17:35:21.972 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /mnt 2024-03-28T17:35:21.972 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /var/tmp 2024-03-28T17:35:21.972 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /home 2024-03-28T17:35:21.972 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /root 2024-03-28T17:35:21.972 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /tmp 2024-03-28T17:35:21.972 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:35:22.210 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T17:35:22.265 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T17:35:22.265 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:22.265 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T17:35:22.265 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:35:23.928 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T17:35:24.016 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T17:35:24.016 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:24.016 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T17:35:24.017 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T17:35:24.017 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T17:35:24.017 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:35:24.270 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T17:35:24.348 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T17:35:24.348 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:24.348 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T17:35:24.348 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T17:35:24.348 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T17:35:24.348 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:35:24.375 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T17:35:24.376 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-28T17:35:24.376 INFO:teuthology.orchestra.run.smithi071.stdout:Upgraded: 2024-03-28T17:35:24.376 INFO:teuthology.orchestra.run.smithi071.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T17:35:24.376 INFO:teuthology.orchestra.run.smithi071.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T17:35:24.376 INFO:teuthology.orchestra.run.smithi071.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T17:35:24.376 INFO:teuthology.orchestra.run.smithi071.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T17:35:24.376 INFO:teuthology.orchestra.run.smithi071.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T17:35:24.376 INFO:teuthology.orchestra.run.smithi071.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T17:35:24.376 INFO:teuthology.orchestra.run.smithi071.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T17:35:24.376 INFO:teuthology.orchestra.run.smithi071.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T17:35:24.377 INFO:teuthology.orchestra.run.smithi071.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T17:35:24.377 INFO:teuthology.orchestra.run.smithi071.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T17:35:24.377 INFO:teuthology.orchestra.run.smithi071.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T17:35:24.377 INFO:teuthology.orchestra.run.smithi071.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T17:35:24.377 INFO:teuthology.orchestra.run.smithi071.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T17:35:24.377 INFO:teuthology.orchestra.run.smithi071.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.377 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.377 INFO:teuthology.orchestra.run.smithi071.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T17:35:24.377 INFO:teuthology.orchestra.run.smithi071.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T17:35:24.377 INFO:teuthology.orchestra.run.smithi071.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T17:35:24.377 INFO:teuthology.orchestra.run.smithi071.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T17:35:24.377 INFO:teuthology.orchestra.run.smithi071.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T17:35:24.378 INFO:teuthology.orchestra.run.smithi071.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T17:35:24.378 INFO:teuthology.orchestra.run.smithi071.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T17:35:24.378 INFO:teuthology.orchestra.run.smithi071.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T17:35:24.378 INFO:teuthology.orchestra.run.smithi071.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T17:35:24.378 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-28T17:35:24.378 INFO:teuthology.orchestra.run.smithi071.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T17:35:24.378 INFO:teuthology.orchestra.run.smithi071.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T17:35:24.378 INFO:teuthology.orchestra.run.smithi071.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T17:35:24.378 INFO:teuthology.orchestra.run.smithi071.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T17:35:24.378 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.378 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.379 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.379 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.379 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:24.379 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.379 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.379 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.379 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:24.379 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:24.379 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:24.379 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:24.379 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:24.380 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.380 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.380 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:24.380 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.380 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.380 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.380 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:24.380 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:24.380 INFO:teuthology.orchestra.run.smithi071.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.380 INFO:teuthology.orchestra.run.smithi071.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.380 INFO:teuthology.orchestra.run.smithi071.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:24.381 INFO:teuthology.orchestra.run.smithi071.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T17:35:24.381 INFO:teuthology.orchestra.run.smithi071.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T17:35:24.381 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T17:35:24.381 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T17:35:24.381 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T17:35:24.381 INFO:teuthology.orchestra.run.smithi071.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T17:35:24.381 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T17:35:24.381 INFO:teuthology.orchestra.run.smithi071.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T17:35:24.381 INFO:teuthology.orchestra.run.smithi071.stdout: jq-1.6-16.el9.x86_64 2024-03-28T17:35:24.381 INFO:teuthology.orchestra.run.smithi071.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T17:35:24.381 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T17:35:24.382 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T17:35:24.382 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.382 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.382 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.382 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T17:35:24.382 INFO:teuthology.orchestra.run.smithi071.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T17:35:24.382 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T17:35:24.382 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T17:35:24.382 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T17:35:24.382 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.382 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.382 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T17:35:24.383 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.383 INFO:teuthology.orchestra.run.smithi071.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T17:35:24.383 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T17:35:24.383 INFO:teuthology.orchestra.run.smithi071.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T17:35:24.383 INFO:teuthology.orchestra.run.smithi071.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T17:35:24.383 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T17:35:24.383 INFO:teuthology.orchestra.run.smithi071.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T17:35:24.383 INFO:teuthology.orchestra.run.smithi071.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T17:35:24.383 INFO:teuthology.orchestra.run.smithi071.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T17:35:24.383 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T17:35:24.383 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T17:35:24.383 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T17:35:24.384 INFO:teuthology.orchestra.run.smithi071.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T17:35:24.384 INFO:teuthology.orchestra.run.smithi071.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T17:35:24.384 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T17:35:24.384 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T17:35:24.384 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T17:35:24.384 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T17:35:24.384 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T17:35:24.384 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T17:35:24.384 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T17:35:24.384 INFO:teuthology.orchestra.run.smithi071.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T17:35:24.384 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T17:35:24.384 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T17:35:24.385 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T17:35:24.385 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.385 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.385 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.385 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T17:35:24.385 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T17:35:24.385 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T17:35:24.385 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T17:35:24.385 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T17:35:24.385 INFO:teuthology.orchestra.run.smithi071.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T17:35:24.385 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T17:35:24.386 INFO:teuthology.orchestra.run.smithi071.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T17:35:24.386 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T17:35:24.386 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T17:35:24.386 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T17:35:24.386 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T17:35:24.386 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T17:35:24.386 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T17:35:24.386 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T17:35:24.386 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T17:35:24.386 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T17:35:24.387 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T17:35:24.387 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T17:35:24.387 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T17:35:24.387 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T17:35:24.387 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T17:35:24.387 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T17:35:24.387 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T17:35:24.387 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T17:35:24.387 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T17:35:24.387 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T17:35:24.387 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T17:35:24.387 INFO:teuthology.orchestra.run.smithi071.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T17:35:24.388 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T17:35:24.388 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T17:35:24.388 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T17:35:24.388 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T17:35:24.388 INFO:teuthology.orchestra.run.smithi071.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T17:35:24.388 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T17:35:24.388 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T17:35:24.388 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T17:35:24.388 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T17:35:24.388 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T17:35:24.388 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T17:35:24.388 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.389 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.389 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T17:35:24.389 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T17:35:24.389 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T17:35:24.389 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.389 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T17:35:24.389 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T17:35:24.389 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T17:35:24.389 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T17:35:24.389 INFO:teuthology.orchestra.run.smithi071.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T17:35:24.389 INFO:teuthology.orchestra.run.smithi071.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T17:35:24.389 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T17:35:24.390 INFO:teuthology.orchestra.run.smithi071.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T17:35:24.390 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T17:35:24.390 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T17:35:24.390 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T17:35:24.390 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T17:35:24.390 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T17:35:24.390 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T17:35:24.390 INFO:teuthology.orchestra.run.smithi071.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T17:35:24.390 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.390 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.390 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:24.391 INFO:teuthology.orchestra.run.smithi071.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T17:35:24.391 INFO:teuthology.orchestra.run.smithi071.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T17:35:24.391 INFO:teuthology.orchestra.run.smithi071.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T17:35:24.391 INFO:teuthology.orchestra.run.smithi071.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T17:35:24.391 INFO:teuthology.orchestra.run.smithi071.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T17:35:24.391 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T17:35:24.391 INFO:teuthology.orchestra.run.smithi071.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T17:35:24.392 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-28T17:35:24.392 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-28T17:35:24.854 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T17:35:24.868 DEBUG:teuthology.parallel:result is None 2024-03-28T17:35:24.906 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T17:35:24.906 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T17:35:24.906 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T17:35:24.906 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T17:35:24.906 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T17:35:24.906 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T17:35:24.906 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T17:35:24.906 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T17:35:24.906 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T17:35:24.906 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T17:35:24.907 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T17:35:24.907 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T17:35:24.907 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T17:35:24.907 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T17:35:24.907 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T17:35:24.907 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T17:35:24.907 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T17:35:24.907 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T17:35:24.907 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T17:35:24.907 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T17:35:24.907 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T17:35:24.907 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T17:35:24.907 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T17:35:24.907 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T17:35:24.907 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T17:35:24.908 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T17:35:24.908 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T17:35:24.908 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T17:35:24.908 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T17:35:24.908 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T17:35:24.908 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T17:35:24.923 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T17:35:24.923 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T17:35:24.923 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T17:35:24.923 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T17:35:24.923 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T17:35:24.923 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T17:35:24.924 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T17:35:24.924 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T17:35:24.924 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T17:35:24.924 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T17:35:24.924 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T17:35:24.924 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T17:35:24.924 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T17:35:24.924 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T17:35:24.924 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T17:35:24.924 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T17:35:24.924 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T17:35:24.924 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T17:35:24.924 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T17:35:24.924 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T17:35:24.924 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T17:35:24.925 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T17:35:24.925 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T17:35:24.925 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T17:35:24.925 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T17:35:24.925 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T17:35:24.925 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T17:35:24.925 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T17:35:24.925 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T17:35:24.925 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T17:35:24.925 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T17:35:24.925 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T17:35:24.925 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T17:35:24.925 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T17:35:24.925 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T17:35:24.926 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T17:35:24.926 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T17:35:24.926 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T17:35:24.926 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T17:35:24.926 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T17:35:24.926 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T17:35:24.926 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T17:35:24.926 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T17:35:24.926 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T17:35:24.926 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T17:35:24.926 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T17:35:24.926 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T17:35:24.927 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T17:35:24.927 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T17:35:24.927 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T17:35:24.927 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T17:35:24.927 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T17:35:24.927 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T17:35:24.927 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T17:35:24.927 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T17:35:24.927 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T17:35:24.927 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T17:35:24.927 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T17:35:24.927 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T17:35:24.927 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T17:35:24.927 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T17:35:24.928 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T17:35:24.928 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T17:35:24.928 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T17:35:24.928 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T17:35:24.928 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T17:35:24.928 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T17:35:24.928 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T17:35:24.928 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T17:35:24.928 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T17:35:24.928 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T17:35:24.928 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T17:35:24.928 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T17:35:24.928 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T17:35:24.928 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T17:35:24.929 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T17:35:24.929 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T17:35:24.929 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T17:35:24.929 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T17:35:24.929 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T17:35:24.929 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T17:35:24.929 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T17:35:24.929 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T17:35:24.929 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T17:35:24.929 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T17:35:24.929 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T17:35:24.929 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T17:35:24.929 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T17:35:24.929 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T17:35:24.930 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T17:35:24.930 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T17:35:24.930 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T17:35:24.930 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T17:35:24.930 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T17:35:24.930 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T17:35:24.930 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T17:35:24.930 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T17:35:24.930 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T17:35:24.930 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T17:35:24.930 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T17:35:24.930 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T17:35:24.930 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T17:35:24.931 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T17:35:24.931 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T17:35:24.931 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T17:35:24.931 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T17:35:24.931 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T17:35:24.931 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T17:35:24.931 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T17:35:24.931 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T17:35:24.931 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T17:35:24.931 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T17:35:24.931 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T17:35:24.931 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T17:35:24.931 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T17:35:24.932 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T17:35:24.932 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T17:35:24.932 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T17:35:24.932 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T17:35:24.932 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T17:35:24.932 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T17:35:24.932 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T17:35:24.932 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T17:35:24.932 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T17:35:24.932 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T17:35:24.932 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T17:35:24.932 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T17:35:24.932 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T17:35:24.932 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T17:35:24.932 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T17:35:24.933 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T17:35:24.933 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T17:35:24.933 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T17:35:24.933 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T17:35:24.933 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T17:35:24.933 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T17:35:24.933 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T17:35:24.933 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T17:35:24.933 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T17:35:24.933 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T17:35:24.933 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T17:35:24.933 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T17:35:24.933 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T17:35:24.933 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T17:35:24.933 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T17:35:24.934 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T17:35:24.934 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T17:35:24.934 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T17:35:24.934 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T17:35:24.934 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T17:35:24.934 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T17:35:24.934 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T17:35:24.934 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T17:35:24.934 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T17:35:24.934 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T17:35:24.934 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T17:35:24.934 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T17:35:24.934 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T17:35:24.934 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T17:35:24.934 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T17:35:24.935 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T17:35:24.935 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T17:35:24.935 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T17:35:24.935 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T17:35:24.935 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T17:35:24.935 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T17:35:24.935 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T17:35:24.935 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T17:35:24.935 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:24.935 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T17:35:24.936 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T17:35:24.936 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T17:35:24.936 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:35:25.073 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T17:35:25.661 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T17:35:25.700 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T17:35:25.752 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T17:35:25.857 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T17:35:25.897 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T17:35:26.231 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T17:35:26.265 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T17:35:27.429 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T17:35:27.457 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T17:35:27.606 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T17:35:27.787 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T17:35:27.826 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T17:35:27.903 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T17:35:27.903 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:27.903 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T17:35:27.903 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T17:35:27.903 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T17:35:27.903 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:35:27.968 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T17:35:28.009 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T17:35:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T17:35:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-28T17:35:28.018 INFO:teuthology.orchestra.run.smithi151.stdout:Upgraded: 2024-03-28T17:35:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T17:35:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T17:35:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T17:35:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T17:35:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T17:35:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T17:35:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T17:35:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T17:35:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T17:35:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T17:35:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T17:35:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T17:35:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T17:35:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T17:35:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T17:35:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T17:35:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T17:35:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T17:35:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T17:35:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T17:35:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T17:35:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T17:35:28.020 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-28T17:35:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T17:35:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T17:35:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T17:35:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T17:35:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T17:35:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T17:35:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T17:35:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T17:35:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T17:35:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T17:35:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T17:35:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T17:35:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: jq-1.6-16.el9.x86_64 2024-03-28T17:35:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T17:35:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T17:35:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T17:35:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T17:35:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T17:35:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T17:35:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T17:35:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T17:35:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T17:35:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T17:35:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T17:35:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T17:35:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T17:35:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T17:35:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T17:35:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T17:35:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T17:35:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T17:35:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T17:35:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T17:35:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T17:35:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T17:35:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T17:35:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T17:35:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T17:35:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T17:35:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T17:35:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T17:35:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T17:35:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T17:35:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T17:35:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T17:35:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T17:35:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T17:35:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T17:35:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T17:35:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T17:35:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T17:35:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T17:35:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T17:35:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T17:35:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T17:35:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T17:35:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T17:35:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T17:35:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T17:35:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T17:35:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T17:35:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T17:35:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T17:35:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T17:35:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T17:35:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T17:35:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T17:35:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T17:35:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T17:35:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T17:35:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T17:35:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T17:35:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T17:35:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T17:35:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T17:35:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T17:35:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T17:35:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T17:35:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T17:35:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T17:35:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T17:35:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T17:35:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T17:35:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T17:35:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T17:35:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T17:35:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T17:35:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T17:35:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T17:35:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T17:35:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T17:35:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T17:35:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T17:35:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T17:35:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T17:35:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T17:35:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T17:35:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T17:35:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T17:35:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T17:35:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T17:35:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T17:35:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T17:35:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T17:35:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T17:35:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T17:35:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T17:35:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T17:35:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T17:35:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T17:35:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T17:35:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-28T17:35:28.031 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-28T17:35:28.073 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T17:35:28.499 DEBUG:teuthology.parallel:result is None 2024-03-28T17:35:29.076 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T17:35:29.111 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T17:35:29.188 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T17:35:29.189 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:29.189 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T17:35:29.189 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T17:35:29.189 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T17:35:29.189 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:35:29.237 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T17:35:29.309 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T17:35:29.309 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:29.310 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T17:35:29.310 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:35:29.388 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T17:35:29.467 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T17:35:29.467 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:29.467 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T17:35:29.467 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T17:35:29.467 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T17:35:29.467 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:35:29.856 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T17:35:29.944 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T17:35:29.944 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T17:35:29.944 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T17:35:29.944 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T17:35:29.944 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T17:35:29.944 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:35:35.056 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T17:35:35.113 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T17:35:35.154 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T17:35:35.193 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T17:35:35.491 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T17:35:35.619 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T17:35:35.672 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T17:35:35.732 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T17:35:35.804 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T17:35:35.851 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T17:35:35.903 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T17:35:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T17:35:36.032 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T17:35:36.130 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T17:35:36.243 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T17:35:36.281 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T17:35:37.162 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T17:35:37.495 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T17:35:37.563 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T17:35:39.029 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T17:35:39.126 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T17:35:39.180 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T17:35:39.322 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T17:35:39.355 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T17:35:39.412 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T17:35:39.439 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T17:35:39.482 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T17:35:39.482 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T17:35:39.626 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T17:35:39.676 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T17:35:39.676 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T17:35:39.814 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T17:35:39.895 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T17:35:39.938 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T17:35:39.971 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T17:35:40.003 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T17:35:40.074 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T17:35:40.155 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T17:35:40.201 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T17:35:40.234 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T17:35:40.267 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T17:35:40.301 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T17:35:40.359 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T17:35:40.397 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T17:35:40.398 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T17:35:40.734 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T17:35:40.735 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T17:35:40.899 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T17:35:40.929 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T17:35:40.983 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T17:35:41.015 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T17:35:41.055 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T17:35:41.178 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T17:35:41.469 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T17:35:41.469 INFO:teuthology.orchestra.run.smithi045.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-28T17:35:41.469 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:35:46.042 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T17:35:46.042 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T17:35:46.042 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T17:35:46.042 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T17:35:46.042 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T17:35:46.042 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T17:35:46.042 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T17:35:46.042 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T17:35:46.042 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T17:35:46.042 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T17:35:46.043 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T17:35:46.043 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T17:35:46.043 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T17:35:46.043 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T17:35:46.043 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T17:35:46.043 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T17:35:46.043 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T17:35:46.043 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T17:35:46.043 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T17:35:46.043 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T17:35:46.043 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T17:35:46.043 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T17:35:46.043 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T17:35:46.043 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T17:35:46.044 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T17:35:46.044 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T17:35:46.044 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T17:35:46.059 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T17:35:46.059 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T17:35:46.059 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T17:35:46.059 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T17:35:46.059 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T17:35:46.060 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T17:35:46.060 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T17:35:46.060 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T17:35:46.060 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T17:35:46.060 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T17:35:46.060 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T17:35:46.060 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T17:35:46.060 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T17:35:46.060 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T17:35:46.060 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T17:35:46.060 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T17:35:46.060 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T17:35:46.060 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T17:35:46.061 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T17:35:46.061 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T17:35:46.061 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T17:35:46.061 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T17:35:46.061 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T17:35:46.061 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T17:35:46.061 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T17:35:46.061 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T17:35:46.061 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T17:35:46.061 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T17:35:46.061 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T17:35:46.061 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T17:35:46.061 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T17:35:46.061 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T17:35:46.062 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T17:35:46.062 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T17:35:46.062 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T17:35:46.062 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T17:35:46.062 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T17:35:46.062 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T17:35:46.062 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T17:35:46.062 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T17:35:46.062 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T17:35:46.062 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T17:35:46.062 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T17:35:46.062 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T17:35:46.062 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T17:35:46.063 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T17:35:46.063 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T17:35:46.063 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T17:35:46.063 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T17:35:46.063 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T17:35:46.063 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T17:35:46.063 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T17:35:46.063 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T17:35:46.063 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T17:35:46.063 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T17:35:46.063 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T17:35:46.063 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T17:35:46.064 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T17:35:46.064 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T17:35:46.064 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T17:35:46.064 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T17:35:46.064 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T17:35:46.064 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T17:35:46.064 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T17:35:46.064 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T17:35:46.064 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T17:35:46.064 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T17:35:46.064 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T17:35:46.064 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T17:35:46.064 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T17:35:46.064 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T17:35:46.064 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T17:35:46.065 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T17:35:46.065 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T17:35:46.065 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T17:35:46.065 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T17:35:46.065 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T17:35:46.065 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T17:35:46.065 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T17:35:46.065 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T17:35:46.065 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T17:35:46.065 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T17:35:46.065 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T17:35:46.065 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T17:35:46.065 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T17:35:46.065 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T17:35:46.066 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T17:35:46.066 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T17:35:46.066 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T17:35:46.066 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T17:35:46.066 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T17:35:46.066 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T17:35:46.066 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T17:35:46.066 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T17:35:46.066 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T17:35:46.066 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T17:35:46.066 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T17:35:46.066 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T17:35:46.066 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T17:35:46.066 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T17:35:46.067 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T17:35:46.067 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T17:35:46.067 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T17:35:46.067 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T17:35:46.067 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T17:35:46.067 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T17:35:46.067 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T17:35:46.067 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T17:35:46.067 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T17:35:46.067 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T17:35:46.067 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T17:35:46.067 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T17:35:46.067 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T17:35:46.068 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T17:35:46.068 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T17:35:46.068 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T17:35:46.068 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T17:35:46.068 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T17:35:46.068 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T17:35:46.068 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T17:35:46.068 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T17:35:46.068 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T17:35:46.068 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T17:35:46.068 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T17:35:46.068 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T17:35:46.068 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T17:35:46.069 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T17:35:46.069 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T17:35:46.069 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T17:35:46.069 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T17:35:46.069 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T17:35:46.069 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T17:35:46.069 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T17:35:46.069 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T17:35:46.069 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T17:35:46.069 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T17:35:46.069 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T17:35:46.069 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T17:35:46.069 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T17:35:46.069 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T17:35:46.069 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T17:35:46.070 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T17:35:46.070 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T17:35:46.070 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T17:35:46.070 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T17:35:46.070 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T17:35:46.070 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T17:35:46.070 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T17:35:46.070 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T17:35:46.070 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T17:35:46.070 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T17:35:46.070 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T17:35:46.070 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T17:35:46.070 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T17:35:46.071 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T17:35:46.071 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T17:35:46.071 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T17:35:46.071 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T17:35:46.071 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T17:35:46.071 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T17:35:46.071 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T17:35:46.071 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T17:35:46.071 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T17:35:46.071 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T17:35:46.071 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T17:35:46.071 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T17:35:46.071 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T17:35:46.071 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T17:35:46.072 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T17:35:46.072 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T17:35:48.647 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T17:35:48.647 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:35:48.648 INFO:teuthology.orchestra.run.smithi045.stdout:Upgraded: 2024-03-28T17:35:48.648 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T17:35:48.648 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T17:35:48.648 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T17:35:48.648 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T17:35:48.648 INFO:teuthology.orchestra.run.smithi045.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T17:35:48.648 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T17:35:48.648 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T17:35:48.648 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T17:35:48.648 INFO:teuthology.orchestra.run.smithi045.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T17:35:48.648 INFO:teuthology.orchestra.run.smithi045.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T17:35:48.648 INFO:teuthology.orchestra.run.smithi045.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T17:35:48.649 INFO:teuthology.orchestra.run.smithi045.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T17:35:48.649 INFO:teuthology.orchestra.run.smithi045.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T17:35:48.649 INFO:teuthology.orchestra.run.smithi045.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.649 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.649 INFO:teuthology.orchestra.run.smithi045.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T17:35:48.649 INFO:teuthology.orchestra.run.smithi045.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T17:35:48.649 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T17:35:48.649 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T17:35:48.649 INFO:teuthology.orchestra.run.smithi045.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T17:35:48.649 INFO:teuthology.orchestra.run.smithi045.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T17:35:48.649 INFO:teuthology.orchestra.run.smithi045.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T17:35:48.649 INFO:teuthology.orchestra.run.smithi045.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T17:35:48.649 INFO:teuthology.orchestra.run.smithi045.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T17:35:48.649 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-28T17:35:48.650 INFO:teuthology.orchestra.run.smithi045.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T17:35:48.650 INFO:teuthology.orchestra.run.smithi045.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T17:35:48.650 INFO:teuthology.orchestra.run.smithi045.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T17:35:48.650 INFO:teuthology.orchestra.run.smithi045.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T17:35:48.650 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.650 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.650 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.650 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.650 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:48.650 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.650 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.650 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.650 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:48.650 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:48.651 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:48.651 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:48.651 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:48.651 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.651 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.651 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:48.651 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.651 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.651 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.651 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:48.651 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:48.651 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.651 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.651 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T17:35:48.651 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T17:35:48.652 INFO:teuthology.orchestra.run.smithi045.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T17:35:48.652 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T17:35:48.652 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T17:35:48.652 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T17:35:48.652 INFO:teuthology.orchestra.run.smithi045.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T17:35:48.652 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T17:35:48.652 INFO:teuthology.orchestra.run.smithi045.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T17:35:48.652 INFO:teuthology.orchestra.run.smithi045.stdout: jq-1.6-16.el9.x86_64 2024-03-28T17:35:48.652 INFO:teuthology.orchestra.run.smithi045.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T17:35:48.652 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T17:35:48.652 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T17:35:48.652 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.652 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.652 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.652 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T17:35:48.653 INFO:teuthology.orchestra.run.smithi045.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T17:35:48.653 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T17:35:48.653 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T17:35:48.653 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T17:35:48.653 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.653 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.653 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T17:35:48.653 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.653 INFO:teuthology.orchestra.run.smithi045.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T17:35:48.653 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T17:35:48.653 INFO:teuthology.orchestra.run.smithi045.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T17:35:48.653 INFO:teuthology.orchestra.run.smithi045.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T17:35:48.653 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T17:35:48.653 INFO:teuthology.orchestra.run.smithi045.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T17:35:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T17:35:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T17:35:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T17:35:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T17:35:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T17:35:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T17:35:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T17:35:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T17:35:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T17:35:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T17:35:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T17:35:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T17:35:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T17:35:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T17:35:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T17:35:48.655 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T17:35:48.655 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T17:35:48.655 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T17:35:48.655 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.655 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.655 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.655 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T17:35:48.655 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T17:35:48.655 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T17:35:48.655 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T17:35:48.655 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T17:35:48.655 INFO:teuthology.orchestra.run.smithi045.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T17:35:48.655 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T17:35:48.655 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T17:35:48.655 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T17:35:48.656 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T17:35:48.656 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T17:35:48.656 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T17:35:48.656 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T17:35:48.656 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T17:35:48.656 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T17:35:48.656 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T17:35:48.656 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T17:35:48.656 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T17:35:48.656 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T17:35:48.656 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T17:35:48.656 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T17:35:48.656 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T17:35:48.656 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T17:35:48.657 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T17:35:48.657 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T17:35:48.657 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T17:35:48.657 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T17:35:48.657 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T17:35:48.657 INFO:teuthology.orchestra.run.smithi045.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T17:35:48.657 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T17:35:48.657 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T17:35:48.657 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T17:35:48.657 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T17:35:48.657 INFO:teuthology.orchestra.run.smithi045.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T17:35:48.657 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T17:35:48.657 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T17:35:48.657 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T17:35:48.658 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T17:35:48.658 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T17:35:48.658 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T17:35:48.658 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.658 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.658 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T17:35:48.658 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T17:35:48.658 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T17:35:48.658 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.658 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T17:35:48.658 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T17:35:48.658 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T17:35:48.658 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T17:35:48.658 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T17:35:48.659 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T17:35:48.659 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T17:35:48.659 INFO:teuthology.orchestra.run.smithi045.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T17:35:48.659 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T17:35:48.659 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T17:35:48.659 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T17:35:48.659 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T17:35:48.659 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T17:35:48.659 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T17:35:48.659 INFO:teuthology.orchestra.run.smithi045.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T17:35:48.659 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.659 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.659 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T17:35:48.659 INFO:teuthology.orchestra.run.smithi045.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T17:35:48.659 INFO:teuthology.orchestra.run.smithi045.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T17:35:48.660 INFO:teuthology.orchestra.run.smithi045.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T17:35:48.660 INFO:teuthology.orchestra.run.smithi045.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T17:35:48.660 INFO:teuthology.orchestra.run.smithi045.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T17:35:48.660 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T17:35:48.660 INFO:teuthology.orchestra.run.smithi045.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T17:35:48.660 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:35:48.660 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-28T17:35:49.389 DEBUG:teuthology.parallel:result is None 2024-03-28T17:35:49.389 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:35:49.527 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T17:35:49.585 INFO:teuthology.orchestra.run.smithi045.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T17:35:49.586 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T17:35:49.586 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T17:35:49.588 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:35:49.801 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T17:35:49.826 INFO:teuthology.orchestra.run.smithi071.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T17:35:49.827 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T17:35:49.827 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T17:35:49.829 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:35:49.967 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T17:35:49.992 INFO:teuthology.orchestra.run.smithi151.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T17:35:49.992 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T17:35:49.992 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T17:35:49.994 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-28T17:35:50.008 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-28T17:35:50.008 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T17:35:50.086 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-28T17:35:50.087 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T17:35:50.121 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-28T17:35:50.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T17:35:50.156 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-28T17:35:50.156 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-28T17:35:50.156 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T17:35:50.234 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T17:35:50.347 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-28T17:35:50.348 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T17:35:50.379 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T17:35:50.448 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-28T17:35:50.448 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T17:35:50.480 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T17:35:50.549 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-28T17:35:50.549 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-28T17:35:50.549 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T17:35:50.635 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T17:35:50.756 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-28T17:35:50.756 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T17:35:50.784 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T17:35:50.851 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-28T17:35:50.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T17:35:50.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T17:35:50.948 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-28T17:35:50.948 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-28T17:35:50.948 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T17:35:51.034 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T17:35:51.156 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-28T17:35:51.156 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T17:35:51.183 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T17:35:51.249 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-28T17:35:51.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T17:35:51.276 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T17:35:51.343 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-28T17:35:51.683 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-03-28T17:35:51.683 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:35:51.683 INFO:tasks.cephadm:Cluster fsid is 9f4888ec-ed29-11ee-95d0-87774f69a715 2024-03-28T17:35:51.683 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-28T17:35:51.684 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.45', 'mon.b': '172.21.15.71', 'mon.c': '172.21.15.151'} 2024-03-28T17:35:51.684 INFO:tasks.cephadm:First mon is mon.a on smithi045 2024-03-28T17:35:51.684 INFO:tasks.cephadm:First mgr is x 2024-03-28T17:35:51.684 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-28T17:35:51.684 DEBUG:teuthology.orchestra.run.smithi045:> sudo hostname $(hostname -s) 2024-03-28T17:35:51.762 DEBUG:teuthology.orchestra.run.smithi071:> sudo hostname $(hostname -s) 2024-03-28T17:35:51.789 DEBUG:teuthology.orchestra.run.smithi151:> sudo hostname $(hostname -s) 2024-03-28T17:35:51.817 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-28T17:35:51.817 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:35:52.034 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77877/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2145-g3be9a5c9', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2145.g3be9a5c9'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-22 05:51:42.267384', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-mchangir-testing-main-20240318.032620', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-28T17:35:52.308 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-mchangir-testing-main-20240318.032620, sha1 3be9a5c9bc793e11e2800a8c0c696e8b46742033 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:35:52.309 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm 2024-03-28T17:35:52.309 DEBUG:teuthology.orchestra.run.smithi045:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T17:35:52.973 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 17:35 /home/ubuntu/cephtest/cephadm 2024-03-28T17:35:52.974 DEBUG:teuthology.orchestra.run.smithi071:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T17:35:53.582 INFO:teuthology.orchestra.run.smithi071.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 17:35 /home/ubuntu/cephtest/cephadm 2024-03-28T17:35:53.583 DEBUG:teuthology.orchestra.run.smithi151:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T17:35:54.187 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 17:35 /home/ubuntu/cephtest/cephadm 2024-03-28T17:35:54.188 DEBUG:teuthology.orchestra.run.smithi045:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T17:35:54.238 DEBUG:teuthology.orchestra.run.smithi071:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T17:35:54.257 DEBUG:teuthology.orchestra.run.smithi151:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T17:35:54.290 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 on all hosts... 2024-03-28T17:35:54.290 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T17:35:54.295 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T17:35:54.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T17:35:55.097 INFO:teuthology.orchestra.run.smithi071.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T17:35:55.108 INFO:teuthology.orchestra.run.smithi151.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T17:35:55.964 INFO:teuthology.orchestra.run.smithi045.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T17:36:22.641 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-03-28T17:36:22.641 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T17:36:22.641 INFO:teuthology.orchestra.run.smithi151.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T17:36:22.641 INFO:teuthology.orchestra.run.smithi151.stdout: "repo_digests": [ 2024-03-28T17:36:22.641 INFO:teuthology.orchestra.run.smithi151.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T17:36:22.641 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-03-28T17:36:22.641 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-03-28T17:36:30.564 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-03-28T17:36:30.564 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T17:36:30.564 INFO:teuthology.orchestra.run.smithi071.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T17:36:30.564 INFO:teuthology.orchestra.run.smithi071.stdout: "repo_digests": [ 2024-03-28T17:36:30.564 INFO:teuthology.orchestra.run.smithi071.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T17:36:30.564 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-03-28T17:36:30.565 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-03-28T17:36:39.136 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:36:39.136 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T17:36:39.136 INFO:teuthology.orchestra.run.smithi045.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T17:36:39.136 INFO:teuthology.orchestra.run.smithi045.stdout: "repo_digests": [ 2024-03-28T17:36:39.136 INFO:teuthology.orchestra.run.smithi045.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T17:36:39.136 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-28T17:36:39.136 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:36:39.185 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /etc/ceph 2024-03-28T17:36:39.275 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph 2024-03-28T17:36:39.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /etc/ceph 2024-03-28T17:36:39.345 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 777 /etc/ceph 2024-03-28T17:36:39.440 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 777 /etc/ceph 2024-03-28T17:36:39.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 777 /etc/ceph 2024-03-28T17:36:39.502 INFO:tasks.cephadm:Writing seed config... 2024-03-28T17:36:39.505 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-28T17:36:39.505 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-28T17:36:39.505 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-28T17:36:39.505 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-28T17:36:39.505 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-28T17:36:39.505 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-28T17:36:39.505 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-28T17:36:39.505 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-28T17:36:39.505 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-28T17:36:39.506 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-28T17:36:39.506 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-28T17:36:39.506 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-28T17:36:39.506 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-28T17:36:39.506 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-28T17:36:39.506 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-28T17:36:39.506 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-28T17:36:39.506 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-28T17:36:39.506 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-28T17:36:39.506 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-28T17:36:39.506 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-28T17:36:39.506 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-28T17:36:39.506 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-28T17:36:39.506 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-28T17:36:39.506 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-28T17:36:39.507 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-28T17:36:39.507 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-28T17:36:39.507 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-28T17:36:39.507 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-28T17:36:39.507 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-28T17:36:39.507 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-28T17:36:39.507 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-28T17:36:39.507 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-28T17:36:39.507 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-28T17:36:39.507 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-28T17:36:39.507 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-28T17:36:39.507 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-28T17:36:39.507 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-28T17:36:39.507 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-28T17:36:39.508 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-28T17:36:39.508 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-28T17:36:39.508 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-28T17:36:39.508 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-28T17:36:39.508 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-28T17:36:39.508 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-28T17:36:39.508 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-28T17:36:39.508 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-28T17:36:39.509 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-28T17:36:39.568 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 9f4888ec-ed29-11ee-95d0-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-28T17:36:39.568 DEBUG:teuthology.orchestra.run.smithi045:mon.a> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mon.a.service 2024-03-28T17:36:39.615 DEBUG:teuthology.orchestra.run.smithi045:mgr.x> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mgr.x.service 2024-03-28T17:36:39.662 INFO:tasks.cephadm:Bootstrapping... 2024-03-28T17:36:39.662 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 -v bootstrap --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.45 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-28T17:36:39.939 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-28T17:36:39.939 INFO:teuthology.orchestra.run.smithi045.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033', '-v', 'bootstrap', '--fsid', '9f4888ec-ed29-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.45', '--skip-admin-label'] 2024-03-28T17:36:40.004 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T17:36:40.005 INFO:teuthology.orchestra.run.smithi045.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-28T17:36:40.005 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying podman|docker is present... 2024-03-28T17:36:40.059 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T17:36:40.060 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying lvm2 is present... 2024-03-28T17:36:40.060 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying time synchronization is in place... 2024-03-28T17:36:40.077 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-28T17:36:40.077 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-28T17:36:40.096 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-28T17:36:40.096 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-03-28T17:36:40.120 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout enabled 2024-03-28T17:36:40.142 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout active 2024-03-28T17:36:40.143 INFO:teuthology.orchestra.run.smithi045.stdout:Unit chronyd.service is enabled and running 2024-03-28T17:36:40.143 INFO:teuthology.orchestra.run.smithi045.stdout:Repeating the final host check... 2024-03-28T17:36:40.202 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T17:36:40.202 INFO:teuthology.orchestra.run.smithi045.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-28T17:36:40.203 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl is present 2024-03-28T17:36:40.203 INFO:teuthology.orchestra.run.smithi045.stdout:lvcreate is present 2024-03-28T17:36:40.222 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-28T17:36:40.222 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-28T17:36:40.242 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-28T17:36:40.242 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-03-28T17:36:40.266 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout enabled 2024-03-28T17:36:40.289 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout active 2024-03-28T17:36:40.290 INFO:teuthology.orchestra.run.smithi045.stdout:Unit chronyd.service is enabled and running 2024-03-28T17:36:40.290 INFO:teuthology.orchestra.run.smithi045.stdout:Host looks OK 2024-03-28T17:36:40.290 INFO:teuthology.orchestra.run.smithi045.stdout:Cluster fsid: 9f4888ec-ed29-11ee-95d0-87774f69a715 2024-03-28T17:36:40.291 INFO:teuthology.orchestra.run.smithi045.stdout:Acquiring lock 139623127375296 on /run/cephadm/9f4888ec-ed29-11ee-95d0-87774f69a715.lock 2024-03-28T17:36:40.291 INFO:teuthology.orchestra.run.smithi045.stdout:Lock 139623127375296 acquired on /run/cephadm/9f4888ec-ed29-11ee-95d0-87774f69a715.lock 2024-03-28T17:36:40.292 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying IP 172.21.15.45 port 3300 ... 2024-03-28T17:36:40.294 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying IP 172.21.15.45 port 6789 ... 2024-03-28T17:36:40.294 INFO:teuthology.orchestra.run.smithi045.stdout:Base mon IP(s) is [172.21.15.45:3300, 172.21.15.45:6789], mon addrv is [v2:172.21.15.45:3300,v1:172.21.15.45:6789] 2024-03-28T17:36:40.310 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.45 metric 100 2024-03-28T17:36:40.310 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.45 metric 100 2024-03-28T17:36:40.323 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-28T17:36:40.323 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-28T17:36:40.323 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-28T17:36:40.334 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-28T17:36:40.335 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-28T17:36:40.335 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-28T17:36:40.336 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-28T17:36:40.336 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2024-03-28T17:36:40.336 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-28T17:36:40.338 INFO:teuthology.orchestra.run.smithi045.stdout:Mon IP `172.21.15.45` is in CIDR network `172.21.0.0/20` 2024-03-28T17:36:40.339 INFO:teuthology.orchestra.run.smithi045.stdout:Mon IP `172.21.15.45` is in CIDR network `172.21.0.0/20` 2024-03-28T17:36:40.339 INFO:teuthology.orchestra.run.smithi045.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-28T17:36:40.342 INFO:teuthology.orchestra.run.smithi045.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-28T17:36:40.344 INFO:teuthology.orchestra.run.smithi045.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T17:36:41.062 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stdout c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-28T17:36:41.062 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T17:36:41.062 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Getting image source signatures 2024-03-28T17:36:41.062 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Copying blob sha256:4dd83c6ee8921fe17396d77c31e205985ae94c87d3f752465b5f04aaf9ae340a 2024-03-28T17:36:41.063 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-28T17:36:41.063 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Copying config sha256:c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-28T17:36:41.063 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-28T17:36:42.256 INFO:teuthology.orchestra.run.smithi045.stdout:ceph: stdout ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:36:42.256 INFO:teuthology.orchestra.run.smithi045.stdout:Ceph version: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:36:42.256 INFO:teuthology.orchestra.run.smithi045.stdout:Extracting ceph user uid/gid from container image... 2024-03-28T17:36:43.504 INFO:teuthology.orchestra.run.smithi045.stdout:stat: stdout 167 167 2024-03-28T17:36:43.504 INFO:teuthology.orchestra.run.smithi045.stdout:Creating initial keys... 2024-03-28T17:36:44.559 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph-authtool: stdout AQCrqgVmpYkEORAAV1uXX6hLCOduhvBuE6Ofdg== 2024-03-28T17:36:45.714 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph-authtool: stdout AQCtqgVmtTWbBBAASxEpzP1BYg2WTdqsvw7/fg== 2024-03-28T17:36:46.842 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph-authtool: stdout AQCuqgVmNhR6DBAAhNaSY82grWbO6Vbz4TewUg== 2024-03-28T17:36:46.844 INFO:teuthology.orchestra.run.smithi045.stdout:Creating initial monmap... 2024-03-28T17:36:47.936 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-28T17:36:47.936 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-28T17:36:47.936 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 9f4888ec-ed29-11ee-95d0-87774f69a715 2024-03-28T17:36:47.936 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-28T17:36:47.936 INFO:teuthology.orchestra.run.smithi045.stdout:monmaptool for a [v2:172.21.15.45:3300,v1:172.21.15.45:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-28T17:36:47.937 INFO:teuthology.orchestra.run.smithi045.stdout:setting min_mon_release = quincy 2024-03-28T17:36:47.937 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: set fsid to 9f4888ec-ed29-11ee-95d0-87774f69a715 2024-03-28T17:36:47.937 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-28T17:36:47.937 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:36:47.937 INFO:teuthology.orchestra.run.smithi045.stdout:Creating mon... 2024-03-28T17:36:49.207 INFO:teuthology.orchestra.run.smithi045.stdout:create mon.a on 2024-03-28T17:36:49.602 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-28T17:36:49.848 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-28T17:36:50.151 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-9f4888ec-ed29-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-9f4888ec-ed29-11ee-95d0-87774f69a715.target. 2024-03-28T17:36:50.151 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-9f4888ec-ed29-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-9f4888ec-ed29-11ee-95d0-87774f69a715.target. 2024-03-28T17:36:50.588 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mon.a 2024-03-28T17:36:50.589 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-28T17:36:50.885 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9f4888ec-ed29-11ee-95d0-87774f69a715.target.wants/ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@.service. 2024-03-28T17:36:51.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:51 smithi045 podman[31387]: 2024-03-28 17:36:51.19014935 +0000 UTC m=+0.032023131 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:36:51.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:51 smithi045 podman[31387]: 2024-03-28 17:36:51.533506715 +0000 UTC m=+0.375380528 container init d7e6716907445770a6014748e6f3f72a8bf4fd8e2f24c24b96794804e2f0402c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mon-a, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T17:36:51.635 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T17:36:51.635 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout disabled 2024-03-28T17:36:51.650 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T17:36:51.650 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-03-28T17:36:51.650 INFO:teuthology.orchestra.run.smithi045.stdout:firewalld.service is not enabled 2024-03-28T17:36:51.650 INFO:teuthology.orchestra.run.smithi045.stdout:Not possible to enable service . firewalld.service is not available 2024-03-28T17:36:51.651 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mon to start... 2024-03-28T17:36:51.651 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mon... 2024-03-28T17:36:51.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:51 smithi045 podman[31387]: 2024-03-28 17:36:51.543422827 +0000 UTC m=+0.385296600 container start d7e6716907445770a6014748e6f3f72a8bf4fd8e2f24c24b96794804e2f0402c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-28T17:36:51.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:51 smithi045 bash[31387]: d7e6716907445770a6014748e6f3f72a8bf4fd8e2f24c24b96794804e2f0402c 2024-03-28T17:36:51.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:51 smithi045 systemd[1]: Started Ceph mon.a for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:36:51.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:51 smithi045 ceph-mon[31405]: mkfs 9f4888ec-ed29-11ee-95d0-87774f69a715 2024-03-28T17:36:51.799 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:51 smithi045 ceph-mon[31405]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T17:36:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:52 smithi045 ceph-mon[31405]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T17:36:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:52 smithi045 ceph-mon[31405]: monmap e1: 1 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T17:36:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:52 smithi045 ceph-mon[31405]: fsmap 2024-03-28T17:36:52.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:52 smithi045 ceph-mon[31405]: osdmap e1: 0 total, 0 up, 0 in 2024-03-28T17:36:52.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:52 smithi045 ceph-mon[31405]: mgrmap e1: no daemons active 2024-03-28T17:36:52.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:52 smithi045 ceph-mon[31405]: from='client.? 172.21.15.45:0/1828931079' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-28T17:36:53.061 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout cluster: 2024-03-28T17:36:53.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout id: 9f4888ec-ed29-11ee-95d0-87774f69a715 2024-03-28T17:36:53.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-28T17:36:53.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-28T17:36:53.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout services: 2024-03-28T17:36:53.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.739727s) 2024-03-28T17:36:53.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-28T17:36:53.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-28T17:36:53.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-28T17:36:53.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout data: 2024-03-28T17:36:53.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-28T17:36:53.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-28T17:36:53.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-28T17:36:53.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout pgs: 2024-03-28T17:36:53.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-28T17:36:53.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.264+0000 7f6cbc400700 1 Processor -- start 2024-03-28T17:36:53.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.264+0000 7f6cbc400700 1 -- start start 2024-03-28T17:36:53.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.265+0000 7f6cbc400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb4108610 0x7f6cb410aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:36:53.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.265+0000 7f6cbc400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb407be30 con 0x7f6cb4108610 2024-03-28T17:36:53.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.266+0000 7f6cb9a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb4108610 0x7f6cb410aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:36:53.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.266+0000 7f6cb9a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb4108610 0x7f6cb410aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50612/0 (socket says 172.21.15.45:50612) 2024-03-28T17:36:53.064 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.266+0000 7f6cb9a00700 1 -- 172.21.15.45:0/3415691680 learned_addr learned my addr 172.21.15.45:0/3415691680 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:36:53.064 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.267+0000 7f6cb9a00700 1 -- 172.21.15.45:0/3415691680 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb410af40 con 0x7f6cb4108610 2024-03-28T17:36:53.064 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.267+0000 7f6cb9a00700 1 --2- 172.21.15.45:0/3415691680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb4108610 0x7f6cb410aa00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6ca80098a0 tx=0x7f6ca8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43b0a862f73ca1a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:36:53.064 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.267+0000 7f6cb3e00700 1 -- 172.21.15.45:0/3415691680 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ca8004160 con 0x7f6cb4108610 2024-03-28T17:36:53.064 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.268+0000 7f6cb3e00700 1 -- 172.21.15.45:0/3415691680 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca8004300 con 0x7f6cb4108610 2024-03-28T17:36:53.064 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.268+0000 7f6cb3e00700 1 -- 172.21.15.45:0/3415691680 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ca80036a0 con 0x7f6cb4108610 2024-03-28T17:36:53.064 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.268+0000 7f6cbc400700 1 -- 172.21.15.45:0/3415691680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb4108610 msgr2=0x7f6cb410aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:36:53.064 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.268+0000 7f6cbc400700 1 --2- 172.21.15.45:0/3415691680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb4108610 0x7f6cb410aa00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6ca80098a0 tx=0x7f6ca8009bb0 comp rx=0 tx=0).stop 2024-03-28T17:36:53.064 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.268+0000 7f6cbc400700 1 -- 172.21.15.45:0/3415691680 shutdown_connections 2024-03-28T17:36:53.064 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.268+0000 7f6cbc400700 1 --2- 172.21.15.45:0/3415691680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb4108610 0x7f6cb410aa00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:36:53.064 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.268+0000 7f6cbc400700 1 -- 172.21.15.45:0/3415691680 >> 172.21.15.45:0/3415691680 conn(0x7f6cb4101fc0 msgr2=0x7f6cb4104400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:36:53.064 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.269+0000 7f6cbc400700 1 -- 172.21.15.45:0/3415691680 shutdown_connections 2024-03-28T17:36:53.064 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.269+0000 7f6cbc400700 1 -- 172.21.15.45:0/3415691680 wait complete. 2024-03-28T17:36:53.065 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.270+0000 7f6cbc400700 1 Processor -- start 2024-03-28T17:36:53.065 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.270+0000 7f6cbc400700 1 -- start start 2024-03-28T17:36:53.065 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.271+0000 7f6cbc400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb4108610 0x7f6cb419d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:36:53.065 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.271+0000 7f6cbc400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb410b3b0 con 0x7f6cb4108610 2024-03-28T17:36:53.065 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.271+0000 7f6cb9a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb4108610 0x7f6cb419d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:36:53.065 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.271+0000 7f6cb9a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb4108610 0x7f6cb419d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50616/0 (socket says 172.21.15.45:50616) 2024-03-28T17:36:53.065 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.271+0000 7f6cb9a00700 1 -- 172.21.15.45:0/1828931079 learned_addr learned my addr 172.21.15.45:0/1828931079 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:36:53.065 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.271+0000 7f6cb9a00700 1 -- 172.21.15.45:0/1828931079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb419d9d0 con 0x7f6cb4108610 2024-03-28T17:36:53.065 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.272+0000 7f6cb9a00700 1 --2- 172.21.15.45:0/1828931079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb4108610 0x7f6cb419d490 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8000c00 tx=0x7f6ca8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:36:53.065 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.272+0000 7f6cb2000700 1 -- 172.21.15.45:0/1828931079 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ca8003db0 con 0x7f6cb4108610 2024-03-28T17:36:53.065 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.272+0000 7f6cb2000700 1 -- 172.21.15.45:0/1828931079 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca8011630 con 0x7f6cb4108610 2024-03-28T17:36:53.065 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.272+0000 7f6cbc400700 1 -- 172.21.15.45:0/1828931079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb419dc60 con 0x7f6cb4108610 2024-03-28T17:36:53.066 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.272+0000 7f6cb2000700 1 -- 172.21.15.45:0/1828931079 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ca8011950 con 0x7f6cb4108610 2024-03-28T17:36:53.066 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.272+0000 7f6cbc400700 1 -- 172.21.15.45:0/1828931079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb41a0950 con 0x7f6cb4108610 2024-03-28T17:36:53.066 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.272+0000 7f6cb2000700 1 -- 172.21.15.45:0/1828931079 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f6ca801a460 con 0x7f6cb4108610 2024-03-28T17:36:53.066 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.273+0000 7f6cb2000700 1 -- 172.21.15.45:0/1828931079 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6ca801b780 con 0x7f6cb4108610 2024-03-28T17:36:53.066 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.273+0000 7f6cbc400700 1 -- 172.21.15.45:0/1828931079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb40637c0 con 0x7f6cb4108610 2024-03-28T17:36:53.066 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.278+0000 7f6cb2000700 1 -- 172.21.15.45:0/1828931079 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6ca801b980 con 0x7f6cb4108610 2024-03-28T17:36:53.066 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.360+0000 7f6cbc400700 1 -- 172.21.15.45:0/1828931079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f6cb402dc80 con 0x7f6cb4108610 2024-03-28T17:36:53.066 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.361+0000 7f6cb2000700 1 -- 172.21.15.45:0/1828931079 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f6ca801bb60 con 0x7f6cb4108610 2024-03-28T17:36:53.066 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.362+0000 7f6cbc400700 1 -- 172.21.15.45:0/1828931079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb4108610 msgr2=0x7f6cb419d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:36:53.066 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.362+0000 7f6cbc400700 1 --2- 172.21.15.45:0/1828931079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb4108610 0x7f6cb419d490 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8000c00 tx=0x7f6ca8004750 comp rx=0 tx=0).stop 2024-03-28T17:36:53.066 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.362+0000 7f6cbc400700 1 -- 172.21.15.45:0/1828931079 shutdown_connections 2024-03-28T17:36:53.066 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.362+0000 7f6cbc400700 1 --2- 172.21.15.45:0/1828931079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cb4108610 0x7f6cb419d490 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:36:53.067 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.362+0000 7f6cbc400700 1 -- 172.21.15.45:0/1828931079 >> 172.21.15.45:0/1828931079 conn(0x7f6cb4101fc0 msgr2=0x7f6cb4102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:36:53.067 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.362+0000 7f6cbc400700 1 -- 172.21.15.45:0/1828931079 shutdown_connections 2024-03-28T17:36:53.067 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:52.363+0000 7f6cbc400700 1 -- 172.21.15.45:0/1828931079 wait complete. 2024-03-28T17:36:53.067 INFO:teuthology.orchestra.run.smithi045.stdout:mon is available 2024-03-28T17:36:53.067 INFO:teuthology.orchestra.run.smithi045.stdout:Assimilating anything we can from ceph.conf... 2024-03-28T17:36:53.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:53 smithi045 ceph-mon[31405]: from='client.? 172.21.15.45:0/3527415420' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-28T17:36:53.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:53 smithi045 ceph-mon[31405]: from='client.? 172.21.15.45:0/3527415420' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-28T17:36:54.387 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-28T17:36:54.389 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [global] 2024-03-28T17:36:54.389 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout fsid = 9f4888ec-ed29-11ee-95d0-87774f69a715 2024-03-28T17:36:54.390 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.45:3300,v1:172.21.15.45:6789] 2024-03-28T17:36:54.390 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-28T17:36:54.390 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-28T17:36:54.390 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-28T17:36:54.390 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-28T17:36:54.390 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-28T17:36:54.390 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-28T17:36:54.390 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-28T17:36:54.390 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-28T17:36:54.390 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [osd] 2024-03-28T17:36:54.390 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-28T17:36:54.390 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-28T17:36:54.390 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-28T17:36:54.391 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-28T17:36:54.391 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-28T17:36:54.391 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.654+0000 7fa52aa00700 1 Processor -- start 2024-03-28T17:36:54.391 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.654+0000 7fa52aa00700 1 -- start start 2024-03-28T17:36:54.395 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.654+0000 7fa52aa00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52407de70 0x7fa52407c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:36:54.395 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.654+0000 7fa52aa00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52407c8f0 con 0x7fa52407de70 2024-03-28T17:36:54.395 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.654+0000 7fa523e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52407de70 0x7fa52407c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:36:54.395 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.654+0000 7fa523e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52407de70 0x7fa52407c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50628/0 (socket says 172.21.15.45:50628) 2024-03-28T17:36:54.395 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.654+0000 7fa523e00700 1 -- 172.21.15.45:0/2104839248 learned_addr learned my addr 172.21.15.45:0/2104839248 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:36:54.395 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.655+0000 7fa523e00700 1 -- 172.21.15.45:0/2104839248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52407ca70 con 0x7fa52407de70 2024-03-28T17:36:54.395 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.655+0000 7fa523e00700 1 --2- 172.21.15.45:0/2104839248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52407de70 0x7fa52407c320 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa5140098a0 tx=0x7fa514009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd56133266b7c736 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:36:54.395 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.655+0000 7fa522a00700 1 -- 172.21.15.45:0/2104839248 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa514004160 con 0x7fa52407de70 2024-03-28T17:36:54.396 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.656+0000 7fa522a00700 1 -- 172.21.15.45:0/2104839248 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa514004300 con 0x7fa52407de70 2024-03-28T17:36:54.396 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.656+0000 7fa522a00700 1 -- 172.21.15.45:0/2104839248 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5140036a0 con 0x7fa52407de70 2024-03-28T17:36:54.396 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.656+0000 7fa52aa00700 1 -- 172.21.15.45:0/2104839248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52407de70 msgr2=0x7fa52407c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:36:54.396 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.656+0000 7fa52aa00700 1 --2- 172.21.15.45:0/2104839248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52407de70 0x7fa52407c320 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa5140098a0 tx=0x7fa514009bb0 comp rx=0 tx=0).stop 2024-03-28T17:36:54.396 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.656+0000 7fa52aa00700 1 -- 172.21.15.45:0/2104839248 shutdown_connections 2024-03-28T17:36:54.396 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.656+0000 7fa52aa00700 1 --2- 172.21.15.45:0/2104839248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52407de70 0x7fa52407c320 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:36:54.396 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.656+0000 7fa52aa00700 1 -- 172.21.15.45:0/2104839248 >> 172.21.15.45:0/2104839248 conn(0x7fa524103110 msgr2=0x7fa524105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:36:54.396 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.657+0000 7fa52aa00700 1 -- 172.21.15.45:0/2104839248 shutdown_connections 2024-03-28T17:36:54.396 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.657+0000 7fa52aa00700 1 -- 172.21.15.45:0/2104839248 wait complete. 2024-03-28T17:36:54.396 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.658+0000 7fa52aa00700 1 Processor -- start 2024-03-28T17:36:54.396 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.658+0000 7fa52aa00700 1 -- start start 2024-03-28T17:36:54.396 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.658+0000 7fa52aa00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52407de70 0x7fa52419d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:36:54.396 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.658+0000 7fa52aa00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52407cf40 con 0x7fa52407de70 2024-03-28T17:36:54.397 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.659+0000 7fa523e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52407de70 0x7fa52419d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:36:54.397 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.659+0000 7fa523e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52407de70 0x7fa52419d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50634/0 (socket says 172.21.15.45:50634) 2024-03-28T17:36:54.397 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.659+0000 7fa523e00700 1 -- 172.21.15.45:0/3527415420 learned_addr learned my addr 172.21.15.45:0/3527415420 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:36:54.397 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.659+0000 7fa523e00700 1 -- 172.21.15.45:0/3527415420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52419d950 con 0x7fa52407de70 2024-03-28T17:36:54.397 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.659+0000 7fa523e00700 1 --2- 172.21.15.45:0/3527415420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52407de70 0x7fa52419d410 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa514000c00 tx=0x7fa514004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:36:54.397 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.660+0000 7fa520c00700 1 -- 172.21.15.45:0/3527415420 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa514003db0 con 0x7fa52407de70 2024-03-28T17:36:54.397 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.660+0000 7fa520c00700 1 -- 172.21.15.45:0/3527415420 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa514011630 con 0x7fa52407de70 2024-03-28T17:36:54.397 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.660+0000 7fa52aa00700 1 -- 172.21.15.45:0/3527415420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa52419dbe0 con 0x7fa52407de70 2024-03-28T17:36:54.397 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.660+0000 7fa520c00700 1 -- 172.21.15.45:0/3527415420 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa514011950 con 0x7fa52407de70 2024-03-28T17:36:54.397 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.660+0000 7fa52aa00700 1 -- 172.21.15.45:0/3527415420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5241a08d0 con 0x7fa52407de70 2024-03-28T17:36:54.397 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.660+0000 7fa520c00700 1 -- 172.21.15.45:0/3527415420 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fa51401a460 con 0x7fa52407de70 2024-03-28T17:36:54.397 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.660+0000 7fa520c00700 1 -- 172.21.15.45:0/3527415420 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa51401b780 con 0x7fa52407de70 2024-03-28T17:36:54.397 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.661+0000 7fa52aa00700 1 -- 172.21.15.45:0/3527415420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5240637c0 con 0x7fa52407de70 2024-03-28T17:36:54.397 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.664+0000 7fa520c00700 1 -- 172.21.15.45:0/3527415420 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fa51401ba20 con 0x7fa52407de70 2024-03-28T17:36:54.398 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.743+0000 7fa52aa00700 1 -- 172.21.15.45:0/3527415420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fa52402dd30 con 0x7fa52407de70 2024-03-28T17:36:54.398 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.749+0000 7fa520c00700 1 -- 172.21.15.45:0/3527415420 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fa51401bc00 con 0x7fa52407de70 2024-03-28T17:36:54.398 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.749+0000 7fa52aa00700 1 -- 172.21.15.45:0/3527415420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52407de70 msgr2=0x7fa52419d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:36:54.398 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.749+0000 7fa52aa00700 1 --2- 172.21.15.45:0/3527415420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52407de70 0x7fa52419d410 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa514000c00 tx=0x7fa514004750 comp rx=0 tx=0).stop 2024-03-28T17:36:54.399 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.749+0000 7fa520c00700 1 -- 172.21.15.45:0/3527415420 <== mon.0 v2:172.21.15.45:3300/0 8 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa51401bde0 con 0x7fa52407de70 2024-03-28T17:36:54.399 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.750+0000 7fa52aa00700 1 -- 172.21.15.45:0/3527415420 shutdown_connections 2024-03-28T17:36:54.399 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.750+0000 7fa52aa00700 1 --2- 172.21.15.45:0/3527415420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52407de70 0x7fa52419d410 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:36:54.399 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.750+0000 7fa52aa00700 1 -- 172.21.15.45:0/3527415420 >> 172.21.15.45:0/3527415420 conn(0x7fa524103110 msgr2=0x7fa524103f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:36:54.399 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.750+0000 7fa52aa00700 1 -- 172.21.15.45:0/3527415420 shutdown_connections 2024-03-28T17:36:54.399 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:53.750+0000 7fa52aa00700 1 -- 172.21.15.45:0/3527415420 wait complete. 2024-03-28T17:36:54.399 INFO:teuthology.orchestra.run.smithi045.stdout:Generating new minimal ceph.conf... 2024-03-28T17:36:55.341 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:55 smithi045 ceph-mon[31405]: from='client.? 172.21.15.45:0/3772447771' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:36:55.628 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.950+0000 7f10fe200700 1 Processor -- start 2024-03-28T17:36:55.628 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.950+0000 7f10fe200700 1 -- start start 2024-03-28T17:36:55.628 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.950+0000 7f10fe200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f807e860 0x7f10f807ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:36:55.628 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.950+0000 7f10fe200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f807f250 con 0x7f10f807e860 2024-03-28T17:36:55.629 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.950+0000 7f10f7400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f807e860 0x7f10f807ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:36:55.629 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.950+0000 7f10f7400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f807e860 0x7f10f807ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50638/0 (socket says 172.21.15.45:50638) 2024-03-28T17:36:55.629 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.950+0000 7f10f7400700 1 -- 172.21.15.45:0/1286624081 learned_addr learned my addr 172.21.15.45:0/1286624081 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:36:55.629 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.951+0000 7f10f7400700 1 -- 172.21.15.45:0/1286624081 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f807fab0 con 0x7f10f807e860 2024-03-28T17:36:55.630 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.951+0000 7f10f7400700 1 --2- 172.21.15.45:0/1286624081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f807e860 0x7f10f807ec80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f10e80098a0 tx=0x7f10e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=887c0a246d0df398 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:36:55.631 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.951+0000 7f10f6000700 1 -- 172.21.15.45:0/1286624081 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f10e8004160 con 0x7f10f807e860 2024-03-28T17:36:55.631 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.951+0000 7f10f6000700 1 -- 172.21.15.45:0/1286624081 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f10e8004300 con 0x7f10f807e860 2024-03-28T17:36:55.631 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.951+0000 7f10f6000700 1 -- 172.21.15.45:0/1286624081 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f10e8011b50 con 0x7f10f807e860 2024-03-28T17:36:55.631 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.952+0000 7f10fe200700 1 -- 172.21.15.45:0/1286624081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f807e860 msgr2=0x7f10f807ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:36:55.631 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.952+0000 7f10fe200700 1 --2- 172.21.15.45:0/1286624081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f807e860 0x7f10f807ec80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f10e80098a0 tx=0x7f10e8009bb0 comp rx=0 tx=0).stop 2024-03-28T17:36:55.631 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.952+0000 7f10fe200700 1 -- 172.21.15.45:0/1286624081 shutdown_connections 2024-03-28T17:36:55.631 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.952+0000 7f10fe200700 1 --2- 172.21.15.45:0/1286624081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f807e860 0x7f10f807ec80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:36:55.631 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.952+0000 7f10fe200700 1 -- 172.21.15.45:0/1286624081 >> 172.21.15.45:0/1286624081 conn(0x7f10f807d0d0 msgr2=0x7f10f807d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:36:55.631 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.952+0000 7f10fe200700 1 -- 172.21.15.45:0/1286624081 shutdown_connections 2024-03-28T17:36:55.631 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.952+0000 7f10fe200700 1 -- 172.21.15.45:0/1286624081 wait complete. 2024-03-28T17:36:55.631 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.953+0000 7f10fe200700 1 Processor -- start 2024-03-28T17:36:55.631 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.953+0000 7f10fe200700 1 -- start start 2024-03-28T17:36:55.631 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.954+0000 7f10fe200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f807e860 0x7f10f81a2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:36:55.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.954+0000 7f10fe200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f8080e70 con 0x7f10f807e860 2024-03-28T17:36:55.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.954+0000 7f10f7400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f807e860 0x7f10f81a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:36:55.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.954+0000 7f10f7400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f807e860 0x7f10f81a2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50654/0 (socket says 172.21.15.45:50654) 2024-03-28T17:36:55.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.954+0000 7f10f7400700 1 -- 172.21.15.45:0/3772447771 learned_addr learned my addr 172.21.15.45:0/3772447771 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:36:55.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.954+0000 7f10f7400700 1 -- 172.21.15.45:0/3772447771 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f81a2650 con 0x7f10f807e860 2024-03-28T17:36:55.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.955+0000 7f10f7400700 1 --2- 172.21.15.45:0/3772447771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f807e860 0x7f10f81a2110 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f10e8009fb0 tx=0x7f10e8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:36:55.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.955+0000 7f10efe00700 1 -- 172.21.15.45:0/3772447771 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f10e80038b0 con 0x7f10f807e860 2024-03-28T17:36:55.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.955+0000 7f10efe00700 1 -- 172.21.15.45:0/3772447771 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f10e8011e50 con 0x7f10f807e860 2024-03-28T17:36:55.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.955+0000 7f10fe200700 1 -- 172.21.15.45:0/3772447771 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f81a28e0 con 0x7f10f807e860 2024-03-28T17:36:55.635 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.955+0000 7f10efe00700 1 -- 172.21.15.45:0/3772447771 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f10e8019420 con 0x7f10f807e860 2024-03-28T17:36:55.635 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.955+0000 7f10fe200700 1 -- 172.21.15.45:0/3772447771 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f81a55d0 con 0x7f10f807e860 2024-03-28T17:36:55.635 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.956+0000 7f10efe00700 1 -- 172.21.15.45:0/3772447771 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f10e8019a80 con 0x7f10f807e860 2024-03-28T17:36:55.635 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.956+0000 7f10efe00700 1 -- 172.21.15.45:0/3772447771 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f10e8024730 con 0x7f10f807e860 2024-03-28T17:36:55.635 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.957+0000 7f10fe200700 1 -- 172.21.15.45:0/3772447771 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f80637c0 con 0x7f10f807e860 2024-03-28T17:36:55.635 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:54.959+0000 7f10efe00700 1 -- 172.21.15.45:0/3772447771 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f10e8043dd0 con 0x7f10f807e860 2024-03-28T17:36:55.635 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:55.037+0000 7f10fe200700 1 -- 172.21.15.45:0/3772447771 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f10f802dc30 con 0x7f10f807e860 2024-03-28T17:36:55.635 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:55.038+0000 7f10efe00700 1 -- 172.21.15.45:0/3772447771 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f10e8032030 con 0x7f10f807e860 2024-03-28T17:36:55.635 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:55.039+0000 7f10fe200700 1 -- 172.21.15.45:0/3772447771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f807e860 msgr2=0x7f10f81a2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:36:55.635 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:55.039+0000 7f10fe200700 1 --2- 172.21.15.45:0/3772447771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f807e860 0x7f10f81a2110 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f10e8009fb0 tx=0x7f10e8011bf0 comp rx=0 tx=0).stop 2024-03-28T17:36:55.635 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:55.039+0000 7f10fe200700 1 -- 172.21.15.45:0/3772447771 shutdown_connections 2024-03-28T17:36:55.635 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:55.039+0000 7f10fe200700 1 --2- 172.21.15.45:0/3772447771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f807e860 0x7f10f81a2110 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:36:55.635 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:55.039+0000 7f10fe200700 1 -- 172.21.15.45:0/3772447771 >> 172.21.15.45:0/3772447771 conn(0x7f10f807d0d0 msgr2=0x7f10f8198db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:36:55.635 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:55.039+0000 7f10fe200700 1 -- 172.21.15.45:0/3772447771 shutdown_connections 2024-03-28T17:36:55.635 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:36:55.039+0000 7f10fe200700 1 -- 172.21.15.45:0/3772447771 wait complete. 2024-03-28T17:36:55.636 INFO:teuthology.orchestra.run.smithi045.stdout:Restarting the monitor... 2024-03-28T17:36:55.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:55 smithi045 systemd[1]: Stopping Ceph mon.a for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:36:56.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mon-a[31401]: 2024-03-28T17:36:56.094+0000 7f021f200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T17:36:56.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mon-a[31401]: 2024-03-28T17:36:56.094+0000 7f021f200700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-28T17:36:56.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:56 smithi045 podman[31651]: 2024-03-28 17:36:56.260261901 +0000 UTC m=+0.286394227 container stop d7e6716907445770a6014748e6f3f72a8bf4fd8e2f24c24b96794804e2f0402c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mon-a, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-28T17:36:56.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:56 smithi045 podman[31651]: 2024-03-28 17:36:56.288300191 +0000 UTC m=+0.314432471 container died d7e6716907445770a6014748e6f3f72a8bf4fd8e2f24c24b96794804e2f0402c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mon-a, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-28T17:36:56.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:56 smithi045 podman[31651]: 2024-03-28 17:36:56.688088322 +0000 UTC m=+0.714220639 container cleanup d7e6716907445770a6014748e6f3f72a8bf4fd8e2f24c24b96794804e2f0402c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mon-a, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD) 2024-03-28T17:36:56.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:56 smithi045 bash[31651]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mon-a 2024-03-28T17:36:57.413 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:56 smithi045 podman[31665]: 2024-03-28 17:36:56.980916263 +0000 UTC m=+0.700151191 container remove d7e6716907445770a6014748e6f3f72a8bf4fd8e2f24c24b96794804e2f0402c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mon-a, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T17:36:57.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:57 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-28T17:36:57.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:57 smithi045 systemd[1]: Stopped Ceph mon.a for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:36:57.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:57 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mon.a.service: Consumed 1.896s CPU time. 2024-03-28T17:36:57.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:57 smithi045 systemd[1]: Starting Ceph mon.a for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:36:58.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:36:57 smithi045 podman[31755]: 2024-03-28 17:36:57.722724773 +0000 UTC m=+0.032010712 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:37:02.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:02 smithi045 podman[31755]: 2024-03-28 17:37:02.189785359 +0000 UTC m=+4.499071335 container create 5eba5f57af179cc3fe3a114ab5c934ea76010a075ab38f9f2b3c9cfd469ad4af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mon-a, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-28T17:37:05.628 INFO:teuthology.orchestra.run.smithi045.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-28T17:37:05.877 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 podman[31755]: 2024-03-28 17:37:05.549306998 +0000 UTC m=+7.858592997 container init 5eba5f57af179cc3fe3a114ab5c934ea76010a075ab38f9f2b3c9cfd469ad4af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-28T17:37:05.877 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 podman[31755]: 2024-03-28 17:37:05.560211128 +0000 UTC m=+7.869497110 container start 5eba5f57af179cc3fe3a114ab5c934ea76010a075ab38f9f2b3c9cfd469ad4af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mon-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-28T17:37:05.877 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: set uid:gid to 167:167 (ceph:ceph) 2024-03-28T17:37:05.877 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev), process ceph-mon, pid 2 2024-03-28T17:37:05.877 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: pidfile_write: ignore empty --pid-file 2024-03-28T17:37:05.877 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: load: jerasure load: lrc 2024-03-28T17:37:05.878 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 bash[31755]: 5eba5f57af179cc3fe3a114ab5c934ea76010a075ab38f9f2b3c9cfd469ad4af 2024-03-28T17:37:05.878 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: RocksDB version: 7.9.2 2024-03-28T17:37:05.878 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Git sha 0 2024-03-28T17:37:05.878 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-28T17:37:05.878 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: DB SUMMARY 2024-03-28T17:37:05.878 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: DB Session ID: X9F4YX5H0URZ7E6WEBTV 2024-03-28T17:37:05.878 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: CURRENT file: CURRENT 2024-03-28T17:37:05.878 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: IDENTITY file: IDENTITY 2024-03-28T17:37:05.878 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-28T17:37:05.878 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-28T17:37:05.878 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 100660 ; 2024-03-28T17:37:05.878 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.error_if_exists: 0 2024-03-28T17:37:05.878 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.create_if_missing: 0 2024-03-28T17:37:05.878 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.paranoid_checks: 1 2024-03-28T17:37:05.879 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-28T17:37:05.879 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-28T17:37:05.879 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-28T17:37:05.879 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.env: 0x558355925b20 2024-03-28T17:37:05.879 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.fs: PosixFileSystem 2024-03-28T17:37:05.879 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.info_log: 0x55835646d960 2024-03-28T17:37:05.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_file_opening_threads: 16 2024-03-28T17:37:05.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.statistics: (nil) 2024-03-28T17:37:05.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.use_fsync: 0 2024-03-28T17:37:05.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_log_file_size: 0 2024-03-28T17:37:05.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-28T17:37:05.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-28T17:37:05.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.keep_log_file_num: 1000 2024-03-28T17:37:05.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.recycle_log_file_num: 0 2024-03-28T17:37:05.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.allow_fallocate: 1 2024-03-28T17:37:05.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.allow_mmap_reads: 0 2024-03-28T17:37:05.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.allow_mmap_writes: 0 2024-03-28T17:37:05.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.use_direct_reads: 0 2024-03-28T17:37:05.881 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-28T17:37:05.881 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.create_missing_column_families: 0 2024-03-28T17:37:05.881 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.db_log_dir: 2024-03-28T17:37:05.881 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.wal_dir: 2024-03-28T17:37:05.881 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-28T17:37:05.881 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-28T17:37:05.881 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-28T17:37:05.881 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-28T17:37:05.881 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-28T17:37:05.881 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-28T17:37:05.881 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.advise_random_on_open: 1 2024-03-28T17:37:05.881 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.db_write_buffer_size: 0 2024-03-28T17:37:05.881 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.write_buffer_manager: 0x5583564401e0 2024-03-28T17:37:05.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-28T17:37:05.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-28T17:37:05.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-28T17:37:05.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.rate_limiter: (nil) 2024-03-28T17:37:05.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-28T17:37:05.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.wal_recovery_mode: 2 2024-03-28T17:37:05.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.enable_thread_tracking: 0 2024-03-28T17:37:05.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.enable_pipelined_write: 0 2024-03-28T17:37:05.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.unordered_write: 0 2024-03-28T17:37:05.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-28T17:37:05.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-28T17:37:05.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-28T17:37:05.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-28T17:37:05.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.row_cache: None 2024-03-28T17:37:05.883 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.wal_filter: None 2024-03-28T17:37:05.883 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-28T17:37:05.883 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.allow_ingest_behind: 0 2024-03-28T17:37:05.883 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.two_write_queues: 0 2024-03-28T17:37:05.883 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.manual_wal_flush: 0 2024-03-28T17:37:05.883 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.wal_compression: 0 2024-03-28T17:37:05.883 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.atomic_flush: 0 2024-03-28T17:37:05.883 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-28T17:37:05.883 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-28T17:37:05.883 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-28T17:37:05.883 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.log_readahead_size: 0 2024-03-28T17:37:05.883 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-28T17:37:05.884 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.best_efforts_recovery: 0 2024-03-28T17:37:05.884 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-28T17:37:05.884 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-28T17:37:05.884 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.allow_data_in_errors: 0 2024-03-28T17:37:05.884 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.db_host_id: __hostname__ 2024-03-28T17:37:05.884 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-28T17:37:05.884 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_background_jobs: 2 2024-03-28T17:37:05.884 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_background_compactions: -1 2024-03-28T17:37:05.884 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_subcompactions: 1 2024-03-28T17:37:05.884 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-28T17:37:05.884 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-28T17:37:05.884 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-28T17:37:05.884 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_total_wal_size: 0 2024-03-28T17:37:05.884 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-28T17:37:05.884 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-28T17:37:05.884 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-28T17:37:05.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-28T17:37:05.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_open_files: -1 2024-03-28T17:37:05.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.bytes_per_sync: 0 2024-03-28T17:37:05.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-28T17:37:05.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-28T17:37:05.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.compaction_readahead_size: 0 2024-03-28T17:37:05.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_background_flushes: -1 2024-03-28T17:37:05.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Compression algorithms supported: 2024-03-28T17:37:05.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-28T17:37:05.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: kZSTD supported: 0 2024-03-28T17:37:05.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: kXpressCompression supported: 0 2024-03-28T17:37:05.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: kLZ4HCCompression supported: 1 2024-03-28T17:37:05.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: kZlibCompression supported: 1 2024-03-28T17:37:05.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: kSnappyCompression supported: 1 2024-03-28T17:37:05.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: kLZ4Compression supported: 1 2024-03-28T17:37:05.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: kBZip2Compression supported: 0 2024-03-28T17:37:05.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-28T17:37:05.886 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-28T17:37:05.886 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-28T17:37:05.886 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-28T17:37:05.886 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-28T17:37:05.886 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.merge_operator: 2024-03-28T17:37:05.886 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.compaction_filter: None 2024-03-28T17:37:05.886 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.compaction_filter_factory: None 2024-03-28T17:37:05.886 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.sst_partitioner_factory: None 2024-03-28T17:37:05.886 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-28T17:37:05.886 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-28T17:37:05.886 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55835646da60) 2024-03-28T17:37:05.887 INFO:journalctl@ceph.mon.a.smithi045.stdout: cache_index_and_filter_blocks: 1 2024-03-28T17:37:05.887 INFO:journalctl@ceph.mon.a.smithi045.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-28T17:37:05.887 INFO:journalctl@ceph.mon.a.smithi045.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-28T17:37:05.887 INFO:journalctl@ceph.mon.a.smithi045.stdout: pin_top_level_index_and_filter: 1 2024-03-28T17:37:05.887 INFO:journalctl@ceph.mon.a.smithi045.stdout: index_type: 0 2024-03-28T17:37:05.887 INFO:journalctl@ceph.mon.a.smithi045.stdout: data_block_index_type: 0 2024-03-28T17:37:05.887 INFO:journalctl@ceph.mon.a.smithi045.stdout: index_shortening: 1 2024-03-28T17:37:05.887 INFO:journalctl@ceph.mon.a.smithi045.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-28T17:37:05.887 INFO:journalctl@ceph.mon.a.smithi045.stdout: checksum: 4 2024-03-28T17:37:05.887 INFO:journalctl@ceph.mon.a.smithi045.stdout: no_block_cache: 0 2024-03-28T17:37:05.887 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_cache: 0x558356473090 2024-03-28T17:37:05.887 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_cache_name: BinnedLRUCache 2024-03-28T17:37:05.888 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_cache_options: 2024-03-28T17:37:05.888 INFO:journalctl@ceph.mon.a.smithi045.stdout: capacity : 536870912 2024-03-28T17:37:05.888 INFO:journalctl@ceph.mon.a.smithi045.stdout: num_shard_bits : 4 2024-03-28T17:37:05.888 INFO:journalctl@ceph.mon.a.smithi045.stdout: strict_capacity_limit : 0 2024-03-28T17:37:05.888 INFO:journalctl@ceph.mon.a.smithi045.stdout: high_pri_pool_ratio: 0.000 2024-03-28T17:37:05.888 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_cache_compressed: (nil) 2024-03-28T17:37:05.888 INFO:journalctl@ceph.mon.a.smithi045.stdout: persistent_cache: (nil) 2024-03-28T17:37:05.888 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_size: 4096 2024-03-28T17:37:05.888 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_size_deviation: 10 2024-03-28T17:37:05.888 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_restart_interval: 16 2024-03-28T17:37:05.888 INFO:journalctl@ceph.mon.a.smithi045.stdout: index_block_restart_interval: 1 2024-03-28T17:37:05.888 INFO:journalctl@ceph.mon.a.smithi045.stdout: metadata_block_size: 4096 2024-03-28T17:37:05.888 INFO:journalctl@ceph.mon.a.smithi045.stdout: partition_filters: 0 2024-03-28T17:37:05.888 INFO:journalctl@ceph.mon.a.smithi045.stdout: use_delta_encoding: 1 2024-03-28T17:37:05.888 INFO:journalctl@ceph.mon.a.smithi045.stdout: filter_policy: bloomfilter 2024-03-28T17:37:05.888 INFO:journalctl@ceph.mon.a.smithi045.stdout: whole_key_filtering: 1 2024-03-28T17:37:05.889 INFO:journalctl@ceph.mon.a.smithi045.stdout: verify_compression: 0 2024-03-28T17:37:05.889 INFO:journalctl@ceph.mon.a.smithi045.stdout: read_amp_bytes_per_bit: 0 2024-03-28T17:37:05.889 INFO:journalctl@ceph.mon.a.smithi045.stdout: format_version: 5 2024-03-28T17:37:05.889 INFO:journalctl@ceph.mon.a.smithi045.stdout: enable_index_compression: 1 2024-03-28T17:37:05.889 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_align: 0 2024-03-28T17:37:05.889 INFO:journalctl@ceph.mon.a.smithi045.stdout: max_auto_readahead_size: 262144 2024-03-28T17:37:05.889 INFO:journalctl@ceph.mon.a.smithi045.stdout: prepopulate_block_cache: 0 2024-03-28T17:37:05.890 INFO:journalctl@ceph.mon.a.smithi045.stdout: initial_auto_readahead_size: 8192 2024-03-28T17:37:05.890 INFO:journalctl@ceph.mon.a.smithi045.stdout: num_file_reads_for_auto_readahead: 2 2024-03-28T17:37:05.890 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.write_buffer_size: 33554432 2024-03-28T17:37:05.890 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_write_buffer_number: 2 2024-03-28T17:37:05.890 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.compression: NoCompression 2024-03-28T17:37:05.890 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.bottommost_compression: Disabled 2024-03-28T17:37:05.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.prefix_extractor: nullptr 2024-03-28T17:37:05.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-28T17:37:05.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.num_levels: 7 2024-03-28T17:37:05.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-28T17:37:05.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-28T17:37:05.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-28T17:37:05.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-28T17:37:05.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-28T17:37:05.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-28T17:37:05.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-28T17:37:05.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-28T17:37:05.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-28T17:37:05.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-28T17:37:05.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-28T17:37:05.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-28T17:37:05.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-28T17:37:05.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.compression_opts.level: 32767 2024-03-28T17:37:05.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.compression_opts.strategy: 0 2024-03-28T17:37:05.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-28T17:37:05.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-28T17:37:05.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-28T17:37:05.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-28T17:37:05.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.compression_opts.enabled: false 2024-03-28T17:37:05.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-28T17:37:05.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-28T17:37:05.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-28T17:37:05.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-28T17:37:05.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.target_file_size_base: 67108864 2024-03-28T17:37:05.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-28T17:37:05.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-28T17:37:05.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-28T17:37:05.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-28T17:37:05.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-28T17:37:05.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-28T17:37:05.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-28T17:37:05.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-28T17:37:05.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-28T17:37:05.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-28T17:37:05.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-28T17:37:05.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-28T17:37:05.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-28T17:37:05.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-28T17:37:05.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.arena_block_size: 1048576 2024-03-28T17:37:05.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-28T17:37:05.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-28T17:37:05.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.disable_auto_compactions: 0 2024-03-28T17:37:05.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-28T17:37:05.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-28T17:37:05.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-28T17:37:05.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-28T17:37:05.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-28T17:37:05.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-28T17:37:05.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-28T17:37:05.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-28T17:37:05.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-28T17:37:05.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-28T17:37:05.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.table_properties_collectors: 2024-03-28T17:37:05.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.inplace_update_support: 0 2024-03-28T17:37:05.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-28T17:37:05.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-28T17:37:05.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-28T17:37:05.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-28T17:37:05.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.bloom_locality: 0 2024-03-28T17:37:05.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.max_successive_merges: 0 2024-03-28T17:37:05.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-28T17:37:05.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.paranoid_file_checks: 0 2024-03-28T17:37:05.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.force_consistency_checks: 1 2024-03-28T17:37:05.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.report_bg_io_stats: 0 2024-03-28T17:37:05.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.ttl: 2592000 2024-03-28T17:37:05.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-28T17:37:05.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-28T17:37:05.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-28T17:37:05.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.enable_blob_files: false 2024-03-28T17:37:05.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.min_blob_size: 0 2024-03-28T17:37:05.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.blob_file_size: 268435456 2024-03-28T17:37:05.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-28T17:37:05.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-28T17:37:05.896 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-28T17:37:05.896 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-28T17:37:05.896 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-28T17:37:05.896 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.blob_file_starting_level: 0 2024-03-28T17:37:05.896 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-28T17:37:05.896 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-28T17:37:05.896 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-28T17:37:05.896 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: db7fc777-4238-4a6a-b5e0-4ed40ce5bae9 2024-03-28T17:37:05.896 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711647425620256, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-28T17:37:05.896 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-28T17:37:05.896 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711647425623221, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 95789, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 318, "table_properties": {"data_size": 93775, "index_size": 296, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13912, "raw_average_key_size": 50, "raw_value_size": 86219, "raw_average_value_size": 311, "num_data_blocks": 11, "num_entries": 277, "num_filter_entries": 277, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711647425, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "db7fc777-4238-4a6a-b5e0-4ed40ce5bae9", "db_session_id": "X9F4YX5H0URZ7E6WEBTV", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-28T17:37:05.896 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711647425623396, "job": 1, "event": "recovery_finished"} 2024-03-28T17:37:05.896 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-28T17:37:05.896 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 systemd[1]: Started Ceph mon.a for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:37:05.896 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-28T17:37:05.897 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558356566000 2024-03-28T17:37:05.897 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: DB pointer 0x558356550000 2024-03-28T17:37:05.897 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-28T17:37:05.897 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-28T17:37:05.897 INFO:journalctl@ceph.mon.a.smithi045.stdout: ** DB Stats ** 2024-03-28T17:37:05.897 INFO:journalctl@ceph.mon.a.smithi045.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T17:37:05.897 INFO:journalctl@ceph.mon.a.smithi045.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-28T17:37:05.897 INFO:journalctl@ceph.mon.a.smithi045.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T17:37:05.897 INFO:journalctl@ceph.mon.a.smithi045.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T17:37:05.897 INFO:journalctl@ceph.mon.a.smithi045.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-28T17:37:05.897 INFO:journalctl@ceph.mon.a.smithi045.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T17:37:05.897 INFO:journalctl@ceph.mon.a.smithi045.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T17:37:05.897 INFO:journalctl@ceph.mon.a.smithi045.stdout: 2024-03-28T17:37:05.897 INFO:journalctl@ceph.mon.a.smithi045.stdout: ** Compaction Stats [default] ** 2024-03-28T17:37:05.897 INFO:journalctl@ceph.mon.a.smithi045.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T17:37:05.897 INFO:journalctl@ceph.mon.a.smithi045.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-28T17:37:05.898 INFO:journalctl@ceph.mon.a.smithi045.stdout: L0 2/0 95.38 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 47.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-28T17:37:05.898 INFO:journalctl@ceph.mon.a.smithi045.stdout: Sum 2/0 95.38 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 47.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-28T17:37:05.898 INFO:journalctl@ceph.mon.a.smithi045.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 47.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-28T17:37:05.898 INFO:journalctl@ceph.mon.a.smithi045.stdout: 2024-03-28T17:37:05.898 INFO:journalctl@ceph.mon.a.smithi045.stdout: ** Compaction Stats [default] ** 2024-03-28T17:37:05.898 INFO:journalctl@ceph.mon.a.smithi045.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T17:37:05.898 INFO:journalctl@ceph.mon.a.smithi045.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-28T17:37:05.898 INFO:journalctl@ceph.mon.a.smithi045.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 47.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-28T17:37:05.898 INFO:journalctl@ceph.mon.a.smithi045.stdout: 2024-03-28T17:37:05.898 INFO:journalctl@ceph.mon.a.smithi045.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-28T17:37:05.898 INFO:journalctl@ceph.mon.a.smithi045.stdout: 2024-03-28T17:37:05.898 INFO:journalctl@ceph.mon.a.smithi045.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T17:37:05.898 INFO:journalctl@ceph.mon.a.smithi045.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-28T17:37:05.898 INFO:journalctl@ceph.mon.a.smithi045.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-28T17:37:05.899 INFO:journalctl@ceph.mon.a.smithi045.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-28T17:37:05.899 INFO:journalctl@ceph.mon.a.smithi045.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-28T17:37:05.899 INFO:journalctl@ceph.mon.a.smithi045.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-28T17:37:05.899 INFO:journalctl@ceph.mon.a.smithi045.stdout: Cumulative compaction: 0.00 GB write, 10.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T17:37:05.899 INFO:journalctl@ceph.mon.a.smithi045.stdout: Interval compaction: 0.00 GB write, 10.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T17:37:05.899 INFO:journalctl@ceph.mon.a.smithi045.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-28T17:37:05.899 INFO:journalctl@ceph.mon.a.smithi045.stdout: Block cache BinnedLRUCache@0x558356473090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-03-28T17:37:05.899 INFO:journalctl@ceph.mon.a.smithi045.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-28T17:37:05.899 INFO:journalctl@ceph.mon.a.smithi045.stdout: 2024-03-28T17:37:05.899 INFO:journalctl@ceph.mon.a.smithi045.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-28T17:37:05.899 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: starting mon.a rank 0 at public addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] at bind addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 2024-03-28T17:37:05.899 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: mon.a@-1(???) e1 preinit fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 2024-03-28T17:37:05.899 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: mon.a@-1(???).mds e1 new map 2024-03-28T17:37:05.899 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: mon.a@-1(???).mds e1 print_map 2024-03-28T17:37:05.899 INFO:journalctl@ceph.mon.a.smithi045.stdout: e1 2024-03-28T17:37:05.900 INFO:journalctl@ceph.mon.a.smithi045.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T17:37:05.900 INFO:journalctl@ceph.mon.a.smithi045.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T17:37:05.900 INFO:journalctl@ceph.mon.a.smithi045.stdout: legacy client fscid: -1 2024-03-28T17:37:05.900 INFO:journalctl@ceph.mon.a.smithi045.stdout: 2024-03-28T17:37:05.900 INFO:journalctl@ceph.mon.a.smithi045.stdout: No filesystems configured 2024-03-28T17:37:05.900 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-28T17:37:05.900 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T17:37:05.900 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T17:37:05.900 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T17:37:05.900 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-28T17:37:05.900 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T17:37:05.900 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: monmap e1: 1 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T17:37:05.900 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: fsmap 2024-03-28T17:37:05.900 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: osdmap e1: 0 total, 0 up, 0 in 2024-03-28T17:37:05.900 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:05 smithi045 ceph-mon[31773]: mgrmap e1: no daemons active 2024-03-28T17:37:07.014 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.235+0000 7febbe000700 1 Processor -- start 2024-03-28T17:37:07.015 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.235+0000 7febbe000700 1 -- start start 2024-03-28T17:37:07.015 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.236+0000 7febbe000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb8107990 0x7febb8107db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:07.015 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.236+0000 7febbe000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb8108380 con 0x7febb8107990 2024-03-28T17:37:07.015 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.237+0000 7febbcc00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb8107990 0x7febb8107db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:07.015 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.237+0000 7febbcc00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb8107990 0x7febb8107db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49830/0 (socket says 172.21.15.45:49830) 2024-03-28T17:37:07.015 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.237+0000 7febbcc00700 1 -- 172.21.15.45:0/1077515119 learned_addr learned my addr 172.21.15.45:0/1077515119 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:07.015 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.237+0000 7febbcc00700 1 -- 172.21.15.45:0/1077515119 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb8108af0 con 0x7febb8107990 2024-03-28T17:37:07.015 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.238+0000 7febbcc00700 1 --2- 172.21.15.45:0/1077515119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb8107990 0x7febb8107db0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7feb9c0098a0 tx=0x7feb9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d18e7e3d7dcd5db8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:07.015 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.238+0000 7febb7e00700 1 -- 172.21.15.45:0/1077515119 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb9c004160 con 0x7febb8107990 2024-03-28T17:37:07.015 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.238+0000 7febb7e00700 1 -- 172.21.15.45:0/1077515119 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb9c004300 con 0x7febb8107990 2024-03-28T17:37:07.015 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.238+0000 7febb7e00700 1 -- 172.21.15.45:0/1077515119 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb9c011b50 con 0x7febb8107990 2024-03-28T17:37:07.015 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.239+0000 7febbe000700 1 -- 172.21.15.45:0/1077515119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb8107990 msgr2=0x7febb8107db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:07.016 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.239+0000 7febbe000700 1 --2- 172.21.15.45:0/1077515119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb8107990 0x7febb8107db0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7feb9c0098a0 tx=0x7feb9c009bb0 comp rx=0 tx=0).stop 2024-03-28T17:37:07.016 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.239+0000 7febbe000700 1 -- 172.21.15.45:0/1077515119 shutdown_connections 2024-03-28T17:37:07.016 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.239+0000 7febbe000700 1 --2- 172.21.15.45:0/1077515119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb8107990 0x7febb8107db0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:07.016 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.239+0000 7febbe000700 1 -- 172.21.15.45:0/1077515119 >> 172.21.15.45:0/1077515119 conn(0x7febb8102f50 msgr2=0x7febb8105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:07.016 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.239+0000 7febbe000700 1 -- 172.21.15.45:0/1077515119 shutdown_connections 2024-03-28T17:37:07.016 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.239+0000 7febbe000700 1 -- 172.21.15.45:0/1077515119 wait complete. 2024-03-28T17:37:07.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.241+0000 7febbe000700 1 Processor -- start 2024-03-28T17:37:07.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.241+0000 7febbe000700 1 -- start start 2024-03-28T17:37:07.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.241+0000 7febbe000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb8107990 0x7febb819db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:07.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.242+0000 7febbe000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb810bea0 con 0x7febb8107990 2024-03-28T17:37:07.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.242+0000 7febbcc00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb8107990 0x7febb819db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:07.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.242+0000 7febbcc00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb8107990 0x7febb819db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49832/0 (socket says 172.21.15.45:49832) 2024-03-28T17:37:07.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.242+0000 7febbcc00700 1 -- 172.21.15.45:0/2899867483 learned_addr learned my addr 172.21.15.45:0/2899867483 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:07.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.242+0000 7febbcc00700 1 -- 172.21.15.45:0/2899867483 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb819e0b0 con 0x7febb8107990 2024-03-28T17:37:07.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.243+0000 7febbcc00700 1 --2- 172.21.15.45:0/2899867483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb8107990 0x7febb819db70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7feb9c009fb0 tx=0x7feb9c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:07.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.243+0000 7febb6000700 1 -- 172.21.15.45:0/2899867483 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb9c0038b0 con 0x7febb8107990 2024-03-28T17:37:07.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.243+0000 7febb6000700 1 -- 172.21.15.45:0/2899867483 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb9c011e50 con 0x7febb8107990 2024-03-28T17:37:07.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.243+0000 7febb6000700 1 -- 172.21.15.45:0/2899867483 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb9c019550 con 0x7febb8107990 2024-03-28T17:37:07.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.243+0000 7febbe000700 1 -- 172.21.15.45:0/2899867483 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb819e340 con 0x7febb8107990 2024-03-28T17:37:07.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.243+0000 7febbe000700 1 -- 172.21.15.45:0/2899867483 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb81a1030 con 0x7febb8107990 2024-03-28T17:37:07.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.244+0000 7febb6000700 1 -- 172.21.15.45:0/2899867483 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7feb9c019be0 con 0x7febb8107990 2024-03-28T17:37:07.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.244+0000 7febb6000700 1 -- 172.21.15.45:0/2899867483 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7feb9c024730 con 0x7febb8107990 2024-03-28T17:37:07.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.244+0000 7febbe000700 1 -- 172.21.15.45:0/2899867483 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb80637c0 con 0x7febb8107990 2024-03-28T17:37:07.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.248+0000 7febb6000700 1 -- 172.21.15.45:0/2899867483 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7feb9c043dd0 con 0x7febb8107990 2024-03-28T17:37:07.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.324+0000 7febbe000700 1 -- 172.21.15.45:0/2899867483 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7febb81a1200 con 0x7febb8107990 2024-03-28T17:37:07.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.327+0000 7febb6000700 1 -- 172.21.15.45:0/2899867483 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7feb9c032030 con 0x7febb8107990 2024-03-28T17:37:07.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.329+0000 7febbe000700 1 -- 172.21.15.45:0/2899867483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb8107990 msgr2=0x7febb819db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:07.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.329+0000 7febbe000700 1 --2- 172.21.15.45:0/2899867483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb8107990 0x7febb819db70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7feb9c009fb0 tx=0x7feb9c011bf0 comp rx=0 tx=0).stop 2024-03-28T17:37:07.021 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.329+0000 7febbe000700 1 -- 172.21.15.45:0/2899867483 shutdown_connections 2024-03-28T17:37:07.021 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.329+0000 7febbe000700 1 --2- 172.21.15.45:0/2899867483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb8107990 0x7febb819db70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:07.021 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.329+0000 7febbe000700 1 -- 172.21.15.45:0/2899867483 >> 172.21.15.45:0/2899867483 conn(0x7febb8102f50 msgr2=0x7febb8103a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:07.021 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.329+0000 7febbe000700 1 -- 172.21.15.45:0/2899867483 shutdown_connections 2024-03-28T17:37:07.021 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:06.329+0000 7febbe000700 1 -- 172.21.15.45:0/2899867483 wait complete. 2024-03-28T17:37:07.021 INFO:teuthology.orchestra.run.smithi045.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-28T17:37:07.053 INFO:teuthology.orchestra.run.smithi045.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-28T17:37:07.053 INFO:teuthology.orchestra.run.smithi045.stdout:Creating mgr... 2024-03-28T17:37:07.055 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-28T17:37:07.056 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-28T17:37:07.501 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mgr.x 2024-03-28T17:37:07.502 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mgr.x.service: Unit ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mgr.x.service not loaded. 2024-03-28T17:37:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:07 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/2899867483' entity='client.admin' 2024-03-28T17:37:07.784 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9f4888ec-ed29-11ee-95d0-87774f69a715.target.wants/ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@.service. 2024-03-28T17:37:08.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:07 smithi045 systemd[1]: Starting Ceph mgr.x for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:37:08.436 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:08 smithi045 podman[31987]: 2024-03-28 17:37:08.089466083 +0000 UTC m=+0.038978905 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:37:08.436 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:08 smithi045 podman[31987]: 2024-03-28 17:37:08.18935213 +0000 UTC m=+0.138864960 container create 9e7be486edd5abfa536ab5fb9641304b33321e81a141a82715a2b806d6da5025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-28T17:37:08.436 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:08 smithi045 podman[31987]: 2024-03-28 17:37:08.434706917 +0000 UTC m=+0.384219764 container init 9e7be486edd5abfa536ab5fb9641304b33321e81a141a82715a2b806d6da5025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-28T17:37:08.558 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T17:37:08.558 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout disabled 2024-03-28T17:37:08.574 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T17:37:08.574 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-03-28T17:37:08.574 INFO:teuthology.orchestra.run.smithi045.stdout:firewalld.service is not enabled 2024-03-28T17:37:08.574 INFO:teuthology.orchestra.run.smithi045.stdout:Not possible to enable service . firewalld.service is not available 2024-03-28T17:37:08.594 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T17:37:08.594 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout disabled 2024-03-28T17:37:08.609 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T17:37:08.609 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-03-28T17:37:08.609 INFO:teuthology.orchestra.run.smithi045.stdout:firewalld.service is not enabled 2024-03-28T17:37:08.609 INFO:teuthology.orchestra.run.smithi045.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-28T17:37:08.609 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mgr to start... 2024-03-28T17:37:08.609 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mgr... 2024-03-28T17:37:08.723 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:08 smithi045 podman[31987]: 2024-03-28 17:37:08.445745229 +0000 UTC m=+0.395258082 container start 9e7be486edd5abfa536ab5fb9641304b33321e81a141a82715a2b806d6da5025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-28T17:37:08.723 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:08 smithi045 bash[31987]: 9e7be486edd5abfa536ab5fb9641304b33321e81a141a82715a2b806d6da5025 2024-03-28T17:37:08.723 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:08 smithi045 systemd[1]: Started Ceph mgr.x for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:37:09.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:09 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/4097235738' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T17:37:09.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:09 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:09.272+0000 7fdd18099200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T17:37:09.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:09 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:09.404+0000 7fdd18099200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T17:37:10.523 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-28T17:37:10.524 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-28T17:37:10.524 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsid": "9f4888ec-ed29-11ee-95d0-87774f69a715", 2024-03-28T17:37:10.524 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T17:37:10.524 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T17:37:10.524 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T17:37:10.524 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T17:37:10.524 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:10.524 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T17:37:10.524 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T17:37:10.524 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 0 2024-03-28T17:37:10.524 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:37:10.524 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T17:37:10.524 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T17:37:10.525 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:37:10.525 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-28T17:37:10.525 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T17:37:10.525 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:37:10.525 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T17:37:10.525 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T17:37:10.526 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:10.526 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T17:37:10.526 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:37:10.526 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T17:37:10.526 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T17:37:10.527 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T17:37:10.527 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T17:37:10.527 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T17:37:10.527 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T17:37:10.527 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:10.534 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T17:37:10.534 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T17:37:10.534 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T17:37:10.535 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T17:37:10.535 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T17:37:10.535 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T17:37:10.535 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T17:37:10.535 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T17:37:10.535 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T17:37:10.535 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:10.535 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T17:37:10.535 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:37:10.535 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T17:37:10.535 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T17:37:10.535 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:10.535 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T17:37:10.535 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T17:37:10.535 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T17:37:10.536 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T17:37:10.536 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T17:37:10.536 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T17:37:10.536 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T17:37:10.536 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:37:10.536 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T17:37:10.536 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:10.536 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T17:37:10.536 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:37:10.536 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T17:36:51.623792+0000", 2024-03-28T17:37:10.536 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T17:37:10.536 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:10.536 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T17:37:10.536 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-28T17:37:10.536 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.240+0000 7fae82e00700 1 Processor -- start 2024-03-28T17:37:10.537 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.241+0000 7fae82e00700 1 -- start start 2024-03-28T17:37:10.537 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.241+0000 7fae82e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae7c107b50 0x7fae7c107f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:10.537 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.241+0000 7fae82e00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c108540 con 0x7fae7c107b50 2024-03-28T17:37:10.537 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.241+0000 7fae7be00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae7c107b50 0x7fae7c107f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:10.537 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.242+0000 7fae7be00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae7c107b50 0x7fae7c107f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49856/0 (socket says 172.21.15.45:49856) 2024-03-28T17:37:10.537 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.242+0000 7fae7be00700 1 -- 172.21.15.45:0/401089689 learned_addr learned my addr 172.21.15.45:0/401089689 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:10.537 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.242+0000 7fae7be00700 1 -- 172.21.15.45:0/401089689 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7c108d00 con 0x7fae7c107b50 2024-03-28T17:37:10.537 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.242+0000 7fae7be00700 1 --2- 172.21.15.45:0/401089689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae7c107b50 0x7fae7c107f70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fae6c0098a0 tx=0x7fae6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=701e0cf549531ab7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:10.537 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.243+0000 7fae7aa00700 1 -- 172.21.15.45:0/401089689 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae6c004160 con 0x7fae7c107b50 2024-03-28T17:37:10.537 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.243+0000 7fae7aa00700 1 -- 172.21.15.45:0/401089689 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae6c004300 con 0x7fae7c107b50 2024-03-28T17:37:10.537 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.243+0000 7fae7aa00700 1 -- 172.21.15.45:0/401089689 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae6c011b50 con 0x7fae7c107b50 2024-03-28T17:37:10.537 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.244+0000 7fae82e00700 1 -- 172.21.15.45:0/401089689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae7c107b50 msgr2=0x7fae7c107f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:10.537 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.244+0000 7fae82e00700 1 --2- 172.21.15.45:0/401089689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae7c107b50 0x7fae7c107f70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fae6c0098a0 tx=0x7fae6c009bb0 comp rx=0 tx=0).stop 2024-03-28T17:37:10.537 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.244+0000 7fae82e00700 1 -- 172.21.15.45:0/401089689 shutdown_connections 2024-03-28T17:37:10.538 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.244+0000 7fae82e00700 1 --2- 172.21.15.45:0/401089689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae7c107b50 0x7fae7c107f70 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:10.538 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.244+0000 7fae82e00700 1 -- 172.21.15.45:0/401089689 >> 172.21.15.45:0/401089689 conn(0x7fae7c103130 msgr2=0x7fae7c105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:10.538 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.244+0000 7fae82e00700 1 -- 172.21.15.45:0/401089689 shutdown_connections 2024-03-28T17:37:10.538 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.244+0000 7fae82e00700 1 -- 172.21.15.45:0/401089689 wait complete. 2024-03-28T17:37:10.538 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.246+0000 7fae82e00700 1 Processor -- start 2024-03-28T17:37:10.538 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.246+0000 7fae82e00700 1 -- start start 2024-03-28T17:37:10.538 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.247+0000 7fae82e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae7c107b50 0x7fae7c07db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:10.538 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.247+0000 7fae82e00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c10c0b0 con 0x7fae7c107b50 2024-03-28T17:37:10.538 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.247+0000 7fae7be00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae7c107b50 0x7fae7c07db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:10.538 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.247+0000 7fae7be00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae7c107b50 0x7fae7c07db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49866/0 (socket says 172.21.15.45:49866) 2024-03-28T17:37:10.538 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.247+0000 7fae7be00700 1 -- 172.21.15.45:0/4097235738 learned_addr learned my addr 172.21.15.45:0/4097235738 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:10.538 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.247+0000 7fae7be00700 1 -- 172.21.15.45:0/4097235738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7c07e0b0 con 0x7fae7c107b50 2024-03-28T17:37:10.538 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.248+0000 7fae7be00700 1 --2- 172.21.15.45:0/4097235738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae7c107b50 0x7fae7c07db70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fae6c011850 tx=0x7fae6c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:10.539 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.248+0000 7fae78c00700 1 -- 172.21.15.45:0/4097235738 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae6c003af0 con 0x7fae7c107b50 2024-03-28T17:37:10.539 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.248+0000 7fae78c00700 1 -- 172.21.15.45:0/4097235738 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae6c020070 con 0x7fae7c107b50 2024-03-28T17:37:10.539 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.248+0000 7fae78c00700 1 -- 172.21.15.45:0/4097235738 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae6c0195a0 con 0x7fae7c107b50 2024-03-28T17:37:10.539 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.248+0000 7fae82e00700 1 -- 172.21.15.45:0/4097235738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae7c07c2a0 con 0x7fae7c107b50 2024-03-28T17:37:10.539 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.248+0000 7fae82e00700 1 -- 172.21.15.45:0/4097235738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae7c07c7e0 con 0x7fae7c107b50 2024-03-28T17:37:10.539 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.249+0000 7fae78c00700 1 -- 172.21.15.45:0/4097235738 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fae6c019850 con 0x7fae7c107b50 2024-03-28T17:37:10.539 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.249+0000 7fae78c00700 1 -- 172.21.15.45:0/4097235738 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fae6c011e00 con 0x7fae7c107b50 2024-03-28T17:37:10.539 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.249+0000 7fae82e00700 1 -- 172.21.15.45:0/4097235738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae7c0637c0 con 0x7fae7c107b50 2024-03-28T17:37:10.539 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.252+0000 7fae78c00700 1 -- 172.21.15.45:0/4097235738 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fae6c003c90 con 0x7fae7c107b50 2024-03-28T17:37:10.539 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.333+0000 7fae82e00700 1 -- 172.21.15.45:0/4097235738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fae7c07cb20 con 0x7fae7c107b50 2024-03-28T17:37:10.539 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.333+0000 7fae78c00700 1 -- 172.21.15.45:0/4097235738 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fae6c019cd0 con 0x7fae7c107b50 2024-03-28T17:37:10.539 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.335+0000 7fae82e00700 1 -- 172.21.15.45:0/4097235738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae7c107b50 msgr2=0x7fae7c07db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:10.539 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.335+0000 7fae82e00700 1 --2- 172.21.15.45:0/4097235738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae7c107b50 0x7fae7c07db70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fae6c011850 tx=0x7fae6c011c10 comp rx=0 tx=0).stop 2024-03-28T17:37:10.539 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.336+0000 7fae82e00700 1 -- 172.21.15.45:0/4097235738 shutdown_connections 2024-03-28T17:37:10.540 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.336+0000 7fae82e00700 1 --2- 172.21.15.45:0/4097235738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae7c107b50 0x7fae7c07db70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:10.540 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.336+0000 7fae82e00700 1 -- 172.21.15.45:0/4097235738 >> 172.21.15.45:0/4097235738 conn(0x7fae7c103130 msgr2=0x7fae7c103cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:10.540 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.336+0000 7fae82e00700 1 -- 172.21.15.45:0/4097235738 shutdown_connections 2024-03-28T17:37:10.540 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:09.336+0000 7fae82e00700 1 -- 172.21.15.45:0/4097235738 wait complete. 2024-03-28T17:37:10.540 INFO:teuthology.orchestra.run.smithi045.stdout:mgr not available, waiting (1/15)... 2024-03-28T17:37:10.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:10 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:10.390+0000 7fdd18099200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T17:37:10.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:10 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:10.518+0000 7fdd18099200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T17:37:11.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:10 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:10.708+0000 7fdd18099200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T17:37:11.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:10 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:10.839+0000 7fdd18099200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T17:37:11.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:10 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:10.954+0000 7fdd18099200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T17:37:11.560 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:11.297+0000 7fdd18099200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T17:37:11.560 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:11.443+0000 7fdd18099200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T17:37:11.560 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:11.559+0000 7fdd18099200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T17:37:11.908 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:11.686+0000 7fdd18099200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T17:37:11.908 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:11.907+0000 7fdd18099200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T17:37:13.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:13 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/2737681050' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T17:37:13.605 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:13 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:13.290+0000 7fdd18099200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T17:37:13.605 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:13 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:13.473+0000 7fdd18099200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T17:37:13.897 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-28T17:37:13.897 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-28T17:37:13.897 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsid": "9f4888ec-ed29-11ee-95d0-87774f69a715", 2024-03-28T17:37:13.897 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T17:37:13.897 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T17:37:13.897 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T17:37:13.898 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T17:37:13.898 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:13.898 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T17:37:13.898 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T17:37:13.898 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 0 2024-03-28T17:37:13.898 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:37:13.898 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T17:37:13.898 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T17:37:13.898 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:37:13.898 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-03-28T17:37:13.898 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T17:37:13.898 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:37:13.898 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T17:37:13.899 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T17:37:13.899 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:13.899 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T17:37:13.899 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:37:13.899 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T17:37:13.899 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T17:37:13.899 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T17:37:13.899 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T17:37:13.899 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T17:37:13.899 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T17:37:13.899 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:13.899 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T17:37:13.899 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T17:37:13.900 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T17:37:13.900 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T17:37:13.900 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T17:37:13.900 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T17:37:13.900 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T17:37:13.900 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T17:37:13.900 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T17:37:13.900 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:13.900 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T17:37:13.900 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:37:13.900 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T17:37:13.900 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T17:37:13.900 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:13.901 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T17:37:13.901 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T17:37:13.901 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T17:37:13.901 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T17:37:13.901 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T17:37:13.901 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T17:37:13.901 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T17:37:13.901 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:37:13.901 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T17:37:13.901 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:13.901 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T17:37:13.901 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:37:13.901 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T17:36:51.623792+0000", 2024-03-28T17:37:13.901 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T17:37:13.902 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:13.902 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T17:37:13.902 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-28T17:37:13.902 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.141+0000 7f2e0d800700 1 Processor -- start 2024-03-28T17:37:13.902 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.141+0000 7f2e0d800700 1 -- start start 2024-03-28T17:37:13.902 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.141+0000 7f2e0d800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e08107b70 0x7f2e08107f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:13.902 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.141+0000 7f2e0d800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e08108560 con 0x7f2e08107b70 2024-03-28T17:37:13.902 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.141+0000 7f2e06a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e08107b70 0x7f2e08107f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:13.902 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.141+0000 7f2e06a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e08107b70 0x7f2e08107f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49868/0 (socket says 172.21.15.45:49868) 2024-03-28T17:37:13.902 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.141+0000 7f2e06a00700 1 -- 172.21.15.45:0/563644595 learned_addr learned my addr 172.21.15.45:0/563644595 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:13.902 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.142+0000 7f2e06a00700 1 -- 172.21.15.45:0/563644595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e08108d20 con 0x7f2e08107b70 2024-03-28T17:37:13.902 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.142+0000 7f2e06a00700 1 --2- 172.21.15.45:0/563644595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e08107b70 0x7f2e08107f90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2df80098a0 tx=0x7f2df8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2564c1af5e7390c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:13.903 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.142+0000 7f2e06000700 1 -- 172.21.15.45:0/563644595 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2df8004160 con 0x7f2e08107b70 2024-03-28T17:37:13.903 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.142+0000 7f2e06000700 1 -- 172.21.15.45:0/563644595 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2df8004300 con 0x7f2e08107b70 2024-03-28T17:37:13.903 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.142+0000 7f2e06000700 1 -- 172.21.15.45:0/563644595 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2df8011b50 con 0x7f2e08107b70 2024-03-28T17:37:13.903 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.143+0000 7f2e0d800700 1 -- 172.21.15.45:0/563644595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e08107b70 msgr2=0x7f2e08107f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:13.903 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.143+0000 7f2e0d800700 1 --2- 172.21.15.45:0/563644595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e08107b70 0x7f2e08107f90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2df80098a0 tx=0x7f2df8009bb0 comp rx=0 tx=0).stop 2024-03-28T17:37:13.903 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.143+0000 7f2e0d800700 1 -- 172.21.15.45:0/563644595 shutdown_connections 2024-03-28T17:37:13.903 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.143+0000 7f2e0d800700 1 --2- 172.21.15.45:0/563644595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e08107b70 0x7f2e08107f90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:13.903 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.143+0000 7f2e0d800700 1 -- 172.21.15.45:0/563644595 >> 172.21.15.45:0/563644595 conn(0x7f2e08103110 msgr2=0x7f2e08105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:13.903 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.144+0000 7f2e0d800700 1 -- 172.21.15.45:0/563644595 shutdown_connections 2024-03-28T17:37:13.903 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.144+0000 7f2e0d800700 1 -- 172.21.15.45:0/563644595 wait complete. 2024-03-28T17:37:13.903 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.145+0000 7f2e0d800700 1 Processor -- start 2024-03-28T17:37:13.903 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.145+0000 7f2e0d800700 1 -- start start 2024-03-28T17:37:13.903 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.145+0000 7f2e0d800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e08107b70 0x7f2e0819dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:13.903 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.145+0000 7f2e0d800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0810c0d0 con 0x7f2e08107b70 2024-03-28T17:37:13.904 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.146+0000 7f2e06a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e08107b70 0x7f2e0819dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:13.904 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.146+0000 7f2e06a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e08107b70 0x7f2e0819dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49882/0 (socket says 172.21.15.45:49882) 2024-03-28T17:37:13.904 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.146+0000 7f2e06a00700 1 -- 172.21.15.45:0/2737681050 learned_addr learned my addr 172.21.15.45:0/2737681050 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:13.904 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.146+0000 7f2e06a00700 1 -- 172.21.15.45:0/2737681050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0819e280 con 0x7f2e08107b70 2024-03-28T17:37:13.904 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.146+0000 7f2e06a00700 1 --2- 172.21.15.45:0/2737681050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e08107b70 0x7f2e0819dd40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2df8011850 tx=0x7f2df8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:13.904 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.147+0000 7f2dffe00700 1 -- 172.21.15.45:0/2737681050 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2df8003af0 con 0x7f2e08107b70 2024-03-28T17:37:13.904 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.147+0000 7f2e0d800700 1 -- 172.21.15.45:0/2737681050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e0819e510 con 0x7f2e08107b70 2024-03-28T17:37:13.904 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.147+0000 7f2dffe00700 1 -- 172.21.15.45:0/2737681050 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2df8020070 con 0x7f2e08107b70 2024-03-28T17:37:13.904 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.147+0000 7f2dffe00700 1 -- 172.21.15.45:0/2737681050 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2df80195a0 con 0x7f2e08107b70 2024-03-28T17:37:13.904 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.147+0000 7f2e0d800700 1 -- 172.21.15.45:0/2737681050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e081a1200 con 0x7f2e08107b70 2024-03-28T17:37:13.904 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.147+0000 7f2dffe00700 1 -- 172.21.15.45:0/2737681050 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f2df8019c30 con 0x7f2e08107b70 2024-03-28T17:37:13.904 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.147+0000 7f2dffe00700 1 -- 172.21.15.45:0/2737681050 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2df8011e00 con 0x7f2e08107b70 2024-03-28T17:37:13.904 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.148+0000 7f2dfcc00700 1 -- 172.21.15.45:0/2737681050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de0005330 con 0x7f2e08107b70 2024-03-28T17:37:13.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.151+0000 7f2dffe00700 1 -- 172.21.15.45:0/2737681050 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f2df8003c90 con 0x7f2e08107b70 2024-03-28T17:37:13.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.231+0000 7f2dfcc00700 1 -- 172.21.15.45:0/2737681050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2de00058f0 con 0x7f2e08107b70 2024-03-28T17:37:13.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.232+0000 7f2dffe00700 1 -- 172.21.15.45:0/2737681050 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f2df800bb20 con 0x7f2e08107b70 2024-03-28T17:37:13.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.233+0000 7f2dfcc00700 1 -- 172.21.15.45:0/2737681050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e08107b70 msgr2=0x7f2e0819dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:13.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.233+0000 7f2dfcc00700 1 --2- 172.21.15.45:0/2737681050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e08107b70 0x7f2e0819dd40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2df8011850 tx=0x7f2df8011c10 comp rx=0 tx=0).stop 2024-03-28T17:37:13.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.233+0000 7f2dfcc00700 1 -- 172.21.15.45:0/2737681050 shutdown_connections 2024-03-28T17:37:13.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.233+0000 7f2dfcc00700 1 --2- 172.21.15.45:0/2737681050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e08107b70 0x7f2e0819dd40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:13.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.233+0000 7f2dfcc00700 1 -- 172.21.15.45:0/2737681050 >> 172.21.15.45:0/2737681050 conn(0x7f2e08103110 msgr2=0x7f2e08103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:13.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.234+0000 7f2dfcc00700 1 -- 172.21.15.45:0/2737681050 shutdown_connections 2024-03-28T17:37:13.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:13.234+0000 7f2dfcc00700 1 -- 172.21.15.45:0/2737681050 wait complete. 2024-03-28T17:37:13.905 INFO:teuthology.orchestra.run.smithi045.stdout:mgr not available, waiting (2/15)... 2024-03-28T17:37:14.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:13 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:13.904+0000 7fdd18099200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T17:37:14.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:14 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:14.025+0000 7fdd18099200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T17:37:14.539 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:14 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:14.287+0000 7fdd18099200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T17:37:14.916 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:14 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:14.538+0000 7fdd18099200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T17:37:15.249 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:14 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:14.925+0000 7fdd18099200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T17:37:15.249 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:15.248+0000 7fdd18099200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T17:37:15.915 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:15.561+0000 7fdd18099200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T17:37:16.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:16 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3236879681' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T17:37:16.834 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:16 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:16.652+0000 7fdd18099200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T17:37:17.146 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-28T17:37:17.146 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-28T17:37:17.146 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsid": "9f4888ec-ed29-11ee-95d0-87774f69a715", 2024-03-28T17:37:17.146 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T17:37:17.146 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T17:37:17.146 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T17:37:17.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T17:37:17.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:17.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T17:37:17.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T17:37:17.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 0 2024-03-28T17:37:17.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:37:17.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T17:37:17.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T17:37:17.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:37:17.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-28T17:37:17.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T17:37:17.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:37:17.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T17:37:17.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T17:37:17.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:17.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T17:37:17.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:37:17.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T17:37:17.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T17:37:17.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T17:37:17.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T17:37:17.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T17:37:17.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T17:37:17.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:17.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T17:37:17.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T17:37:17.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T17:37:17.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T17:37:17.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T17:37:17.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T17:37:17.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T17:37:17.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T17:37:17.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T17:37:17.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:17.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T17:37:17.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:37:17.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T17:37:17.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T17:37:17.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:17.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T17:37:17.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T17:37:17.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T17:37:17.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T17:37:17.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T17:37:17.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T17:37:17.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T17:37:17.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:37:17.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T17:37:17.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:17.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T17:37:17.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:37:17.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T17:36:51.623792+0000", 2024-03-28T17:37:17.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T17:37:17.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:17.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T17:37:17.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-28T17:37:17.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.452+0000 7effd8e00700 1 Processor -- start 2024-03-28T17:37:17.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.452+0000 7effd8e00700 1 -- start start 2024-03-28T17:37:17.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.452+0000 7effd8e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd407de90 0x7effd407c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:17.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.452+0000 7effd8e00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd407c910 con 0x7effd407de90 2024-03-28T17:37:17.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.452+0000 7effd1e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd407de90 0x7effd407c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:17.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.453+0000 7effd1e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd407de90 0x7effd407c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59904/0 (socket says 172.21.15.45:59904) 2024-03-28T17:37:17.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.453+0000 7effd1e00700 1 -- 172.21.15.45:0/4235692465 learned_addr learned my addr 172.21.15.45:0/4235692465 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:17.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.453+0000 7effd1e00700 1 -- 172.21.15.45:0/4235692465 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd407ca90 con 0x7effd407de90 2024-03-28T17:37:17.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.453+0000 7effd1e00700 1 --2- 172.21.15.45:0/4235692465 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd407de90 0x7effd407c340 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7effc40098a0 tx=0x7effc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51da1bcb454685e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:17.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.454+0000 7effd0a00700 1 -- 172.21.15.45:0/4235692465 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7effc4004160 con 0x7effd407de90 2024-03-28T17:37:17.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.454+0000 7effd0a00700 1 -- 172.21.15.45:0/4235692465 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effc4004300 con 0x7effd407de90 2024-03-28T17:37:17.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.454+0000 7effd0a00700 1 -- 172.21.15.45:0/4235692465 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7effc4011b50 con 0x7effd407de90 2024-03-28T17:37:17.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.454+0000 7effd8e00700 1 -- 172.21.15.45:0/4235692465 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd407de90 msgr2=0x7effd407c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:17.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.454+0000 7effd8e00700 1 --2- 172.21.15.45:0/4235692465 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd407de90 0x7effd407c340 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7effc40098a0 tx=0x7effc4009bb0 comp rx=0 tx=0).stop 2024-03-28T17:37:17.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.455+0000 7effd8e00700 1 -- 172.21.15.45:0/4235692465 shutdown_connections 2024-03-28T17:37:17.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.455+0000 7effd8e00700 1 --2- 172.21.15.45:0/4235692465 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd407de90 0x7effd407c340 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:17.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.455+0000 7effd8e00700 1 -- 172.21.15.45:0/4235692465 >> 172.21.15.45:0/4235692465 conn(0x7effd4103110 msgr2=0x7effd4105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:17.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.455+0000 7effd8e00700 1 -- 172.21.15.45:0/4235692465 shutdown_connections 2024-03-28T17:37:17.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.455+0000 7effd8e00700 1 -- 172.21.15.45:0/4235692465 wait complete. 2024-03-28T17:37:17.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.456+0000 7effd8e00700 1 Processor -- start 2024-03-28T17:37:17.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.456+0000 7effd8e00700 1 -- start start 2024-03-28T17:37:17.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.456+0000 7effd8e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd407de90 0x7effd419dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:17.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.456+0000 7effd8e00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd410e140 con 0x7effd407de90 2024-03-28T17:37:17.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.457+0000 7effd1e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd407de90 0x7effd419dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:17.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.457+0000 7effd1e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd407de90 0x7effd419dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59908/0 (socket says 172.21.15.45:59908) 2024-03-28T17:37:17.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.457+0000 7effd1e00700 1 -- 172.21.15.45:0/3236879681 learned_addr learned my addr 172.21.15.45:0/3236879681 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:17.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.457+0000 7effd1e00700 1 -- 172.21.15.45:0/3236879681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd419e250 con 0x7effd407de90 2024-03-28T17:37:17.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.457+0000 7effd1e00700 1 --2- 172.21.15.45:0/3236879681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd407de90 0x7effd419dd10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7effc4011b60 tx=0x7effc4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:17.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.458+0000 7effcaa00700 1 -- 172.21.15.45:0/3236879681 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7effc4003970 con 0x7effd407de90 2024-03-28T17:37:17.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.458+0000 7effcaa00700 1 -- 172.21.15.45:0/3236879681 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effc4003b10 con 0x7effd407de90 2024-03-28T17:37:17.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.458+0000 7effcaa00700 1 -- 172.21.15.45:0/3236879681 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7effc4024610 con 0x7effd407de90 2024-03-28T17:37:17.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.458+0000 7effd8e00700 1 -- 172.21.15.45:0/3236879681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd419e4e0 con 0x7effd407de90 2024-03-28T17:37:17.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.458+0000 7effd8e00700 1 -- 172.21.15.45:0/3236879681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd41a11c0 con 0x7effd407de90 2024-03-28T17:37:17.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.458+0000 7effcaa00700 1 -- 172.21.15.45:0/3236879681 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7effc40247b0 con 0x7effd407de90 2024-03-28T17:37:17.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.458+0000 7effcaa00700 1 -- 172.21.15.45:0/3236879681 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7effc4011e00 con 0x7effd407de90 2024-03-28T17:37:17.153 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.459+0000 7effd8e00700 1 -- 172.21.15.45:0/3236879681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effd40637c0 con 0x7effd407de90 2024-03-28T17:37:17.153 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.461+0000 7effcaa00700 1 -- 172.21.15.45:0/3236879681 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7effc401a7c0 con 0x7effd407de90 2024-03-28T17:37:17.153 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.540+0000 7effd8e00700 1 -- 172.21.15.45:0/3236879681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7effd41a1760 con 0x7effd407de90 2024-03-28T17:37:17.153 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.541+0000 7effcaa00700 1 -- 172.21.15.45:0/3236879681 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7effc402ddf0 con 0x7effd407de90 2024-03-28T17:37:17.153 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.542+0000 7effd8e00700 1 -- 172.21.15.45:0/3236879681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd407de90 msgr2=0x7effd419dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:17.153 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.542+0000 7effd8e00700 1 --2- 172.21.15.45:0/3236879681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd407de90 0x7effd419dd10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7effc4011b60 tx=0x7effc4011c40 comp rx=0 tx=0).stop 2024-03-28T17:37:17.153 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.542+0000 7effd8e00700 1 -- 172.21.15.45:0/3236879681 shutdown_connections 2024-03-28T17:37:17.153 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.542+0000 7effd8e00700 1 --2- 172.21.15.45:0/3236879681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd407de90 0x7effd419dd10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:17.153 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.542+0000 7effd8e00700 1 -- 172.21.15.45:0/3236879681 >> 172.21.15.45:0/3236879681 conn(0x7effd4103110 msgr2=0x7effd4103cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:17.153 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.542+0000 7effd8e00700 1 -- 172.21.15.45:0/3236879681 shutdown_connections 2024-03-28T17:37:17.153 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:16.543+0000 7effd8e00700 1 -- 172.21.15.45:0/3236879681 wait complete. 2024-03-28T17:37:17.153 INFO:teuthology.orchestra.run.smithi045.stdout:mgr not available, waiting (3/15)... 2024-03-28T17:37:17.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:17.390+0000 7fdd18099200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T17:37:17.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:17.522+0000 7fdd18099200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T17:37:18.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:17.810+0000 7fdd18099200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T17:37:18.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:17 smithi045 ceph-mon[31773]: Activating manager daemon x 2024-03-28T17:37:18.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:17 smithi045 ceph-mon[31773]: mgrmap e2: x(active, starting, since 0.0123945s) 2024-03-28T17:37:18.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:17 smithi045 ceph-mon[31773]: from='mgr.14100 172.21.15.45:0/3128174898' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T17:37:18.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:17 smithi045 ceph-mon[31773]: from='mgr.14100 172.21.15.45:0/3128174898' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T17:37:18.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:17 smithi045 ceph-mon[31773]: from='mgr.14100 172.21.15.45:0/3128174898' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T17:37:18.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:17 smithi045 ceph-mon[31773]: from='mgr.14100 172.21.15.45:0/3128174898' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T17:37:18.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:17 smithi045 ceph-mon[31773]: from='mgr.14100 172.21.15.45:0/3128174898' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T17:37:18.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:17 smithi045 ceph-mon[31773]: Manager daemon x is now available 2024-03-28T17:37:18.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:17 smithi045 ceph-mon[31773]: from='mgr.14100 172.21.15.45:0/3128174898' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T17:37:18.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:17 smithi045 ceph-mon[31773]: from='mgr.14100 172.21.15.45:0/3128174898' entity='mgr.x' 2024-03-28T17:37:18.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:17 smithi045 ceph-mon[31773]: from='mgr.14100 172.21.15.45:0/3128174898' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T17:37:18.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:17 smithi045 ceph-mon[31773]: from='mgr.14100 172.21.15.45:0/3128174898' entity='mgr.x' 2024-03-28T17:37:19.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:18 smithi045 ceph-mon[31773]: from='mgr.14100 172.21.15.45:0/3128174898' entity='mgr.x' 2024-03-28T17:37:19.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:18 smithi045 ceph-mon[31773]: mgrmap e3: x(active, since 1.0221s) 2024-03-28T17:37:20.867 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-28T17:37:20.868 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-28T17:37:20.868 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsid": "9f4888ec-ed29-11ee-95d0-87774f69a715", 2024-03-28T17:37:20.868 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T17:37:20.868 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T17:37:20.868 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T17:37:20.868 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T17:37:20.868 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:20.868 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T17:37:20.868 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T17:37:20.868 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 0 2024-03-28T17:37:20.868 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:37:20.868 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T17:37:20.870 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T17:37:20.870 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:37:20.870 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-28T17:37:20.870 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T17:37:20.870 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:37:20.870 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T17:37:20.870 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T17:37:20.870 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:20.870 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T17:37:20.870 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:37:20.871 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T17:37:20.871 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T17:37:20.871 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T17:37:20.880 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T17:37:20.880 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T17:37:20.880 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T17:37:20.880 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:20.880 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T17:37:20.880 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T17:37:20.880 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T17:37:20.880 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T17:37:20.880 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T17:37:20.880 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T17:37:20.880 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T17:37:20.880 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T17:37:20.881 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T17:37:20.881 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:20.881 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T17:37:20.881 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:37:20.881 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T17:37:20.881 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T17:37:20.881 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:20.881 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T17:37:20.881 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T17:37:20.881 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T17:37:20.881 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T17:37:20.881 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T17:37:20.881 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T17:37:20.881 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T17:37:20.882 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-28T17:37:20.882 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T17:37:20.882 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:20.882 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T17:37:20.882 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T17:37:20.882 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T17:36:51.623792+0000", 2024-03-28T17:37:20.882 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T17:37:20.882 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-28T17:37:20.882 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T17:37:20.882 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-28T17:37:20.882 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.752+0000 7f5d97200700 1 Processor -- start 2024-03-28T17:37:20.882 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.752+0000 7f5d97200700 1 -- start start 2024-03-28T17:37:20.882 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.752+0000 7f5d97200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d98109b20 0x7f5d98109f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:20.883 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.752+0000 7f5d97200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9810a510 con 0x7f5d98109b20 2024-03-28T17:37:20.883 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.753+0000 7f5d95e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d98109b20 0x7f5d98109f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:20.883 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.753+0000 7f5d95e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d98109b20 0x7f5d98109f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59980/0 (socket says 172.21.15.45:59980) 2024-03-28T17:37:20.883 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.753+0000 7f5d95e00700 1 -- 172.21.15.45:0/3819362444 learned_addr learned my addr 172.21.15.45:0/3819362444 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:20.883 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.753+0000 7f5d95e00700 1 -- 172.21.15.45:0/3819362444 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d9810acd0 con 0x7f5d98109b20 2024-03-28T17:37:20.883 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.754+0000 7f5d95e00700 1 --2- 172.21.15.45:0/3819362444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d98109b20 0x7f5d98109f40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5d88009e80 tx=0x7f5d8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f39838cf4f39ca0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:20.883 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.754+0000 7f5d95400700 1 -- 172.21.15.45:0/3819362444 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d88017070 con 0x7f5d98109b20 2024-03-28T17:37:20.883 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.754+0000 7f5d95400700 1 -- 172.21.15.45:0/3819362444 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d8800b8c0 con 0x7f5d98109b20 2024-03-28T17:37:20.883 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.754+0000 7f5d95400700 1 -- 172.21.15.45:0/3819362444 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d880045c0 con 0x7f5d98109b20 2024-03-28T17:37:20.883 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.755+0000 7f5d97200700 1 -- 172.21.15.45:0/3819362444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d98109b20 msgr2=0x7f5d98109f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:20.883 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.755+0000 7f5d97200700 1 --2- 172.21.15.45:0/3819362444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d98109b20 0x7f5d98109f40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5d88009e80 tx=0x7f5d8800b330 comp rx=0 tx=0).stop 2024-03-28T17:37:20.883 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.755+0000 7f5d97200700 1 -- 172.21.15.45:0/3819362444 shutdown_connections 2024-03-28T17:37:20.883 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.755+0000 7f5d97200700 1 --2- 172.21.15.45:0/3819362444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d98109b20 0x7f5d98109f40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:20.883 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.755+0000 7f5d97200700 1 -- 172.21.15.45:0/3819362444 >> 172.21.15.45:0/3819362444 conn(0x7f5d9807cf10 msgr2=0x7f5d9807d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:20.884 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.755+0000 7f5d97200700 1 -- 172.21.15.45:0/3819362444 shutdown_connections 2024-03-28T17:37:20.884 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.755+0000 7f5d97200700 1 -- 172.21.15.45:0/3819362444 wait complete. 2024-03-28T17:37:20.884 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.756+0000 7f5d97200700 1 Processor -- start 2024-03-28T17:37:20.884 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.756+0000 7f5d97200700 1 -- start start 2024-03-28T17:37:20.884 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.756+0000 7f5d97200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d98109b20 0x7f5d9819da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:20.884 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.756+0000 7f5d97200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9810e080 con 0x7f5d98109b20 2024-03-28T17:37:20.884 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.757+0000 7f5d95e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d98109b20 0x7f5d9819da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:20.884 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.757+0000 7f5d95e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d98109b20 0x7f5d9819da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59984/0 (socket says 172.21.15.45:59984) 2024-03-28T17:37:20.884 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.757+0000 7f5d95e00700 1 -- 172.21.15.45:0/3793682361 learned_addr learned my addr 172.21.15.45:0/3793682361 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:20.884 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.757+0000 7f5d95e00700 1 -- 172.21.15.45:0/3793682361 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d9819dfc0 con 0x7f5d98109b20 2024-03-28T17:37:20.884 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.757+0000 7f5d95e00700 1 --2- 172.21.15.45:0/3793682361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d98109b20 0x7f5d9819da80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5d88009530 tx=0x7f5d8800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:20.884 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.758+0000 7f5d8f400700 1 -- 172.21.15.45:0/3793682361 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d88020020 con 0x7f5d98109b20 2024-03-28T17:37:20.884 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.758+0000 7f5d8f400700 1 -- 172.21.15.45:0/3793682361 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d88003b50 con 0x7f5d98109b20 2024-03-28T17:37:20.884 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.758+0000 7f5d8f400700 1 -- 172.21.15.45:0/3793682361 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d88017040 con 0x7f5d98109b20 2024-03-28T17:37:20.885 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.758+0000 7f5d97200700 1 -- 172.21.15.45:0/3793682361 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d9819e250 con 0x7f5d98109b20 2024-03-28T17:37:20.885 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.758+0000 7f5d97200700 1 -- 172.21.15.45:0/3793682361 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d981a0f40 con 0x7f5d98109b20 2024-03-28T17:37:20.885 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.759+0000 7f5d8f400700 1 -- 172.21.15.45:0/3793682361 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 3) v1 ==== 45989+0+0 (secure 0 0 0) 0x7f5d88023020 con 0x7f5d98109b20 2024-03-28T17:37:20.885 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.759+0000 7f5d97200700 1 -- 172.21.15.45:0/3793682361 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d980637c0 con 0x7f5d98109b20 2024-03-28T17:37:20.885 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.759+0000 7f5d8f400700 1 --2- 172.21.15.45:0/3793682361 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f5d7c038e00 0x7f5d7c03b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:20.885 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.760+0000 7f5d8f400700 1 -- 172.21.15.45:0/3793682361 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5d8800f050 con 0x7f5d98109b20 2024-03-28T17:37:20.885 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.760+0000 7f5d8d400700 1 --2- 172.21.15.45:0/3793682361 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f5d7c038e00 0x7f5d7c03b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:20.885 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.761+0000 7f5d8d400700 1 --2- 172.21.15.45:0/3793682361 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f5d7c038e00 0x7f5d7c03b2c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5d80006fd0 tx=0x7f5d80006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:20.885 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.766+0000 7f5d8f400700 1 -- 172.21.15.45:0/3793682361 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5d8802a020 con 0x7f5d98109b20 2024-03-28T17:37:20.885 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:19.879+0000 7f5d8f400700 1 -- 172.21.15.45:0/3793682361 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f5d88012430 con 0x7f5d98109b20 2024-03-28T17:37:20.885 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:20.085+0000 7f5d97200700 1 -- 172.21.15.45:0/3793682361 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5d981a1590 con 0x7f5d98109b20 2024-03-28T17:37:20.885 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:20.086+0000 7f5d8f400700 1 -- 172.21.15.45:0/3793682361 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f5d8802d350 con 0x7f5d98109b20 2024-03-28T17:37:20.885 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:20.088+0000 7f5d97200700 1 -- 172.21.15.45:0/3793682361 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f5d7c038e00 msgr2=0x7f5d7c03b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:20.885 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:20.088+0000 7f5d97200700 1 --2- 172.21.15.45:0/3793682361 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f5d7c038e00 0x7f5d7c03b2c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5d80006fd0 tx=0x7f5d80006e40 comp rx=0 tx=0).stop 2024-03-28T17:37:20.886 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:20.088+0000 7f5d97200700 1 -- 172.21.15.45:0/3793682361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d98109b20 msgr2=0x7f5d9819da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:20.886 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:20.088+0000 7f5d97200700 1 --2- 172.21.15.45:0/3793682361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d98109b20 0x7f5d9819da80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5d88009530 tx=0x7f5d8800bd50 comp rx=0 tx=0).stop 2024-03-28T17:37:20.886 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:20.089+0000 7f5d97200700 1 -- 172.21.15.45:0/3793682361 shutdown_connections 2024-03-28T17:37:20.886 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:20.089+0000 7f5d97200700 1 --2- 172.21.15.45:0/3793682361 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f5d7c038e00 0x7f5d7c03b2c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:20.886 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:20.089+0000 7f5d97200700 1 --2- 172.21.15.45:0/3793682361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d98109b20 0x7f5d9819da80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:20.886 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:20.089+0000 7f5d97200700 1 -- 172.21.15.45:0/3793682361 >> 172.21.15.45:0/3793682361 conn(0x7f5d9807cf10 msgr2=0x7f5d98107580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:20.886 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:20.089+0000 7f5d97200700 1 -- 172.21.15.45:0/3793682361 shutdown_connections 2024-03-28T17:37:20.886 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:20.089+0000 7f5d97200700 1 -- 172.21.15.45:0/3793682361 wait complete. 2024-03-28T17:37:20.886 INFO:teuthology.orchestra.run.smithi045.stdout:mgr is available 2024-03-28T17:37:21.158 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:20 smithi045 ceph-mon[31773]: mgrmap e4: x(active, since 2s) 2024-03-28T17:37:21.158 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:20 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3793682361' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T17:37:22.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:21 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/630716166' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-28T17:37:22.431 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-28T17:37:22.431 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [global] 2024-03-28T17:37:22.431 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout fsid = 9f4888ec-ed29-11ee-95d0-87774f69a715 2024-03-28T17:37:22.431 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-28T17:37:22.431 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-28T17:37:22.431 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-28T17:37:22.431 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-28T17:37:22.433 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-28T17:37:22.433 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-28T17:37:22.433 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-28T17:37:22.433 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-28T17:37:22.434 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [osd] 2024-03-28T17:37:22.434 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-28T17:37:22.434 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-28T17:37:22.434 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-28T17:37:22.434 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-28T17:37:22.434 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-28T17:37:22.434 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.477+0000 7f26efc00700 1 Processor -- start 2024-03-28T17:37:22.434 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.477+0000 7f26efc00700 1 -- start start 2024-03-28T17:37:22.434 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.477+0000 7f26efc00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26e8107b70 0x7f26e8107f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:22.434 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.477+0000 7f26efc00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e8108560 con 0x7f26e8107b70 2024-03-28T17:37:22.434 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.478+0000 7f26ed200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26e8107b70 0x7f26e8107f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:22.434 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.478+0000 7f26ed200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26e8107b70 0x7f26e8107f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59986/0 (socket says 172.21.15.45:59986) 2024-03-28T17:37:22.434 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.478+0000 7f26ed200700 1 -- 172.21.15.45:0/44827548 learned_addr learned my addr 172.21.15.45:0/44827548 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:22.438 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.478+0000 7f26ed200700 1 -- 172.21.15.45:0/44827548 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e8108d20 con 0x7f26e8107b70 2024-03-28T17:37:22.438 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.479+0000 7f26ed200700 1 --2- 172.21.15.45:0/44827548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26e8107b70 0x7f26e8107f90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f26d80098a0 tx=0x7f26d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9309f1a217882132 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:22.438 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.479+0000 7f26e7e00700 1 -- 172.21.15.45:0/44827548 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26d8004160 con 0x7f26e8107b70 2024-03-28T17:37:22.439 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.479+0000 7f26e7e00700 1 -- 172.21.15.45:0/44827548 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26d8004300 con 0x7f26e8107b70 2024-03-28T17:37:22.439 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.479+0000 7f26e7e00700 1 -- 172.21.15.45:0/44827548 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26d8011b50 con 0x7f26e8107b70 2024-03-28T17:37:22.439 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.480+0000 7f26efc00700 1 -- 172.21.15.45:0/44827548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26e8107b70 msgr2=0x7f26e8107f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:22.439 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.480+0000 7f26efc00700 1 --2- 172.21.15.45:0/44827548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26e8107b70 0x7f26e8107f90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f26d80098a0 tx=0x7f26d8009bb0 comp rx=0 tx=0).stop 2024-03-28T17:37:22.439 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.480+0000 7f26efc00700 1 -- 172.21.15.45:0/44827548 shutdown_connections 2024-03-28T17:37:22.439 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.480+0000 7f26efc00700 1 --2- 172.21.15.45:0/44827548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26e8107b70 0x7f26e8107f90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:22.439 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.480+0000 7f26efc00700 1 -- 172.21.15.45:0/44827548 >> 172.21.15.45:0/44827548 conn(0x7f26e8103110 msgr2=0x7f26e8105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:22.439 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.480+0000 7f26efc00700 1 -- 172.21.15.45:0/44827548 shutdown_connections 2024-03-28T17:37:22.439 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.480+0000 7f26efc00700 1 -- 172.21.15.45:0/44827548 wait complete. 2024-03-28T17:37:22.439 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.481+0000 7f26efc00700 1 Processor -- start 2024-03-28T17:37:22.439 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.481+0000 7f26efc00700 1 -- start start 2024-03-28T17:37:22.439 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.482+0000 7f26efc00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26e8107b70 0x7f26e807db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:22.439 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.482+0000 7f26efc00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e810c0d0 con 0x7f26e8107b70 2024-03-28T17:37:22.439 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.482+0000 7f26ed200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26e8107b70 0x7f26e807db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:22.440 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.482+0000 7f26ed200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26e8107b70 0x7f26e807db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59988/0 (socket says 172.21.15.45:59988) 2024-03-28T17:37:22.440 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.482+0000 7f26ed200700 1 -- 172.21.15.45:0/630716166 learned_addr learned my addr 172.21.15.45:0/630716166 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:22.440 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.482+0000 7f26ed200700 1 -- 172.21.15.45:0/630716166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e807e0b0 con 0x7f26e8107b70 2024-03-28T17:37:22.440 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.483+0000 7f26ed200700 1 --2- 172.21.15.45:0/630716166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26e8107b70 0x7f26e807db70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f26d8011850 tx=0x7f26d8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:22.440 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.483+0000 7f26e6000700 1 -- 172.21.15.45:0/630716166 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26d8003af0 con 0x7f26e8107b70 2024-03-28T17:37:22.440 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.483+0000 7f26e6000700 1 -- 172.21.15.45:0/630716166 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26d8020070 con 0x7f26e8107b70 2024-03-28T17:37:22.440 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.483+0000 7f26efc00700 1 -- 172.21.15.45:0/630716166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e807c2a0 con 0x7f26e8107b70 2024-03-28T17:37:22.440 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.483+0000 7f26e6000700 1 -- 172.21.15.45:0/630716166 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26d80195a0 con 0x7f26e8107b70 2024-03-28T17:37:22.440 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.483+0000 7f26efc00700 1 -- 172.21.15.45:0/630716166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e807c7e0 con 0x7f26e8107b70 2024-03-28T17:37:22.440 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.484+0000 7f26e6000700 1 -- 172.21.15.45:0/630716166 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f26d801a430 con 0x7f26e8107b70 2024-03-28T17:37:22.440 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.484+0000 7f26efc00700 1 -- 172.21.15.45:0/630716166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e80637c0 con 0x7f26e8107b70 2024-03-28T17:37:22.440 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.485+0000 7f26e6000700 1 --2- 172.21.15.45:0/630716166 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f26cc038f20 0x7f26cc03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:22.440 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.485+0000 7f26e6000700 1 -- 172.21.15.45:0/630716166 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f26d8011e00 con 0x7f26e8107b70 2024-03-28T17:37:22.441 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.485+0000 7f26dfe00700 1 --2- 172.21.15.45:0/630716166 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f26cc038f20 0x7f26cc03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:22.441 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.486+0000 7f26dfe00700 1 --2- 172.21.15.45:0/630716166 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f26cc038f20 0x7f26cc03b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f26d0006fd0 tx=0x7f26d0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:22.441 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.490+0000 7f26e6000700 1 -- 172.21.15.45:0/630716166 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f26d8024e80 con 0x7f26e8107b70 2024-03-28T17:37:22.441 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.718+0000 7f26efc00700 1 -- 172.21.15.45:0/630716166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f26e807ca60 con 0x7f26e8107b70 2024-03-28T17:37:22.441 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.719+0000 7f26e6000700 1 -- 172.21.15.45:0/630716166 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f26d804cbc0 con 0x7f26e8107b70 2024-03-28T17:37:22.442 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.722+0000 7f26efc00700 1 -- 172.21.15.45:0/630716166 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f26cc038f20 msgr2=0x7f26cc03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:22.442 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.722+0000 7f26efc00700 1 --2- 172.21.15.45:0/630716166 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f26cc038f20 0x7f26cc03b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f26d0006fd0 tx=0x7f26d0006e40 comp rx=0 tx=0).stop 2024-03-28T17:37:22.442 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.722+0000 7f26efc00700 1 -- 172.21.15.45:0/630716166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26e8107b70 msgr2=0x7f26e807db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:22.443 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.722+0000 7f26efc00700 1 --2- 172.21.15.45:0/630716166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26e8107b70 0x7f26e807db70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f26d8011850 tx=0x7f26d8011c10 comp rx=0 tx=0).stop 2024-03-28T17:37:22.443 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.723+0000 7f26efc00700 1 -- 172.21.15.45:0/630716166 shutdown_connections 2024-03-28T17:37:22.443 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.723+0000 7f26efc00700 1 --2- 172.21.15.45:0/630716166 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f26cc038f20 0x7f26cc03b3e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:22.443 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.723+0000 7f26efc00700 1 --2- 172.21.15.45:0/630716166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26e8107b70 0x7f26e807db70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:22.443 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.723+0000 7f26efc00700 1 -- 172.21.15.45:0/630716166 >> 172.21.15.45:0/630716166 conn(0x7f26e8103110 msgr2=0x7f26e8103cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:22.443 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.723+0000 7f26efc00700 1 -- 172.21.15.45:0/630716166 shutdown_connections 2024-03-28T17:37:22.443 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:21.723+0000 7f26efc00700 1 -- 172.21.15.45:0/630716166 wait complete. 2024-03-28T17:37:22.443 INFO:teuthology.orchestra.run.smithi045.stdout:Enabling cephadm module... 2024-03-28T17:37:23.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:23 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3484549436' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-28T17:37:23.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: ignoring --setuser ceph since I am not root 2024-03-28T17:37:23.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: ignoring --setgroup ceph since I am not root 2024-03-28T17:37:24.048 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.031+0000 7f91cc600700 1 Processor -- start 2024-03-28T17:37:24.048 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.032+0000 7f91cc600700 1 -- start start 2024-03-28T17:37:24.048 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.032+0000 7f91cc600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c40822c0 0x7f91c4110ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:24.048 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.032+0000 7f91cc600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c41112b0 con 0x7f91c40822c0 2024-03-28T17:37:24.049 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.032+0000 7f91c9c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c40822c0 0x7f91c4110ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:24.049 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.033+0000 7f91c9c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c40822c0 0x7f91c4110ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59990/0 (socket says 172.21.15.45:59990) 2024-03-28T17:37:24.049 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.033+0000 7f91c9c00700 1 -- 172.21.15.45:0/2157076875 learned_addr learned my addr 172.21.15.45:0/2157076875 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:24.049 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.033+0000 7f91c9c00700 1 -- 172.21.15.45:0/2157076875 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c4111430 con 0x7f91c40822c0 2024-03-28T17:37:24.049 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.033+0000 7f91c9c00700 1 --2- 172.21.15.45:0/2157076875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c40822c0 0x7f91c4110ce0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f91b80098a0 tx=0x7f91b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b81645249c077d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:24.049 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.034+0000 7f91c3e00700 1 -- 172.21.15.45:0/2157076875 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91b8004160 con 0x7f91c40822c0 2024-03-28T17:37:24.049 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.034+0000 7f91c3e00700 1 -- 172.21.15.45:0/2157076875 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f91b8004300 con 0x7f91c40822c0 2024-03-28T17:37:24.049 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.034+0000 7f91c3e00700 1 -- 172.21.15.45:0/2157076875 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91b8011b50 con 0x7f91c40822c0 2024-03-28T17:37:24.049 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.034+0000 7f91cc600700 1 -- 172.21.15.45:0/2157076875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c40822c0 msgr2=0x7f91c4110ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:24.049 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.034+0000 7f91cc600700 1 --2- 172.21.15.45:0/2157076875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c40822c0 0x7f91c4110ce0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f91b80098a0 tx=0x7f91b8009bb0 comp rx=0 tx=0).stop 2024-03-28T17:37:24.049 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.035+0000 7f91cc600700 1 -- 172.21.15.45:0/2157076875 shutdown_connections 2024-03-28T17:37:24.049 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.035+0000 7f91cc600700 1 --2- 172.21.15.45:0/2157076875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c40822c0 0x7f91c4110ce0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:24.049 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.035+0000 7f91cc600700 1 -- 172.21.15.45:0/2157076875 >> 172.21.15.45:0/2157076875 conn(0x7f91c407d0b0 msgr2=0x7f91c407f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:24.050 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.035+0000 7f91cc600700 1 -- 172.21.15.45:0/2157076875 shutdown_connections 2024-03-28T17:37:24.050 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.035+0000 7f91cc600700 1 -- 172.21.15.45:0/2157076875 wait complete. 2024-03-28T17:37:24.050 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.036+0000 7f91cc600700 1 Processor -- start 2024-03-28T17:37:24.050 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.036+0000 7f91cc600700 1 -- start start 2024-03-28T17:37:24.050 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.036+0000 7f91cc600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c40822c0 0x7f91c41a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:24.050 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.036+0000 7f91cc600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c41124c0 con 0x7f91c40822c0 2024-03-28T17:37:24.050 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.037+0000 7f91c9c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c40822c0 0x7f91c41a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:24.050 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.037+0000 7f91c9c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c40822c0 0x7f91c41a20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60006/0 (socket says 172.21.15.45:60006) 2024-03-28T17:37:24.050 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.037+0000 7f91c9c00700 1 -- 172.21.15.45:0/3484549436 learned_addr learned my addr 172.21.15.45:0/3484549436 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:24.050 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.037+0000 7f91c9c00700 1 -- 172.21.15.45:0/3484549436 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c41a2630 con 0x7f91c40822c0 2024-03-28T17:37:24.050 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.037+0000 7f91c9c00700 1 --2- 172.21.15.45:0/3484549436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c40822c0 0x7f91c41a20f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f91b8011b60 tx=0x7f91b8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:24.050 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.038+0000 7f91c2000700 1 -- 172.21.15.45:0/3484549436 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91b8003970 con 0x7f91c40822c0 2024-03-28T17:37:24.050 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.038+0000 7f91c2000700 1 -- 172.21.15.45:0/3484549436 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f91b8003b10 con 0x7f91c40822c0 2024-03-28T17:37:24.050 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.038+0000 7f91c2000700 1 -- 172.21.15.45:0/3484549436 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91b8024610 con 0x7f91c40822c0 2024-03-28T17:37:24.051 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.038+0000 7f91cc600700 1 -- 172.21.15.45:0/3484549436 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c41a28c0 con 0x7f91c40822c0 2024-03-28T17:37:24.051 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.038+0000 7f91cc600700 1 -- 172.21.15.45:0/3484549436 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c41a55b0 con 0x7f91c40822c0 2024-03-28T17:37:24.051 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.039+0000 7f91c2000700 1 -- 172.21.15.45:0/3484549436 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f91b801a9f0 con 0x7f91c40822c0 2024-03-28T17:37:24.051 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.039+0000 7f91cc600700 1 -- 172.21.15.45:0/3484549436 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91c40637c0 con 0x7f91c40822c0 2024-03-28T17:37:24.051 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.040+0000 7f91c2000700 1 --2- 172.21.15.45:0/3484549436 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f91ac038f20 0x7f91ac03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:24.051 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.040+0000 7f91c2000700 1 -- 172.21.15.45:0/3484549436 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f91b8011e00 con 0x7f91c40822c0 2024-03-28T17:37:24.051 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.040+0000 7f91c9200700 1 --2- 172.21.15.45:0/3484549436 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f91ac038f20 0x7f91ac03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:24.051 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.041+0000 7f91c9200700 1 --2- 172.21.15.45:0/3484549436 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f91ac038f20 0x7f91ac03b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f91b0006fd0 tx=0x7f91b0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:24.051 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.045+0000 7f91c2000700 1 -- 172.21.15.45:0/3484549436 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f91b800b3e0 con 0x7f91c40822c0 2024-03-28T17:37:24.051 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.333+0000 7f91cc600700 1 -- 172.21.15.45:0/3484549436 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f91c41a5780 con 0x7f91c40822c0 2024-03-28T17:37:24.051 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.387+0000 7f91c2000700 1 -- 172.21.15.45:0/3484549436 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f91c41a5780 con 0x7f91c40822c0 2024-03-28T17:37:24.051 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.390+0000 7f91c2000700 1 -- 172.21.15.45:0/3484549436 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f91b80172b0 con 0x7f91c40822c0 2024-03-28T17:37:24.051 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.391+0000 7f91cc600700 1 -- 172.21.15.45:0/3484549436 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f91ac038f20 msgr2=0x7f91ac03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:24.052 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.391+0000 7f91cc600700 1 --2- 172.21.15.45:0/3484549436 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f91ac038f20 0x7f91ac03b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f91b0006fd0 tx=0x7f91b0006e40 comp rx=0 tx=0).stop 2024-03-28T17:37:24.052 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.391+0000 7f91cc600700 1 -- 172.21.15.45:0/3484549436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c40822c0 msgr2=0x7f91c41a20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:24.052 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.391+0000 7f91cc600700 1 --2- 172.21.15.45:0/3484549436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c40822c0 0x7f91c41a20f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f91b8011b60 tx=0x7f91b8011c40 comp rx=0 tx=0).stop 2024-03-28T17:37:24.052 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.393+0000 7f91cc600700 1 -- 172.21.15.45:0/3484549436 shutdown_connections 2024-03-28T17:37:24.052 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.393+0000 7f91cc600700 1 --2- 172.21.15.45:0/3484549436 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f91ac038f20 0x7f91ac03b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:24.052 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.393+0000 7f91cc600700 1 --2- 172.21.15.45:0/3484549436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c40822c0 0x7f91c41a20f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:24.052 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.393+0000 7f91cc600700 1 -- 172.21.15.45:0/3484549436 >> 172.21.15.45:0/3484549436 conn(0x7f91c407d0b0 msgr2=0x7f91c407dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:24.052 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.393+0000 7f91cc600700 1 -- 172.21.15.45:0/3484549436 shutdown_connections 2024-03-28T17:37:24.052 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:23.393+0000 7f91cc600700 1 -- 172.21.15.45:0/3484549436 wait complete. 2024-03-28T17:37:24.320 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:24.193+0000 7f0e22c40200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T17:37:24.639 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:24 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3484549436' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-28T17:37:24.639 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:24 smithi045 ceph-mon[31773]: mgrmap e5: x(active, since 5s) 2024-03-28T17:37:24.639 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:24.319+0000 7f0e22c40200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T17:37:25.270 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:25 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:25.144+0000 7f0e22c40200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T17:37:25.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:25 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1012439050' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-28T17:37:25.582 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:25 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:25.269+0000 7f0e22c40200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T17:37:25.582 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:25 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:25.453+0000 7f0e22c40200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T17:37:25.687 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-28T17:37:25.688 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-28T17:37:25.688 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T17:37:25.688 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-28T17:37:25.688 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-28T17:37:25.688 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-28T17:37:25.688 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.652+0000 7f7169400700 1 Processor -- start 2024-03-28T17:37:25.688 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.652+0000 7f7169400700 1 -- start start 2024-03-28T17:37:25.688 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.652+0000 7f7169400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7164108660 0x7f716410aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:25.688 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.652+0000 7f7169400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716407bdf0 con 0x7f7164108660 2024-03-28T17:37:25.688 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.652+0000 7f7162800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7164108660 0x7f716410aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:25.688 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.653+0000 7f7162800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7164108660 0x7f716410aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60042/0 (socket says 172.21.15.45:60042) 2024-03-28T17:37:25.688 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.653+0000 7f7162800700 1 -- 172.21.15.45:0/304160567 learned_addr learned my addr 172.21.15.45:0/304160567 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:25.689 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.653+0000 7f7162800700 1 -- 172.21.15.45:0/304160567 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716410af90 con 0x7f7164108660 2024-03-28T17:37:25.689 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.653+0000 7f7162800700 1 --2- 172.21.15.45:0/304160567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7164108660 0x7f716410aa50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f71540098a0 tx=0x7f7154009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aa1cd09feb1bfdc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:25.689 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.654+0000 7f7161400700 1 -- 172.21.15.45:0/304160567 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7154004160 con 0x7f7164108660 2024-03-28T17:37:25.689 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.654+0000 7f7161400700 1 -- 172.21.15.45:0/304160567 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7154004300 con 0x7f7164108660 2024-03-28T17:37:25.689 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.654+0000 7f7161400700 1 -- 172.21.15.45:0/304160567 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7154011b50 con 0x7f7164108660 2024-03-28T17:37:25.689 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.654+0000 7f7169400700 1 -- 172.21.15.45:0/304160567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7164108660 msgr2=0x7f716410aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:25.689 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.654+0000 7f7169400700 1 --2- 172.21.15.45:0/304160567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7164108660 0x7f716410aa50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f71540098a0 tx=0x7f7154009bb0 comp rx=0 tx=0).stop 2024-03-28T17:37:25.689 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.655+0000 7f7169400700 1 -- 172.21.15.45:0/304160567 shutdown_connections 2024-03-28T17:37:25.689 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.655+0000 7f7169400700 1 --2- 172.21.15.45:0/304160567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7164108660 0x7f716410aa50 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:25.689 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.655+0000 7f7169400700 1 -- 172.21.15.45:0/304160567 >> 172.21.15.45:0/304160567 conn(0x7f7164101ff0 msgr2=0x7f7164104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:25.689 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.655+0000 7f7169400700 1 -- 172.21.15.45:0/304160567 shutdown_connections 2024-03-28T17:37:25.689 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.655+0000 7f7169400700 1 -- 172.21.15.45:0/304160567 wait complete. 2024-03-28T17:37:25.689 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.656+0000 7f7169400700 1 Processor -- start 2024-03-28T17:37:25.690 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.656+0000 7f7169400700 1 -- start start 2024-03-28T17:37:25.690 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.656+0000 7f7169400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7164108660 0x7f7164107ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:25.690 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.656+0000 7f7169400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716410bf00 con 0x7f7164108660 2024-03-28T17:37:25.690 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.657+0000 7f7162800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7164108660 0x7f7164107ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:25.690 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.657+0000 7f7162800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7164108660 0x7f7164107ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60046/0 (socket says 172.21.15.45:60046) 2024-03-28T17:37:25.690 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.657+0000 7f7162800700 1 -- 172.21.15.45:0/1012439050 learned_addr learned my addr 172.21.15.45:0/1012439050 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:25.690 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.657+0000 7f7162800700 1 -- 172.21.15.45:0/1012439050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7164108420 con 0x7f7164108660 2024-03-28T17:37:25.690 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.657+0000 7f7162800700 1 --2- 172.21.15.45:0/1012439050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7164108660 0x7f7164107ee0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7154011b60 tx=0x7f7154011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:25.690 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.657+0000 7f715b400700 1 -- 172.21.15.45:0/1012439050 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7154003970 con 0x7f7164108660 2024-03-28T17:37:25.690 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.658+0000 7f715b400700 1 -- 172.21.15.45:0/1012439050 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7154003b10 con 0x7f7164108660 2024-03-28T17:37:25.690 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.658+0000 7f715b400700 1 -- 172.21.15.45:0/1012439050 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7154024610 con 0x7f7164108660 2024-03-28T17:37:25.690 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.658+0000 7f7169400700 1 -- 172.21.15.45:0/1012439050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7164106610 con 0x7f7164108660 2024-03-28T17:37:25.690 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.658+0000 7f7169400700 1 -- 172.21.15.45:0/1012439050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7164106b50 con 0x7f7164108660 2024-03-28T17:37:25.691 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.659+0000 7f7169400700 1 -- 172.21.15.45:0/1012439050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71640637c0 con 0x7f7164108660 2024-03-28T17:37:25.691 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.659+0000 7f715b400700 1 -- 172.21.15.45:0/1012439050 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f715401a9f0 con 0x7f7164108660 2024-03-28T17:37:25.691 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.659+0000 7f715b400700 1 --2- 172.21.15.45:0/1012439050 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f7148038fd0 0x7f714803b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:25.691 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.660+0000 7f715b400700 1 -- 172.21.15.45:0/1012439050 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7154011e00 con 0x7f7164108660 2024-03-28T17:37:25.691 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.660+0000 7f7161e00700 1 -- 172.21.15.45:0/1012439050 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f7148038fd0 msgr2=0x7f714803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/4186185610 2024-03-28T17:37:25.691 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.660+0000 7f7161e00700 1 --2- 172.21.15.45:0/1012439050 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f7148038fd0 0x7f714803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T17:37:25.691 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.665+0000 7f715b400700 1 -- 172.21.15.45:0/1012439050 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7154019de0 con 0x7f7164108660 2024-03-28T17:37:25.691 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.861+0000 7f7161e00700 1 -- 172.21.15.45:0/1012439050 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f7148038fd0 msgr2=0x7f714803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/4186185610 2024-03-28T17:37:25.691 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.861+0000 7f7161e00700 1 --2- 172.21.15.45:0/1012439050 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f7148038fd0 0x7f714803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T17:37:25.691 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.950+0000 7f7169400700 1 -- 172.21.15.45:0/1012439050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f7164107480 con 0x7f7164108660 2024-03-28T17:37:25.691 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.950+0000 7f715b400700 1 -- 172.21.15.45:0/1012439050 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f715400b5a0 con 0x7f7164108660 2024-03-28T17:37:25.691 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.954+0000 7f7169400700 1 -- 172.21.15.45:0/1012439050 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f7148038fd0 msgr2=0x7f714803b490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T17:37:25.691 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.954+0000 7f7169400700 1 --2- 172.21.15.45:0/1012439050 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f7148038fd0 0x7f714803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:25.691 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.954+0000 7f7169400700 1 -- 172.21.15.45:0/1012439050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7164108660 msgr2=0x7f7164107ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:25.692 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.954+0000 7f7169400700 1 --2- 172.21.15.45:0/1012439050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7164108660 0x7f7164107ee0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7154011b60 tx=0x7f7154011c40 comp rx=0 tx=0).stop 2024-03-28T17:37:25.692 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.954+0000 7f7169400700 1 -- 172.21.15.45:0/1012439050 shutdown_connections 2024-03-28T17:37:25.692 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.954+0000 7f7169400700 1 --2- 172.21.15.45:0/1012439050 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f7148038fd0 0x7f714803b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:25.692 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.954+0000 7f7169400700 1 --2- 172.21.15.45:0/1012439050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7164108660 0x7f7164107ee0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:25.692 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.954+0000 7f7169400700 1 -- 172.21.15.45:0/1012439050 >> 172.21.15.45:0/1012439050 conn(0x7f7164101ff0 msgr2=0x7f7164102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:25.692 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.954+0000 7f7169400700 1 -- 172.21.15.45:0/1012439050 shutdown_connections 2024-03-28T17:37:25.692 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:24.954+0000 7f7169400700 1 -- 172.21.15.45:0/1012439050 wait complete. 2024-03-28T17:37:25.692 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for the mgr to restart... 2024-03-28T17:37:25.692 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mgr epoch 5... 2024-03-28T17:37:25.835 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:25 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:25.581+0000 7f0e22c40200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T17:37:25.835 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:25 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:25.696+0000 7f0e22c40200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T17:37:26.146 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:26.047+0000 7f0e22c40200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T17:37:26.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:26.197+0000 7f0e22c40200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T17:37:26.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:26.313+0000 7f0e22c40200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T17:37:26.824 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:26.441+0000 7f0e22c40200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T17:37:26.824 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:26.662+0000 7f0e22c40200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T17:37:28.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:27 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:27.966+0000 7f0e22c40200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T17:37:28.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:28.142+0000 7f0e22c40200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T17:37:28.916 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:28.567+0000 7f0e22c40200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T17:37:28.916 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:28.688+0000 7f0e22c40200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T17:37:29.205 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:28.952+0000 7f0e22c40200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T17:37:29.590 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:29 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:29.204+0000 7f0e22c40200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T17:37:29.590 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:29 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:29.588+0000 7f0e22c40200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T17:37:30.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:29 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:29.911+0000 7f0e22c40200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T17:37:30.665 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:30 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:30.223+0000 7f0e22c40200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T17:37:31.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:31.282+0000 7f0e22c40200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T17:37:32.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:31.995+0000 7f0e22c40200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T17:37:32.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:32 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:32.129+0000 7f0e22c40200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T17:37:32.916 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:32 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:32.420+0000 7f0e22c40200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T17:37:32.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:32 smithi045 ceph-mon[31773]: Active manager daemon x restarted 2024-03-28T17:37:32.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:32 smithi045 ceph-mon[31773]: Activating manager daemon x 2024-03-28T17:37:32.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:32 smithi045 ceph-mon[31773]: osdmap e2: 0 total, 0 up, 0 in 2024-03-28T17:37:32.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:32 smithi045 ceph-mon[31773]: mgrmap e6: x(active, starting, since 0.00963107s) 2024-03-28T17:37:32.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:32 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T17:37:32.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:32 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T17:37:32.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:32 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T17:37:32.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:32 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T17:37:32.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:32 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T17:37:32.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:32 smithi045 ceph-mon[31773]: Manager daemon x is now available 2024-03-28T17:37:32.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:32 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' 2024-03-28T17:37:33.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:33 smithi045 ceph-mon[31773]: Found migration_current of "None". Setting to last migration. 2024-03-28T17:37:33.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:33 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' 2024-03-28T17:37:33.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:33 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:37:33.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:33 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:37:33.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:33 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:37:33.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:33 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T17:37:33.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:33 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T17:37:33.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:33 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' 2024-03-28T17:37:33.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:33 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' 2024-03-28T17:37:33.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:33 smithi045 ceph-mon[31773]: mgrmap e7: x(active, since 1.01675s) 2024-03-28T17:37:34.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-28T17:37:34.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-28T17:37:34.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-28T17:37:34.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-28T17:37:34.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.213+0000 7f9dd0600700 1 Processor -- start 2024-03-28T17:37:34.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.213+0000 7f9dd0600700 1 -- start start 2024-03-28T17:37:34.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.214+0000 7f9dd0600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dc807de10 0x7f9dc807c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:34.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.214+0000 7f9dd0600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc807c800 con 0x7f9dc807de10 2024-03-28T17:37:34.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.214+0000 7f9dcdc00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dc807de10 0x7f9dc807c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:34.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.214+0000 7f9dcdc00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dc807de10 0x7f9dc807c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48924/0 (socket says 172.21.15.45:48924) 2024-03-28T17:37:34.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.214+0000 7f9dcdc00700 1 -- 172.21.15.45:0/4130218361 learned_addr learned my addr 172.21.15.45:0/4130218361 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:34.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.215+0000 7f9dcdc00700 1 -- 172.21.15.45:0/4130218361 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc807c980 con 0x7f9dc807de10 2024-03-28T17:37:34.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.215+0000 7f9dcdc00700 1 --2- 172.21.15.45:0/4130218361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dc807de10 0x7f9dc807c2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc0098a0 tx=0x7f9dbc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=872d235aac25dddc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:34.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.215+0000 7f9dc7e00700 1 -- 172.21.15.45:0/4130218361 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9dbc004160 con 0x7f9dc807de10 2024-03-28T17:37:34.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.215+0000 7f9dc7e00700 1 -- 172.21.15.45:0/4130218361 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9dbc004300 con 0x7f9dc807de10 2024-03-28T17:37:34.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.215+0000 7f9dc7e00700 1 -- 172.21.15.45:0/4130218361 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9dbc011b50 con 0x7f9dc807de10 2024-03-28T17:37:34.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.216+0000 7f9dd0600700 1 -- 172.21.15.45:0/4130218361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dc807de10 msgr2=0x7f9dc807c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:34.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.216+0000 7f9dd0600700 1 --2- 172.21.15.45:0/4130218361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dc807de10 0x7f9dc807c2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc0098a0 tx=0x7f9dbc009bb0 comp rx=0 tx=0).stop 2024-03-28T17:37:34.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.216+0000 7f9dd0600700 1 -- 172.21.15.45:0/4130218361 shutdown_connections 2024-03-28T17:37:34.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.216+0000 7f9dd0600700 1 --2- 172.21.15.45:0/4130218361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dc807de10 0x7f9dc807c2c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:34.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.216+0000 7f9dd0600700 1 -- 172.21.15.45:0/4130218361 >> 172.21.15.45:0/4130218361 conn(0x7f9dc8103110 msgr2=0x7f9dc8105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:34.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.216+0000 7f9dd0600700 1 -- 172.21.15.45:0/4130218361 shutdown_connections 2024-03-28T17:37:34.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.216+0000 7f9dd0600700 1 -- 172.21.15.45:0/4130218361 wait complete. 2024-03-28T17:37:34.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.217+0000 7f9dd0600700 1 Processor -- start 2024-03-28T17:37:34.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.217+0000 7f9dd0600700 1 -- start start 2024-03-28T17:37:34.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.218+0000 7f9dd0600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dc807de10 0x7f9dc819dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:34.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.218+0000 7f9dd0600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc810e160 con 0x7f9dc807de10 2024-03-28T17:37:34.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.218+0000 7f9dcdc00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dc807de10 0x7f9dc819dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:34.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.218+0000 7f9dcdc00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dc807de10 0x7f9dc819dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48928/0 (socket says 172.21.15.45:48928) 2024-03-28T17:37:34.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.218+0000 7f9dcdc00700 1 -- 172.21.15.45:0/139945563 learned_addr learned my addr 172.21.15.45:0/139945563 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:34.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.218+0000 7f9dcdc00700 1 -- 172.21.15.45:0/139945563 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc819e250 con 0x7f9dc807de10 2024-03-28T17:37:34.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.219+0000 7f9dcdc00700 1 --2- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dc807de10 0x7f9dc819dd10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc011b60 tx=0x7f9dbc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:34.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.219+0000 7f9dc6000700 1 -- 172.21.15.45:0/139945563 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9dbc003970 con 0x7f9dc807de10 2024-03-28T17:37:34.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.219+0000 7f9dc6000700 1 -- 172.21.15.45:0/139945563 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9dbc003b10 con 0x7f9dc807de10 2024-03-28T17:37:34.126 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.219+0000 7f9dc6000700 1 -- 172.21.15.45:0/139945563 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9dbc024610 con 0x7f9dc807de10 2024-03-28T17:37:34.126 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.219+0000 7f9dd0600700 1 -- 172.21.15.45:0/139945563 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc819e4e0 con 0x7f9dc807de10 2024-03-28T17:37:34.126 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.219+0000 7f9dd0600700 1 -- 172.21.15.45:0/139945563 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc81a11c0 con 0x7f9dc807de10 2024-03-28T17:37:34.126 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.221+0000 7f9dc6000700 1 -- 172.21.15.45:0/139945563 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f9dbc01a9f0 con 0x7f9dc807de10 2024-03-28T17:37:34.126 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.221+0000 7f9dc6000700 1 --2- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f9db0038f70 0x7f9db003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:34.126 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.221+0000 7f9dc6000700 1 -- 172.21.15.45:0/139945563 --> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9db003bb40 con 0x7f9db0038f70 2024-03-28T17:37:34.126 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.221+0000 7f9dc6000700 1 -- 172.21.15.45:0/139945563 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9dbc011e00 con 0x7f9dc807de10 2024-03-28T17:37:34.126 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.221+0000 7f9dcd200700 1 -- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f9db0038f70 msgr2=0x7f9db003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/4186185610 2024-03-28T17:37:34.126 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.221+0000 7f9dcd200700 1 --2- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f9db0038f70 0x7f9db003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T17:37:34.126 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.422+0000 7f9dcd200700 1 -- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f9db0038f70 msgr2=0x7f9db003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/4186185610 2024-03-28T17:37:34.126 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.422+0000 7f9dcd200700 1 --2- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f9db0038f70 0x7f9db003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T17:37:34.126 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.823+0000 7f9dcd200700 1 -- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f9db0038f70 msgr2=0x7f9db003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/4186185610 2024-03-28T17:37:34.126 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:26.823+0000 7f9dcd200700 1 --2- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f9db0038f70 0x7f9db003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T17:37:34.126 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:27.624+0000 7f9dcd200700 1 -- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f9db0038f70 msgr2=0x7f9db003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/4186185610 2024-03-28T17:37:34.127 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:27.624+0000 7f9dcd200700 1 --2- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f9db0038f70 0x7f9db003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T17:37:34.127 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:29.226+0000 7f9dcd200700 1 -- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f9db0038f70 msgr2=0x7f9db003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/4186185610 2024-03-28T17:37:34.127 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:29.226+0000 7f9dcd200700 1 --2- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f9db0038f70 0x7f9db003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T17:37:34.127 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:32.429+0000 7f9dc6000700 1 -- 172.21.15.45:0/139945563 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mgrmap(e 6) v1 ==== 45873+0+0 (secure 0 0 0) 0x7f9dbc01a430 con 0x7f9dc807de10 2024-03-28T17:37:34.127 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:32.430+0000 7f9dc6000700 1 -- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f9db0038f70 msgr2=0x7f9db003b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T17:37:34.127 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:32.430+0000 7f9dcd200700 1 -- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f9db0038f70 msgr2=0x7f9db003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/4186185610 2024-03-28T17:37:34.127 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:32.430+0000 7f9dcd200700 1 --2- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f9db0038f70 0x7f9db003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-28T17:37:34.127 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:32.430+0000 7f9dc6000700 1 --2- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f9db0038f70 0x7f9db003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:34.127 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:33.435+0000 7f9dc6000700 1 -- 172.21.15.45:0/139945563 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7f9dbc04efe0 con 0x7f9dc807de10 2024-03-28T17:37:34.127 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:33.435+0000 7f9dc6000700 1 --2- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f9db003c860 0x7f9db003ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:34.127 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:33.436+0000 7f9dc6000700 1 -- 172.21.15.45:0/139945563 --> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9db003bb40 con 0x7f9db003c860 2024-03-28T17:37:34.127 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:33.437+0000 7f9dcd200700 1 --2- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f9db003c860 0x7f9db003ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:34.127 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:33.439+0000 7f9dcd200700 1 --2- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f9db003c860 0x7f9db003ec50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9db4003a10 tx=0x7f9db40092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:34.127 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:33.441+0000 7f9dc6000700 1 -- 172.21.15.45:0/139945563 <== mgr.14120 v2:172.21.15.45:6800/3951447106 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f9db003bb40 con 0x7f9db003c860 2024-03-28T17:37:34.128 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:33.450+0000 7f9dd0600700 1 -- 172.21.15.45:0/139945563 --> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f9dc80637c0 con 0x7f9db003c860 2024-03-28T17:37:34.128 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:33.451+0000 7f9dc6000700 1 -- 172.21.15.45:0/139945563 <== mgr.14120 v2:172.21.15.45:6800/3951447106 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f9dc80637c0 con 0x7f9db003c860 2024-03-28T17:37:34.128 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:33.451+0000 7f9dd0600700 1 -- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f9db003c860 msgr2=0x7f9db003ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:34.128 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:33.451+0000 7f9dd0600700 1 --2- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f9db003c860 0x7f9db003ec50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9db4003a10 tx=0x7f9db40092b0 comp rx=0 tx=0).stop 2024-03-28T17:37:34.128 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:33.451+0000 7f9dd0600700 1 -- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dc807de10 msgr2=0x7f9dc819dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:34.128 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:33.451+0000 7f9dd0600700 1 --2- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dc807de10 0x7f9dc819dd10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc011b60 tx=0x7f9dbc011c40 comp rx=0 tx=0).stop 2024-03-28T17:37:34.128 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:33.451+0000 7f9dd0600700 1 -- 172.21.15.45:0/139945563 shutdown_connections 2024-03-28T17:37:34.128 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:33.451+0000 7f9dd0600700 1 --2- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f9db003c860 0x7f9db003ec50 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:34.128 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:33.451+0000 7f9dd0600700 1 --2- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dc807de10 0x7f9dc819dd10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:34.128 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:33.451+0000 7f9dd0600700 1 --2- 172.21.15.45:0/139945563 >> [v2:172.21.15.45:6800/4186185610,v1:172.21.15.45:6801/4186185610] conn(0x7f9db0038f70 0x7f9db003b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:34.128 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:33.451+0000 7f9dd0600700 1 -- 172.21.15.45:0/139945563 >> 172.21.15.45:0/139945563 conn(0x7f9dc8103110 msgr2=0x7f9dc8103cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:34.128 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:33.452+0000 7f9dd0600700 1 -- 172.21.15.45:0/139945563 shutdown_connections 2024-03-28T17:37:34.128 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:33.452+0000 7f9dd0600700 1 -- 172.21.15.45:0/139945563 wait complete. 2024-03-28T17:37:34.128 INFO:teuthology.orchestra.run.smithi045.stdout:mgr epoch 5 is available 2024-03-28T17:37:34.129 INFO:teuthology.orchestra.run.smithi045.stdout:Setting orchestrator backend to cephadm... 2024-03-28T17:37:34.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:34 smithi045 ceph-mon[31773]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-28T17:37:34.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:34 smithi045 ceph-mon[31773]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-28T17:37:35.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:35 smithi045 ceph-mon[31773]: [28/Mar/2024:17:37:34] ENGINE Bus STARTING 2024-03-28T17:37:35.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:35 smithi045 ceph-mon[31773]: [28/Mar/2024:17:37:34] ENGINE Serving on http://172.21.15.45:8765 2024-03-28T17:37:35.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:35 smithi045 ceph-mon[31773]: mgrmap e8: x(active, since 2s) 2024-03-28T17:37:35.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:35 smithi045 ceph-mon[31773]: [28/Mar/2024:17:37:34] ENGINE Serving on https://172.21.15.45:7150 2024-03-28T17:37:35.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:35 smithi045 ceph-mon[31773]: [28/Mar/2024:17:37:34] ENGINE Bus STARTED 2024-03-28T17:37:35.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:35 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:37:35.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:35 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' 2024-03-28T17:37:35.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:35 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:37:35.719 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.712+0000 7fd735e00700 1 Processor -- start 2024-03-28T17:37:35.719 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.712+0000 7fd735e00700 1 -- start start 2024-03-28T17:37:35.719 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.713+0000 7fd735e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd730109d10 0x7fd73010a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:35.720 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.713+0000 7fd735e00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73010a700 con 0x7fd730109d10 2024-03-28T17:37:35.720 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.713+0000 7fd72ea00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd730109d10 0x7fd73010a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:35.720 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.713+0000 7fd72ea00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd730109d10 0x7fd73010a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49008/0 (socket says 172.21.15.45:49008) 2024-03-28T17:37:35.720 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.713+0000 7fd72ea00700 1 -- 172.21.15.45:0/1998714478 learned_addr learned my addr 172.21.15.45:0/1998714478 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:35.721 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.714+0000 7fd72ea00700 1 -- 172.21.15.45:0/1998714478 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73010af10 con 0x7fd730109d10 2024-03-28T17:37:35.721 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.714+0000 7fd72ea00700 1 --2- 172.21.15.45:0/1998714478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd730109d10 0x7fd73010a130 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd7200098a0 tx=0x7fd720009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3395482acd1595a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:35.721 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.714+0000 7fd72d600700 1 -- 172.21.15.45:0/1998714478 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd720004160 con 0x7fd730109d10 2024-03-28T17:37:35.722 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.714+0000 7fd72d600700 1 -- 172.21.15.45:0/1998714478 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd720004300 con 0x7fd730109d10 2024-03-28T17:37:35.722 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.714+0000 7fd72d600700 1 -- 172.21.15.45:0/1998714478 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd720011b50 con 0x7fd730109d10 2024-03-28T17:37:35.722 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.715+0000 7fd735e00700 1 -- 172.21.15.45:0/1998714478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd730109d10 msgr2=0x7fd73010a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:35.722 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.715+0000 7fd735e00700 1 --2- 172.21.15.45:0/1998714478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd730109d10 0x7fd73010a130 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd7200098a0 tx=0x7fd720009bb0 comp rx=0 tx=0).stop 2024-03-28T17:37:35.722 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.715+0000 7fd735e00700 1 -- 172.21.15.45:0/1998714478 shutdown_connections 2024-03-28T17:37:35.723 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.715+0000 7fd735e00700 1 --2- 172.21.15.45:0/1998714478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd730109d10 0x7fd73010a130 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:35.723 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.715+0000 7fd735e00700 1 -- 172.21.15.45:0/1998714478 >> 172.21.15.45:0/1998714478 conn(0x7fd73007d100 msgr2=0x7fd73007d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:35.724 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.715+0000 7fd735e00700 1 -- 172.21.15.45:0/1998714478 shutdown_connections 2024-03-28T17:37:35.724 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.715+0000 7fd735e00700 1 -- 172.21.15.45:0/1998714478 wait complete. 2024-03-28T17:37:35.724 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.717+0000 7fd735e00700 1 Processor -- start 2024-03-28T17:37:35.724 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.718+0000 7fd735e00700 1 -- start start 2024-03-28T17:37:35.725 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.718+0000 7fd735e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd730109d10 0x7fd73019dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:35.725 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.718+0000 7fd735e00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73010e2c0 con 0x7fd730109d10 2024-03-28T17:37:35.725 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.718+0000 7fd72ea00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd730109d10 0x7fd73019dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:35.725 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.718+0000 7fd72ea00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd730109d10 0x7fd73019dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49018/0 (socket says 172.21.15.45:49018) 2024-03-28T17:37:35.725 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.718+0000 7fd72ea00700 1 -- 172.21.15.45:0/1173799836 learned_addr learned my addr 172.21.15.45:0/1173799836 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:35.725 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.719+0000 7fd72ea00700 1 -- 172.21.15.45:0/1173799836 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73019e290 con 0x7fd730109d10 2024-03-28T17:37:35.726 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.719+0000 7fd72ea00700 1 --2- 172.21.15.45:0/1173799836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd730109d10 0x7fd73019dd50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd720011850 tx=0x7fd720011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:35.726 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.720+0000 7fd734a00700 1 -- 172.21.15.45:0/1173799836 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd720003af0 con 0x7fd730109d10 2024-03-28T17:37:35.726 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.720+0000 7fd734a00700 1 -- 172.21.15.45:0/1173799836 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd720020070 con 0x7fd730109d10 2024-03-28T17:37:35.727 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.720+0000 7fd734a00700 1 -- 172.21.15.45:0/1173799836 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd7200195a0 con 0x7fd730109d10 2024-03-28T17:37:35.727 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.720+0000 7fd735e00700 1 -- 172.21.15.45:0/1173799836 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd73019e520 con 0x7fd730109d10 2024-03-28T17:37:35.727 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.720+0000 7fd735e00700 1 -- 172.21.15.45:0/1173799836 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7301a1210 con 0x7fd730109d10 2024-03-28T17:37:35.727 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.721+0000 7fd734a00700 1 -- 172.21.15.45:0/1173799836 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fd720019850 con 0x7fd730109d10 2024-03-28T17:37:35.728 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.721+0000 7fd735e00700 1 -- 172.21.15.45:0/1173799836 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7300637c0 con 0x7fd730109d10 2024-03-28T17:37:35.728 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.721+0000 7fd734a00700 1 --2- 172.21.15.45:0/1173799836 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fd714038f70 0x7fd71403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:35.728 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.722+0000 7fd734a00700 1 -- 172.21.15.45:0/1173799836 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd720011e00 con 0x7fd730109d10 2024-03-28T17:37:35.728 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.722+0000 7fd72e000700 1 --2- 172.21.15.45:0/1173799836 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fd714038f70 0x7fd71403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:35.728 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.723+0000 7fd72e000700 1 --2- 172.21.15.45:0/1173799836 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fd714038f70 0x7fd71403b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd71c006fd0 tx=0x7fd71c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:35.728 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.728+0000 7fd734a00700 1 -- 172.21.15.45:0/1173799836 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd720019b40 con 0x7fd730109d10 2024-03-28T17:37:35.728 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.977+0000 7fd735e00700 1 -- 172.21.15.45:0/1173799836 --> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fd730107bf0 con 0x7fd714038f70 2024-03-28T17:37:35.729 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.989+0000 7fd734a00700 1 -- 172.21.15.45:0/1173799836 <== mgr.14120 v2:172.21.15.45:6800/3951447106 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd730107bf0 con 0x7fd714038f70 2024-03-28T17:37:35.729 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.992+0000 7fd735e00700 1 -- 172.21.15.45:0/1173799836 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fd714038f70 msgr2=0x7fd71403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:35.730 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.992+0000 7fd735e00700 1 --2- 172.21.15.45:0/1173799836 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fd714038f70 0x7fd71403b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd71c006fd0 tx=0x7fd71c006e40 comp rx=0 tx=0).stop 2024-03-28T17:37:35.730 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.992+0000 7fd735e00700 1 -- 172.21.15.45:0/1173799836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd730109d10 msgr2=0x7fd73019dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:35.730 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.992+0000 7fd735e00700 1 --2- 172.21.15.45:0/1173799836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd730109d10 0x7fd73019dd50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd720011850 tx=0x7fd720011c10 comp rx=0 tx=0).stop 2024-03-28T17:37:35.730 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.993+0000 7fd735e00700 1 -- 172.21.15.45:0/1173799836 shutdown_connections 2024-03-28T17:37:35.730 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.993+0000 7fd735e00700 1 --2- 172.21.15.45:0/1173799836 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fd714038f70 0x7fd71403b430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:35.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.993+0000 7fd735e00700 1 --2- 172.21.15.45:0/1173799836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd730109d10 0x7fd73019dd50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:35.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.993+0000 7fd735e00700 1 -- 172.21.15.45:0/1173799836 >> 172.21.15.45:0/1173799836 conn(0x7fd73007d100 msgr2=0x7fd7301074e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:35.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.993+0000 7fd735e00700 1 -- 172.21.15.45:0/1173799836 shutdown_connections 2024-03-28T17:37:35.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:34.993+0000 7fd735e00700 1 -- 172.21.15.45:0/1173799836 wait complete. 2024-03-28T17:37:36.605 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:36 smithi045 ceph-mon[31773]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:37:37.243 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-28T17:37:37.243 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.352+0000 7efdc2600700 1 Processor -- start 2024-03-28T17:37:37.243 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.352+0000 7efdc2600700 1 -- start start 2024-03-28T17:37:37.243 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.353+0000 7efdc2600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdbc07dde0 0x7efdbc07c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:37.243 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.353+0000 7efdc2600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc07c820 con 0x7efdbc07dde0 2024-03-28T17:37:37.245 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.353+0000 7efdc1200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdbc07dde0 0x7efdbc07c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:37.245 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.353+0000 7efdc1200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdbc07dde0 0x7efdbc07c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53680/0 (socket says 172.21.15.45:53680) 2024-03-28T17:37:37.245 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.353+0000 7efdc1200700 1 -- 172.21.15.45:0/4059700259 learned_addr learned my addr 172.21.15.45:0/4059700259 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:37.245 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.354+0000 7efdc1200700 1 -- 172.21.15.45:0/4059700259 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdbc07c9a0 con 0x7efdbc07dde0 2024-03-28T17:37:37.245 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.354+0000 7efdc1200700 1 --2- 172.21.15.45:0/4059700259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdbc07dde0 0x7efdbc07c2e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efdb00098a0 tx=0x7efdb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aafa340175095c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:37.245 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.354+0000 7efdbb400700 1 -- 172.21.15.45:0/4059700259 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efdb0004160 con 0x7efdbc07dde0 2024-03-28T17:37:37.245 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.354+0000 7efdbb400700 1 -- 172.21.15.45:0/4059700259 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efdb0004300 con 0x7efdbc07dde0 2024-03-28T17:37:37.245 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.354+0000 7efdbb400700 1 -- 172.21.15.45:0/4059700259 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efdb0011b50 con 0x7efdbc07dde0 2024-03-28T17:37:37.245 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.355+0000 7efdc2600700 1 -- 172.21.15.45:0/4059700259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdbc07dde0 msgr2=0x7efdbc07c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:37.245 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.355+0000 7efdc2600700 1 --2- 172.21.15.45:0/4059700259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdbc07dde0 0x7efdbc07c2e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efdb00098a0 tx=0x7efdb0009bb0 comp rx=0 tx=0).stop 2024-03-28T17:37:37.246 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.355+0000 7efdc2600700 1 -- 172.21.15.45:0/4059700259 shutdown_connections 2024-03-28T17:37:37.248 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.355+0000 7efdc2600700 1 --2- 172.21.15.45:0/4059700259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdbc07dde0 0x7efdbc07c2e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:37.248 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.355+0000 7efdc2600700 1 -- 172.21.15.45:0/4059700259 >> 172.21.15.45:0/4059700259 conn(0x7efdbc102f50 msgr2=0x7efdbc105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:37.248 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.355+0000 7efdc2600700 1 -- 172.21.15.45:0/4059700259 shutdown_connections 2024-03-28T17:37:37.248 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.355+0000 7efdc2600700 1 -- 172.21.15.45:0/4059700259 wait complete. 2024-03-28T17:37:37.248 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.356+0000 7efdc2600700 1 Processor -- start 2024-03-28T17:37:37.248 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.357+0000 7efdc2600700 1 -- start start 2024-03-28T17:37:37.248 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.357+0000 7efdc2600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdbc07dde0 0x7efdbc19da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:37.249 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.357+0000 7efdc2600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc10df00 con 0x7efdbc07dde0 2024-03-28T17:37:37.249 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.357+0000 7efdc1200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdbc07dde0 0x7efdbc19da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:37.249 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.357+0000 7efdc1200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdbc07dde0 0x7efdbc19da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53694/0 (socket says 172.21.15.45:53694) 2024-03-28T17:37:37.249 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.357+0000 7efdc1200700 1 -- 172.21.15.45:0/2517091929 learned_addr learned my addr 172.21.15.45:0/2517091929 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:37.249 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.358+0000 7efdc1200700 1 -- 172.21.15.45:0/2517091929 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdbc19dfb0 con 0x7efdbc07dde0 2024-03-28T17:37:37.249 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.358+0000 7efdc1200700 1 --2- 172.21.15.45:0/2517091929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdbc07dde0 0x7efdbc19da70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efdb0011b60 tx=0x7efdb0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:37.249 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.358+0000 7efdb9600700 1 -- 172.21.15.45:0/2517091929 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efdb0003b20 con 0x7efdbc07dde0 2024-03-28T17:37:37.249 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.358+0000 7efdb9600700 1 -- 172.21.15.45:0/2517091929 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efdb0003cc0 con 0x7efdbc07dde0 2024-03-28T17:37:37.249 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.358+0000 7efdb9600700 1 -- 172.21.15.45:0/2517091929 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efdb0024840 con 0x7efdbc07dde0 2024-03-28T17:37:37.249 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.358+0000 7efdc2600700 1 -- 172.21.15.45:0/2517091929 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdbc19e240 con 0x7efdbc07dde0 2024-03-28T17:37:37.249 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.359+0000 7efdc2600700 1 -- 172.21.15.45:0/2517091929 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdbc1a0f30 con 0x7efdbc07dde0 2024-03-28T17:37:37.249 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.360+0000 7efdb9600700 1 -- 172.21.15.45:0/2517091929 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7efdb0022020 con 0x7efdbc07dde0 2024-03-28T17:37:37.249 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.360+0000 7efdc2600700 1 -- 172.21.15.45:0/2517091929 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdbc0637c0 con 0x7efdbc07dde0 2024-03-28T17:37:37.251 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.360+0000 7efdb9600700 1 --2- 172.21.15.45:0/2517091929 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7efda4038f70 0x7efda403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:37.251 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.360+0000 7efdb9600700 1 -- 172.21.15.45:0/2517091929 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7efdb0011e00 con 0x7efdbc07dde0 2024-03-28T17:37:37.252 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.361+0000 7efdbbe00700 1 --2- 172.21.15.45:0/2517091929 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7efda4038f70 0x7efda403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:37.252 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.361+0000 7efdbbe00700 1 --2- 172.21.15.45:0/2517091929 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7efda4038f70 0x7efda403b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efda8006fd0 tx=0x7efda8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:37.252 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.366+0000 7efdb9600700 1 -- 172.21.15.45:0/2517091929 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efdb000b3e0 con 0x7efdbc07dde0 2024-03-28T17:37:37.252 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.604+0000 7efdc2600700 1 -- 172.21.15.45:0/2517091929 --> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7efdbc1041d0 con 0x7efda4038f70 2024-03-28T17:37:37.252 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.607+0000 7efdb9600700 1 -- 172.21.15.45:0/2517091929 <== mgr.14120 v2:172.21.15.45:6800/3951447106 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7efdbc1041d0 con 0x7efda4038f70 2024-03-28T17:37:37.252 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.610+0000 7efdc2600700 1 -- 172.21.15.45:0/2517091929 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7efda4038f70 msgr2=0x7efda403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:37.252 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.610+0000 7efdc2600700 1 --2- 172.21.15.45:0/2517091929 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7efda4038f70 0x7efda403b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efda8006fd0 tx=0x7efda8006e40 comp rx=0 tx=0).stop 2024-03-28T17:37:37.252 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.610+0000 7efdc2600700 1 -- 172.21.15.45:0/2517091929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdbc07dde0 msgr2=0x7efdbc19da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:37.252 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.610+0000 7efdc2600700 1 --2- 172.21.15.45:0/2517091929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdbc07dde0 0x7efdbc19da70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efdb0011b60 tx=0x7efdb0011c40 comp rx=0 tx=0).stop 2024-03-28T17:37:37.252 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.610+0000 7efdc2600700 1 -- 172.21.15.45:0/2517091929 shutdown_connections 2024-03-28T17:37:37.252 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.610+0000 7efdc2600700 1 --2- 172.21.15.45:0/2517091929 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7efda4038f70 0x7efda403b430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:37.252 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.610+0000 7efdc2600700 1 --2- 172.21.15.45:0/2517091929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdbc07dde0 0x7efdbc19da70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:37.252 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.610+0000 7efdc2600700 1 -- 172.21.15.45:0/2517091929 >> 172.21.15.45:0/2517091929 conn(0x7efdbc102f50 msgr2=0x7efdbc103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:37.253 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.610+0000 7efdc2600700 1 -- 172.21.15.45:0/2517091929 shutdown_connections 2024-03-28T17:37:37.253 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:36.610+0000 7efdc2600700 1 -- 172.21.15.45:0/2517091929 wait complete. 2024-03-28T17:37:37.253 INFO:teuthology.orchestra.run.smithi045.stdout:Generating ssh key... 2024-03-28T17:37:37.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:37 smithi045 ceph-mon[31773]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:37:38.810 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:38 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: Generating public/private rsa key pair. 2024-03-28T17:37:38.810 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:38 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: Your identification has been saved in /tmp/tmpxzutzztw/key. 2024-03-28T17:37:38.810 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:38 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: Your public key has been saved in /tmp/tmpxzutzztw/key.pub. 2024-03-28T17:37:38.810 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:38 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: The key fingerprint is: 2024-03-28T17:37:38.810 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:38 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: SHA256:MJd4WEUC7d4GB9XFtrVMkLVHpWNm2g68VPic8/xeLP4 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715 2024-03-28T17:37:38.810 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:38 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: The key's randomart image is: 2024-03-28T17:37:38.810 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:38 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: +---[RSA 3072]----+ 2024-03-28T17:37:38.810 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:38 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: | .oo++. +=.+| 2024-03-28T17:37:38.810 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:38 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: | +oo .oo+o| 2024-03-28T17:37:38.811 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:38 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: | =.+. ..X+o| 2024-03-28T17:37:38.811 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:38 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: | =o .. X.=.| 2024-03-28T17:37:38.811 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:38 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: | .S+ = * | 2024-03-28T17:37:38.811 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:38 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: | . o. + = | 2024-03-28T17:37:38.811 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:38 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: | . . o =| 2024-03-28T17:37:38.811 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:38 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: | . .o| 2024-03-28T17:37:38.811 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:38 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: | .oE| 2024-03-28T17:37:38.811 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:38 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: +----[SHA256]-----+ 2024-03-28T17:37:39.132 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.872+0000 7f3073600700 1 Processor -- start 2024-03-28T17:37:39.132 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.873+0000 7f3073600700 1 -- start start 2024-03-28T17:37:39.132 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.873+0000 7f3073600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f306c07de90 0x7f306c07c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:39.133 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.873+0000 7f3073600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306c07c910 con 0x7f306c07de90 2024-03-28T17:37:39.133 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.874+0000 7f3070c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f306c07de90 0x7f306c07c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:39.133 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.874+0000 7f3070c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f306c07de90 0x7f306c07c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53700/0 (socket says 172.21.15.45:53700) 2024-03-28T17:37:39.133 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.874+0000 7f3070c00700 1 -- 172.21.15.45:0/1953431443 learned_addr learned my addr 172.21.15.45:0/1953431443 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:39.133 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.874+0000 7f3070c00700 1 -- 172.21.15.45:0/1953431443 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306c07ca90 con 0x7f306c07de90 2024-03-28T17:37:39.133 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.875+0000 7f3070c00700 1 --2- 172.21.15.45:0/1953431443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f306c07de90 0x7f306c07c340 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f30600098a0 tx=0x7f3060009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b409a9eb7cc16670 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:39.133 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.875+0000 7f306b400700 1 -- 172.21.15.45:0/1953431443 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3060004160 con 0x7f306c07de90 2024-03-28T17:37:39.133 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.875+0000 7f306b400700 1 -- 172.21.15.45:0/1953431443 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3060004300 con 0x7f306c07de90 2024-03-28T17:37:39.133 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.875+0000 7f306b400700 1 -- 172.21.15.45:0/1953431443 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3060011b50 con 0x7f306c07de90 2024-03-28T17:37:39.134 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.876+0000 7f3073600700 1 -- 172.21.15.45:0/1953431443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f306c07de90 msgr2=0x7f306c07c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:39.134 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.876+0000 7f3073600700 1 --2- 172.21.15.45:0/1953431443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f306c07de90 0x7f306c07c340 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f30600098a0 tx=0x7f3060009bb0 comp rx=0 tx=0).stop 2024-03-28T17:37:39.134 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.876+0000 7f3073600700 1 -- 172.21.15.45:0/1953431443 shutdown_connections 2024-03-28T17:37:39.134 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.876+0000 7f3073600700 1 --2- 172.21.15.45:0/1953431443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f306c07de90 0x7f306c07c340 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:39.134 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.876+0000 7f3073600700 1 -- 172.21.15.45:0/1953431443 >> 172.21.15.45:0/1953431443 conn(0x7f306c103110 msgr2=0x7f306c105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:39.134 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.876+0000 7f3073600700 1 -- 172.21.15.45:0/1953431443 shutdown_connections 2024-03-28T17:37:39.134 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.876+0000 7f3073600700 1 -- 172.21.15.45:0/1953431443 wait complete. 2024-03-28T17:37:39.134 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.877+0000 7f3073600700 1 Processor -- start 2024-03-28T17:37:39.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.877+0000 7f3073600700 1 -- start start 2024-03-28T17:37:39.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.878+0000 7f3073600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f306c07de90 0x7f306c19dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:39.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.878+0000 7f3073600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306c10e140 con 0x7f306c07de90 2024-03-28T17:37:39.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.878+0000 7f3070c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f306c07de90 0x7f306c19dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:39.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.878+0000 7f3070c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f306c07de90 0x7f306c19dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53716/0 (socket says 172.21.15.45:53716) 2024-03-28T17:37:39.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.878+0000 7f3070c00700 1 -- 172.21.15.45:0/2830105274 learned_addr learned my addr 172.21.15.45:0/2830105274 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:39.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.878+0000 7f3070c00700 1 -- 172.21.15.45:0/2830105274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306c19e250 con 0x7f306c07de90 2024-03-28T17:37:39.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.879+0000 7f3070c00700 1 --2- 172.21.15.45:0/2830105274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f306c07de90 0x7f306c19dd10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3060011b60 tx=0x7f3060011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:39.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.879+0000 7f3069600700 1 -- 172.21.15.45:0/2830105274 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3060003970 con 0x7f306c07de90 2024-03-28T17:37:39.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.879+0000 7f3069600700 1 -- 172.21.15.45:0/2830105274 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3060003b10 con 0x7f306c07de90 2024-03-28T17:37:39.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.879+0000 7f3073600700 1 -- 172.21.15.45:0/2830105274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f306c19e4e0 con 0x7f306c07de90 2024-03-28T17:37:39.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.879+0000 7f3069600700 1 -- 172.21.15.45:0/2830105274 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3060024610 con 0x7f306c07de90 2024-03-28T17:37:39.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.879+0000 7f3073600700 1 -- 172.21.15.45:0/2830105274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f306c1a11c0 con 0x7f306c07de90 2024-03-28T17:37:39.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.880+0000 7f3069600700 1 -- 172.21.15.45:0/2830105274 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f306001a9f0 con 0x7f306c07de90 2024-03-28T17:37:39.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.880+0000 7f3073600700 1 -- 172.21.15.45:0/2830105274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f306c0637c0 con 0x7f306c07de90 2024-03-28T17:37:39.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.881+0000 7f3069600700 1 --2- 172.21.15.45:0/2830105274 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f3054038f70 0x7f305403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:39.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.881+0000 7f3069600700 1 -- 172.21.15.45:0/2830105274 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f3060011e00 con 0x7f306c07de90 2024-03-28T17:37:39.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.881+0000 7f306be00700 1 --2- 172.21.15.45:0/2830105274 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f3054038f70 0x7f305403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:39.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.882+0000 7f306be00700 1 --2- 172.21.15.45:0/2830105274 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f3054038f70 0x7f305403b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3058006fd0 tx=0x7f3058006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:39.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:37.886+0000 7f3069600700 1 -- 172.21.15.45:0/2830105274 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3060019e10 con 0x7f306c07de90 2024-03-28T17:37:39.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:38.112+0000 7f3073600700 1 -- 172.21.15.45:0/2830105274 --> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f306c1043d0 con 0x7f3054038f70 2024-03-28T17:37:39.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:38.432+0000 7f3069600700 1 -- 172.21.15.45:0/2830105274 <== mgr.14120 v2:172.21.15.45:6800/3951447106 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f306c1043d0 con 0x7f3054038f70 2024-03-28T17:37:39.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:38.435+0000 7f3073600700 1 -- 172.21.15.45:0/2830105274 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f3054038f70 msgr2=0x7f305403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:39.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:38.435+0000 7f3073600700 1 --2- 172.21.15.45:0/2830105274 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f3054038f70 0x7f305403b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3058006fd0 tx=0x7f3058006e40 comp rx=0 tx=0).stop 2024-03-28T17:37:39.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:38.435+0000 7f3073600700 1 -- 172.21.15.45:0/2830105274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f306c07de90 msgr2=0x7f306c19dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:39.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:38.435+0000 7f3073600700 1 --2- 172.21.15.45:0/2830105274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f306c07de90 0x7f306c19dd10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3060011b60 tx=0x7f3060011c40 comp rx=0 tx=0).stop 2024-03-28T17:37:39.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:38.436+0000 7f3073600700 1 -- 172.21.15.45:0/2830105274 shutdown_connections 2024-03-28T17:37:39.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:38.436+0000 7f3073600700 1 --2- 172.21.15.45:0/2830105274 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f3054038f70 0x7f305403b430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:39.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:38.436+0000 7f3073600700 1 --2- 172.21.15.45:0/2830105274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f306c07de90 0x7f306c19dd10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:39.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:38.436+0000 7f3073600700 1 -- 172.21.15.45:0/2830105274 >> 172.21.15.45:0/2830105274 conn(0x7f306c103110 msgr2=0x7f306c103cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:39.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:38.436+0000 7f3073600700 1 -- 172.21.15.45:0/2830105274 shutdown_connections 2024-03-28T17:37:39.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:38.436+0000 7f3073600700 1 -- 172.21.15.45:0/2830105274 wait complete. 2024-03-28T17:37:39.731 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:39 smithi045 ceph-mon[31773]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:37:39.731 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:39 smithi045 ceph-mon[31773]: Generating ssh key... 2024-03-28T17:37:39.731 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:39 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' 2024-03-28T17:37:39.731 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:39 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' 2024-03-28T17:37:40.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:40 smithi045 ceph-mon[31773]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:37:40.609 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFbuyxvgdUhfq0nwnk1xUkPjL73wGaJmY8CRinLud8x75kIvoa/j+BE4X/1fsKx0oo1G8793Y+h7EP31Qxmg260wB5tg3SVbCZOjTbJ7oC/Dxct0scCzxLVdJ5xG+G8dz+CR5FBuj/0cBZhCV0MqvOVBQBcVzkH7MWkKkmdjkL7nxmi29H5p8QouGgSCEEllBv1MsdmXV5fy7YT3SlpZUKdELPaST+EXTJjCinXw1FwKfx21QxdFe6OpbLfOxQDklYA2WIPD300j3jQ611N9eBuGsvXn+oeFRan1omDnUuHAA+PqwsY/MeREjRfptyAQOhoMZ/D6XIiBHtQm5H4iQ+tqH+7l5DTojsf+iHny1jNu/DKrLQJwFaJx4RobxINQV7T68I9OR/onSNp4LOZMzsCgYbxzkn3UAqemB+a9uzDnTujOXqpsREQ0EgUXfZ6ESvEJHarmFW3pu2CVLMQlM96KXTFtBFXA+TAvrTkB5klBCBSEdJZomOwnLuGxBc7nc= ceph-9f4888ec-ed29-11ee-95d0-87774f69a715 2024-03-28T17:37:40.614 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.729+0000 7f7762000700 1 Processor -- start 2024-03-28T17:37:40.614 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.730+0000 7f7762000700 1 -- start start 2024-03-28T17:37:40.614 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.730+0000 7f7762000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f775c107b50 0x7f775c107f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:40.614 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.730+0000 7f7762000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775c108540 con 0x7f775c107b50 2024-03-28T17:37:40.614 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.731+0000 7f775b400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f775c107b50 0x7f775c107f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:40.614 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.731+0000 7f775b400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f775c107b50 0x7f775c107f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53720/0 (socket says 172.21.15.45:53720) 2024-03-28T17:37:40.614 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.731+0000 7f775b400700 1 -- 172.21.15.45:0/3889147375 learned_addr learned my addr 172.21.15.45:0/3889147375 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:40.614 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.731+0000 7f775b400700 1 -- 172.21.15.45:0/3889147375 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775c108d00 con 0x7f775c107b50 2024-03-28T17:37:40.614 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.732+0000 7f775b400700 1 --2- 172.21.15.45:0/3889147375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f775c107b50 0x7f775c107f70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f774c0098a0 tx=0x7f774c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c092e458d552a11a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:40.614 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.732+0000 7f775a000700 1 -- 172.21.15.45:0/3889147375 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f774c004160 con 0x7f775c107b50 2024-03-28T17:37:40.614 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.732+0000 7f775a000700 1 -- 172.21.15.45:0/3889147375 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f774c004300 con 0x7f775c107b50 2024-03-28T17:37:40.615 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.732+0000 7f775a000700 1 -- 172.21.15.45:0/3889147375 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f774c011b50 con 0x7f775c107b50 2024-03-28T17:37:40.615 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.733+0000 7f7762000700 1 -- 172.21.15.45:0/3889147375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f775c107b50 msgr2=0x7f775c107f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:40.615 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.733+0000 7f7762000700 1 --2- 172.21.15.45:0/3889147375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f775c107b50 0x7f775c107f70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f774c0098a0 tx=0x7f774c009bb0 comp rx=0 tx=0).stop 2024-03-28T17:37:40.615 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.733+0000 7f7762000700 1 -- 172.21.15.45:0/3889147375 shutdown_connections 2024-03-28T17:37:40.615 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.733+0000 7f7762000700 1 --2- 172.21.15.45:0/3889147375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f775c107b50 0x7f775c107f70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:40.615 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.733+0000 7f7762000700 1 -- 172.21.15.45:0/3889147375 >> 172.21.15.45:0/3889147375 conn(0x7f775c1030f0 msgr2=0x7f775c105530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:40.615 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.733+0000 7f7762000700 1 -- 172.21.15.45:0/3889147375 shutdown_connections 2024-03-28T17:37:40.615 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.733+0000 7f7762000700 1 -- 172.21.15.45:0/3889147375 wait complete. 2024-03-28T17:37:40.615 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.734+0000 7f7762000700 1 Processor -- start 2024-03-28T17:37:40.615 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.735+0000 7f7762000700 1 -- start start 2024-03-28T17:37:40.615 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.735+0000 7f7762000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f775c107b50 0x7f775c07db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:40.615 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.735+0000 7f7762000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775c10c0b0 con 0x7f775c107b50 2024-03-28T17:37:40.615 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.735+0000 7f775b400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f775c107b50 0x7f775c07db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:40.616 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.735+0000 7f775b400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f775c107b50 0x7f775c07db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53728/0 (socket says 172.21.15.45:53728) 2024-03-28T17:37:40.616 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.735+0000 7f775b400700 1 -- 172.21.15.45:0/3588308376 learned_addr learned my addr 172.21.15.45:0/3588308376 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:40.616 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.735+0000 7f775b400700 1 -- 172.21.15.45:0/3588308376 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775c07e0b0 con 0x7f775c107b50 2024-03-28T17:37:40.616 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.736+0000 7f775b400700 1 --2- 172.21.15.45:0/3588308376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f775c107b50 0x7f775c07db70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f774c011850 tx=0x7f774c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:40.616 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.736+0000 7f7753e00700 1 -- 172.21.15.45:0/3588308376 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f774c003af0 con 0x7f775c107b50 2024-03-28T17:37:40.616 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.736+0000 7f7753e00700 1 -- 172.21.15.45:0/3588308376 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f774c020070 con 0x7f775c107b50 2024-03-28T17:37:40.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.736+0000 7f7762000700 1 -- 172.21.15.45:0/3588308376 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f775c07c2a0 con 0x7f775c107b50 2024-03-28T17:37:40.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.736+0000 7f7753e00700 1 -- 172.21.15.45:0/3588308376 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f774c0195a0 con 0x7f775c107b50 2024-03-28T17:37:40.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.736+0000 7f7762000700 1 -- 172.21.15.45:0/3588308376 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f775c07c7e0 con 0x7f775c107b50 2024-03-28T17:37:40.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.738+0000 7f7762000700 1 -- 172.21.15.45:0/3588308376 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f775c0637c0 con 0x7f775c107b50 2024-03-28T17:37:40.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.738+0000 7f7753e00700 1 -- 172.21.15.45:0/3588308376 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f774c019850 con 0x7f775c107b50 2024-03-28T17:37:40.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.738+0000 7f7753e00700 1 --2- 172.21.15.45:0/3588308376 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f7740038f70 0x7f774003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:40.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.738+0000 7f7753e00700 1 -- 172.21.15.45:0/3588308376 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f774c011e00 con 0x7f775c107b50 2024-03-28T17:37:40.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.738+0000 7f775aa00700 1 --2- 172.21.15.45:0/3588308376 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f7740038f70 0x7f774003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:40.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.739+0000 7f775aa00700 1 --2- 172.21.15.45:0/3588308376 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f7740038f70 0x7f774003b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7748006fd0 tx=0x7f7748006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:40.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.744+0000 7f7753e00700 1 -- 172.21.15.45:0/3588308376 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f774c019b40 con 0x7f775c107b50 2024-03-28T17:37:40.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.970+0000 7f7762000700 1 -- 172.21.15.45:0/3588308376 --> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f775c104470 con 0x7f7740038f70 2024-03-28T17:37:40.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.972+0000 7f7753e00700 1 -- 172.21.15.45:0/3588308376 <== mgr.14120 v2:172.21.15.45:6800/3951447106 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f775c104470 con 0x7f7740038f70 2024-03-28T17:37:40.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.974+0000 7f7762000700 1 -- 172.21.15.45:0/3588308376 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f7740038f70 msgr2=0x7f774003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:40.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.974+0000 7f7762000700 1 --2- 172.21.15.45:0/3588308376 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f7740038f70 0x7f774003b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7748006fd0 tx=0x7f7748006e40 comp rx=0 tx=0).stop 2024-03-28T17:37:40.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.974+0000 7f7762000700 1 -- 172.21.15.45:0/3588308376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f775c107b50 msgr2=0x7f775c07db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:40.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.974+0000 7f7762000700 1 --2- 172.21.15.45:0/3588308376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f775c107b50 0x7f775c07db70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f774c011850 tx=0x7f774c011c10 comp rx=0 tx=0).stop 2024-03-28T17:37:40.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.975+0000 7f7762000700 1 -- 172.21.15.45:0/3588308376 shutdown_connections 2024-03-28T17:37:40.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.975+0000 7f7762000700 1 --2- 172.21.15.45:0/3588308376 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f7740038f70 0x7f774003b430 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:40.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.975+0000 7f7762000700 1 --2- 172.21.15.45:0/3588308376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f775c107b50 0x7f775c07db70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:40.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.975+0000 7f7762000700 1 -- 172.21.15.45:0/3588308376 >> 172.21.15.45:0/3588308376 conn(0x7f775c1030f0 msgr2=0x7f775c103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:40.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.975+0000 7f7762000700 1 -- 172.21.15.45:0/3588308376 shutdown_connections 2024-03-28T17:37:40.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:39.975+0000 7f7762000700 1 -- 172.21.15.45:0/3588308376 wait complete. 2024-03-28T17:37:40.621 INFO:teuthology.orchestra.run.smithi045.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-28T17:37:40.621 INFO:teuthology.orchestra.run.smithi045.stdout:Adding key to root@localhost authorized_keys... 2024-03-28T17:37:40.621 INFO:teuthology.orchestra.run.smithi045.stdout:Adding host smithi045... 2024-03-28T17:37:42.737 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:42 smithi045 ceph-mon[31773]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi045", "addr": "172.21.15.45", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:37:43.763 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:43 smithi045 ceph-mon[31773]: Deploying cephadm binary to smithi045 2024-03-28T17:37:48.904 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Added host 'smithi045' with addr '172.21.15.45' 2024-03-28T17:37:48.904 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.289+0000 7f6d12a00700 1 Processor -- start 2024-03-28T17:37:48.904 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.290+0000 7f6d12a00700 1 -- start start 2024-03-28T17:37:48.904 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.290+0000 7f6d12a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d0c109d90 0x7f6d0c10a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:48.904 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.290+0000 7f6d12a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0c10a780 con 0x7f6d0c109d90 2024-03-28T17:37:48.904 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.290+0000 7f6d0be00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d0c109d90 0x7f6d0c10a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:48.904 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.291+0000 7f6d0be00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d0c109d90 0x7f6d0c10a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53744/0 (socket says 172.21.15.45:53744) 2024-03-28T17:37:48.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.291+0000 7f6d0be00700 1 -- 172.21.15.45:0/2557413250 learned_addr learned my addr 172.21.15.45:0/2557413250 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:48.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.291+0000 7f6d0be00700 1 -- 172.21.15.45:0/2557413250 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d0c10aef0 con 0x7f6d0c109d90 2024-03-28T17:37:48.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.292+0000 7f6d0be00700 1 --2- 172.21.15.45:0/2557413250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d0c109d90 0x7f6d0c10a1b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc0098a0 tx=0x7f6cfc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=456634f85ca78578 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:48.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.292+0000 7f6d0aa00700 1 -- 172.21.15.45:0/2557413250 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6cfc004160 con 0x7f6d0c109d90 2024-03-28T17:37:48.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.292+0000 7f6d0aa00700 1 -- 172.21.15.45:0/2557413250 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6cfc004300 con 0x7f6d0c109d90 2024-03-28T17:37:48.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.292+0000 7f6d0aa00700 1 -- 172.21.15.45:0/2557413250 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6cfc011b50 con 0x7f6d0c109d90 2024-03-28T17:37:48.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.293+0000 7f6d12a00700 1 -- 172.21.15.45:0/2557413250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d0c109d90 msgr2=0x7f6d0c10a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:48.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.293+0000 7f6d12a00700 1 --2- 172.21.15.45:0/2557413250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d0c109d90 0x7f6d0c10a1b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc0098a0 tx=0x7f6cfc009bb0 comp rx=0 tx=0).stop 2024-03-28T17:37:48.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.293+0000 7f6d12a00700 1 -- 172.21.15.45:0/2557413250 shutdown_connections 2024-03-28T17:37:48.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.293+0000 7f6d12a00700 1 --2- 172.21.15.45:0/2557413250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d0c109d90 0x7f6d0c10a1b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:48.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.293+0000 7f6d12a00700 1 -- 172.21.15.45:0/2557413250 >> 172.21.15.45:0/2557413250 conn(0x7f6d0c07d200 msgr2=0x7f6d0c07d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:48.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.293+0000 7f6d12a00700 1 -- 172.21.15.45:0/2557413250 shutdown_connections 2024-03-28T17:37:48.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.293+0000 7f6d12a00700 1 -- 172.21.15.45:0/2557413250 wait complete. 2024-03-28T17:37:48.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.294+0000 7f6d12a00700 1 Processor -- start 2024-03-28T17:37:48.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.294+0000 7f6d12a00700 1 -- start start 2024-03-28T17:37:48.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.295+0000 7f6d12a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d0c109d90 0x7f6d0c19dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:48.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.295+0000 7f6d12a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0c10e2a0 con 0x7f6d0c109d90 2024-03-28T17:37:48.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.295+0000 7f6d0be00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d0c109d90 0x7f6d0c19dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:48.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.295+0000 7f6d0be00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d0c109d90 0x7f6d0c19dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53748/0 (socket says 172.21.15.45:53748) 2024-03-28T17:37:48.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.295+0000 7f6d0be00700 1 -- 172.21.15.45:0/2664783296 learned_addr learned my addr 172.21.15.45:0/2664783296 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:48.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.295+0000 7f6d0be00700 1 -- 172.21.15.45:0/2664783296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d0c19e270 con 0x7f6d0c109d90 2024-03-28T17:37:48.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.296+0000 7f6d0be00700 1 --2- 172.21.15.45:0/2664783296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d0c109d90 0x7f6d0c19dd30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc011850 tx=0x7f6cfc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:48.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.296+0000 7f6d08c00700 1 -- 172.21.15.45:0/2664783296 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6cfc003af0 con 0x7f6d0c109d90 2024-03-28T17:37:48.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.296+0000 7f6d08c00700 1 -- 172.21.15.45:0/2664783296 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6cfc020070 con 0x7f6d0c109d90 2024-03-28T17:37:48.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.296+0000 7f6d08c00700 1 -- 172.21.15.45:0/2664783296 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6cfc0195a0 con 0x7f6d0c109d90 2024-03-28T17:37:48.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.296+0000 7f6d12a00700 1 -- 172.21.15.45:0/2664783296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d0c19e500 con 0x7f6d0c109d90 2024-03-28T17:37:48.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.296+0000 7f6d12a00700 1 -- 172.21.15.45:0/2664783296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d0c1a11f0 con 0x7f6d0c109d90 2024-03-28T17:37:48.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.297+0000 7f6d08c00700 1 -- 172.21.15.45:0/2664783296 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f6cfc019850 con 0x7f6d0c109d90 2024-03-28T17:37:48.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.297+0000 7f6d12a00700 1 -- 172.21.15.45:0/2664783296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d0c0637c0 con 0x7f6d0c109d90 2024-03-28T17:37:48.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.298+0000 7f6d08c00700 1 --2- 172.21.15.45:0/2664783296 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f6cf0038f70 0x7f6cf003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:48.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.298+0000 7f6d08c00700 1 -- 172.21.15.45:0/2664783296 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6cfc011e00 con 0x7f6d0c109d90 2024-03-28T17:37:48.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.298+0000 7f6d0b400700 1 --2- 172.21.15.45:0/2664783296 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f6cf0038f70 0x7f6cf003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:48.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.299+0000 7f6d0b400700 1 --2- 172.21.15.45:0/2664783296 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f6cf0038f70 0x7f6cf003b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8006fd0 tx=0x7f6cf8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:48.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.303+0000 7f6d08c00700 1 -- 172.21.15.45:0/2664783296 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6cfc019b40 con 0x7f6d0c109d90 2024-03-28T17:37:48.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:41.534+0000 7f6d12a00700 1 -- 172.21.15.45:0/2664783296 --> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi045", "addr": "172.21.15.45", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d0c107c30 con 0x7f6cf0038f70 2024-03-28T17:37:48.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:48.205+0000 7f6d08c00700 1 -- 172.21.15.45:0/2664783296 <== mgr.14120 v2:172.21.15.45:6800/3951447106 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f6d0c107c30 con 0x7f6cf0038f70 2024-03-28T17:37:48.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:48.210+0000 7f6d12a00700 1 -- 172.21.15.45:0/2664783296 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f6cf0038f70 msgr2=0x7f6cf003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:48.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:48.210+0000 7f6d12a00700 1 --2- 172.21.15.45:0/2664783296 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f6cf0038f70 0x7f6cf003b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8006fd0 tx=0x7f6cf8006e40 comp rx=0 tx=0).stop 2024-03-28T17:37:48.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:48.210+0000 7f6d12a00700 1 -- 172.21.15.45:0/2664783296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d0c109d90 msgr2=0x7f6d0c19dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:48.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:48.210+0000 7f6d12a00700 1 --2- 172.21.15.45:0/2664783296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d0c109d90 0x7f6d0c19dd30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc011850 tx=0x7f6cfc011c10 comp rx=0 tx=0).stop 2024-03-28T17:37:48.908 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:48.210+0000 7f6d12a00700 1 -- 172.21.15.45:0/2664783296 shutdown_connections 2024-03-28T17:37:48.908 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:48.210+0000 7f6d12a00700 1 --2- 172.21.15.45:0/2664783296 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f6cf0038f70 0x7f6cf003b430 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:48.908 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:48.210+0000 7f6d12a00700 1 --2- 172.21.15.45:0/2664783296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d0c109d90 0x7f6d0c19dd30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:48.908 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:48.210+0000 7f6d12a00700 1 -- 172.21.15.45:0/2664783296 >> 172.21.15.45:0/2664783296 conn(0x7f6d0c07d200 msgr2=0x7f6d0c107520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:48.908 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:48.210+0000 7f6d12a00700 1 -- 172.21.15.45:0/2664783296 shutdown_connections 2024-03-28T17:37:48.908 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:48.211+0000 7f6d12a00700 1 -- 172.21.15.45:0/2664783296 wait complete. 2024-03-28T17:37:48.908 INFO:teuthology.orchestra.run.smithi045.stdout:Deploying unmanaged mon service... 2024-03-28T17:37:49.520 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:49 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' 2024-03-28T17:37:49.520 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:49 smithi045 ceph-mon[31773]: Added host smithi045 2024-03-28T17:37:49.520 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:49 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:37:51.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-28T17:37:51.021 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.612+0000 7fa644200700 1 Processor -- start 2024-03-28T17:37:51.021 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.612+0000 7fa644200700 1 -- start start 2024-03-28T17:37:51.021 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.612+0000 7fa644200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa63c07de70 0x7fa63c07c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:51.021 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.612+0000 7fa644200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63c07c8f0 con 0x7fa63c07de70 2024-03-28T17:37:51.022 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.613+0000 7fa641800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa63c07de70 0x7fa63c07c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:51.022 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.613+0000 7fa641800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa63c07de70 0x7fa63c07c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39816/0 (socket says 172.21.15.45:39816) 2024-03-28T17:37:51.022 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.613+0000 7fa641800700 1 -- 172.21.15.45:0/154324556 learned_addr learned my addr 172.21.15.45:0/154324556 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:51.024 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.613+0000 7fa641800700 1 -- 172.21.15.45:0/154324556 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa63c07ca70 con 0x7fa63c07de70 2024-03-28T17:37:51.024 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.614+0000 7fa641800700 1 --2- 172.21.15.45:0/154324556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa63c07de70 0x7fa63c07c320 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa630009e80 tx=0x7fa63000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9235089766bc63cf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:51.024 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.614+0000 7fa63be00700 1 -- 172.21.15.45:0/154324556 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa630017070 con 0x7fa63c07de70 2024-03-28T17:37:51.024 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.614+0000 7fa63be00700 1 -- 172.21.15.45:0/154324556 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa63000b8c0 con 0x7fa63c07de70 2024-03-28T17:37:51.024 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.614+0000 7fa63be00700 1 -- 172.21.15.45:0/154324556 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6300045c0 con 0x7fa63c07de70 2024-03-28T17:37:51.024 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.615+0000 7fa644200700 1 -- 172.21.15.45:0/154324556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa63c07de70 msgr2=0x7fa63c07c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:51.024 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.615+0000 7fa644200700 1 --2- 172.21.15.45:0/154324556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa63c07de70 0x7fa63c07c320 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa630009e80 tx=0x7fa63000b330 comp rx=0 tx=0).stop 2024-03-28T17:37:51.024 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.615+0000 7fa644200700 1 -- 172.21.15.45:0/154324556 shutdown_connections 2024-03-28T17:37:51.024 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.615+0000 7fa644200700 1 --2- 172.21.15.45:0/154324556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa63c07de70 0x7fa63c07c320 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:51.025 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.615+0000 7fa644200700 1 -- 172.21.15.45:0/154324556 >> 172.21.15.45:0/154324556 conn(0x7fa63c103110 msgr2=0x7fa63c105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:51.025 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.615+0000 7fa644200700 1 -- 172.21.15.45:0/154324556 shutdown_connections 2024-03-28T17:37:51.025 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.615+0000 7fa644200700 1 -- 172.21.15.45:0/154324556 wait complete. 2024-03-28T17:37:51.025 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.616+0000 7fa644200700 1 Processor -- start 2024-03-28T17:37:51.025 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.616+0000 7fa644200700 1 -- start start 2024-03-28T17:37:51.025 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.617+0000 7fa644200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa63c07de70 0x7fa63c19dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:51.025 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.617+0000 7fa644200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63c10e100 con 0x7fa63c07de70 2024-03-28T17:37:51.025 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.617+0000 7fa641800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa63c07de70 0x7fa63c19dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:51.025 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.617+0000 7fa641800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa63c07de70 0x7fa63c19dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39826/0 (socket says 172.21.15.45:39826) 2024-03-28T17:37:51.028 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.617+0000 7fa641800700 1 -- 172.21.15.45:0/2894324366 learned_addr learned my addr 172.21.15.45:0/2894324366 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:51.028 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.617+0000 7fa641800700 1 -- 172.21.15.45:0/2894324366 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa63c19e200 con 0x7fa63c07de70 2024-03-28T17:37:51.028 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.618+0000 7fa641800700 1 --2- 172.21.15.45:0/2894324366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa63c07de70 0x7fa63c19dcc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa630009530 tx=0x7fa63000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:51.028 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.618+0000 7fa63a000700 1 -- 172.21.15.45:0/2894324366 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa630020020 con 0x7fa63c07de70 2024-03-28T17:37:51.028 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.618+0000 7fa63a000700 1 -- 172.21.15.45:0/2894324366 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa630003b50 con 0x7fa63c07de70 2024-03-28T17:37:51.028 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.618+0000 7fa63a000700 1 -- 172.21.15.45:0/2894324366 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa630017040 con 0x7fa63c07de70 2024-03-28T17:37:51.028 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.618+0000 7fa644200700 1 -- 172.21.15.45:0/2894324366 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa63c19e490 con 0x7fa63c07de70 2024-03-28T17:37:51.029 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.618+0000 7fa644200700 1 -- 172.21.15.45:0/2894324366 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa63c1a1180 con 0x7fa63c07de70 2024-03-28T17:37:51.029 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.620+0000 7fa63a000700 1 -- 172.21.15.45:0/2894324366 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fa630023020 con 0x7fa63c07de70 2024-03-28T17:37:51.029 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.620+0000 7fa644200700 1 -- 172.21.15.45:0/2894324366 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa63c0637c0 con 0x7fa63c07de70 2024-03-28T17:37:51.029 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.620+0000 7fa63a000700 1 --2- 172.21.15.45:0/2894324366 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fa624038f70 0x7fa62403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:51.029 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.620+0000 7fa63a000700 1 -- 172.21.15.45:0/2894324366 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa63000f050 con 0x7fa63c07de70 2024-03-28T17:37:51.029 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.620+0000 7fa640e00700 1 --2- 172.21.15.45:0/2894324366 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fa624038f70 0x7fa62403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:51.029 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.621+0000 7fa640e00700 1 --2- 172.21.15.45:0/2894324366 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fa624038f70 0x7fa62403b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa628006fd0 tx=0x7fa628006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:51.029 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.626+0000 7fa63a000700 1 -- 172.21.15.45:0/2894324366 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa63002a020 con 0x7fa63c07de70 2024-03-28T17:37:51.029 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.871+0000 7fa644200700 1 -- 172.21.15.45:0/2894324366 --> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fa63c104360 con 0x7fa624038f70 2024-03-28T17:37:51.029 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.879+0000 7fa63a000700 1 -- 172.21.15.45:0/2894324366 <== mgr.14120 v2:172.21.15.45:6800/3951447106 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa63c104360 con 0x7fa624038f70 2024-03-28T17:37:51.029 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.882+0000 7fa644200700 1 -- 172.21.15.45:0/2894324366 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fa624038f70 msgr2=0x7fa62403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:51.029 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.882+0000 7fa644200700 1 --2- 172.21.15.45:0/2894324366 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fa624038f70 0x7fa62403b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa628006fd0 tx=0x7fa628006e40 comp rx=0 tx=0).stop 2024-03-28T17:37:51.029 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.882+0000 7fa644200700 1 -- 172.21.15.45:0/2894324366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa63c07de70 msgr2=0x7fa63c19dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:51.029 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.882+0000 7fa644200700 1 --2- 172.21.15.45:0/2894324366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa63c07de70 0x7fa63c19dcc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa630009530 tx=0x7fa63000bd50 comp rx=0 tx=0).stop 2024-03-28T17:37:51.029 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.882+0000 7fa644200700 1 -- 172.21.15.45:0/2894324366 shutdown_connections 2024-03-28T17:37:51.030 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.882+0000 7fa644200700 1 --2- 172.21.15.45:0/2894324366 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fa624038f70 0x7fa62403b430 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:51.030 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.882+0000 7fa644200700 1 --2- 172.21.15.45:0/2894324366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa63c07de70 0x7fa63c19dcc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:51.030 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.882+0000 7fa644200700 1 -- 172.21.15.45:0/2894324366 >> 172.21.15.45:0/2894324366 conn(0x7fa63c103110 msgr2=0x7fa63c103c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:51.030 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.882+0000 7fa644200700 1 -- 172.21.15.45:0/2894324366 shutdown_connections 2024-03-28T17:37:51.030 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:49.882+0000 7fa644200700 1 -- 172.21.15.45:0/2894324366 wait complete. 2024-03-28T17:37:51.030 INFO:teuthology.orchestra.run.smithi045.stdout:Deploying unmanaged mgr service... 2024-03-28T17:37:51.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:50 smithi045 ceph-mon[31773]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:37:51.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:50 smithi045 ceph-mon[31773]: Saving service mon spec with placement count:5 2024-03-28T17:37:51.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:50 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' 2024-03-28T17:37:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:52 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' 2024-03-28T17:37:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:52 smithi045 ceph-mon[31773]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:37:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:52 smithi045 ceph-mon[31773]: Saving service mgr spec with placement count:2 2024-03-28T17:37:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:52 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' 2024-03-28T17:37:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:52 smithi045 ceph-mon[31773]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:37:53.029 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-28T17:37:53.029 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.023+0000 7fac68c00700 1 Processor -- start 2024-03-28T17:37:53.030 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.024+0000 7fac68c00700 1 -- start start 2024-03-28T17:37:53.030 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.024+0000 7fac68c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac640822c0 0x7fac64110ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:53.030 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.024+0000 7fac68c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac641112b0 con 0x7fac640822c0 2024-03-28T17:37:53.030 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.025+0000 7fac61e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac640822c0 0x7fac64110ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:53.030 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.025+0000 7fac61e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac640822c0 0x7fac64110ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39832/0 (socket says 172.21.15.45:39832) 2024-03-28T17:37:53.030 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.025+0000 7fac61e00700 1 -- 172.21.15.45:0/1253715079 learned_addr learned my addr 172.21.15.45:0/1253715079 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:53.030 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.025+0000 7fac61e00700 1 -- 172.21.15.45:0/1253715079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac64111430 con 0x7fac640822c0 2024-03-28T17:37:53.030 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.026+0000 7fac61e00700 1 --2- 172.21.15.45:0/1253715079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac640822c0 0x7fac64110ce0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fac540098a0 tx=0x7fac54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a9788f41ae4cc3c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:53.030 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.026+0000 7fac61400700 1 -- 172.21.15.45:0/1253715079 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac54004160 con 0x7fac640822c0 2024-03-28T17:37:53.030 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.026+0000 7fac61400700 1 -- 172.21.15.45:0/1253715079 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac54004300 con 0x7fac640822c0 2024-03-28T17:37:53.030 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.026+0000 7fac61400700 1 -- 172.21.15.45:0/1253715079 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac54011b20 con 0x7fac640822c0 2024-03-28T17:37:53.030 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.027+0000 7fac68c00700 1 -- 172.21.15.45:0/1253715079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac640822c0 msgr2=0x7fac64110ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:53.030 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.027+0000 7fac68c00700 1 --2- 172.21.15.45:0/1253715079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac640822c0 0x7fac64110ce0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fac540098a0 tx=0x7fac54009bb0 comp rx=0 tx=0).stop 2024-03-28T17:37:53.030 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.027+0000 7fac68c00700 1 -- 172.21.15.45:0/1253715079 shutdown_connections 2024-03-28T17:37:53.031 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.027+0000 7fac68c00700 1 --2- 172.21.15.45:0/1253715079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac640822c0 0x7fac64110ce0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:53.031 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.027+0000 7fac68c00700 1 -- 172.21.15.45:0/1253715079 >> 172.21.15.45:0/1253715079 conn(0x7fac6407d0b0 msgr2=0x7fac6407f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:53.031 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.027+0000 7fac68c00700 1 -- 172.21.15.45:0/1253715079 shutdown_connections 2024-03-28T17:37:53.031 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.027+0000 7fac68c00700 1 -- 172.21.15.45:0/1253715079 wait complete. 2024-03-28T17:37:53.031 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.028+0000 7fac68c00700 1 Processor -- start 2024-03-28T17:37:53.031 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.028+0000 7fac68c00700 1 -- start start 2024-03-28T17:37:53.031 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.029+0000 7fac68c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac640822c0 0x7fac641a2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:53.031 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.029+0000 7fac68c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac64112500 con 0x7fac640822c0 2024-03-28T17:37:53.031 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.029+0000 7fac61e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac640822c0 0x7fac641a2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:53.031 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.029+0000 7fac61e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac640822c0 0x7fac641a2080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39834/0 (socket says 172.21.15.45:39834) 2024-03-28T17:37:53.031 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.029+0000 7fac61e00700 1 -- 172.21.15.45:0/876428104 learned_addr learned my addr 172.21.15.45:0/876428104 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:53.031 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.029+0000 7fac61e00700 1 -- 172.21.15.45:0/876428104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac641a25c0 con 0x7fac640822c0 2024-03-28T17:37:53.031 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.030+0000 7fac61e00700 1 --2- 172.21.15.45:0/876428104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac640822c0 0x7fac641a2080 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fac54011b60 tx=0x7fac54011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:53.031 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.030+0000 7fac5b400700 1 -- 172.21.15.45:0/876428104 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac54003b60 con 0x7fac640822c0 2024-03-28T17:37:53.032 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.030+0000 7fac5b400700 1 -- 172.21.15.45:0/876428104 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac5401a910 con 0x7fac640822c0 2024-03-28T17:37:53.032 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.030+0000 7fac68c00700 1 -- 172.21.15.45:0/876428104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac641a2850 con 0x7fac640822c0 2024-03-28T17:37:53.032 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.030+0000 7fac5b400700 1 -- 172.21.15.45:0/876428104 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac54024930 con 0x7fac640822c0 2024-03-28T17:37:53.032 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.030+0000 7fac68c00700 1 -- 172.21.15.45:0/876428104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac641a2cb0 con 0x7fac640822c0 2024-03-28T17:37:53.032 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.032+0000 7fac68c00700 1 -- 172.21.15.45:0/876428104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac640637c0 con 0x7fac640822c0 2024-03-28T17:37:53.032 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.032+0000 7fac5b400700 1 -- 172.21.15.45:0/876428104 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fac54022020 con 0x7fac640822c0 2024-03-28T17:37:53.032 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.032+0000 7fac5b400700 1 --2- 172.21.15.45:0/876428104 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fac48038f70 0x7fac4803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:53.032 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.032+0000 7fac5b400700 1 -- 172.21.15.45:0/876428104 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fac5404d170 con 0x7fac640822c0 2024-03-28T17:37:53.032 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.033+0000 7fac59400700 1 --2- 172.21.15.45:0/876428104 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fac48038f70 0x7fac4803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:53.032 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.033+0000 7fac59400700 1 --2- 172.21.15.45:0/876428104 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fac48038f70 0x7fac4803b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fac4c006fd0 tx=0x7fac4c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:53.032 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.038+0000 7fac5b400700 1 -- 172.21.15.45:0/876428104 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fac5402d770 con 0x7fac640822c0 2024-03-28T17:37:53.032 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.280+0000 7fac68c00700 1 -- 172.21.15.45:0/876428104 --> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fac6407e320 con 0x7fac48038f70 2024-03-28T17:37:53.032 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.287+0000 7fac5b400700 1 -- 172.21.15.45:0/876428104 <== mgr.14120 v2:172.21.15.45:6800/3951447106 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fac6407e320 con 0x7fac48038f70 2024-03-28T17:37:53.032 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.290+0000 7fac68c00700 1 -- 172.21.15.45:0/876428104 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fac48038f70 msgr2=0x7fac4803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:53.032 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.290+0000 7fac68c00700 1 --2- 172.21.15.45:0/876428104 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fac48038f70 0x7fac4803b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fac4c006fd0 tx=0x7fac4c006e40 comp rx=0 tx=0).stop 2024-03-28T17:37:53.033 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.290+0000 7fac68c00700 1 -- 172.21.15.45:0/876428104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac640822c0 msgr2=0x7fac641a2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:53.033 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.290+0000 7fac68c00700 1 --2- 172.21.15.45:0/876428104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac640822c0 0x7fac641a2080 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fac54011b60 tx=0x7fac54011c40 comp rx=0 tx=0).stop 2024-03-28T17:37:53.033 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.290+0000 7fac68c00700 1 -- 172.21.15.45:0/876428104 shutdown_connections 2024-03-28T17:37:53.033 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.290+0000 7fac68c00700 1 --2- 172.21.15.45:0/876428104 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fac48038f70 0x7fac4803b430 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:53.033 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.290+0000 7fac68c00700 1 --2- 172.21.15.45:0/876428104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac640822c0 0x7fac641a2080 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:53.033 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.290+0000 7fac68c00700 1 -- 172.21.15.45:0/876428104 >> 172.21.15.45:0/876428104 conn(0x7fac6407d0b0 msgr2=0x7fac6407dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:53.033 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.291+0000 7fac68c00700 1 -- 172.21.15.45:0/876428104 shutdown_connections 2024-03-28T17:37:53.033 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:52.291+0000 7fac68c00700 1 -- 172.21.15.45:0/876428104 wait complete. 2024-03-28T17:37:53.846 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:53 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' 2024-03-28T17:37:55.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:54 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/4161197235' entity='client.admin' 2024-03-28T17:37:55.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:54 smithi045 ceph-mon[31773]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:37:55.188 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.590+0000 7feb85600700 1 Processor -- start 2024-03-28T17:37:55.189 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.590+0000 7feb85600700 1 -- start start 2024-03-28T17:37:55.189 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.591+0000 7feb85600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb8007de10 0x7feb8007c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:55.189 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.591+0000 7feb85600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8007c800 con 0x7feb8007de10 2024-03-28T17:37:55.189 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.591+0000 7feb7ea00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb8007de10 0x7feb8007c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:55.189 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.591+0000 7feb7ea00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb8007de10 0x7feb8007c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39838/0 (socket says 172.21.15.45:39838) 2024-03-28T17:37:55.189 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.591+0000 7feb7ea00700 1 -- 172.21.15.45:0/1395717307 learned_addr learned my addr 172.21.15.45:0/1395717307 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:55.189 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.592+0000 7feb7ea00700 1 -- 172.21.15.45:0/1395717307 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8007c980 con 0x7feb8007de10 2024-03-28T17:37:55.189 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.592+0000 7feb7ea00700 1 --2- 172.21.15.45:0/1395717307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb8007de10 0x7feb8007c2c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7feb700098a0 tx=0x7feb70009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6a5907984ccb829 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:55.189 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.592+0000 7feb7d600700 1 -- 172.21.15.45:0/1395717307 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb70004160 con 0x7feb8007de10 2024-03-28T17:37:55.189 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.592+0000 7feb7d600700 1 -- 172.21.15.45:0/1395717307 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feb70004300 con 0x7feb8007de10 2024-03-28T17:37:55.189 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.592+0000 7feb7d600700 1 -- 172.21.15.45:0/1395717307 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb70011b20 con 0x7feb8007de10 2024-03-28T17:37:55.189 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.593+0000 7feb85600700 1 -- 172.21.15.45:0/1395717307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb8007de10 msgr2=0x7feb8007c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:55.189 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.593+0000 7feb85600700 1 --2- 172.21.15.45:0/1395717307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb8007de10 0x7feb8007c2c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7feb700098a0 tx=0x7feb70009bb0 comp rx=0 tx=0).stop 2024-03-28T17:37:55.189 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.593+0000 7feb85600700 1 -- 172.21.15.45:0/1395717307 shutdown_connections 2024-03-28T17:37:55.190 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.593+0000 7feb85600700 1 --2- 172.21.15.45:0/1395717307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb8007de10 0x7feb8007c2c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:55.190 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.593+0000 7feb85600700 1 -- 172.21.15.45:0/1395717307 >> 172.21.15.45:0/1395717307 conn(0x7feb80103110 msgr2=0x7feb80105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:55.190 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.594+0000 7feb85600700 1 -- 172.21.15.45:0/1395717307 shutdown_connections 2024-03-28T17:37:55.190 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.594+0000 7feb85600700 1 -- 172.21.15.45:0/1395717307 wait complete. 2024-03-28T17:37:55.190 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.595+0000 7feb85600700 1 Processor -- start 2024-03-28T17:37:55.190 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.595+0000 7feb85600700 1 -- start start 2024-03-28T17:37:55.190 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.595+0000 7feb85600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb8007de10 0x7feb8019dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:55.190 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.595+0000 7feb85600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8010e150 con 0x7feb8007de10 2024-03-28T17:37:55.190 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.596+0000 7feb7ea00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb8007de10 0x7feb8019dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:55.190 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.596+0000 7feb7ea00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb8007de10 0x7feb8019dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39844/0 (socket says 172.21.15.45:39844) 2024-03-28T17:37:55.190 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.596+0000 7feb7ea00700 1 -- 172.21.15.45:0/4161197235 learned_addr learned my addr 172.21.15.45:0/4161197235 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:55.190 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.596+0000 7feb7ea00700 1 -- 172.21.15.45:0/4161197235 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8019e1c0 con 0x7feb8007de10 2024-03-28T17:37:55.190 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.596+0000 7feb7ea00700 1 --2- 172.21.15.45:0/4161197235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb8007de10 0x7feb8019dc80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7feb70011b60 tx=0x7feb70011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:55.190 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.597+0000 7feb77400700 1 -- 172.21.15.45:0/4161197235 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb70003b60 con 0x7feb8007de10 2024-03-28T17:37:55.191 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.597+0000 7feb77400700 1 -- 172.21.15.45:0/4161197235 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feb7001a910 con 0x7feb8007de10 2024-03-28T17:37:55.191 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.597+0000 7feb85600700 1 -- 172.21.15.45:0/4161197235 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb8019e450 con 0x7feb8007de10 2024-03-28T17:37:55.191 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.597+0000 7feb77400700 1 -- 172.21.15.45:0/4161197235 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb70024930 con 0x7feb8007de10 2024-03-28T17:37:55.191 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.597+0000 7feb85600700 1 -- 172.21.15.45:0/4161197235 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb8019e8b0 con 0x7feb8007de10 2024-03-28T17:37:55.191 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.598+0000 7feb77400700 1 -- 172.21.15.45:0/4161197235 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7feb70022020 con 0x7feb8007de10 2024-03-28T17:37:55.191 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.598+0000 7feb85600700 1 -- 172.21.15.45:0/4161197235 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb800637c0 con 0x7feb8007de10 2024-03-28T17:37:55.191 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.599+0000 7feb77400700 1 --2- 172.21.15.45:0/4161197235 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7feb64038f70 0x7feb6403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:55.191 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.599+0000 7feb77400700 1 -- 172.21.15.45:0/4161197235 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7feb7004d0c0 con 0x7feb8007de10 2024-03-28T17:37:55.191 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.601+0000 7feb7e000700 1 --2- 172.21.15.45:0/4161197235 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7feb64038f70 0x7feb6403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:55.191 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.601+0000 7feb7e000700 1 --2- 172.21.15.45:0/4161197235 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7feb64038f70 0x7feb6403b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7feb6c006fd0 tx=0x7feb6c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:55.191 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.605+0000 7feb77400700 1 -- 172.21.15.45:0/4161197235 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7feb7004ca50 con 0x7feb8007de10 2024-03-28T17:37:55.191 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.845+0000 7feb85600700 1 -- 172.21.15.45:0/4161197235 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7feb801a16f0 con 0x7feb8007de10 2024-03-28T17:37:55.191 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.848+0000 7feb77400700 1 -- 172.21.15.45:0/4161197235 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7feb7000bc70 con 0x7feb8007de10 2024-03-28T17:37:55.191 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.852+0000 7feb85600700 1 -- 172.21.15.45:0/4161197235 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7feb64038f70 msgr2=0x7feb6403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:55.191 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.852+0000 7feb85600700 1 --2- 172.21.15.45:0/4161197235 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7feb64038f70 0x7feb6403b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7feb6c006fd0 tx=0x7feb6c006e40 comp rx=0 tx=0).stop 2024-03-28T17:37:55.192 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.852+0000 7feb85600700 1 -- 172.21.15.45:0/4161197235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb8007de10 msgr2=0x7feb8019dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:55.192 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.852+0000 7feb85600700 1 --2- 172.21.15.45:0/4161197235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb8007de10 0x7feb8019dc80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7feb70011b60 tx=0x7feb70011c40 comp rx=0 tx=0).stop 2024-03-28T17:37:55.192 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.852+0000 7feb85600700 1 -- 172.21.15.45:0/4161197235 shutdown_connections 2024-03-28T17:37:55.192 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.852+0000 7feb85600700 1 --2- 172.21.15.45:0/4161197235 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7feb64038f70 0x7feb6403b430 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:55.192 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.852+0000 7feb85600700 1 --2- 172.21.15.45:0/4161197235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb8007de10 0x7feb8019dc80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:55.192 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.852+0000 7feb85600700 1 -- 172.21.15.45:0/4161197235 >> 172.21.15.45:0/4161197235 conn(0x7feb80103110 msgr2=0x7feb80103be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:55.192 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.853+0000 7feb85600700 1 -- 172.21.15.45:0/4161197235 shutdown_connections 2024-03-28T17:37:55.192 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:53.853+0000 7feb85600700 1 -- 172.21.15.45:0/4161197235 wait complete. 2024-03-28T17:37:56.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:56 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' 2024-03-28T17:37:56.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:56 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/4051395163' entity='client.admin' 2024-03-28T17:37:56.710 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.761+0000 7f2786600700 1 Processor -- start 2024-03-28T17:37:56.710 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.762+0000 7f2786600700 1 -- start start 2024-03-28T17:37:56.710 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.762+0000 7f2786600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2780107b30 0x7f2780107f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:56.710 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.762+0000 7f2786600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2780108520 con 0x7f2780107b30 2024-03-28T17:37:56.710 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.762+0000 7f277f400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2780107b30 0x7f2780107f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:56.711 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.763+0000 7f277f400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2780107b30 0x7f2780107f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39848/0 (socket says 172.21.15.45:39848) 2024-03-28T17:37:56.711 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.763+0000 7f277f400700 1 -- 172.21.15.45:0/250240779 learned_addr learned my addr 172.21.15.45:0/250240779 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:56.711 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.763+0000 7f277f400700 1 -- 172.21.15.45:0/250240779 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2780108ce0 con 0x7f2780107b30 2024-03-28T17:37:56.711 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.763+0000 7f277f400700 1 --2- 172.21.15.45:0/250240779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2780107b30 0x7f2780107f50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f27700098a0 tx=0x7f2770009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcf01e6afcfd6184 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:56.711 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.764+0000 7f277e000700 1 -- 172.21.15.45:0/250240779 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2770004160 con 0x7f2780107b30 2024-03-28T17:37:56.711 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.764+0000 7f277e000700 1 -- 172.21.15.45:0/250240779 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2770004300 con 0x7f2780107b30 2024-03-28T17:37:56.711 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.764+0000 7f277e000700 1 -- 172.21.15.45:0/250240779 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2770011b20 con 0x7f2780107b30 2024-03-28T17:37:56.711 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.765+0000 7f2786600700 1 -- 172.21.15.45:0/250240779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2780107b30 msgr2=0x7f2780107f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:56.711 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.765+0000 7f2786600700 1 --2- 172.21.15.45:0/250240779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2780107b30 0x7f2780107f50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f27700098a0 tx=0x7f2770009bb0 comp rx=0 tx=0).stop 2024-03-28T17:37:56.711 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.765+0000 7f2786600700 1 -- 172.21.15.45:0/250240779 shutdown_connections 2024-03-28T17:37:56.711 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.765+0000 7f2786600700 1 --2- 172.21.15.45:0/250240779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2780107b30 0x7f2780107f50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:56.712 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.765+0000 7f2786600700 1 -- 172.21.15.45:0/250240779 >> 172.21.15.45:0/250240779 conn(0x7f27801030f0 msgr2=0x7f2780105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:56.712 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.765+0000 7f2786600700 1 -- 172.21.15.45:0/250240779 shutdown_connections 2024-03-28T17:37:56.712 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.765+0000 7f2786600700 1 -- 172.21.15.45:0/250240779 wait complete. 2024-03-28T17:37:56.712 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.766+0000 7f2786600700 1 Processor -- start 2024-03-28T17:37:56.712 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.766+0000 7f2786600700 1 -- start start 2024-03-28T17:37:56.712 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.767+0000 7f2786600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2780107b30 0x7f278019dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:56.712 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.767+0000 7f2786600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278010c090 con 0x7f2780107b30 2024-03-28T17:37:56.712 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.767+0000 7f277f400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2780107b30 0x7f278019dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:56.712 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.767+0000 7f277f400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2780107b30 0x7f278019dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39856/0 (socket says 172.21.15.45:39856) 2024-03-28T17:37:56.712 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.767+0000 7f277f400700 1 -- 172.21.15.45:0/4051395163 learned_addr learned my addr 172.21.15.45:0/4051395163 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:56.712 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.767+0000 7f277f400700 1 -- 172.21.15.45:0/4051395163 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278019e1b0 con 0x7f2780107b30 2024-03-28T17:37:56.712 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.768+0000 7f277f400700 1 --2- 172.21.15.45:0/4051395163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2780107b30 0x7f278019dc70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f27700117e0 tx=0x7f2770011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:56.713 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.768+0000 7f2777e00700 1 -- 172.21.15.45:0/4051395163 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2770003bf0 con 0x7f2780107b30 2024-03-28T17:37:56.713 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.768+0000 7f2777e00700 1 -- 172.21.15.45:0/4051395163 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f277001a910 con 0x7f2780107b30 2024-03-28T17:37:56.713 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.768+0000 7f2777e00700 1 -- 172.21.15.45:0/4051395163 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2770024970 con 0x7f2780107b30 2024-03-28T17:37:56.713 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.768+0000 7f2786600700 1 -- 172.21.15.45:0/4051395163 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f278019e440 con 0x7f2780107b30 2024-03-28T17:37:56.713 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.768+0000 7f2786600700 1 -- 172.21.15.45:0/4051395163 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f278019e8a0 con 0x7f2780107b30 2024-03-28T17:37:56.713 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.769+0000 7f2786600700 1 -- 172.21.15.45:0/4051395163 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27800637c0 con 0x7f2780107b30 2024-03-28T17:37:56.713 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.770+0000 7f2777e00700 1 -- 172.21.15.45:0/4051395163 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f2770022020 con 0x7f2780107b30 2024-03-28T17:37:56.713 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.770+0000 7f2777e00700 1 --2- 172.21.15.45:0/4051395163 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f2764038f20 0x7f276403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:56.713 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.770+0000 7f2777e00700 1 -- 172.21.15.45:0/4051395163 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f277004d690 con 0x7f2780107b30 2024-03-28T17:37:56.713 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.770+0000 7f277ea00700 1 --2- 172.21.15.45:0/4051395163 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f2764038f20 0x7f276403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:56.713 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.771+0000 7f277ea00700 1 --2- 172.21.15.45:0/4051395163 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f2764038f20 0x7f276403b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f276c006fd0 tx=0x7f276c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:56.714 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:55.776+0000 7f2777e00700 1 -- 172.21.15.45:0/4051395163 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2770017ca0 con 0x7f2780107b30 2024-03-28T17:37:56.714 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:56.001+0000 7f2786600700 1 -- 172.21.15.45:0/4051395163 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f27801a1310 con 0x7f2780107b30 2024-03-28T17:37:56.714 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:56.005+0000 7f2777e00700 1 -- 172.21.15.45:0/4051395163 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f277004cfe0 con 0x7f2780107b30 2024-03-28T17:37:56.714 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:56.009+0000 7f2786600700 1 -- 172.21.15.45:0/4051395163 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f2764038f20 msgr2=0x7f276403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:56.714 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:56.009+0000 7f2786600700 1 --2- 172.21.15.45:0/4051395163 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f2764038f20 0x7f276403b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f276c006fd0 tx=0x7f276c006e40 comp rx=0 tx=0).stop 2024-03-28T17:37:56.714 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:56.009+0000 7f2786600700 1 -- 172.21.15.45:0/4051395163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2780107b30 msgr2=0x7f278019dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:56.714 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:56.009+0000 7f2786600700 1 --2- 172.21.15.45:0/4051395163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2780107b30 0x7f278019dc70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f27700117e0 tx=0x7f2770011ba0 comp rx=0 tx=0).stop 2024-03-28T17:37:56.714 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:56.009+0000 7f2786600700 1 -- 172.21.15.45:0/4051395163 shutdown_connections 2024-03-28T17:37:56.714 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:56.009+0000 7f2786600700 1 --2- 172.21.15.45:0/4051395163 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f2764038f20 0x7f276403b3e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:56.714 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:56.009+0000 7f2786600700 1 --2- 172.21.15.45:0/4051395163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2780107b30 0x7f278019dc70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:56.714 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:56.009+0000 7f2786600700 1 -- 172.21.15.45:0/4051395163 >> 172.21.15.45:0/4051395163 conn(0x7f27801030f0 msgr2=0x7f2780103c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:56.714 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:56.009+0000 7f2786600700 1 -- 172.21.15.45:0/4051395163 shutdown_connections 2024-03-28T17:37:56.714 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:56.009+0000 7f2786600700 1 -- 172.21.15.45:0/4051395163 wait complete. 2024-03-28T17:37:56.715 INFO:teuthology.orchestra.run.smithi045.stdout:Enabling the dashboard module... 2024-03-28T17:37:57.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:57 smithi045 ceph-mon[31773]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:37:58.474 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:58 smithi045 ceph-mon[31773]: from='mgr.14120 172.21.15.45:0/1742156594' entity='mgr.x' 2024-03-28T17:37:58.474 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:58 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/467368595' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-28T17:37:58.728 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: ignoring --setuser ceph since I am not root 2024-03-28T17:37:58.728 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: ignoring --setgroup ceph since I am not root 2024-03-28T17:37:59.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:59 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/467368595' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-28T17:37:59.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:37:59 smithi045 ceph-mon[31773]: mgrmap e9: x(active, since 25s) 2024-03-28T17:37:59.399 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:59 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:59.171+0000 7ffadd16e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T17:37:59.399 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:37:59 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:37:59.295+0000 7ffadd16e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T17:37:59.926 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.314+0000 7fe1adc00700 1 Processor -- start 2024-03-28T17:37:59.926 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.314+0000 7fe1adc00700 1 -- start start 2024-03-28T17:37:59.926 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.314+0000 7fe1adc00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1a807de30 0x7fe1a807c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:59.926 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.314+0000 7fe1adc00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a807c820 con 0x7fe1a807de30 2024-03-28T17:37:59.931 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.315+0000 7fe1a6a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1a807de30 0x7fe1a807c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:59.931 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.315+0000 7fe1a6a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1a807de30 0x7fe1a807c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43896/0 (socket says 172.21.15.45:43896) 2024-03-28T17:37:59.931 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.315+0000 7fe1a6a00700 1 -- 172.21.15.45:0/3758825818 learned_addr learned my addr 172.21.15.45:0/3758825818 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:59.931 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.315+0000 7fe1a6a00700 1 -- 172.21.15.45:0/3758825818 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a807c9a0 con 0x7fe1a807de30 2024-03-28T17:37:59.931 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.316+0000 7fe1a6a00700 1 --2- 172.21.15.45:0/3758825818 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1a807de30 0x7fe1a807c2e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe1980098a0 tx=0x7fe198009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=932e1666a0cb4fa4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:59.931 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.316+0000 7fe1a5600700 1 -- 172.21.15.45:0/3758825818 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe198004160 con 0x7fe1a807de30 2024-03-28T17:37:59.931 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.316+0000 7fe1a5600700 1 -- 172.21.15.45:0/3758825818 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe198004300 con 0x7fe1a807de30 2024-03-28T17:37:59.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.316+0000 7fe1a5600700 1 -- 172.21.15.45:0/3758825818 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe198011b20 con 0x7fe1a807de30 2024-03-28T17:37:59.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.317+0000 7fe1adc00700 1 -- 172.21.15.45:0/3758825818 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1a807de30 msgr2=0x7fe1a807c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:59.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.317+0000 7fe1adc00700 1 --2- 172.21.15.45:0/3758825818 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1a807de30 0x7fe1a807c2e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe1980098a0 tx=0x7fe198009bb0 comp rx=0 tx=0).stop 2024-03-28T17:37:59.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.317+0000 7fe1adc00700 1 -- 172.21.15.45:0/3758825818 shutdown_connections 2024-03-28T17:37:59.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.317+0000 7fe1adc00700 1 --2- 172.21.15.45:0/3758825818 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1a807de30 0x7fe1a807c2e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:59.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.317+0000 7fe1adc00700 1 -- 172.21.15.45:0/3758825818 >> 172.21.15.45:0/3758825818 conn(0x7fe1a8103110 msgr2=0x7fe1a8105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:59.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.317+0000 7fe1adc00700 1 -- 172.21.15.45:0/3758825818 shutdown_connections 2024-03-28T17:37:59.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.317+0000 7fe1adc00700 1 -- 172.21.15.45:0/3758825818 wait complete. 2024-03-28T17:37:59.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.318+0000 7fe1adc00700 1 Processor -- start 2024-03-28T17:37:59.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.319+0000 7fe1adc00700 1 -- start start 2024-03-28T17:37:59.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.319+0000 7fe1adc00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1a807de30 0x7fe1a819dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:59.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.319+0000 7fe1adc00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a810e130 con 0x7fe1a807de30 2024-03-28T17:37:59.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.319+0000 7fe1a6a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1a807de30 0x7fe1a819dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:59.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.319+0000 7fe1a6a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1a807de30 0x7fe1a819dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43906/0 (socket says 172.21.15.45:43906) 2024-03-28T17:37:59.933 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.319+0000 7fe1a6a00700 1 -- 172.21.15.45:0/467368595 learned_addr learned my addr 172.21.15.45:0/467368595 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:37:59.933 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.320+0000 7fe1a6a00700 1 -- 172.21.15.45:0/467368595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a819e210 con 0x7fe1a807de30 2024-03-28T17:37:59.933 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.320+0000 7fe1a6a00700 1 --2- 172.21.15.45:0/467368595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1a807de30 0x7fe1a819dcd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe198004660 tx=0x7fe198011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:59.933 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.320+0000 7fe19f400700 1 -- 172.21.15.45:0/467368595 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe198003ca0 con 0x7fe1a807de30 2024-03-28T17:37:59.933 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.321+0000 7fe1adc00700 1 -- 172.21.15.45:0/467368595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a819e4a0 con 0x7fe1a807de30 2024-03-28T17:37:59.933 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.321+0000 7fe19f400700 1 -- 172.21.15.45:0/467368595 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe19801a910 con 0x7fe1a807de30 2024-03-28T17:37:59.933 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.321+0000 7fe19f400700 1 -- 172.21.15.45:0/467368595 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe198024970 con 0x7fe1a807de30 2024-03-28T17:37:59.933 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.321+0000 7fe1adc00700 1 -- 172.21.15.45:0/467368595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a819e900 con 0x7fe1a807de30 2024-03-28T17:37:59.933 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.322+0000 7fe19f400700 1 -- 172.21.15.45:0/467368595 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fe198022020 con 0x7fe1a807de30 2024-03-28T17:37:59.933 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.322+0000 7fe1adc00700 1 -- 172.21.15.45:0/467368595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a80637c0 con 0x7fe1a807de30 2024-03-28T17:37:59.933 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.322+0000 7fe19f400700 1 --2- 172.21.15.45:0/467368595 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fe18c039110 0x7fe18c03b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:37:59.933 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.322+0000 7fe19f400700 1 -- 172.21.15.45:0/467368595 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe19804d130 con 0x7fe1a807de30 2024-03-28T17:37:59.933 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.323+0000 7fe1a6000700 1 --2- 172.21.15.45:0/467368595 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fe18c039110 0x7fe18c03b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:37:59.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.323+0000 7fe1a6000700 1 --2- 172.21.15.45:0/467368595 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fe18c039110 0x7fe18c03b5d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe194006fd0 tx=0x7fe194006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:37:59.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.328+0000 7fe19f400700 1 -- 172.21.15.45:0/467368595 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe198019c70 con 0x7fe1a807de30 2024-03-28T17:37:59.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:57.619+0000 7fe1adc00700 1 -- 172.21.15.45:0/467368595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fe1a81a1360 con 0x7fe1a807de30 2024-03-28T17:37:59.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:58.350+0000 7fe19f400700 1 -- 172.21.15.45:0/467368595 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fe19800bc40 con 0x7fe1a807de30 2024-03-28T17:37:59.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:58.355+0000 7fe19f400700 1 -- 172.21.15.45:0/467368595 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fe198024cd0 con 0x7fe1a807de30 2024-03-28T17:37:59.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:58.356+0000 7fe1adc00700 1 -- 172.21.15.45:0/467368595 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fe18c039110 msgr2=0x7fe18c03b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:59.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:58.356+0000 7fe1adc00700 1 --2- 172.21.15.45:0/467368595 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fe18c039110 0x7fe18c03b5d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe194006fd0 tx=0x7fe194006e40 comp rx=0 tx=0).stop 2024-03-28T17:37:59.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:58.356+0000 7fe1adc00700 1 -- 172.21.15.45:0/467368595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1a807de30 msgr2=0x7fe1a819dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:37:59.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:58.356+0000 7fe1adc00700 1 --2- 172.21.15.45:0/467368595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1a807de30 0x7fe1a819dcd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe198004660 tx=0x7fe198011ce0 comp rx=0 tx=0).stop 2024-03-28T17:37:59.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:58.356+0000 7fe1adc00700 1 -- 172.21.15.45:0/467368595 shutdown_connections 2024-03-28T17:37:59.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:58.356+0000 7fe1adc00700 1 --2- 172.21.15.45:0/467368595 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7fe18c039110 0x7fe18c03b5d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:59.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:58.356+0000 7fe1adc00700 1 --2- 172.21.15.45:0/467368595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1a807de30 0x7fe1a819dcd0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:37:59.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:58.356+0000 7fe1adc00700 1 -- 172.21.15.45:0/467368595 >> 172.21.15.45:0/467368595 conn(0x7fe1a8103110 msgr2=0x7fe1a8103c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:37:59.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:58.356+0000 7fe1adc00700 1 -- 172.21.15.45:0/467368595 shutdown_connections 2024-03-28T17:37:59.935 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:37:58.356+0000 7fe1adc00700 1 -- 172.21.15.45:0/467368595 wait complete. 2024-03-28T17:38:00.230 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:38:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:38:00.117+0000 7ffadd16e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T17:38:00.525 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:38:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:38:00.246+0000 7ffadd16e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T17:38:00.526 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:38:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:38:00.439+0000 7ffadd16e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T17:38:00.830 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:38:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:38:00.579+0000 7ffadd16e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T17:38:00.831 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:38:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:38:00.694+0000 7ffadd16e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T17:38:01.149 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:00 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1301707641' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-28T17:38:01.150 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:38:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:38:01.039+0000 7ffadd16e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T17:38:01.415 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:38:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:38:01.188+0000 7ffadd16e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T17:38:01.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:38:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:38:01.304+0000 7ffadd16e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T17:38:01.507 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-28T17:38:01.507 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-28T17:38:01.508 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T17:38:01.508 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-28T17:38:01.508 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-28T17:38:01.508 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-28T17:38:01.508 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.523+0000 7f1a12a00700 1 Processor -- start 2024-03-28T17:38:01.508 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.523+0000 7f1a12a00700 1 -- start start 2024-03-28T17:38:01.508 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.524+0000 7f1a12a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c07de90 0x7f1a0c07c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:01.508 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.524+0000 7f1a12a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0c07c910 con 0x7f1a0c07de90 2024-03-28T17:38:01.508 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.524+0000 7f1a0be00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c07de90 0x7f1a0c07c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:01.508 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.525+0000 7f1a0be00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c07de90 0x7f1a0c07c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43932/0 (socket says 172.21.15.45:43932) 2024-03-28T17:38:01.508 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.525+0000 7f1a0be00700 1 -- 172.21.15.45:0/2320163491 learned_addr learned my addr 172.21.15.45:0/2320163491 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:38:01.508 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.525+0000 7f1a0be00700 1 -- 172.21.15.45:0/2320163491 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a0c07ca90 con 0x7f1a0c07de90 2024-03-28T17:38:01.509 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.526+0000 7f1a0be00700 1 --2- 172.21.15.45:0/2320163491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c07de90 0x7f1a0c07c340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f19fc0098a0 tx=0x7f19fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c42478695c320aa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:01.509 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.526+0000 7f1a0aa00700 1 -- 172.21.15.45:0/2320163491 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19fc004160 con 0x7f1a0c07de90 2024-03-28T17:38:01.509 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.526+0000 7f1a0aa00700 1 -- 172.21.15.45:0/2320163491 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19fc004300 con 0x7f1a0c07de90 2024-03-28T17:38:01.509 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.526+0000 7f1a0aa00700 1 -- 172.21.15.45:0/2320163491 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19fc011b20 con 0x7f1a0c07de90 2024-03-28T17:38:01.509 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.527+0000 7f1a12a00700 1 -- 172.21.15.45:0/2320163491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c07de90 msgr2=0x7f1a0c07c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:01.509 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.527+0000 7f1a12a00700 1 --2- 172.21.15.45:0/2320163491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c07de90 0x7f1a0c07c340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f19fc0098a0 tx=0x7f19fc009bb0 comp rx=0 tx=0).stop 2024-03-28T17:38:01.509 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.527+0000 7f1a12a00700 1 -- 172.21.15.45:0/2320163491 shutdown_connections 2024-03-28T17:38:01.509 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.527+0000 7f1a12a00700 1 --2- 172.21.15.45:0/2320163491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c07de90 0x7f1a0c07c340 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:01.509 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.527+0000 7f1a12a00700 1 -- 172.21.15.45:0/2320163491 >> 172.21.15.45:0/2320163491 conn(0x7f1a0c1030f0 msgr2=0x7f1a0c105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:01.509 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.527+0000 7f1a12a00700 1 -- 172.21.15.45:0/2320163491 shutdown_connections 2024-03-28T17:38:01.509 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.527+0000 7f1a12a00700 1 -- 172.21.15.45:0/2320163491 wait complete. 2024-03-28T17:38:01.509 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.528+0000 7f1a12a00700 1 Processor -- start 2024-03-28T17:38:01.509 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.529+0000 7f1a12a00700 1 -- start start 2024-03-28T17:38:01.510 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.529+0000 7f1a12a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c07de90 0x7f1a0c19dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:01.510 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.529+0000 7f1a12a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0c10e150 con 0x7f1a0c07de90 2024-03-28T17:38:01.510 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.529+0000 7f1a0be00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c07de90 0x7f1a0c19dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:01.510 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.529+0000 7f1a0be00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c07de90 0x7f1a0c19dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43934/0 (socket says 172.21.15.45:43934) 2024-03-28T17:38:01.510 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.529+0000 7f1a0be00700 1 -- 172.21.15.45:0/1301707641 learned_addr learned my addr 172.21.15.45:0/1301707641 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:38:01.510 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.530+0000 7f1a0be00700 1 -- 172.21.15.45:0/1301707641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a0c19e170 con 0x7f1a0c07de90 2024-03-28T17:38:01.510 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.530+0000 7f1a0be00700 1 --2- 172.21.15.45:0/1301707641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c07de90 0x7f1a0c19dc30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f19fc0117e0 tx=0x7f19fc011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:01.510 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.530+0000 7f1a08c00700 1 -- 172.21.15.45:0/1301707641 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19fc003ba0 con 0x7f1a0c07de90 2024-03-28T17:38:01.510 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.530+0000 7f1a08c00700 1 -- 172.21.15.45:0/1301707641 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19fc01a910 con 0x7f1a0c07de90 2024-03-28T17:38:01.510 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.530+0000 7f1a12a00700 1 -- 172.21.15.45:0/1301707641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a0c19e400 con 0x7f1a0c07de90 2024-03-28T17:38:01.510 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.530+0000 7f1a08c00700 1 -- 172.21.15.45:0/1301707641 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19fc0248f0 con 0x7f1a0c07de90 2024-03-28T17:38:01.510 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.531+0000 7f1a12a00700 1 -- 172.21.15.45:0/1301707641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a0c19e860 con 0x7f1a0c07de90 2024-03-28T17:38:01.510 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.532+0000 7f1a08c00700 1 -- 172.21.15.45:0/1301707641 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f19fc022020 con 0x7f1a0c07de90 2024-03-28T17:38:01.511 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.532+0000 7f1a12a00700 1 -- 172.21.15.45:0/1301707641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a0c0637c0 con 0x7f1a0c07de90 2024-03-28T17:38:01.511 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.532+0000 7f1a08c00700 1 --2- 172.21.15.45:0/1301707641 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f19f0038f70 0x7f19f003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:01.511 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.532+0000 7f1a08c00700 1 -- 172.21.15.45:0/1301707641 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f19fc04d6e0 con 0x7f1a0c07de90 2024-03-28T17:38:01.511 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.532+0000 7f1a0b400700 1 -- 172.21.15.45:0/1301707641 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f19f0038f70 msgr2=0x7f19f003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/3951447106 2024-03-28T17:38:01.511 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.533+0000 7f1a0b400700 1 --2- 172.21.15.45:0/1301707641 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f19f0038f70 0x7f19f003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T17:38:01.511 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.538+0000 7f1a08c00700 1 -- 172.21.15.45:0/1301707641 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f19fc00b3e0 con 0x7f1a0c07de90 2024-03-28T17:38:01.511 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.734+0000 7f1a0b400700 1 -- 172.21.15.45:0/1301707641 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f19f0038f70 msgr2=0x7f19f003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/3951447106 2024-03-28T17:38:01.511 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.734+0000 7f1a0b400700 1 --2- 172.21.15.45:0/1301707641 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f19f0038f70 0x7f19f003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T17:38:01.513 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.828+0000 7f1a12a00700 1 -- 172.21.15.45:0/1301707641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f1a0c1a16a0 con 0x7f1a0c07de90 2024-03-28T17:38:01.513 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.829+0000 7f1a08c00700 1 -- 172.21.15.45:0/1301707641 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f19fc0172b0 con 0x7f1a0c07de90 2024-03-28T17:38:01.513 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.833+0000 7f1a12a00700 1 -- 172.21.15.45:0/1301707641 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f19f0038f70 msgr2=0x7f19f003b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T17:38:01.513 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.833+0000 7f1a12a00700 1 --2- 172.21.15.45:0/1301707641 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f19f0038f70 0x7f19f003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:01.513 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.833+0000 7f1a12a00700 1 -- 172.21.15.45:0/1301707641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c07de90 msgr2=0x7f1a0c19dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:01.513 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.833+0000 7f1a12a00700 1 --2- 172.21.15.45:0/1301707641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c07de90 0x7f1a0c19dc30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f19fc0117e0 tx=0x7f19fc011ba0 comp rx=0 tx=0).stop 2024-03-28T17:38:01.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.833+0000 7f1a12a00700 1 -- 172.21.15.45:0/1301707641 shutdown_connections 2024-03-28T17:38:01.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.833+0000 7f1a12a00700 1 --2- 172.21.15.45:0/1301707641 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f19f0038f70 0x7f19f003b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:01.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.833+0000 7f1a12a00700 1 --2- 172.21.15.45:0/1301707641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c07de90 0x7f1a0c19dc30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:01.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.833+0000 7f1a12a00700 1 -- 172.21.15.45:0/1301707641 >> 172.21.15.45:0/1301707641 conn(0x7f1a0c1030f0 msgr2=0x7f1a0c103bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:01.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.833+0000 7f1a12a00700 1 -- 172.21.15.45:0/1301707641 shutdown_connections 2024-03-28T17:38:01.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:00.833+0000 7f1a12a00700 1 -- 172.21.15.45:0/1301707641 wait complete. 2024-03-28T17:38:01.514 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for the mgr to restart... 2024-03-28T17:38:01.514 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mgr epoch 9... 2024-03-28T17:38:01.691 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:38:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:38:01.433+0000 7ffadd16e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T17:38:01.691 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:38:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:38:01.661+0000 7ffadd16e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T17:38:03.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:38:02 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:38:02.985+0000 7ffadd16e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T17:38:03.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:38:03 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:38:03.161+0000 7ffadd16e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T17:38:03.916 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:38:03 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:38:03.588+0000 7ffadd16e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T17:38:03.917 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:38:03 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:38:03.711+0000 7ffadd16e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T17:38:04.232 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:38:03 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:38:03.978+0000 7ffadd16e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T17:38:04.619 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:38:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:38:04.231+0000 7ffadd16e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T17:38:04.915 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:38:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:38:04.618+0000 7ffadd16e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T17:38:05.258 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:38:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:38:04.942+0000 7ffadd16e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T17:38:05.665 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:38:05 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:38:05.256+0000 7ffadd16e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T17:38:06.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:38:06 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:38:06.324+0000 7ffadd16e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T17:38:07.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:38:07 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:38:07.070+0000 7ffadd16e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T17:38:07.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:38:07 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:38:07.204+0000 7ffadd16e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T17:38:07.680 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:38:07 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:38:07.501+0000 7ffadd16e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T17:38:07.680 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:07 smithi045 ceph-mon[31773]: Active manager daemon x restarted 2024-03-28T17:38:07.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:07 smithi045 ceph-mon[31773]: Activating manager daemon x 2024-03-28T17:38:07.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:07 smithi045 ceph-mon[31773]: osdmap e3: 0 total, 0 up, 0 in 2024-03-28T17:38:07.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:07 smithi045 ceph-mon[31773]: mgrmap e10: x(active, starting, since 0.00898606s) 2024-03-28T17:38:07.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:07 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T17:38:07.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:07 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T17:38:07.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:07 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T17:38:07.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:07 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T17:38:07.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:07 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T17:38:07.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:07 smithi045 ceph-mon[31773]: Manager daemon x is now available 2024-03-28T17:38:08.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:38:08.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:38:08.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T17:38:08.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T17:38:08.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:08 smithi045 ceph-mon[31773]: mgrmap e11: x(active, since 1.01209s) 2024-03-28T17:38:09.239 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-28T17:38:09.239 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-28T17:38:09.239 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-28T17:38:09.239 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-28T17:38:09.239 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.097+0000 7f0d40200700 1 Processor -- start 2024-03-28T17:38:09.239 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.098+0000 7f0d40200700 1 -- start start 2024-03-28T17:38:09.239 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.098+0000 7f0d40200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d38108600 0x7f0d3810a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:09.239 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.098+0000 7f0d40200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3807be50 con 0x7f0d38108600 2024-03-28T17:38:09.239 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.098+0000 7f0d3d800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d38108600 0x7f0d3810a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:09.239 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.099+0000 7f0d3d800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d38108600 0x7f0d3810a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43942/0 (socket says 172.21.15.45:43942) 2024-03-28T17:38:09.239 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.099+0000 7f0d3d800700 1 -- 172.21.15.45:0/1444700198 learned_addr learned my addr 172.21.15.45:0/1444700198 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:38:09.240 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.099+0000 7f0d3d800700 1 -- 172.21.15.45:0/1444700198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d3810af30 con 0x7f0d38108600 2024-03-28T17:38:09.240 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.099+0000 7f0d3d800700 1 --2- 172.21.15.45:0/1444700198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d38108600 0x7f0d3810a9f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c0098a0 tx=0x7f0d2c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7afbd043e85ec60d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:09.240 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.100+0000 7f0d37e00700 1 -- 172.21.15.45:0/1444700198 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d2c004160 con 0x7f0d38108600 2024-03-28T17:38:09.240 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.100+0000 7f0d37e00700 1 -- 172.21.15.45:0/1444700198 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d2c004300 con 0x7f0d38108600 2024-03-28T17:38:09.240 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.100+0000 7f0d37e00700 1 -- 172.21.15.45:0/1444700198 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d2c011b20 con 0x7f0d38108600 2024-03-28T17:38:09.240 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.100+0000 7f0d40200700 1 -- 172.21.15.45:0/1444700198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d38108600 msgr2=0x7f0d3810a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:09.240 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.100+0000 7f0d40200700 1 --2- 172.21.15.45:0/1444700198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d38108600 0x7f0d3810a9f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c0098a0 tx=0x7f0d2c009bb0 comp rx=0 tx=0).stop 2024-03-28T17:38:09.240 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.100+0000 7f0d40200700 1 -- 172.21.15.45:0/1444700198 shutdown_connections 2024-03-28T17:38:09.240 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.101+0000 7f0d40200700 1 --2- 172.21.15.45:0/1444700198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d38108600 0x7f0d3810a9f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:09.240 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.101+0000 7f0d40200700 1 -- 172.21.15.45:0/1444700198 >> 172.21.15.45:0/1444700198 conn(0x7f0d38101ff0 msgr2=0x7f0d38104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:09.240 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.101+0000 7f0d40200700 1 -- 172.21.15.45:0/1444700198 shutdown_connections 2024-03-28T17:38:09.240 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.101+0000 7f0d40200700 1 -- 172.21.15.45:0/1444700198 wait complete. 2024-03-28T17:38:09.240 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.102+0000 7f0d40200700 1 Processor -- start 2024-03-28T17:38:09.240 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.102+0000 7f0d40200700 1 -- start start 2024-03-28T17:38:09.241 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.102+0000 7f0d40200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d38108600 0x7f0d38107e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:09.241 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.102+0000 7f0d40200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3810bf40 con 0x7f0d38108600 2024-03-28T17:38:09.241 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.102+0000 7f0d3d800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d38108600 0x7f0d38107e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:09.241 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.102+0000 7f0d3d800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d38108600 0x7f0d38107e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43944/0 (socket says 172.21.15.45:43944) 2024-03-28T17:38:09.241 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.102+0000 7f0d3d800700 1 -- 172.21.15.45:0/2686369900 learned_addr learned my addr 172.21.15.45:0/2686369900 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:38:09.241 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.103+0000 7f0d3d800700 1 -- 172.21.15.45:0/2686369900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d381083c0 con 0x7f0d38108600 2024-03-28T17:38:09.241 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.103+0000 7f0d3d800700 1 --2- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d38108600 0x7f0d38107e80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c011b60 tx=0x7f0d2c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:09.241 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.103+0000 7f0d36000700 1 -- 172.21.15.45:0/2686369900 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d2c003b60 con 0x7f0d38108600 2024-03-28T17:38:09.241 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.103+0000 7f0d36000700 1 -- 172.21.15.45:0/2686369900 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d2c01a910 con 0x7f0d38108600 2024-03-28T17:38:09.241 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.103+0000 7f0d40200700 1 -- 172.21.15.45:0/2686369900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d381065b0 con 0x7f0d38108600 2024-03-28T17:38:09.241 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.103+0000 7f0d36000700 1 -- 172.21.15.45:0/2686369900 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d2c024930 con 0x7f0d38108600 2024-03-28T17:38:09.241 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.104+0000 7f0d40200700 1 -- 172.21.15.45:0/2686369900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d38106a70 con 0x7f0d38108600 2024-03-28T17:38:09.241 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.105+0000 7f0d36000700 1 -- 172.21.15.45:0/2686369900 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f0d2c022020 con 0x7f0d38108600 2024-03-28T17:38:09.241 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.105+0000 7f0d36000700 1 --2- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f0d20039020 0x7f0d2003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:09.241 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.105+0000 7f0d36000700 1 -- 172.21.15.45:0/2686369900 --> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d2003bbf0 con 0x7f0d20039020 2024-03-28T17:38:09.242 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.105+0000 7f0d36000700 1 -- 172.21.15.45:0/2686369900 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0d2c04d130 con 0x7f0d38108600 2024-03-28T17:38:09.242 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.105+0000 7f0d3ce00700 1 -- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f0d20039020 msgr2=0x7f0d2003b4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/3951447106 2024-03-28T17:38:09.242 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.105+0000 7f0d3ce00700 1 --2- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f0d20039020 0x7f0d2003b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T17:38:09.242 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.306+0000 7f0d3ce00700 1 -- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f0d20039020 msgr2=0x7f0d2003b4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/3951447106 2024-03-28T17:38:09.242 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.306+0000 7f0d3ce00700 1 --2- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f0d20039020 0x7f0d2003b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T17:38:09.242 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.707+0000 7f0d3ce00700 1 -- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f0d20039020 msgr2=0x7f0d2003b4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/3951447106 2024-03-28T17:38:09.242 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:02.708+0000 7f0d3ce00700 1 --2- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f0d20039020 0x7f0d2003b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T17:38:09.242 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:03.509+0000 7f0d3ce00700 1 -- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f0d20039020 msgr2=0x7f0d2003b4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/3951447106 2024-03-28T17:38:09.242 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:03.509+0000 7f0d3ce00700 1 --2- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f0d20039020 0x7f0d2003b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T17:38:09.242 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:05.111+0000 7f0d3ce00700 1 -- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f0d20039020 msgr2=0x7f0d2003b4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/3951447106 2024-03-28T17:38:09.242 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:05.111+0000 7f0d3ce00700 1 --2- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f0d20039020 0x7f0d2003b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T17:38:09.242 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:07.510+0000 7f0d36000700 1 -- 172.21.15.45:0/2686369900 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mgrmap(e 10) v1 ==== 45886+0+0 (secure 0 0 0) 0x7f0d2c024ad0 con 0x7f0d38108600 2024-03-28T17:38:09.243 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:07.510+0000 7f0d36000700 1 -- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f0d20039020 msgr2=0x7f0d2003b4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T17:38:09.243 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:07.510+0000 7f0d36000700 1 --2- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f0d20039020 0x7f0d2003b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:09.243 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:08.513+0000 7f0d36000700 1 -- 172.21.15.45:0/2686369900 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7f0d2c04ef80 con 0x7f0d38108600 2024-03-28T17:38:09.243 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:08.514+0000 7f0d36000700 1 --2- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f0d2003cad0 0x7f0d2003eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:09.243 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:08.514+0000 7f0d36000700 1 -- 172.21.15.45:0/2686369900 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d2003bbf0 con 0x7f0d2003cad0 2024-03-28T17:38:09.243 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:08.517+0000 7f0d3ce00700 1 --2- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f0d2003cad0 0x7f0d2003eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:09.243 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:08.518+0000 7f0d3ce00700 1 --2- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f0d2003cad0 0x7f0d2003eec0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0d24003a10 tx=0x7f0d240092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:09.243 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:08.519+0000 7f0d36000700 1 -- 172.21.15.45:0/2686369900 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f0d2003bbf0 con 0x7f0d2003cad0 2024-03-28T17:38:09.243 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:08.528+0000 7f0d40200700 1 -- 172.21.15.45:0/2686369900 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f0d380637c0 con 0x7f0d2003cad0 2024-03-28T17:38:09.243 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:08.528+0000 7f0d36000700 1 -- 172.21.15.45:0/2686369900 <== mgr.14152 v2:172.21.15.45:6800/1246060167 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f0d380637c0 con 0x7f0d2003cad0 2024-03-28T17:38:09.243 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:08.529+0000 7f0d40200700 1 -- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f0d2003cad0 msgr2=0x7f0d2003eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:09.243 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:08.529+0000 7f0d40200700 1 --2- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f0d2003cad0 0x7f0d2003eec0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0d24003a10 tx=0x7f0d240092b0 comp rx=0 tx=0).stop 2024-03-28T17:38:09.243 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:08.529+0000 7f0d40200700 1 -- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d38108600 msgr2=0x7f0d38107e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:09.244 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:08.529+0000 7f0d40200700 1 --2- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d38108600 0x7f0d38107e80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c011b60 tx=0x7f0d2c011c40 comp rx=0 tx=0).stop 2024-03-28T17:38:09.244 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:08.529+0000 7f0d40200700 1 -- 172.21.15.45:0/2686369900 shutdown_connections 2024-03-28T17:38:09.244 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:08.529+0000 7f0d40200700 1 --2- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f0d2003cad0 0x7f0d2003eec0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:09.244 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:08.529+0000 7f0d40200700 1 --2- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d38108600 0x7f0d38107e80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:09.244 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:08.529+0000 7f0d40200700 1 --2- 172.21.15.45:0/2686369900 >> [v2:172.21.15.45:6800/3951447106,v1:172.21.15.45:6801/3951447106] conn(0x7f0d20039020 0x7f0d2003b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:09.244 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:08.529+0000 7f0d40200700 1 -- 172.21.15.45:0/2686369900 >> 172.21.15.45:0/2686369900 conn(0x7f0d38101ff0 msgr2=0x7f0d38102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:09.244 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:08.530+0000 7f0d40200700 1 -- 172.21.15.45:0/2686369900 shutdown_connections 2024-03-28T17:38:09.244 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:08.530+0000 7f0d40200700 1 -- 172.21.15.45:0/2686369900 wait complete. 2024-03-28T17:38:09.244 INFO:teuthology.orchestra.run.smithi045.stdout:mgr epoch 9 is available 2024-03-28T17:38:09.244 INFO:teuthology.orchestra.run.smithi045.stdout:Generating a dashboard self-signed certificate... 2024-03-28T17:38:09.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:09 smithi045 ceph-mon[31773]: [28/Mar/2024:17:38:09] ENGINE Bus STARTING 2024-03-28T17:38:09.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:09 smithi045 ceph-mon[31773]: [28/Mar/2024:17:38:09] ENGINE Serving on http://172.21.15.45:8765 2024-03-28T17:38:09.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:09 smithi045 ceph-mon[31773]: [28/Mar/2024:17:38:09] ENGINE Serving on https://172.21.15.45:7150 2024-03-28T17:38:09.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:09 smithi045 ceph-mon[31773]: [28/Mar/2024:17:38:09] ENGINE Bus STARTED 2024-03-28T17:38:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:10 smithi045 ceph-mon[31773]: mgrmap e12: x(active, since 2s) 2024-03-28T17:38:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:10 smithi045 ceph-mon[31773]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:38:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:10 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:10.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:10 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:11.079 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-28T17:38:11.082 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.817+0000 7f3c5be00700 1 Processor -- start 2024-03-28T17:38:11.083 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.817+0000 7f3c5be00700 1 -- start start 2024-03-28T17:38:11.083 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.818+0000 7f3c5be00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c54109d30 0x7f3c5410a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:11.083 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.818+0000 7f3c5be00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5410a720 con 0x7f3c54109d30 2024-03-28T17:38:11.083 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.818+0000 7f3c59400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c54109d30 0x7f3c5410a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:11.083 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.818+0000 7f3c59400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c54109d30 0x7f3c5410a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52632/0 (socket says 172.21.15.45:52632) 2024-03-28T17:38:11.083 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.818+0000 7f3c59400700 1 -- 172.21.15.45:0/3698210220 learned_addr learned my addr 172.21.15.45:0/3698210220 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:38:11.083 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.819+0000 7f3c59400700 1 -- 172.21.15.45:0/3698210220 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5410aee0 con 0x7f3c54109d30 2024-03-28T17:38:11.083 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.819+0000 7f3c59400700 1 --2- 172.21.15.45:0/3698210220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c54109d30 0x7f3c5410a150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3c480098a0 tx=0x7f3c48009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0405c22c1c401a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:11.083 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.819+0000 7f3c53e00700 1 -- 172.21.15.45:0/3698210220 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c48004160 con 0x7f3c54109d30 2024-03-28T17:38:11.083 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.819+0000 7f3c53e00700 1 -- 172.21.15.45:0/3698210220 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c48004300 con 0x7f3c54109d30 2024-03-28T17:38:11.083 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.820+0000 7f3c53e00700 1 -- 172.21.15.45:0/3698210220 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c48011b20 con 0x7f3c54109d30 2024-03-28T17:38:11.084 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.820+0000 7f3c5be00700 1 -- 172.21.15.45:0/3698210220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c54109d30 msgr2=0x7f3c5410a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:11.084 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.820+0000 7f3c5be00700 1 --2- 172.21.15.45:0/3698210220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c54109d30 0x7f3c5410a150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3c480098a0 tx=0x7f3c48009bb0 comp rx=0 tx=0).stop 2024-03-28T17:38:11.084 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.820+0000 7f3c5be00700 1 -- 172.21.15.45:0/3698210220 shutdown_connections 2024-03-28T17:38:11.084 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.820+0000 7f3c5be00700 1 --2- 172.21.15.45:0/3698210220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c54109d30 0x7f3c5410a150 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:11.084 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.820+0000 7f3c5be00700 1 -- 172.21.15.45:0/3698210220 >> 172.21.15.45:0/3698210220 conn(0x7f3c5407d260 msgr2=0x7f3c5407d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:11.084 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.820+0000 7f3c5be00700 1 -- 172.21.15.45:0/3698210220 shutdown_connections 2024-03-28T17:38:11.084 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.820+0000 7f3c5be00700 1 -- 172.21.15.45:0/3698210220 wait complete. 2024-03-28T17:38:11.084 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.821+0000 7f3c5be00700 1 Processor -- start 2024-03-28T17:38:11.084 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.821+0000 7f3c5be00700 1 -- start start 2024-03-28T17:38:11.084 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.822+0000 7f3c5be00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c5419dec0 0x7f3c5419e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:11.084 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.822+0000 7f3c5be00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5410e290 con 0x7f3c5419dec0 2024-03-28T17:38:11.084 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.822+0000 7f3c59400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c5419dec0 0x7f3c5419e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:11.084 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.822+0000 7f3c59400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c5419dec0 0x7f3c5419e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52634/0 (socket says 172.21.15.45:52634) 2024-03-28T17:38:11.084 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.822+0000 7f3c59400700 1 -- 172.21.15.45:0/223409969 learned_addr learned my addr 172.21.15.45:0/223409969 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:38:11.084 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.822+0000 7f3c59400700 1 -- 172.21.15.45:0/223409969 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5419e820 con 0x7f3c5419dec0 2024-03-28T17:38:11.087 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.823+0000 7f3c59400700 1 --2- 172.21.15.45:0/223409969 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c5419dec0 0x7f3c5419e2e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3c5410ad40 tx=0x7f3c48011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:11.087 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.823+0000 7f3c52000700 1 -- 172.21.15.45:0/223409969 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c480036a0 con 0x7f3c5419dec0 2024-03-28T17:38:11.088 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.823+0000 7f3c52000700 1 -- 172.21.15.45:0/223409969 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c4801a460 con 0x7f3c5419dec0 2024-03-28T17:38:11.088 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.823+0000 7f3c5be00700 1 -- 172.21.15.45:0/223409969 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c5419eab0 con 0x7f3c5419dec0 2024-03-28T17:38:11.088 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.823+0000 7f3c52000700 1 -- 172.21.15.45:0/223409969 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c4801b440 con 0x7f3c5419dec0 2024-03-28T17:38:11.088 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.823+0000 7f3c5be00700 1 -- 172.21.15.45:0/223409969 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c541a1590 con 0x7f3c5419dec0 2024-03-28T17:38:11.088 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.824+0000 7f3c5be00700 1 -- 172.21.15.45:0/223409969 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c540637c0 con 0x7f3c5419dec0 2024-03-28T17:38:11.088 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.824+0000 7f3c52000700 1 -- 172.21.15.45:0/223409969 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f3c48003840 con 0x7f3c5419dec0 2024-03-28T17:38:11.088 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.825+0000 7f3c52000700 1 --2- 172.21.15.45:0/223409969 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3c3c038f70 0x7f3c3c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:11.088 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.825+0000 7f3c52000700 1 -- 172.21.15.45:0/223409969 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3c4804d640 con 0x7f3c5419dec0 2024-03-28T17:38:11.089 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.825+0000 7f3c58a00700 1 --2- 172.21.15.45:0/223409969 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3c3c038f70 0x7f3c3c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:11.089 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.825+0000 7f3c58a00700 1 --2- 172.21.15.45:0/223409969 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3c3c038f70 0x7f3c3c03b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3c40006fd0 tx=0x7f3c40006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:11.089 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:09.830+0000 7f3c52000700 1 -- 172.21.15.45:0/223409969 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3c4802a2b0 con 0x7f3c5419dec0 2024-03-28T17:38:11.089 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:10.055+0000 7f3c5be00700 1 -- 172.21.15.45:0/223409969 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c54107d00 con 0x7f3c3c038f70 2024-03-28T17:38:11.089 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:10.192+0000 7f3c52000700 1 -- 172.21.15.45:0/223409969 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f3c54107d00 con 0x7f3c3c038f70 2024-03-28T17:38:11.089 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:10.194+0000 7f3c5be00700 1 -- 172.21.15.45:0/223409969 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3c3c038f70 msgr2=0x7f3c3c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:11.089 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:10.194+0000 7f3c5be00700 1 --2- 172.21.15.45:0/223409969 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3c3c038f70 0x7f3c3c03b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3c40006fd0 tx=0x7f3c40006e40 comp rx=0 tx=0).stop 2024-03-28T17:38:11.089 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:10.195+0000 7f3c5be00700 1 -- 172.21.15.45:0/223409969 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c5419dec0 msgr2=0x7f3c5419e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:11.089 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:10.195+0000 7f3c5be00700 1 --2- 172.21.15.45:0/223409969 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c5419dec0 0x7f3c5419e2e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3c5410ad40 tx=0x7f3c48011ba0 comp rx=0 tx=0).stop 2024-03-28T17:38:11.090 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:10.195+0000 7f3c5be00700 1 -- 172.21.15.45:0/223409969 shutdown_connections 2024-03-28T17:38:11.090 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:10.195+0000 7f3c5be00700 1 --2- 172.21.15.45:0/223409969 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3c3c038f70 0x7f3c3c03b430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:11.090 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:10.195+0000 7f3c5be00700 1 --2- 172.21.15.45:0/223409969 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c5419dec0 0x7f3c5419e2e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:11.090 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:10.195+0000 7f3c5be00700 1 -- 172.21.15.45:0/223409969 >> 172.21.15.45:0/223409969 conn(0x7f3c5407d260 msgr2=0x7f3c541075b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:11.090 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:10.195+0000 7f3c5be00700 1 -- 172.21.15.45:0/223409969 shutdown_connections 2024-03-28T17:38:11.090 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:10.195+0000 7f3c5be00700 1 -- 172.21.15.45:0/223409969 wait complete. 2024-03-28T17:38:11.090 INFO:teuthology.orchestra.run.smithi045.stdout:Creating initial admin user... 2024-03-28T17:38:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:11 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:12.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$qpMkH6BoXLszFDhMiQFYVeG2uGztYhwzj5V9CzIuzawu41Y5XpRhK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711647492, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-28T17:38:12.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.658+0000 7f513e000700 1 Processor -- start 2024-03-28T17:38:12.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.658+0000 7f513e000700 1 -- start start 2024-03-28T17:38:12.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.659+0000 7f513e000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f513807de90 0x7f513807c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:12.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.659+0000 7f513e000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513807c910 con 0x7f513807de90 2024-03-28T17:38:12.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.659+0000 7f5137400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f513807de90 0x7f513807c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:12.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.659+0000 7f5137400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f513807de90 0x7f513807c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52642/0 (socket says 172.21.15.45:52642) 2024-03-28T17:38:12.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.659+0000 7f5137400700 1 -- 172.21.15.45:0/207017114 learned_addr learned my addr 172.21.15.45:0/207017114 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:38:12.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.660+0000 7f5137400700 1 -- 172.21.15.45:0/207017114 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513807ca90 con 0x7f513807de90 2024-03-28T17:38:12.905 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.660+0000 7f5137400700 1 --2- 172.21.15.45:0/207017114 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f513807de90 0x7f513807c340 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f51280098a0 tx=0x7f5128009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f25a71ed567541e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:12.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.661+0000 7f5136000700 1 -- 172.21.15.45:0/207017114 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5128004160 con 0x7f513807de90 2024-03-28T17:38:12.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.661+0000 7f5136000700 1 -- 172.21.15.45:0/207017114 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5128004300 con 0x7f513807de90 2024-03-28T17:38:12.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.661+0000 7f5136000700 1 -- 172.21.15.45:0/207017114 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5128011b20 con 0x7f513807de90 2024-03-28T17:38:12.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.661+0000 7f513e000700 1 -- 172.21.15.45:0/207017114 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f513807de90 msgr2=0x7f513807c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:12.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.661+0000 7f513e000700 1 --2- 172.21.15.45:0/207017114 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f513807de90 0x7f513807c340 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f51280098a0 tx=0x7f5128009bb0 comp rx=0 tx=0).stop 2024-03-28T17:38:12.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.661+0000 7f513e000700 1 -- 172.21.15.45:0/207017114 shutdown_connections 2024-03-28T17:38:12.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.661+0000 7f513e000700 1 --2- 172.21.15.45:0/207017114 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f513807de90 0x7f513807c340 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:12.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.661+0000 7f513e000700 1 -- 172.21.15.45:0/207017114 >> 172.21.15.45:0/207017114 conn(0x7f5138103110 msgr2=0x7f5138105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:12.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.662+0000 7f513e000700 1 -- 172.21.15.45:0/207017114 shutdown_connections 2024-03-28T17:38:12.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.662+0000 7f513e000700 1 -- 172.21.15.45:0/207017114 wait complete. 2024-03-28T17:38:12.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.663+0000 7f513e000700 1 Processor -- start 2024-03-28T17:38:12.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.663+0000 7f513e000700 1 -- start start 2024-03-28T17:38:12.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.663+0000 7f513e000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f513807de90 0x7f513819dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:12.906 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.663+0000 7f513e000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513810e190 con 0x7f513807de90 2024-03-28T17:38:12.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.664+0000 7f5137400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f513807de90 0x7f513819dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:12.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.664+0000 7f5137400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f513807de90 0x7f513819dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52650/0 (socket says 172.21.15.45:52650) 2024-03-28T17:38:12.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.664+0000 7f5137400700 1 -- 172.21.15.45:0/1267496373 learned_addr learned my addr 172.21.15.45:0/1267496373 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:38:12.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.665+0000 7f5137400700 1 -- 172.21.15.45:0/1267496373 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513819e1e0 con 0x7f513807de90 2024-03-28T17:38:12.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.665+0000 7f5137400700 1 --2- 172.21.15.45:0/1267496373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f513807de90 0x7f513819dca0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f51280116a0 tx=0x7f5128011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:12.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.666+0000 7f512fe00700 1 -- 172.21.15.45:0/1267496373 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5128003a40 con 0x7f513807de90 2024-03-28T17:38:12.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.666+0000 7f512fe00700 1 -- 172.21.15.45:0/1267496373 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5128003be0 con 0x7f513807de90 2024-03-28T17:38:12.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.666+0000 7f512fe00700 1 -- 172.21.15.45:0/1267496373 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f512801a750 con 0x7f513807de90 2024-03-28T17:38:12.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.666+0000 7f513e000700 1 -- 172.21.15.45:0/1267496373 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f513819e470 con 0x7f513807de90 2024-03-28T17:38:12.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.666+0000 7f513e000700 1 -- 172.21.15.45:0/1267496373 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513819e8d0 con 0x7f513807de90 2024-03-28T17:38:12.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.667+0000 7f513e000700 1 -- 172.21.15.45:0/1267496373 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51380637c0 con 0x7f513807de90 2024-03-28T17:38:12.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.668+0000 7f512fe00700 1 -- 172.21.15.45:0/1267496373 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f5128024aa0 con 0x7f513807de90 2024-03-28T17:38:12.907 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.668+0000 7f512fe00700 1 --2- 172.21.15.45:0/1267496373 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f511c038f80 0x7f511c03b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:12.908 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.668+0000 7f512fe00700 1 -- 172.21.15.45:0/1267496373 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f512804d550 con 0x7f513807de90 2024-03-28T17:38:12.908 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.669+0000 7f5136a00700 1 --2- 172.21.15.45:0/1267496373 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f511c038f80 0x7f511c03b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:12.908 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.669+0000 7f5136a00700 1 --2- 172.21.15.45:0/1267496373 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f511c038f80 0x7f511c03b440 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5124006fd0 tx=0x7f5124006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:12.908 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.674+0000 7f512fe00700 1 -- 172.21.15.45:0/1267496373 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5128011e20 con 0x7f513807de90 2024-03-28T17:38:12.908 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:11.921+0000 7f513e000700 1 -- 172.21.15.45:0/1267496373 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f51381a12e0 con 0x7f511c038f80 2024-03-28T17:38:12.908 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:12.191+0000 7f512fe00700 1 -- 172.21.15.45:0/1267496373 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f51381a12e0 con 0x7f511c038f80 2024-03-28T17:38:12.908 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:12.195+0000 7f513e000700 1 -- 172.21.15.45:0/1267496373 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f511c038f80 msgr2=0x7f511c03b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:12.908 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:12.195+0000 7f513e000700 1 --2- 172.21.15.45:0/1267496373 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f511c038f80 0x7f511c03b440 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5124006fd0 tx=0x7f5124006e40 comp rx=0 tx=0).stop 2024-03-28T17:38:12.908 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:12.195+0000 7f513e000700 1 -- 172.21.15.45:0/1267496373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f513807de90 msgr2=0x7f513819dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:12.908 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:12.195+0000 7f513e000700 1 --2- 172.21.15.45:0/1267496373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f513807de90 0x7f513819dca0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f51280116a0 tx=0x7f5128011ba0 comp rx=0 tx=0).stop 2024-03-28T17:38:12.908 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:12.195+0000 7f513e000700 1 -- 172.21.15.45:0/1267496373 shutdown_connections 2024-03-28T17:38:12.908 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:12.195+0000 7f513e000700 1 --2- 172.21.15.45:0/1267496373 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f511c038f80 0x7f511c03b440 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:12.908 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:12.195+0000 7f513e000700 1 --2- 172.21.15.45:0/1267496373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f513807de90 0x7f513819dca0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:12.908 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:12.195+0000 7f513e000700 1 -- 172.21.15.45:0/1267496373 >> 172.21.15.45:0/1267496373 conn(0x7f5138103110 msgr2=0x7f5138103c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:12.909 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:12.196+0000 7f513e000700 1 -- 172.21.15.45:0/1267496373 shutdown_connections 2024-03-28T17:38:12.909 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:12.196+0000 7f513e000700 1 -- 172.21.15.45:0/1267496373 wait complete. 2024-03-28T17:38:12.909 INFO:teuthology.orchestra.run.smithi045.stdout:Fetching dashboard port number... 2024-03-28T17:38:13.503 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:13 smithi045 ceph-mon[31773]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:38:13.503 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:13 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:13.504 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:13 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:14.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:14 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1165818829' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-28T17:38:15.282 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 8443 2024-03-28T17:38:15.282 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.587+0000 7f824ce00700 1 Processor -- start 2024-03-28T17:38:15.282 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.588+0000 7f824ce00700 1 -- start start 2024-03-28T17:38:15.283 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.588+0000 7f824ce00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f824807ddc0 0x7f824807c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:15.283 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.588+0000 7f824ce00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824807c800 con 0x7f824807ddc0 2024-03-28T17:38:15.283 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.588+0000 7f8247400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f824807ddc0 0x7f824807c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:15.283 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.589+0000 7f8247400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f824807ddc0 0x7f824807c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52652/0 (socket says 172.21.15.45:52652) 2024-03-28T17:38:15.283 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.589+0000 7f8247400700 1 -- 172.21.15.45:0/2404027082 learned_addr learned my addr 172.21.15.45:0/2404027082 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:38:15.283 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.589+0000 7f8247400700 1 -- 172.21.15.45:0/2404027082 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f824807c980 con 0x7f824807ddc0 2024-03-28T17:38:15.283 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.589+0000 7f8247400700 1 --2- 172.21.15.45:0/2404027082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f824807ddc0 0x7f824807c2c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f82380098a0 tx=0x7f8238009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9944b328777d6a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:15.283 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.590+0000 7f8246000700 1 -- 172.21.15.45:0/2404027082 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8238004160 con 0x7f824807ddc0 2024-03-28T17:38:15.283 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.590+0000 7f8246000700 1 -- 172.21.15.45:0/2404027082 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8238004300 con 0x7f824807ddc0 2024-03-28T17:38:15.283 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.590+0000 7f8246000700 1 -- 172.21.15.45:0/2404027082 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8238011b20 con 0x7f824807ddc0 2024-03-28T17:38:15.283 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.590+0000 7f824ce00700 1 -- 172.21.15.45:0/2404027082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f824807ddc0 msgr2=0x7f824807c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:15.283 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.590+0000 7f824ce00700 1 --2- 172.21.15.45:0/2404027082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f824807ddc0 0x7f824807c2c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f82380098a0 tx=0x7f8238009bb0 comp rx=0 tx=0).stop 2024-03-28T17:38:15.283 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.591+0000 7f824ce00700 1 -- 172.21.15.45:0/2404027082 shutdown_connections 2024-03-28T17:38:15.283 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.591+0000 7f824ce00700 1 --2- 172.21.15.45:0/2404027082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f824807ddc0 0x7f824807c2c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:15.284 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.591+0000 7f824ce00700 1 -- 172.21.15.45:0/2404027082 >> 172.21.15.45:0/2404027082 conn(0x7f8248102f50 msgr2=0x7f8248105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:15.284 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.591+0000 7f824ce00700 1 -- 172.21.15.45:0/2404027082 shutdown_connections 2024-03-28T17:38:15.284 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.591+0000 7f824ce00700 1 -- 172.21.15.45:0/2404027082 wait complete. 2024-03-28T17:38:15.284 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.592+0000 7f824ce00700 1 Processor -- start 2024-03-28T17:38:15.284 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.592+0000 7f824ce00700 1 -- start start 2024-03-28T17:38:15.284 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.592+0000 7f824ce00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f824807ddc0 0x7f824819da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:15.284 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.592+0000 7f824ce00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824810df10 con 0x7f824807ddc0 2024-03-28T17:38:15.284 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.593+0000 7f8247400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f824807ddc0 0x7f824819da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:15.284 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.593+0000 7f8247400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f824807ddc0 0x7f824819da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52668/0 (socket says 172.21.15.45:52668) 2024-03-28T17:38:15.284 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.593+0000 7f8247400700 1 -- 172.21.15.45:0/1165818829 learned_addr learned my addr 172.21.15.45:0/1165818829 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:38:15.284 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.593+0000 7f8247400700 1 -- 172.21.15.45:0/1165818829 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f824819df80 con 0x7f824807ddc0 2024-03-28T17:38:15.284 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.593+0000 7f8247400700 1 --2- 172.21.15.45:0/1165818829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f824807ddc0 0x7f824819da40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f82380117e0 tx=0x7f8238011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:15.284 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.594+0000 7f823fe00700 1 -- 172.21.15.45:0/1165818829 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8238003ba0 con 0x7f824807ddc0 2024-03-28T17:38:15.284 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.594+0000 7f823fe00700 1 -- 172.21.15.45:0/1165818829 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f823801a910 con 0x7f824807ddc0 2024-03-28T17:38:15.285 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.594+0000 7f823fe00700 1 -- 172.21.15.45:0/1165818829 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f82380248f0 con 0x7f824807ddc0 2024-03-28T17:38:15.285 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.594+0000 7f824ce00700 1 -- 172.21.15.45:0/1165818829 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f824819e210 con 0x7f824807ddc0 2024-03-28T17:38:15.285 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.594+0000 7f824ce00700 1 -- 172.21.15.45:0/1165818829 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f824819e670 con 0x7f824807ddc0 2024-03-28T17:38:15.285 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.595+0000 7f824ce00700 1 -- 172.21.15.45:0/1165818829 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82480637c0 con 0x7f824807ddc0 2024-03-28T17:38:15.285 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.595+0000 7f823fe00700 1 -- 172.21.15.45:0/1165818829 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f8238022020 con 0x7f824807ddc0 2024-03-28T17:38:15.285 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.596+0000 7f823fe00700 1 --2- 172.21.15.45:0/1165818829 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f822c038fd0 0x7f822c03b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:15.285 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.596+0000 7f823fe00700 1 -- 172.21.15.45:0/1165818829 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f823804d3c0 con 0x7f824807ddc0 2024-03-28T17:38:15.285 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.596+0000 7f8246a00700 1 --2- 172.21.15.45:0/1165818829 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f822c038fd0 0x7f822c03b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:15.285 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.597+0000 7f8246a00700 1 --2- 172.21.15.45:0/1165818829 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f822c038fd0 0x7f822c03b490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8234006fd0 tx=0x7f8234006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:15.285 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.601+0000 7f823fe00700 1 -- 172.21.15.45:0/1165818829 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f823801ab70 con 0x7f824807ddc0 2024-03-28T17:38:15.285 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.843+0000 7f824ce00700 1 -- 172.21.15.45:0/1165818829 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f82481a1100 con 0x7f824807ddc0 2024-03-28T17:38:15.285 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.844+0000 7f823fe00700 1 -- 172.21.15.45:0/1165818829 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f823802d3f0 con 0x7f824807ddc0 2024-03-28T17:38:15.285 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.847+0000 7f824ce00700 1 -- 172.21.15.45:0/1165818829 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f822c038fd0 msgr2=0x7f822c03b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:15.285 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.847+0000 7f824ce00700 1 --2- 172.21.15.45:0/1165818829 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f822c038fd0 0x7f822c03b490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8234006fd0 tx=0x7f8234006e40 comp rx=0 tx=0).stop 2024-03-28T17:38:15.286 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.847+0000 7f824ce00700 1 -- 172.21.15.45:0/1165818829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f824807ddc0 msgr2=0x7f824819da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:15.286 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.847+0000 7f824ce00700 1 --2- 172.21.15.45:0/1165818829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f824807ddc0 0x7f824819da40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f82380117e0 tx=0x7f8238011ba0 comp rx=0 tx=0).stop 2024-03-28T17:38:15.286 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.847+0000 7f824ce00700 1 -- 172.21.15.45:0/1165818829 shutdown_connections 2024-03-28T17:38:15.286 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.847+0000 7f824ce00700 1 --2- 172.21.15.45:0/1165818829 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f822c038fd0 0x7f822c03b490 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:15.286 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.847+0000 7f824ce00700 1 --2- 172.21.15.45:0/1165818829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f824807ddc0 0x7f824819da40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:15.286 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.847+0000 7f824ce00700 1 -- 172.21.15.45:0/1165818829 >> 172.21.15.45:0/1165818829 conn(0x7f8248102f50 msgr2=0x7f8248103a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:15.286 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.848+0000 7f824ce00700 1 -- 172.21.15.45:0/1165818829 shutdown_connections 2024-03-28T17:38:15.286 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:13.848+0000 7f824ce00700 1 -- 172.21.15.45:0/1165818829 wait complete. 2024-03-28T17:38:15.312 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T17:38:15.312 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout disabled 2024-03-28T17:38:15.335 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T17:38:15.336 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-03-28T17:38:15.336 INFO:teuthology.orchestra.run.smithi045.stdout:firewalld.service is not enabled 2024-03-28T17:38:15.336 INFO:teuthology.orchestra.run.smithi045.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-28T17:38:15.343 INFO:teuthology.orchestra.run.smithi045.stdout:Ceph Dashboard is now available at: 2024-03-28T17:38:15.343 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:38:15.343 INFO:teuthology.orchestra.run.smithi045.stdout: URL: https://smithi045.front.sepia.ceph.com:8443/ 2024-03-28T17:38:15.343 INFO:teuthology.orchestra.run.smithi045.stdout: User: admin 2024-03-28T17:38:15.343 INFO:teuthology.orchestra.run.smithi045.stdout: Password: 9d9tt9wgfg 2024-03-28T17:38:15.343 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:38:15.344 INFO:teuthology.orchestra.run.smithi045.stdout:Saving cluster configuration to /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config directory 2024-03-28T17:38:15.346 INFO:teuthology.orchestra.run.smithi045.stdout:Enabling autotune for osd_memory_target 2024-03-28T17:38:15.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:15 smithi045 ceph-mon[31773]: mgrmap e13: x(active, since 6s) 2024-03-28T17:38:17.134 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.120+0000 7fe325c00700 1 Processor -- start 2024-03-28T17:38:17.134 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.120+0000 7fe325c00700 1 -- start start 2024-03-28T17:38:17.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.121+0000 7fe325c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe320109d90 0x7fe32010a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:17.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.121+0000 7fe325c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32010a780 con 0x7fe320109d90 2024-03-28T17:38:17.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.121+0000 7fe31ea00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe320109d90 0x7fe32010a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:17.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.122+0000 7fe31ea00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe320109d90 0x7fe32010a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59918/0 (socket says 172.21.15.45:59918) 2024-03-28T17:38:17.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.122+0000 7fe31ea00700 1 -- 172.21.15.45:0/991645155 learned_addr learned my addr 172.21.15.45:0/991645155 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:38:17.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.122+0000 7fe31ea00700 1 -- 172.21.15.45:0/991645155 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32010aef0 con 0x7fe320109d90 2024-03-28T17:38:17.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.123+0000 7fe31ea00700 1 --2- 172.21.15.45:0/991645155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe320109d90 0x7fe32010a1b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe3100098a0 tx=0x7fe310009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f0c4ca6a4ff10d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:17.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.123+0000 7fe31d600700 1 -- 172.21.15.45:0/991645155 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe310004160 con 0x7fe320109d90 2024-03-28T17:38:17.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.123+0000 7fe31d600700 1 -- 172.21.15.45:0/991645155 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe310004300 con 0x7fe320109d90 2024-03-28T17:38:17.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.123+0000 7fe31d600700 1 -- 172.21.15.45:0/991645155 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe310011b20 con 0x7fe320109d90 2024-03-28T17:38:17.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.123+0000 7fe325c00700 1 -- 172.21.15.45:0/991645155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe320109d90 msgr2=0x7fe32010a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:17.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.124+0000 7fe325c00700 1 --2- 172.21.15.45:0/991645155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe320109d90 0x7fe32010a1b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe3100098a0 tx=0x7fe310009bb0 comp rx=0 tx=0).stop 2024-03-28T17:38:17.135 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.124+0000 7fe325c00700 1 -- 172.21.15.45:0/991645155 shutdown_connections 2024-03-28T17:38:17.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.124+0000 7fe325c00700 1 --2- 172.21.15.45:0/991645155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe320109d90 0x7fe32010a1b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:17.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.124+0000 7fe325c00700 1 -- 172.21.15.45:0/991645155 >> 172.21.15.45:0/991645155 conn(0x7fe32007d200 msgr2=0x7fe32007d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:17.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.124+0000 7fe325c00700 1 -- 172.21.15.45:0/991645155 shutdown_connections 2024-03-28T17:38:17.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.124+0000 7fe325c00700 1 -- 172.21.15.45:0/991645155 wait complete. 2024-03-28T17:38:17.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.125+0000 7fe325c00700 1 Processor -- start 2024-03-28T17:38:17.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.125+0000 7fe325c00700 1 -- start start 2024-03-28T17:38:17.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.125+0000 7fe325c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe320109d90 0x7fe32019dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:17.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.125+0000 7fe325c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32010e2a0 con 0x7fe320109d90 2024-03-28T17:38:17.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.126+0000 7fe31ea00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe320109d90 0x7fe32019dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:17.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.126+0000 7fe31ea00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe320109d90 0x7fe32019dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59922/0 (socket says 172.21.15.45:59922) 2024-03-28T17:38:17.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.126+0000 7fe31ea00700 1 -- 172.21.15.45:0/2868411145 learned_addr learned my addr 172.21.15.45:0/2868411145 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:38:17.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.126+0000 7fe31ea00700 1 -- 172.21.15.45:0/2868411145 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32019e1e0 con 0x7fe320109d90 2024-03-28T17:38:17.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.126+0000 7fe31ea00700 1 --2- 172.21.15.45:0/2868411145 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe320109d90 0x7fe32019dca0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe310000c00 tx=0x7fe310011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:17.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.127+0000 7fe317400700 1 -- 172.21.15.45:0/2868411145 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe310003b60 con 0x7fe320109d90 2024-03-28T17:38:17.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.127+0000 7fe317400700 1 -- 172.21.15.45:0/2868411145 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe31001a910 con 0x7fe320109d90 2024-03-28T17:38:17.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.127+0000 7fe317400700 1 -- 172.21.15.45:0/2868411145 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe310024930 con 0x7fe320109d90 2024-03-28T17:38:17.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.127+0000 7fe325c00700 1 -- 172.21.15.45:0/2868411145 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32019e470 con 0x7fe320109d90 2024-03-28T17:38:17.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.127+0000 7fe325c00700 1 -- 172.21.15.45:0/2868411145 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe32019e8d0 con 0x7fe320109d90 2024-03-28T17:38:17.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.128+0000 7fe325c00700 1 -- 172.21.15.45:0/2868411145 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3200637c0 con 0x7fe320109d90 2024-03-28T17:38:17.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.128+0000 7fe317400700 1 -- 172.21.15.45:0/2868411145 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fe310022020 con 0x7fe320109d90 2024-03-28T17:38:17.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.128+0000 7fe317400700 1 --2- 172.21.15.45:0/2868411145 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe304039070 0x7fe30403b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:17.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.129+0000 7fe317400700 1 -- 172.21.15.45:0/2868411145 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe31004d2b0 con 0x7fe320109d90 2024-03-28T17:38:17.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.129+0000 7fe31e000700 1 --2- 172.21.15.45:0/2868411145 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe304039070 0x7fe30403b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:17.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.130+0000 7fe31e000700 1 --2- 172.21.15.45:0/2868411145 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe304039070 0x7fe30403b530 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe30c006fd0 tx=0x7fe30c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:17.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.134+0000 7fe317400700 1 -- 172.21.15.45:0/2868411145 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe31001ab70 con 0x7fe320109d90 2024-03-28T17:38:17.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.360+0000 7fe325c00700 1 -- 172.21.15.45:0/2868411145 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fe3201a1340 con 0x7fe320109d90 2024-03-28T17:38:17.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.361+0000 7fe317400700 1 -- 172.21.15.45:0/2868411145 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fe31002d3f0 con 0x7fe320109d90 2024-03-28T17:38:17.138 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.364+0000 7fe325c00700 1 -- 172.21.15.45:0/2868411145 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe304039070 msgr2=0x7fe30403b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:17.138 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.364+0000 7fe325c00700 1 --2- 172.21.15.45:0/2868411145 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe304039070 0x7fe30403b530 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe30c006fd0 tx=0x7fe30c006e40 comp rx=0 tx=0).stop 2024-03-28T17:38:17.138 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.364+0000 7fe325c00700 1 -- 172.21.15.45:0/2868411145 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe320109d90 msgr2=0x7fe32019dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:17.138 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.364+0000 7fe325c00700 1 --2- 172.21.15.45:0/2868411145 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe320109d90 0x7fe32019dca0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe310000c00 tx=0x7fe310011c10 comp rx=0 tx=0).stop 2024-03-28T17:38:17.138 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.364+0000 7fe325c00700 1 -- 172.21.15.45:0/2868411145 shutdown_connections 2024-03-28T17:38:17.138 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.364+0000 7fe325c00700 1 --2- 172.21.15.45:0/2868411145 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe304039070 0x7fe30403b530 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:17.138 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.364+0000 7fe325c00700 1 --2- 172.21.15.45:0/2868411145 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe320109d90 0x7fe32019dca0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:17.138 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.364+0000 7fe325c00700 1 -- 172.21.15.45:0/2868411145 >> 172.21.15.45:0/2868411145 conn(0x7fe32007d200 msgr2=0x7fe3201074b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:17.138 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.364+0000 7fe325c00700 1 -- 172.21.15.45:0/2868411145 shutdown_connections 2024-03-28T17:38:17.138 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:16.365+0000 7fe325c00700 1 -- 172.21.15.45:0/2868411145 wait complete. 2024-03-28T17:38:18.840 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.760+0000 7f8482200700 1 Processor -- start 2024-03-28T17:38:18.840 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.760+0000 7f8482200700 1 -- start start 2024-03-28T17:38:18.840 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.761+0000 7f8482200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f847c108680 0x7f847c10aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:18.840 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.761+0000 7f8482200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c07bdd0 con 0x7f847c108680 2024-03-28T17:38:18.840 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.761+0000 7f847b400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f847c108680 0x7f847c10aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:18.840 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.761+0000 7f847b400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f847c108680 0x7f847c10aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59936/0 (socket says 172.21.15.45:59936) 2024-03-28T17:38:18.840 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.761+0000 7f847b400700 1 -- 172.21.15.45:0/3648222110 learned_addr learned my addr 172.21.15.45:0/3648222110 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:38:18.840 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.762+0000 7f847b400700 1 -- 172.21.15.45:0/3648222110 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c10afb0 con 0x7f847c108680 2024-03-28T17:38:18.840 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.762+0000 7f847b400700 1 --2- 172.21.15.45:0/3648222110 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f847c108680 0x7f847c10aa70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f846c0098a0 tx=0x7f846c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55a45da6dd70e5b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:18.840 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.763+0000 7f847a000700 1 -- 172.21.15.45:0/3648222110 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f846c004160 con 0x7f847c108680 2024-03-28T17:38:18.840 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.763+0000 7f847a000700 1 -- 172.21.15.45:0/3648222110 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f846c004300 con 0x7f847c108680 2024-03-28T17:38:18.840 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.763+0000 7f847a000700 1 -- 172.21.15.45:0/3648222110 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f846c011b20 con 0x7f847c108680 2024-03-28T17:38:18.841 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.764+0000 7f8482200700 1 -- 172.21.15.45:0/3648222110 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f847c108680 msgr2=0x7f847c10aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:18.841 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.764+0000 7f8482200700 1 --2- 172.21.15.45:0/3648222110 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f847c108680 0x7f847c10aa70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f846c0098a0 tx=0x7f846c009bb0 comp rx=0 tx=0).stop 2024-03-28T17:38:18.841 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.764+0000 7f8482200700 1 -- 172.21.15.45:0/3648222110 shutdown_connections 2024-03-28T17:38:18.841 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.764+0000 7f8482200700 1 --2- 172.21.15.45:0/3648222110 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f847c108680 0x7f847c10aa70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:18.841 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.764+0000 7f8482200700 1 -- 172.21.15.45:0/3648222110 >> 172.21.15.45:0/3648222110 conn(0x7f847c101ff0 msgr2=0x7f847c104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:18.841 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.764+0000 7f8482200700 1 -- 172.21.15.45:0/3648222110 shutdown_connections 2024-03-28T17:38:18.841 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.764+0000 7f8482200700 1 -- 172.21.15.45:0/3648222110 wait complete. 2024-03-28T17:38:18.841 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.765+0000 7f8482200700 1 Processor -- start 2024-03-28T17:38:18.841 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.765+0000 7f8482200700 1 -- start start 2024-03-28T17:38:18.841 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.766+0000 7f8482200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f847c108680 0x7f847c19dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:18.841 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.766+0000 7f8482200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c10bf50 con 0x7f847c108680 2024-03-28T17:38:18.841 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.766+0000 7f847b400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f847c108680 0x7f847c19dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:18.841 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.766+0000 7f847b400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f847c108680 0x7f847c19dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59938/0 (socket says 172.21.15.45:59938) 2024-03-28T17:38:18.841 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.766+0000 7f847b400700 1 -- 172.21.15.45:0/3337050887 learned_addr learned my addr 172.21.15.45:0/3337050887 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:38:18.842 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.766+0000 7f847b400700 1 -- 172.21.15.45:0/3337050887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c19e260 con 0x7f847c108680 2024-03-28T17:38:18.842 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.767+0000 7f847b400700 1 --2- 172.21.15.45:0/3337050887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f847c108680 0x7f847c19dd20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f846c011b60 tx=0x7f846c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:18.842 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.767+0000 7f8473e00700 1 -- 172.21.15.45:0/3337050887 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f846c003b60 con 0x7f847c108680 2024-03-28T17:38:18.842 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.767+0000 7f8473e00700 1 -- 172.21.15.45:0/3337050887 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f846c01a910 con 0x7f847c108680 2024-03-28T17:38:18.842 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.767+0000 7f8473e00700 1 -- 172.21.15.45:0/3337050887 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f846c024930 con 0x7f847c108680 2024-03-28T17:38:18.842 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.767+0000 7f8482200700 1 -- 172.21.15.45:0/3337050887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847c19e4f0 con 0x7f847c108680 2024-03-28T17:38:18.842 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.767+0000 7f8482200700 1 -- 172.21.15.45:0/3337050887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f847c19e950 con 0x7f847c108680 2024-03-28T17:38:18.842 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.768+0000 7f8473e00700 1 -- 172.21.15.45:0/3337050887 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f846c022020 con 0x7f847c108680 2024-03-28T17:38:18.842 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.768+0000 7f8482200700 1 -- 172.21.15.45:0/3337050887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f847c0637c0 con 0x7f847c108680 2024-03-28T17:38:18.842 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.769+0000 7f8473e00700 1 --2- 172.21.15.45:0/3337050887 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8460039070 0x7f846003b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:18.842 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.769+0000 7f8473e00700 1 -- 172.21.15.45:0/3337050887 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f846c04d2c0 con 0x7f847c108680 2024-03-28T17:38:18.842 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.769+0000 7f847aa00700 1 --2- 172.21.15.45:0/3337050887 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8460039070 0x7f846003b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:18.843 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.770+0000 7f847aa00700 1 --2- 172.21.15.45:0/3337050887 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8460039070 0x7f846003b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8468006fd0 tx=0x7f8468006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:18.843 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:17.774+0000 7f8473e00700 1 -- 172.21.15.45:0/3337050887 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f846c00b3e0 con 0x7f847c108680 2024-03-28T17:38:18.843 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:18.131+0000 7f8482200700 1 -- 172.21.15.45:0/3337050887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f847c1a1360 con 0x7f847c108680 2024-03-28T17:38:18.843 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:18.136+0000 7f8473e00700 1 -- 172.21.15.45:0/3337050887 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f846c02dde0 con 0x7f847c108680 2024-03-28T17:38:18.843 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:18.139+0000 7f8482200700 1 -- 172.21.15.45:0/3337050887 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8460039070 msgr2=0x7f846003b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:18.843 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:18.139+0000 7f8482200700 1 --2- 172.21.15.45:0/3337050887 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8460039070 0x7f846003b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8468006fd0 tx=0x7f8468006e40 comp rx=0 tx=0).stop 2024-03-28T17:38:18.843 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:18.139+0000 7f8482200700 1 -- 172.21.15.45:0/3337050887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f847c108680 msgr2=0x7f847c19dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:18.843 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:18.139+0000 7f8482200700 1 --2- 172.21.15.45:0/3337050887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f847c108680 0x7f847c19dd20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f846c011b60 tx=0x7f846c011c40 comp rx=0 tx=0).stop 2024-03-28T17:38:18.843 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:18.139+0000 7f8482200700 1 -- 172.21.15.45:0/3337050887 shutdown_connections 2024-03-28T17:38:18.843 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:18.139+0000 7f8482200700 1 --2- 172.21.15.45:0/3337050887 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8460039070 0x7f846003b530 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:18.843 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:18.139+0000 7f8482200700 1 --2- 172.21.15.45:0/3337050887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f847c108680 0x7f847c19dd20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:18.843 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:18.139+0000 7f8482200700 1 -- 172.21.15.45:0/3337050887 >> 172.21.15.45:0/3337050887 conn(0x7f847c101ff0 msgr2=0x7f847c102a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:18.843 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:18.140+0000 7f8482200700 1 -- 172.21.15.45:0/3337050887 shutdown_connections 2024-03-28T17:38:18.843 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-28T17:38:18.140+0000 7f8482200700 1 -- 172.21.15.45:0/3337050887 wait complete. 2024-03-28T17:38:18.843 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-28T17:38:18.844 INFO:teuthology.orchestra.run.smithi045.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-28T17:38:18.844 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:38:18.844 INFO:teuthology.orchestra.run.smithi045.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-28T17:38:18.844 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:38:18.844 INFO:teuthology.orchestra.run.smithi045.stdout:Or, if you are only running a single cluster on this host: 2024-03-28T17:38:18.844 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:38:18.844 INFO:teuthology.orchestra.run.smithi045.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-28T17:38:18.844 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:38:18.844 INFO:teuthology.orchestra.run.smithi045.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-28T17:38:18.844 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:38:18.844 INFO:teuthology.orchestra.run.smithi045.stdout: ceph telemetry on 2024-03-28T17:38:18.844 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:38:18.844 INFO:teuthology.orchestra.run.smithi045.stdout:For more information see: 2024-03-28T17:38:18.844 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:38:18.845 INFO:teuthology.orchestra.run.smithi045.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-28T17:38:18.845 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:38:18.845 INFO:teuthology.orchestra.run.smithi045.stdout:Bootstrap complete. 2024-03-28T17:38:18.943 INFO:tasks.cephadm:Fetching config... 2024-03-28T17:38:18.944 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-28T17:38:18.944 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-28T17:38:18.991 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-28T17:38:18.991 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-28T17:38:18.991 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-28T17:38:19.078 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-28T17:38:19.078 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-28T17:38:19.078 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-28T17:38:19.202 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-28T17:38:19.202 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-28T17:38:19.202 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-28T17:38:19.291 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-28T17:38:19.291 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFbuyxvgdUhfq0nwnk1xUkPjL73wGaJmY8CRinLud8x75kIvoa/j+BE4X/1fsKx0oo1G8793Y+h7EP31Qxmg260wB5tg3SVbCZOjTbJ7oC/Dxct0scCzxLVdJ5xG+G8dz+CR5FBuj/0cBZhCV0MqvOVBQBcVzkH7MWkKkmdjkL7nxmi29H5p8QouGgSCEEllBv1MsdmXV5fy7YT3SlpZUKdELPaST+EXTJjCinXw1FwKfx21QxdFe6OpbLfOxQDklYA2WIPD300j3jQ611N9eBuGsvXn+oeFRan1omDnUuHAA+PqwsY/MeREjRfptyAQOhoMZ/D6XIiBHtQm5H4iQ+tqH+7l5DTojsf+iHny1jNu/DKrLQJwFaJx4RobxINQV7T68I9OR/onSNp4LOZMzsCgYbxzkn3UAqemB+a9uzDnTujOXqpsREQ0EgUXfZ6ESvEJHarmFW3pu2CVLMQlM96KXTFtBFXA+TAvrTkB5klBCBSEdJZomOwnLuGxBc7nc= ceph-9f4888ec-ed29-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T17:38:19.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:19 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3337050887' entity='client.admin' 2024-03-28T17:38:19.475 INFO:teuthology.orchestra.run.smithi045.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFbuyxvgdUhfq0nwnk1xUkPjL73wGaJmY8CRinLud8x75kIvoa/j+BE4X/1fsKx0oo1G8793Y+h7EP31Qxmg260wB5tg3SVbCZOjTbJ7oC/Dxct0scCzxLVdJ5xG+G8dz+CR5FBuj/0cBZhCV0MqvOVBQBcVzkH7MWkKkmdjkL7nxmi29H5p8QouGgSCEEllBv1MsdmXV5fy7YT3SlpZUKdELPaST+EXTJjCinXw1FwKfx21QxdFe6OpbLfOxQDklYA2WIPD300j3jQ611N9eBuGsvXn+oeFRan1omDnUuHAA+PqwsY/MeREjRfptyAQOhoMZ/D6XIiBHtQm5H4iQ+tqH+7l5DTojsf+iHny1jNu/DKrLQJwFaJx4RobxINQV7T68I9OR/onSNp4LOZMzsCgYbxzkn3UAqemB+a9uzDnTujOXqpsREQ0EgUXfZ6ESvEJHarmFW3pu2CVLMQlM96KXTFtBFXA+TAvrTkB5klBCBSEdJZomOwnLuGxBc7nc= ceph-9f4888ec-ed29-11ee-95d0-87774f69a715 2024-03-28T17:38:19.546 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFbuyxvgdUhfq0nwnk1xUkPjL73wGaJmY8CRinLud8x75kIvoa/j+BE4X/1fsKx0oo1G8793Y+h7EP31Qxmg260wB5tg3SVbCZOjTbJ7oC/Dxct0scCzxLVdJ5xG+G8dz+CR5FBuj/0cBZhCV0MqvOVBQBcVzkH7MWkKkmdjkL7nxmi29H5p8QouGgSCEEllBv1MsdmXV5fy7YT3SlpZUKdELPaST+EXTJjCinXw1FwKfx21QxdFe6OpbLfOxQDklYA2WIPD300j3jQ611N9eBuGsvXn+oeFRan1omDnUuHAA+PqwsY/MeREjRfptyAQOhoMZ/D6XIiBHtQm5H4iQ+tqH+7l5DTojsf+iHny1jNu/DKrLQJwFaJx4RobxINQV7T68I9OR/onSNp4LOZMzsCgYbxzkn3UAqemB+a9uzDnTujOXqpsREQ0EgUXfZ6ESvEJHarmFW3pu2CVLMQlM96KXTFtBFXA+TAvrTkB5klBCBSEdJZomOwnLuGxBc7nc= ceph-9f4888ec-ed29-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T17:38:19.594 INFO:teuthology.orchestra.run.smithi071.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFbuyxvgdUhfq0nwnk1xUkPjL73wGaJmY8CRinLud8x75kIvoa/j+BE4X/1fsKx0oo1G8793Y+h7EP31Qxmg260wB5tg3SVbCZOjTbJ7oC/Dxct0scCzxLVdJ5xG+G8dz+CR5FBuj/0cBZhCV0MqvOVBQBcVzkH7MWkKkmdjkL7nxmi29H5p8QouGgSCEEllBv1MsdmXV5fy7YT3SlpZUKdELPaST+EXTJjCinXw1FwKfx21QxdFe6OpbLfOxQDklYA2WIPD300j3jQ611N9eBuGsvXn+oeFRan1omDnUuHAA+PqwsY/MeREjRfptyAQOhoMZ/D6XIiBHtQm5H4iQ+tqH+7l5DTojsf+iHny1jNu/DKrLQJwFaJx4RobxINQV7T68I9OR/onSNp4LOZMzsCgYbxzkn3UAqemB+a9uzDnTujOXqpsREQ0EgUXfZ6ESvEJHarmFW3pu2CVLMQlM96KXTFtBFXA+TAvrTkB5klBCBSEdJZomOwnLuGxBc7nc= ceph-9f4888ec-ed29-11ee-95d0-87774f69a715 2024-03-28T17:38:19.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFbuyxvgdUhfq0nwnk1xUkPjL73wGaJmY8CRinLud8x75kIvoa/j+BE4X/1fsKx0oo1G8793Y+h7EP31Qxmg260wB5tg3SVbCZOjTbJ7oC/Dxct0scCzxLVdJ5xG+G8dz+CR5FBuj/0cBZhCV0MqvOVBQBcVzkH7MWkKkmdjkL7nxmi29H5p8QouGgSCEEllBv1MsdmXV5fy7YT3SlpZUKdELPaST+EXTJjCinXw1FwKfx21QxdFe6OpbLfOxQDklYA2WIPD300j3jQ611N9eBuGsvXn+oeFRan1omDnUuHAA+PqwsY/MeREjRfptyAQOhoMZ/D6XIiBHtQm5H4iQ+tqH+7l5DTojsf+iHny1jNu/DKrLQJwFaJx4RobxINQV7T68I9OR/onSNp4LOZMzsCgYbxzkn3UAqemB+a9uzDnTujOXqpsREQ0EgUXfZ6ESvEJHarmFW3pu2CVLMQlM96KXTFtBFXA+TAvrTkB5klBCBSEdJZomOwnLuGxBc7nc= ceph-9f4888ec-ed29-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T17:38:19.657 INFO:teuthology.orchestra.run.smithi151.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFbuyxvgdUhfq0nwnk1xUkPjL73wGaJmY8CRinLud8x75kIvoa/j+BE4X/1fsKx0oo1G8793Y+h7EP31Qxmg260wB5tg3SVbCZOjTbJ7oC/Dxct0scCzxLVdJ5xG+G8dz+CR5FBuj/0cBZhCV0MqvOVBQBcVzkH7MWkKkmdjkL7nxmi29H5p8QouGgSCEEllBv1MsdmXV5fy7YT3SlpZUKdELPaST+EXTJjCinXw1FwKfx21QxdFe6OpbLfOxQDklYA2WIPD300j3jQ611N9eBuGsvXn+oeFRan1omDnUuHAA+PqwsY/MeREjRfptyAQOhoMZ/D6XIiBHtQm5H4iQ+tqH+7l5DTojsf+iHny1jNu/DKrLQJwFaJx4RobxINQV7T68I9OR/onSNp4LOZMzsCgYbxzkn3UAqemB+a9uzDnTujOXqpsREQ0EgUXfZ6ESvEJHarmFW3pu2CVLMQlM96KXTFtBFXA+TAvrTkB5klBCBSEdJZomOwnLuGxBc7nc= ceph-9f4888ec-ed29-11ee-95d0-87774f69a715 2024-03-28T17:38:19.672 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-28T17:38:20.114 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:38:22.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.941+0000 7f7c24400700 1 -- 172.21.15.45:0/4132931322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c1c076140 msgr2=0x7f7c1c0745a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:22.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.941+0000 7f7c24400700 1 --2- 172.21.15.45:0/4132931322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c1c076140 0x7f7c1c0745a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7c10009910 tx=0x7f7c10009c20 comp rx=0 tx=0).stop 2024-03-28T17:38:22.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.941+0000 7f7c24400700 1 -- 172.21.15.45:0/4132931322 shutdown_connections 2024-03-28T17:38:22.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.941+0000 7f7c24400700 1 --2- 172.21.15.45:0/4132931322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c1c076140 0x7f7c1c0745a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:22.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.941+0000 7f7c24400700 1 -- 172.21.15.45:0/4132931322 >> 172.21.15.45:0/4132931322 conn(0x7f7c1c0fd160 msgr2=0x7f7c1c0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:22.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.941+0000 7f7c24400700 1 -- 172.21.15.45:0/4132931322 shutdown_connections 2024-03-28T17:38:22.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.941+0000 7f7c24400700 1 -- 172.21.15.45:0/4132931322 wait complete. 2024-03-28T17:38:22.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.942+0000 7f7c24400700 1 Processor -- start 2024-03-28T17:38:22.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.943+0000 7f7c24400700 1 -- start start 2024-03-28T17:38:22.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.943+0000 7f7c24400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c1c076140 0x7f7c1c199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:22.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.943+0000 7f7c24400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1c109e40 con 0x7f7c1c076140 2024-03-28T17:38:22.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.943+0000 7f7c21a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c1c076140 0x7f7c1c199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:22.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.943+0000 7f7c21a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c1c076140 0x7f7c1c199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59964/0 (socket says 172.21.15.45:59964) 2024-03-28T17:38:22.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.943+0000 7f7c21a00700 1 -- 172.21.15.45:0/316395923 learned_addr learned my addr 172.21.15.45:0/316395923 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:38:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.944+0000 7f7c21a00700 1 -- 172.21.15.45:0/316395923 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1c199990 con 0x7f7c1c076140 2024-03-28T17:38:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.944+0000 7f7c21a00700 1 --2- 172.21.15.45:0/316395923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c1c076140 0x7f7c1c199450 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7c10005950 tx=0x7f7c10005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.945+0000 7f7c1a000700 1 -- 172.21.15.45:0/316395923 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c10026070 con 0x7f7c1c076140 2024-03-28T17:38:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.945+0000 7f7c1a000700 1 -- 172.21.15.45:0/316395923 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c1000bc30 con 0x7f7c1c076140 2024-03-28T17:38:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.945+0000 7f7c1a000700 1 -- 172.21.15.45:0/316395923 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c10003b30 con 0x7f7c1c076140 2024-03-28T17:38:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.945+0000 7f7c24400700 1 -- 172.21.15.45:0/316395923 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c1c199c20 con 0x7f7c1c076140 2024-03-28T17:38:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.945+0000 7f7c24400700 1 -- 172.21.15.45:0/316395923 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c1c19c910 con 0x7f7c1c076140 2024-03-28T17:38:22.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.946+0000 7f7c24400700 1 -- 172.21.15.45:0/316395923 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c1c10a5c0 con 0x7f7c1c076140 2024-03-28T17:38:22.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.946+0000 7f7c1a000700 1 -- 172.21.15.45:0/316395923 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f7c10029070 con 0x7f7c1c076140 2024-03-28T17:38:22.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.947+0000 7f7c1a000700 1 --2- 172.21.15.45:0/316395923 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f7c04039020 0x7f7c0403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:22.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.947+0000 7f7c1a000700 1 -- 172.21.15.45:0/316395923 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7c1001e070 con 0x7f7c1c076140 2024-03-28T17:38:22.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.947+0000 7f7c21000700 1 --2- 172.21.15.45:0/316395923 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f7c04039020 0x7f7c0403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:22.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.948+0000 7f7c21000700 1 --2- 172.21.15.45:0/316395923 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f7c04039020 0x7f7c0403b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7c08006fd0 tx=0x7f7c08006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:22.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:22.952+0000 7f7c1a000700 1 -- 172.21.15.45:0/316395923 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7c10013070 con 0x7f7c1c076140 2024-03-28T17:38:23.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:23.182+0000 7f7c24400700 1 -- 172.21.15.45:0/316395923 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f7c1c02db40 con 0x7f7c1c076140 2024-03-28T17:38:23.184 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:23 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:23.184 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:23 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:23.184 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:23 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi045", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:38:23.184 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:23 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:23.184 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:23 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:38:23.184 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:23 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:23.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:23.186+0000 7f7c1a000700 1 -- 172.21.15.45:0/316395923 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f7c10030020 con 0x7f7c1c076140 2024-03-28T17:38:23.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:23.189+0000 7f7c24400700 1 -- 172.21.15.45:0/316395923 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f7c04039020 msgr2=0x7f7c0403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:23.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:23.189+0000 7f7c24400700 1 --2- 172.21.15.45:0/316395923 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f7c04039020 0x7f7c0403b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7c08006fd0 tx=0x7f7c08006e40 comp rx=0 tx=0).stop 2024-03-28T17:38:23.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:23.189+0000 7f7c24400700 1 -- 172.21.15.45:0/316395923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c1c076140 msgr2=0x7f7c1c199450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:23.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:23.189+0000 7f7c24400700 1 --2- 172.21.15.45:0/316395923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c1c076140 0x7f7c1c199450 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7c10005950 tx=0x7f7c10005030 comp rx=0 tx=0).stop 2024-03-28T17:38:23.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:23.190+0000 7f7c24400700 1 -- 172.21.15.45:0/316395923 shutdown_connections 2024-03-28T17:38:23.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:23.190+0000 7f7c24400700 1 --2- 172.21.15.45:0/316395923 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f7c04039020 0x7f7c0403b4e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:23.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:23.190+0000 7f7c24400700 1 --2- 172.21.15.45:0/316395923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c1c076140 0x7f7c1c199450 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:23.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:23.190+0000 7f7c24400700 1 -- 172.21.15.45:0/316395923 >> 172.21.15.45:0/316395923 conn(0x7f7c1c0fd160 msgr2=0x7f7c1c0fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:23.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:23.190+0000 7f7c24400700 1 -- 172.21.15.45:0/316395923 shutdown_connections 2024-03-28T17:38:23.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:23.190+0000 7f7c24400700 1 -- 172.21.15.45:0/316395923 wait complete. 2024-03-28T17:38:24.056 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-28T17:38:24.057 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-28T17:38:24.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:24 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/316395923' entity='client.admin' 2024-03-28T17:38:24.349 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:24 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:38:24.350 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:24 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:25.077 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:38:26.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.991+0000 7fde48600700 1 -- 172.21.15.45:0/4107088084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde40101bc0 msgr2=0x7fde40101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:26.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.991+0000 7fde48600700 1 --2- 172.21.15.45:0/4107088084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde40101bc0 0x7fde40101fe0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fde3400b0a0 tx=0x7fde3400b3b0 comp rx=0 tx=0).stop 2024-03-28T17:38:26.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.992+0000 7fde48600700 1 -- 172.21.15.45:0/4107088084 shutdown_connections 2024-03-28T17:38:26.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.992+0000 7fde48600700 1 --2- 172.21.15.45:0/4107088084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde40101bc0 0x7fde40101fe0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:26.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.992+0000 7fde48600700 1 -- 172.21.15.45:0/4107088084 >> 172.21.15.45:0/4107088084 conn(0x7fde400fd160 msgr2=0x7fde400ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:26.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.992+0000 7fde48600700 1 -- 172.21.15.45:0/4107088084 shutdown_connections 2024-03-28T17:38:26.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.992+0000 7fde48600700 1 -- 172.21.15.45:0/4107088084 wait complete. 2024-03-28T17:38:26.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.993+0000 7fde48600700 1 Processor -- start 2024-03-28T17:38:26.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.993+0000 7fde48600700 1 -- start start 2024-03-28T17:38:26.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.993+0000 7fde48600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde40101bc0 0x7fde40075e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:26.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.993+0000 7fde48600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde40107c50 con 0x7fde40101bc0 2024-03-28T17:38:26.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.994+0000 7fde45c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde40101bc0 0x7fde40075e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:26.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.994+0000 7fde45c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde40101bc0 0x7fde40075e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60258/0 (socket says 172.21.15.45:60258) 2024-03-28T17:38:26.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.994+0000 7fde45c00700 1 -- 172.21.15.45:0/3418253969 learned_addr learned my addr 172.21.15.45:0/3418253969 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:38:26.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.994+0000 7fde45c00700 1 -- 172.21.15.45:0/3418253969 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde40076390 con 0x7fde40101bc0 2024-03-28T17:38:26.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.994+0000 7fde45c00700 1 --2- 172.21.15.45:0/3418253969 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde40101bc0 0x7fde40075e50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fde34007890 tx=0x7fde34009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:26.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.995+0000 7fde3e000700 1 -- 172.21.15.45:0/3418253969 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde34017040 con 0x7fde40101bc0 2024-03-28T17:38:26.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.995+0000 7fde3e000700 1 -- 172.21.15.45:0/3418253969 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde3400bd50 con 0x7fde40101bc0 2024-03-28T17:38:26.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.995+0000 7fde48600700 1 -- 172.21.15.45:0/3418253969 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde40074580 con 0x7fde40101bc0 2024-03-28T17:38:26.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.995+0000 7fde3e000700 1 -- 172.21.15.45:0/3418253969 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde3401d7a0 con 0x7fde40101bc0 2024-03-28T17:38:26.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.995+0000 7fde48600700 1 -- 172.21.15.45:0/3418253969 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde40074ac0 con 0x7fde40101bc0 2024-03-28T17:38:26.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.996+0000 7fde48600700 1 -- 172.21.15.45:0/3418253969 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde401083d0 con 0x7fde40101bc0 2024-03-28T17:38:26.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.996+0000 7fde3e000700 1 -- 172.21.15.45:0/3418253969 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fde34024020 con 0x7fde40101bc0 2024-03-28T17:38:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.996+0000 7fde3e000700 1 --2- 172.21.15.45:0/3418253969 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fde28039010 0x7fde2803b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.996+0000 7fde3e000700 1 -- 172.21.15.45:0/3418253969 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fde34022020 con 0x7fde40101bc0 2024-03-28T17:38:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.997+0000 7fde45200700 1 --2- 172.21.15.45:0/3418253969 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fde28039010 0x7fde2803b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:26.997+0000 7fde45200700 1 --2- 172.21.15.45:0/3418253969 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fde28039010 0x7fde2803b4d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fde2c006fd0 tx=0x7fde2c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:27.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:27.002+0000 7fde3e000700 1 -- 172.21.15.45:0/3418253969 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fde3402b020 con 0x7fde40101bc0 2024-03-28T17:38:27.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:27.235+0000 7fde48600700 1 -- 172.21.15.45:0/3418253969 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fde40190780 con 0x7fde28039010 2024-03-28T17:38:27.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:27.243+0000 7fde3e000700 1 -- 172.21.15.45:0/3418253969 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fde40190780 con 0x7fde28039010 2024-03-28T17:38:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:27.246+0000 7fde48600700 1 -- 172.21.15.45:0/3418253969 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fde28039010 msgr2=0x7fde2803b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:27.246+0000 7fde48600700 1 --2- 172.21.15.45:0/3418253969 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fde28039010 0x7fde2803b4d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fde2c006fd0 tx=0x7fde2c006e40 comp rx=0 tx=0).stop 2024-03-28T17:38:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:27.246+0000 7fde48600700 1 -- 172.21.15.45:0/3418253969 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde40101bc0 msgr2=0x7fde40075e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:27.246+0000 7fde48600700 1 --2- 172.21.15.45:0/3418253969 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde40101bc0 0x7fde40075e50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fde34007890 tx=0x7fde34009790 comp rx=0 tx=0).stop 2024-03-28T17:38:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:27.246+0000 7fde48600700 1 -- 172.21.15.45:0/3418253969 shutdown_connections 2024-03-28T17:38:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:27.246+0000 7fde48600700 1 --2- 172.21.15.45:0/3418253969 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fde28039010 0x7fde2803b4d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:27.246+0000 7fde48600700 1 --2- 172.21.15.45:0/3418253969 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde40101bc0 0x7fde40075e50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:27.246+0000 7fde48600700 1 -- 172.21.15.45:0/3418253969 >> 172.21.15.45:0/3418253969 conn(0x7fde400fd160 msgr2=0x7fde400fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:27.246+0000 7fde48600700 1 -- 172.21.15.45:0/3418253969 shutdown_connections 2024-03-28T17:38:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:27.247+0000 7fde48600700 1 -- 172.21.15.45:0/3418253969 wait complete. 2024-03-28T17:38:28.078 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi071 2024-03-28T17:38:28.079 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-28T17:38:28.079 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/etc/ceph/ceph.conf 2024-03-28T17:38:28.097 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-28T17:38:28.097 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T17:38:28.154 INFO:tasks.cephadm:Adding host smithi071 to orchestrator... 2024-03-28T17:38:28.154 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph orch host add smithi071 2024-03-28T17:38:28.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:28 smithi045 ceph-mon[31773]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:38:28.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:28 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:28.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:28 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:38:28.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:28 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:38:28.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:28 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:38:28.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:28 smithi045 ceph-mon[31773]: Updating smithi045:/etc/ceph/ceph.conf 2024-03-28T17:38:28.582 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:38:29.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:29 smithi045 ceph-mon[31773]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:38:29.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:29 smithi045 ceph-mon[31773]: Updating smithi045:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:38:30.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:30 smithi045 ceph-mon[31773]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-03-28T17:38:30.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:30 smithi045 ceph-mon[31773]: Updating smithi045:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T17:38:30.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:30 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:30.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:30 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:30.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:30 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:30.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.790+0000 7f38ff600700 1 -- 172.21.15.45:0/688059092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38f8103ff0 msgr2=0x7f38f81043d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:30.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.790+0000 7f38ff600700 1 --2- 172.21.15.45:0/688059092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38f8103ff0 0x7f38f81043d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f38ec009910 tx=0x7f38ec009c20 comp rx=0 tx=0).stop 2024-03-28T17:38:30.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.790+0000 7f38ff600700 1 -- 172.21.15.45:0/688059092 shutdown_connections 2024-03-28T17:38:30.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.790+0000 7f38ff600700 1 --2- 172.21.15.45:0/688059092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38f8103ff0 0x7f38f81043d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:30.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.790+0000 7f38ff600700 1 -- 172.21.15.45:0/688059092 >> 172.21.15.45:0/688059092 conn(0x7f38f80ff880 msgr2=0x7f38f8101ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:30.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.790+0000 7f38ff600700 1 -- 172.21.15.45:0/688059092 shutdown_connections 2024-03-28T17:38:30.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.790+0000 7f38ff600700 1 -- 172.21.15.45:0/688059092 wait complete. 2024-03-28T17:38:30.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.791+0000 7f38ff600700 1 Processor -- start 2024-03-28T17:38:30.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.791+0000 7f38ff600700 1 -- start start 2024-03-28T17:38:30.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.792+0000 7f38ff600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38f8103ff0 0x7f38f819f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:30.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.792+0000 7f38ff600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f810e3c0 con 0x7f38f8103ff0 2024-03-28T17:38:30.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.792+0000 7f38fcc00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38f8103ff0 0x7f38f819f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:30.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.792+0000 7f38fcc00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38f8103ff0 0x7f38f819f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60268/0 (socket says 172.21.15.45:60268) 2024-03-28T17:38:30.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.792+0000 7f38fcc00700 1 -- 172.21.15.45:0/3340692375 learned_addr learned my addr 172.21.15.45:0/3340692375 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:38:30.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.792+0000 7f38fcc00700 1 -- 172.21.15.45:0/3340692375 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f819fb20 con 0x7f38f8103ff0 2024-03-28T17:38:30.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.793+0000 7f38fcc00700 1 --2- 172.21.15.45:0/3340692375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38f8103ff0 0x7f38f819f5e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f38ec005850 tx=0x7f38ec005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:30.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.793+0000 7f38f5600700 1 -- 172.21.15.45:0/3340692375 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f38ec026070 con 0x7f38f8103ff0 2024-03-28T17:38:30.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.793+0000 7f38f5600700 1 -- 172.21.15.45:0/3340692375 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38ec00bbd0 con 0x7f38f8103ff0 2024-03-28T17:38:30.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.793+0000 7f38f5600700 1 -- 172.21.15.45:0/3340692375 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f38ec003970 con 0x7f38f8103ff0 2024-03-28T17:38:30.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.793+0000 7f38ff600700 1 -- 172.21.15.45:0/3340692375 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f819fe10 con 0x7f38f8103ff0 2024-03-28T17:38:30.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.793+0000 7f38ff600700 1 -- 172.21.15.45:0/3340692375 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f8199880 con 0x7f38f8103ff0 2024-03-28T17:38:30.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.794+0000 7f38f5600700 1 -- 172.21.15.45:0/3340692375 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f38ec029070 con 0x7f38f8103ff0 2024-03-28T17:38:30.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.794+0000 7f38ff600700 1 -- 172.21.15.45:0/3340692375 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38f81054d0 con 0x7f38f8103ff0 2024-03-28T17:38:30.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.795+0000 7f38f5600700 1 --2- 172.21.15.45:0/3340692375 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f38e0039020 0x7f38e003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:30.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.795+0000 7f38f5600700 1 -- 172.21.15.45:0/3340692375 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f38ec01e070 con 0x7f38f8103ff0 2024-03-28T17:38:30.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.795+0000 7f38f7e00700 1 --2- 172.21.15.45:0/3340692375 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f38e0039020 0x7f38e003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:30.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.796+0000 7f38f7e00700 1 --2- 172.21.15.45:0/3340692375 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f38e0039020 0x7f38e003b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f38e4006fd0 tx=0x7f38e4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:30.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:30.801+0000 7f38f5600700 1 -- 172.21.15.45:0/3340692375 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f38ec013070 con 0x7f38f8103ff0 2024-03-28T17:38:31.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:31.042+0000 7f38ff600700 1 -- 172.21.15.45:0/3340692375 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi071", "target": ["mon-mgr", ""]}) v1 -- 0x7f38f819ac10 con 0x7f38e0039020 2024-03-28T17:38:31.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:31 smithi045 ceph-mon[31773]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:38:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:32 smithi045 ceph-mon[31773]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi071", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:38:33.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:33 smithi045 ceph-mon[31773]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:38:33.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:33 smithi045 ceph-mon[31773]: Deploying cephadm binary to smithi071 2024-03-28T17:38:35.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:35 smithi045 ceph-mon[31773]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:38:36.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:36.246+0000 7f38f5600700 1 -- 172.21.15.45:0/3340692375 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f38f819ac10 con 0x7f38e0039020 2024-03-28T17:38:36.247 INFO:teuthology.orchestra.run.smithi045.stdout:Added host 'smithi071' with addr '172.21.15.71' 2024-03-28T17:38:36.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:36.250+0000 7f38ff600700 1 -- 172.21.15.45:0/3340692375 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f38e0039020 msgr2=0x7f38e003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:36.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:36.250+0000 7f38ff600700 1 --2- 172.21.15.45:0/3340692375 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f38e0039020 0x7f38e003b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f38e4006fd0 tx=0x7f38e4006e40 comp rx=0 tx=0).stop 2024-03-28T17:38:36.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:36.250+0000 7f38ff600700 1 -- 172.21.15.45:0/3340692375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38f8103ff0 msgr2=0x7f38f819f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:36.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:36.250+0000 7f38ff600700 1 --2- 172.21.15.45:0/3340692375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38f8103ff0 0x7f38f819f5e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f38ec005850 tx=0x7f38ec005030 comp rx=0 tx=0).stop 2024-03-28T17:38:36.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:36.250+0000 7f38ff600700 1 -- 172.21.15.45:0/3340692375 shutdown_connections 2024-03-28T17:38:36.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:36.250+0000 7f38ff600700 1 --2- 172.21.15.45:0/3340692375 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f38e0039020 0x7f38e003b4e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:36.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:36.250+0000 7f38ff600700 1 --2- 172.21.15.45:0/3340692375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38f8103ff0 0x7f38f819f5e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:36.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:36.250+0000 7f38ff600700 1 -- 172.21.15.45:0/3340692375 >> 172.21.15.45:0/3340692375 conn(0x7f38f80ff880 msgr2=0x7f38f81059c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:36.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:36.250+0000 7f38ff600700 1 -- 172.21.15.45:0/3340692375 shutdown_connections 2024-03-28T17:38:36.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:36.250+0000 7f38ff600700 1 -- 172.21.15.45:0/3340692375 wait complete. 2024-03-28T17:38:37.052 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-28T17:38:37.543 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:38:37.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:37 smithi045 ceph-mon[31773]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:38:37.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:37 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:37.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:37 smithi045 ceph-mon[31773]: Added host smithi071 2024-03-28T17:38:37.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:37 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:38:37.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:37 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:38.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:38 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:39.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.495+0000 7f46dbc00700 1 -- 172.21.15.45:0/3155106451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d4102d20 msgr2=0x7f46d4103100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:39.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.495+0000 7f46dbc00700 1 --2- 172.21.15.45:0/3155106451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d4102d20 0x7f46d4103100 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f46c8009910 tx=0x7f46c8009c20 comp rx=0 tx=0).stop 2024-03-28T17:38:39.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.495+0000 7f46dbc00700 1 -- 172.21.15.45:0/3155106451 shutdown_connections 2024-03-28T17:38:39.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.495+0000 7f46dbc00700 1 --2- 172.21.15.45:0/3155106451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d4102d20 0x7f46d4103100 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:39.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.495+0000 7f46dbc00700 1 -- 172.21.15.45:0/3155106451 >> 172.21.15.45:0/3155106451 conn(0x7f46d40776d0 msgr2=0x7f46d40fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:39.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.495+0000 7f46dbc00700 1 -- 172.21.15.45:0/3155106451 shutdown_connections 2024-03-28T17:38:39.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.495+0000 7f46dbc00700 1 -- 172.21.15.45:0/3155106451 wait complete. 2024-03-28T17:38:39.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.497+0000 7f46dbc00700 1 Processor -- start 2024-03-28T17:38:39.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.497+0000 7f46dbc00700 1 -- start start 2024-03-28T17:38:39.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.497+0000 7f46dbc00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d4102d20 0x7f46d4197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:39.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.497+0000 7f46dbc00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d410c210 con 0x7f46d4102d20 2024-03-28T17:38:39.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.498+0000 7f46d9200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d4102d20 0x7f46d4197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:39.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.498+0000 7f46d9200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d4102d20 0x7f46d4197740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36306/0 (socket says 172.21.15.45:36306) 2024-03-28T17:38:39.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.498+0000 7f46d9200700 1 -- 172.21.15.45:0/4213385526 learned_addr learned my addr 172.21.15.45:0/4213385526 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:38:39.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.498+0000 7f46d9200700 1 -- 172.21.15.45:0/4213385526 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d4197c80 con 0x7f46d4102d20 2024-03-28T17:38:39.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.499+0000 7f46d9200700 1 --2- 172.21.15.45:0/4213385526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d4102d20 0x7f46d4197740 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f46c8006010 tx=0x7f46c8005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:39.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.499+0000 7f46d1600700 1 -- 172.21.15.45:0/4213385526 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f46c8026070 con 0x7f46d4102d20 2024-03-28T17:38:39.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.499+0000 7f46d1600700 1 -- 172.21.15.45:0/4213385526 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46c800bc30 con 0x7f46d4102d20 2024-03-28T17:38:39.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.499+0000 7f46d1600700 1 -- 172.21.15.45:0/4213385526 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f46c8003ab0 con 0x7f46d4102d20 2024-03-28T17:38:39.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.499+0000 7f46dbc00700 1 -- 172.21.15.45:0/4213385526 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d4197f10 con 0x7f46d4102d20 2024-03-28T17:38:39.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.499+0000 7f46dbc00700 1 -- 172.21.15.45:0/4213385526 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d4198370 con 0x7f46d4102d20 2024-03-28T17:38:39.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.501+0000 7f46d1600700 1 -- 172.21.15.45:0/4213385526 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f46c8029070 con 0x7f46d4102d20 2024-03-28T17:38:39.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.501+0000 7f46dbc00700 1 -- 172.21.15.45:0/4213385526 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d4069e60 con 0x7f46d4102d20 2024-03-28T17:38:39.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.501+0000 7f46d1600700 1 --2- 172.21.15.45:0/4213385526 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f46bc039020 0x7f46bc03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:39.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.501+0000 7f46d1600700 1 -- 172.21.15.45:0/4213385526 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f46c801e070 con 0x7f46d4102d20 2024-03-28T17:38:39.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.501+0000 7f46d3e00700 1 --2- 172.21.15.45:0/4213385526 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f46bc039020 0x7f46bc03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:39.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.502+0000 7f46d3e00700 1 --2- 172.21.15.45:0/4213385526 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f46bc039020 0x7f46bc03b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f46c0006fd0 tx=0x7f46c0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:39.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.507+0000 7f46d1600700 1 -- 172.21.15.45:0/4213385526 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f46c8013070 con 0x7f46d4102d20 2024-03-28T17:38:39.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:39 smithi045 ceph-mon[31773]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:38:39.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:39 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:39.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.776+0000 7f46dbc00700 1 -- 172.21.15.45:0/4213385526 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f46d4198aa0 con 0x7f46bc039020 2024-03-28T17:38:39.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.778+0000 7f46d1600700 1 -- 172.21.15.45:0/4213385526 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f46d4198aa0 con 0x7f46bc039020 2024-03-28T17:38:39.778 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:38:39.778 INFO:teuthology.orchestra.run.smithi045.stdout:[{"addr": "172.21.15.45", "hostname": "smithi045", "labels": [], "status": ""}, {"addr": "172.21.15.71", "hostname": "smithi071", "labels": [], "status": ""}] 2024-03-28T17:38:39.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.781+0000 7f46dbc00700 1 -- 172.21.15.45:0/4213385526 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f46bc039020 msgr2=0x7f46bc03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:39.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.781+0000 7f46dbc00700 1 --2- 172.21.15.45:0/4213385526 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f46bc039020 0x7f46bc03b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f46c0006fd0 tx=0x7f46c0006e40 comp rx=0 tx=0).stop 2024-03-28T17:38:39.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.781+0000 7f46dbc00700 1 -- 172.21.15.45:0/4213385526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d4102d20 msgr2=0x7f46d4197740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:39.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.781+0000 7f46dbc00700 1 --2- 172.21.15.45:0/4213385526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d4102d20 0x7f46d4197740 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f46c8006010 tx=0x7f46c8005050 comp rx=0 tx=0).stop 2024-03-28T17:38:39.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.781+0000 7f46dbc00700 1 -- 172.21.15.45:0/4213385526 shutdown_connections 2024-03-28T17:38:39.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.781+0000 7f46dbc00700 1 --2- 172.21.15.45:0/4213385526 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f46bc039020 0x7f46bc03b4e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:39.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.781+0000 7f46dbc00700 1 --2- 172.21.15.45:0/4213385526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d4102d20 0x7f46d4197740 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:39.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.781+0000 7f46dbc00700 1 -- 172.21.15.45:0/4213385526 >> 172.21.15.45:0/4213385526 conn(0x7f46d40776d0 msgr2=0x7f46d4103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:39.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.781+0000 7f46dbc00700 1 -- 172.21.15.45:0/4213385526 shutdown_connections 2024-03-28T17:38:39.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:39.781+0000 7f46dbc00700 1 -- 172.21.15.45:0/4213385526 wait complete. 2024-03-28T17:38:40.474 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi151 2024-03-28T17:38:40.474 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-28T17:38:40.475 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/etc/ceph/ceph.conf 2024-03-28T17:38:40.492 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-28T17:38:40.493 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T17:38:40.551 INFO:tasks.cephadm:Adding host smithi151 to orchestrator... 2024-03-28T17:38:40.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph orch host add smithi151 2024-03-28T17:38:40.998 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:38:41.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:41 smithi045 ceph-mon[31773]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:38:41.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:41 smithi045 ceph-mon[31773]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:38:43.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.068+0000 7f3d6d400700 1 -- 172.21.15.45:0/1405410222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d68103e40 msgr2=0x7f3d68104220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:43.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.068+0000 7f3d6d400700 1 --2- 172.21.15.45:0/1405410222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d68103e40 0x7f3d68104220 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3d58009910 tx=0x7f3d58009c20 comp rx=0 tx=0).stop 2024-03-28T17:38:43.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.068+0000 7f3d6d400700 1 -- 172.21.15.45:0/1405410222 shutdown_connections 2024-03-28T17:38:43.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.068+0000 7f3d6d400700 1 --2- 172.21.15.45:0/1405410222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d68103e40 0x7f3d68104220 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:43.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.068+0000 7f3d6d400700 1 -- 172.21.15.45:0/1405410222 >> 172.21.15.45:0/1405410222 conn(0x7f3d680ff6d0 msgr2=0x7f3d68101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:43.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.068+0000 7f3d6d400700 1 -- 172.21.15.45:0/1405410222 shutdown_connections 2024-03-28T17:38:43.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.068+0000 7f3d6d400700 1 -- 172.21.15.45:0/1405410222 wait complete. 2024-03-28T17:38:43.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.069+0000 7f3d6d400700 1 Processor -- start 2024-03-28T17:38:43.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.069+0000 7f3d6d400700 1 -- start start 2024-03-28T17:38:43.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.070+0000 7f3d6d400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d68103e40 0x7f3d681996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:43.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.070+0000 7f3d6d400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6810e210 con 0x7f3d68103e40 2024-03-28T17:38:43.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.070+0000 7f3d67e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d68103e40 0x7f3d681996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:43.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.070+0000 7f3d67e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d68103e40 0x7f3d681996d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36316/0 (socket says 172.21.15.45:36316) 2024-03-28T17:38:43.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.070+0000 7f3d67e00700 1 -- 172.21.15.45:0/2741098261 learned_addr learned my addr 172.21.15.45:0/2741098261 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:38:43.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.070+0000 7f3d67e00700 1 -- 172.21.15.45:0/2741098261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d68199c10 con 0x7f3d68103e40 2024-03-28T17:38:43.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.071+0000 7f3d67e00700 1 --2- 172.21.15.45:0/2741098261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d68103e40 0x7f3d681996d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3d58005850 tx=0x7f3d58005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:43.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.071+0000 7f3d64c00700 1 -- 172.21.15.45:0/2741098261 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d58026070 con 0x7f3d68103e40 2024-03-28T17:38:43.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.071+0000 7f3d64c00700 1 -- 172.21.15.45:0/2741098261 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d5800bbd0 con 0x7f3d68103e40 2024-03-28T17:38:43.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.071+0000 7f3d6d400700 1 -- 172.21.15.45:0/2741098261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d68199f00 con 0x7f3d68103e40 2024-03-28T17:38:43.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.071+0000 7f3d64c00700 1 -- 172.21.15.45:0/2741098261 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d58003970 con 0x7f3d68103e40 2024-03-28T17:38:43.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.071+0000 7f3d6d400700 1 -- 172.21.15.45:0/2741098261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d6819dfa0 con 0x7f3d68103e40 2024-03-28T17:38:43.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.072+0000 7f3d6d400700 1 -- 172.21.15.45:0/2741098261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d68105320 con 0x7f3d68103e40 2024-03-28T17:38:43.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.072+0000 7f3d64c00700 1 -- 172.21.15.45:0/2741098261 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f3d58029070 con 0x7f3d68103e40 2024-03-28T17:38:43.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.073+0000 7f3d64c00700 1 --2- 172.21.15.45:0/2741098261 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3d4c039020 0x7f3d4c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:43.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.073+0000 7f3d64c00700 1 -- 172.21.15.45:0/2741098261 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3d5801e070 con 0x7f3d68103e40 2024-03-28T17:38:43.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.073+0000 7f3d67400700 1 --2- 172.21.15.45:0/2741098261 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3d4c039020 0x7f3d4c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:43.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.074+0000 7f3d67400700 1 --2- 172.21.15.45:0/2741098261 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3d4c039020 0x7f3d4c03b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3d54006fd0 tx=0x7f3d54006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:43.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.078+0000 7f3d64c00700 1 -- 172.21.15.45:0/2741098261 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3d58013070 con 0x7f3d68103e40 2024-03-28T17:38:43.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:43 smithi045 ceph-mon[31773]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:38:43.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:43 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:43.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:43 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:43.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:43 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:43.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:43 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:43.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:43 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi071", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:38:43.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:43 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:38:43.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:43 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:38:43.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:43 smithi045 ceph-mon[31773]: Updating smithi071:/etc/ceph/ceph.conf 2024-03-28T17:38:43.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:43 smithi045 ceph-mon[31773]: Updating smithi071:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:38:43.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:43.313+0000 7f3d6d400700 1 -- 172.21.15.45:0/2741098261 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi151", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d6819aa30 con 0x7f3d4c039020 2024-03-28T17:38:44.669 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:44 smithi045 ceph-mon[31773]: Updating smithi071:/etc/ceph/ceph.client.admin.keyring 2024-03-28T17:38:44.669 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:44 smithi045 ceph-mon[31773]: Updating smithi071:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T17:38:44.670 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:44 smithi045 ceph-mon[31773]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi151", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:38:44.670 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:44.670 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:44.670 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:45.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:45 smithi045 ceph-mon[31773]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:38:45.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:45 smithi045 ceph-mon[31773]: Deploying cephadm binary to smithi151 2024-03-28T17:38:47.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:47 smithi045 ceph-mon[31773]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:38:48.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:48.334+0000 7f3d64c00700 1 -- 172.21.15.45:0/2741098261 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f3d6819aa30 con 0x7f3d4c039020 2024-03-28T17:38:48.335 INFO:teuthology.orchestra.run.smithi045.stdout:Added host 'smithi151' with addr '172.21.15.151' 2024-03-28T17:38:48.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:48.338+0000 7f3d6d400700 1 -- 172.21.15.45:0/2741098261 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3d4c039020 msgr2=0x7f3d4c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:48.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:48.338+0000 7f3d6d400700 1 --2- 172.21.15.45:0/2741098261 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3d4c039020 0x7f3d4c03b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3d54006fd0 tx=0x7f3d54006e40 comp rx=0 tx=0).stop 2024-03-28T17:38:48.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:48.338+0000 7f3d6d400700 1 -- 172.21.15.45:0/2741098261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d68103e40 msgr2=0x7f3d681996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:48.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:48.338+0000 7f3d6d400700 1 --2- 172.21.15.45:0/2741098261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d68103e40 0x7f3d681996d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3d58005850 tx=0x7f3d58005030 comp rx=0 tx=0).stop 2024-03-28T17:38:48.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:48.338+0000 7f3d6d400700 1 -- 172.21.15.45:0/2741098261 shutdown_connections 2024-03-28T17:38:48.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:48.338+0000 7f3d6d400700 1 --2- 172.21.15.45:0/2741098261 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3d4c039020 0x7f3d4c03b4e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:48.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:48.338+0000 7f3d6d400700 1 --2- 172.21.15.45:0/2741098261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d68103e40 0x7f3d681996d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:48.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:48.338+0000 7f3d6d400700 1 -- 172.21.15.45:0/2741098261 >> 172.21.15.45:0/2741098261 conn(0x7f3d680ff6d0 msgr2=0x7f3d68105810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:48.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:48.339+0000 7f3d6d400700 1 -- 172.21.15.45:0/2741098261 shutdown_connections 2024-03-28T17:38:48.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:48.339+0000 7f3d6d400700 1 -- 172.21.15.45:0/2741098261 wait complete. 2024-03-28T17:38:49.039 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-28T17:38:49.504 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:38:49.652 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:49 smithi045 ceph-mon[31773]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:38:49.652 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:49 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:49.652 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:49 smithi045 ceph-mon[31773]: Added host smithi151 2024-03-28T17:38:49.652 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:49 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:38:49.652 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:49 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:50.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:50 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:51.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.594+0000 7f31ef400700 1 -- 172.21.15.45:0/1528011284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31e8101dc0 msgr2=0x7f31e81021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:51.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.594+0000 7f31ef400700 1 --2- 172.21.15.45:0/1528011284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31e8101dc0 0x7f31e81021a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f31d8009910 tx=0x7f31d8009c20 comp rx=0 tx=0).stop 2024-03-28T17:38:51.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.594+0000 7f31ef400700 1 -- 172.21.15.45:0/1528011284 shutdown_connections 2024-03-28T17:38:51.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.594+0000 7f31ef400700 1 --2- 172.21.15.45:0/1528011284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31e8101dc0 0x7f31e81021a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:51.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.594+0000 7f31ef400700 1 -- 172.21.15.45:0/1528011284 >> 172.21.15.45:0/1528011284 conn(0x7f31e80fd670 msgr2=0x7f31e80ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:51.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.594+0000 7f31ef400700 1 -- 172.21.15.45:0/1528011284 shutdown_connections 2024-03-28T17:38:51.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.594+0000 7f31ef400700 1 -- 172.21.15.45:0/1528011284 wait complete. 2024-03-28T17:38:51.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.595+0000 7f31ef400700 1 Processor -- start 2024-03-28T17:38:51.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.595+0000 7f31ef400700 1 -- start start 2024-03-28T17:38:51.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.596+0000 7f31ef400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31e8101dc0 0x7f31e819e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:51.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.596+0000 7f31ef400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e810c190 con 0x7f31e8101dc0 2024-03-28T17:38:51.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.596+0000 7f31eca00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31e8101dc0 0x7f31e819e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:51.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.596+0000 7f31eca00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31e8101dc0 0x7f31e819e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53636/0 (socket says 172.21.15.45:53636) 2024-03-28T17:38:51.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.596+0000 7f31eca00700 1 -- 172.21.15.45:0/1397715324 learned_addr learned my addr 172.21.15.45:0/1397715324 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:38:51.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.597+0000 7f31eca00700 1 -- 172.21.15.45:0/1397715324 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e819e950 con 0x7f31e8101dc0 2024-03-28T17:38:51.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.597+0000 7f31eca00700 1 --2- 172.21.15.45:0/1397715324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31e8101dc0 0x7f31e819e410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f31d8005850 tx=0x7f31d8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:51.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.597+0000 7f31e6000700 1 -- 172.21.15.45:0/1397715324 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f31d8026070 con 0x7f31e8101dc0 2024-03-28T17:38:51.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.597+0000 7f31e6000700 1 -- 172.21.15.45:0/1397715324 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31d800bbd0 con 0x7f31e8101dc0 2024-03-28T17:38:51.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.597+0000 7f31e6000700 1 -- 172.21.15.45:0/1397715324 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f31d8003970 con 0x7f31e8101dc0 2024-03-28T17:38:51.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.597+0000 7f31ef400700 1 -- 172.21.15.45:0/1397715324 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31e819ec40 con 0x7f31e8101dc0 2024-03-28T17:38:51.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.597+0000 7f31ef400700 1 -- 172.21.15.45:0/1397715324 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31e81a25b0 con 0x7f31e8101dc0 2024-03-28T17:38:51.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.599+0000 7f31e6000700 1 -- 172.21.15.45:0/1397715324 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f31d8029070 con 0x7f31e8101dc0 2024-03-28T17:38:51.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.599+0000 7f31ef400700 1 -- 172.21.15.45:0/1397715324 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31e81032a0 con 0x7f31e8101dc0 2024-03-28T17:38:51.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.599+0000 7f31e6000700 1 --2- 172.21.15.45:0/1397715324 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f31cc039020 0x7f31cc03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:51.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.599+0000 7f31e6000700 1 -- 172.21.15.45:0/1397715324 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f31d801e070 con 0x7f31e8101dc0 2024-03-28T17:38:51.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.599+0000 7f31dfe00700 1 --2- 172.21.15.45:0/1397715324 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f31cc039020 0x7f31cc03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:51.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.600+0000 7f31dfe00700 1 --2- 172.21.15.45:0/1397715324 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f31cc039020 0x7f31cc03b4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f31d0006fd0 tx=0x7f31d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.605+0000 7f31e6000700 1 -- 172.21.15.45:0/1397715324 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f31d8013070 con 0x7f31e8101dc0 2024-03-28T17:38:51.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:51 smithi045 ceph-mon[31773]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:38:51.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:51 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:51.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.841+0000 7f31ef400700 1 -- 172.21.15.45:0/1397715324 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f31e819f100 con 0x7f31cc039020 2024-03-28T17:38:51.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.843+0000 7f31e6000700 1 -- 172.21.15.45:0/1397715324 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+239 (secure 0 0 0) 0x7f31e819f100 con 0x7f31cc039020 2024-03-28T17:38:51.844 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:38:51.844 INFO:teuthology.orchestra.run.smithi045.stdout:[{"addr": "172.21.15.45", "hostname": "smithi045", "labels": [], "status": ""}, {"addr": "172.21.15.71", "hostname": "smithi071", "labels": [], "status": ""}, {"addr": "172.21.15.151", "hostname": "smithi151", "labels": [], "status": ""}] 2024-03-28T17:38:51.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.846+0000 7f31ef400700 1 -- 172.21.15.45:0/1397715324 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f31cc039020 msgr2=0x7f31cc03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:51.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.846+0000 7f31ef400700 1 --2- 172.21.15.45:0/1397715324 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f31cc039020 0x7f31cc03b4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f31d0006fd0 tx=0x7f31d0006e40 comp rx=0 tx=0).stop 2024-03-28T17:38:51.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.846+0000 7f31ef400700 1 -- 172.21.15.45:0/1397715324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31e8101dc0 msgr2=0x7f31e819e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:51.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.846+0000 7f31ef400700 1 --2- 172.21.15.45:0/1397715324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31e8101dc0 0x7f31e819e410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f31d8005850 tx=0x7f31d8005030 comp rx=0 tx=0).stop 2024-03-28T17:38:51.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.846+0000 7f31ef400700 1 -- 172.21.15.45:0/1397715324 shutdown_connections 2024-03-28T17:38:51.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.846+0000 7f31ef400700 1 --2- 172.21.15.45:0/1397715324 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f31cc039020 0x7f31cc03b4e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:51.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.846+0000 7f31ef400700 1 --2- 172.21.15.45:0/1397715324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31e8101dc0 0x7f31e819e410 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:51.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.846+0000 7f31ef400700 1 -- 172.21.15.45:0/1397715324 >> 172.21.15.45:0/1397715324 conn(0x7f31e80fd670 msgr2=0x7f31e80743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:51.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.847+0000 7f31ef400700 1 -- 172.21.15.45:0/1397715324 shutdown_connections 2024-03-28T17:38:51.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:51.847+0000 7f31ef400700 1 -- 172.21.15.45:0/1397715324 wait complete. 2024-03-28T17:38:52.587 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-28T17:38:52.587 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-28T17:38:53.047 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:38:53.465 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:53 smithi045 ceph-mon[31773]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:38:53.465 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:53 smithi045 ceph-mon[31773]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:38:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.048+0000 7fd115800700 1 -- 172.21.15.45:0/88764014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd110103fe0 msgr2=0x7fd1101043c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.048+0000 7fd115800700 1 --2- 172.21.15.45:0/88764014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd110103fe0 0x7fd1101043c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd100009910 tx=0x7fd100009c20 comp rx=0 tx=0).stop 2024-03-28T17:38:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.048+0000 7fd115800700 1 -- 172.21.15.45:0/88764014 shutdown_connections 2024-03-28T17:38:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.048+0000 7fd115800700 1 --2- 172.21.15.45:0/88764014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd110103fe0 0x7fd1101043c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.048+0000 7fd115800700 1 -- 172.21.15.45:0/88764014 >> 172.21.15.45:0/88764014 conn(0x7fd1100ff870 msgr2=0x7fd110101c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.048+0000 7fd115800700 1 -- 172.21.15.45:0/88764014 shutdown_connections 2024-03-28T17:38:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.048+0000 7fd115800700 1 -- 172.21.15.45:0/88764014 wait complete. 2024-03-28T17:38:55.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.051+0000 7fd115800700 1 Processor -- start 2024-03-28T17:38:55.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.051+0000 7fd115800700 1 -- start start 2024-03-28T17:38:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.052+0000 7fd115800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd110103fe0 0x7fd11019f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:55.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.052+0000 7fd115800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11010e3b0 con 0x7fd110103fe0 2024-03-28T17:38:55.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.052+0000 7fd10ea00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd110103fe0 0x7fd11019f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:55.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.052+0000 7fd10ea00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd110103fe0 0x7fd11019f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53656/0 (socket says 172.21.15.45:53656) 2024-03-28T17:38:55.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.052+0000 7fd10ea00700 1 -- 172.21.15.45:0/1282791936 learned_addr learned my addr 172.21.15.45:0/1282791936 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:38:55.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.053+0000 7fd10ea00700 1 -- 172.21.15.45:0/1282791936 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11019fb60 con 0x7fd110103fe0 2024-03-28T17:38:55.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.053+0000 7fd10ea00700 1 --2- 172.21.15.45:0/1282791936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd110103fe0 0x7fd11019f620 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd100006010 tx=0x7fd100005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:55.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.053+0000 7fd107400700 1 -- 172.21.15.45:0/1282791936 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd100026070 con 0x7fd110103fe0 2024-03-28T17:38:55.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.053+0000 7fd107400700 1 -- 172.21.15.45:0/1282791936 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd10000bbd0 con 0x7fd110103fe0 2024-03-28T17:38:55.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.053+0000 7fd107400700 1 -- 172.21.15.45:0/1282791936 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd100003910 con 0x7fd110103fe0 2024-03-28T17:38:55.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.053+0000 7fd115800700 1 -- 172.21.15.45:0/1282791936 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd11019fe50 con 0x7fd110103fe0 2024-03-28T17:38:55.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.054+0000 7fd115800700 1 -- 172.21.15.45:0/1282791936 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1101998c0 con 0x7fd110103fe0 2024-03-28T17:38:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.055+0000 7fd107400700 1 -- 172.21.15.45:0/1282791936 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fd100029070 con 0x7fd110103fe0 2024-03-28T17:38:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.055+0000 7fd115800700 1 -- 172.21.15.45:0/1282791936 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1101054c0 con 0x7fd110103fe0 2024-03-28T17:38:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.055+0000 7fd107400700 1 --2- 172.21.15.45:0/1282791936 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd0f4038fc0 0x7fd0f403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.055+0000 7fd107400700 1 -- 172.21.15.45:0/1282791936 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd10001e070 con 0x7fd110103fe0 2024-03-28T17:38:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.056+0000 7fd10e000700 1 --2- 172.21.15.45:0/1282791936 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd0f4038fc0 0x7fd0f403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:55.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.056+0000 7fd10e000700 1 --2- 172.21.15.45:0/1282791936 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd0f4038fc0 0x7fd0f403b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc006fd0 tx=0x7fd0fc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:55.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.061+0000 7fd107400700 1 -- 172.21.15.45:0/1282791936 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd100013070 con 0x7fd110103fe0 2024-03-28T17:38:55.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:55.310+0000 7fd115800700 1 -- 172.21.15.45:0/1282791936 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fd11019ac20 con 0x7fd110103fe0 2024-03-28T17:38:55.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:55 smithi045 ceph-mon[31773]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:38:55.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:55 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:55.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:55 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:55.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:55 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:55.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:55 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:55.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:55 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:38:55.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:55 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:38:55.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:55 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:38:55.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:55 smithi045 ceph-mon[31773]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-28T17:38:56.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:56.243+0000 7fd107400700 1 -- 172.21.15.45:0/1282791936 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd10000fe20 con 0x7fd110103fe0 2024-03-28T17:38:56.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:56.247+0000 7fd115800700 1 -- 172.21.15.45:0/1282791936 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd0f4038fc0 msgr2=0x7fd0f403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:56.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:56.247+0000 7fd115800700 1 --2- 172.21.15.45:0/1282791936 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd0f4038fc0 0x7fd0f403b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc006fd0 tx=0x7fd0fc006e40 comp rx=0 tx=0).stop 2024-03-28T17:38:56.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:56.247+0000 7fd115800700 1 -- 172.21.15.45:0/1282791936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd110103fe0 msgr2=0x7fd11019f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:56.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:56.247+0000 7fd115800700 1 --2- 172.21.15.45:0/1282791936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd110103fe0 0x7fd11019f620 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd100006010 tx=0x7fd100005030 comp rx=0 tx=0).stop 2024-03-28T17:38:56.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:56.247+0000 7fd115800700 1 -- 172.21.15.45:0/1282791936 shutdown_connections 2024-03-28T17:38:56.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:56.247+0000 7fd115800700 1 --2- 172.21.15.45:0/1282791936 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd0f4038fc0 0x7fd0f403b480 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:56.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:56.247+0000 7fd115800700 1 --2- 172.21.15.45:0/1282791936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd110103fe0 0x7fd11019f620 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:56.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:56.247+0000 7fd115800700 1 -- 172.21.15.45:0/1282791936 >> 172.21.15.45:0/1282791936 conn(0x7fd1100ff870 msgr2=0x7fd1101059b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:56.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:56.248+0000 7fd115800700 1 -- 172.21.15.45:0/1282791936 shutdown_connections 2024-03-28T17:38:56.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:38:56.248+0000 7fd115800700 1 -- 172.21.15.45:0/1282791936 wait complete. 2024-03-28T17:38:56.250 INFO:teuthology.orchestra.run.smithi045.stderr:adjusted tunables profile to default 2024-03-28T17:38:56.631 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:56 smithi045 ceph-mon[31773]: Updating smithi151:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:38:56.631 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:56 smithi045 ceph-mon[31773]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-03-28T17:38:56.631 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:56 smithi045 ceph-mon[31773]: Updating smithi151:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T17:38:56.631 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:56 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1282791936' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-28T17:38:56.631 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:56 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:56.632 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:56 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:56.632 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:56 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:57.007 INFO:tasks.cephadm:Adding mon.a on smithi045 2024-03-28T17:38:57.008 INFO:tasks.cephadm:Adding mon.b on smithi071 2024-03-28T17:38:57.008 INFO:tasks.cephadm:Adding mon.c on smithi151 2024-03-28T17:38:57.008 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi045:172.21.15.45=a;smithi071:172.21.15.71=b;smithi151:172.21.15.151=c' 2024-03-28T17:38:57.245 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:38:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:57 smithi045 ceph-mon[31773]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:38:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:57 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1282791936' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-28T17:38:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:57 smithi045 ceph-mon[31773]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T17:38:58.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.678+0000 7f8ae5186700 1 -- 172.21.15.151:0/2171070533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ae0106b70 msgr2=0x7f8ae0106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:58.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.678+0000 7f8ae5186700 1 --2- 172.21.15.151:0/2171070533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ae0106b70 0x7f8ae0106f50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4009910 tx=0x7f8ad4009c20 comp rx=0 tx=0).stop 2024-03-28T17:38:58.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.678+0000 7f8ae5186700 1 -- 172.21.15.151:0/2171070533 shutdown_connections 2024-03-28T17:38:58.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.678+0000 7f8ae5186700 1 --2- 172.21.15.151:0/2171070533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ae0106b70 0x7f8ae0106f50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:58.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.678+0000 7f8ae5186700 1 -- 172.21.15.151:0/2171070533 >> 172.21.15.151:0/2171070533 conn(0x7f8ae00fbce0 msgr2=0x7f8ae00fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:58.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.678+0000 7f8ae5186700 1 -- 172.21.15.151:0/2171070533 shutdown_connections 2024-03-28T17:38:58.679 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.678+0000 7f8ae5186700 1 -- 172.21.15.151:0/2171070533 wait complete. 2024-03-28T17:38:58.680 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.679+0000 7f8ae5186700 1 Processor -- start 2024-03-28T17:38:58.680 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.679+0000 7f8ae5186700 1 -- start start 2024-03-28T17:38:58.680 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.679+0000 7f8ae5186700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ae0106b70 0x7f8ae0197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:58.680 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.679+0000 7f8ae5186700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae010c2f0 con 0x7f8ae0106b70 2024-03-28T17:38:58.680 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.680+0000 7f8aded9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ae0106b70 0x7f8ae0197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:58.680 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.680+0000 7f8aded9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ae0106b70 0x7f8ae0197780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.151:52288/0 (socket says 172.21.15.151:52288) 2024-03-28T17:38:58.680 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.680+0000 7f8aded9d700 1 -- 172.21.15.151:0/2232425868 learned_addr learned my addr 172.21.15.151:0/2232425868 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-03-28T17:38:58.681 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.680+0000 7f8aded9d700 1 -- 172.21.15.151:0/2232425868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae0197cc0 con 0x7f8ae0106b70 2024-03-28T17:38:58.681 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.681+0000 7f8aded9d700 1 --2- 172.21.15.151:0/2232425868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ae0106b70 0x7f8ae0197780 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4005850 tx=0x7f8ad4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:58.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.681+0000 7f8ac7fff700 1 -- 172.21.15.151:0/2232425868 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ad4026070 con 0x7f8ae0106b70 2024-03-28T17:38:58.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.681+0000 7f8ae5186700 1 -- 172.21.15.151:0/2232425868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae0197f50 con 0x7f8ae0106b70 2024-03-28T17:38:58.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.681+0000 7f8ae5186700 1 -- 172.21.15.151:0/2232425868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae01983b0 con 0x7f8ae0106b70 2024-03-28T17:38:58.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.681+0000 7f8ac7fff700 1 -- 172.21.15.151:0/2232425868 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ad400bc30 con 0x7f8ae0106b70 2024-03-28T17:38:58.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.681+0000 7f8ac7fff700 1 -- 172.21.15.151:0/2232425868 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ad40039b0 con 0x7f8ae0106b70 2024-03-28T17:38:58.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.682+0000 7f8ae5186700 1 -- 172.21.15.151:0/2232425868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae01033c0 con 0x7f8ae0106b70 2024-03-28T17:38:58.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.683+0000 7f8ac7fff700 1 -- 172.21.15.151:0/2232425868 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f8ad4029070 con 0x7f8ae0106b70 2024-03-28T17:38:58.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.683+0000 7f8ac7fff700 1 --2- 172.21.15.151:0/2232425868 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8ac8038fc0 0x7f8ac803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:38:58.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.683+0000 7f8ac7fff700 1 -- 172.21.15.151:0/2232425868 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8ad401e070 con 0x7f8ae0106b70 2024-03-28T17:38:58.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.684+0000 7f8ade59c700 1 --2- 172.21.15.151:0/2232425868 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8ac8038fc0 0x7f8ac803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:38:58.684 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.684+0000 7f8ade59c700 1 --2- 172.21.15.151:0/2232425868 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8ac8038fc0 0x7f8ac803b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0006fd0 tx=0x7f8ad0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:38:58.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.699+0000 7f8ac7fff700 1 -- 172.21.15.151:0/2232425868 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8ad4013070 con 0x7f8ae0106b70 2024-03-28T17:38:58.906 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.905+0000 7f8ae5186700 1 -- 172.21.15.151:0/2232425868 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi045:172.21.15.45=a;smithi071:172.21.15.71=b;smithi151:172.21.15.151=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ae0198ba0 con 0x7f8ac8038fc0 2024-03-28T17:38:58.921 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.921+0000 7f8ac7fff700 1 -- 172.21.15.151:0/2232425868 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f8ae0198ba0 con 0x7f8ac8038fc0 2024-03-28T17:38:58.921 INFO:teuthology.orchestra.run.smithi151.stdout:Scheduled mon update... 2024-03-28T17:38:58.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.922+0000 7f8ae5186700 1 -- 172.21.15.151:0/2232425868 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8ac8038fc0 msgr2=0x7f8ac803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:58.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.922+0000 7f8ae5186700 1 --2- 172.21.15.151:0/2232425868 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8ac8038fc0 0x7f8ac803b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0006fd0 tx=0x7f8ad0006e40 comp rx=0 tx=0).stop 2024-03-28T17:38:58.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.922+0000 7f8ae5186700 1 -- 172.21.15.151:0/2232425868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ae0106b70 msgr2=0x7f8ae0197780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:38:58.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.922+0000 7f8ae5186700 1 --2- 172.21.15.151:0/2232425868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ae0106b70 0x7f8ae0197780 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4005850 tx=0x7f8ad4005030 comp rx=0 tx=0).stop 2024-03-28T17:38:58.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.922+0000 7f8ae5186700 1 -- 172.21.15.151:0/2232425868 shutdown_connections 2024-03-28T17:38:58.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.922+0000 7f8ae5186700 1 --2- 172.21.15.151:0/2232425868 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8ac8038fc0 0x7f8ac803b480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:58.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.922+0000 7f8ae5186700 1 --2- 172.21.15.151:0/2232425868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ae0106b70 0x7f8ae0197780 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:38:58.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.922+0000 7f8ae5186700 1 -- 172.21.15.151:0/2232425868 >> 172.21.15.151:0/2232425868 conn(0x7f8ae00fbce0 msgr2=0x7f8ae01095d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:38:58.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.922+0000 7f8ae5186700 1 -- 172.21.15.151:0/2232425868 shutdown_connections 2024-03-28T17:38:58.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:38:58.922+0000 7f8ae5186700 1 -- 172.21.15.151:0/2232425868 wait complete. 2024-03-28T17:38:59.573 DEBUG:teuthology.orchestra.run.smithi071:mon.b> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mon.b.service 2024-03-28T17:38:59.576 DEBUG:teuthology.orchestra.run.smithi151:mon.c> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mon.c.service 2024-03-28T17:38:59.579 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T17:38:59.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T17:38:59.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:59 smithi045 ceph-mon[31773]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:38:59.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:59.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:38:59.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:38:59.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:38:59.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:38:59.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T17:38:59.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:38:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:00.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:00 smithi045 ceph-mon[31773]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi045:172.21.15.45=a;smithi071:172.21.15.71=b;smithi151:172.21.15.151=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:39:00.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:00 smithi045 ceph-mon[31773]: Saving service mon spec with placement smithi045:172.21.15.45=a;smithi071:172.21.15.71=b;smithi151:172.21.15.151=c;count:3 2024-03-28T17:39:00.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:00 smithi045 ceph-mon[31773]: Deploying daemon mon.c on smithi151 2024-03-28T17:39:00.908 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:39:01.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:01 smithi045 ceph-mon[31773]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:03.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:03 smithi045 ceph-mon[31773]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:03.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.688+0000 7fc3f19f3700 1 -- 172.21.15.151:0/3851296928 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3ec106a90 msgr2=0x7fc3ec106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:39:03.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.688+0000 7fc3f19f3700 1 --2- 172.21.15.151:0/3851296928 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3ec106a90 0x7fc3ec106e70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc009910 tx=0x7fc3dc009c20 comp rx=0 tx=0).stop 2024-03-28T17:39:03.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.688+0000 7fc3f19f3700 1 -- 172.21.15.151:0/3851296928 shutdown_connections 2024-03-28T17:39:03.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.688+0000 7fc3f19f3700 1 --2- 172.21.15.151:0/3851296928 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3ec106a90 0x7fc3ec106e70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:03.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.688+0000 7fc3f19f3700 1 -- 172.21.15.151:0/3851296928 >> 172.21.15.151:0/3851296928 conn(0x7fc3ec0fc5e0 msgr2=0x7fc3ec0fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:39:03.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.688+0000 7fc3f19f3700 1 -- 172.21.15.151:0/3851296928 shutdown_connections 2024-03-28T17:39:03.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.688+0000 7fc3f19f3700 1 -- 172.21.15.151:0/3851296928 wait complete. 2024-03-28T17:39:03.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.688+0000 7fc3f19f3700 1 Processor -- start 2024-03-28T17:39:03.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.688+0000 7fc3f19f3700 1 -- start start 2024-03-28T17:39:03.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.689+0000 7fc3f19f3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3ec106a90 0x7fc3ec196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:39:03.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.689+0000 7fc3f19f3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec10b0d0 con 0x7fc3ec106a90 2024-03-28T17:39:03.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.689+0000 7fc3eaffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3ec106a90 0x7fc3ec196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:39:03.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.689+0000 7fc3eaffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3ec106a90 0x7fc3ec196320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.151:52296/0 (socket says 172.21.15.151:52296) 2024-03-28T17:39:03.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.689+0000 7fc3eaffd700 1 -- 172.21.15.151:0/178064672 learned_addr learned my addr 172.21.15.151:0/178064672 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-03-28T17:39:03.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.690+0000 7fc3eaffd700 1 -- 172.21.15.151:0/178064672 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ec06ada0 con 0x7fc3ec106a90 2024-03-28T17:39:03.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.690+0000 7fc3eaffd700 1 --2- 172.21.15.151:0/178064672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3ec106a90 0x7fc3ec196320 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc005850 tx=0x7fc3dc005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:39:03.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.691+0000 7fc3f09f1700 1 -- 172.21.15.151:0/178064672 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc3dc026070 con 0x7fc3ec106a90 2024-03-28T17:39:03.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.691+0000 7fc3f09f1700 1 -- 172.21.15.151:0/178064672 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3dc00bbd0 con 0x7fc3ec106a90 2024-03-28T17:39:03.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.691+0000 7fc3f09f1700 1 -- 172.21.15.151:0/178064672 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc3dc003970 con 0x7fc3ec106a90 2024-03-28T17:39:03.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.691+0000 7fc3f19f3700 1 -- 172.21.15.151:0/178064672 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3ec196860 con 0x7fc3ec106a90 2024-03-28T17:39:03.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.691+0000 7fc3f19f3700 1 -- 172.21.15.151:0/178064672 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3ec196d20 con 0x7fc3ec106a90 2024-03-28T17:39:03.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.692+0000 7fc3f19f3700 1 -- 172.21.15.151:0/178064672 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3ec1021d0 con 0x7fc3ec106a90 2024-03-28T17:39:03.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.692+0000 7fc3f09f1700 1 -- 172.21.15.151:0/178064672 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fc3dc029070 con 0x7fc3ec106a90 2024-03-28T17:39:03.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.692+0000 7fc3f09f1700 1 --2- 172.21.15.151:0/178064672 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc3d4039020 0x7fc3d403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:39:03.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.692+0000 7fc3f09f1700 1 -- 172.21.15.151:0/178064672 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc3dc01e070 con 0x7fc3ec106a90 2024-03-28T17:39:03.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.693+0000 7fc3ea7fc700 1 --2- 172.21.15.151:0/178064672 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc3d4039020 0x7fc3d403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:39:03.695 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.694+0000 7fc3ea7fc700 1 --2- 172.21.15.151:0/178064672 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc3d4039020 0x7fc3d403b4e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0006fd0 tx=0x7fc3e0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:39:03.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:03.708+0000 7fc3f09f1700 1 -- 172.21.15.151:0/178064672 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc3dc013070 con 0x7fc3ec106a90 2024-03-28T17:39:03.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:03 smithi151 systemd[1]: Starting Ceph mon.c for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:39:04.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:04.012+0000 7fc3f19f3700 1 -- 172.21.15.151:0/178064672 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc3ec06a590 con 0x7fc3ec106a90 2024-03-28T17:39:04.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:04.013+0000 7fc3f09f1700 1 -- 172.21.15.151:0/178064672 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+745 (secure 0 0 0) 0x7fc3dc030030 con 0x7fc3ec106a90 2024-03-28T17:39:04.014 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-28T17:39:04.014 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":1,"fsid":"9f4888ec-ed29-11ee-95d0-87774f69a715","modified":"2024-03-28T17:36:47.322376Z","created":"2024-03-28T17:36:47.322376Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-28T17:39:04.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:04.015+0000 7fc3f19f3700 1 -- 172.21.15.151:0/178064672 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc3d4039020 msgr2=0x7fc3d403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:39:04.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:04.015+0000 7fc3f19f3700 1 --2- 172.21.15.151:0/178064672 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc3d4039020 0x7fc3d403b4e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0006fd0 tx=0x7fc3e0006e40 comp rx=0 tx=0).stop 2024-03-28T17:39:04.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:04.015+0000 7fc3f19f3700 1 -- 172.21.15.151:0/178064672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3ec106a90 msgr2=0x7fc3ec196320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:39:04.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:04.015+0000 7fc3f19f3700 1 --2- 172.21.15.151:0/178064672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3ec106a90 0x7fc3ec196320 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc005850 tx=0x7fc3dc005030 comp rx=0 tx=0).stop 2024-03-28T17:39:04.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:04.015+0000 7fc3f19f3700 1 -- 172.21.15.151:0/178064672 shutdown_connections 2024-03-28T17:39:04.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:04.015+0000 7fc3f19f3700 1 --2- 172.21.15.151:0/178064672 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc3d4039020 0x7fc3d403b4e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:04.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:04.015+0000 7fc3f19f3700 1 --2- 172.21.15.151:0/178064672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3ec106a90 0x7fc3ec196320 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:04.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:04.015+0000 7fc3f19f3700 1 -- 172.21.15.151:0/178064672 >> 172.21.15.151:0/178064672 conn(0x7fc3ec0fc5e0 msgr2=0x7fc3ec1085d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:39:04.016 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:04.015+0000 7fc3f19f3700 1 -- 172.21.15.151:0/178064672 shutdown_connections 2024-03-28T17:39:04.017 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:04.015+0000 7fc3f19f3700 1 -- 172.21.15.151:0/178064672 wait complete. 2024-03-28T17:39:04.017 INFO:teuthology.orchestra.run.smithi151.stderr:dumped monmap epoch 1 2024-03-28T17:39:04.255 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:03 smithi151 podman[36447]: 2024-03-28 17:39:03.841525373 +0000 UTC m=+0.019558171 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:39:04.255 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:03 smithi151 podman[36447]: 2024-03-28 17:39:03.943446295 +0000 UTC m=+0.121479082 container create cb1942f6cc8799216e83b61907889a22c7baab7dc8081b943e7d68e666434264 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mon-c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-28T17:39:04.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 podman[36447]: 2024-03-28 17:39:04.255443486 +0000 UTC m=+0.433476288 container init cb1942f6cc8799216e83b61907889a22c7baab7dc8081b943e7d68e666434264 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mon-c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-28T17:39:04.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 podman[36447]: 2024-03-28 17:39:04.258782772 +0000 UTC m=+0.436815559 container start cb1942f6cc8799216e83b61907889a22c7baab7dc8081b943e7d68e666434264 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mon-c, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-28T17:39:04.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: set uid:gid to 167:167 (ceph:ceph) 2024-03-28T17:39:04.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev), process ceph-mon, pid 2 2024-03-28T17:39:04.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: pidfile_write: ignore empty --pid-file 2024-03-28T17:39:04.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: load: jerasure load: lrc 2024-03-28T17:39:04.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: RocksDB version: 7.9.2 2024-03-28T17:39:04.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Git sha 0 2024-03-28T17:39:04.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-28T17:39:04.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: DB SUMMARY 2024-03-28T17:39:04.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: DB Session ID: H6EP7E0CTBCEXMTRVOYJ 2024-03-28T17:39:04.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: CURRENT file: CURRENT 2024-03-28T17:39:04.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: IDENTITY file: IDENTITY 2024-03-28T17:39:04.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-28T17:39:04.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-28T17:39:04.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-28T17:39:04.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.error_if_exists: 0 2024-03-28T17:39:04.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.create_if_missing: 0 2024-03-28T17:39:04.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.paranoid_checks: 1 2024-03-28T17:39:04.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-28T17:39:04.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-28T17:39:04.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-28T17:39:04.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.env: 0x563faddecb20 2024-03-28T17:39:04.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.fs: PosixFileSystem 2024-03-28T17:39:04.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.info_log: 0x563faf387920 2024-03-28T17:39:04.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_file_opening_threads: 16 2024-03-28T17:39:04.577 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.statistics: (nil) 2024-03-28T17:39:04.577 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.use_fsync: 0 2024-03-28T17:39:04.577 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_log_file_size: 0 2024-03-28T17:39:04.577 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-28T17:39:04.578 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-28T17:39:04.578 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.keep_log_file_num: 1000 2024-03-28T17:39:04.578 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.recycle_log_file_num: 0 2024-03-28T17:39:04.578 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.allow_fallocate: 1 2024-03-28T17:39:04.578 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.allow_mmap_reads: 0 2024-03-28T17:39:04.578 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.allow_mmap_writes: 0 2024-03-28T17:39:04.578 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.use_direct_reads: 0 2024-03-28T17:39:04.578 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-28T17:39:04.578 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.create_missing_column_families: 0 2024-03-28T17:39:04.578 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.db_log_dir: 2024-03-28T17:39:04.578 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.wal_dir: 2024-03-28T17:39:04.578 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-28T17:39:04.578 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-28T17:39:04.578 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-28T17:39:04.579 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-28T17:39:04.579 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-28T17:39:04.579 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-28T17:39:04.579 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.advise_random_on_open: 1 2024-03-28T17:39:04.579 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.db_write_buffer_size: 0 2024-03-28T17:39:04.579 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.write_buffer_manager: 0x563faf35a1e0 2024-03-28T17:39:04.579 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-28T17:39:04.579 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-28T17:39:04.579 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-28T17:39:04.579 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.rate_limiter: (nil) 2024-03-28T17:39:04.579 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-28T17:39:04.579 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.wal_recovery_mode: 2 2024-03-28T17:39:04.580 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.enable_thread_tracking: 0 2024-03-28T17:39:04.580 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.enable_pipelined_write: 0 2024-03-28T17:39:04.580 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.unordered_write: 0 2024-03-28T17:39:04.580 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-28T17:39:04.580 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-28T17:39:04.580 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-28T17:39:04.580 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-28T17:39:04.580 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.row_cache: None 2024-03-28T17:39:04.580 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.wal_filter: None 2024-03-28T17:39:04.580 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-28T17:39:04.580 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.allow_ingest_behind: 0 2024-03-28T17:39:04.580 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.two_write_queues: 0 2024-03-28T17:39:04.580 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.manual_wal_flush: 0 2024-03-28T17:39:04.580 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.wal_compression: 0 2024-03-28T17:39:04.580 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.atomic_flush: 0 2024-03-28T17:39:04.580 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-28T17:39:04.581 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-28T17:39:04.581 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-28T17:39:04.581 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.log_readahead_size: 0 2024-03-28T17:39:04.581 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-28T17:39:04.581 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.best_efforts_recovery: 0 2024-03-28T17:39:04.581 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-28T17:39:04.581 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-28T17:39:04.581 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.allow_data_in_errors: 0 2024-03-28T17:39:04.581 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.db_host_id: __hostname__ 2024-03-28T17:39:04.581 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-28T17:39:04.581 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_background_jobs: 2 2024-03-28T17:39:04.581 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_background_compactions: -1 2024-03-28T17:39:04.581 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_subcompactions: 1 2024-03-28T17:39:04.581 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-28T17:39:04.581 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-28T17:39:04.581 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-28T17:39:04.582 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_total_wal_size: 0 2024-03-28T17:39:04.582 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-28T17:39:04.582 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-28T17:39:04.582 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-28T17:39:04.582 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-28T17:39:04.582 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_open_files: -1 2024-03-28T17:39:04.582 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.bytes_per_sync: 0 2024-03-28T17:39:04.582 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-28T17:39:04.582 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-28T17:39:04.582 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.compaction_readahead_size: 0 2024-03-28T17:39:04.582 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_background_flushes: -1 2024-03-28T17:39:04.582 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Compression algorithms supported: 2024-03-28T17:39:04.582 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-28T17:39:04.582 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: kZSTD supported: 0 2024-03-28T17:39:04.582 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: kXpressCompression supported: 0 2024-03-28T17:39:04.583 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: kLZ4HCCompression supported: 1 2024-03-28T17:39:04.583 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: kZlibCompression supported: 1 2024-03-28T17:39:04.583 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: kSnappyCompression supported: 1 2024-03-28T17:39:04.583 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: kLZ4Compression supported: 1 2024-03-28T17:39:04.583 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: kBZip2Compression supported: 0 2024-03-28T17:39:04.583 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-28T17:39:04.583 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-28T17:39:04.583 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-28T17:39:04.583 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-28T17:39:04.583 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-28T17:39:04.583 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.merge_operator: 2024-03-28T17:39:04.583 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.compaction_filter: None 2024-03-28T17:39:04.583 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.compaction_filter_factory: None 2024-03-28T17:39:04.584 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.sst_partitioner_factory: None 2024-03-28T17:39:04.584 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-28T17:39:04.584 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-28T17:39:04.584 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563faf387a20) 2024-03-28T17:39:04.584 INFO:journalctl@ceph.mon.c.smithi151.stdout: cache_index_and_filter_blocks: 1 2024-03-28T17:39:04.584 INFO:journalctl@ceph.mon.c.smithi151.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-28T17:39:04.584 INFO:journalctl@ceph.mon.c.smithi151.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-28T17:39:04.584 INFO:journalctl@ceph.mon.c.smithi151.stdout: pin_top_level_index_and_filter: 1 2024-03-28T17:39:04.584 INFO:journalctl@ceph.mon.c.smithi151.stdout: index_type: 0 2024-03-28T17:39:04.584 INFO:journalctl@ceph.mon.c.smithi151.stdout: data_block_index_type: 0 2024-03-28T17:39:04.584 INFO:journalctl@ceph.mon.c.smithi151.stdout: index_shortening: 1 2024-03-28T17:39:04.584 INFO:journalctl@ceph.mon.c.smithi151.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-28T17:39:04.584 INFO:journalctl@ceph.mon.c.smithi151.stdout: checksum: 4 2024-03-28T17:39:04.584 INFO:journalctl@ceph.mon.c.smithi151.stdout: no_block_cache: 0 2024-03-28T17:39:04.585 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_cache: 0x563faf38d090 2024-03-28T17:39:04.585 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_cache_name: BinnedLRUCache 2024-03-28T17:39:04.585 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_cache_options: 2024-03-28T17:39:04.585 INFO:journalctl@ceph.mon.c.smithi151.stdout: capacity : 536870912 2024-03-28T17:39:04.585 INFO:journalctl@ceph.mon.c.smithi151.stdout: num_shard_bits : 4 2024-03-28T17:39:04.585 INFO:journalctl@ceph.mon.c.smithi151.stdout: strict_capacity_limit : 0 2024-03-28T17:39:04.585 INFO:journalctl@ceph.mon.c.smithi151.stdout: high_pri_pool_ratio: 0.000 2024-03-28T17:39:04.585 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_cache_compressed: (nil) 2024-03-28T17:39:04.585 INFO:journalctl@ceph.mon.c.smithi151.stdout: persistent_cache: (nil) 2024-03-28T17:39:04.585 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_size: 4096 2024-03-28T17:39:04.585 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_size_deviation: 10 2024-03-28T17:39:04.585 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_restart_interval: 16 2024-03-28T17:39:04.585 INFO:journalctl@ceph.mon.c.smithi151.stdout: index_block_restart_interval: 1 2024-03-28T17:39:04.585 INFO:journalctl@ceph.mon.c.smithi151.stdout: metadata_block_size: 4096 2024-03-28T17:39:04.585 INFO:journalctl@ceph.mon.c.smithi151.stdout: partition_filters: 0 2024-03-28T17:39:04.585 INFO:journalctl@ceph.mon.c.smithi151.stdout: use_delta_encoding: 1 2024-03-28T17:39:04.586 INFO:journalctl@ceph.mon.c.smithi151.stdout: filter_policy: bloomfilter 2024-03-28T17:39:04.586 INFO:journalctl@ceph.mon.c.smithi151.stdout: whole_key_filtering: 1 2024-03-28T17:39:04.586 INFO:journalctl@ceph.mon.c.smithi151.stdout: verify_compression: 0 2024-03-28T17:39:04.586 INFO:journalctl@ceph.mon.c.smithi151.stdout: read_amp_bytes_per_bit: 0 2024-03-28T17:39:04.586 INFO:journalctl@ceph.mon.c.smithi151.stdout: format_version: 5 2024-03-28T17:39:04.586 INFO:journalctl@ceph.mon.c.smithi151.stdout: enable_index_compression: 1 2024-03-28T17:39:04.586 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_align: 0 2024-03-28T17:39:04.586 INFO:journalctl@ceph.mon.c.smithi151.stdout: max_auto_readahead_size: 262144 2024-03-28T17:39:04.586 INFO:journalctl@ceph.mon.c.smithi151.stdout: prepopulate_block_cache: 0 2024-03-28T17:39:04.586 INFO:journalctl@ceph.mon.c.smithi151.stdout: initial_auto_readahead_size: 8192 2024-03-28T17:39:04.586 INFO:journalctl@ceph.mon.c.smithi151.stdout: num_file_reads_for_auto_readahead: 2 2024-03-28T17:39:04.586 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.write_buffer_size: 33554432 2024-03-28T17:39:04.586 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_write_buffer_number: 2 2024-03-28T17:39:04.586 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.compression: NoCompression 2024-03-28T17:39:04.586 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.bottommost_compression: Disabled 2024-03-28T17:39:04.587 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.prefix_extractor: nullptr 2024-03-28T17:39:04.587 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-28T17:39:04.587 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.num_levels: 7 2024-03-28T17:39:04.587 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-28T17:39:04.587 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-28T17:39:04.587 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-28T17:39:04.587 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-28T17:39:04.587 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-28T17:39:04.587 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-28T17:39:04.587 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-28T17:39:04.587 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-28T17:39:04.587 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-28T17:39:04.587 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-28T17:39:04.587 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-28T17:39:04.587 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-28T17:39:04.587 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-28T17:39:04.587 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.compression_opts.level: 32767 2024-03-28T17:39:04.588 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.compression_opts.strategy: 0 2024-03-28T17:39:04.588 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-28T17:39:04.588 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-28T17:39:04.588 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-28T17:39:04.588 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-28T17:39:04.588 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.compression_opts.enabled: false 2024-03-28T17:39:04.588 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-28T17:39:04.588 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-28T17:39:04.588 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-28T17:39:04.588 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-28T17:39:04.588 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.target_file_size_base: 67108864 2024-03-28T17:39:04.588 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-28T17:39:04.588 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-28T17:39:04.588 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-28T17:39:04.588 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-28T17:39:04.589 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-28T17:39:04.589 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-28T17:39:04.589 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-28T17:39:04.589 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-28T17:39:04.589 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-28T17:39:04.589 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-28T17:39:04.589 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-28T17:39:04.589 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-28T17:39:04.589 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-28T17:39:04.589 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-28T17:39:04.589 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.arena_block_size: 1048576 2024-03-28T17:39:04.589 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-28T17:39:04.589 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-28T17:39:04.589 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.disable_auto_compactions: 0 2024-03-28T17:39:04.589 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-28T17:39:04.589 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-28T17:39:04.590 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-28T17:39:04.590 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-28T17:39:04.590 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-28T17:39:04.590 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-28T17:39:04.590 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-28T17:39:04.590 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-28T17:39:04.590 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-28T17:39:04.590 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-28T17:39:04.590 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.table_properties_collectors: 2024-03-28T17:39:04.590 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.inplace_update_support: 0 2024-03-28T17:39:04.590 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-28T17:39:04.590 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-28T17:39:04.590 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-28T17:39:04.590 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-28T17:39:04.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.bloom_locality: 0 2024-03-28T17:39:04.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.max_successive_merges: 0 2024-03-28T17:39:04.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-28T17:39:04.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.paranoid_file_checks: 0 2024-03-28T17:39:04.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.force_consistency_checks: 1 2024-03-28T17:39:04.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.report_bg_io_stats: 0 2024-03-28T17:39:04.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.ttl: 2592000 2024-03-28T17:39:04.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-28T17:39:04.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-28T17:39:04.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-28T17:39:04.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.enable_blob_files: false 2024-03-28T17:39:04.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.min_blob_size: 0 2024-03-28T17:39:04.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.blob_file_size: 268435456 2024-03-28T17:39:04.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-28T17:39:04.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-28T17:39:04.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-28T17:39:04.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-28T17:39:04.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-28T17:39:04.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.blob_file_starting_level: 0 2024-03-28T17:39:04.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-28T17:39:04.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-28T17:39:04.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-28T17:39:04.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ac993ecd-3d1f-4bec-b9e0-df0fe47c5ead 2024-03-28T17:39:04.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711647544288333, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-28T17:39:04.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-28T17:39:04.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711647544288735, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711647544, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ac993ecd-3d1f-4bec-b9e0-df0fe47c5ead", "db_session_id": "H6EP7E0CTBCEXMTRVOYJ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-28T17:39:04.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711647544288815, "job": 1, "event": "recovery_finished"} 2024-03-28T17:39:04.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-28T17:39:04.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-28T17:39:04.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563faf480000 2024-03-28T17:39:04.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: DB pointer 0x563faf46a000 2024-03-28T17:39:04.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-28T17:39:04.593 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-28T17:39:04.593 INFO:journalctl@ceph.mon.c.smithi151.stdout: ** DB Stats ** 2024-03-28T17:39:04.593 INFO:journalctl@ceph.mon.c.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T17:39:04.593 INFO:journalctl@ceph.mon.c.smithi151.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-28T17:39:04.593 INFO:journalctl@ceph.mon.c.smithi151.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T17:39:04.593 INFO:journalctl@ceph.mon.c.smithi151.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T17:39:04.593 INFO:journalctl@ceph.mon.c.smithi151.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-28T17:39:04.593 INFO:journalctl@ceph.mon.c.smithi151.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T17:39:04.593 INFO:journalctl@ceph.mon.c.smithi151.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T17:39:04.593 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2024-03-28T17:39:04.593 INFO:journalctl@ceph.mon.c.smithi151.stdout: ** Compaction Stats [default] ** 2024-03-28T17:39:04.593 INFO:journalctl@ceph.mon.c.smithi151.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T17:39:04.593 INFO:journalctl@ceph.mon.c.smithi151.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-28T17:39:04.593 INFO:journalctl@ceph.mon.c.smithi151.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T17:39:04.594 INFO:journalctl@ceph.mon.c.smithi151.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T17:39:04.594 INFO:journalctl@ceph.mon.c.smithi151.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T17:39:04.594 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2024-03-28T17:39:04.594 INFO:journalctl@ceph.mon.c.smithi151.stdout: ** Compaction Stats [default] ** 2024-03-28T17:39:04.594 INFO:journalctl@ceph.mon.c.smithi151.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T17:39:04.594 INFO:journalctl@ceph.mon.c.smithi151.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-28T17:39:04.594 INFO:journalctl@ceph.mon.c.smithi151.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T17:39:04.595 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2024-03-28T17:39:04.595 INFO:journalctl@ceph.mon.c.smithi151.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-28T17:39:04.595 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2024-03-28T17:39:04.595 INFO:journalctl@ceph.mon.c.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T17:39:04.595 INFO:journalctl@ceph.mon.c.smithi151.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-28T17:39:04.595 INFO:journalctl@ceph.mon.c.smithi151.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-28T17:39:04.595 INFO:journalctl@ceph.mon.c.smithi151.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-28T17:39:04.595 INFO:journalctl@ceph.mon.c.smithi151.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-28T17:39:04.595 INFO:journalctl@ceph.mon.c.smithi151.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-28T17:39:04.595 INFO:journalctl@ceph.mon.c.smithi151.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T17:39:04.595 INFO:journalctl@ceph.mon.c.smithi151.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T17:39:04.595 INFO:journalctl@ceph.mon.c.smithi151.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-28T17:39:04.595 INFO:journalctl@ceph.mon.c.smithi151.stdout: Block cache BinnedLRUCache@0x563faf38d090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-28T17:39:04.595 INFO:journalctl@ceph.mon.c.smithi151.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-28T17:39:04.595 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2024-03-28T17:39:04.595 INFO:journalctl@ceph.mon.c.smithi151.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-28T17:39:04.596 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-28T17:39:04.596 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: using public_addr v2:172.21.15.151:0/0 -> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2024-03-28T17:39:04.596 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: starting mon.c rank -1 at public addrs [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] at bind addrs [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 2024-03-28T17:39:04.596 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: mon.c@-1(???) e0 preinit fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 2024-03-28T17:39:04.596 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: mon.c@-1(synchronizing).mds e1 new map 2024-03-28T17:39:04.596 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-28T17:39:04.596 INFO:journalctl@ceph.mon.c.smithi151.stdout: e1 2024-03-28T17:39:04.596 INFO:journalctl@ceph.mon.c.smithi151.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T17:39:04.596 INFO:journalctl@ceph.mon.c.smithi151.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T17:39:04.596 INFO:journalctl@ceph.mon.c.smithi151.stdout: legacy client fscid: -1 2024-03-28T17:39:04.596 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2024-03-28T17:39:04.596 INFO:journalctl@ceph.mon.c.smithi151.stdout: No filesystems configured 2024-03-28T17:39:04.596 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-28T17:39:04.596 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-28T17:39:04.596 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-28T17:39:04.596 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-28T17:39:04.597 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-28T17:39:04.597 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-28T17:39:04.597 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-28T17:39:04.597 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T17:39:04.597 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T17:39:04.597 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T17:39:04.597 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:04.597 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi071", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:39:04.597 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:04.597 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: Deploying cephadm binary to smithi071 2024-03-28T17:39:04.597 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:04.597 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:04.597 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.597 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: Added host smithi071 2024-03-28T17:39:04.598 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:39:04.598 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.598 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.598 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:04.598 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.598 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:04.598 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:39:04.598 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:04.598 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.598 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.598 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.598 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.598 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi071", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:39:04.598 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:04.599 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:39:04.599 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: Updating smithi071:/etc/ceph/ceph.conf 2024-03-28T17:39:04.599 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: Updating smithi071:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:39:04.599 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: Updating smithi071:/etc/ceph/ceph.client.admin.keyring 2024-03-28T17:39:04.599 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: Updating smithi071:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T17:39:04.599 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi151", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:39:04.599 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.599 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.599 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.599 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:04.599 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: Deploying cephadm binary to smithi151 2024-03-28T17:39:04.599 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:04.599 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:04.599 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.599 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: Added host smithi151 2024-03-28T17:39:04.599 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:39:04.600 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.600 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.600 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:04.600 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.600 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:04.600 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:39:04.600 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:04.600 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.600 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.600 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.600 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.600 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:39:04.600 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:04.600 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:39:04.600 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-28T17:39:04.600 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: Updating smithi151:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:39:04.600 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-03-28T17:39:04.601 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: Updating smithi151:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T17:39:04.601 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1282791936' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-28T17:39:04.601 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.601 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.601 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.601 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:04.601 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1282791936' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-28T17:39:04.601 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T17:39:04.601 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:04.601 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.601 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:39:04.601 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:04.601 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:39:04.601 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:04.601 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T17:39:04.601 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:04.602 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi045:172.21.15.45=a;smithi071:172.21.15.71=b;smithi151:172.21.15.151=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:39:04.602 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: Saving service mon spec with placement smithi045:172.21.15.45=a;smithi071:172.21.15.71=b;smithi151:172.21.15.151=c;count:3 2024-03-28T17:39:04.602 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: Deploying daemon mon.c on smithi151 2024-03-28T17:39:04.602 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:04.602 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:04.602 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: from='client.? 172.21.15.151:0/178064672' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T17:39:04.602 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 ceph-mon[36481]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-28T17:39:04.602 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 bash[36447]: cb1942f6cc8799216e83b61907889a22c7baab7dc8081b943e7d68e666434264 2024-03-28T17:39:04.602 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:04 smithi151 systemd[1]: Started Ceph mon.c for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:39:04.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:04 smithi045 ceph-mon[31773]: from='client.? 172.21.15.151:0/178064672' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T17:39:05.792 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T17:39:05.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T17:39:06.030 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T17:39:09.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:09 smithi045 ceph-mon[31773]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:09.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T17:39:09.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:09 smithi045 ceph-mon[31773]: mon.a calling monitor election 2024-03-28T17:39:09.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:09.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:09.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:09 smithi045 ceph-mon[31773]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:09.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:09.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:09 smithi045 ceph-mon[31773]: mon.c calling monitor election 2024-03-28T17:39:09.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:09.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:09 smithi045 ceph-mon[31773]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:09.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:09.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:09.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:09 smithi045 ceph-mon[31773]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T17:39:09.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:09 smithi045 ceph-mon[31773]: monmap e2: 2 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],c=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T17:39:09.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:09 smithi045 ceph-mon[31773]: fsmap 2024-03-28T17:39:09.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:09 smithi045 ceph-mon[31773]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T17:39:09.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:09 smithi045 ceph-mon[31773]: mgrmap e13: x(active, since 61s) 2024-03-28T17:39:09.669 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:09 smithi045 ceph-mon[31773]: overall HEALTH_OK 2024-03-28T17:39:09.669 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:09.669 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:09.669 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:09.669 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T17:39:09.669 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:09.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:09 smithi151 ceph-mon[36481]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:09.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T17:39:09.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:09 smithi151 ceph-mon[36481]: mon.a calling monitor election 2024-03-28T17:39:09.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:09.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:09.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:09 smithi151 ceph-mon[36481]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:09.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:09.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:09 smithi151 ceph-mon[36481]: mon.c calling monitor election 2024-03-28T17:39:09.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:09.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:09 smithi151 ceph-mon[36481]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:09.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:09.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:09.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:09 smithi151 ceph-mon[36481]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T17:39:09.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:09 smithi151 ceph-mon[36481]: monmap e2: 2 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],c=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T17:39:09.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:09 smithi151 ceph-mon[36481]: fsmap 2024-03-28T17:39:09.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:09 smithi151 ceph-mon[36481]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T17:39:09.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:09 smithi151 ceph-mon[36481]: mgrmap e13: x(active, since 61s) 2024-03-28T17:39:09.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:09 smithi151 ceph-mon[36481]: overall HEALTH_OK 2024-03-28T17:39:09.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:09.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:09.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:09.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T17:39:09.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:10.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.333+0000 7fae99c44700 1 -- 172.21.15.151:0/544685207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae7c0035c0 msgr2=0x7fae7c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:39:10.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.333+0000 7fae99c44700 1 --2- 172.21.15.151:0/544685207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae7c0035c0 0x7fae7c005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fae94100ad0 tx=0x7fae8400feb0 comp rx=0 tx=0).stop 2024-03-28T17:39:10.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.333+0000 7fae99c44700 1 -- 172.21.15.151:0/544685207 shutdown_connections 2024-03-28T17:39:10.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.333+0000 7fae99c44700 1 --2- 172.21.15.151:0/544685207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae7c0035c0 0x7fae7c005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:10.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.333+0000 7fae99c44700 1 -- 172.21.15.151:0/544685207 >> 172.21.15.151:0/544685207 conn(0x7fae940fc6b0 msgr2=0x7fae940fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:39:10.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.333+0000 7fae99c44700 1 -- 172.21.15.151:0/544685207 shutdown_connections 2024-03-28T17:39:10.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.333+0000 7fae99c44700 1 -- 172.21.15.151:0/544685207 wait complete. 2024-03-28T17:39:10.335 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.334+0000 7fae99c44700 1 Processor -- start 2024-03-28T17:39:10.336 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.335+0000 7fae99c44700 1 -- start start 2024-03-28T17:39:10.337 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.335+0000 7fae99c44700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae7c0035c0 0x7fae941a0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:39:10.337 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.335+0000 7fae99c44700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fae941a0580 0x7fae941a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:39:10.337 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.335+0000 7fae99c44700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9406bea0 con 0x7fae7c0035c0 2024-03-28T17:39:10.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.335+0000 7fae99c44700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9406bd20 con 0x7fae941a0580 2024-03-28T17:39:10.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.335+0000 7fae92ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fae941a0580 0x7fae941a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:39:10.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.335+0000 7fae92ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fae941a0580 0x7fae941a48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53794/0 (socket says 172.21.15.151:53794) 2024-03-28T17:39:10.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.335+0000 7fae92ffd700 1 -- 172.21.15.151:0/2712905098 learned_addr learned my addr 172.21.15.151:0/2712905098 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-03-28T17:39:10.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.335+0000 7fae92ffd700 1 -- 172.21.15.151:0/2712905098 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fae941a0580 msgr2=0x7fae941a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-28T17:39:10.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.335+0000 7fae92ffd700 1 -- 172.21.15.151:0/2712905098 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fae941a0580 msgr2=0x7fae941a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T17:39:10.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.335+0000 7fae92ffd700 1 --2- 172.21.15.151:0/2712905098 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fae941a0580 0x7fae941a48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T17:39:10.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.335+0000 7fae92ffd700 1 --2- 172.21.15.151:0/2712905098 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fae941a0580 0x7fae941a48c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T17:39:10.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.335+0000 7fae937fe700 1 --2- 172.21.15.151:0/2712905098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae7c0035c0 0x7fae941a0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:39:10.338 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.336+0000 7fae937fe700 1 -- 172.21.15.151:0/2712905098 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fae941a0580 msgr2=0x7fae941a48c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T17:39:10.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.336+0000 7fae937fe700 1 --2- 172.21.15.151:0/2712905098 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fae941a0580 0x7fae941a48c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:10.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.336+0000 7fae937fe700 1 -- 172.21.15.151:0/2712905098 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae941a0bc0 con 0x7fae7c0035c0 2024-03-28T17:39:10.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.336+0000 7fae937fe700 1 --2- 172.21.15.151:0/2712905098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae7c0035c0 0x7fae941a0040 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fae94100ad0 tx=0x7fae840055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:39:10.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.337+0000 7fae90ff9700 1 -- 172.21.15.151:0/2712905098 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fae84004b60 con 0x7fae7c0035c0 2024-03-28T17:39:10.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.337+0000 7fae99c44700 1 -- 172.21.15.151:0/2712905098 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae941a4e00 con 0x7fae7c0035c0 2024-03-28T17:39:10.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.337+0000 7fae99c44700 1 -- 172.21.15.151:0/2712905098 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae941a5360 con 0x7fae7c0035c0 2024-03-28T17:39:10.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.337+0000 7fae90ff9700 1 -- 172.21.15.151:0/2712905098 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae84013e40 con 0x7fae7c0035c0 2024-03-28T17:39:10.339 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.337+0000 7fae90ff9700 1 -- 172.21.15.151:0/2712905098 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fae840119b0 con 0x7fae7c0035c0 2024-03-28T17:39:10.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.338+0000 7fae99c44700 1 -- 172.21.15.151:0/2712905098 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae941022e0 con 0x7fae7c0035c0 2024-03-28T17:39:10.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.338+0000 7fae90ff9700 1 -- 172.21.15.151:0/2712905098 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fae84023070 con 0x7fae7c0035c0 2024-03-28T17:39:10.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.339+0000 7fae90ff9700 1 --2- 172.21.15.151:0/2712905098 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fae80038f00 0x7fae8003b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:39:10.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.339+0000 7fae90ff9700 1 -- 172.21.15.151:0/2712905098 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fae84020050 con 0x7fae7c0035c0 2024-03-28T17:39:10.340 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.339+0000 7fae92ffd700 1 --2- 172.21.15.151:0/2712905098 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fae80038f00 0x7fae8003b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:39:10.341 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.340+0000 7fae92ffd700 1 --2- 172.21.15.151:0/2712905098 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fae80038f00 0x7fae8003b3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fae88009180 tx=0x7fae88008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:39:10.354 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.353+0000 7fae90ff9700 1 -- 172.21.15.151:0/2712905098 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fae84028020 con 0x7fae7c0035c0 2024-03-28T17:39:10.635 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.635+0000 7fae99c44700 1 -- 172.21.15.151:0/2712905098 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae941a5610 con 0x7fae7c0035c0 2024-03-28T17:39:10.635 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:10 smithi151 ceph-mon[36481]: Deploying daemon mon.b on smithi071 2024-03-28T17:39:10.635 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:10 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:10.636 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.635+0000 7fae90ff9700 1 -- 172.21.15.151:0/2712905098 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1015 (secure 0 0 0) 0x7fae8400f8b0 con 0x7fae7c0035c0 2024-03-28T17:39:10.636 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-28T17:39:10.636 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":2,"fsid":"9f4888ec-ed29-11ee-95d0-87774f69a715","modified":"2024-03-28T17:39:04.322453Z","created":"2024-03-28T17:36:47.322376Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-28T17:39:10.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.638+0000 7fae99c44700 1 -- 172.21.15.151:0/2712905098 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fae80038f00 msgr2=0x7fae8003b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:39:10.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.638+0000 7fae99c44700 1 --2- 172.21.15.151:0/2712905098 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fae80038f00 0x7fae8003b3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fae88009180 tx=0x7fae88008040 comp rx=0 tx=0).stop 2024-03-28T17:39:10.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.638+0000 7fae99c44700 1 -- 172.21.15.151:0/2712905098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae7c0035c0 msgr2=0x7fae941a0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:39:10.638 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.638+0000 7fae99c44700 1 --2- 172.21.15.151:0/2712905098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae7c0035c0 0x7fae941a0040 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fae94100ad0 tx=0x7fae840055a0 comp rx=0 tx=0).stop 2024-03-28T17:39:10.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.638+0000 7fae99c44700 1 -- 172.21.15.151:0/2712905098 shutdown_connections 2024-03-28T17:39:10.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.638+0000 7fae99c44700 1 --2- 172.21.15.151:0/2712905098 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fae80038f00 0x7fae8003b3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:10.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.638+0000 7fae99c44700 1 --2- 172.21.15.151:0/2712905098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae7c0035c0 0x7fae941a0040 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:10.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.638+0000 7fae99c44700 1 --2- 172.21.15.151:0/2712905098 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fae941a0580 0x7fae941a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:10.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.638+0000 7fae99c44700 1 -- 172.21.15.151:0/2712905098 >> 172.21.15.151:0/2712905098 conn(0x7fae940fc6b0 msgr2=0x7fae940fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:39:10.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.638+0000 7fae99c44700 1 -- 172.21.15.151:0/2712905098 shutdown_connections 2024-03-28T17:39:10.639 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:10.638+0000 7fae99c44700 1 -- 172.21.15.151:0/2712905098 wait complete. 2024-03-28T17:39:10.639 INFO:teuthology.orchestra.run.smithi151.stderr:dumped monmap epoch 2 2024-03-28T17:39:10.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:10 smithi045 ceph-mon[31773]: Deploying daemon mon.b on smithi071 2024-03-28T17:39:10.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:10 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:11 smithi151 ceph-mon[36481]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:11 smithi151 ceph-mon[36481]: from='client.? 172.21.15.151:0/2712905098' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T17:39:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:11 smithi045 ceph-mon[31773]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:11 smithi045 ceph-mon[31773]: from='client.? 172.21.15.151:0/2712905098' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T17:39:12.225 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T17:39:12.226 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T17:39:12.463 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T17:39:13.303 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: mon.b@-1(synchronizing).mds e1 new map 2024-03-28T17:39:13.303 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-28T17:39:13.303 INFO:journalctl@ceph.mon.b.smithi071.stdout: e1 2024-03-28T17:39:13.303 INFO:journalctl@ceph.mon.b.smithi071.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T17:39:13.304 INFO:journalctl@ceph.mon.b.smithi071.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T17:39:13.304 INFO:journalctl@ceph.mon.b.smithi071.stdout: legacy client fscid: -1 2024-03-28T17:39:13.304 INFO:journalctl@ceph.mon.b.smithi071.stdout: 2024-03-28T17:39:13.304 INFO:journalctl@ceph.mon.b.smithi071.stdout: No filesystems configured 2024-03-28T17:39:13.304 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-28T17:39:13.304 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-28T17:39:13.304 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-28T17:39:13.304 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-28T17:39:13.304 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-28T17:39:13.304 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-28T17:39:13.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-28T17:39:13.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T17:39:13.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T17:39:13.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T17:39:13.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:13.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi071", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:39:13.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:13.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: Deploying cephadm binary to smithi071 2024-03-28T17:39:13.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:13.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:13.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: Added host smithi071 2024-03-28T17:39:13.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:39:13.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:13.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:39:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi071", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:39:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:39:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: Updating smithi071:/etc/ceph/ceph.conf 2024-03-28T17:39:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: Updating smithi071:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:39:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: Updating smithi071:/etc/ceph/ceph.client.admin.keyring 2024-03-28T17:39:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: Updating smithi071:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T17:39:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi151", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:39:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: Deploying cephadm binary to smithi151 2024-03-28T17:39:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: Added host smithi151 2024-03-28T17:39:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:39:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:39:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:13.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:39:13.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:13.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:39:13.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-28T17:39:13.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: Updating smithi151:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:39:13.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-03-28T17:39:13.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: Updating smithi151:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T17:39:13.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1282791936' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-28T17:39:13.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:13.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1282791936' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-28T17:39:13.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T17:39:13.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:13.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:39:13.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:13.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:39:13.567 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.567 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T17:39:13.567 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:13.567 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi045:172.21.15.45=a;smithi071:172.21.15.71=b;smithi151:172.21.15.151=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:39:13.567 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: Saving service mon spec with placement smithi045:172.21.15.45=a;smithi071:172.21.15.71=b;smithi151:172.21.15.151=c;count:3 2024-03-28T17:39:13.567 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: Deploying daemon mon.c on smithi151 2024-03-28T17:39:13.567 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:13.567 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:13.567 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='client.? 172.21.15.151:0/178064672' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T17:39:13.567 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:13.567 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T17:39:13.567 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: mon.a calling monitor election 2024-03-28T17:39:13.567 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:13.567 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:13.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:13.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:13.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: mon.c calling monitor election 2024-03-28T17:39:13.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:13.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:13.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:13.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:13.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T17:39:13.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: monmap e2: 2 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],c=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T17:39:13.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: fsmap 2024-03-28T17:39:13.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T17:39:13.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: mgrmap e13: x(active, since 61s) 2024-03-28T17:39:13.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: overall HEALTH_OK 2024-03-28T17:39:13.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:13.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T17:39:13.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:13.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: Deploying daemon mon.b on smithi071 2024-03-28T17:39:13.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:13.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:13.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: from='client.? 172.21.15.151:0/2712905098' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T17:39:13.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 ceph-mon[36597]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-28T17:39:13.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 bash[36578]: 43886e29daec95f30719da38fbf830c8aff89fb8ad0cdbbdc35425aa79e3a259 2024-03-28T17:39:13.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:13 smithi071 systemd[1]: Started Ceph mon.b for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:39:18.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.398+0000 7fec138db700 1 -- 172.21.15.151:0/677825142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febf80057d0 msgr2=0x7febf8007c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:39:18.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.398+0000 7fec138db700 1 --2- 172.21.15.151:0/677825142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febf80057d0 0x7febf8007c60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fec0c100ac0 tx=0x7fec0800fe60 comp rx=0 tx=0).stop 2024-03-28T17:39:18.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.398+0000 7fec138db700 1 -- 172.21.15.151:0/677825142 shutdown_connections 2024-03-28T17:39:18.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.398+0000 7fec138db700 1 --2- 172.21.15.151:0/677825142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febf80057d0 0x7febf8007c60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:18.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.398+0000 7fec138db700 1 -- 172.21.15.151:0/677825142 >> 172.21.15.151:0/677825142 conn(0x7fec0c0fc6a0 msgr2=0x7fec0c0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:39:18.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.398+0000 7fec138db700 1 -- 172.21.15.151:0/677825142 shutdown_connections 2024-03-28T17:39:18.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.399+0000 7fec138db700 1 -- 172.21.15.151:0/677825142 wait complete. 2024-03-28T17:39:18.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.399+0000 7fec138db700 1 Processor -- start 2024-03-28T17:39:18.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.400+0000 7fec138db700 1 -- start start 2024-03-28T17:39:18.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.400+0000 7fec138db700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febf80057d0 0x7fec0c195720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:39:18.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.400+0000 7fec138db700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec0c195c60 0x7fec0c199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:39:18.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.400+0000 7fec138db700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec0c19a530 0x7fec0c19a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:39:18.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.400+0000 7fec138db700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0c107bf0 con 0x7fec0c19a530 2024-03-28T17:39:18.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.400+0000 7fec138db700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0c107a70 con 0x7fec0c195c60 2024-03-28T17:39:18.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.400+0000 7fec138db700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0c107d70 con 0x7febf80057d0 2024-03-28T17:39:18.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.400+0000 7fec11677700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febf80057d0 0x7fec0c195720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:39:18.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.400+0000 7fec11677700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febf80057d0 0x7fec0c195720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:50900/0 (socket says 172.21.15.151:50900) 2024-03-28T17:39:18.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.400+0000 7fec11677700 1 -- 172.21.15.151:0/1076202001 learned_addr learned my addr 172.21.15.151:0/1076202001 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-03-28T17:39:18.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.400+0000 7fec11677700 1 -- 172.21.15.151:0/1076202001 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febf80057d0 msgr2=0x7fec0c195720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-28T17:39:18.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.400+0000 7fec11677700 1 -- 172.21.15.151:0/1076202001 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febf80057d0 msgr2=0x7fec0c195720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T17:39:18.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.400+0000 7fec11677700 1 --2- 172.21.15.151:0/1076202001 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febf80057d0 0x7fec0c195720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T17:39:18.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.400+0000 7fec11677700 1 --2- 172.21.15.151:0/1076202001 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febf80057d0 0x7fec0c195720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T17:39:18.403 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.400+0000 7fec10e76700 1 --2- 172.21.15.151:0/1076202001 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec0c195c60 0x7fec0c199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:39:18.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.400+0000 7fec10e76700 1 -- 172.21.15.151:0/1076202001 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec0c195c60 msgr2=0x7fec0c199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-28T17:39:18.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.400+0000 7fec10e76700 1 -- 172.21.15.151:0/1076202001 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec0c195c60 msgr2=0x7fec0c199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T17:39:18.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.400+0000 7fec10e76700 1 --2- 172.21.15.151:0/1076202001 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec0c195c60 0x7fec0c199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T17:39:18.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.400+0000 7fec10e76700 1 --2- 172.21.15.151:0/1076202001 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec0c195c60 0x7fec0c199ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T17:39:18.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.400+0000 7fec11e78700 1 --2- 172.21.15.151:0/1076202001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec0c19a530 0x7fec0c19a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:39:18.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.401+0000 7fec11e78700 1 -- 172.21.15.151:0/1076202001 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febf80057d0 msgr2=0x7fec0c195720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T17:39:18.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.401+0000 7fec11e78700 1 --2- 172.21.15.151:0/1076202001 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febf80057d0 0x7fec0c195720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:18.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.401+0000 7fec11e78700 1 -- 172.21.15.151:0/1076202001 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec0c195c60 msgr2=0x7fec0c199ff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T17:39:18.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.401+0000 7fec11e78700 1 --2- 172.21.15.151:0/1076202001 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec0c195c60 0x7fec0c199ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:18.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.401+0000 7fec11e78700 1 -- 172.21.15.151:0/1076202001 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0c1a22a0 con 0x7fec0c19a530 2024-03-28T17:39:18.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.402+0000 7fec11e78700 1 --2- 172.21.15.151:0/1076202001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec0c19a530 0x7fec0c19a9e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fec0400bd50 tx=0x7fec0400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:39:18.404 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.402+0000 7fec027fc700 1 -- 172.21.15.151:0/1076202001 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec04019070 con 0x7fec0c19a530 2024-03-28T17:39:18.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.402+0000 7fec138db700 1 -- 172.21.15.151:0/1076202001 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec0c1a24d0 con 0x7fec0c19a530 2024-03-28T17:39:18.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.403+0000 7fec138db700 1 -- 172.21.15.151:0/1076202001 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec0c1a2a10 con 0x7fec0c19a530 2024-03-28T17:39:18.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.403+0000 7fec027fc700 1 -- 172.21.15.151:0/1076202001 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec0400cd10 con 0x7fec0c19a530 2024-03-28T17:39:18.405 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.403+0000 7fec027fc700 1 -- 172.21.15.151:0/1076202001 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec04023060 con 0x7fec0c19a530 2024-03-28T17:39:18.406 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.404+0000 7fec138db700 1 -- 172.21.15.151:0/1076202001 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf8002140 con 0x7fec0c19a530 2024-03-28T17:39:18.406 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.404+0000 7fec027fc700 1 -- 172.21.15.151:0/1076202001 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fec0401c070 con 0x7fec0c19a530 2024-03-28T17:39:18.406 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.405+0000 7fec027fc700 1 --2- 172.21.15.151:0/1076202001 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7febf0039450 0x7febf003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:39:18.406 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.405+0000 7fec027fc700 1 -- 172.21.15.151:0/1076202001 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fec0404bad0 con 0x7fec0c19a530 2024-03-28T17:39:18.406 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.405+0000 7fec11677700 1 --2- 172.21.15.151:0/1076202001 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7febf0039450 0x7febf003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:39:18.406 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.406+0000 7fec11677700 1 --2- 172.21.15.151:0/1076202001 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7febf0039450 0x7febf003b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fec08005b40 tx=0x7fec08005ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:39:18.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.421+0000 7fec027fc700 1 -- 172.21.15.151:0/1076202001 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fec0401d910 con 0x7fec0c19a530 2024-03-28T17:39:18.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:18 smithi045 ceph-mon[31773]: mon.c calling monitor election 2024-03-28T17:39:18.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:18 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T17:39:18.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:18 smithi045 ceph-mon[31773]: mon.a calling monitor election 2024-03-28T17:39:18.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:18 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:18.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:18 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:18.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:18 smithi045 ceph-mon[31773]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:18.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:18 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:18.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:18 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:18.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:18 smithi045 ceph-mon[31773]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:18.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:18 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:18.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:18 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:18.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:18 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:18.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:18 smithi045 ceph-mon[31773]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T17:39:18.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:18 smithi045 ceph-mon[31773]: monmap e3: 3 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],b=[v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0],c=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T17:39:18.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:18 smithi045 ceph-mon[31773]: fsmap 2024-03-28T17:39:18.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:18 smithi045 ceph-mon[31773]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T17:39:18.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:18 smithi045 ceph-mon[31773]: mgrmap e13: x(active, since 70s) 2024-03-28T17:39:18.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:18 smithi045 ceph-mon[31773]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-28T17:39:18.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:18 smithi045 ceph-mon[31773]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-28T17:39:18.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:18 smithi045 ceph-mon[31773]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-28T17:39:18.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:18 smithi045 ceph-mon[31773]: mon.b (rank 2) addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] is down (out of quorum) 2024-03-28T17:39:18.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:18 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:18.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:18 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:18.707 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.706+0000 7fec138db700 1 -- 172.21.15.151:0/1076202001 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7febf8002c30 con 0x7fec0c19a530 2024-03-28T17:39:18.707 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:18 smithi151 ceph-mon[36481]: mon.c calling monitor election 2024-03-28T17:39:18.707 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:18 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T17:39:18.707 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:18 smithi151 ceph-mon[36481]: mon.a calling monitor election 2024-03-28T17:39:18.707 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:18 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:18.707 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:18 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:18.708 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:18 smithi151 ceph-mon[36481]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:18.708 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:18 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:18.708 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:18 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:18.708 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:18 smithi151 ceph-mon[36481]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:18.708 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:18 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:18.708 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:18 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:18.708 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:18 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:18.708 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:18 smithi151 ceph-mon[36481]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T17:39:18.708 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:18 smithi151 ceph-mon[36481]: monmap e3: 3 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],b=[v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0],c=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T17:39:18.708 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:18 smithi151 ceph-mon[36481]: fsmap 2024-03-28T17:39:18.708 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:18 smithi151 ceph-mon[36481]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T17:39:18.708 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:18 smithi151 ceph-mon[36481]: mgrmap e13: x(active, since 70s) 2024-03-28T17:39:18.708 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:18 smithi151 ceph-mon[36481]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-28T17:39:18.709 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:18 smithi151 ceph-mon[36481]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-28T17:39:18.709 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:18 smithi151 ceph-mon[36481]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-28T17:39:18.709 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:18 smithi151 ceph-mon[36481]: mon.b (rank 2) addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] is down (out of quorum) 2024-03-28T17:39:18.709 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:18 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:18.709 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:18 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:18.709 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.707+0000 7fec027fc700 1 -- 172.21.15.151:0/1076202001 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1279 (secure 0 0 0) 0x7fec0404a460 con 0x7fec0c19a530 2024-03-28T17:39:18.710 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-28T17:39:18.710 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":3,"fsid":"9f4888ec-ed29-11ee-95d0-87774f69a715","modified":"2024-03-28T17:39:13.308931Z","created":"2024-03-28T17:36:47.322376Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-28T17:39:18.710 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.710+0000 7fec138db700 1 -- 172.21.15.151:0/1076202001 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7febf0039450 msgr2=0x7febf003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:39:18.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.710+0000 7fec138db700 1 --2- 172.21.15.151:0/1076202001 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7febf0039450 0x7febf003b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fec08005b40 tx=0x7fec08005ad0 comp rx=0 tx=0).stop 2024-03-28T17:39:18.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.710+0000 7fec138db700 1 -- 172.21.15.151:0/1076202001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec0c19a530 msgr2=0x7fec0c19a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:39:18.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.710+0000 7fec138db700 1 --2- 172.21.15.151:0/1076202001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec0c19a530 0x7fec0c19a9e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fec0400bd50 tx=0x7fec0400be30 comp rx=0 tx=0).stop 2024-03-28T17:39:18.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.710+0000 7fec138db700 1 -- 172.21.15.151:0/1076202001 shutdown_connections 2024-03-28T17:39:18.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.710+0000 7fec138db700 1 --2- 172.21.15.151:0/1076202001 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7febf0039450 0x7febf003b910 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:18.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.710+0000 7fec138db700 1 --2- 172.21.15.151:0/1076202001 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febf80057d0 0x7fec0c195720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:18.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.710+0000 7fec138db700 1 --2- 172.21.15.151:0/1076202001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec0c19a530 0x7fec0c19a9e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:18.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.710+0000 7fec138db700 1 --2- 172.21.15.151:0/1076202001 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec0c195c60 0x7fec0c199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:18.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.710+0000 7fec138db700 1 -- 172.21.15.151:0/1076202001 >> 172.21.15.151:0/1076202001 conn(0x7fec0c0fc6a0 msgr2=0x7fec0c0fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:39:18.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.710+0000 7fec138db700 1 -- 172.21.15.151:0/1076202001 shutdown_connections 2024-03-28T17:39:18.711 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:18.710+0000 7fec138db700 1 -- 172.21.15.151:0/1076202001 wait complete. 2024-03-28T17:39:18.711 INFO:teuthology.orchestra.run.smithi151.stderr:dumped monmap epoch 3 2024-03-28T17:39:19.315 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-28T17:39:19.315 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-28T17:39:19.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:19 smithi151 ceph-mon[36481]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:19.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:19 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:19.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:19 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:19.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:19 smithi151 ceph-mon[36481]: from='client.? 172.21.15.151:0/1076202001' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T17:39:19.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:19 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:39:19.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:19 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:19 smithi045 ceph-mon[31773]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:19 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:19 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:19 smithi045 ceph-mon[31773]: from='client.? 172.21.15.151:0/1076202001' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T17:39:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:19 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:39:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:19 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:19.792 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:39:20.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: mon.c calling monitor election 2024-03-28T17:39:20.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T17:39:20.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: mon.a calling monitor election 2024-03-28T17:39:20.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:20.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:39:20.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:20.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:20.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:20.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:20.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:20.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:20.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:20.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T17:39:20.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: monmap e3: 3 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],b=[v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0],c=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T17:39:20.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: fsmap 2024-03-28T17:39:20.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T17:39:20.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: mgrmap e13: x(active, since 70s) 2024-03-28T17:39:20.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-28T17:39:20.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-28T17:39:20.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-28T17:39:20.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: mon.b (rank 2) addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] is down (out of quorum) 2024-03-28T17:39:20.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:20.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:20.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:20.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:20.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:20.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: from='client.? 172.21.15.151:0/1076202001' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T17:39:20.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:39:20.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:21.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:20 smithi045 ceph-mon[31773]: mon.b calling monitor election 2024-03-28T17:39:21.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:20 smithi045 ceph-mon[31773]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:21.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:20 smithi045 ceph-mon[31773]: Updating smithi045:/etc/ceph/ceph.conf 2024-03-28T17:39:21.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:20 smithi045 ceph-mon[31773]: Updating smithi071:/etc/ceph/ceph.conf 2024-03-28T17:39:21.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:20 smithi045 ceph-mon[31773]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-28T17:39:21.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:20 smithi045 ceph-mon[31773]: Updating smithi071:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:39:21.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:20 smithi045 ceph-mon[31773]: Updating smithi151:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:39:21.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:20 smithi045 ceph-mon[31773]: mon.b calling monitor election 2024-03-28T17:39:21.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:20 smithi045 ceph-mon[31773]: mon.c calling monitor election 2024-03-28T17:39:21.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:20 smithi045 ceph-mon[31773]: mon.a calling monitor election 2024-03-28T17:39:21.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:20 smithi045 ceph-mon[31773]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T17:39:21.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:20 smithi045 ceph-mon[31773]: monmap e3: 3 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],b=[v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0],c=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T17:39:21.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:20 smithi045 ceph-mon[31773]: fsmap 2024-03-28T17:39:21.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:20 smithi045 ceph-mon[31773]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T17:39:21.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:20 smithi045 ceph-mon[31773]: mgrmap e13: x(active, since 72s) 2024-03-28T17:39:21.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:20 smithi045 ceph-mon[31773]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-28T17:39:21.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:20 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:39:21.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:20 smithi045 ceph-mon[31773]: overall HEALTH_OK 2024-03-28T17:39:21.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:20 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:21.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:20 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:21.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:20 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:21.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:20 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:21.291 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:20 smithi151 ceph-mon[36481]: mon.b calling monitor election 2024-03-28T17:39:21.291 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:20 smithi151 ceph-mon[36481]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:21.291 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:20 smithi151 ceph-mon[36481]: Updating smithi045:/etc/ceph/ceph.conf 2024-03-28T17:39:21.292 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:20 smithi151 ceph-mon[36481]: Updating smithi071:/etc/ceph/ceph.conf 2024-03-28T17:39:21.292 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:20 smithi151 ceph-mon[36481]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-28T17:39:21.292 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:20 smithi151 ceph-mon[36481]: Updating smithi071:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:39:21.292 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:20 smithi151 ceph-mon[36481]: Updating smithi151:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:39:21.292 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:20 smithi151 ceph-mon[36481]: mon.b calling monitor election 2024-03-28T17:39:21.292 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:20 smithi151 ceph-mon[36481]: mon.c calling monitor election 2024-03-28T17:39:21.292 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:20 smithi151 ceph-mon[36481]: mon.a calling monitor election 2024-03-28T17:39:21.292 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:20 smithi151 ceph-mon[36481]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T17:39:21.292 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:20 smithi151 ceph-mon[36481]: monmap e3: 3 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],b=[v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0],c=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T17:39:21.292 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:20 smithi151 ceph-mon[36481]: fsmap 2024-03-28T17:39:21.292 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:20 smithi151 ceph-mon[36481]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T17:39:21.292 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:20 smithi151 ceph-mon[36481]: mgrmap e13: x(active, since 72s) 2024-03-28T17:39:21.292 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:20 smithi151 ceph-mon[36481]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-28T17:39:21.292 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:20 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:39:21.292 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:20 smithi151 ceph-mon[36481]: overall HEALTH_OK 2024-03-28T17:39:21.293 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:20 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:21.293 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:20 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:21.293 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:20 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:21.293 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:20 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:21.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: mon.b calling monitor election 2024-03-28T17:39:21.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:21.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: Updating smithi045:/etc/ceph/ceph.conf 2024-03-28T17:39:21.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: Updating smithi071:/etc/ceph/ceph.conf 2024-03-28T17:39:21.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-28T17:39:21.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: Updating smithi071:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:39:21.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: Updating smithi151:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:39:21.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: mon.b calling monitor election 2024-03-28T17:39:21.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: mon.c calling monitor election 2024-03-28T17:39:21.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: mon.a calling monitor election 2024-03-28T17:39:21.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T17:39:21.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: monmap e3: 3 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],b=[v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0],c=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T17:39:21.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: fsmap 2024-03-28T17:39:21.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T17:39:21.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: mgrmap e13: x(active, since 72s) 2024-03-28T17:39:21.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-28T17:39:21.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:39:21.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: overall HEALTH_OK 2024-03-28T17:39:21.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:21.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:21.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:21.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:22.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.038+0000 7f637fa00700 1 -- 172.21.15.45:0/487070155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6378106a60 msgr2=0x7f6378106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:39:22.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.038+0000 7f637fa00700 1 --2- 172.21.15.45:0/487070155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6378106a60 0x7f6378106e40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f636c009910 tx=0x7f636c009c20 comp rx=0 tx=0).stop 2024-03-28T17:39:22.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.038+0000 7f637fa00700 1 -- 172.21.15.45:0/487070155 shutdown_connections 2024-03-28T17:39:22.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.038+0000 7f637fa00700 1 --2- 172.21.15.45:0/487070155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6378106a60 0x7f6378106e40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:22.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.038+0000 7f637fa00700 1 -- 172.21.15.45:0/487070155 >> 172.21.15.45:0/487070155 conn(0x7f63780776d0 msgr2=0x7f63780fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:39:22.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.038+0000 7f637fa00700 1 -- 172.21.15.45:0/487070155 shutdown_connections 2024-03-28T17:39:22.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.038+0000 7f637fa00700 1 -- 172.21.15.45:0/487070155 wait complete. 2024-03-28T17:39:22.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.040+0000 7f637fa00700 1 Processor -- start 2024-03-28T17:39:22.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.040+0000 7f637fa00700 1 -- start start 2024-03-28T17:39:22.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.041+0000 7f637fa00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6378106a60 0x7f637819f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:39:22.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.041+0000 7f637fa00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f637819fd60 0x7f63781999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:39:22.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.041+0000 7f637fa00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6378199f20 0x7f637819a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:39:22.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.041+0000 7f637fa00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637810b0a0 con 0x7f6378199f20 2024-03-28T17:39:22.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.041+0000 7f637fa00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637810af20 con 0x7f637819fd60 2024-03-28T17:39:22.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.041+0000 7f637fa00700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637810b220 con 0x7f6378106a60 2024-03-28T17:39:22.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.041+0000 7f6377e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f637819fd60 0x7f63781999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:39:22.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.041+0000 7f637da00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6378199f20 0x7f637819a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:39:22.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.041+0000 7f637d000700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6378106a60 0x7f637819f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:39:22.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.041+0000 7f6377e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f637819fd60 0x7f63781999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:48836/0 (socket says 172.21.15.45:48836) 2024-03-28T17:39:22.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.041+0000 7f6377e00700 1 -- 172.21.15.45:0/3483933953 learned_addr learned my addr 172.21.15.45:0/3483933953 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:39:22.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.042+0000 7f637d000700 1 -- 172.21.15.45:0/3483933953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f637819fd60 msgr2=0x7f63781999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:39:22.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.042+0000 7f637d000700 1 --2- 172.21.15.45:0/3483933953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f637819fd60 0x7f63781999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:22.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.042+0000 7f637d000700 1 -- 172.21.15.45:0/3483933953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6378199f20 msgr2=0x7f637819a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:39:22.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.042+0000 7f637d000700 1 --2- 172.21.15.45:0/3483933953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6378199f20 0x7f637819a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:22.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.042+0000 7f637d000700 1 -- 172.21.15.45:0/3483933953 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63781a66f0 con 0x7f6378106a60 2024-03-28T17:39:22.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.042+0000 7f6377e00700 1 --2- 172.21.15.45:0/3483933953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f637819fd60 0x7f63781999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:39:22.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.042+0000 7f637d000700 1 --2- 172.21.15.45:0/3483933953 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6378106a60 0x7f637819f820 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f636c005950 tx=0x7f636c004f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:39:22.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.042+0000 7f6375600700 1 -- 172.21.15.45:0/3483933953 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c026070 con 0x7f6378106a60 2024-03-28T17:39:22.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.042+0000 7f6375600700 1 -- 172.21.15.45:0/3483933953 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f636c00b6f0 con 0x7f6378106a60 2024-03-28T17:39:22.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.042+0000 7f6375600700 1 -- 172.21.15.45:0/3483933953 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c009e60 con 0x7f6378106a60 2024-03-28T17:39:22.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.042+0000 7f637fa00700 1 -- 172.21.15.45:0/3483933953 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63781a6920 con 0x7f6378106a60 2024-03-28T17:39:22.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.043+0000 7f637fa00700 1 -- 172.21.15.45:0/3483933953 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63781a6e10 con 0x7f6378106a60 2024-03-28T17:39:22.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.044+0000 7f637fa00700 1 -- 172.21.15.45:0/3483933953 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63780637c0 con 0x7f6378106a60 2024-03-28T17:39:22.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.044+0000 7f6375600700 1 -- 172.21.15.45:0/3483933953 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f636c029070 con 0x7f6378106a60 2024-03-28T17:39:22.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.045+0000 7f6375600700 1 --2- 172.21.15.45:0/3483933953 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6360039450 0x7f636003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:39:22.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.045+0000 7f6375600700 1 -- 172.21.15.45:0/3483933953 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f636c04cb20 con 0x7f6378106a60 2024-03-28T17:39:22.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.045+0000 7f6377e00700 1 --2- 172.21.15.45:0/3483933953 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6360039450 0x7f636003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:39:22.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.046+0000 7f6377e00700 1 --2- 172.21.15.45:0/3483933953 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6360039450 0x7f636003b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6364009f70 tx=0x7f63640092c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:39:22.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.050+0000 7f6375600700 1 -- 172.21.15.45:0/3483933953 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f636c013070 con 0x7f6378106a60 2024-03-28T17:39:22.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.279+0000 7f637fa00700 1 -- 172.21.15.45:0/3483933953 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f63781a70a0 con 0x7f6378106a60 2024-03-28T17:39:22.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.280+0000 7f6375600700 1 -- 172.21.15.45:0/3483933953 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+273 (secure 0 0 0) 0x7f636c030090 con 0x7f6378106a60 2024-03-28T17:39:22.281 INFO:teuthology.orchestra.run.smithi045.stdout:# minimal ceph.conf for 9f4888ec-ed29-11ee-95d0-87774f69a715 2024-03-28T17:39:22.281 INFO:teuthology.orchestra.run.smithi045.stdout:[global] 2024-03-28T17:39:22.281 INFO:teuthology.orchestra.run.smithi045.stdout: fsid = 9f4888ec-ed29-11ee-95d0-87774f69a715 2024-03-28T17:39:22.281 INFO:teuthology.orchestra.run.smithi045.stdout: mon_host = [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2024-03-28T17:39:22.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.283+0000 7f637fa00700 1 -- 172.21.15.45:0/3483933953 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6360039450 msgr2=0x7f636003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:39:22.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.283+0000 7f637fa00700 1 --2- 172.21.15.45:0/3483933953 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6360039450 0x7f636003b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6364009f70 tx=0x7f63640092c0 comp rx=0 tx=0).stop 2024-03-28T17:39:22.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.283+0000 7f637fa00700 1 -- 172.21.15.45:0/3483933953 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6378106a60 msgr2=0x7f637819f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:39:22.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.283+0000 7f637fa00700 1 --2- 172.21.15.45:0/3483933953 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6378106a60 0x7f637819f820 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f636c005950 tx=0x7f636c004f80 comp rx=0 tx=0).stop 2024-03-28T17:39:22.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.283+0000 7f637fa00700 1 -- 172.21.15.45:0/3483933953 shutdown_connections 2024-03-28T17:39:22.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.283+0000 7f637fa00700 1 --2- 172.21.15.45:0/3483933953 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6360039450 0x7f636003b910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:22.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.283+0000 7f637fa00700 1 --2- 172.21.15.45:0/3483933953 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6378106a60 0x7f637819f820 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:22.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.283+0000 7f637fa00700 1 --2- 172.21.15.45:0/3483933953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6378199f20 0x7f637819a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:22.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.283+0000 7f637fa00700 1 --2- 172.21.15.45:0/3483933953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f637819fd60 0x7f63781999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:22.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.283+0000 7f637fa00700 1 -- 172.21.15.45:0/3483933953 >> 172.21.15.45:0/3483933953 conn(0x7f63780776d0 msgr2=0x7f63780695c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:39:22.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.284+0000 7f637fa00700 1 -- 172.21.15.45:0/3483933953 shutdown_connections 2024-03-28T17:39:22.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:22.284+0000 7f637fa00700 1 -- 172.21.15.45:0/3483933953 wait complete. 2024-03-28T17:39:22.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:22 smithi045 ceph-mon[31773]: Updating smithi045:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:39:22.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:22 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:22.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:22 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:22.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:22 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:22.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:22 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:22.303 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:22 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:22.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:22 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:22.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:22 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:22.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:22 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:22.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:22 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T17:39:22.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:22 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T17:39:22.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:22 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:22.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:22 smithi071 ceph-mon[36597]: Updating smithi045:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:39:22.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:22 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:22.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:22 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:22.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:22 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:22.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:22 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:22.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:22 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:22.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:22 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:22.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:22 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:22.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:22 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:22.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:22 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T17:39:22.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:22 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T17:39:22.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:22 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:22 smithi151 ceph-mon[36481]: Updating smithi045:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:39:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:22 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:22 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:22 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:22.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:22 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:39:22.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:22 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:22.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:22 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:22.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:22 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:22.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:22 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:22.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:22 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T17:39:22.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:22 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T17:39:22.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:22 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:23.505 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-28T17:39:23.505 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-28T17:39:23.505 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T17:39:23.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:23 smithi071 ceph-mon[36597]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:23.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:23 smithi071 ceph-mon[36597]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T17:39:23.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:23 smithi071 ceph-mon[36597]: Reconfiguring daemon mon.a on smithi045 2024-03-28T17:39:23.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:23 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3483933953' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:23.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:23 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:23.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:23 smithi151 ceph-mon[36481]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:23.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:23 smithi151 ceph-mon[36481]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T17:39:23.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:23 smithi151 ceph-mon[36481]: Reconfiguring daemon mon.a on smithi045 2024-03-28T17:39:23.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:23 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3483933953' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:23.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:23 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:23.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:23 smithi045 ceph-mon[31773]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:23.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:23 smithi045 ceph-mon[31773]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T17:39:23.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:23 smithi045 ceph-mon[31773]: Reconfiguring daemon mon.a on smithi045 2024-03-28T17:39:23.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:23 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3483933953' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:23.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:23 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:23.605 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-28T17:39:23.605 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T17:39:23.736 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-28T17:39:23.736 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T17:39:23.771 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-28T17:39:23.771 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T17:39:23.842 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-28T17:39:23.842 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T17:39:23.877 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-28T17:39:23.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T17:39:23.948 INFO:tasks.cephadm:Adding mgr.x on smithi045 2024-03-28T17:39:23.948 INFO:tasks.cephadm:Adding mgr.y on smithi071 2024-03-28T17:39:23.948 INFO:tasks.cephadm:Adding mgr.z on smithi151 2024-03-28T17:39:23.948 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph orch apply mgr '3;smithi045=x;smithi071=y;smithi151=z' 2024-03-28T17:39:24.222 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T17:39:25.495 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:25 smithi151 ceph-mon[36481]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:25.496 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:25.496 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:25.496 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:25 smithi151 ceph-mon[36481]: Reconfiguring mon.b (monmap changed)... 2024-03-28T17:39:25.496 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T17:39:25.496 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T17:39:25.496 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:25.496 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:25 smithi151 ceph-mon[36481]: Reconfiguring daemon mon.b on smithi071 2024-03-28T17:39:25.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:25 smithi071 ceph-mon[36597]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:25 smithi071 ceph-mon[36597]: Reconfiguring mon.b (monmap changed)... 2024-03-28T17:39:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T17:39:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T17:39:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:25 smithi071 ceph-mon[36597]: Reconfiguring daemon mon.b on smithi071 2024-03-28T17:39:25.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:25 smithi045 ceph-mon[31773]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:25.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:25.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:25.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:25 smithi045 ceph-mon[31773]: Reconfiguring mon.b (monmap changed)... 2024-03-28T17:39:25.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T17:39:25.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T17:39:25.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:25.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:25 smithi045 ceph-mon[31773]: Reconfiguring daemon mon.b on smithi071 2024-03-28T17:39:25.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.727+0000 7f47e0b11700 1 -- 172.21.15.151:0/4031353567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47dc100bd0 msgr2=0x7f47dc100ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:39:25.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.727+0000 7f47e0b11700 1 --2- 172.21.15.151:0/4031353567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47dc100bd0 0x7f47dc100ff0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f47cc009910 tx=0x7f47cc009c20 comp rx=0 tx=0).stop 2024-03-28T17:39:25.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.727+0000 7f47e0b11700 1 -- 172.21.15.151:0/4031353567 shutdown_connections 2024-03-28T17:39:25.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.727+0000 7f47e0b11700 1 --2- 172.21.15.151:0/4031353567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47dc100bd0 0x7f47dc100ff0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:25.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.727+0000 7f47e0b11700 1 -- 172.21.15.151:0/4031353567 >> 172.21.15.151:0/4031353567 conn(0x7f47dc0fc170 msgr2=0x7f47dc0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:39:25.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.727+0000 7f47e0b11700 1 -- 172.21.15.151:0/4031353567 shutdown_connections 2024-03-28T17:39:25.730 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.727+0000 7f47e0b11700 1 -- 172.21.15.151:0/4031353567 wait complete. 2024-03-28T17:39:25.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.728+0000 7f47e0b11700 1 Processor -- start 2024-03-28T17:39:25.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.728+0000 7f47e0b11700 1 -- start start 2024-03-28T17:39:25.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.728+0000 7f47e0b11700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47dc100bd0 0x7f47dc196420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:39:25.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.728+0000 7f47e0b11700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47dc196960 0x7f47dc19b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:39:25.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.728+0000 7f47e0b11700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47dc196de0 0x7f47dc19df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:39:25.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.728+0000 7f47e0b11700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc106c60 con 0x7f47dc100bd0 2024-03-28T17:39:25.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.728+0000 7f47e0b11700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc106ae0 con 0x7f47dc196960 2024-03-28T17:39:25.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.728+0000 7f47e0b11700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc106de0 con 0x7f47dc196de0 2024-03-28T17:39:25.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.728+0000 7f47dad9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47dc196de0 0x7f47dc19df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:39:25.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.728+0000 7f47dad9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47dc196de0 0x7f47dc19df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40528/0 (socket says 172.21.15.151:40528) 2024-03-28T17:39:25.731 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.728+0000 7f47dad9d700 1 -- 172.21.15.151:0/1136185044 learned_addr learned my addr 172.21.15.151:0/1136185044 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-03-28T17:39:25.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.728+0000 7f47d9d9b700 1 --2- 172.21.15.151:0/1136185044 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47dc196960 0x7f47dc19b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:39:25.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.728+0000 7f47dad9d700 1 -- 172.21.15.151:0/1136185044 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47dc196960 msgr2=0x7f47dc19b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:39:25.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.728+0000 7f47dad9d700 1 --2- 172.21.15.151:0/1136185044 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47dc196960 0x7f47dc19b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:25.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.728+0000 7f47dad9d700 1 -- 172.21.15.151:0/1136185044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47dc100bd0 msgr2=0x7f47dc196420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:39:25.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.728+0000 7f47dad9d700 1 --2- 172.21.15.151:0/1136185044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47dc100bd0 0x7f47dc196420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:25.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.728+0000 7f47dad9d700 1 -- 172.21.15.151:0/1136185044 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47dc19e830 con 0x7f47dc196de0 2024-03-28T17:39:25.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.729+0000 7f47dad9d700 1 --2- 172.21.15.151:0/1136185044 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47dc196de0 0x7f47dc19df60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f47d400ba50 tx=0x7f47d400bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:39:25.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.729+0000 7f47cb7fe700 1 -- 172.21.15.151:0/1136185044 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d4019070 con 0x7f47dc196de0 2024-03-28T17:39:25.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.729+0000 7f47e0b11700 1 -- 172.21.15.151:0/1136185044 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47dc19eb20 con 0x7f47dc196de0 2024-03-28T17:39:25.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.729+0000 7f47cb7fe700 1 -- 172.21.15.151:0/1136185044 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47d4014440 con 0x7f47dc196de0 2024-03-28T17:39:25.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.729+0000 7f47cb7fe700 1 -- 172.21.15.151:0/1136185044 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d40075b0 con 0x7f47dc196de0 2024-03-28T17:39:25.732 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.729+0000 7f47e0b11700 1 -- 172.21.15.151:0/1136185044 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47dc19f060 con 0x7f47dc196de0 2024-03-28T17:39:25.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.730+0000 7f47cb7fe700 1 -- 172.21.15.151:0/1136185044 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f47d401c070 con 0x7f47dc196de0 2024-03-28T17:39:25.733 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.730+0000 7f47e0b11700 1 -- 172.21.15.151:0/1136185044 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47dc0637c0 con 0x7f47dc196de0 2024-03-28T17:39:25.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.730+0000 7f47cb7fe700 1 --2- 172.21.15.151:0/1136185044 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f47c4039450 0x7f47c403b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:39:25.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.730+0000 7f47cb7fe700 1 -- 172.21.15.151:0/1136185044 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f47d404b5f0 con 0x7f47dc196de0 2024-03-28T17:39:25.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.731+0000 7f47da59c700 1 --2- 172.21.15.151:0/1136185044 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f47c4039450 0x7f47c403b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:39:25.734 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.732+0000 7f47da59c700 1 --2- 172.21.15.151:0/1136185044 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f47c4039450 0x7f47c403b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f47cc005e00 tx=0x7f47cc0052d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:39:25.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.735+0000 7f47cb7fe700 1 -- 172.21.15.151:0/1136185044 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f47d4012500 con 0x7f47dc196de0 2024-03-28T17:39:25.941 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.940+0000 7f47e0b11700 1 -- 172.21.15.151:0/1136185044 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi045=x;smithi071=y;smithi151=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f47dc062660 con 0x7f47c4039450 2024-03-28T17:39:25.954 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.953+0000 7f47cb7fe700 1 -- 172.21.15.151:0/1136185044 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f47dc062660 con 0x7f47c4039450 2024-03-28T17:39:25.954 INFO:teuthology.orchestra.run.smithi151.stdout:Scheduled mgr update... 2024-03-28T17:39:25.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.955+0000 7f47e0b11700 1 -- 172.21.15.151:0/1136185044 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f47c4039450 msgr2=0x7f47c403b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:39:25.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.955+0000 7f47e0b11700 1 --2- 172.21.15.151:0/1136185044 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f47c4039450 0x7f47c403b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f47cc005e00 tx=0x7f47cc0052d0 comp rx=0 tx=0).stop 2024-03-28T17:39:25.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.955+0000 7f47e0b11700 1 -- 172.21.15.151:0/1136185044 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47dc196de0 msgr2=0x7f47dc19df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:39:25.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.955+0000 7f47e0b11700 1 --2- 172.21.15.151:0/1136185044 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47dc196de0 0x7f47dc19df60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f47d400ba50 tx=0x7f47d400bd60 comp rx=0 tx=0).stop 2024-03-28T17:39:25.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.956+0000 7f47e0b11700 1 -- 172.21.15.151:0/1136185044 shutdown_connections 2024-03-28T17:39:25.956 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.956+0000 7f47e0b11700 1 --2- 172.21.15.151:0/1136185044 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f47c4039450 0x7f47c403b910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:25.957 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.956+0000 7f47e0b11700 1 --2- 172.21.15.151:0/1136185044 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47dc196de0 0x7f47dc19df60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:25.957 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.956+0000 7f47e0b11700 1 --2- 172.21.15.151:0/1136185044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47dc100bd0 0x7f47dc196420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:25.957 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.956+0000 7f47e0b11700 1 --2- 172.21.15.151:0/1136185044 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47dc196960 0x7f47dc19b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:25.957 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.956+0000 7f47e0b11700 1 -- 172.21.15.151:0/1136185044 >> 172.21.15.151:0/1136185044 conn(0x7f47dc0fc170 msgr2=0x7f47dc0fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:39:25.957 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.956+0000 7f47e0b11700 1 -- 172.21.15.151:0/1136185044 shutdown_connections 2024-03-28T17:39:25.957 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:39:25.956+0000 7f47e0b11700 1 -- 172.21.15.151:0/1136185044 wait complete. 2024-03-28T17:39:26.589 DEBUG:teuthology.orchestra.run.smithi071:mgr.y> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mgr.y.service 2024-03-28T17:39:26.592 DEBUG:teuthology.orchestra.run.smithi151:mgr.z> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mgr.z.service 2024-03-28T17:39:26.594 INFO:tasks.cephadm:Deploying OSDs... 2024-03-28T17:39:26.594 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-28T17:39:26.594 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T17:39:26.655 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T17:39:26.655 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_1 2024-03-28T17:39:26.712 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T17:39:26.713 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T17:39:26.713 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 635 Links: 1 2024-03-28T17:39:26.713 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T17:39:26.713 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T17:39:26.713 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-03-28 17:38:23.909642054 +0000 2024-03-28T17:39:26.713 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-03-28 17:32:53.182607526 +0000 2024-03-28T17:39:26.713 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-03-28 17:32:53.182607526 +0000 2024-03-28T17:39:26.713 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-03-28 17:32:53.182607526 +0000 2024-03-28T17:39:26.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T17:39:26.807 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-03-28T17:39:26.807 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-03-28T17:39:26.807 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000272906 s, 1.9 MB/s 2024-03-28T17:39:26.812 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T17:39:26.869 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_2 2024-03-28T17:39:26.918 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T17:39:26.918 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T17:39:26.918 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 654 Links: 1 2024-03-28T17:39:26.918 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T17:39:26.918 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T17:39:26.918 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-03-28 17:38:24.011640537 +0000 2024-03-28T17:39:26.918 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-03-28 17:32:53.744599008 +0000 2024-03-28T17:39:26.919 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-03-28 17:32:53.744599008 +0000 2024-03-28T17:39:26.919 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-03-28 17:32:53.744599008 +0000 2024-03-28T17:39:26.919 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T17:39:27.009 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-03-28T17:39:27.009 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-03-28T17:39:27.009 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.00029256 s, 1.8 MB/s 2024-03-28T17:39:27.014 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T17:39:27.070 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_3 2024-03-28T17:39:27.120 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T17:39:27.120 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T17:39:27.120 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 675 Links: 1 2024-03-28T17:39:27.120 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T17:39:27.120 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T17:39:27.121 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-03-28 17:38:24.073639615 +0000 2024-03-28T17:39:27.121 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-03-28 17:32:54.390589217 +0000 2024-03-28T17:39:27.121 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-03-28 17:32:54.390589217 +0000 2024-03-28T17:39:27.121 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-03-28 17:32:54.390589217 +0000 2024-03-28T17:39:27.121 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T17:39:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:26 smithi045 ceph-mon[31773]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:26 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:26 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:26 smithi045 ceph-mon[31773]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi045=x;smithi071=y;smithi151=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:39:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:26 smithi045 ceph-mon[31773]: Saving service mgr spec with placement smithi045=x;smithi071=y;smithi151=z;count:3 2024-03-28T17:39:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:26 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:26 smithi045 ceph-mon[31773]: Reconfiguring mon.c (monmap changed)... 2024-03-28T17:39:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:26 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T17:39:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:26 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T17:39:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:26 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:26 smithi045 ceph-mon[31773]: Reconfiguring daemon mon.c on smithi151 2024-03-28T17:39:27.212 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-03-28T17:39:27.212 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-03-28T17:39:27.212 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000269119 s, 1.9 MB/s 2024-03-28T17:39:27.217 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T17:39:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:26 smithi071 ceph-mon[36597]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:26 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:26 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:26 smithi071 ceph-mon[36597]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi045=x;smithi071=y;smithi151=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:39:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:26 smithi071 ceph-mon[36597]: Saving service mgr spec with placement smithi045=x;smithi071=y;smithi151=z;count:3 2024-03-28T17:39:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:26 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:26 smithi071 ceph-mon[36597]: Reconfiguring mon.c (monmap changed)... 2024-03-28T17:39:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:26 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T17:39:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:26 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T17:39:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:26 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:26 smithi071 ceph-mon[36597]: Reconfiguring daemon mon.c on smithi151 2024-03-28T17:39:27.317 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_4 2024-03-28T17:39:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:26 smithi151 ceph-mon[36481]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:26 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:26 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:26 smithi151 ceph-mon[36481]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi045=x;smithi071=y;smithi151=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:39:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:26 smithi151 ceph-mon[36481]: Saving service mgr spec with placement smithi045=x;smithi071=y;smithi151=z;count:3 2024-03-28T17:39:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:26 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:26 smithi151 ceph-mon[36481]: Reconfiguring mon.c (monmap changed)... 2024-03-28T17:39:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:26 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T17:39:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:26 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T17:39:27.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:26 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:27.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:26 smithi151 ceph-mon[36481]: Reconfiguring daemon mon.c on smithi151 2024-03-28T17:39:27.369 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T17:39:27.369 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T17:39:27.370 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 691 Links: 1 2024-03-28T17:39:27.370 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T17:39:27.370 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T17:39:27.370 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-03-28 17:38:24.126638827 +0000 2024-03-28T17:39:27.370 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-03-28 17:32:54.951580715 +0000 2024-03-28T17:39:27.370 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-03-28 17:32:54.951580715 +0000 2024-03-28T17:39:27.370 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-03-28 17:32:54.951580715 +0000 2024-03-28T17:39:27.370 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T17:39:27.464 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-03-28T17:39:27.464 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-03-28T17:39:27.464 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000265769 s, 1.9 MB/s 2024-03-28T17:39:27.469 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T17:39:27.528 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-28T17:39:27.529 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T17:39:27.547 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T17:39:27.547 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_1 2024-03-28T17:39:27.605 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T17:39:27.605 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T17:39:27.605 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-03-28T17:39:27.605 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T17:39:27.605 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T17:39:27.605 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-28 17:39:12.771162800 +0000 2024-03-28T17:39:27.605 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-28 17:31:34.894499758 +0000 2024-03-28T17:39:27.605 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-28 17:31:34.894499758 +0000 2024-03-28T17:39:27.605 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-28 17:31:34.894499758 +0000 2024-03-28T17:39:27.605 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T17:39:27.674 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-28T17:39:27.674 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-28T17:39:27.674 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000206486 s, 2.5 MB/s 2024-03-28T17:39:27.675 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T17:39:27.734 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_2 2024-03-28T17:39:27.791 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T17:39:27.791 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T17:39:27.791 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 681 Links: 1 2024-03-28T17:39:27.791 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T17:39:27.791 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T17:39:27.791 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-28 17:39:12.771162800 +0000 2024-03-28T17:39:27.791 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-28 17:31:35.226493987 +0000 2024-03-28T17:39:27.791 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-28 17:31:35.226493987 +0000 2024-03-28T17:39:27.792 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-28 17:31:35.226493987 +0000 2024-03-28T17:39:27.792 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T17:39:27.860 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-28T17:39:27.860 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-28T17:39:27.861 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000212416 s, 2.4 MB/s 2024-03-28T17:39:27.862 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T17:39:27.919 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_3 2024-03-28T17:39:27.976 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T17:39:27.977 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T17:39:27.977 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-03-28T17:39:27.977 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T17:39:27.977 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T17:39:27.977 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-28 17:39:12.772162783 +0000 2024-03-28T17:39:27.977 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-28 17:31:35.550488354 +0000 2024-03-28T17:39:27.977 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-28 17:31:35.550488354 +0000 2024-03-28T17:39:27.977 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-28 17:31:35.550488354 +0000 2024-03-28T17:39:27.977 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T17:39:28.046 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-28T17:39:28.046 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-28T17:39:28.046 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000203697 s, 2.5 MB/s 2024-03-28T17:39:28.047 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T17:39:28.105 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_4 2024-03-28T17:39:28.161 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T17:39:28.162 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T17:39:28.162 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-03-28T17:39:28.162 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T17:39:28.162 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T17:39:28.162 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-28 17:39:12.772162783 +0000 2024-03-28T17:39:28.162 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-28 17:31:35.890482444 +0000 2024-03-28T17:39:28.162 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-28 17:31:35.890482444 +0000 2024-03-28T17:39:28.162 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-28 17:31:35.890482444 +0000 2024-03-28T17:39:28.163 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T17:39:28.229 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-28T17:39:28.229 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-28T17:39:28.229 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000208315 s, 2.5 MB/s 2024-03-28T17:39:28.231 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T17:39:28.288 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-28T17:39:28.288 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T17:39:28.304 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T17:39:28.304 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_1 2024-03-28T17:39:28.359 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T17:39:28.359 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T17:39:28.359 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-03-28T17:39:28.359 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T17:39:28.359 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T17:39:28.359 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-28 17:39:03.629475253 +0000 2024-03-28T17:39:28.359 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-28 17:31:44.503483307 +0000 2024-03-28T17:39:28.359 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-28 17:31:44.503483307 +0000 2024-03-28T17:39:28.359 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-28 17:31:44.503483307 +0000 2024-03-28T17:39:28.360 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T17:39:28.425 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-28T17:39:28.425 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-28T17:39:28.425 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000246803 s, 2.1 MB/s 2024-03-28T17:39:28.426 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T17:39:28.482 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_2 2024-03-28T17:39:28.537 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T17:39:28.537 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T17:39:28.537 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-03-28T17:39:28.537 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T17:39:28.538 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T17:39:28.538 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-28 17:39:03.630475241 +0000 2024-03-28T17:39:28.538 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-28 17:31:44.851479003 +0000 2024-03-28T17:39:28.538 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-28 17:31:44.851479003 +0000 2024-03-28T17:39:28.538 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-28 17:31:44.851479003 +0000 2024-03-28T17:39:28.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T17:39:28.603 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-28T17:39:28.603 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-28T17:39:28.603 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000220605 s, 2.3 MB/s 2024-03-28T17:39:28.604 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T17:39:28.660 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_3 2024-03-28T17:39:28.715 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T17:39:28.715 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T17:39:28.715 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-03-28T17:39:28.715 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T17:39:28.715 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T17:39:28.715 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-28 17:39:03.630475241 +0000 2024-03-28T17:39:28.715 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-28 17:31:45.188474836 +0000 2024-03-28T17:39:28.715 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-28 17:31:45.188474836 +0000 2024-03-28T17:39:28.715 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-28 17:31:45.188474836 +0000 2024-03-28T17:39:28.716 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T17:39:28.780 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-28T17:39:28.780 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-28T17:39:28.780 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000184463 s, 2.8 MB/s 2024-03-28T17:39:28.781 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T17:39:28.837 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_4 2024-03-28T17:39:28.893 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T17:39:28.893 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T17:39:28.893 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-03-28T17:39:28.893 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T17:39:28.893 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T17:39:28.893 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-28 17:39:03.631475229 +0000 2024-03-28T17:39:28.893 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-28 17:31:45.527470644 +0000 2024-03-28T17:39:28.893 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-28 17:31:45.527470644 +0000 2024-03-28T17:39:28.893 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-28 17:31:45.527470644 +0000 2024-03-28T17:39:28.894 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T17:39:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:28 smithi045 ceph-mon[31773]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:28 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:28 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:28 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:39:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:28 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:28 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:39:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:28 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:28 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T17:39:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:28 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T17:39:28.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:28 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T17:39:28.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:28 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:28.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:28 smithi045 ceph-mon[31773]: Deploying daemon mgr.z on smithi151 2024-03-28T17:39:28.957 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:28 smithi151 ceph-mon[36481]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:28.957 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:28 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:28.957 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:28 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:28.958 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:28 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:39:28.958 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:28 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:28.958 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:28 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:39:28.958 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:28 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:28.958 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:28 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T17:39:28.958 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:28 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T17:39:28.958 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:28 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T17:39:28.958 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:28 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:28.958 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:28 smithi151 ceph-mon[36481]: Deploying daemon mgr.z on smithi151 2024-03-28T17:39:28.959 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-28T17:39:28.959 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-28T17:39:28.960 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000196942 s, 2.6 MB/s 2024-03-28T17:39:28.961 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T17:39:29.016 INFO:tasks.cephadm:Deploying osd.0 on smithi045 with /dev/vg_nvme/lv_4... 2024-03-28T17:39:29.017 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T17:39:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:28 smithi071 ceph-mon[36597]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:28 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:28 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:28 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:39:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:28 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:28 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:39:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:28 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:28 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T17:39:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:28 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T17:39:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:28 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T17:39:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:28 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:28 smithi071 ceph-mon[36597]: Deploying daemon mgr.z on smithi151 2024-03-28T17:39:29.475 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:39:30.323 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:30 smithi151 systemd[1]: Started Ceph mgr.z for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:39:31.073 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:30 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:30.603+0000 7fc60567a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T17:39:31.073 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:30 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:30.694+0000 7fc60567a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T17:39:31.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:31 smithi045 ceph-mon[31773]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:31.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:31 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:31.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:31 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:31.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:31 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:31.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:31 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T17:39:31.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:31 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T17:39:31.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:31 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T17:39:31.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:31 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:31.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:31 smithi045 ceph-mon[31773]: Deploying daemon mgr.y on smithi071 2024-03-28T17:39:31.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:31 smithi071 ceph-mon[36597]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:31.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:31 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:31.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:31 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:31.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:31 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:31.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:31 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T17:39:31.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:31 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T17:39:31.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:31 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T17:39:31.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:31 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:31.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:31 smithi071 ceph-mon[36597]: Deploying daemon mgr.y on smithi071 2024-03-28T17:39:31.573 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:31 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:31.338+0000 7fc60567a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T17:39:31.573 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:31 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:31.435+0000 7fc60567a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T17:39:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:31 smithi151 ceph-mon[36481]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:31 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:31.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:31 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:31.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:31 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:31.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:31 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T17:39:31.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:31 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T17:39:31.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:31 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T17:39:31.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:31 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:31.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:31 smithi151 ceph-mon[36481]: Deploying daemon mgr.y on smithi071 2024-03-28T17:39:32.039 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:31 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:31.582+0000 7fc60567a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T17:39:32.039 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:31 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:31.681+0000 7fc60567a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T17:39:32.039 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:31 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:31.769+0000 7fc60567a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T17:39:32.039 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:32 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:32.037+0000 7fc60567a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T17:39:32.313 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:32 smithi071 systemd[1]: Starting Ceph mgr.y for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:39:32.323 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:32 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:32.148+0000 7fc60567a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T17:39:32.323 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:32 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:32.233+0000 7fc60567a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T17:39:32.357 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T17:39:32.357 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T17:39:32.357 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2024-03-28T17:39:32.357 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2024-03-28T17:39:32.357 INFO:teuthology.orchestra.run.smithi045.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0389965 s, 269 MB/s 2024-03-28T17:39:32.357 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 2024-03-28T17:39:32.357 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2024-03-28T17:39:32.589 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:32 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:32.327+0000 7fc60567a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T17:39:32.589 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:32 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:32.507+0000 7fc60567a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T17:39:32.728 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:32 smithi071 podman[37958]: 2024-03-28 17:39:32.331591829 +0000 UTC m=+0.017441384 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:39:32.728 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:32 smithi071 podman[37958]: 2024-03-28 17:39:32.434969705 +0000 UTC m=+0.120819242 container create 21216561732b5a072ab9e9d8b95a353f500f5f5bd2566a9228603404fdb18341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD) 2024-03-28T17:39:32.729 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:32 smithi071 podman[37958]: 2024-03-28 17:39:32.645497363 +0000 UTC m=+0.331346935 container init 21216561732b5a072ab9e9d8b95a353f500f5f5bd2566a9228603404fdb18341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-28T17:39:32.729 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:32 smithi071 podman[37958]: 2024-03-28 17:39:32.648298208 +0000 UTC m=+0.334147759 container start 21216561732b5a072ab9e9d8b95a353f500f5f5bd2566a9228603404fdb18341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T17:39:32.729 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:32 smithi071 ceph-mgr[37978]: -- 172.21.15.71:0/2724479822 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5614f2bec5a0 con 0x5614f2bc0c00 2024-03-28T17:39:32.729 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:32 smithi071 bash[37958]: 21216561732b5a072ab9e9d8b95a353f500f5f5bd2566a9228603404fdb18341 2024-03-28T17:39:33.035 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:32 smithi071 systemd[1]: Started Ceph mgr.y for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:39:33.215 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi045:vg_nvme/lv_4 2024-03-28T17:39:33.297 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:33 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:33.197+0000 7fba88141200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T17:39:33.297 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:33 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:33.296+0000 7fba88141200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T17:39:33.297 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:33 smithi071 ceph-mon[36597]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:33.297 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:33 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:33.297 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:33 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:33.297 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:33 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:33.297 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:33 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:33.297 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:33 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:39:33.405 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:33 smithi045 ceph-mon[31773]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:33.406 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:33 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:33.406 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:33 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:33.406 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:33 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:33.406 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:33 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:33.406 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:33 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:39:33.503 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:33 smithi151 ceph-mon[36481]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:33.503 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:33 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:33.503 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:33 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:33.503 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:33 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:33.503 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:33 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:33.503 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:33 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:39:33.673 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:39:33.823 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:33 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:33.502+0000 7fc60567a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T17:39:33.823 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:33 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:33.643+0000 7fc60567a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T17:39:34.243 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:34 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:33.999+0000 7fba88141200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T17:39:34.243 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:34 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:34.097+0000 7fba88141200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T17:39:34.291 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:34 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:34.000+0000 7fc60567a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T17:39:34.291 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:34 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:34.094+0000 7fc60567a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T17:39:34.291 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:34 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:34.290+0000 7fc60567a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T17:39:34.560 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:34 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:34.242+0000 7fba88141200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T17:39:34.560 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:34 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:34.362+0000 7fba88141200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T17:39:34.561 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:34 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:34.447+0000 7fba88141200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T17:39:34.573 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:34 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:34.483+0000 7fc60567a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T17:39:34.996 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:34 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:34.711+0000 7fba88141200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T17:39:34.996 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:34 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:34.818+0000 7fba88141200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T17:39:34.996 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:34 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:34.903+0000 7fba88141200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T17:39:34.996 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:34 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:34.995+0000 7fba88141200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T17:39:35.031 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:34 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:34.770+0000 7fc60567a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T17:39:35.031 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:35.030+0000 7fc60567a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T17:39:35.310 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:35 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:35.167+0000 7fba88141200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T17:39:35.323 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:35.263+0000 7fc60567a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T17:39:35.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:35 smithi045 ceph-mon[31773]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:35.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:35 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:35.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:35 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:35.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:35 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:35.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:35 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:35.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:35 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:35.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:35 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:39:35.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:35 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:35.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:35 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T17:39:35.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:35 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T17:39:35.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:35 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:35.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.779+0000 7f9efd800700 1 -- 172.21.15.45:0/932597474 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ef81021f0 msgr2=0x7f9ef8102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:39:35.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.779+0000 7f9efd800700 1 --2- 172.21.15.45:0/932597474 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ef81021f0 0x7f9ef8102610 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009990 tx=0x7f9ee8009ca0 comp rx=0 tx=0).stop 2024-03-28T17:39:35.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.780+0000 7f9efd800700 1 -- 172.21.15.45:0/932597474 shutdown_connections 2024-03-28T17:39:35.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.780+0000 7f9efd800700 1 --2- 172.21.15.45:0/932597474 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ef81021f0 0x7f9ef8102610 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:35.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.780+0000 7f9efd800700 1 --2- 172.21.15.45:0/932597474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ef8103d90 0x7f9ef810a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:35.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.780+0000 7f9efd800700 1 --2- 172.21.15.45:0/932597474 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ef81033f0 0x7f9ef8103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:35.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.780+0000 7f9efd800700 1 -- 172.21.15.45:0/932597474 >> 172.21.15.45:0/932597474 conn(0x7f9ef80fd790 msgr2=0x7f9ef80ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:39:35.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.780+0000 7f9efd800700 1 -- 172.21.15.45:0/932597474 shutdown_connections 2024-03-28T17:39:35.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.780+0000 7f9efd800700 1 -- 172.21.15.45:0/932597474 wait complete. 2024-03-28T17:39:35.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.781+0000 7f9efd800700 1 Processor -- start 2024-03-28T17:39:35.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.781+0000 7f9efd800700 1 -- start start 2024-03-28T17:39:35.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.782+0000 7f9efd800700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ef81021f0 0x7f9ef8198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:39:35.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.782+0000 7f9efd800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ef81033f0 0x7f9ef8198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:39:35.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.782+0000 7f9efd800700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ef8103d90 0x7f9ef819f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:39:35.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.782+0000 7f9efd800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef810de00 con 0x7f9ef81033f0 2024-03-28T17:39:35.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.782+0000 7f9efd800700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef810dc80 con 0x7f9ef8103d90 2024-03-28T17:39:35.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.782+0000 7f9efd800700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef810df80 con 0x7f9ef81021f0 2024-03-28T17:39:35.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.782+0000 7f9ef6a00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ef81021f0 0x7f9ef8198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:39:35.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.782+0000 7f9eee000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ef81033f0 0x7f9ef8198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:39:35.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.782+0000 7f9ef6a00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ef81021f0 0x7f9ef8198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:52318/0 (socket says 172.21.15.45:52318) 2024-03-28T17:39:35.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.782+0000 7f9ef6a00700 1 -- 172.21.15.45:0/1043600372 learned_addr learned my addr 172.21.15.45:0/1043600372 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:39:35.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.782+0000 7f9ef7400700 1 --2- 172.21.15.45:0/1043600372 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ef8103d90 0x7f9ef819f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:39:35.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.782+0000 7f9ef6a00700 1 -- 172.21.15.45:0/1043600372 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ef8103d90 msgr2=0x7f9ef819f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:39:35.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.782+0000 7f9ef6a00700 1 --2- 172.21.15.45:0/1043600372 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ef8103d90 0x7f9ef819f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:35.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.782+0000 7f9ef6a00700 1 -- 172.21.15.45:0/1043600372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ef81033f0 msgr2=0x7f9ef8198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:39:35.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.782+0000 7f9ef6a00700 1 --2- 172.21.15.45:0/1043600372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ef81033f0 0x7f9ef8198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:39:35.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.782+0000 7f9ef6a00700 1 -- 172.21.15.45:0/1043600372 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef81a0080 con 0x7f9ef81021f0 2024-03-28T17:39:35.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.782+0000 7f9ef7400700 1 --2- 172.21.15.45:0/1043600372 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ef8103d90 0x7f9ef819f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:39:35.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.783+0000 7f9ef6a00700 1 --2- 172.21.15.45:0/1043600372 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ef81021f0 0x7f9ef8198420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8005b40 tx=0x7f9ee800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:39:35.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.783+0000 7f9eefe00700 1 -- 172.21.15.45:0/1043600372 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8026070 con 0x7f9ef81021f0 2024-03-28T17:39:35.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.783+0000 7f9eefe00700 1 -- 172.21.15.45:0/1043600372 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ee800fe50 con 0x7f9ef81021f0 2024-03-28T17:39:35.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.783+0000 7f9efd800700 1 -- 172.21.15.45:0/1043600372 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ef81a0310 con 0x7f9ef81021f0 2024-03-28T17:39:35.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.783+0000 7f9eefe00700 1 -- 172.21.15.45:0/1043600372 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8017650 con 0x7f9ef81021f0 2024-03-28T17:39:35.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.783+0000 7f9efd800700 1 -- 172.21.15.45:0/1043600372 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ef81a07c0 con 0x7f9ef81021f0 2024-03-28T17:39:35.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.784+0000 7f9eefe00700 1 -- 172.21.15.45:0/1043600372 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f9ee8029070 con 0x7f9ef81021f0 2024-03-28T17:39:35.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.784+0000 7f9efd800700 1 -- 172.21.15.45:0/1043600372 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ef8068e50 con 0x7f9ef81021f0 2024-03-28T17:39:35.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.785+0000 7f9eefe00700 1 --2- 172.21.15.45:0/1043600372 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9edc039200 0x7f9edc03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:39:35.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.785+0000 7f9eefe00700 1 -- 172.21.15.45:0/1043600372 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9ee804d0c0 con 0x7f9ef81021f0 2024-03-28T17:39:35.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.785+0000 7f9eee000700 1 --2- 172.21.15.45:0/1043600372 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9edc039200 0x7f9edc03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:39:35.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.786+0000 7f9eee000700 1 --2- 172.21.15.45:0/1043600372 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9edc039200 0x7f9edc03b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0009680 tx=0x7f9ee0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:39:35.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:35.790+0000 7f9eefe00700 1 -- 172.21.15.45:0/1043600372 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9ee8013070 con 0x7f9ef81021f0 2024-03-28T17:39:35.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:35 smithi071 ceph-mon[36597]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:35.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:35 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:35.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:35 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:35.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:35 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:35.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:35 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:35.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:35 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:35.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:35 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:39:35.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:35 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:35.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:35 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T17:39:35.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:35 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T17:39:35.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:35 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:35.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:35 smithi151 ceph-mon[36481]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:35.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:35 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:35.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:35 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:35.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:35 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:35.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:35 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:35.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:35 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:35.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:35 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:39:35.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:35 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:35.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:35 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T17:39:35.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:35 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T17:39:35.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:35 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:36.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:36.036+0000 7f9efd800700 1 -- 172.21.15.45:0/1043600372 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ef80625d0 con 0x7f9edc039200 2024-03-28T17:39:36.323 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:36.058+0000 7fc60567a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T17:39:36.383 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:36 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:36.120+0000 7fba88141200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T17:39:36.384 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:36 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:36.248+0000 7fba88141200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T17:39:36.639 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:36 smithi045 ceph-mon[31773]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T17:39:36.639 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:36 smithi045 ceph-mon[31773]: Reconfiguring daemon mgr.x on smithi045 2024-03-28T17:39:36.639 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:36 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:39:36.639 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:36 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:39:36.639 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:36 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:36.664 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:36 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:36.576+0000 7fba88141200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T17:39:36.665 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:36 smithi071 ceph-mon[36597]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T17:39:36.665 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:36 smithi071 ceph-mon[36597]: Reconfiguring daemon mgr.x on smithi045 2024-03-28T17:39:36.665 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:36 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:39:36.665 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:36 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:39:36.665 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:36 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:36.724 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:36.627+0000 7fc60567a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T17:39:36.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:36 smithi151 ceph-mon[36481]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T17:39:36.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:36 smithi151 ceph-mon[36481]: Reconfiguring daemon mgr.x on smithi045 2024-03-28T17:39:36.725 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:36 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:39:36.725 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:36 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:39:36.725 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:36 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:37.036 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:36 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:36.664+0000 7fba88141200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T17:39:37.036 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:36 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:36.855+0000 7fba88141200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T17:39:37.036 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:37 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:37.036+0000 7fba88141200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T17:39:37.073 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:36.724+0000 7fc60567a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T17:39:37.073 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:39:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:39:36.936+0000 7fc60567a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T17:39:37.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:37.404+0000 7f9eefe00700 1 -- 172.21.15.45:0/1043600372 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mgrmap(e 14) v1 ==== 91698+0+0 (secure 0 0 0) 0x7f9ee8009d10 con 0x7f9ef81021f0 2024-03-28T17:39:37.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:37 smithi045 ceph-mon[31773]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:37.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:37 smithi045 ceph-mon[31773]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:39:37.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:37 smithi045 ceph-mon[31773]: Standby manager daemon z started 2024-03-28T17:39:37.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:37 smithi045 ceph-mon[31773]: from='mgr.? 172.21.15.151:0/2132524673' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T17:39:37.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:37 smithi045 ceph-mon[31773]: from='mgr.? 172.21.15.151:0/2132524673' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T17:39:37.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:37 smithi045 ceph-mon[31773]: from='mgr.? 172.21.15.151:0/2132524673' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T17:39:37.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:37 smithi045 ceph-mon[31773]: from='mgr.? 172.21.15.151:0/2132524673' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T17:39:37.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:37 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:37.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:37 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:37.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:37 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:39:37.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:37 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:37.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:37 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:39:37.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:37 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:37.573 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:37 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:37.316+0000 7fba88141200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T17:39:37.573 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:37 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:37.572+0000 7fba88141200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T17:39:37.573 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:37 smithi071 ceph-mon[36597]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:37.573 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:37 smithi071 ceph-mon[36597]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:39:37.573 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:37 smithi071 ceph-mon[36597]: Standby manager daemon z started 2024-03-28T17:39:37.573 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:37 smithi071 ceph-mon[36597]: from='mgr.? 172.21.15.151:0/2132524673' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T17:39:37.573 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:37 smithi071 ceph-mon[36597]: from='mgr.? 172.21.15.151:0/2132524673' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T17:39:37.573 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:37 smithi071 ceph-mon[36597]: from='mgr.? 172.21.15.151:0/2132524673' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T17:39:37.573 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:37 smithi071 ceph-mon[36597]: from='mgr.? 172.21.15.151:0/2132524673' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T17:39:37.574 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:37 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:37.574 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:37 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:37.574 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:37 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:39:37.574 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:37 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:37.574 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:37 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:39:37.574 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:37 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:37 smithi151 ceph-mon[36481]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:37.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:37 smithi151 ceph-mon[36481]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:39:37.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:37 smithi151 ceph-mon[36481]: Standby manager daemon z started 2024-03-28T17:39:37.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:37 smithi151 ceph-mon[36481]: from='mgr.? 172.21.15.151:0/2132524673' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T17:39:37.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:37 smithi151 ceph-mon[36481]: from='mgr.? 172.21.15.151:0/2132524673' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T17:39:37.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:37 smithi151 ceph-mon[36481]: from='mgr.? 172.21.15.151:0/2132524673' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T17:39:37.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:37 smithi151 ceph-mon[36481]: from='mgr.? 172.21.15.151:0/2132524673' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T17:39:37.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:37 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:37.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:37 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:37.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:37 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:39:37.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:37 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:37.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:37 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:39:37.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:37 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:38.060 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:37 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:37.805+0000 7fba88141200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T17:39:38.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:38 smithi045 ceph-mon[31773]: mgrmap e14: x(active, since 89s), standbys: z 2024-03-28T17:39:38.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:38 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T17:39:38.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:38 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:38.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:38 smithi071 ceph-mon[36597]: mgrmap e14: x(active, since 89s), standbys: z 2024-03-28T17:39:38.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:38 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T17:39:38.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:38 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:38.811 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:38 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:38.590+0000 7fba88141200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T17:39:38.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:38 smithi151 ceph-mon[36481]: mgrmap e14: x(active, since 89s), standbys: z 2024-03-28T17:39:38.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:38 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T17:39:38.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:38 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:39.412 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:39 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:39.146+0000 7fba88141200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T17:39:39.412 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:39 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:39.241+0000 7fba88141200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T17:39:39.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:39 smithi045 ceph-mon[31773]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:39 smithi071 ceph-mon[36597]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:39.811 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:39:39 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:39:39.455+0000 7fba88141200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T17:39:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:39 smithi151 ceph-mon[36481]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:40.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:39:40.435+0000 7f9eefe00700 1 -- 172.21.15.45:0/1043600372 <== mon.1 v2:172.21.15.151:3300/0 8 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f9ee8013d20 con 0x7f9ef81021f0 2024-03-28T17:39:40.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:40 smithi045 ceph-mon[31773]: Standby manager daemon y started 2024-03-28T17:39:40.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:40 smithi045 ceph-mon[31773]: from='mgr.? 172.21.15.71:0/3016179938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T17:39:40.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:40 smithi045 ceph-mon[31773]: from='mgr.? 172.21.15.71:0/3016179938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T17:39:40.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:40 smithi045 ceph-mon[31773]: from='mgr.? 172.21.15.71:0/3016179938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T17:39:40.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:40 smithi045 ceph-mon[31773]: from='mgr.? 172.21.15.71:0/3016179938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T17:39:40.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:40 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/800063379' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74c56bc2-4169-4919-afd4-2ebd236f23e7"}]: dispatch 2024-03-28T17:39:40.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:40 smithi045 ceph-mon[31773]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74c56bc2-4169-4919-afd4-2ebd236f23e7"}]: dispatch 2024-03-28T17:39:40.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:40 smithi045 ceph-mon[31773]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "74c56bc2-4169-4919-afd4-2ebd236f23e7"}]': finished 2024-03-28T17:39:40.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:40 smithi045 ceph-mon[31773]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T17:39:40.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:40 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T17:39:40.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:40 smithi071 ceph-mon[36597]: Standby manager daemon y started 2024-03-28T17:39:40.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:40 smithi071 ceph-mon[36597]: from='mgr.? 172.21.15.71:0/3016179938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T17:39:40.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:40 smithi071 ceph-mon[36597]: from='mgr.? 172.21.15.71:0/3016179938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T17:39:40.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:40 smithi071 ceph-mon[36597]: from='mgr.? 172.21.15.71:0/3016179938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T17:39:40.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:40 smithi071 ceph-mon[36597]: from='mgr.? 172.21.15.71:0/3016179938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T17:39:40.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:40 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/800063379' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74c56bc2-4169-4919-afd4-2ebd236f23e7"}]: dispatch 2024-03-28T17:39:40.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:40 smithi071 ceph-mon[36597]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74c56bc2-4169-4919-afd4-2ebd236f23e7"}]: dispatch 2024-03-28T17:39:40.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:40 smithi071 ceph-mon[36597]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "74c56bc2-4169-4919-afd4-2ebd236f23e7"}]': finished 2024-03-28T17:39:40.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:40 smithi071 ceph-mon[36597]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T17:39:40.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:40 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T17:39:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:40 smithi151 ceph-mon[36481]: Standby manager daemon y started 2024-03-28T17:39:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:40 smithi151 ceph-mon[36481]: from='mgr.? 172.21.15.71:0/3016179938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T17:39:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:40 smithi151 ceph-mon[36481]: from='mgr.? 172.21.15.71:0/3016179938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T17:39:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:40 smithi151 ceph-mon[36481]: from='mgr.? 172.21.15.71:0/3016179938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T17:39:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:40 smithi151 ceph-mon[36481]: from='mgr.? 172.21.15.71:0/3016179938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T17:39:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:40 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/800063379' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74c56bc2-4169-4919-afd4-2ebd236f23e7"}]: dispatch 2024-03-28T17:39:40.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:40 smithi151 ceph-mon[36481]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74c56bc2-4169-4919-afd4-2ebd236f23e7"}]: dispatch 2024-03-28T17:39:40.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:40 smithi151 ceph-mon[36481]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "74c56bc2-4169-4919-afd4-2ebd236f23e7"}]': finished 2024-03-28T17:39:40.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:40 smithi151 ceph-mon[36481]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T17:39:40.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:40 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T17:39:41.731 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:41 smithi045 ceph-mon[31773]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:41.731 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:41 smithi045 ceph-mon[31773]: mgrmap e15: x(active, since 92s), standbys: z, y 2024-03-28T17:39:41.731 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:41 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T17:39:41.731 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:41 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3470798564' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:39:41.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:41 smithi071 ceph-mon[36597]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:41.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:41 smithi071 ceph-mon[36597]: mgrmap e15: x(active, since 92s), standbys: z, y 2024-03-28T17:39:41.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:41 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T17:39:41.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:41 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3470798564' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:39:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:41 smithi151 ceph-mon[36481]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:41 smithi151 ceph-mon[36481]: mgrmap e15: x(active, since 92s), standbys: z, y 2024-03-28T17:39:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:41 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T17:39:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:41 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3470798564' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:39:43.774 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:43 smithi071 ceph-mon[36597]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:43.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:43 smithi151 ceph-mon[36481]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:43.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:43 smithi045 ceph-mon[31773]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:45.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:45 smithi071 ceph-mon[36597]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:45.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:45 smithi151 ceph-mon[36481]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:45.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:45 smithi045 ceph-mon[31773]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:47.513 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:47 smithi045 ceph-mon[31773]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:47.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:47 smithi071 ceph-mon[36597]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:47 smithi151 ceph-mon[36481]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:49.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:49 smithi071 ceph-mon[36597]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:49.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:49 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T17:39:49.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:49 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:49.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:49 smithi151 ceph-mon[36481]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:49.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:49 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T17:39:49.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:49 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:49.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:49 smithi045 ceph-mon[31773]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:49.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:49 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T17:39:49.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:49 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:39:50.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:50 smithi151 ceph-mon[36481]: Deploying daemon osd.0 on smithi045 2024-03-28T17:39:50.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:50 smithi071 ceph-mon[36597]: Deploying daemon osd.0 on smithi045 2024-03-28T17:39:50.848 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:50 smithi045 ceph-mon[31773]: Deploying daemon osd.0 on smithi045 2024-03-28T17:39:51.742 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:51 smithi045 ceph-mon[31773]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:51 smithi071 ceph-mon[36597]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:51 smithi151 ceph-mon[36481]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:53.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:53 smithi071 ceph-mon[36597]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:53.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:53 smithi151 ceph-mon[36481]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:53.859 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:53 smithi045 ceph-mon[31773]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:55.727 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:55 smithi045 ceph-mon[31773]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:55.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:55 smithi071 ceph-mon[36597]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:55.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:55 smithi151 ceph-mon[36481]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:57 smithi071 ceph-mon[36597]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:57.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:57 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:57.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:57 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:57.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:57 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:39:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:57 smithi151 ceph-mon[36481]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:57 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:57 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:57 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:39:57.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:57 smithi045 ceph-mon[31773]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:57.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:57 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:57.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:57 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:39:57.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:57 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:39:59.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:39:59 smithi045 ceph-mon[31773]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:59.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:39:59 smithi071 ceph-mon[36597]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:39:59.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:39:59 smithi151 ceph-mon[36481]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:40:00.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:00 smithi045 ceph-mon[31773]: overall HEALTH_OK 2024-03-28T17:40:00.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:00 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:00.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:00 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:00.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:00 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:00.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:00 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:40:00.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:00 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:00.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:00 smithi071 ceph-mon[36597]: overall HEALTH_OK 2024-03-28T17:40:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:00 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:00 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:00 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:00 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:40:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:00 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:00 smithi151 ceph-mon[36481]: overall HEALTH_OK 2024-03-28T17:40:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:00 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:00 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:00 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:00 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:40:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:00 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:01.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:01 smithi045 ceph-mon[31773]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:40:01.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:01 smithi045 ceph-mon[31773]: from='osd.0 [v2:172.21.15.45:6802/3795754023,v1:172.21.15.45:6803/3795754023]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T17:40:01.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:01 smithi071 ceph-mon[36597]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:40:01.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:01 smithi071 ceph-mon[36597]: from='osd.0 [v2:172.21.15.45:6802/3795754023,v1:172.21.15.45:6803/3795754023]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T17:40:01.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:01 smithi151 ceph-mon[36481]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:40:01.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:01 smithi151 ceph-mon[36481]: from='osd.0 [v2:172.21.15.45:6802/3795754023,v1:172.21.15.45:6803/3795754023]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T17:40:02.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:02.431+0000 7f9eefe00700 1 -- 172.21.15.45:0/1043600372 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9ef80625d0 con 0x7f9edc039200 2024-03-28T17:40:02.433 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 0 on host 'smithi045' 2024-03-28T17:40:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:02.435+0000 7f9efd800700 1 -- 172.21.15.45:0/1043600372 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9edc039200 msgr2=0x7f9edc03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:40:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:02.435+0000 7f9efd800700 1 --2- 172.21.15.45:0/1043600372 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9edc039200 0x7f9edc03b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0009680 tx=0x7f9ee0009260 comp rx=0 tx=0).stop 2024-03-28T17:40:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:02.435+0000 7f9efd800700 1 -- 172.21.15.45:0/1043600372 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ef81021f0 msgr2=0x7f9ef8198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:40:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:02.435+0000 7f9efd800700 1 --2- 172.21.15.45:0/1043600372 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ef81021f0 0x7f9ef8198420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8005b40 tx=0x7f9ee800f690 comp rx=0 tx=0).stop 2024-03-28T17:40:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:02.435+0000 7f9efd800700 1 -- 172.21.15.45:0/1043600372 shutdown_connections 2024-03-28T17:40:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:02.435+0000 7f9efd800700 1 --2- 172.21.15.45:0/1043600372 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9edc039200 0x7f9edc03b6c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:02.435+0000 7f9efd800700 1 --2- 172.21.15.45:0/1043600372 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ef81021f0 0x7f9ef8198420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:02.435+0000 7f9efd800700 1 --2- 172.21.15.45:0/1043600372 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ef8103d90 0x7f9ef819f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:02.435+0000 7f9efd800700 1 --2- 172.21.15.45:0/1043600372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ef81033f0 0x7f9ef8198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:02.435+0000 7f9efd800700 1 -- 172.21.15.45:0/1043600372 >> 172.21.15.45:0/1043600372 conn(0x7f9ef80fd790 msgr2=0x7f9ef81066f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:40:02.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:02.435+0000 7f9efd800700 1 -- 172.21.15.45:0/1043600372 shutdown_connections 2024-03-28T17:40:02.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:02.435+0000 7f9efd800700 1 -- 172.21.15.45:0/1043600372 wait complete. 2024-03-28T17:40:02.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:02 smithi045 ceph-mon[31773]: from='osd.0 [v2:172.21.15.45:6802/3795754023,v1:172.21.15.45:6803/3795754023]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T17:40:02.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:02 smithi045 ceph-mon[31773]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T17:40:02.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:02 smithi045 ceph-mon[31773]: from='osd.0 [v2:172.21.15.45:6802/3795754023,v1:172.21.15.45:6803/3795754023]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-28T17:40:02.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:02 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T17:40:02.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:02 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:02.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:02 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:02.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:02 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:40:02.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:02 smithi045 ceph-mon[31773]: from='osd.0 [v2:172.21.15.45:6802/3795754023,v1:172.21.15.45:6803/3795754023]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-28T17:40:02.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:02 smithi045 ceph-mon[31773]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T17:40:02.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:02 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T17:40:02.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:02 smithi071 ceph-mon[36597]: from='osd.0 [v2:172.21.15.45:6802/3795754023,v1:172.21.15.45:6803/3795754023]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T17:40:02.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:02 smithi071 ceph-mon[36597]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T17:40:02.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:02 smithi071 ceph-mon[36597]: from='osd.0 [v2:172.21.15.45:6802/3795754023,v1:172.21.15.45:6803/3795754023]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-28T17:40:02.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:02 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T17:40:02.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:02 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:02.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:02 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:02.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:02 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:40:02.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:02 smithi071 ceph-mon[36597]: from='osd.0 [v2:172.21.15.45:6802/3795754023,v1:172.21.15.45:6803/3795754023]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-28T17:40:02.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:02 smithi071 ceph-mon[36597]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T17:40:02.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:02 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T17:40:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:02 smithi151 ceph-mon[36481]: from='osd.0 [v2:172.21.15.45:6802/3795754023,v1:172.21.15.45:6803/3795754023]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T17:40:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:02 smithi151 ceph-mon[36481]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T17:40:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:02 smithi151 ceph-mon[36481]: from='osd.0 [v2:172.21.15.45:6802/3795754023,v1:172.21.15.45:6803/3795754023]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-28T17:40:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:02 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T17:40:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:02 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:02 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:02 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:40:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:02 smithi151 ceph-mon[36481]: from='osd.0 [v2:172.21.15.45:6802/3795754023,v1:172.21.15.45:6803/3795754023]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-28T17:40:02.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:02 smithi151 ceph-mon[36481]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T17:40:02.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:02 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T17:40:03.382 DEBUG:teuthology.orchestra.run.smithi045:osd.0> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@osd.0.service 2024-03-28T17:40:03.388 INFO:tasks.cephadm:Deploying osd.1 on smithi071 with /dev/vg_nvme/lv_4... 2024-03-28T17:40:03.388 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T17:40:03.624 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T17:40:03.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:03 smithi045 ceph-mon[31773]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:40:03.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:03 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T17:40:03.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:03 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T17:40:03.745 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:03 smithi071 ceph-mon[36597]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:40:03.745 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:03 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T17:40:03.746 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:03 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T17:40:03.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:03 smithi151 ceph-mon[36481]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:40:03.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:03 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T17:40:03.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:03 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T17:40:04.634 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:04 smithi071 ceph-mon[36597]: purged_snaps scrub starts 2024-03-28T17:40:04.634 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:04 smithi071 ceph-mon[36597]: purged_snaps scrub ok 2024-03-28T17:40:04.634 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:04 smithi071 ceph-mon[36597]: osd.0 [v2:172.21.15.45:6802/3795754023,v1:172.21.15.45:6803/3795754023] boot 2024-03-28T17:40:04.634 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:04 smithi071 ceph-mon[36597]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T17:40:04.634 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:04 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T17:40:04.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:04 smithi045 ceph-mon[31773]: purged_snaps scrub starts 2024-03-28T17:40:04.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:04 smithi045 ceph-mon[31773]: purged_snaps scrub ok 2024-03-28T17:40:04.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:04 smithi045 ceph-mon[31773]: osd.0 [v2:172.21.15.45:6802/3795754023,v1:172.21.15.45:6803/3795754023] boot 2024-03-28T17:40:04.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:04 smithi045 ceph-mon[31773]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T17:40:04.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:04 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T17:40:04.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:04 smithi151 ceph-mon[36481]: purged_snaps scrub starts 2024-03-28T17:40:04.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:04 smithi151 ceph-mon[36481]: purged_snaps scrub ok 2024-03-28T17:40:04.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:04 smithi151 ceph-mon[36481]: osd.0 [v2:172.21.15.45:6802/3795754023,v1:172.21.15.45:6803/3795754023] boot 2024-03-28T17:40:04.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:04 smithi151 ceph-mon[36481]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T17:40:04.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T17:40:05.460 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T17:40:05.461 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T17:40:05.461 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10+0 records in 2024-03-28T17:40:05.461 INFO:teuthology.orchestra.run.smithi071.stderr:10+0 records out 2024-03-28T17:40:05.461 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113915 s, 920 MB/s 2024-03-28T17:40:05.461 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping successful for: 2024-03-28T17:40:05.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:05 smithi045 ceph-mon[31773]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:40:05.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:05 smithi045 ceph-mon[31773]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T17:40:05.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:05 smithi071 ceph-mon[36597]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:40:05.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:05 smithi071 ceph-mon[36597]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T17:40:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:05 smithi151 ceph-mon[36481]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T17:40:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:05 smithi151 ceph-mon[36481]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T17:40:06.056 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi071:vg_nvme/lv_4 2024-03-28T17:40:06.287 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T17:40:07.633 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:07 smithi071 ceph-mon[36597]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:07.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:07 smithi151 ceph-mon[36481]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:07.842 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.842+0000 7f046c8f7700 1 -- 172.21.15.71:0/1730589424 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0468102c50 msgr2=0x7f0468109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:40:07.842 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.842+0000 7f046c8f7700 1 --2- 172.21.15.71:0/1730589424 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0468102c50 0x7f0468109500 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f04600099c0 tx=0x7f0460009cd0 comp rx=0 tx=0).stop 2024-03-28T17:40:07.842 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.842+0000 7f046c8f7700 1 -- 172.21.15.71:0/1730589424 shutdown_connections 2024-03-28T17:40:07.842 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.842+0000 7f046c8f7700 1 --2- 172.21.15.71:0/1730589424 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04681010b0 0x7f04681014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:07.843 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.842+0000 7f046c8f7700 1 --2- 172.21.15.71:0/1730589424 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0468102c50 0x7f0468109500 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:07.843 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.842+0000 7f046c8f7700 1 --2- 172.21.15.71:0/1730589424 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04681022b0 0x7f0468102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:07.843 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.842+0000 7f046c8f7700 1 -- 172.21.15.71:0/1730589424 >> 172.21.15.71:0/1730589424 conn(0x7f04680fc650 msgr2=0x7f04680fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:40:07.843 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.842+0000 7f046c8f7700 1 -- 172.21.15.71:0/1730589424 shutdown_connections 2024-03-28T17:40:07.843 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.842+0000 7f046c8f7700 1 -- 172.21.15.71:0/1730589424 wait complete. 2024-03-28T17:40:07.844 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.843+0000 7f046c8f7700 1 Processor -- start 2024-03-28T17:40:07.844 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.843+0000 7f046c8f7700 1 -- start start 2024-03-28T17:40:07.845 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.843+0000 7f046c8f7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04681010b0 0x7f046819a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:40:07.845 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.843+0000 7f046c8f7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04681022b0 0x7f046819ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:40:07.845 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.843+0000 7f046c8f7700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0468102c50 0x7f04681a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:40:07.845 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.843+0000 7f046c8f7700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046810cd90 con 0x7f04681022b0 2024-03-28T17:40:07.845 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.843+0000 7f046c8f7700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046810cc10 con 0x7f0468102c50 2024-03-28T17:40:07.845 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.843+0000 7f046c8f7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046810cf10 con 0x7f04681010b0 2024-03-28T17:40:07.845 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.843+0000 7f0466d9d700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0468102c50 0x7f04681a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:40:07.845 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.843+0000 7f0466d9d700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0468102c50 0x7f04681a1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:53414/0 (socket says 172.21.15.71:53414) 2024-03-28T17:40:07.845 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.843+0000 7f0466d9d700 1 -- 172.21.15.71:0/4265825825 learned_addr learned my addr 172.21.15.71:0/4265825825 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-03-28T17:40:07.845 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.843+0000 7f046659c700 1 --2- 172.21.15.71:0/4265825825 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04681010b0 0x7f046819a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:40:07.845 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.843+0000 7f0466d9d700 1 -- 172.21.15.71:0/4265825825 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04681010b0 msgr2=0x7f046819a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:40:07.845 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.843+0000 7f0466d9d700 1 --2- 172.21.15.71:0/4265825825 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04681010b0 0x7f046819a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:07.845 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.843+0000 7f0466d9d700 1 -- 172.21.15.71:0/4265825825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04681022b0 msgr2=0x7f046819ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:40:07.846 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.843+0000 7f0466d9d700 1 --2- 172.21.15.71:0/4265825825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04681022b0 0x7f046819ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:07.846 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.843+0000 7f0466d9d700 1 -- 172.21.15.71:0/4265825825 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04681a2380 con 0x7f0468102c50 2024-03-28T17:40:07.846 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.843+0000 7f0466d9d700 1 --2- 172.21.15.71:0/4265825825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0468102c50 0x7f04681a1ce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0460009990 tx=0x7f04600047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:40:07.846 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.844+0000 7f045ffff700 1 -- 172.21.15.71:0/4265825825 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0460003d30 con 0x7f0468102c50 2024-03-28T17:40:07.846 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.844+0000 7f046c8f7700 1 -- 172.21.15.71:0/4265825825 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04681a2610 con 0x7f0468102c50 2024-03-28T17:40:07.846 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.844+0000 7f045ffff700 1 -- 172.21.15.71:0/4265825825 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0460004830 con 0x7f0468102c50 2024-03-28T17:40:07.846 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.844+0000 7f045ffff700 1 -- 172.21.15.71:0/4265825825 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0460005590 con 0x7f0468102c50 2024-03-28T17:40:07.846 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.844+0000 7f046c8f7700 1 -- 172.21.15.71:0/4265825825 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04681a2a80 con 0x7f0468102c50 2024-03-28T17:40:07.847 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.845+0000 7f046c8f7700 1 -- 172.21.15.71:0/4265825825 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04681a2d40 con 0x7f0468102c50 2024-03-28T17:40:07.847 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.846+0000 7f045ffff700 1 -- 172.21.15.71:0/4265825825 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f04600049e0 con 0x7f0468102c50 2024-03-28T17:40:07.847 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.846+0000 7f045ffff700 1 --2- 172.21.15.71:0/4265825825 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f04540a2800 0x7f04540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:40:07.847 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.846+0000 7f045ffff700 1 -- 172.21.15.71:0/4265825825 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f04600ce140 con 0x7f0468102c50 2024-03-28T17:40:07.847 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.847+0000 7f046659c700 1 --2- 172.21.15.71:0/4265825825 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f04540a2800 0x7f04540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:40:07.848 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.848+0000 7f046659c700 1 --2- 172.21.15.71:0/4265825825 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f04540a2800 0x7f04540a4cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0468102110 tx=0x7f045800d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:40:07.851 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:07.851+0000 7f045ffff700 1 -- 172.21.15.71:0/4265825825 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0460013070 con 0x7f0468102c50 2024-03-28T17:40:07.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:07 smithi045 ceph-mon[31773]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:08.061 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:08.060+0000 7f046c8f7700 1 -- 172.21.15.71:0/4265825825 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f04680fe820 con 0x7f04540a2800 2024-03-28T17:40:09.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:08 smithi071 ceph-mon[36597]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:09.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:08 smithi071 ceph-mon[36597]: Detected new or changed devices on smithi045 2024-03-28T17:40:09.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:09.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:09.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:40:09.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:08 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi045 to 17095M 2024-03-28T17:40:09.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:09.256 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:09.256 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:40:09.256 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:09.256 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:08 smithi071 ceph-mon[36597]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:40:09.256 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:40:09.256 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:40:09.256 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:09.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:08 smithi151 ceph-mon[36481]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:09.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:08 smithi151 ceph-mon[36481]: Detected new or changed devices on smithi045 2024-03-28T17:40:09.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:09.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:09.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:40:09.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:08 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi045 to 17095M 2024-03-28T17:40:09.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:09.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:09.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:40:09.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:09.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:08 smithi151 ceph-mon[36481]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:40:09.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:40:09.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:40:09.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:09.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:08 smithi045 ceph-mon[31773]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:09.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:08 smithi045 ceph-mon[31773]: Detected new or changed devices on smithi045 2024-03-28T17:40:09.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:09.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:09.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:40:09.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:08 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi045 to 17095M 2024-03-28T17:40:09.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:09.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:09.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:40:09.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:09.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:08 smithi045 ceph-mon[31773]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:40:09.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:40:09.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:40:09.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:11.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:10 smithi071 ceph-mon[36597]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:11.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:10 smithi071 ceph-mon[36597]: from='client.? 172.21.15.71:0/2926945902' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78dcc21b-75b2-453d-b690-17b55b8ed1a0"}]: dispatch 2024-03-28T17:40:11.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:10 smithi071 ceph-mon[36597]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78dcc21b-75b2-453d-b690-17b55b8ed1a0"}]: dispatch 2024-03-28T17:40:11.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:10 smithi071 ceph-mon[36597]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "78dcc21b-75b2-453d-b690-17b55b8ed1a0"}]': finished 2024-03-28T17:40:11.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:10 smithi071 ceph-mon[36597]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T17:40:11.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:10 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T17:40:11.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:10 smithi151 ceph-mon[36481]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:11.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:10 smithi151 ceph-mon[36481]: from='client.? 172.21.15.71:0/2926945902' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78dcc21b-75b2-453d-b690-17b55b8ed1a0"}]: dispatch 2024-03-28T17:40:11.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:10 smithi151 ceph-mon[36481]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78dcc21b-75b2-453d-b690-17b55b8ed1a0"}]: dispatch 2024-03-28T17:40:11.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:10 smithi151 ceph-mon[36481]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "78dcc21b-75b2-453d-b690-17b55b8ed1a0"}]': finished 2024-03-28T17:40:11.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:10 smithi151 ceph-mon[36481]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T17:40:11.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:10 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T17:40:11.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:10 smithi045 ceph-mon[31773]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:11.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:10 smithi045 ceph-mon[31773]: from='client.? 172.21.15.71:0/2926945902' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78dcc21b-75b2-453d-b690-17b55b8ed1a0"}]: dispatch 2024-03-28T17:40:11.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:10 smithi045 ceph-mon[31773]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78dcc21b-75b2-453d-b690-17b55b8ed1a0"}]: dispatch 2024-03-28T17:40:11.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:10 smithi045 ceph-mon[31773]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "78dcc21b-75b2-453d-b690-17b55b8ed1a0"}]': finished 2024-03-28T17:40:11.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:10 smithi045 ceph-mon[31773]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T17:40:11.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:10 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T17:40:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:11 smithi071 ceph-mon[36597]: from='client.? 172.21.15.71:0/1419665578' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:40:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:11 smithi151 ceph-mon[36481]: from='client.? 172.21.15.71:0/1419665578' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:40:12.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:11 smithi045 ceph-mon[31773]: from='client.? 172.21.15.71:0/1419665578' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:40:13.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:12 smithi071 ceph-mon[36597]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:13.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:12 smithi151 ceph-mon[36481]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:13.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:12 smithi045 ceph-mon[31773]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:15.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:14 smithi071 ceph-mon[36597]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:15.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:14 smithi151 ceph-mon[36481]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:15.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:14 smithi045 ceph-mon[31773]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:17.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:16 smithi071 ceph-mon[36597]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:16 smithi151 ceph-mon[36481]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:16 smithi045 ceph-mon[31773]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:18.293 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:17 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T17:40:18.293 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:17 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:18.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:17 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T17:40:18.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:17 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:18.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:17 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T17:40:18.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:17 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:19.099 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:19 smithi071 ceph-mon[36597]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:19.099 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:19 smithi071 ceph-mon[36597]: Deploying daemon osd.1 on smithi071 2024-03-28T17:40:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:19 smithi151 ceph-mon[36481]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:19 smithi151 ceph-mon[36481]: Deploying daemon osd.1 on smithi071 2024-03-28T17:40:19.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:19 smithi045 ceph-mon[31773]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:19.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:19 smithi045 ceph-mon[31773]: Deploying daemon osd.1 on smithi071 2024-03-28T17:40:21.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:21 smithi045 ceph-mon[31773]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:21.190 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:21 smithi071 ceph-mon[36597]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:21.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:21 smithi151 ceph-mon[36481]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:23 smithi071 ceph-mon[36597]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:23 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:40:23.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:23 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:23.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:23 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:23.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:23 smithi151 ceph-mon[36481]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:23.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:23 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:40:23.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:23 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:23.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:23 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:23.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:23 smithi045 ceph-mon[31773]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:23.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:23 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:40:23.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:23 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:23.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:23 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:25.110 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:25 smithi071 ceph-mon[36597]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:25.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:25 smithi151 ceph-mon[36481]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:25.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:25 smithi045 ceph-mon[31773]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:26.601 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:26.600+0000 7f045ffff700 1 -- 172.21.15.71:0/4265825825 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f04680fe820 con 0x7f04540a2800 2024-03-28T17:40:26.601 INFO:teuthology.orchestra.run.smithi071.stdout:Created osd(s) 1 on host 'smithi071' 2024-03-28T17:40:26.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:26.603+0000 7f046c8f7700 1 -- 172.21.15.71:0/4265825825 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f04540a2800 msgr2=0x7f04540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:40:26.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:26.603+0000 7f046c8f7700 1 --2- 172.21.15.71:0/4265825825 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f04540a2800 0x7f04540a4cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0468102110 tx=0x7f045800d040 comp rx=0 tx=0).stop 2024-03-28T17:40:26.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:26.603+0000 7f046c8f7700 1 -- 172.21.15.71:0/4265825825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0468102c50 msgr2=0x7f04681a1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:40:26.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:26.603+0000 7f046c8f7700 1 --2- 172.21.15.71:0/4265825825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0468102c50 0x7f04681a1ce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0460009990 tx=0x7f04600047b0 comp rx=0 tx=0).stop 2024-03-28T17:40:26.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:26.603+0000 7f046c8f7700 1 -- 172.21.15.71:0/4265825825 shutdown_connections 2024-03-28T17:40:26.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:26.603+0000 7f046c8f7700 1 --2- 172.21.15.71:0/4265825825 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f04540a2800 0x7f04540a4cc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:26.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:26.603+0000 7f046c8f7700 1 --2- 172.21.15.71:0/4265825825 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04681010b0 0x7f046819a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:26.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:26.603+0000 7f046c8f7700 1 --2- 172.21.15.71:0/4265825825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0468102c50 0x7f04681a1ce0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:26.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:26.603+0000 7f046c8f7700 1 --2- 172.21.15.71:0/4265825825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04681022b0 0x7f046819ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:26.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:26.603+0000 7f046c8f7700 1 -- 172.21.15.71:0/4265825825 >> 172.21.15.71:0/4265825825 conn(0x7f04680fc650 msgr2=0x7f04680fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:40:26.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:26.603+0000 7f046c8f7700 1 -- 172.21.15.71:0/4265825825 shutdown_connections 2024-03-28T17:40:26.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:40:26.603+0000 7f046c8f7700 1 -- 172.21.15.71:0/4265825825 wait complete. 2024-03-28T17:40:26.790 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:26 smithi071 ceph-mon[36597]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:26.790 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:26 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:26.790 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:26 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:26.790 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:26 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:26.790 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:26 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:40:26.790 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:26 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:26.790 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:26 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:26.790 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:26 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:26.790 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:26 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:40:26.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:26 smithi045 ceph-mon[31773]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:26.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:26 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:26.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:26 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:26.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:26 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:26.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:26 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:40:26.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:26 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:26.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:26 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:26.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:26 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:26.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:26 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:40:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:26 smithi151 ceph-mon[36481]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:26 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:27.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:26 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:27.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:26 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:27.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:26 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:40:27.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:26 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:27.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:26 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:27.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:26 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:27.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:26 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:40:27.260 DEBUG:teuthology.orchestra.run.smithi071:osd.1> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@osd.1.service 2024-03-28T17:40:27.263 INFO:tasks.cephadm:Deploying osd.2 on smithi151 with /dev/vg_nvme/lv_4... 2024-03-28T17:40:27.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T17:40:27.508 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T17:40:27.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:27 smithi045 ceph-mon[31773]: from='osd.1 [v2:172.21.15.71:6800/2014529906,v1:172.21.15.71:6801/2014529906]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T17:40:27.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:27 smithi045 ceph-mon[31773]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T17:40:27.938 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:27 smithi151 ceph-mon[36481]: from='osd.1 [v2:172.21.15.71:6800/2014529906,v1:172.21.15.71:6801/2014529906]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T17:40:27.938 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:27 smithi151 ceph-mon[36481]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T17:40:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:27 smithi071 ceph-mon[36597]: from='osd.1 [v2:172.21.15.71:6800/2014529906,v1:172.21.15.71:6801/2014529906]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T17:40:28.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:27 smithi071 ceph-mon[36597]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T17:40:28.794 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:28 smithi151 ceph-mon[36481]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:28.794 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:28 smithi151 ceph-mon[36481]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T17:40:28.794 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:28 smithi151 ceph-mon[36481]: from='osd.1 [v2:172.21.15.71:6800/2014529906,v1:172.21.15.71:6801/2014529906]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:40:28.794 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:28 smithi151 ceph-mon[36481]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T17:40:28.795 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:28 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T17:40:28.795 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:28 smithi151 ceph-mon[36481]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:40:28.988 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:28 smithi071 ceph-mon[36597]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:28.988 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:28 smithi071 ceph-mon[36597]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T17:40:28.988 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:28 smithi071 ceph-mon[36597]: from='osd.1 [v2:172.21.15.71:6800/2014529906,v1:172.21.15.71:6801/2014529906]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:40:28.988 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:28 smithi071 ceph-mon[36597]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T17:40:28.988 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:28 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T17:40:28.988 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:28 smithi071 ceph-mon[36597]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:40:29.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:28 smithi045 ceph-mon[31773]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:29.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:28 smithi045 ceph-mon[31773]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T17:40:29.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:28 smithi045 ceph-mon[31773]: from='osd.1 [v2:172.21.15.71:6800/2014529906,v1:172.21.15.71:6801/2014529906]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:40:29.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:28 smithi045 ceph-mon[31773]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T17:40:29.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:28 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T17:40:29.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:28 smithi045 ceph-mon[31773]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:40:29.311 INFO:journalctl@ceph.osd.1.smithi071.stdout:Mar 28 17:40:29 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-osd-1[41195]: 2024-03-28T17:40:29.152+0000 7fee1262e700 -1 osd.1 0 waiting for initial osdmap 2024-03-28T17:40:29.311 INFO:journalctl@ceph.osd.1.smithi071.stdout:Mar 28 17:40:29 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-osd-1[41195]: 2024-03-28T17:40:29.157+0000 7fee0a419700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T17:40:29.320 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T17:40:29.320 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T17:40:29.320 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2024-03-28T17:40:29.321 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2024-03-28T17:40:29.321 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115131 s, 911 MB/s 2024-03-28T17:40:29.321 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2024-03-28T17:40:29.982 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:29 smithi151 ceph-mon[36481]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-28T17:40:29.982 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:29 smithi151 ceph-mon[36481]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T17:40:29.982 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:29 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T17:40:29.982 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:29 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T17:40:29.982 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:29 smithi151 ceph-mon[36481]: from='osd.1 ' entity='osd.1' 2024-03-28T17:40:30.011 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi151:vg_nvme/lv_4 2024-03-28T17:40:30.033 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:29 smithi071 ceph-mon[36597]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-28T17:40:30.033 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:29 smithi071 ceph-mon[36597]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T17:40:30.033 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:29 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T17:40:30.033 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:29 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T17:40:30.033 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:29 smithi071 ceph-mon[36597]: from='osd.1 ' entity='osd.1' 2024-03-28T17:40:30.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:29 smithi045 ceph-mon[31773]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-28T17:40:30.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:29 smithi045 ceph-mon[31773]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T17:40:30.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:29 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T17:40:30.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:29 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T17:40:30.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:29 smithi045 ceph-mon[31773]: from='osd.1 ' entity='osd.1' 2024-03-28T17:40:30.247 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T17:40:31.020 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:30 smithi151 ceph-mon[36481]: purged_snaps scrub starts 2024-03-28T17:40:31.021 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:30 smithi151 ceph-mon[36481]: purged_snaps scrub ok 2024-03-28T17:40:31.021 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:30 smithi151 ceph-mon[36481]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:31.021 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:30 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T17:40:31.021 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:30 smithi151 ceph-mon[36481]: osd.1 [v2:172.21.15.71:6800/2014529906,v1:172.21.15.71:6801/2014529906] boot 2024-03-28T17:40:31.021 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:30 smithi151 ceph-mon[36481]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T17:40:31.021 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:30 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T17:40:31.021 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:30 smithi151 ceph-mon[36481]: Detected new or changed devices on smithi071 2024-03-28T17:40:31.021 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:30 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:31.021 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:30 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:31.021 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:30 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:40:31.021 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:30 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi071 to 17080M 2024-03-28T17:40:31.021 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:30 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:31.021 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:30 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:31.021 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:30 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:40:31.022 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:30 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:31.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:30 smithi071 ceph-mon[36597]: purged_snaps scrub starts 2024-03-28T17:40:31.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:30 smithi071 ceph-mon[36597]: purged_snaps scrub ok 2024-03-28T17:40:31.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:30 smithi071 ceph-mon[36597]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:31.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:30 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T17:40:31.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:30 smithi071 ceph-mon[36597]: osd.1 [v2:172.21.15.71:6800/2014529906,v1:172.21.15.71:6801/2014529906] boot 2024-03-28T17:40:31.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:30 smithi071 ceph-mon[36597]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T17:40:31.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:30 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T17:40:31.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:30 smithi071 ceph-mon[36597]: Detected new or changed devices on smithi071 2024-03-28T17:40:31.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:30 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:31.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:30 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:31.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:30 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:40:31.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:30 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi071 to 17080M 2024-03-28T17:40:31.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:30 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:31.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:30 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:31.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:30 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:40:31.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:30 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:30 smithi045 ceph-mon[31773]: purged_snaps scrub starts 2024-03-28T17:40:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:30 smithi045 ceph-mon[31773]: purged_snaps scrub ok 2024-03-28T17:40:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:30 smithi045 ceph-mon[31773]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T17:40:31.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:30 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T17:40:31.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:30 smithi045 ceph-mon[31773]: osd.1 [v2:172.21.15.71:6800/2014529906,v1:172.21.15.71:6801/2014529906] boot 2024-03-28T17:40:31.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:30 smithi045 ceph-mon[31773]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T17:40:31.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:30 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T17:40:31.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:30 smithi045 ceph-mon[31773]: Detected new or changed devices on smithi071 2024-03-28T17:40:31.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:30 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:31.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:30 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:31.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:30 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:40:31.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:30 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi071 to 17080M 2024-03-28T17:40:31.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:30 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:31.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:30 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:31.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:30 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:40:31.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:30 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:31.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.761+0000 7f842f04a700 1 -- 172.21.15.151:0/4277381804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84281010e0 msgr2=0x7f8428101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:40:31.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.761+0000 7f842f04a700 1 --2- 172.21.15.151:0/4277381804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84281010e0 0x7f8428101500 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f841c009990 tx=0x7f841c009ca0 comp rx=0 tx=0).stop 2024-03-28T17:40:31.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.761+0000 7f842f04a700 1 -- 172.21.15.151:0/4277381804 shutdown_connections 2024-03-28T17:40:31.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.761+0000 7f842f04a700 1 --2- 172.21.15.151:0/4277381804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84281010e0 0x7f8428101500 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:31.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.761+0000 7f842f04a700 1 --2- 172.21.15.151:0/4277381804 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8428102c80 0x7f8428109530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:31.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.761+0000 7f842f04a700 1 --2- 172.21.15.151:0/4277381804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84281022e0 0x7f8428102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:31.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.761+0000 7f842f04a700 1 -- 172.21.15.151:0/4277381804 >> 172.21.15.151:0/4277381804 conn(0x7f84280fc680 msgr2=0x7f84280feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:40:31.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.761+0000 7f842f04a700 1 -- 172.21.15.151:0/4277381804 shutdown_connections 2024-03-28T17:40:31.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.761+0000 7f842f04a700 1 -- 172.21.15.151:0/4277381804 wait complete. 2024-03-28T17:40:31.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.762+0000 7f842f04a700 1 Processor -- start 2024-03-28T17:40:31.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.762+0000 7f842f04a700 1 -- start start 2024-03-28T17:40:31.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.762+0000 7f842f04a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84281010e0 0x7f8428197450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:40:31.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.762+0000 7f842f04a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84281022e0 0x7f8428197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:40:31.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.762+0000 7f842f04a700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8428102c80 0x7f842819ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:40:31.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.762+0000 7f842f04a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842810cda0 con 0x7f84281010e0 2024-03-28T17:40:31.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.762+0000 7f842f04a700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842810cc20 con 0x7f8428102c80 2024-03-28T17:40:31.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.762+0000 7f842f04a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842810cf20 con 0x7f84281022e0 2024-03-28T17:40:31.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.762+0000 7f8427fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84281022e0 0x7f8428197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:40:31.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.762+0000 7f8427fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84281022e0 0x7f8428197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40654/0 (socket says 172.21.15.151:40654) 2024-03-28T17:40:31.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.762+0000 7f8427fff700 1 -- 172.21.15.151:0/1827366690 learned_addr learned my addr 172.21.15.151:0/1827366690 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-03-28T17:40:31.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.762+0000 7f8427fff700 1 -- 172.21.15.151:0/1827366690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8428102c80 msgr2=0x7f842819ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:40:31.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.762+0000 7f842d5e7700 1 --2- 172.21.15.151:0/1827366690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8428102c80 0x7f842819ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:40:31.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.762+0000 7f8427fff700 1 --2- 172.21.15.151:0/1827366690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8428102c80 0x7f842819ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:31.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.762+0000 7f8427fff700 1 -- 172.21.15.151:0/1827366690 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84281010e0 msgr2=0x7f8428197450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:40:31.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.762+0000 7f8427fff700 1 --2- 172.21.15.151:0/1827366690 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84281010e0 0x7f8428197450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:31.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.762+0000 7f8427fff700 1 -- 172.21.15.151:0/1827366690 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f842819f0b0 con 0x7f84281022e0 2024-03-28T17:40:31.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.762+0000 7f8427fff700 1 --2- 172.21.15.151:0/1827366690 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84281022e0 0x7f8428197990 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f841400b560 tx=0x7f841400b920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:40:31.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.763+0000 7f8425ffb700 1 -- 172.21.15.151:0/1827366690 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8414015020 con 0x7f84281022e0 2024-03-28T17:40:31.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.763+0000 7f8425ffb700 1 -- 172.21.15.151:0/1827366690 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8414011d10 con 0x7f84281022e0 2024-03-28T17:40:31.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.763+0000 7f8425ffb700 1 -- 172.21.15.151:0/1827366690 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841400f2e0 con 0x7f84281022e0 2024-03-28T17:40:31.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.763+0000 7f842f04a700 1 -- 172.21.15.151:0/1827366690 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f842819f3a0 con 0x7f84281022e0 2024-03-28T17:40:31.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.763+0000 7f842f04a700 1 -- 172.21.15.151:0/1827366690 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f842819f9b0 con 0x7f84281022e0 2024-03-28T17:40:31.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.764+0000 7f842f04a700 1 -- 172.21.15.151:0/1827366690 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8428068e50 con 0x7f84281022e0 2024-03-28T17:40:31.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.765+0000 7f8425ffb700 1 -- 172.21.15.151:0/1827366690 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f8414018020 con 0x7f84281022e0 2024-03-28T17:40:31.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.765+0000 7f8425ffb700 1 --2- 172.21.15.151:0/1827366690 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f84180a2880 0x7f84180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:40:31.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.765+0000 7f8425ffb700 1 -- 172.21.15.151:0/1827366690 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f84140cc990 con 0x7f84281022e0 2024-03-28T17:40:31.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.766+0000 7f842cde6700 1 --2- 172.21.15.151:0/1827366690 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f84180a2880 0x7f84180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:40:31.768 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.767+0000 7f842cde6700 1 --2- 172.21.15.151:0/1827366690 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f84180a2880 0x7f84180a4d40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f841c0098e0 tx=0x7f841c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:40:31.770 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.770+0000 7f8425ffb700 1 -- 172.21.15.151:0/1827366690 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f841401f020 con 0x7f84281022e0 2024-03-28T17:40:31.974 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:31.973+0000 7f842f04a700 1 -- 172.21.15.151:0/1827366690 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f84280625d0 con 0x7f84180a2880 2024-03-28T17:40:32.877 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:32 smithi151 ceph-mon[36481]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:32.877 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:32 smithi151 ceph-mon[36481]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T17:40:32.877 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:32 smithi151 ceph-mon[36481]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:40:32.877 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:32 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:40:32.878 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:32 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:40:32.878 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:32 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:32 smithi071 ceph-mon[36597]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:32 smithi071 ceph-mon[36597]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T17:40:33.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:32 smithi071 ceph-mon[36597]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:40:33.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:32 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:40:33.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:32 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:40:33.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:32 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:33.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:32 smithi045 ceph-mon[31773]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:33.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:32 smithi045 ceph-mon[31773]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T17:40:33.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:32 smithi045 ceph-mon[31773]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:40:33.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:32 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:40:33.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:32 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:40:33.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:32 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:35.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:34 smithi071 ceph-mon[36597]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:35.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:34 smithi071 ceph-mon[36597]: from='client.? 172.21.15.151:0/3795719583' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca9f3f36-b71e-4962-a94f-cedf2f12eb7f"}]: dispatch 2024-03-28T17:40:35.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:34 smithi071 ceph-mon[36597]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca9f3f36-b71e-4962-a94f-cedf2f12eb7f"}]: dispatch 2024-03-28T17:40:35.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:34 smithi071 ceph-mon[36597]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca9f3f36-b71e-4962-a94f-cedf2f12eb7f"}]': finished 2024-03-28T17:40:35.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:34 smithi071 ceph-mon[36597]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T17:40:35.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:34 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T17:40:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:34 smithi151 ceph-mon[36481]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:34 smithi151 ceph-mon[36481]: from='client.? 172.21.15.151:0/3795719583' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca9f3f36-b71e-4962-a94f-cedf2f12eb7f"}]: dispatch 2024-03-28T17:40:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:34 smithi151 ceph-mon[36481]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca9f3f36-b71e-4962-a94f-cedf2f12eb7f"}]: dispatch 2024-03-28T17:40:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:34 smithi151 ceph-mon[36481]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca9f3f36-b71e-4962-a94f-cedf2f12eb7f"}]': finished 2024-03-28T17:40:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:34 smithi151 ceph-mon[36481]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T17:40:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:34 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T17:40:35.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:34 smithi045 ceph-mon[31773]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:35.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:34 smithi045 ceph-mon[31773]: from='client.? 172.21.15.151:0/3795719583' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca9f3f36-b71e-4962-a94f-cedf2f12eb7f"}]: dispatch 2024-03-28T17:40:35.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:34 smithi045 ceph-mon[31773]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca9f3f36-b71e-4962-a94f-cedf2f12eb7f"}]: dispatch 2024-03-28T17:40:35.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:34 smithi045 ceph-mon[31773]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca9f3f36-b71e-4962-a94f-cedf2f12eb7f"}]': finished 2024-03-28T17:40:35.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:34 smithi045 ceph-mon[31773]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T17:40:35.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:34 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T17:40:36.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:35 smithi071 ceph-mon[36597]: from='client.? 172.21.15.151:0/3758203777' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:40:36.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:35 smithi151 ceph-mon[36481]: from='client.? 172.21.15.151:0/3758203777' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:40:36.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:35 smithi045 ceph-mon[31773]: from='client.? 172.21.15.151:0/3758203777' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:40:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:36 smithi071 ceph-mon[36597]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:37.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:36 smithi151 ceph-mon[36481]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:36 smithi045 ceph-mon[31773]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:39.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:38 smithi071 ceph-mon[36597]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:39.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:38 smithi151 ceph-mon[36481]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:39.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:38 smithi045 ceph-mon[31773]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:40.969 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:40 smithi045 ceph-mon[31773]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:41.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:40 smithi071 ceph-mon[36597]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:41.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:40 smithi151 ceph-mon[36481]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:41 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T17:40:42.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:41 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:42.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:41 smithi071 ceph-mon[36597]: Deploying daemon osd.2 on smithi151 2024-03-28T17:40:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:41 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T17:40:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:41 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:41 smithi151 ceph-mon[36481]: Deploying daemon osd.2 on smithi151 2024-03-28T17:40:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:41 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T17:40:42.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:41 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:42.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:41 smithi045 ceph-mon[31773]: Deploying daemon osd.2 on smithi151 2024-03-28T17:40:42.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:42 smithi045 ceph-mon[31773]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:43.031 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:42 smithi151 ceph-mon[36481]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:42 smithi071 ceph-mon[36597]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:44.962 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:44 smithi151 ceph-mon[36481]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:45.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:44 smithi071 ceph-mon[36597]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:45.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:44 smithi045 ceph-mon[31773]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:46 smithi071 ceph-mon[36597]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:46 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:47.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:46 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:47.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:46 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:40:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:46 smithi151 ceph-mon[36481]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:46 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:46 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:46 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:40:47.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:46 smithi045 ceph-mon[31773]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:47.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:46 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:47.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:46 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:47.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:46 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:40:48.901 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:48 smithi151 ceph-mon[36481]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:49.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:48 smithi071 ceph-mon[36597]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:49.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:48 smithi045 ceph-mon[31773]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:50.457 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:50.457+0000 7f8425ffb700 1 -- 172.21.15.151:0/1827366690 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f84280625d0 con 0x7f84180a2880 2024-03-28T17:40:50.458 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 2 on host 'smithi151' 2024-03-28T17:40:50.460 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:50.459+0000 7f842f04a700 1 -- 172.21.15.151:0/1827366690 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f84180a2880 msgr2=0x7f84180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:40:50.460 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:50.459+0000 7f842f04a700 1 --2- 172.21.15.151:0/1827366690 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f84180a2880 0x7f84180a4d40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f841c0098e0 tx=0x7f841c005fb0 comp rx=0 tx=0).stop 2024-03-28T17:40:50.460 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:50.459+0000 7f842f04a700 1 -- 172.21.15.151:0/1827366690 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84281022e0 msgr2=0x7f8428197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:40:50.460 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:50.459+0000 7f842f04a700 1 --2- 172.21.15.151:0/1827366690 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84281022e0 0x7f8428197990 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f841400b560 tx=0x7f841400b920 comp rx=0 tx=0).stop 2024-03-28T17:40:50.460 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:50.459+0000 7f842f04a700 1 -- 172.21.15.151:0/1827366690 shutdown_connections 2024-03-28T17:40:50.460 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:50.459+0000 7f842f04a700 1 --2- 172.21.15.151:0/1827366690 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f84180a2880 0x7f84180a4d40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:50.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:50.459+0000 7f842f04a700 1 --2- 172.21.15.151:0/1827366690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8428102c80 0x7f842819ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:50.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:50.459+0000 7f842f04a700 1 --2- 172.21.15.151:0/1827366690 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84281010e0 0x7f8428197450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:50.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:50.459+0000 7f842f04a700 1 --2- 172.21.15.151:0/1827366690 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84281022e0 0x7f8428197990 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:50.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:50.459+0000 7f842f04a700 1 -- 172.21.15.151:0/1827366690 >> 172.21.15.151:0/1827366690 conn(0x7f84280fc680 msgr2=0x7f8428105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:40:50.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:50.459+0000 7f842f04a700 1 -- 172.21.15.151:0/1827366690 shutdown_connections 2024-03-28T17:40:50.461 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:40:50.459+0000 7f842f04a700 1 -- 172.21.15.151:0/1827366690 wait complete. 2024-03-28T17:40:50.759 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:50 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:50.759 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:50 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:50.759 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:50 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:50.759 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:50 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:40:50.759 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:50 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:50.759 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:50 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:50.759 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:50 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:50.760 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:50 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:40:50.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:50 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:50.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:50 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:50.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:50 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:50.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:50 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:40:50.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:50 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:50.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:50 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:50.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:50 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:50.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:50 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:40:50.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:50 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:50.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:50 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:50.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:50 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:50.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:50 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:40:50.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:50 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:50.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:50 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:50.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:50 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:50.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:50 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:40:51.094 DEBUG:teuthology.orchestra.run.smithi151:osd.2> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@osd.2.service 2024-03-28T17:40:51.096 INFO:tasks.cephadm:Deploying osd.3 on smithi045 with /dev/vg_nvme/lv_3... 2024-03-28T17:40:51.097 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T17:40:51.567 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:40:51.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:51 smithi151 ceph-mon[36481]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:51.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:51 smithi151 ceph-mon[36481]: from='osd.2 [v2:172.21.15.151:6800/1850933775,v1:172.21.15.151:6801/1850933775]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T17:40:51.613 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:51 smithi151 ceph-mon[36481]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T17:40:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:51 smithi071 ceph-mon[36597]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:51.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:51 smithi071 ceph-mon[36597]: from='osd.2 [v2:172.21.15.151:6800/1850933775,v1:172.21.15.151:6801/1850933775]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T17:40:51.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:51 smithi071 ceph-mon[36597]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T17:40:51.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:51 smithi045 ceph-mon[31773]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:51.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:51 smithi045 ceph-mon[31773]: from='osd.2 [v2:172.21.15.151:6800/1850933775,v1:172.21.15.151:6801/1850933775]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T17:40:51.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:51 smithi045 ceph-mon[31773]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T17:40:52.731 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:52 smithi045 ceph-mon[31773]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T17:40:52.732 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:52 smithi045 ceph-mon[31773]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T17:40:52.732 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:52 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T17:40:52.732 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:52 smithi045 ceph-mon[31773]: from='osd.2 [v2:172.21.15.151:6800/1850933775,v1:172.21.15.151:6801/1850933775]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:40:52.732 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:52 smithi045 ceph-mon[31773]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:40:52.732 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:52 smithi045 ceph-mon[31773]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:52.805 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:52 smithi151 ceph-mon[36481]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T17:40:52.805 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:52 smithi151 ceph-mon[36481]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T17:40:52.806 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:52 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T17:40:52.806 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:52 smithi151 ceph-mon[36481]: from='osd.2 [v2:172.21.15.151:6800/1850933775,v1:172.21.15.151:6801/1850933775]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:40:52.806 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:52 smithi151 ceph-mon[36481]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:40:52.806 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:52 smithi151 ceph-mon[36481]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:52.806 INFO:journalctl@ceph.osd.2.smithi151.stdout:Mar 28 17:40:52 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-osd-2[41636]: 2024-03-28T17:40:52.566+0000 7f5a2ac8e700 -1 osd.2 0 waiting for initial osdmap 2024-03-28T17:40:52.806 INFO:journalctl@ceph.osd.2.smithi151.stdout:Mar 28 17:40:52 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-osd-2[41636]: 2024-03-28T17:40:52.572+0000 7f5a26a81700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T17:40:53.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:52 smithi071 ceph-mon[36597]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T17:40:53.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:52 smithi071 ceph-mon[36597]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T17:40:53.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:52 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T17:40:53.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:52 smithi071 ceph-mon[36597]: from='osd.2 [v2:172.21.15.151:6800/1850933775,v1:172.21.15.151:6801/1850933775]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:40:53.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:52 smithi071 ceph-mon[36597]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:40:53.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:52 smithi071 ceph-mon[36597]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:53.765 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:53 smithi151 ceph-mon[36481]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-28T17:40:53.765 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:53 smithi151 ceph-mon[36481]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T17:40:53.765 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:53 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T17:40:53.765 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:53 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T17:40:53.765 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:53 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T17:40:53.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:53 smithi045 ceph-mon[31773]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-28T17:40:53.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:53 smithi045 ceph-mon[31773]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T17:40:53.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:53 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T17:40:53.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:53 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T17:40:53.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:53 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T17:40:54.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:53 smithi071 ceph-mon[36597]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-28T17:40:54.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:53 smithi071 ceph-mon[36597]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T17:40:54.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:53 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T17:40:54.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:53 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T17:40:54.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:53 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T17:40:54.310 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T17:40:54.311 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T17:40:54.311 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2024-03-28T17:40:54.311 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2024-03-28T17:40:54.311 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.032966 s, 318 MB/s 2024-03-28T17:40:54.311 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2024-03-28T17:40:54.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:54 smithi045 ceph-mon[31773]: purged_snaps scrub starts 2024-03-28T17:40:54.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:54 smithi045 ceph-mon[31773]: purged_snaps scrub ok 2024-03-28T17:40:54.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:54 smithi045 ceph-mon[31773]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:54.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:54 smithi045 ceph-mon[31773]: osd.2 [v2:172.21.15.151:6800/1850933775,v1:172.21.15.151:6801/1850933775] boot 2024-03-28T17:40:54.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:54 smithi045 ceph-mon[31773]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T17:40:54.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:54 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T17:40:54.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:54 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T17:40:54.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:54 smithi045 ceph-mon[31773]: Detected new or changed devices on smithi151 2024-03-28T17:40:54.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:54 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:54.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:54 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:54.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:54 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:40:54.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:54 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi151 to 17080M 2024-03-28T17:40:54.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:54 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:54.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:54 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:54.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:54 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:40:54.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:54 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:55.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:54 smithi071 ceph-mon[36597]: purged_snaps scrub starts 2024-03-28T17:40:55.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:54 smithi071 ceph-mon[36597]: purged_snaps scrub ok 2024-03-28T17:40:55.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:54 smithi071 ceph-mon[36597]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:55.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:54 smithi071 ceph-mon[36597]: osd.2 [v2:172.21.15.151:6800/1850933775,v1:172.21.15.151:6801/1850933775] boot 2024-03-28T17:40:55.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:54 smithi071 ceph-mon[36597]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T17:40:55.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:54 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T17:40:55.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:54 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T17:40:55.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:54 smithi071 ceph-mon[36597]: Detected new or changed devices on smithi151 2024-03-28T17:40:55.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:54 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:55.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:54 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:55.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:54 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:40:55.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:54 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi151 to 17080M 2024-03-28T17:40:55.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:54 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:55.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:54 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:55.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:54 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:40:55.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:54 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:55.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:54 smithi151 ceph-mon[36481]: purged_snaps scrub starts 2024-03-28T17:40:55.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:54 smithi151 ceph-mon[36481]: purged_snaps scrub ok 2024-03-28T17:40:55.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:54 smithi151 ceph-mon[36481]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T17:40:55.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:54 smithi151 ceph-mon[36481]: osd.2 [v2:172.21.15.151:6800/1850933775,v1:172.21.15.151:6801/1850933775] boot 2024-03-28T17:40:55.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:54 smithi151 ceph-mon[36481]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T17:40:55.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:54 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T17:40:55.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:54 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T17:40:55.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:54 smithi151 ceph-mon[36481]: Detected new or changed devices on smithi151 2024-03-28T17:40:55.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:54 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:55.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:54 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:55.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:54 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:40:55.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:54 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi151 to 17080M 2024-03-28T17:40:55.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:54 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:55.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:54 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:55.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:54 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:40:55.075 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:54 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:55.097 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi045:vg_nvme/lv_3 2024-03-28T17:40:55.553 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:40:55.907 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:55 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T17:40:55.907 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:55 smithi045 ceph-mon[31773]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T17:40:55.907 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:55 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T17:40:56.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:55 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T17:40:56.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:55 smithi071 ceph-mon[36597]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T17:40:56.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:55 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T17:40:56.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:55 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T17:40:56.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:55 smithi151 ceph-mon[36481]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T17:40:56.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:55 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T17:40:56.486 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 28 17:40:56 smithi045 sudo[50603]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T17:40:56.486 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 28 17:40:56 smithi045 sudo[50603]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T17:40:56.486 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 28 17:40:56 smithi045 sudo[50603]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T17:40:56.486 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 28 17:40:56 smithi045 sudo[50603]: pam_unix(sudo:session): session closed for user root 2024-03-28T17:40:56.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:56 smithi045 ceph-mon[31773]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:40:56.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:56 smithi045 ceph-mon[31773]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T17:40:56.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:56 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T17:40:56.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:56 smithi045 ceph-mon[31773]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T17:40:57.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:56 smithi071 ceph-mon[36597]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:40:57.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:56 smithi071 ceph-mon[36597]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T17:40:57.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:56 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T17:40:57.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:56 smithi071 ceph-mon[36597]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T17:40:57.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:56 smithi151 ceph-mon[36481]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:40:57.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:56 smithi151 ceph-mon[36481]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T17:40:57.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:56 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T17:40:57.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:56 smithi151 ceph-mon[36481]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T17:40:57.118 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 28 17:40:56 smithi045 sudo[50612]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T17:40:57.119 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 28 17:40:56 smithi045 sudo[50612]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T17:40:57.119 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 28 17:40:56 smithi045 sudo[50612]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T17:40:57.119 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 28 17:40:56 smithi045 sudo[50612]: pam_unix(sudo:session): session closed for user root 2024-03-28T17:40:57.488 INFO:journalctl@ceph.osd.1.smithi071.stdout:Mar 28 17:40:57 smithi071 sudo[45352]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T17:40:57.488 INFO:journalctl@ceph.osd.1.smithi071.stdout:Mar 28 17:40:57 smithi071 sudo[45352]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T17:40:57.488 INFO:journalctl@ceph.osd.1.smithi071.stdout:Mar 28 17:40:57 smithi071 sudo[45352]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T17:40:57.488 INFO:journalctl@ceph.osd.1.smithi071.stdout:Mar 28 17:40:57 smithi071 sudo[45352]: pam_unix(sudo:session): session closed for user root 2024-03-28T17:40:57.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:57 smithi045 ceph-mon[31773]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T17:40:57.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:57 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:57.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:57 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.674+0000 7f3284800700 1 -- 172.21.15.45:0/1717483592 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f327c105610 msgr2=0x7f327c105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:40:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.674+0000 7f3284800700 1 --2- 172.21.15.45:0/1717483592 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f327c105610 0x7f327c105a70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f326c0099c0 tx=0x7f326c009cd0 comp rx=0 tx=0).stop 2024-03-28T17:40:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.674+0000 7f3284800700 1 -- 172.21.15.45:0/1717483592 shutdown_connections 2024-03-28T17:40:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.674+0000 7f3284800700 1 --2- 172.21.15.45:0/1717483592 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f327c104410 0x7f327c104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.674+0000 7f3284800700 1 --2- 172.21.15.45:0/1717483592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f327c105fb0 0x7f327c10c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.674+0000 7f3284800700 1 --2- 172.21.15.45:0/1717483592 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f327c105610 0x7f327c105a70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.674+0000 7f3284800700 1 -- 172.21.15.45:0/1717483592 >> 172.21.15.45:0/1717483592 conn(0x7f327c0ff9b0 msgr2=0x7f327c101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:40:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.674+0000 7f3284800700 1 -- 172.21.15.45:0/1717483592 shutdown_connections 2024-03-28T17:40:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.674+0000 7f3284800700 1 -- 172.21.15.45:0/1717483592 wait complete. 2024-03-28T17:40:57.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.675+0000 7f3284800700 1 Processor -- start 2024-03-28T17:40:57.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.676+0000 7f3284800700 1 -- start start 2024-03-28T17:40:57.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.676+0000 7f3284800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f327c104410 0x7f327c19a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:40:57.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.676+0000 7f3284800700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f327c105610 0x7f327c19ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:40:57.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.676+0000 7f3284800700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f327c105fb0 0x7f327c1a1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:40:57.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.676+0000 7f3284800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327c110030 con 0x7f327c104410 2024-03-28T17:40:57.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.676+0000 7f3284800700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327c10feb0 con 0x7f327c105610 2024-03-28T17:40:57.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.676+0000 7f3284800700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327c1101b0 con 0x7f327c105fb0 2024-03-28T17:40:57.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.676+0000 7f3281e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f327c104410 0x7f327c19a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:40:57.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.677+0000 7f3281400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f327c105610 0x7f327c19ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:40:57.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.677+0000 7f3281400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f327c105610 0x7f327c19ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:60764/0 (socket says 172.21.15.45:60764) 2024-03-28T17:40:57.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.677+0000 7f3281400700 1 -- 172.21.15.45:0/2977770301 learned_addr learned my addr 172.21.15.45:0/2977770301 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:40:57.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.677+0000 7f3282800700 1 --2- 172.21.15.45:0/2977770301 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f327c105fb0 0x7f327c1a1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:40:57.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.677+0000 7f3281400700 1 -- 172.21.15.45:0/2977770301 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f327c105fb0 msgr2=0x7f327c1a1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:40:57.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.677+0000 7f3281400700 1 --2- 172.21.15.45:0/2977770301 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f327c105fb0 0x7f327c1a1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:57.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.677+0000 7f3281400700 1 -- 172.21.15.45:0/2977770301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f327c104410 msgr2=0x7f327c19a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:40:57.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.677+0000 7f3281400700 1 --2- 172.21.15.45:0/2977770301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f327c104410 0x7f327c19a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:40:57.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.677+0000 7f3281400700 1 -- 172.21.15.45:0/2977770301 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327c1a2270 con 0x7f327c105610 2024-03-28T17:40:57.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.677+0000 7f3282800700 1 --2- 172.21.15.45:0/2977770301 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f327c105fb0 0x7f327c1a1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:40:57.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.677+0000 7f3281400700 1 --2- 172.21.15.45:0/2977770301 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f327c105610 0x7f327c19ab50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f326c005b40 tx=0x7f326c009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:40:57.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.678+0000 7f3276a00700 1 -- 172.21.15.45:0/2977770301 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c027070 con 0x7f327c105610 2024-03-28T17:40:57.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.678+0000 7f3284800700 1 -- 172.21.15.45:0/2977770301 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f327c1a2500 con 0x7f327c105610 2024-03-28T17:40:57.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.678+0000 7f3276a00700 1 -- 172.21.15.45:0/2977770301 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f326c00fb40 con 0x7f327c105610 2024-03-28T17:40:57.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.678+0000 7f3276a00700 1 -- 172.21.15.45:0/2977770301 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c017650 con 0x7f327c105610 2024-03-28T17:40:57.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.678+0000 7f3284800700 1 -- 172.21.15.45:0/2977770301 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f327c1a29b0 con 0x7f327c105610 2024-03-28T17:40:57.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.679+0000 7f3284800700 1 -- 172.21.15.45:0/2977770301 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f327c068e50 con 0x7f327c105610 2024-03-28T17:40:57.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.681+0000 7f3276a00700 1 -- 172.21.15.45:0/2977770301 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f326c02a070 con 0x7f327c105610 2024-03-28T17:40:57.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.682+0000 7f3276a00700 1 --2- 172.21.15.45:0/2977770301 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f32640a2800 0x7f32640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:40:57.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.682+0000 7f3276a00700 1 -- 172.21.15.45:0/2977770301 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3237+0+0 (secure 0 0 0) 0x7f326c0ce530 con 0x7f327c105610 2024-03-28T17:40:57.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.682+0000 7f3281e00700 1 --2- 172.21.15.45:0/2977770301 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f32640a2800 0x7f32640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:40:57.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.683+0000 7f3281e00700 1 --2- 172.21.15.45:0/2977770301 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f32640a2800 0x7f32640a4cc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f327c105470 tx=0x7f3270009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:40:57.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.686+0000 7f3276a00700 1 -- 172.21.15.45:0/2977770301 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f326c05e740 con 0x7f327c105610 2024-03-28T17:40:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:57 smithi071 ceph-mon[36597]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T17:40:57.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:57 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:57.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:57 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:57.811 INFO:journalctl@ceph.osd.1.smithi071.stdout:Mar 28 17:40:57 smithi071 sudo[45355]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T17:40:57.811 INFO:journalctl@ceph.osd.1.smithi071.stdout:Mar 28 17:40:57 smithi071 sudo[45355]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T17:40:57.811 INFO:journalctl@ceph.osd.1.smithi071.stdout:Mar 28 17:40:57 smithi071 sudo[45355]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T17:40:57.811 INFO:journalctl@ceph.osd.1.smithi071.stdout:Mar 28 17:40:57 smithi071 sudo[45355]: pam_unix(sudo:session): session closed for user root 2024-03-28T17:40:57.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:57.913+0000 7f3284800700 1 -- 172.21.15.45:0/2977770301 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f327c062690 con 0x7f32640a2800 2024-03-28T17:40:58.032 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:57 smithi151 ceph-mon[36481]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T17:40:58.032 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:57 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:58.032 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:57 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:58.033 INFO:journalctl@ceph.osd.2.smithi151.stdout:Mar 28 17:40:57 smithi151 sudo[45283]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T17:40:58.033 INFO:journalctl@ceph.osd.2.smithi151.stdout:Mar 28 17:40:57 smithi151 sudo[45283]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T17:40:58.033 INFO:journalctl@ceph.osd.2.smithi151.stdout:Mar 28 17:40:57 smithi151 sudo[45283]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T17:40:58.033 INFO:journalctl@ceph.osd.2.smithi151.stdout:Mar 28 17:40:57 smithi151 sudo[45283]: pam_unix(sudo:session): session closed for user root 2024-03-28T17:40:58.323 INFO:journalctl@ceph.osd.2.smithi151.stdout:Mar 28 17:40:58 smithi151 sudo[45286]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T17:40:58.323 INFO:journalctl@ceph.osd.2.smithi151.stdout:Mar 28 17:40:58 smithi151 sudo[45286]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T17:40:58.323 INFO:journalctl@ceph.osd.2.smithi151.stdout:Mar 28 17:40:58 smithi151 sudo[45286]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T17:40:58.323 INFO:journalctl@ceph.osd.2.smithi151.stdout:Mar 28 17:40:58 smithi151 sudo[45286]: pam_unix(sudo:session): session closed for user root 2024-03-28T17:40:58.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:58 smithi045 sudo[50751]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T17:40:58.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:58 smithi045 sudo[50751]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T17:40:58.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:58 smithi045 sudo[50751]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T17:40:58.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:58 smithi045 sudo[50751]: pam_unix(sudo:session): session closed for user root 2024-03-28T17:40:59.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:40:59.239+0000 7f3276a00700 1 -- 172.21.15.45:0/2977770301 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f326c00fcf0 con 0x7f327c105610 2024-03-28T17:40:59.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:59 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:40:59.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:59 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:40:59.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:59 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:59.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:59 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:59.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:59 smithi071 ceph-mon[36597]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T17:40:59.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:59 smithi045 sudo[50800]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T17:40:59.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:59 smithi045 sudo[50800]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T17:40:59.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:59 smithi045 sudo[50800]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T17:40:59.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:59 smithi045 sudo[50800]: pam_unix(sudo:session): session closed for user root 2024-03-28T17:40:59.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:40:59.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:40:59.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:40:59.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:40:59.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:40:59 smithi045 ceph-mon[31773]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T17:40:59.725 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:59 smithi151 sudo[45289]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T17:40:59.725 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:59 smithi151 sudo[45289]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T17:40:59.725 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:59 smithi151 sudo[45289]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T17:40:59.725 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:59 smithi151 sudo[45289]: pam_unix(sudo:session): session closed for user root 2024-03-28T17:41:00.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:59 smithi151 sudo[45292]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T17:41:00.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:59 smithi151 sudo[45292]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T17:41:00.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:59 smithi151 sudo[45292]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T17:41:00.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:59 smithi151 sudo[45292]: pam_unix(sudo:session): session closed for user root 2024-03-28T17:41:00.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:59 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:41:00.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:59 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:41:00.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:59 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:00.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:59 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:00.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:40:59 smithi151 ceph-mon[36481]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T17:41:00.221 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:59 smithi071 sudo[45358]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T17:41:00.221 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:59 smithi071 sudo[45358]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T17:41:00.221 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:59 smithi071 sudo[45358]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T17:41:00.221 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:40:59 smithi071 sudo[45358]: pam_unix(sudo:session): session closed for user root 2024-03-28T17:41:00.221 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:00 smithi071 sudo[45361]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T17:41:00.221 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:00 smithi071 sudo[45361]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T17:41:00.221 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:00 smithi071 sudo[45361]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T17:41:00.221 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:00 smithi071 sudo[45361]: pam_unix(sudo:session): session closed for user root 2024-03-28T17:41:00.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:00 smithi045 ceph-mon[31773]: pgmap v96: 1 pgs: 1 active+clean; 257 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:00.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:00 smithi045 ceph-mon[31773]: from='client.24182 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:41:00.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:00 smithi045 ceph-mon[31773]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T17:41:00.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:00 smithi045 ceph-mon[31773]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T17:41:00.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:00 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:41:00.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:00 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T17:41:00.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:00 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:41:00.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:00 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:41:00.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:00 smithi045 ceph-mon[31773]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T17:41:00.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:00 smithi045 ceph-mon[31773]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T17:41:00.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:00 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T17:41:00.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:00 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:41:00.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:00 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:41:00.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:00 smithi045 ceph-mon[31773]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T17:41:00.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:00 smithi045 ceph-mon[31773]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T17:41:00.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:00 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T17:41:00.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:00 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:41:00.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:00 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:41:00.542 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:00 smithi045 ceph-mon[31773]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T17:41:00.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:00 smithi071 ceph-mon[36597]: pgmap v96: 1 pgs: 1 active+clean; 257 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:00.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:00 smithi071 ceph-mon[36597]: from='client.24182 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:41:00.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:00 smithi071 ceph-mon[36597]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T17:41:00.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:00 smithi071 ceph-mon[36597]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T17:41:00.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:00 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:41:00.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:00 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T17:41:00.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:00 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:41:00.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:00 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:41:00.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:00 smithi071 ceph-mon[36597]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T17:41:00.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:00 smithi071 ceph-mon[36597]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T17:41:00.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:00 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T17:41:00.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:00 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:41:00.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:00 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:41:00.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:00 smithi071 ceph-mon[36597]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T17:41:00.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:00 smithi071 ceph-mon[36597]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T17:41:00.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:00 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T17:41:00.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:00 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:41:00.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:00 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:41:00.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:00 smithi071 ceph-mon[36597]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T17:41:00.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:00 smithi151 ceph-mon[36481]: pgmap v96: 1 pgs: 1 active+clean; 257 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:00.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:00 smithi151 ceph-mon[36481]: from='client.24182 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:41:00.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:00 smithi151 ceph-mon[36481]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T17:41:00.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:00 smithi151 ceph-mon[36481]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T17:41:00.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:00 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:41:00.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:00 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T17:41:00.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:00 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:41:00.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:00 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:41:00.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:00 smithi151 ceph-mon[36481]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T17:41:00.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:00 smithi151 ceph-mon[36481]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T17:41:00.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:00 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T17:41:00.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:00 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:41:00.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:00 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:41:00.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:00 smithi151 ceph-mon[36481]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T17:41:00.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:00 smithi151 ceph-mon[36481]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T17:41:00.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:00 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T17:41:00.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:00 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:41:00.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:00 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:41:00.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:00 smithi151 ceph-mon[36481]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T17:41:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:01 smithi071 ceph-mon[36597]: pgmap v97: 1 pgs: 1 active+clean; 257 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:01 smithi151 ceph-mon[36481]: pgmap v97: 1 pgs: 1 active+clean; 257 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:01 smithi045 ceph-mon[31773]: pgmap v97: 1 pgs: 1 active+clean; 257 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:02.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:02 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1210902663' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d3f21f01-2973-413e-8daa-9974b8a44af5"}]: dispatch 2024-03-28T17:41:02.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:02 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1210902663' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d3f21f01-2973-413e-8daa-9974b8a44af5"}]': finished 2024-03-28T17:41:02.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:02 smithi071 ceph-mon[36597]: osdmap e22: 4 total, 3 up, 4 in 2024-03-28T17:41:02.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:02 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T17:41:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:02 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1210902663' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d3f21f01-2973-413e-8daa-9974b8a44af5"}]: dispatch 2024-03-28T17:41:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:02 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1210902663' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d3f21f01-2973-413e-8daa-9974b8a44af5"}]': finished 2024-03-28T17:41:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:02 smithi151 ceph-mon[36481]: osdmap e22: 4 total, 3 up, 4 in 2024-03-28T17:41:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:02 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T17:41:02.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:02 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1210902663' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d3f21f01-2973-413e-8daa-9974b8a44af5"}]: dispatch 2024-03-28T17:41:02.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:02 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1210902663' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d3f21f01-2973-413e-8daa-9974b8a44af5"}]': finished 2024-03-28T17:41:02.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:02 smithi045 ceph-mon[31773]: osdmap e22: 4 total, 3 up, 4 in 2024-03-28T17:41:02.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:02 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T17:41:03.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:03 smithi071 ceph-mon[36597]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 15 KiB/s rd, 516 KiB/s wr, 41 op/s 2024-03-28T17:41:03.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:03 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1662996827' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:41:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:03 smithi151 ceph-mon[36481]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 15 KiB/s rd, 516 KiB/s wr, 41 op/s 2024-03-28T17:41:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:03 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1662996827' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:41:03.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:03 smithi045 ceph-mon[31773]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 15 KiB/s rd, 516 KiB/s wr, 41 op/s 2024-03-28T17:41:03.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:03 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1662996827' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:41:05.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:05 smithi071 ceph-mon[36597]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 11 KiB/s rd, 367 KiB/s wr, 29 op/s 2024-03-28T17:41:05.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:05 smithi151 ceph-mon[36481]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 11 KiB/s rd, 367 KiB/s wr, 29 op/s 2024-03-28T17:41:05.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:05 smithi045 ceph-mon[31773]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 11 KiB/s rd, 367 KiB/s wr, 29 op/s 2024-03-28T17:41:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:07 smithi071 ceph-mon[36597]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 8.4 KiB/s rd, 285 KiB/s wr, 23 op/s 2024-03-28T17:41:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:07 smithi151 ceph-mon[36481]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 8.4 KiB/s rd, 285 KiB/s wr, 23 op/s 2024-03-28T17:41:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:07 smithi045 ceph-mon[31773]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 8.4 KiB/s rd, 285 KiB/s wr, 23 op/s 2024-03-28T17:41:09.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:09 smithi045 ceph-mon[31773]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 255 KiB/s wr, 20 op/s 2024-03-28T17:41:09.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:09 smithi071 ceph-mon[36597]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 255 KiB/s wr, 20 op/s 2024-03-28T17:41:09.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:09 smithi151 ceph-mon[36481]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 255 KiB/s wr, 20 op/s 2024-03-28T17:41:11.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:11 smithi045 ceph-mon[31773]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 255 KiB/s wr, 20 op/s 2024-03-28T17:41:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:11 smithi071 ceph-mon[36597]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 255 KiB/s wr, 20 op/s 2024-03-28T17:41:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:11 smithi151 ceph-mon[36481]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 255 KiB/s wr, 20 op/s 2024-03-28T17:41:12.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:12 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T17:41:12.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:12 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:12.322 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:12 smithi045 ceph-mon[31773]: Deploying daemon osd.3 on smithi045 2024-03-28T17:41:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:12 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T17:41:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:12 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:12.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:12 smithi071 ceph-mon[36597]: Deploying daemon osd.3 on smithi045 2024-03-28T17:41:12.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:12 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T17:41:12.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:12 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:12.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:12 smithi151 ceph-mon[36481]: Deploying daemon osd.3 on smithi045 2024-03-28T17:41:13.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:13 smithi071 ceph-mon[36597]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:13.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:13 smithi151 ceph-mon[36481]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:13.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:13 smithi045 ceph-mon[31773]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:15.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:15 smithi045 ceph-mon[31773]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:15.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:15 smithi071 ceph-mon[36597]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:15 smithi151 ceph-mon[36481]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:17.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:17 smithi071 ceph-mon[36597]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:17 smithi151 ceph-mon[36481]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:17.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:17 smithi045 ceph-mon[31773]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:19.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:19 smithi045 ceph-mon[31773]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:19.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:19 smithi071 ceph-mon[36597]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:19.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:19 smithi151 ceph-mon[36481]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:21.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:21 smithi045 ceph-mon[31773]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:21.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:21 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:21.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:21 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:21.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:21 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:41:21.375 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:21 smithi151 ceph-mon[36481]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:21.375 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:21 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:21.375 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:21 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:21.376 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:21 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:41:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:21 smithi071 ceph-mon[36597]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:21 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:21.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:21 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:21.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:21 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:41:23.264 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:23 smithi045 ceph-mon[31773]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:23.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:23 smithi071 ceph-mon[36597]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:23.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:23 smithi151 ceph-mon[36481]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:24.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:24 smithi045 ceph-mon[31773]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:24.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:24 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:24.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:24 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:24.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:24 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:24.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:24 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:41:24.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:24 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:24.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:24 smithi045 ceph-mon[31773]: from='osd.3 [v2:172.21.15.45:6810/3021569975,v1:172.21.15.45:6811/3021569975]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T17:41:25.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:41:25.010+0000 7f3276a00700 1 -- 172.21.15.45:0/2977770301 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f327c062690 con 0x7f32640a2800 2024-03-28T17:41:25.011 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 3 on host 'smithi045' 2024-03-28T17:41:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:41:25.014+0000 7f3284800700 1 -- 172.21.15.45:0/2977770301 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f32640a2800 msgr2=0x7f32640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:41:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:41:25.014+0000 7f3284800700 1 --2- 172.21.15.45:0/2977770301 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f32640a2800 0x7f32640a4cc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f327c105470 tx=0x7f3270009260 comp rx=0 tx=0).stop 2024-03-28T17:41:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:41:25.014+0000 7f3284800700 1 -- 172.21.15.45:0/2977770301 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f327c105610 msgr2=0x7f327c19ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:41:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:41:25.014+0000 7f3284800700 1 --2- 172.21.15.45:0/2977770301 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f327c105610 0x7f327c19ab50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f326c005b40 tx=0x7f326c009580 comp rx=0 tx=0).stop 2024-03-28T17:41:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:41:25.015+0000 7f3284800700 1 -- 172.21.15.45:0/2977770301 shutdown_connections 2024-03-28T17:41:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:41:25.015+0000 7f3284800700 1 --2- 172.21.15.45:0/2977770301 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f32640a2800 0x7f32640a4cc0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:41:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:41:25.015+0000 7f3284800700 1 --2- 172.21.15.45:0/2977770301 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f327c105fb0 0x7f327c1a1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:41:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:41:25.015+0000 7f3284800700 1 --2- 172.21.15.45:0/2977770301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f327c104410 0x7f327c19a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:41:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:41:25.015+0000 7f3284800700 1 --2- 172.21.15.45:0/2977770301 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f327c105610 0x7f327c19ab50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:41:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:41:25.015+0000 7f3284800700 1 -- 172.21.15.45:0/2977770301 >> 172.21.15.45:0/2977770301 conn(0x7f327c0ff9b0 msgr2=0x7f327c108910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:41:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:41:25.015+0000 7f3284800700 1 -- 172.21.15.45:0/2977770301 shutdown_connections 2024-03-28T17:41:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:41:25.015+0000 7f3284800700 1 -- 172.21.15.45:0/2977770301 wait complete. 2024-03-28T17:41:25.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:24 smithi071 ceph-mon[36597]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:25.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:24 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:25.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:24 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:25.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:24 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:25.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:24 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:41:25.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:24 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:25.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:24 smithi071 ceph-mon[36597]: from='osd.3 [v2:172.21.15.45:6810/3021569975,v1:172.21.15.45:6811/3021569975]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T17:41:25.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:24 smithi151 ceph-mon[36481]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:25.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:24 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:25.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:24 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:25.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:24 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:25.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:24 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:41:25.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:24 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:25.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:24 smithi151 ceph-mon[36481]: from='osd.3 [v2:172.21.15.45:6810/3021569975,v1:172.21.15.45:6811/3021569975]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T17:41:25.825 DEBUG:teuthology.orchestra.run.smithi045:osd.3> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@osd.3.service 2024-03-28T17:41:25.830 INFO:tasks.cephadm:Deploying osd.4 on smithi071 with /dev/vg_nvme/lv_3... 2024-03-28T17:41:25.831 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T17:41:26.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:25 smithi071 ceph-mon[36597]: from='osd.3 [v2:172.21.15.45:6810/3021569975,v1:172.21.15.45:6811/3021569975]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T17:41:26.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:25 smithi071 ceph-mon[36597]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T17:41:26.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:25 smithi071 ceph-mon[36597]: from='osd.3 [v2:172.21.15.45:6810/3021569975,v1:172.21.15.45:6811/3021569975]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-28T17:41:26.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T17:41:26.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:26.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:26.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:41:26.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:25 smithi151 ceph-mon[36481]: from='osd.3 [v2:172.21.15.45:6810/3021569975,v1:172.21.15.45:6811/3021569975]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T17:41:26.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:25 smithi151 ceph-mon[36481]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T17:41:26.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:25 smithi151 ceph-mon[36481]: from='osd.3 [v2:172.21.15.45:6810/3021569975,v1:172.21.15.45:6811/3021569975]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-28T17:41:26.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T17:41:26.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:26.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:26.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:41:26.074 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T17:41:26.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:25 smithi045 ceph-mon[31773]: from='osd.3 [v2:172.21.15.45:6810/3021569975,v1:172.21.15.45:6811/3021569975]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T17:41:26.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:25 smithi045 ceph-mon[31773]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T17:41:26.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:25 smithi045 ceph-mon[31773]: from='osd.3 [v2:172.21.15.45:6810/3021569975,v1:172.21.15.45:6811/3021569975]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-28T17:41:26.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T17:41:26.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:26.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:26.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:41:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:26 smithi071 ceph-mon[36597]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:26.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:26 smithi071 ceph-mon[36597]: from='osd.3 [v2:172.21.15.45:6810/3021569975,v1:172.21.15.45:6811/3021569975]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-28T17:41:26.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:26 smithi071 ceph-mon[36597]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T17:41:26.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:26 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T17:41:26.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:26 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T17:41:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:26 smithi151 ceph-mon[36481]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:26 smithi151 ceph-mon[36481]: from='osd.3 [v2:172.21.15.45:6810/3021569975,v1:172.21.15.45:6811/3021569975]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-28T17:41:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:26 smithi151 ceph-mon[36481]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T17:41:27.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:26 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T17:41:27.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:26 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T17:41:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:26 smithi045 ceph-mon[31773]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:26 smithi045 ceph-mon[31773]: from='osd.3 [v2:172.21.15.45:6810/3021569975,v1:172.21.15.45:6811/3021569975]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-28T17:41:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:26 smithi045 ceph-mon[31773]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T17:41:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:26 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T17:41:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:26 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T17:41:27.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:27 smithi071 ceph-mon[36597]: purged_snaps scrub starts 2024-03-28T17:41:27.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:27 smithi071 ceph-mon[36597]: purged_snaps scrub ok 2024-03-28T17:41:27.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:27 smithi071 ceph-mon[36597]: osd.3 [v2:172.21.15.45:6810/3021569975,v1:172.21.15.45:6811/3021569975] boot 2024-03-28T17:41:27.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:27 smithi071 ceph-mon[36597]: osdmap e25: 4 total, 4 up, 4 in 2024-03-28T17:41:27.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:27 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T17:41:27.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:27 smithi045 ceph-mon[31773]: purged_snaps scrub starts 2024-03-28T17:41:27.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:27 smithi045 ceph-mon[31773]: purged_snaps scrub ok 2024-03-28T17:41:27.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:27 smithi045 ceph-mon[31773]: osd.3 [v2:172.21.15.45:6810/3021569975,v1:172.21.15.45:6811/3021569975] boot 2024-03-28T17:41:27.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:27 smithi045 ceph-mon[31773]: osdmap e25: 4 total, 4 up, 4 in 2024-03-28T17:41:27.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:27 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T17:41:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:27 smithi151 ceph-mon[36481]: purged_snaps scrub starts 2024-03-28T17:41:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:27 smithi151 ceph-mon[36481]: purged_snaps scrub ok 2024-03-28T17:41:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:27 smithi151 ceph-mon[36481]: osd.3 [v2:172.21.15.45:6810/3021569975,v1:172.21.15.45:6811/3021569975] boot 2024-03-28T17:41:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:27 smithi151 ceph-mon[36481]: osdmap e25: 4 total, 4 up, 4 in 2024-03-28T17:41:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:27 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T17:41:28.093 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T17:41:28.094 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T17:41:28.094 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10+0 records in 2024-03-28T17:41:28.094 INFO:teuthology.orchestra.run.smithi071.stderr:10+0 records out 2024-03-28T17:41:28.094 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114761 s, 914 MB/s 2024-03-28T17:41:28.094 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping successful for: 2024-03-28T17:41:28.690 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi071:vg_nvme/lv_3 2024-03-28T17:41:28.896 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:28 smithi045 ceph-mon[31773]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:28.896 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:28 smithi045 ceph-mon[31773]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T17:41:28.935 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T17:41:29.056 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:28 smithi071 ceph-mon[36597]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:29.056 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:28 smithi071 ceph-mon[36597]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T17:41:29.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:28 smithi151 ceph-mon[36481]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T17:41:29.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:28 smithi151 ceph-mon[36481]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T17:41:30.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:29 smithi071 ceph-mon[36597]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T17:41:30.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:29 smithi151 ceph-mon[36481]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T17:41:30.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:29 smithi045 ceph-mon[31773]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T17:41:30.647 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.644+0000 7fe595429700 1 -- 172.21.15.71:0/2958512631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe590103e50 msgr2=0x7fe59010a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:41:30.647 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.644+0000 7fe595429700 1 --2- 172.21.15.71:0/2958512631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe590103e50 0x7fe59010a700 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe58800b0a0 tx=0x7fe58800b3b0 comp rx=0 tx=0).stop 2024-03-28T17:41:30.647 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.645+0000 7fe595429700 1 -- 172.21.15.71:0/2958512631 shutdown_connections 2024-03-28T17:41:30.648 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.645+0000 7fe595429700 1 --2- 172.21.15.71:0/2958512631 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5901004d0 0x7fe5901008f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:41:30.648 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.645+0000 7fe595429700 1 --2- 172.21.15.71:0/2958512631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe590103e50 0x7fe59010a700 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:41:30.648 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.645+0000 7fe595429700 1 --2- 172.21.15.71:0/2958512631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5901034b0 0x7fe590103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:41:30.648 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.645+0000 7fe595429700 1 -- 172.21.15.71:0/2958512631 >> 172.21.15.71:0/2958512631 conn(0x7fe5900fbe30 msgr2=0x7fe5900fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:41:30.648 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.645+0000 7fe595429700 1 -- 172.21.15.71:0/2958512631 shutdown_connections 2024-03-28T17:41:30.648 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.645+0000 7fe595429700 1 -- 172.21.15.71:0/2958512631 wait complete. 2024-03-28T17:41:30.648 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.645+0000 7fe595429700 1 Processor -- start 2024-03-28T17:41:30.648 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.645+0000 7fe595429700 1 -- start start 2024-03-28T17:41:30.648 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.645+0000 7fe595429700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5901004d0 0x7fe590198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:41:30.648 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.645+0000 7fe595429700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5901034b0 0x7fe590198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:41:30.649 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.645+0000 7fe595429700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe590103e50 0x7fe59019f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:41:30.649 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.645+0000 7fe595429700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59010dee0 con 0x7fe5901004d0 2024-03-28T17:41:30.649 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.646+0000 7fe595429700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59010dd60 con 0x7fe590103e50 2024-03-28T17:41:30.649 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.646+0000 7fe595429700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59010e060 con 0x7fe5901034b0 2024-03-28T17:41:30.649 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.646+0000 7fe58f7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe590103e50 0x7fe59019f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:41:30.649 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.646+0000 7fe58f7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe590103e50 0x7fe59019f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50748/0 (socket says 172.21.15.71:50748) 2024-03-28T17:41:30.649 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.646+0000 7fe58f7fe700 1 -- 172.21.15.71:0/1735461141 learned_addr learned my addr 172.21.15.71:0/1735461141 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-03-28T17:41:30.649 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.646+0000 7fe58e7fc700 1 --2- 172.21.15.71:0/1735461141 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5901034b0 0x7fe590198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:41:30.649 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.646+0000 7fe58effd700 1 --2- 172.21.15.71:0/1735461141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5901004d0 0x7fe590198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:41:30.649 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.646+0000 7fe58f7fe700 1 -- 172.21.15.71:0/1735461141 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5901034b0 msgr2=0x7fe590198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:41:30.649 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.646+0000 7fe58f7fe700 1 --2- 172.21.15.71:0/1735461141 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5901034b0 0x7fe590198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:41:30.649 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.646+0000 7fe58f7fe700 1 -- 172.21.15.71:0/1735461141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5901004d0 msgr2=0x7fe590198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:41:30.649 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.646+0000 7fe58f7fe700 1 --2- 172.21.15.71:0/1735461141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5901004d0 0x7fe590198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:41:30.649 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.646+0000 7fe58f7fe700 1 -- 172.21.15.71:0/1735461141 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5901a0090 con 0x7fe590103e50 2024-03-28T17:41:30.649 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.646+0000 7fe58e7fc700 1 --2- 172.21.15.71:0/1735461141 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5901034b0 0x7fe590198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:41:30.650 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.646+0000 7fe58f7fe700 1 --2- 172.21.15.71:0/1735461141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe590103e50 0x7fe59019f9f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe588007f00 tx=0x7fe58800bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:41:30.650 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.646+0000 7fe577fff700 1 -- 172.21.15.71:0/1735461141 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe588021030 con 0x7fe590103e50 2024-03-28T17:41:30.650 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.646+0000 7fe595429700 1 -- 172.21.15.71:0/1735461141 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5901a0320 con 0x7fe590103e50 2024-03-28T17:41:30.650 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.646+0000 7fe577fff700 1 -- 172.21.15.71:0/1735461141 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe588007bb0 con 0x7fe590103e50 2024-03-28T17:41:30.650 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.646+0000 7fe577fff700 1 -- 172.21.15.71:0/1735461141 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58801d6d0 con 0x7fe590103e50 2024-03-28T17:41:30.650 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.646+0000 7fe595429700 1 -- 172.21.15.71:0/1735461141 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5901a0800 con 0x7fe590103e50 2024-03-28T17:41:30.651 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.647+0000 7fe595429700 1 -- 172.21.15.71:0/1735461141 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe590068e50 con 0x7fe590103e50 2024-03-28T17:41:30.651 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.648+0000 7fe577fff700 1 -- 172.21.15.71:0/1735461141 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe588024070 con 0x7fe590103e50 2024-03-28T17:41:30.651 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.649+0000 7fe577fff700 1 --2- 172.21.15.71:0/1735461141 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe5780a2890 0x7fe5780a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:41:30.651 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.649+0000 7fe577fff700 1 -- 172.21.15.71:0/1735461141 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 3669+0+0 (secure 0 0 0) 0x7fe5880cea30 con 0x7fe590103e50 2024-03-28T17:41:30.651 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.649+0000 7fe58effd700 1 --2- 172.21.15.71:0/1735461141 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe5780a2890 0x7fe5780a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:41:30.651 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.650+0000 7fe58effd700 1 --2- 172.21.15.71:0/1735461141 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe5780a2890 0x7fe5780a4d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe580006fd0 tx=0x7fe580008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:41:30.654 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.654+0000 7fe577fff700 1 -- 172.21.15.71:0/1735461141 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe58809cd80 con 0x7fe590103e50 2024-03-28T17:41:30.860 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:30.860+0000 7fe595429700 1 -- 172.21.15.71:0/1735461141 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5900625d0 con 0x7fe5780a2890 2024-03-28T17:41:31.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:30 smithi071 ceph-mon[36597]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:31.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:30 smithi071 ceph-mon[36597]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T17:41:31.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:30 smithi151 ceph-mon[36481]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:31.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:30 smithi151 ceph-mon[36481]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T17:41:31.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:30 smithi045 ceph-mon[31773]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:31.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:30 smithi045 ceph-mon[31773]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T17:41:32.057 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:31 smithi151 ceph-mon[36481]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T17:41:32.057 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:31 smithi151 ceph-mon[36481]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:41:32.057 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:31 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:41:32.057 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:31 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:41:32.057 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:31 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:31 smithi071 ceph-mon[36597]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T17:41:32.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:31 smithi071 ceph-mon[36597]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:41:32.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:31 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:41:32.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:31 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:41:32.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:31 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:32.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:31 smithi045 ceph-mon[31773]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T17:41:32.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:31 smithi045 ceph-mon[31773]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:41:32.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:31 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:41:32.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:31 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:41:32.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:31 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:33.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:33 smithi071 ceph-mon[36597]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T17:41:33.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:33 smithi071 ceph-mon[36597]: Detected new or changed devices on smithi045 2024-03-28T17:41:33.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:33 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:33.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:33 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:33.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:33 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:41:33.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:33 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:41:33.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:33 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi045 to 8547M 2024-03-28T17:41:33.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:33 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:33.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:33 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:33.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:33 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:41:33.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:33 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:33.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:33 smithi151 ceph-mon[36481]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T17:41:33.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:33 smithi151 ceph-mon[36481]: Detected new or changed devices on smithi045 2024-03-28T17:41:33.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:33 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:33.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:33 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:33.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:33 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:41:33.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:33 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:41:33.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:33 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi045 to 8547M 2024-03-28T17:41:33.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:33 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:33.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:33 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:33.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:33 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:41:33.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:33 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:33.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:33 smithi045 ceph-mon[31773]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T17:41:33.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:33 smithi045 ceph-mon[31773]: Detected new or changed devices on smithi045 2024-03-28T17:41:33.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:33 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:33.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:33 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:33.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:33 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:41:33.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:33 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:41:33.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:33 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi045 to 8547M 2024-03-28T17:41:33.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:33 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:33.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:33 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:33.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:33 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:41:33.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:33 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:34.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:34 smithi071 ceph-mon[36597]: from='client.? 172.21.15.71:0/4893825' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a336f42-0605-492b-ad88-56cb02acdcf8"}]: dispatch 2024-03-28T17:41:34.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:34 smithi071 ceph-mon[36597]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a336f42-0605-492b-ad88-56cb02acdcf8"}]: dispatch 2024-03-28T17:41:34.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:34 smithi071 ceph-mon[36597]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a336f42-0605-492b-ad88-56cb02acdcf8"}]': finished 2024-03-28T17:41:34.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:34 smithi071 ceph-mon[36597]: osdmap e30: 5 total, 4 up, 5 in 2024-03-28T17:41:34.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:34 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T17:41:34.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:34 smithi045 ceph-mon[31773]: from='client.? 172.21.15.71:0/4893825' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a336f42-0605-492b-ad88-56cb02acdcf8"}]: dispatch 2024-03-28T17:41:34.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:34 smithi045 ceph-mon[31773]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a336f42-0605-492b-ad88-56cb02acdcf8"}]: dispatch 2024-03-28T17:41:34.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:34 smithi045 ceph-mon[31773]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a336f42-0605-492b-ad88-56cb02acdcf8"}]': finished 2024-03-28T17:41:34.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:34 smithi045 ceph-mon[31773]: osdmap e30: 5 total, 4 up, 5 in 2024-03-28T17:41:34.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:34 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T17:41:34.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:34 smithi151 ceph-mon[36481]: from='client.? 172.21.15.71:0/4893825' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a336f42-0605-492b-ad88-56cb02acdcf8"}]: dispatch 2024-03-28T17:41:34.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:34 smithi151 ceph-mon[36481]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a336f42-0605-492b-ad88-56cb02acdcf8"}]: dispatch 2024-03-28T17:41:34.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:34 smithi151 ceph-mon[36481]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a336f42-0605-492b-ad88-56cb02acdcf8"}]': finished 2024-03-28T17:41:34.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:34 smithi151 ceph-mon[36481]: osdmap e30: 5 total, 4 up, 5 in 2024-03-28T17:41:34.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:34 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T17:41:35.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:35 smithi071 ceph-mon[36597]: pgmap v122: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-28T17:41:35.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:35 smithi071 ceph-mon[36597]: from='client.? 172.21.15.71:0/1061198428' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:41:35.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:35 smithi151 ceph-mon[36481]: pgmap v122: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-28T17:41:35.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:35 smithi151 ceph-mon[36481]: from='client.? 172.21.15.71:0/1061198428' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:41:35.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:35 smithi045 ceph-mon[31773]: pgmap v122: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-28T17:41:35.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:35 smithi045 ceph-mon[31773]: from='client.? 172.21.15.71:0/1061198428' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:41:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:37 smithi071 ceph-mon[36597]: pgmap v124: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T17:41:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:37 smithi151 ceph-mon[36481]: pgmap v124: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T17:41:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:37 smithi045 ceph-mon[31773]: pgmap v124: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T17:41:39.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:39 smithi071 ceph-mon[36597]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-28T17:41:39.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:39 smithi151 ceph-mon[36481]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-28T17:41:39.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:39 smithi045 ceph-mon[31773]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-28T17:41:41.188 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:41 smithi071 ceph-mon[36597]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:41.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:41 smithi151 ceph-mon[36481]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:41.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:41 smithi045 ceph-mon[31773]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:42.191 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:42 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T17:41:42.191 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:42 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:42.191 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:42 smithi071 ceph-mon[36597]: Deploying daemon osd.4 on smithi071 2024-03-28T17:41:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:42 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T17:41:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:42 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:42 smithi151 ceph-mon[36481]: Deploying daemon osd.4 on smithi071 2024-03-28T17:41:42.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:42 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T17:41:42.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:42 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:42.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:42 smithi045 ceph-mon[31773]: Deploying daemon osd.4 on smithi071 2024-03-28T17:41:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:43 smithi045 ceph-mon[31773]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:43.181 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:43 smithi071 ceph-mon[36597]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:43.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:43 smithi151 ceph-mon[36481]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:45.264 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:45 smithi071 ceph-mon[36597]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:45.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:45 smithi151 ceph-mon[36481]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:45.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:45 smithi045 ceph-mon[31773]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:47 smithi071 ceph-mon[36597]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:47 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:47 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:47.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:47 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:41:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:47 smithi151 ceph-mon[36481]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:47 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:47 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:47 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:41:47.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:47 smithi045 ceph-mon[31773]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:47 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:47 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:47 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:41:49.210 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:49 smithi071 ceph-mon[36597]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:49.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:49 smithi151 ceph-mon[36481]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:49.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:49 smithi045 ceph-mon[31773]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:50.993 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:50.992+0000 7fe577fff700 1 -- 172.21.15.71:0/1735461141 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe5900625d0 con 0x7fe5780a2890 2024-03-28T17:41:50.993 INFO:teuthology.orchestra.run.smithi071.stdout:Created osd(s) 4 on host 'smithi071' 2024-03-28T17:41:50.995 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:50.995+0000 7fe595429700 1 -- 172.21.15.71:0/1735461141 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe5780a2890 msgr2=0x7fe5780a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:41:50.995 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:50.995+0000 7fe595429700 1 --2- 172.21.15.71:0/1735461141 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe5780a2890 0x7fe5780a4d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe580006fd0 tx=0x7fe580008040 comp rx=0 tx=0).stop 2024-03-28T17:41:50.996 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:50.995+0000 7fe595429700 1 -- 172.21.15.71:0/1735461141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe590103e50 msgr2=0x7fe59019f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:41:50.996 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:50.995+0000 7fe595429700 1 --2- 172.21.15.71:0/1735461141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe590103e50 0x7fe59019f9f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe588007f00 tx=0x7fe58800bf10 comp rx=0 tx=0).stop 2024-03-28T17:41:50.996 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:50.995+0000 7fe595429700 1 -- 172.21.15.71:0/1735461141 shutdown_connections 2024-03-28T17:41:50.996 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:50.995+0000 7fe595429700 1 --2- 172.21.15.71:0/1735461141 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe5780a2890 0x7fe5780a4d50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:41:50.996 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:50.995+0000 7fe595429700 1 --2- 172.21.15.71:0/1735461141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe590103e50 0x7fe59019f9f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:41:50.996 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:50.995+0000 7fe595429700 1 --2- 172.21.15.71:0/1735461141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5901004d0 0x7fe590198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:41:50.996 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:50.995+0000 7fe595429700 1 --2- 172.21.15.71:0/1735461141 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5901034b0 0x7fe590198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:41:50.996 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:50.995+0000 7fe595429700 1 -- 172.21.15.71:0/1735461141 >> 172.21.15.71:0/1735461141 conn(0x7fe5900fbe30 msgr2=0x7fe5901066e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:41:50.996 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:50.995+0000 7fe595429700 1 -- 172.21.15.71:0/1735461141 shutdown_connections 2024-03-28T17:41:50.996 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:41:50.995+0000 7fe595429700 1 -- 172.21.15.71:0/1735461141 wait complete. 2024-03-28T17:41:51.006 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:50 smithi071 ceph-mon[36597]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:51.006 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:50 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:51.006 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:50 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:51.006 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:50 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:51.006 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:50 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:41:51.006 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:50 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:51.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:50 smithi151 ceph-mon[36481]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:51.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:50 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:51.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:50 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:51.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:50 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:51.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:50 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:41:51.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:50 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:51.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:50 smithi045 ceph-mon[31773]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:51.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:50 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:51.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:50 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:51.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:50 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:51.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:50 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:41:51.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:50 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:51.716 DEBUG:teuthology.orchestra.run.smithi071:osd.4> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@osd.4.service 2024-03-28T17:41:51.719 INFO:tasks.cephadm:Deploying osd.5 on smithi151 with /dev/vg_nvme/lv_3... 2024-03-28T17:41:51.719 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T17:41:51.962 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T17:41:52.034 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:51 smithi071 ceph-mon[36597]: from='osd.4 [v2:172.21.15.71:6808/2069578333,v1:172.21.15.71:6809/2069578333]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T17:41:52.034 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:51 smithi071 ceph-mon[36597]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T17:41:52.034 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:51 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:52.034 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:51 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:52.034 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:51 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:41:52.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:51 smithi151 ceph-mon[36481]: from='osd.4 [v2:172.21.15.71:6808/2069578333,v1:172.21.15.71:6809/2069578333]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T17:41:52.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:51 smithi151 ceph-mon[36481]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T17:41:52.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:51 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:52.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:51 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:52.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:51 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:41:52.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:51 smithi045 ceph-mon[31773]: from='osd.4 [v2:172.21.15.71:6808/2069578333,v1:172.21.15.71:6809/2069578333]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T17:41:52.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:51 smithi045 ceph-mon[31773]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T17:41:52.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:51 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:52.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:51 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:52.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:51 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:41:53.090 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:52 smithi151 ceph-mon[36481]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:53.090 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:52 smithi151 ceph-mon[36481]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T17:41:53.090 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:52 smithi151 ceph-mon[36481]: from='osd.4 [v2:172.21.15.71:6808/2069578333,v1:172.21.15.71:6809/2069578333]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:41:53.090 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:52 smithi151 ceph-mon[36481]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T17:41:53.090 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:52 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T17:41:53.090 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:52 smithi151 ceph-mon[36481]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:41:53.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:52 smithi071 ceph-mon[36597]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:53.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:52 smithi071 ceph-mon[36597]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T17:41:53.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:52 smithi071 ceph-mon[36597]: from='osd.4 [v2:172.21.15.71:6808/2069578333,v1:172.21.15.71:6809/2069578333]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:41:53.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:52 smithi071 ceph-mon[36597]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T17:41:53.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:52 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T17:41:53.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:52 smithi071 ceph-mon[36597]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:41:53.091 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 28 17:41:52 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-osd-4[48186]: 2024-03-28T17:41:52.826+0000 7f4083df6700 -1 osd.4 0 waiting for initial osdmap 2024-03-28T17:41:53.091 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 28 17:41:52 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-osd-4[48186]: 2024-03-28T17:41:52.832+0000 7f407fbe9700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T17:41:53.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:52 smithi045 ceph-mon[31773]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:53.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:52 smithi045 ceph-mon[31773]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T17:41:53.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:52 smithi045 ceph-mon[31773]: from='osd.4 [v2:172.21.15.71:6808/2069578333,v1:172.21.15.71:6809/2069578333]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:41:53.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:52 smithi045 ceph-mon[31773]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T17:41:53.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:52 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T17:41:53.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:52 smithi045 ceph-mon[31773]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:41:53.976 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T17:41:53.976 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T17:41:53.976 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2024-03-28T17:41:53.976 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2024-03-28T17:41:53.976 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131075 s, 800 MB/s 2024-03-28T17:41:53.976 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2024-03-28T17:41:54.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:53 smithi045 ceph-mon[31773]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-28T17:41:54.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:53 smithi045 ceph-mon[31773]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T17:41:54.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:53 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T17:41:54.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:53 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T17:41:54.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:53 smithi071 ceph-mon[36597]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-28T17:41:54.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:53 smithi071 ceph-mon[36597]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T17:41:54.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:53 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T17:41:54.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:53 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T17:41:54.317 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:53 smithi151 ceph-mon[36481]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-28T17:41:54.317 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:53 smithi151 ceph-mon[36481]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T17:41:54.317 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:53 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T17:41:54.317 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:53 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T17:41:54.625 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi151:vg_nvme/lv_3 2024-03-28T17:41:54.863 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T17:41:54.984 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:54 smithi151 ceph-mon[36481]: purged_snaps scrub starts 2024-03-28T17:41:54.985 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:54 smithi151 ceph-mon[36481]: purged_snaps scrub ok 2024-03-28T17:41:54.985 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:54 smithi151 ceph-mon[36481]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:54.985 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:54 smithi151 ceph-mon[36481]: osd.4 [v2:172.21.15.71:6808/2069578333,v1:172.21.15.71:6809/2069578333] boot 2024-03-28T17:41:54.985 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:54 smithi151 ceph-mon[36481]: osdmap e33: 5 total, 5 up, 5 in 2024-03-28T17:41:54.985 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:54 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T17:41:54.985 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:54 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:54.985 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:54 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:54.985 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:54 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:41:54.985 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:54 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:41:54.985 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:54 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:54.985 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:54 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:54.985 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:54 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:41:54.985 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:54 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:55.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:54 smithi071 ceph-mon[36597]: purged_snaps scrub starts 2024-03-28T17:41:55.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:54 smithi071 ceph-mon[36597]: purged_snaps scrub ok 2024-03-28T17:41:55.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:54 smithi071 ceph-mon[36597]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:55.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:54 smithi071 ceph-mon[36597]: osd.4 [v2:172.21.15.71:6808/2069578333,v1:172.21.15.71:6809/2069578333] boot 2024-03-28T17:41:55.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:54 smithi071 ceph-mon[36597]: osdmap e33: 5 total, 5 up, 5 in 2024-03-28T17:41:55.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:54 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T17:41:55.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:54 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:55.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:54 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:55.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:54 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:41:55.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:54 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:41:55.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:54 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:55.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:54 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:55.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:54 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:41:55.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:54 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:55.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:54 smithi045 ceph-mon[31773]: purged_snaps scrub starts 2024-03-28T17:41:55.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:54 smithi045 ceph-mon[31773]: purged_snaps scrub ok 2024-03-28T17:41:55.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:54 smithi045 ceph-mon[31773]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T17:41:55.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:54 smithi045 ceph-mon[31773]: osd.4 [v2:172.21.15.71:6808/2069578333,v1:172.21.15.71:6809/2069578333] boot 2024-03-28T17:41:55.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:54 smithi045 ceph-mon[31773]: osdmap e33: 5 total, 5 up, 5 in 2024-03-28T17:41:55.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:54 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T17:41:55.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:54 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:55.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:54 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:55.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:54 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:41:55.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:54 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:41:55.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:54 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:55.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:54 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:55.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:54 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:41:55.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:54 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:41:55.982 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:55 smithi151 ceph-mon[36481]: Detected new or changed devices on smithi071 2024-03-28T17:41:55.982 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:55 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi071 to 8540M 2024-03-28T17:41:55.982 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:55 smithi151 ceph-mon[36481]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T17:41:56.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:55 smithi045 ceph-mon[31773]: Detected new or changed devices on smithi071 2024-03-28T17:41:56.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:55 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi071 to 8540M 2024-03-28T17:41:56.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:55 smithi045 ceph-mon[31773]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T17:41:56.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:55 smithi071 ceph-mon[36597]: Detected new or changed devices on smithi071 2024-03-28T17:41:56.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:55 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi071 to 8540M 2024-03-28T17:41:56.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:55 smithi071 ceph-mon[36597]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T17:41:56.486 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.486+0000 7f9f6764e700 1 -- 172.21.15.151:0/131427942 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f60102c50 msgr2=0x7f9f60109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:41:56.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.486+0000 7f9f6764e700 1 --2- 172.21.15.151:0/131427942 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f60102c50 0x7f9f60109500 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9f5800b0a0 tx=0x7f9f5800b3b0 comp rx=0 tx=0).stop 2024-03-28T17:41:56.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.486+0000 7f9f6764e700 1 -- 172.21.15.151:0/131427942 shutdown_connections 2024-03-28T17:41:56.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.486+0000 7f9f6764e700 1 --2- 172.21.15.151:0/131427942 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f60102c50 0x7f9f60109500 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:41:56.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.486+0000 7f9f6764e700 1 --2- 172.21.15.151:0/131427942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f601010b0 0x7f9f601014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:41:56.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.486+0000 7f9f6764e700 1 --2- 172.21.15.151:0/131427942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f601022b0 0x7f9f60102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:41:56.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.486+0000 7f9f6764e700 1 -- 172.21.15.151:0/131427942 >> 172.21.15.151:0/131427942 conn(0x7f9f600fc670 msgr2=0x7f9f600fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:41:56.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.486+0000 7f9f6764e700 1 -- 172.21.15.151:0/131427942 shutdown_connections 2024-03-28T17:41:56.487 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.486+0000 7f9f6764e700 1 -- 172.21.15.151:0/131427942 wait complete. 2024-03-28T17:41:56.489 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.487+0000 7f9f6764e700 1 Processor -- start 2024-03-28T17:41:56.489 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.487+0000 7f9f6764e700 1 -- start start 2024-03-28T17:41:56.489 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.487+0000 7f9f6764e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f601010b0 0x7f9f60197290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:41:56.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.487+0000 7f9f6764e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f601022b0 0x7f9f601977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:41:56.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.487+0000 7f9f6764e700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f60102c50 0x7f9f6019e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:41:56.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.487+0000 7f9f6764e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6010cd10 con 0x7f9f601010b0 2024-03-28T17:41:56.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.487+0000 7f9f6764e700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6010cb90 con 0x7f9f60102c50 2024-03-28T17:41:56.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.487+0000 7f9f6764e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6010ce90 con 0x7f9f601022b0 2024-03-28T17:41:56.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.487+0000 7f9f64be9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f601022b0 0x7f9f601977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:41:56.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.487+0000 7f9f64be9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f601022b0 0x7f9f601977d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:34038/0 (socket says 172.21.15.151:34038) 2024-03-28T17:41:56.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.487+0000 7f9f64be9700 1 -- 172.21.15.151:0/3612740795 learned_addr learned my addr 172.21.15.151:0/3612740795 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-03-28T17:41:56.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.487+0000 7f9f65beb700 1 --2- 172.21.15.151:0/3612740795 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f60102c50 0x7f9f6019e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:41:56.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.487+0000 7f9f64be9700 1 -- 172.21.15.151:0/3612740795 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f60102c50 msgr2=0x7f9f6019e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:41:56.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.487+0000 7f9f64be9700 1 --2- 172.21.15.151:0/3612740795 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f60102c50 0x7f9f6019e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:41:56.490 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.487+0000 7f9f64be9700 1 -- 172.21.15.151:0/3612740795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f601010b0 msgr2=0x7f9f60197290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:41:56.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.487+0000 7f9f64be9700 1 --2- 172.21.15.151:0/3612740795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f601010b0 0x7f9f60197290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:41:56.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.487+0000 7f9f64be9700 1 -- 172.21.15.151:0/3612740795 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6019eef0 con 0x7f9f601022b0 2024-03-28T17:41:56.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.488+0000 7f9f64be9700 1 --2- 172.21.15.151:0/3612740795 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f601022b0 0x7f9f601977d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9f5000d9b0 tx=0x7f9f5000dd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:41:56.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.488+0000 7f9f567fc700 1 -- 172.21.15.151:0/3612740795 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f50016070 con 0x7f9f601022b0 2024-03-28T17:41:56.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.488+0000 7f9f567fc700 1 -- 172.21.15.151:0/3612740795 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f50004500 con 0x7f9f601022b0 2024-03-28T17:41:56.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.488+0000 7f9f6764e700 1 -- 172.21.15.151:0/3612740795 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f6019f1e0 con 0x7f9f601022b0 2024-03-28T17:41:56.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.488+0000 7f9f567fc700 1 -- 172.21.15.151:0/3612740795 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f50004d00 con 0x7f9f601022b0 2024-03-28T17:41:56.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.488+0000 7f9f6764e700 1 -- 172.21.15.151:0/3612740795 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f6019f720 con 0x7f9f601022b0 2024-03-28T17:41:56.491 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.489+0000 7f9f6764e700 1 -- 172.21.15.151:0/3612740795 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f60068e50 con 0x7f9f601022b0 2024-03-28T17:41:56.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.490+0000 7f9f567fc700 1 -- 172.21.15.151:0/3612740795 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9f50004020 con 0x7f9f601022b0 2024-03-28T17:41:56.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.490+0000 7f9f567fc700 1 --2- 172.21.15.151:0/3612740795 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9f4c0a2890 0x7f9f4c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:41:56.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.490+0000 7f9f567fc700 1 -- 172.21.15.151:0/3612740795 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f9f500cd940 con 0x7f9f601022b0 2024-03-28T17:41:56.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.491+0000 7f9f653ea700 1 --2- 172.21.15.151:0/3612740795 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9f4c0a2890 0x7f9f4c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:41:56.492 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.492+0000 7f9f653ea700 1 --2- 172.21.15.151:0/3612740795 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9f4c0a2890 0x7f9f4c0a4d50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9f60102110 tx=0x7f9f5c006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:41:56.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.495+0000 7f9f567fc700 1 -- 172.21.15.151:0/3612740795 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9f5001f020 con 0x7f9f601022b0 2024-03-28T17:41:56.697 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:41:56.696+0000 7f9f6764e700 1 -- 172.21.15.151:0/3612740795 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f600625d0 con 0x7f9f4c0a2890 2024-03-28T17:41:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:56 smithi151 ceph-mon[36481]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:41:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:56 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:41:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:56 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:41:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:56 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:57.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:56 smithi045 ceph-mon[31773]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:41:57.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:56 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:41:57.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:56 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:41:57.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:56 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:56 smithi071 ceph-mon[36597]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:41:57.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:56 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:41:57.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:56 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:41:57.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:56 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:41:58.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:57 smithi151 ceph-mon[36481]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:41:58.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:57 smithi045 ceph-mon[31773]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:41:58.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:57 smithi071 ceph-mon[36597]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:41:59.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:58 smithi045 ceph-mon[31773]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:41:59.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:58 smithi071 ceph-mon[36597]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:41:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:58 smithi151 ceph-mon[36481]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:00.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:59 smithi151 ceph-mon[36481]: from='client.? 172.21.15.151:0/1990508354' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4"}]: dispatch 2024-03-28T17:42:00.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:59 smithi151 ceph-mon[36481]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4"}]: dispatch 2024-03-28T17:42:00.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:59 smithi151 ceph-mon[36481]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4"}]': finished 2024-03-28T17:42:00.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:59 smithi151 ceph-mon[36481]: osdmap e35: 6 total, 5 up, 6 in 2024-03-28T17:42:00.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:41:59 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T17:42:00.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:59 smithi045 ceph-mon[31773]: from='client.? 172.21.15.151:0/1990508354' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4"}]: dispatch 2024-03-28T17:42:00.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:59 smithi045 ceph-mon[31773]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4"}]: dispatch 2024-03-28T17:42:00.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:59 smithi045 ceph-mon[31773]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4"}]': finished 2024-03-28T17:42:00.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:59 smithi045 ceph-mon[31773]: osdmap e35: 6 total, 5 up, 6 in 2024-03-28T17:42:00.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:41:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T17:42:00.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:59 smithi071 ceph-mon[36597]: from='client.? 172.21.15.151:0/1990508354' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4"}]: dispatch 2024-03-28T17:42:00.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:59 smithi071 ceph-mon[36597]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4"}]: dispatch 2024-03-28T17:42:00.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:59 smithi071 ceph-mon[36597]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4"}]': finished 2024-03-28T17:42:00.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:59 smithi071 ceph-mon[36597]: osdmap e35: 6 total, 5 up, 6 in 2024-03-28T17:42:00.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:41:59 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T17:42:01.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:00 smithi045 ceph-mon[31773]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:01.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:00 smithi045 ceph-mon[31773]: from='client.? 172.21.15.151:0/738076606' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:42:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:00 smithi071 ceph-mon[36597]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:01.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:00 smithi071 ceph-mon[36597]: from='client.? 172.21.15.151:0/738076606' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:42:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:00 smithi151 ceph-mon[36481]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:00 smithi151 ceph-mon[36481]: from='client.? 172.21.15.151:0/738076606' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:42:03.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:02 smithi045 ceph-mon[31773]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:02 smithi071 ceph-mon[36597]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:03.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:02 smithi151 ceph-mon[36481]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:05.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:04 smithi151 ceph-mon[36481]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:05.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:04 smithi045 ceph-mon[31773]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:05.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:04 smithi071 ceph-mon[36597]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:07.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:06 smithi151 ceph-mon[36481]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:07.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:06 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T17:42:07.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:06 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:06 smithi045 ceph-mon[31773]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:07.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:06 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T17:42:07.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:06 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:06 smithi071 ceph-mon[36597]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:06 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T17:42:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:06 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:08.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:07 smithi151 ceph-mon[36481]: Deploying daemon osd.5 on smithi151 2024-03-28T17:42:08.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:07 smithi045 ceph-mon[31773]: Deploying daemon osd.5 on smithi151 2024-03-28T17:42:08.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:07 smithi071 ceph-mon[36597]: Deploying daemon osd.5 on smithi151 2024-03-28T17:42:09.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:08 smithi045 ceph-mon[31773]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:09.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:08 smithi071 ceph-mon[36597]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:09.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:08 smithi151 ceph-mon[36481]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:11.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:10 smithi045 ceph-mon[31773]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:11.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:10 smithi071 ceph-mon[36597]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:11.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:10 smithi151 ceph-mon[36481]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:13.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:13 smithi071 ceph-mon[36597]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:13.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:13.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:13.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:42:13.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:13 smithi151 ceph-mon[36481]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:13.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:13 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:13.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:13 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:13.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:13 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:42:13.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:13 smithi045 ceph-mon[31773]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:13.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:13 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:13.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:13 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:13.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:13 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:42:15.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:15 smithi071 ceph-mon[36597]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:15.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:15 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:15.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:15 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:15.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:15 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:15.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:15 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:42:15.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:15 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:15 smithi151 ceph-mon[36481]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:15 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:15 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:15 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:15 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:42:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:15 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:15.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:15 smithi045 ceph-mon[31773]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:15.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:15 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:15.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:15 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:15.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:15 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:15.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:15 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:42:15.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:15 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:16.089 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:42:16.088+0000 7f9f567fc700 1 -- 172.21.15.151:0/3612740795 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9f600625d0 con 0x7f9f4c0a2890 2024-03-28T17:42:16.090 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 5 on host 'smithi151' 2024-03-28T17:42:16.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:42:16.091+0000 7f9f6764e700 1 -- 172.21.15.151:0/3612740795 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9f4c0a2890 msgr2=0x7f9f4c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:42:16.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:42:16.091+0000 7f9f6764e700 1 --2- 172.21.15.151:0/3612740795 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9f4c0a2890 0x7f9f4c0a4d50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9f60102110 tx=0x7f9f5c006d20 comp rx=0 tx=0).stop 2024-03-28T17:42:16.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:42:16.091+0000 7f9f6764e700 1 -- 172.21.15.151:0/3612740795 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f601022b0 msgr2=0x7f9f601977d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:42:16.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:42:16.091+0000 7f9f6764e700 1 --2- 172.21.15.151:0/3612740795 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f601022b0 0x7f9f601977d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9f5000d9b0 tx=0x7f9f5000dd70 comp rx=0 tx=0).stop 2024-03-28T17:42:16.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:42:16.091+0000 7f9f6764e700 1 -- 172.21.15.151:0/3612740795 shutdown_connections 2024-03-28T17:42:16.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:42:16.091+0000 7f9f6764e700 1 --2- 172.21.15.151:0/3612740795 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9f4c0a2890 0x7f9f4c0a4d50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:42:16.092 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:42:16.091+0000 7f9f6764e700 1 --2- 172.21.15.151:0/3612740795 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f60102c50 0x7f9f6019e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:42:16.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:42:16.091+0000 7f9f6764e700 1 --2- 172.21.15.151:0/3612740795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f601010b0 0x7f9f60197290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:42:16.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:42:16.091+0000 7f9f6764e700 1 --2- 172.21.15.151:0/3612740795 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f601022b0 0x7f9f601977d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:42:16.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:42:16.091+0000 7f9f6764e700 1 -- 172.21.15.151:0/3612740795 >> 172.21.15.151:0/3612740795 conn(0x7f9f600fc670 msgr2=0x7f9f60105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:42:16.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:42:16.092+0000 7f9f6764e700 1 -- 172.21.15.151:0/3612740795 shutdown_connections 2024-03-28T17:42:16.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:42:16.092+0000 7f9f6764e700 1 -- 172.21.15.151:0/3612740795 wait complete. 2024-03-28T17:42:16.784 DEBUG:teuthology.orchestra.run.smithi151:osd.5> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@osd.5.service 2024-03-28T17:42:16.786 INFO:tasks.cephadm:Deploying osd.6 on smithi045 with /dev/vg_nvme/lv_2... 2024-03-28T17:42:16.786 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T17:42:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:17 smithi045 ceph-mon[31773]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:17 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:17 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:17 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:42:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:17 smithi045 ceph-mon[31773]: from='osd.5 [v2:172.21.15.151:6808/2629217227,v1:172.21.15.151:6809/2629217227]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T17:42:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:17 smithi045 ceph-mon[31773]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T17:42:17.261 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:42:17.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:17 smithi071 ceph-mon[36597]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:17.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:17 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:17.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:17 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:17.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:17 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:42:17.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:17 smithi071 ceph-mon[36597]: from='osd.5 [v2:172.21.15.151:6808/2629217227,v1:172.21.15.151:6809/2629217227]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T17:42:17.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:17 smithi071 ceph-mon[36597]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T17:42:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:17 smithi151 ceph-mon[36481]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:17 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:17 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:17 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:42:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:17 smithi151 ceph-mon[36481]: from='osd.5 [v2:172.21.15.151:6808/2629217227,v1:172.21.15.151:6809/2629217227]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T17:42:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:17 smithi151 ceph-mon[36481]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T17:42:18.203 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:18 smithi151 ceph-mon[36481]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T17:42:18.203 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:18 smithi151 ceph-mon[36481]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T17:42:18.203 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:18 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T17:42:18.203 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:18 smithi151 ceph-mon[36481]: from='osd.5 [v2:172.21.15.151:6808/2629217227,v1:172.21.15.151:6809/2629217227]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:42:18.203 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:18 smithi151 ceph-mon[36481]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:42:18.203 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 28 17:42:18 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-osd-5[48182]: 2024-03-28T17:42:18.074+0000 7fb4606a7700 -1 osd.5 0 waiting for initial osdmap 2024-03-28T17:42:18.204 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 28 17:42:18 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-osd-5[48182]: 2024-03-28T17:42:18.083+0000 7fb45c49a700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T17:42:18.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:18 smithi045 ceph-mon[31773]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T17:42:18.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:18 smithi045 ceph-mon[31773]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T17:42:18.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:18 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T17:42:18.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:18 smithi045 ceph-mon[31773]: from='osd.5 [v2:172.21.15.151:6808/2629217227,v1:172.21.15.151:6809/2629217227]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:42:18.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:18 smithi045 ceph-mon[31773]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:42:18.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:18 smithi071 ceph-mon[36597]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T17:42:18.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:18 smithi071 ceph-mon[36597]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T17:42:18.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:18 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T17:42:18.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:18 smithi071 ceph-mon[36597]: from='osd.5 [v2:172.21.15.151:6808/2629217227,v1:172.21.15.151:6809/2629217227]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:42:18.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:18 smithi071 ceph-mon[36597]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:42:19.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:19 smithi045 ceph-mon[31773]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:19.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:19 smithi045 ceph-mon[31773]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-28T17:42:19.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:19 smithi045 ceph-mon[31773]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T17:42:19.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:19 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T17:42:19.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:19 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T17:42:19.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:19 smithi045 ceph-mon[31773]: osd.5 [v2:172.21.15.151:6808/2629217227,v1:172.21.15.151:6809/2629217227] boot 2024-03-28T17:42:19.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:19 smithi045 ceph-mon[31773]: osdmap e38: 6 total, 6 up, 6 in 2024-03-28T17:42:19.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:19 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T17:42:19.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:19 smithi151 ceph-mon[36481]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:19 smithi151 ceph-mon[36481]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-28T17:42:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:19 smithi151 ceph-mon[36481]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T17:42:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:19 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T17:42:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:19 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T17:42:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:19 smithi151 ceph-mon[36481]: osd.5 [v2:172.21.15.151:6808/2629217227,v1:172.21.15.151:6809/2629217227] boot 2024-03-28T17:42:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:19 smithi151 ceph-mon[36481]: osdmap e38: 6 total, 6 up, 6 in 2024-03-28T17:42:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:19 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T17:42:19.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:19 smithi071 ceph-mon[36597]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-28T17:42:19.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:19 smithi071 ceph-mon[36597]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-28T17:42:19.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:19 smithi071 ceph-mon[36597]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T17:42:19.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:19 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T17:42:19.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:19 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T17:42:19.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:19 smithi071 ceph-mon[36597]: osd.5 [v2:172.21.15.151:6808/2629217227,v1:172.21.15.151:6809/2629217227] boot 2024-03-28T17:42:19.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:19 smithi071 ceph-mon[36597]: osdmap e38: 6 total, 6 up, 6 in 2024-03-28T17:42:19.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:19 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T17:42:20.107 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T17:42:20.107 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T17:42:20.107 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2024-03-28T17:42:20.107 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2024-03-28T17:42:20.107 INFO:teuthology.orchestra.run.smithi045.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0435164 s, 241 MB/s 2024-03-28T17:42:20.107 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2024-03-28T17:42:20.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:20 smithi045 ceph-mon[31773]: purged_snaps scrub starts 2024-03-28T17:42:20.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:20 smithi045 ceph-mon[31773]: purged_snaps scrub ok 2024-03-28T17:42:20.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:20 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:20.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:20 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:20.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:20 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:42:20.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:20 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:42:20.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:20 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:20.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:20 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:20.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:20 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:42:20.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:20 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:20.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:20 smithi045 ceph-mon[31773]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T17:42:20.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:20 smithi071 ceph-mon[36597]: purged_snaps scrub starts 2024-03-28T17:42:20.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:20 smithi071 ceph-mon[36597]: purged_snaps scrub ok 2024-03-28T17:42:20.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:20.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:20.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:42:20.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:42:20.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:20.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:20.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:42:20.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:20.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:20 smithi071 ceph-mon[36597]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T17:42:20.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:20 smithi151 ceph-mon[36481]: purged_snaps scrub starts 2024-03-28T17:42:20.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:20 smithi151 ceph-mon[36481]: purged_snaps scrub ok 2024-03-28T17:42:20.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:20 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:20.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:20 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:20.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:20 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:42:20.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:20 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:42:20.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:20 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:20.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:20 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:20.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:20 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:42:20.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:20 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:20.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:20 smithi151 ceph-mon[36481]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T17:42:21.030 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi045:vg_nvme/lv_2 2024-03-28T17:42:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:21 smithi045 ceph-mon[31773]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:21 smithi045 ceph-mon[31773]: Detected new or changed devices on smithi151 2024-03-28T17:42:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:21 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi151 to 8540M 2024-03-28T17:42:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:21 smithi045 ceph-mon[31773]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T17:42:21.475 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:21 smithi151 ceph-mon[36481]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:21.475 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:21 smithi151 ceph-mon[36481]: Detected new or changed devices on smithi151 2024-03-28T17:42:21.475 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:21 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi151 to 8540M 2024-03-28T17:42:21.475 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:21 smithi151 ceph-mon[36481]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T17:42:21.497 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:42:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:21 smithi071 ceph-mon[36597]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:21.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:21 smithi071 ceph-mon[36597]: Detected new or changed devices on smithi151 2024-03-28T17:42:21.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:21 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi151 to 8540M 2024-03-28T17:42:21.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:21 smithi071 ceph-mon[36597]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T17:42:23.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:23 smithi045 ceph-mon[31773]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 164 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:23.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:23 smithi045 ceph-mon[31773]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T17:42:23.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:23 smithi071 ceph-mon[36597]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 164 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:23.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:23 smithi071 ceph-mon[36597]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T17:42:23.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:23 smithi151 ceph-mon[36481]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 164 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:23.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:23 smithi151 ceph-mon[36481]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T17:42:23.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.882+0000 7fdfe7200700 1 -- 172.21.15.45:0/557455211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdfe0107dd0 msgr2=0x7fdfe010a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:42:23.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.882+0000 7fdfe7200700 1 --2- 172.21.15.45:0/557455211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdfe0107dd0 0x7fdfe010a240 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc00b0a0 tx=0x7fdfdc00b3b0 comp rx=0 tx=0).stop 2024-03-28T17:42:23.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.882+0000 7fdfe7200700 1 -- 172.21.15.45:0/557455211 shutdown_connections 2024-03-28T17:42:23.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.882+0000 7fdfe7200700 1 --2- 172.21.15.45:0/557455211 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfe0102b70 0x7fdfe0104f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:42:23.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.882+0000 7fdfe7200700 1 --2- 172.21.15.45:0/557455211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdfe0107dd0 0x7fdfe010a240 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:42:23.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.882+0000 7fdfe7200700 1 --2- 172.21.15.45:0/557455211 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfe01054a0 0x7fdfe0107890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:42:23.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.882+0000 7fdfe7200700 1 -- 172.21.15.45:0/557455211 >> 172.21.15.45:0/557455211 conn(0x7fdfe00fc560 msgr2=0x7fdfe00fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:42:23.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.882+0000 7fdfe7200700 1 -- 172.21.15.45:0/557455211 shutdown_connections 2024-03-28T17:42:23.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.882+0000 7fdfe7200700 1 -- 172.21.15.45:0/557455211 wait complete. 2024-03-28T17:42:23.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.883+0000 7fdfe7200700 1 Processor -- start 2024-03-28T17:42:23.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.884+0000 7fdfe7200700 1 -- start start 2024-03-28T17:42:23.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.884+0000 7fdfe7200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdfe0102b70 0x7fdfe019a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:42:23.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.884+0000 7fdfe7200700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfe01054a0 0x7fdfe019a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:42:23.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.884+0000 7fdfe7200700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfe0107dd0 0x7fdfe01a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:42:23.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.884+0000 7fdfe7200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe010dd40 con 0x7fdfe0102b70 2024-03-28T17:42:23.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.884+0000 7fdfe7200700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe010dbc0 con 0x7fdfe0107dd0 2024-03-28T17:42:23.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.884+0000 7fdfe7200700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe010dec0 con 0x7fdfe01054a0 2024-03-28T17:42:23.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.884+0000 7fdfe5e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdfe0102b70 0x7fdfe019a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:42:23.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.884+0000 7fdfe5e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdfe0102b70 0x7fdfe019a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42450/0 (socket says 172.21.15.45:42450) 2024-03-28T17:42:23.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.884+0000 7fdfe5e00700 1 -- 172.21.15.45:0/3781508275 learned_addr learned my addr 172.21.15.45:0/3781508275 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:42:23.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.884+0000 7fdfe6800700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfe0107dd0 0x7fdfe01a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:42:23.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.884+0000 7fdfe5400700 1 --2- 172.21.15.45:0/3781508275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfe01054a0 0x7fdfe019a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:42:23.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.885+0000 7fdfe6800700 1 -- 172.21.15.45:0/3781508275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfe01054a0 msgr2=0x7fdfe019a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:42:23.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.885+0000 7fdfe6800700 1 --2- 172.21.15.45:0/3781508275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfe01054a0 0x7fdfe019a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:42:23.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.885+0000 7fdfe6800700 1 -- 172.21.15.45:0/3781508275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdfe0102b70 msgr2=0x7fdfe019a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:42:23.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.885+0000 7fdfe6800700 1 --2- 172.21.15.45:0/3781508275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdfe0102b70 0x7fdfe019a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:42:23.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.885+0000 7fdfe6800700 1 -- 172.21.15.45:0/3781508275 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe01a2100 con 0x7fdfe0107dd0 2024-03-28T17:42:23.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.885+0000 7fdfe6800700 1 --2- 172.21.15.45:0/3781508275 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfe0107dd0 0x7fdfe01a1a60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc007fa0 tx=0x7fdfdc004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:42:23.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.885+0000 7fdfdaa00700 1 -- 172.21.15.45:0/3781508275 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc021070 con 0x7fdfe0107dd0 2024-03-28T17:42:23.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.885+0000 7fdfdaa00700 1 -- 172.21.15.45:0/3781508275 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdfdc007b80 con 0x7fdfe0107dd0 2024-03-28T17:42:23.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.885+0000 7fdfe7200700 1 -- 172.21.15.45:0/3781508275 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe01a2390 con 0x7fdfe0107dd0 2024-03-28T17:42:23.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.885+0000 7fdfdaa00700 1 -- 172.21.15.45:0/3781508275 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc004750 con 0x7fdfe0107dd0 2024-03-28T17:42:23.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.886+0000 7fdfe7200700 1 -- 172.21.15.45:0/3781508275 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe01a2840 con 0x7fdfe0107dd0 2024-03-28T17:42:23.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.887+0000 7fdfe7200700 1 -- 172.21.15.45:0/3781508275 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfe0068e50 con 0x7fdfe0107dd0 2024-03-28T17:42:23.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.889+0000 7fdfdaa00700 1 -- 172.21.15.45:0/3781508275 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdfdc003a00 con 0x7fdfe0107dd0 2024-03-28T17:42:23.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.890+0000 7fdfdaa00700 1 --2- 172.21.15.45:0/3781508275 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdfc80a2890 0x7fdfc80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:42:23.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.890+0000 7fdfdaa00700 1 -- 172.21.15.45:0/3781508275 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4533+0+0 (secure 0 0 0) 0x7fdfdc00f020 con 0x7fdfe0107dd0 2024-03-28T17:42:23.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.890+0000 7fdfe5e00700 1 --2- 172.21.15.45:0/3781508275 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdfc80a2890 0x7fdfc80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:42:23.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.891+0000 7fdfe5e00700 1 --2- 172.21.15.45:0/3781508275 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdfc80a2890 0x7fdfc80a4d50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4009680 tx=0x7fdfd4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:42:23.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:23.893+0000 7fdfdaa00700 1 -- 172.21.15.45:0/3781508275 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdfdc02a020 con 0x7fdfe0107dd0 2024-03-28T17:42:24.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:24.133+0000 7fdfe7200700 1 -- 172.21.15.45:0/3781508275 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fdfe00625d0 con 0x7fdfc80a2890 2024-03-28T17:42:24.134 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:24 smithi045 ceph-mon[31773]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T17:42:24.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:24 smithi071 ceph-mon[36597]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T17:42:24.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:24 smithi151 ceph-mon[36481]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T17:42:25.413 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:25 smithi045 ceph-mon[31773]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 164 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:25.413 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:25 smithi045 ceph-mon[31773]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:42:25.413 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:42:25.413 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:42:25.413 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:25.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:25 smithi071 ceph-mon[36597]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 164 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:25 smithi071 ceph-mon[36597]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:42:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:42:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:42:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:25.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:25 smithi151 ceph-mon[36481]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 164 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:25.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:25 smithi151 ceph-mon[36481]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:42:25.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:42:25.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:42:25.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:27 smithi045 ceph-mon[31773]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:27 smithi071 ceph-mon[36597]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:27 smithi151 ceph-mon[36481]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:29.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:29 smithi045 ceph-mon[31773]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-28T17:42:29.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:29 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1908385822' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58a6c5bd-a8b7-4892-b346-bcd7994bcb6b"}]: dispatch 2024-03-28T17:42:29.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:29 smithi045 ceph-mon[31773]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58a6c5bd-a8b7-4892-b346-bcd7994bcb6b"}]: dispatch 2024-03-28T17:42:29.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:29 smithi045 ceph-mon[31773]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58a6c5bd-a8b7-4892-b346-bcd7994bcb6b"}]': finished 2024-03-28T17:42:29.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:29 smithi045 ceph-mon[31773]: osdmap e43: 7 total, 6 up, 7 in 2024-03-28T17:42:29.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:29 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T17:42:29.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:29 smithi071 ceph-mon[36597]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-28T17:42:29.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:29 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1908385822' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58a6c5bd-a8b7-4892-b346-bcd7994bcb6b"}]: dispatch 2024-03-28T17:42:29.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:29 smithi071 ceph-mon[36597]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58a6c5bd-a8b7-4892-b346-bcd7994bcb6b"}]: dispatch 2024-03-28T17:42:29.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:29 smithi071 ceph-mon[36597]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58a6c5bd-a8b7-4892-b346-bcd7994bcb6b"}]': finished 2024-03-28T17:42:29.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:29 smithi071 ceph-mon[36597]: osdmap e43: 7 total, 6 up, 7 in 2024-03-28T17:42:29.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:29 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T17:42:29.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:29 smithi151 ceph-mon[36481]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-28T17:42:29.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:29 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1908385822' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58a6c5bd-a8b7-4892-b346-bcd7994bcb6b"}]: dispatch 2024-03-28T17:42:29.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:29 smithi151 ceph-mon[36481]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58a6c5bd-a8b7-4892-b346-bcd7994bcb6b"}]: dispatch 2024-03-28T17:42:29.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:29 smithi151 ceph-mon[36481]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58a6c5bd-a8b7-4892-b346-bcd7994bcb6b"}]': finished 2024-03-28T17:42:29.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:29 smithi151 ceph-mon[36481]: osdmap e43: 7 total, 6 up, 7 in 2024-03-28T17:42:29.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:29 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T17:42:30.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:30 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/2254294753' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:42:30.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:30 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/2254294753' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:42:30.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:30 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/2254294753' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:42:31.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:31 smithi045 ceph-mon[31773]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-28T17:42:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:31 smithi071 ceph-mon[36597]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-28T17:42:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:31 smithi151 ceph-mon[36481]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-28T17:42:33.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:33 smithi045 ceph-mon[31773]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-28T17:42:33.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:33 smithi071 ceph-mon[36597]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-28T17:42:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:33 smithi151 ceph-mon[36481]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-28T17:42:35.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:35 smithi045 ceph-mon[31773]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T17:42:35.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:35 smithi071 ceph-mon[36597]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T17:42:35.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:35 smithi151 ceph-mon[36481]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T17:42:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:37 smithi045 ceph-mon[31773]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T17:42:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:37 smithi071 ceph-mon[36597]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T17:42:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:37 smithi151 ceph-mon[36481]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T17:42:39.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:39 smithi045 ceph-mon[31773]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:39.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:39 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T17:42:39.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:39 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:39.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:39 smithi045 ceph-mon[31773]: Deploying daemon osd.6 on smithi045 2024-03-28T17:42:39.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:39 smithi071 ceph-mon[36597]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:39.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:39 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T17:42:39.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:39 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:39.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:39 smithi071 ceph-mon[36597]: Deploying daemon osd.6 on smithi045 2024-03-28T17:42:39.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:39 smithi151 ceph-mon[36481]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:39.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:39 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T17:42:39.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:39 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:39.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:39 smithi151 ceph-mon[36481]: Deploying daemon osd.6 on smithi045 2024-03-28T17:42:41.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:41 smithi045 ceph-mon[31773]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:41 smithi071 ceph-mon[36597]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:41 smithi151 ceph-mon[36481]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:43.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:43 smithi045 ceph-mon[31773]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:43.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:43 smithi071 ceph-mon[36597]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:43.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:43 smithi151 ceph-mon[36481]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:45.387 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:45 smithi045 ceph-mon[31773]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:45.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:45 smithi071 ceph-mon[36597]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:45.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:45 smithi151 ceph-mon[36481]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:47.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:47 smithi045 ceph-mon[31773]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:47 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:47 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:47 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:42:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:47 smithi071 ceph-mon[36597]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:47 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:47 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:47.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:47 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:42:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:47 smithi151 ceph-mon[36481]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:47 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:47 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:47 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:42:49.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:49 smithi071 ceph-mon[36597]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:49 smithi151 ceph-mon[36481]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:49.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:49 smithi045 ceph-mon[31773]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:51.376 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:51 smithi151 ceph-mon[36481]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:51.376 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:51 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:51.376 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:51 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:51.376 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:51 smithi151 ceph-mon[36481]: from='osd.6 [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T17:42:51.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:51 smithi045 ceph-mon[31773]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:51.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:51 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:51.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:51 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:51.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:51 smithi045 ceph-mon[31773]: from='osd.6 [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T17:42:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:51 smithi071 ceph-mon[36597]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:51 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:51 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:51.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:51 smithi071 ceph-mon[36597]: from='osd.6 [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T17:42:51.858 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 6 on host 'smithi045' 2024-03-28T17:42:51.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:51.857+0000 7fdfdaa00700 1 -- 172.21.15.45:0/3781508275 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fdfe00625d0 con 0x7fdfc80a2890 2024-03-28T17:42:51.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:51.861+0000 7fdfe7200700 1 -- 172.21.15.45:0/3781508275 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdfc80a2890 msgr2=0x7fdfc80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:42:51.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:51.861+0000 7fdfe7200700 1 --2- 172.21.15.45:0/3781508275 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdfc80a2890 0x7fdfc80a4d50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4009680 tx=0x7fdfd4009260 comp rx=0 tx=0).stop 2024-03-28T17:42:51.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:51.861+0000 7fdfe7200700 1 -- 172.21.15.45:0/3781508275 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfe0107dd0 msgr2=0x7fdfe01a1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:42:51.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:51.861+0000 7fdfe7200700 1 --2- 172.21.15.45:0/3781508275 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfe0107dd0 0x7fdfe01a1a60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc007fa0 tx=0x7fdfdc004c80 comp rx=0 tx=0).stop 2024-03-28T17:42:51.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:51.861+0000 7fdfe7200700 1 -- 172.21.15.45:0/3781508275 shutdown_connections 2024-03-28T17:42:51.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:51.861+0000 7fdfe7200700 1 --2- 172.21.15.45:0/3781508275 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdfc80a2890 0x7fdfc80a4d50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:42:51.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:51.861+0000 7fdfe7200700 1 --2- 172.21.15.45:0/3781508275 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfe0107dd0 0x7fdfe01a1a60 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:42:51.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:51.861+0000 7fdfe7200700 1 --2- 172.21.15.45:0/3781508275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdfe0102b70 0x7fdfe019a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:42:51.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:51.861+0000 7fdfe7200700 1 --2- 172.21.15.45:0/3781508275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfe01054a0 0x7fdfe019a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:42:51.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:51.861+0000 7fdfe7200700 1 -- 172.21.15.45:0/3781508275 >> 172.21.15.45:0/3781508275 conn(0x7fdfe00fc560 msgr2=0x7fdfe0102250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:42:51.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:51.861+0000 7fdfe7200700 1 -- 172.21.15.45:0/3781508275 shutdown_connections 2024-03-28T17:42:51.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:42:51.861+0000 7fdfe7200700 1 -- 172.21.15.45:0/3781508275 wait complete. 2024-03-28T17:42:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:52 smithi071 ceph-mon[36597]: from='osd.6 [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T17:42:52.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:52 smithi071 ceph-mon[36597]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T17:42:52.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:52 smithi071 ceph-mon[36597]: from='osd.6 [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-28T17:42:52.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:52 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T17:42:52.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:52 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:52.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:52 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:42:52.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:52 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:52.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:52 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:52.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:52 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:42:52.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:52 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:52 smithi151 ceph-mon[36481]: from='osd.6 [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T17:42:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:52 smithi151 ceph-mon[36481]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T17:42:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:52 smithi151 ceph-mon[36481]: from='osd.6 [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-28T17:42:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:52 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T17:42:52.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:52 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:52.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:52 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:42:52.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:52 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:52.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:52 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:52.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:52 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:42:52.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:52 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:52.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:52 smithi045 ceph-mon[31773]: from='osd.6 [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T17:42:52.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:52 smithi045 ceph-mon[31773]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T17:42:52.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:52 smithi045 ceph-mon[31773]: from='osd.6 [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-28T17:42:52.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:52 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T17:42:52.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:52 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:52.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:52 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:42:52.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:52 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:52.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:52 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:52.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:52 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:42:52.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:52 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:52.727 DEBUG:teuthology.orchestra.run.smithi045:osd.6> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@osd.6.service 2024-03-28T17:42:52.732 INFO:tasks.cephadm:Deploying osd.7 on smithi071 with /dev/vg_nvme/lv_2... 2024-03-28T17:42:52.732 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T17:42:52.977 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T17:42:52.988 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 28 17:42:52 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-osd-6[59771]: 2024-03-28T17:42:52.876+0000 7f1370400700 -1 osd.6 0 waiting for initial osdmap 2024-03-28T17:42:52.988 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 28 17:42:52 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-osd-6[59771]: 2024-03-28T17:42:52.901+0000 7f1366200700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T17:42:53.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:53 smithi045 ceph-mon[31773]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:53.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:53 smithi045 ceph-mon[31773]: from='osd.6 [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-28T17:42:53.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:53 smithi045 ceph-mon[31773]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T17:42:53.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:53 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T17:42:53.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:53 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T17:42:53.283 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:53 smithi045 ceph-mon[31773]: from='osd.6 [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039]' entity='osd.6' 2024-03-28T17:42:53.354 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:53 smithi071 ceph-mon[36597]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:53.354 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:53 smithi071 ceph-mon[36597]: from='osd.6 [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-28T17:42:53.354 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:53 smithi071 ceph-mon[36597]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T17:42:53.354 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:53 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T17:42:53.354 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:53 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T17:42:53.355 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:53 smithi071 ceph-mon[36597]: from='osd.6 [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039]' entity='osd.6' 2024-03-28T17:42:53.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:53 smithi151 ceph-mon[36481]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:53.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:53 smithi151 ceph-mon[36481]: from='osd.6 [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-28T17:42:53.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:53 smithi151 ceph-mon[36481]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T17:42:53.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:53 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T17:42:53.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:53 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T17:42:53.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:53 smithi151 ceph-mon[36481]: from='osd.6 [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039]' entity='osd.6' 2024-03-28T17:42:54.412 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:54 smithi071 ceph-mon[36597]: purged_snaps scrub starts 2024-03-28T17:42:54.412 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:54 smithi071 ceph-mon[36597]: purged_snaps scrub ok 2024-03-28T17:42:54.412 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:54 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T17:42:54.412 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:54 smithi071 ceph-mon[36597]: osd.6 [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039] boot 2024-03-28T17:42:54.412 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:54 smithi071 ceph-mon[36597]: osdmap e46: 7 total, 7 up, 7 in 2024-03-28T17:42:54.412 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:54 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T17:42:54.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:54 smithi045 ceph-mon[31773]: purged_snaps scrub starts 2024-03-28T17:42:54.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:54 smithi045 ceph-mon[31773]: purged_snaps scrub ok 2024-03-28T17:42:54.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:54 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T17:42:54.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:54 smithi045 ceph-mon[31773]: osd.6 [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039] boot 2024-03-28T17:42:54.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:54 smithi045 ceph-mon[31773]: osdmap e46: 7 total, 7 up, 7 in 2024-03-28T17:42:54.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:54 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T17:42:54.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:54 smithi151 ceph-mon[36481]: purged_snaps scrub starts 2024-03-28T17:42:54.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:54 smithi151 ceph-mon[36481]: purged_snaps scrub ok 2024-03-28T17:42:54.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:54 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T17:42:54.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:54 smithi151 ceph-mon[36481]: osd.6 [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039] boot 2024-03-28T17:42:54.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:54 smithi151 ceph-mon[36481]: osdmap e46: 7 total, 7 up, 7 in 2024-03-28T17:42:54.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:54 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T17:42:54.960 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T17:42:54.961 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T17:42:54.961 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10+0 records in 2024-03-28T17:42:54.961 INFO:teuthology.orchestra.run.smithi071.stderr:10+0 records out 2024-03-28T17:42:54.961 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129175 s, 812 MB/s 2024-03-28T17:42:54.961 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping successful for: 2024-03-28T17:42:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:55 smithi071 ceph-mon[36597]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:55.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:55 smithi151 ceph-mon[36481]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:55.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:55 smithi045 ceph-mon[31773]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-28T17:42:55.687 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi071:vg_nvme/lv_2 2024-03-28T17:42:55.926 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T17:42:56.300 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:56 smithi071 ceph-mon[36597]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T17:42:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:56 smithi151 ceph-mon[36481]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T17:42:56.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:56 smithi045 ceph-mon[31773]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T17:42:57.459 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.456+0000 7fb47cf27700 1 -- 172.21.15.71:0/2129239702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4781022c0 msgr2=0x7fb478102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:42:57.459 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.456+0000 7fb47cf27700 1 --2- 172.21.15.71:0/2129239702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4781022c0 0x7fb478102720 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb4680099c0 tx=0x7fb468009cd0 comp rx=0 tx=0).stop 2024-03-28T17:42:57.459 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.456+0000 7fb47cf27700 1 -- 172.21.15.71:0/2129239702 shutdown_connections 2024-03-28T17:42:57.459 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.456+0000 7fb47cf27700 1 --2- 172.21.15.71:0/2129239702 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4781010c0 0x7fb4781014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:42:57.459 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.456+0000 7fb47cf27700 1 --2- 172.21.15.71:0/2129239702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb478102c60 0x7fb478109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:42:57.459 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.456+0000 7fb47cf27700 1 --2- 172.21.15.71:0/2129239702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4781022c0 0x7fb478102720 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:42:57.459 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.456+0000 7fb47cf27700 1 -- 172.21.15.71:0/2129239702 >> 172.21.15.71:0/2129239702 conn(0x7fb4780fc660 msgr2=0x7fb4780feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:42:57.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.456+0000 7fb47cf27700 1 -- 172.21.15.71:0/2129239702 shutdown_connections 2024-03-28T17:42:57.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.456+0000 7fb47cf27700 1 -- 172.21.15.71:0/2129239702 wait complete. 2024-03-28T17:42:57.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.457+0000 7fb47cf27700 1 Processor -- start 2024-03-28T17:42:57.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.457+0000 7fb47cf27700 1 -- start start 2024-03-28T17:42:57.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.457+0000 7fb47cf27700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4781010c0 0x7fb478197460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:42:57.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.458+0000 7fb47cf27700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4781022c0 0x7fb4781979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:42:57.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.458+0000 7fb47cf27700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb478102c60 0x7fb47819ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:42:57.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.458+0000 7fb47cf27700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47810cda0 con 0x7fb478102c60 2024-03-28T17:42:57.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.458+0000 7fb47cf27700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47810cc20 con 0x7fb4781010c0 2024-03-28T17:42:57.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.458+0000 7fb47cf27700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47810cf20 con 0x7fb4781022c0 2024-03-28T17:42:57.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.458+0000 7fb47659c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4781010c0 0x7fb478197460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:42:57.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.458+0000 7fb47659c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4781010c0 0x7fb478197460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:37532/0 (socket says 172.21.15.71:37532) 2024-03-28T17:42:57.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.458+0000 7fb47659c700 1 -- 172.21.15.71:0/3803041670 learned_addr learned my addr 172.21.15.71:0/3803041670 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-03-28T17:42:57.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.458+0000 7fb475d9b700 1 --2- 172.21.15.71:0/3803041670 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4781022c0 0x7fb4781979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:42:57.461 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.458+0000 7fb47659c700 1 -- 172.21.15.71:0/3803041670 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4781022c0 msgr2=0x7fb4781979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:42:57.461 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.458+0000 7fb47659c700 1 --2- 172.21.15.71:0/3803041670 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4781022c0 0x7fb4781979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:42:57.461 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.458+0000 7fb47659c700 1 -- 172.21.15.71:0/3803041670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb478102c60 msgr2=0x7fb47819ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:42:57.461 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.458+0000 7fb47659c700 1 --2- 172.21.15.71:0/3803041670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb478102c60 0x7fb47819ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:42:57.461 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.458+0000 7fb47659c700 1 -- 172.21.15.71:0/3803041670 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47819f0c0 con 0x7fb4781010c0 2024-03-28T17:42:57.461 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.458+0000 7fb47659c700 1 --2- 172.21.15.71:0/3803041670 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4781010c0 0x7fb478197460 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00b500 tx=0x7fb46c00b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:42:57.461 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.458+0000 7fb4677fe700 1 -- 172.21.15.71:0/3803041670 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c015020 con 0x7fb4781010c0 2024-03-28T17:42:57.461 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.458+0000 7fb47cf27700 1 -- 172.21.15.71:0/3803041670 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47819f3b0 con 0x7fb4781010c0 2024-03-28T17:42:57.461 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.458+0000 7fb4677fe700 1 -- 172.21.15.71:0/3803041670 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb46c011d10 con 0x7fb4781010c0 2024-03-28T17:42:57.461 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.458+0000 7fb4677fe700 1 -- 172.21.15.71:0/3803041670 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c00f2e0 con 0x7fb4781010c0 2024-03-28T17:42:57.461 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.458+0000 7fb47cf27700 1 -- 172.21.15.71:0/3803041670 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47819f970 con 0x7fb4781010c0 2024-03-28T17:42:57.462 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.459+0000 7fb47cf27700 1 -- 172.21.15.71:0/3803041670 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb478068e50 con 0x7fb4781010c0 2024-03-28T17:42:57.462 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.460+0000 7fb4677fe700 1 -- 172.21.15.71:0/3803041670 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb46c018020 con 0x7fb4781010c0 2024-03-28T17:42:57.462 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.461+0000 7fb4677fe700 1 --2- 172.21.15.71:0/3803041670 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb4600a2890 0x7fb4600a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:42:57.462 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.461+0000 7fb4677fe700 1 -- 172.21.15.71:0/3803041670 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4965+0+0 (secure 0 0 0) 0x7fb46c0cd550 con 0x7fb4781010c0 2024-03-28T17:42:57.462 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.461+0000 7fb475d9b700 1 --2- 172.21.15.71:0/3803041670 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb4600a2890 0x7fb4600a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:42:57.463 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.462+0000 7fb475d9b700 1 --2- 172.21.15.71:0/3803041670 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb4600a2890 0x7fb4600a4d50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb4680039c0 tx=0x7fb468000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:42:57.465 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.465+0000 7fb4677fe700 1 -- 172.21.15.71:0/3803041670 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb46c01f020 con 0x7fb4781010c0 2024-03-28T17:42:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:57 smithi071 ceph-mon[36597]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:42:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:57 smithi151 ceph-mon[36481]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:42:57.660 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:57 smithi045 ceph-mon[31773]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:42:57.666 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:42:57.665+0000 7fb47cf27700 1 -- 172.21.15.71:0/3803041670 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4780625d0 con 0x7fb4600a2890 2024-03-28T17:42:58.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:58 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:42:58.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:58 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:42:58.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:58 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:58.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:58 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:42:58.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:58 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:42:58.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:58 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:58.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:58 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:42:58.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:58 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:42:58.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:58 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:59.530 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:59 smithi071 ceph-mon[36597]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:42:59.531 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:59 smithi071 ceph-mon[36597]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:42:59.531 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:59 smithi071 ceph-mon[36597]: Detected new or changed devices on smithi045 2024-03-28T17:42:59.531 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:59 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:59.531 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:59 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:59.531 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:59 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:42:59.531 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:59 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:42:59.531 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:59 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:42:59.531 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:59 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi045 to 5698M 2024-03-28T17:42:59.531 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:59 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:59.531 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:59 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:59.531 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:59 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:42:59.531 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:42:59 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:59.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:59 smithi151 ceph-mon[36481]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:42:59.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:59 smithi151 ceph-mon[36481]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:42:59.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:59 smithi151 ceph-mon[36481]: Detected new or changed devices on smithi045 2024-03-28T17:42:59.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:59 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:59.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:59 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:59.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:59 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:42:59.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:59 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:42:59.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:59 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:42:59.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:59 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi045 to 5698M 2024-03-28T17:42:59.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:59 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:59.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:59 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:59.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:59 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:42:59.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:42:59 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:59.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:59 smithi045 ceph-mon[31773]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:42:59.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:59 smithi045 ceph-mon[31773]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:42:59.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:59 smithi045 ceph-mon[31773]: Detected new or changed devices on smithi045 2024-03-28T17:42:59.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:59.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:59.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:42:59.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:42:59.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:42:59.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:59 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi045 to 5698M 2024-03-28T17:42:59.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:42:59.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:42:59.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:42:59.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:42:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:01.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:01 smithi071 ceph-mon[36597]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:01.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:01 smithi071 ceph-mon[36597]: from='client.? 172.21.15.71:0/276038174' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7a45d52-807b-4407-bb04-787b2a52a0ac"}]: dispatch 2024-03-28T17:43:01.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:01 smithi071 ceph-mon[36597]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7a45d52-807b-4407-bb04-787b2a52a0ac"}]: dispatch 2024-03-28T17:43:01.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:01 smithi071 ceph-mon[36597]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7a45d52-807b-4407-bb04-787b2a52a0ac"}]': finished 2024-03-28T17:43:01.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:01 smithi071 ceph-mon[36597]: osdmap e48: 8 total, 7 up, 8 in 2024-03-28T17:43:01.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:01 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T17:43:01.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:01 smithi071 ceph-mon[36597]: from='client.? 172.21.15.71:0/951471458' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:43:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:01 smithi151 ceph-mon[36481]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:01 smithi151 ceph-mon[36481]: from='client.? 172.21.15.71:0/276038174' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7a45d52-807b-4407-bb04-787b2a52a0ac"}]: dispatch 2024-03-28T17:43:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:01 smithi151 ceph-mon[36481]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7a45d52-807b-4407-bb04-787b2a52a0ac"}]: dispatch 2024-03-28T17:43:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:01 smithi151 ceph-mon[36481]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7a45d52-807b-4407-bb04-787b2a52a0ac"}]': finished 2024-03-28T17:43:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:01 smithi151 ceph-mon[36481]: osdmap e48: 8 total, 7 up, 8 in 2024-03-28T17:43:01.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:01 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T17:43:01.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:01 smithi151 ceph-mon[36481]: from='client.? 172.21.15.71:0/951471458' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:43:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:01 smithi045 ceph-mon[31773]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:01 smithi045 ceph-mon[31773]: from='client.? 172.21.15.71:0/276038174' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7a45d52-807b-4407-bb04-787b2a52a0ac"}]: dispatch 2024-03-28T17:43:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:01 smithi045 ceph-mon[31773]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7a45d52-807b-4407-bb04-787b2a52a0ac"}]: dispatch 2024-03-28T17:43:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:01 smithi045 ceph-mon[31773]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7a45d52-807b-4407-bb04-787b2a52a0ac"}]': finished 2024-03-28T17:43:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:01 smithi045 ceph-mon[31773]: osdmap e48: 8 total, 7 up, 8 in 2024-03-28T17:43:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:01 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T17:43:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:01 smithi045 ceph-mon[31773]: from='client.? 172.21.15.71:0/951471458' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:43:03.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:03 smithi071 ceph-mon[36597]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:03 smithi151 ceph-mon[36481]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:03.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:03 smithi045 ceph-mon[31773]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:05.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:05 smithi071 ceph-mon[36597]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:05.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:05 smithi151 ceph-mon[36481]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:05.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:05 smithi045 ceph-mon[31773]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:07.511 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:07 smithi071 ceph-mon[36597]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:07 smithi151 ceph-mon[36481]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:07 smithi045 ceph-mon[31773]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:08.557 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T17:43:08.557 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:08.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T17:43:08.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:08.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T17:43:08.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:09.389 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:09 smithi071 ceph-mon[36597]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:09.390 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:09 smithi071 ceph-mon[36597]: Deploying daemon osd.7 on smithi071 2024-03-28T17:43:09.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:09 smithi151 ceph-mon[36481]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:09.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:09 smithi151 ceph-mon[36481]: Deploying daemon osd.7 on smithi071 2024-03-28T17:43:09.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:09 smithi045 ceph-mon[31773]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:09.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:09 smithi045 ceph-mon[31773]: Deploying daemon osd.7 on smithi071 2024-03-28T17:43:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:11 smithi071 ceph-mon[36597]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:11 smithi151 ceph-mon[36481]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:11 smithi045 ceph-mon[31773]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:13.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:13 smithi071 ceph-mon[36597]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:13.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:13 smithi151 ceph-mon[36481]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:13.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:13 smithi045 ceph-mon[31773]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:14.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:14 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:14.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:14 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:14.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:14 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:43:14.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:14 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:14.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:14 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:14.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:14 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:43:14.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:14 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:14.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:14 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:14.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:14 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:43:15.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:15 smithi071 ceph-mon[36597]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:15.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:15 smithi151 ceph-mon[36481]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:15.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:15 smithi045 ceph-mon[31773]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:17.358 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:43:17.358+0000 7fb4677fe700 1 -- 172.21.15.71:0/3803041670 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb4780625d0 con 0x7fb4600a2890 2024-03-28T17:43:17.359 INFO:teuthology.orchestra.run.smithi071.stdout:Created osd(s) 7 on host 'smithi071' 2024-03-28T17:43:17.362 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:43:17.361+0000 7fb47cf27700 1 -- 172.21.15.71:0/3803041670 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb4600a2890 msgr2=0x7fb4600a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:43:17.362 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:43:17.361+0000 7fb47cf27700 1 --2- 172.21.15.71:0/3803041670 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb4600a2890 0x7fb4600a4d50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb4680039c0 tx=0x7fb468000bc0 comp rx=0 tx=0).stop 2024-03-28T17:43:17.362 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:43:17.361+0000 7fb47cf27700 1 -- 172.21.15.71:0/3803041670 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4781010c0 msgr2=0x7fb478197460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:43:17.362 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:43:17.361+0000 7fb47cf27700 1 --2- 172.21.15.71:0/3803041670 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4781010c0 0x7fb478197460 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00b500 tx=0x7fb46c00b8c0 comp rx=0 tx=0).stop 2024-03-28T17:43:17.362 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:43:17.361+0000 7fb47cf27700 1 -- 172.21.15.71:0/3803041670 shutdown_connections 2024-03-28T17:43:17.362 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:43:17.361+0000 7fb47cf27700 1 --2- 172.21.15.71:0/3803041670 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb4600a2890 0x7fb4600a4d50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:43:17.362 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:43:17.361+0000 7fb47cf27700 1 --2- 172.21.15.71:0/3803041670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb478102c60 0x7fb47819ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:43:17.362 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:43:17.361+0000 7fb47cf27700 1 --2- 172.21.15.71:0/3803041670 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4781010c0 0x7fb478197460 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:43:17.362 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:43:17.361+0000 7fb47cf27700 1 --2- 172.21.15.71:0/3803041670 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb4781022c0 0x7fb4781979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:43:17.362 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:43:17.361+0000 7fb47cf27700 1 -- 172.21.15.71:0/3803041670 >> 172.21.15.71:0/3803041670 conn(0x7fb4780fc660 msgr2=0x7fb4781056b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:43:17.362 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:43:17.361+0000 7fb47cf27700 1 -- 172.21.15.71:0/3803041670 shutdown_connections 2024-03-28T17:43:17.363 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:43:17.361+0000 7fb47cf27700 1 -- 172.21.15.71:0/3803041670 wait complete. 2024-03-28T17:43:17.678 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:17 smithi071 ceph-mon[36597]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:17.678 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:17 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:17.678 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:17 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:17.678 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:17 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:17.679 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:17 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:43:17.679 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:17 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:17.679 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:17 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:17.679 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:17 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:17.679 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:17 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:43:17.679 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:17 smithi071 ceph-mon[36597]: from='osd.7 [v2:172.21.15.71:6816/2000211251,v1:172.21.15.71:6817/2000211251]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T17:43:17.679 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:17 smithi071 ceph-mon[36597]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T17:43:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:17 smithi151 ceph-mon[36481]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:17 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:17 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:17 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:17 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:43:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:17 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:17 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:17 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:17.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:17 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:43:17.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:17 smithi151 ceph-mon[36481]: from='osd.7 [v2:172.21.15.71:6816/2000211251,v1:172.21.15.71:6817/2000211251]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T17:43:17.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:17 smithi151 ceph-mon[36481]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T17:43:17.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:17 smithi045 ceph-mon[31773]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:17.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:17 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:17.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:17 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:17.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:17 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:17.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:17 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:43:17.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:17 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:17.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:17 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:17.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:17 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:17.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:17 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:43:17.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:17 smithi045 ceph-mon[31773]: from='osd.7 [v2:172.21.15.71:6816/2000211251,v1:172.21.15.71:6817/2000211251]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T17:43:17.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:17 smithi045 ceph-mon[31773]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T17:43:18.039 DEBUG:teuthology.orchestra.run.smithi071:osd.7> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@osd.7.service 2024-03-28T17:43:18.042 INFO:tasks.cephadm:Deploying osd.8 on smithi151 with /dev/vg_nvme/lv_2... 2024-03-28T17:43:18.042 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T17:43:18.290 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T17:43:19.541 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:19 smithi071 ceph-mon[36597]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:19.541 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:19 smithi071 ceph-mon[36597]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T17:43:19.541 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:19 smithi071 ceph-mon[36597]: from='osd.7 [v2:172.21.15.71:6816/2000211251,v1:172.21.15.71:6817/2000211251]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:43:19.541 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:19 smithi071 ceph-mon[36597]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T17:43:19.542 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:19 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T17:43:19.542 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:19 smithi071 ceph-mon[36597]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:43:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:19 smithi045 ceph-mon[31773]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:19 smithi045 ceph-mon[31773]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T17:43:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:19 smithi045 ceph-mon[31773]: from='osd.7 [v2:172.21.15.71:6816/2000211251,v1:172.21.15.71:6817/2000211251]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:43:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:19 smithi045 ceph-mon[31773]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T17:43:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:19 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T17:43:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:19 smithi045 ceph-mon[31773]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:43:19.708 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:19 smithi151 ceph-mon[36481]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:19.708 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:19 smithi151 ceph-mon[36481]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T17:43:19.708 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:19 smithi151 ceph-mon[36481]: from='osd.7 [v2:172.21.15.71:6816/2000211251,v1:172.21.15.71:6817/2000211251]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:43:19.708 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:19 smithi151 ceph-mon[36481]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T17:43:19.708 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:19 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T17:43:19.708 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:19 smithi151 ceph-mon[36481]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:43:19.796 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 28 17:43:19 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-osd-7[54892]: 2024-03-28T17:43:19.785+0000 7f8eacd10700 -1 osd.7 0 waiting for initial osdmap 2024-03-28T17:43:19.796 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 28 17:43:19 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-osd-7[54892]: 2024-03-28T17:43:19.795+0000 7f8ea4afb700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T17:43:20.418 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T17:43:20.418 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T17:43:20.418 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2024-03-28T17:43:20.418 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2024-03-28T17:43:20.418 INFO:teuthology.orchestra.run.smithi151.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0115636 s, 907 MB/s 2024-03-28T17:43:20.418 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 2024-03-28T17:43:20.418 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2024-03-28T17:43:20.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:20 smithi045 ceph-mon[31773]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-28T17:43:20.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:20 smithi045 ceph-mon[31773]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T17:43:20.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:20 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T17:43:20.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:20 smithi045 ceph-mon[31773]: from='osd.7 ' entity='osd.7' 2024-03-28T17:43:20.722 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:20 smithi151 ceph-mon[36481]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-28T17:43:20.722 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:20 smithi151 ceph-mon[36481]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T17:43:20.722 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:20 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T17:43:20.722 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:20 smithi151 ceph-mon[36481]: from='osd.7 ' entity='osd.7' 2024-03-28T17:43:20.763 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:20 smithi071 ceph-mon[36597]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-28T17:43:20.763 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:20 smithi071 ceph-mon[36597]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T17:43:20.763 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T17:43:20.763 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:20 smithi071 ceph-mon[36597]: from='osd.7 ' entity='osd.7' 2024-03-28T17:43:21.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi151:vg_nvme/lv_2 2024-03-28T17:43:21.356 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T17:43:21.637 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:21 smithi151 ceph-mon[36481]: purged_snaps scrub starts 2024-03-28T17:43:21.638 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:21 smithi151 ceph-mon[36481]: purged_snaps scrub ok 2024-03-28T17:43:21.638 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:21 smithi151 ceph-mon[36481]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:21.638 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:21 smithi151 ceph-mon[36481]: osd.7 [v2:172.21.15.71:6816/2000211251,v1:172.21.15.71:6817/2000211251] boot 2024-03-28T17:43:21.638 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:21 smithi151 ceph-mon[36481]: osdmap e51: 8 total, 8 up, 8 in 2024-03-28T17:43:21.638 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:21 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T17:43:21.638 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:21 smithi151 ceph-mon[36481]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T17:43:21.638 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:21 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:21.638 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:21 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:21.638 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:21 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:43:21.638 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:21 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:43:21.638 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:21 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:43:21.638 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:21 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:21.639 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:21 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:21.639 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:21 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:43:21.639 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:21 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:21.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:21 smithi045 ceph-mon[31773]: purged_snaps scrub starts 2024-03-28T17:43:21.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:21 smithi045 ceph-mon[31773]: purged_snaps scrub ok 2024-03-28T17:43:21.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:21 smithi045 ceph-mon[31773]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:21.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:21 smithi045 ceph-mon[31773]: osd.7 [v2:172.21.15.71:6816/2000211251,v1:172.21.15.71:6817/2000211251] boot 2024-03-28T17:43:21.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:21 smithi045 ceph-mon[31773]: osdmap e51: 8 total, 8 up, 8 in 2024-03-28T17:43:21.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:21 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T17:43:21.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:21 smithi045 ceph-mon[31773]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T17:43:21.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:21 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:21.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:21 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:21.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:21 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:43:21.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:21 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:43:21.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:21 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:43:21.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:21 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:21.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:21 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:21.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:21 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:43:21.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:21 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:21 smithi071 ceph-mon[36597]: purged_snaps scrub starts 2024-03-28T17:43:21.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:21 smithi071 ceph-mon[36597]: purged_snaps scrub ok 2024-03-28T17:43:21.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:21 smithi071 ceph-mon[36597]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-28T17:43:21.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:21 smithi071 ceph-mon[36597]: osd.7 [v2:172.21.15.71:6816/2000211251,v1:172.21.15.71:6817/2000211251] boot 2024-03-28T17:43:21.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:21 smithi071 ceph-mon[36597]: osdmap e51: 8 total, 8 up, 8 in 2024-03-28T17:43:21.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:21 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T17:43:21.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:21 smithi071 ceph-mon[36597]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T17:43:21.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:21 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:21.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:21 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:21.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:21 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:43:21.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:21 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:43:21.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:21 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:43:21.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:21 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:21.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:21 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:21.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:21 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:43:21.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:21 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:22.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:22 smithi045 ceph-mon[31773]: Detected new or changed devices on smithi071 2024-03-28T17:43:22.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:22 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi071 to 5693M 2024-03-28T17:43:22.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:22 smithi045 ceph-mon[31773]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T17:43:22.690 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:22 smithi151 ceph-mon[36481]: Detected new or changed devices on smithi071 2024-03-28T17:43:22.690 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:22 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi071 to 5693M 2024-03-28T17:43:22.690 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:22 smithi151 ceph-mon[36481]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T17:43:22.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:22 smithi071 ceph-mon[36597]: Detected new or changed devices on smithi071 2024-03-28T17:43:22.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:22 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi071 to 5693M 2024-03-28T17:43:22.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:22 smithi071 ceph-mon[36597]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T17:43:23.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.106+0000 7ff239ea7700 1 -- 172.21.15.151:0/3454099713 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff234102310 msgr2=0x7ff234102770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:43:23.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.106+0000 7ff239ea7700 1 --2- 172.21.15.151:0/3454099713 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff234102310 0x7ff234102770 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff228009990 tx=0x7ff228009ca0 comp rx=0 tx=0).stop 2024-03-28T17:43:23.109 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.106+0000 7ff239ea7700 1 -- 172.21.15.151:0/3454099713 shutdown_connections 2024-03-28T17:43:23.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.106+0000 7ff239ea7700 1 --2- 172.21.15.151:0/3454099713 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff234102cb0 0x7ff234109560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:43:23.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.106+0000 7ff239ea7700 1 --2- 172.21.15.151:0/3454099713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff234101110 0x7ff234101530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:43:23.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.106+0000 7ff239ea7700 1 --2- 172.21.15.151:0/3454099713 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff234102310 0x7ff234102770 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:43:23.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.106+0000 7ff239ea7700 1 -- 172.21.15.151:0/3454099713 >> 172.21.15.151:0/3454099713 conn(0x7ff2340fc6b0 msgr2=0x7ff2340feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:43:23.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.106+0000 7ff239ea7700 1 -- 172.21.15.151:0/3454099713 shutdown_connections 2024-03-28T17:43:23.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.106+0000 7ff239ea7700 1 -- 172.21.15.151:0/3454099713 wait complete. 2024-03-28T17:43:23.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.107+0000 7ff239ea7700 1 Processor -- start 2024-03-28T17:43:23.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.107+0000 7ff239ea7700 1 -- start start 2024-03-28T17:43:23.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.107+0000 7ff239ea7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff234101110 0x7ff23406b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:43:23.110 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.107+0000 7ff239ea7700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff234102310 0x7ff23419f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:43:23.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.107+0000 7ff239ea7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff234102cb0 0x7ff2341a1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:43:23.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.107+0000 7ff239ea7700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23410cdf0 con 0x7ff234102cb0 2024-03-28T17:43:23.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.107+0000 7ff239ea7700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23410cc70 con 0x7ff234102310 2024-03-28T17:43:23.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.107+0000 7ff239ea7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23410cf70 con 0x7ff234101110 2024-03-28T17:43:23.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.107+0000 7ff2337fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff234101110 0x7ff23406b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:43:23.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.107+0000 7ff2337fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff234101110 0x7ff23406b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44668/0 (socket says 172.21.15.151:44668) 2024-03-28T17:43:23.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.107+0000 7ff2337fe700 1 -- 172.21.15.151:0/1052898834 learned_addr learned my addr 172.21.15.151:0/1052898834 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-03-28T17:43:23.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.108+0000 7ff232ffd700 1 --2- 172.21.15.151:0/1052898834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff234102310 0x7ff23419f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:43:23.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.108+0000 7ff2337fe700 1 -- 172.21.15.151:0/1052898834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff234102310 msgr2=0x7ff23419f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:43:23.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.108+0000 7ff2337fe700 1 --2- 172.21.15.151:0/1052898834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff234102310 0x7ff23419f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:43:23.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.108+0000 7ff2337fe700 1 -- 172.21.15.151:0/1052898834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff234102cb0 msgr2=0x7ff2341a1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:43:23.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.108+0000 7ff2337fe700 1 --2- 172.21.15.151:0/1052898834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff234102cb0 0x7ff2341a1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:43:23.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.108+0000 7ff2337fe700 1 -- 172.21.15.151:0/1052898834 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2341a23d0 con 0x7ff234101110 2024-03-28T17:43:23.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.108+0000 7ff2337fe700 1 --2- 172.21.15.151:0/1052898834 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff234101110 0x7ff23406b3d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff22400b500 tx=0x7ff22400b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:43:23.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.108+0000 7ff230ff9700 1 -- 172.21.15.151:0/1052898834 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff224015020 con 0x7ff234101110 2024-03-28T17:43:23.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.108+0000 7ff230ff9700 1 -- 172.21.15.151:0/1052898834 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff224011d10 con 0x7ff234101110 2024-03-28T17:43:23.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.108+0000 7ff230ff9700 1 -- 172.21.15.151:0/1052898834 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22400f2e0 con 0x7ff234101110 2024-03-28T17:43:23.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.108+0000 7ff239ea7700 1 -- 172.21.15.151:0/1052898834 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2341a26c0 con 0x7ff234101110 2024-03-28T17:43:23.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.108+0000 7ff239ea7700 1 -- 172.21.15.151:0/1052898834 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2341a2c80 con 0x7ff234101110 2024-03-28T17:43:23.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.109+0000 7ff239ea7700 1 -- 172.21.15.151:0/1052898834 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff234068e50 con 0x7ff234101110 2024-03-28T17:43:23.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.110+0000 7ff230ff9700 1 -- 172.21.15.151:0/1052898834 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff224018020 con 0x7ff234101110 2024-03-28T17:43:23.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.111+0000 7ff230ff9700 1 --2- 172.21.15.151:0/1052898834 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff21c0a2890 0x7ff21c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:43:23.113 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.111+0000 7ff230ff9700 1 -- 172.21.15.151:0/1052898834 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5426+0+0 (secure 0 0 0) 0x7ff2240cdbd0 con 0x7ff234101110 2024-03-28T17:43:23.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.111+0000 7ff232ffd700 1 --2- 172.21.15.151:0/1052898834 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff21c0a2890 0x7ff21c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:43:23.114 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.112+0000 7ff232ffd700 1 --2- 172.21.15.151:0/1052898834 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff21c0a2890 0x7ff21c0a4d50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff2280039c0 tx=0x7ff228005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:43:23.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.114+0000 7ff230ff9700 1 -- 172.21.15.151:0/1052898834 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff22401f020 con 0x7ff234101110 2024-03-28T17:43:23.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:23.318+0000 7ff239ea7700 1 -- 172.21.15.151:0/1052898834 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2340625d0 con 0x7ff21c0a2890 2024-03-28T17:43:23.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:23 smithi151 ceph-mon[36481]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:23.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:23 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:43:23.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:23 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:43:23.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:23 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:23.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:23 smithi045 ceph-mon[31773]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:23.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:23 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:43:23.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:23 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:43:23.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:23 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:23.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:23 smithi071 ceph-mon[36597]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:23.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:23 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:43:23.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:23 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:43:23.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:23 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:24.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:24 smithi151 ceph-mon[36481]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:43:24.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:24 smithi151 ceph-mon[36481]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T17:43:24.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:24 smithi045 ceph-mon[31773]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:43:24.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:24 smithi045 ceph-mon[31773]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T17:43:24.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:24 smithi071 ceph-mon[36597]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:43:24.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:24 smithi071 ceph-mon[36597]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T17:43:25.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:25 smithi045 ceph-mon[31773]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:25.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:25 smithi045 ceph-mon[31773]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T17:43:25.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:25 smithi071 ceph-mon[36597]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:25.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:25 smithi071 ceph-mon[36597]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T17:43:25.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:25 smithi151 ceph-mon[36481]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:25.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:25 smithi151 ceph-mon[36481]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T17:43:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:26 smithi045 ceph-mon[31773]: from='client.? 172.21.15.151:0/1666543789' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88abcb0f-f189-49be-8315-c700c77369f3"}]: dispatch 2024-03-28T17:43:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:26 smithi045 ceph-mon[31773]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88abcb0f-f189-49be-8315-c700c77369f3"}]: dispatch 2024-03-28T17:43:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:26 smithi045 ceph-mon[31773]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88abcb0f-f189-49be-8315-c700c77369f3"}]': finished 2024-03-28T17:43:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:26 smithi045 ceph-mon[31773]: osdmap e56: 9 total, 8 up, 9 in 2024-03-28T17:43:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:26 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T17:43:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:26 smithi071 ceph-mon[36597]: from='client.? 172.21.15.151:0/1666543789' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88abcb0f-f189-49be-8315-c700c77369f3"}]: dispatch 2024-03-28T17:43:26.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:26 smithi071 ceph-mon[36597]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88abcb0f-f189-49be-8315-c700c77369f3"}]: dispatch 2024-03-28T17:43:26.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:26 smithi071 ceph-mon[36597]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88abcb0f-f189-49be-8315-c700c77369f3"}]': finished 2024-03-28T17:43:26.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:26 smithi071 ceph-mon[36597]: osdmap e56: 9 total, 8 up, 9 in 2024-03-28T17:43:26.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:26 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T17:43:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:26 smithi151 ceph-mon[36481]: from='client.? 172.21.15.151:0/1666543789' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88abcb0f-f189-49be-8315-c700c77369f3"}]: dispatch 2024-03-28T17:43:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:26 smithi151 ceph-mon[36481]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88abcb0f-f189-49be-8315-c700c77369f3"}]: dispatch 2024-03-28T17:43:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:26 smithi151 ceph-mon[36481]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88abcb0f-f189-49be-8315-c700c77369f3"}]': finished 2024-03-28T17:43:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:26 smithi151 ceph-mon[36481]: osdmap e56: 9 total, 8 up, 9 in 2024-03-28T17:43:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:26 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T17:43:27.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:27 smithi045 ceph-mon[31773]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:27 smithi045 ceph-mon[31773]: from='client.? 172.21.15.151:0/2666501359' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:43:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:27 smithi071 ceph-mon[36597]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:27 smithi071 ceph-mon[36597]: from='client.? 172.21.15.151:0/2666501359' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:43:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:27 smithi151 ceph-mon[36481]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:27 smithi151 ceph-mon[36481]: from='client.? 172.21.15.151:0/2666501359' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:43:29.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:29 smithi045 ceph-mon[31773]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:29.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:29 smithi071 ceph-mon[36597]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:29.813 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:29 smithi151 ceph-mon[36481]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:31.411 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:31 smithi151 ceph-mon[36481]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:31.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:31 smithi045 ceph-mon[31773]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:31 smithi071 ceph-mon[36597]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:33 smithi151 ceph-mon[36481]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:33.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:33 smithi045 ceph-mon[31773]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:33.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:33 smithi071 ceph-mon[36597]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:34.565 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:34 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T17:43:34.565 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:34 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:34.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:34 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T17:43:34.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:34 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:34.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:34 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T17:43:34.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:34 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:35.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:35 smithi151 ceph-mon[36481]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:35.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:35 smithi151 ceph-mon[36481]: Deploying daemon osd.8 on smithi151 2024-03-28T17:43:35.795 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:35 smithi071 ceph-mon[36597]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:35.795 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:35 smithi071 ceph-mon[36597]: Deploying daemon osd.8 on smithi151 2024-03-28T17:43:35.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:35 smithi045 ceph-mon[31773]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:35.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:35 smithi045 ceph-mon[31773]: Deploying daemon osd.8 on smithi151 2024-03-28T17:43:37.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:37 smithi151 ceph-mon[36481]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:37 smithi071 ceph-mon[36597]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:37.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:37 smithi045 ceph-mon[31773]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:39.700 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:39 smithi151 ceph-mon[36481]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:39.700 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:39 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:39.700 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:39 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:39.700 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:39 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:43:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:39 smithi071 ceph-mon[36597]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:39 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:39 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:39 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:43:39.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:39 smithi045 ceph-mon[31773]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:39.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:39 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:39.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:39 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:39.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:39 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:43:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:41 smithi151 ceph-mon[36481]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:41 smithi071 ceph-mon[36597]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:41 smithi045 ceph-mon[31773]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:42.656 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 8 on host 'smithi151' 2024-03-28T17:43:42.657 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:42.656+0000 7ff230ff9700 1 -- 172.21.15.151:0/1052898834 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff2340625d0 con 0x7ff21c0a2890 2024-03-28T17:43:42.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:42.658+0000 7ff239ea7700 1 -- 172.21.15.151:0/1052898834 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff21c0a2890 msgr2=0x7ff21c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:43:42.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:42.658+0000 7ff239ea7700 1 --2- 172.21.15.151:0/1052898834 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff21c0a2890 0x7ff21c0a4d50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff2280039c0 tx=0x7ff228005fb0 comp rx=0 tx=0).stop 2024-03-28T17:43:42.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:42.658+0000 7ff239ea7700 1 -- 172.21.15.151:0/1052898834 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff234101110 msgr2=0x7ff23406b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:43:42.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:42.658+0000 7ff239ea7700 1 --2- 172.21.15.151:0/1052898834 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff234101110 0x7ff23406b3d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff22400b500 tx=0x7ff22400b8c0 comp rx=0 tx=0).stop 2024-03-28T17:43:42.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:42.659+0000 7ff239ea7700 1 -- 172.21.15.151:0/1052898834 shutdown_connections 2024-03-28T17:43:42.659 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:42.659+0000 7ff239ea7700 1 --2- 172.21.15.151:0/1052898834 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff21c0a2890 0x7ff21c0a4d50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:43:42.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:42.659+0000 7ff239ea7700 1 --2- 172.21.15.151:0/1052898834 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff234101110 0x7ff23406b3d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:43:42.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:42.659+0000 7ff239ea7700 1 --2- 172.21.15.151:0/1052898834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff234102cb0 0x7ff2341a1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:43:42.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:42.659+0000 7ff239ea7700 1 --2- 172.21.15.151:0/1052898834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff234102310 0x7ff23419f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:43:42.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:42.659+0000 7ff239ea7700 1 -- 172.21.15.151:0/1052898834 >> 172.21.15.151:0/1052898834 conn(0x7ff2340fc6b0 msgr2=0x7ff234105690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:43:42.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:42.659+0000 7ff239ea7700 1 -- 172.21.15.151:0/1052898834 shutdown_connections 2024-03-28T17:43:42.660 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:43:42.659+0000 7ff239ea7700 1 -- 172.21.15.151:0/1052898834 wait complete. 2024-03-28T17:43:43.538 DEBUG:teuthology.orchestra.run.smithi151:osd.8> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@osd.8.service 2024-03-28T17:43:43.540 INFO:tasks.cephadm:Deploying osd.9 on smithi045 with /dev/vg_nvme/lv_1... 2024-03-28T17:43:43.540 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T17:43:43.718 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:43 smithi045 ceph-mon[31773]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:43.718 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:43 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:43.719 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:43 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:43.719 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:43 smithi045 ceph-mon[31773]: from='osd.8 [v2:172.21.15.151:6816/3665460137,v1:172.21.15.151:6817/3665460137]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T17:43:43.719 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:43 smithi045 ceph-mon[31773]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T17:43:43.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:43 smithi071 ceph-mon[36597]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:43.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:43 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:43.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:43 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:43.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:43 smithi071 ceph-mon[36597]: from='osd.8 [v2:172.21.15.151:6816/3665460137,v1:172.21.15.151:6817/3665460137]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T17:43:43.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:43 smithi071 ceph-mon[36597]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T17:43:43.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:43 smithi151 ceph-mon[36481]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:43.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:43 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:43.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:43 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:43.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:43 smithi151 ceph-mon[36481]: from='osd.8 [v2:172.21.15.151:6816/3665460137,v1:172.21.15.151:6817/3665460137]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T17:43:43.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:43 smithi151 ceph-mon[36481]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T17:43:44.006 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:43:44.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:44 smithi045 ceph-mon[31773]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:44.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:44 smithi045 ceph-mon[31773]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T17:43:44.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:44 smithi045 ceph-mon[31773]: from='osd.8 [v2:172.21.15.151:6816/3665460137,v1:172.21.15.151:6817/3665460137]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:43:44.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:44 smithi045 ceph-mon[31773]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T17:43:44.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T17:43:44.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:44 smithi045 ceph-mon[31773]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:43:44.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:44.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:45.019 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:44 smithi151 ceph-mon[36481]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:45.019 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:44 smithi151 ceph-mon[36481]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T17:43:45.019 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:44 smithi151 ceph-mon[36481]: from='osd.8 [v2:172.21.15.151:6816/3665460137,v1:172.21.15.151:6817/3665460137]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:43:45.019 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:44 smithi151 ceph-mon[36481]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T17:43:45.019 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T17:43:45.019 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:44 smithi151 ceph-mon[36481]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:43:45.020 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:45.020 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:45.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:44 smithi071 ceph-mon[36597]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:45.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:44 smithi071 ceph-mon[36597]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T17:43:45.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:44 smithi071 ceph-mon[36597]: from='osd.8 [v2:172.21.15.151:6816/3665460137,v1:172.21.15.151:6817/3665460137]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:43:45.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:44 smithi071 ceph-mon[36597]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T17:43:45.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T17:43:45.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:44 smithi071 ceph-mon[36597]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:43:45.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:45.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:45.323 INFO:journalctl@ceph.osd.8.smithi151.stdout:Mar 28 17:43:45 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-osd-8[54799]: 2024-03-28T17:43:45.019+0000 7f5ff565c700 -1 osd.8 0 waiting for initial osdmap 2024-03-28T17:43:45.324 INFO:journalctl@ceph.osd.8.smithi151.stdout:Mar 28 17:43:45 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-osd-8[54799]: 2024-03-28T17:43:45.031+0000 7f5fed447700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T17:43:45.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:45 smithi045 ceph-mon[31773]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-28T17:43:45.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:45 smithi045 ceph-mon[31773]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T17:43:45.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:45 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T17:43:45.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:45 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T17:43:45.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:45 smithi045 ceph-mon[31773]: from='osd.8 ' entity='osd.8' 2024-03-28T17:43:46.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:45 smithi071 ceph-mon[36597]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-28T17:43:46.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:45 smithi071 ceph-mon[36597]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T17:43:46.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:45 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T17:43:46.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:45 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T17:43:46.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:45 smithi071 ceph-mon[36597]: from='osd.8 ' entity='osd.8' 2024-03-28T17:43:46.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:45 smithi151 ceph-mon[36481]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-28T17:43:46.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:45 smithi151 ceph-mon[36481]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T17:43:46.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:45 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T17:43:46.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:45 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T17:43:46.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:45 smithi151 ceph-mon[36481]: from='osd.8 ' entity='osd.8' 2024-03-28T17:43:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:46 smithi151 ceph-mon[36481]: purged_snaps scrub starts 2024-03-28T17:43:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:46 smithi151 ceph-mon[36481]: purged_snaps scrub ok 2024-03-28T17:43:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:46 smithi151 ceph-mon[36481]: pgmap v217: 1 pgs: 1 peering; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:46.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:46 smithi151 ceph-mon[36481]: osd.8 [v2:172.21.15.151:6816/3665460137,v1:172.21.15.151:6817/3665460137] boot 2024-03-28T17:43:46.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:46 smithi151 ceph-mon[36481]: osdmap e59: 9 total, 9 up, 9 in 2024-03-28T17:43:46.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:46 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T17:43:46.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:46 smithi151 ceph-mon[36481]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T17:43:46.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:46 smithi045 ceph-mon[31773]: purged_snaps scrub starts 2024-03-28T17:43:46.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:46 smithi045 ceph-mon[31773]: purged_snaps scrub ok 2024-03-28T17:43:46.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:46 smithi045 ceph-mon[31773]: pgmap v217: 1 pgs: 1 peering; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:46.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:46 smithi045 ceph-mon[31773]: osd.8 [v2:172.21.15.151:6816/3665460137,v1:172.21.15.151:6817/3665460137] boot 2024-03-28T17:43:46.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:46 smithi045 ceph-mon[31773]: osdmap e59: 9 total, 9 up, 9 in 2024-03-28T17:43:46.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:46 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T17:43:46.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:46 smithi045 ceph-mon[31773]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T17:43:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:46 smithi071 ceph-mon[36597]: purged_snaps scrub starts 2024-03-28T17:43:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:46 smithi071 ceph-mon[36597]: purged_snaps scrub ok 2024-03-28T17:43:47.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:46 smithi071 ceph-mon[36597]: pgmap v217: 1 pgs: 1 peering; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T17:43:47.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:46 smithi071 ceph-mon[36597]: osd.8 [v2:172.21.15.151:6816/3665460137,v1:172.21.15.151:6817/3665460137] boot 2024-03-28T17:43:47.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:46 smithi071 ceph-mon[36597]: osdmap e59: 9 total, 9 up, 9 in 2024-03-28T17:43:47.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:46 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T17:43:47.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:46 smithi071 ceph-mon[36597]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T17:43:47.061 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T17:43:47.061 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T17:43:47.062 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2024-03-28T17:43:47.062 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2024-03-28T17:43:47.062 INFO:teuthology.orchestra.run.smithi045.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0348212 s, 301 MB/s 2024-03-28T17:43:47.062 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 2024-03-28T17:43:47.062 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2024-03-28T17:43:47.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:47 smithi045 ceph-mon[31773]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T17:43:47.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:47 smithi045 ceph-mon[31773]: Detected new or changed devices on smithi151 2024-03-28T17:43:47.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:47 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:47.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:47 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:47.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:47 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:43:47.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:47 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:43:47.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:47 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:43:47.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:47 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi151 to 5693M 2024-03-28T17:43:47.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:47 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:47.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:47 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:47.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:47 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:43:47.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:47 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:47.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:47 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:43:48.075 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi045:vg_nvme/lv_1 2024-03-28T17:43:48.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:47 smithi071 ceph-mon[36597]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T17:43:48.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:47 smithi071 ceph-mon[36597]: Detected new or changed devices on smithi151 2024-03-28T17:43:48.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:47 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:47 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:47 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:43:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:47 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:43:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:47 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:43:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:47 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi151 to 5693M 2024-03-28T17:43:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:47 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:47 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:48.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:47 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:43:48.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:47 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:48.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:47 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:43:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:47 smithi151 ceph-mon[36481]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T17:43:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:47 smithi151 ceph-mon[36481]: Detected new or changed devices on smithi151 2024-03-28T17:43:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:47 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:47 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:48.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:47 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:43:48.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:47 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:43:48.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:47 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:43:48.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:47 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi151 to 5693M 2024-03-28T17:43:48.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:47 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:48.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:47 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:48.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:47 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:43:48.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:47 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:48.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:47 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:43:48.540 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:43:49.016 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:48 smithi045 ceph-mon[31773]: pgmap v221: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:43:49.016 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:48 smithi045 ceph-mon[31773]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T17:43:49.016 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:48 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:49.016 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:48 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:43:49.016 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:48 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:49.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:48 smithi071 ceph-mon[36597]: pgmap v221: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:43:49.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:48 smithi071 ceph-mon[36597]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T17:43:49.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:48 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:49.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:48 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:43:49.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:48 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:49.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:48 smithi151 ceph-mon[36481]: pgmap v221: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:43:49.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:48 smithi151 ceph-mon[36481]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T17:43:49.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:48 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:49.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:48 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:43:49.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:48 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:43:50.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:49 smithi045 ceph-mon[31773]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T17:43:50.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:49 smithi071 ceph-mon[36597]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T17:43:50.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:49 smithi151 ceph-mon[36481]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T17:43:50.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.821+0000 7f6833200700 1 -- 172.21.15.45:0/124932135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f682c1055d0 msgr2=0x7f682c105a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:43:50.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.821+0000 7f6833200700 1 --2- 172.21.15.45:0/124932135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f682c1055d0 0x7f682c105a30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f68180099c0 tx=0x7f6818009cd0 comp rx=0 tx=0).stop 2024-03-28T17:43:50.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.821+0000 7f6833200700 1 -- 172.21.15.45:0/124932135 shutdown_connections 2024-03-28T17:43:50.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.821+0000 7f6833200700 1 --2- 172.21.15.45:0/124932135 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f682c105f70 0x7f682c10c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:43:50.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.821+0000 7f6833200700 1 --2- 172.21.15.45:0/124932135 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f682c1043d0 0x7f682c1047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:43:50.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.821+0000 7f6833200700 1 --2- 172.21.15.45:0/124932135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f682c1055d0 0x7f682c105a30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:43:50.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.821+0000 7f6833200700 1 -- 172.21.15.45:0/124932135 >> 172.21.15.45:0/124932135 conn(0x7f682c0ff950 msgr2=0x7f682c101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:43:50.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.822+0000 7f6833200700 1 -- 172.21.15.45:0/124932135 shutdown_connections 2024-03-28T17:43:50.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.822+0000 7f6833200700 1 -- 172.21.15.45:0/124932135 wait complete. 2024-03-28T17:43:50.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.823+0000 7f6833200700 1 Processor -- start 2024-03-28T17:43:50.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.823+0000 7f6833200700 1 -- start start 2024-03-28T17:43:50.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.823+0000 7f6833200700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f682c1043d0 0x7f682c19a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:43:50.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.823+0000 7f6833200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f682c1055d0 0x7f682c19aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:43:50.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.823+0000 7f6833200700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f682c105f70 0x7f682c1a1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:43:50.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.823+0000 7f6833200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682c10ffb0 con 0x7f682c1055d0 2024-03-28T17:43:50.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.823+0000 7f6833200700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682c10fe30 con 0x7f682c105f70 2024-03-28T17:43:50.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.823+0000 7f6833200700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682c110130 con 0x7f682c1043d0 2024-03-28T17:43:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.823+0000 7f682b400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f682c1055d0 0x7f682c19aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:43:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.824+0000 7f682be00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f682c1043d0 0x7f682c19a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:43:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.824+0000 7f682b400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f682c1055d0 0x7f682c19aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40584/0 (socket says 172.21.15.45:40584) 2024-03-28T17:43:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.824+0000 7f682b400700 1 -- 172.21.15.45:0/2880122998 learned_addr learned my addr 172.21.15.45:0/2880122998 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:43:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.824+0000 7f6831200700 1 --2- 172.21.15.45:0/2880122998 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f682c105f70 0x7f682c1a1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:43:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.824+0000 7f682be00700 1 -- 172.21.15.45:0/2880122998 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f682c105f70 msgr2=0x7f682c1a1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:43:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.824+0000 7f682be00700 1 --2- 172.21.15.45:0/2880122998 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f682c105f70 0x7f682c1a1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:43:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.824+0000 7f682be00700 1 -- 172.21.15.45:0/2880122998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f682c1055d0 msgr2=0x7f682c19aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:43:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.824+0000 7f682be00700 1 --2- 172.21.15.45:0/2880122998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f682c1055d0 0x7f682c19aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:43:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.824+0000 7f682be00700 1 -- 172.21.15.45:0/2880122998 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f682c1a21f0 con 0x7f682c1043d0 2024-03-28T17:43:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.824+0000 7f682b400700 1 --2- 172.21.15.45:0/2880122998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f682c1055d0 0x7f682c19aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:43:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.824+0000 7f682be00700 1 --2- 172.21.15.45:0/2880122998 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f682c1043d0 0x7f682c19a590 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f681c00e9f0 tx=0x7f681c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:43:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.825+0000 7f6828c00700 1 -- 172.21.15.45:0/2880122998 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c004220 con 0x7f682c1043d0 2024-03-28T17:43:50.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.825+0000 7f6833200700 1 -- 172.21.15.45:0/2880122998 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f682c1a24e0 con 0x7f682c1043d0 2024-03-28T17:43:50.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.825+0000 7f6828c00700 1 -- 172.21.15.45:0/2880122998 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f681c0043c0 con 0x7f682c1043d0 2024-03-28T17:43:50.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.825+0000 7f6828c00700 1 -- 172.21.15.45:0/2880122998 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c003cb0 con 0x7f682c1043d0 2024-03-28T17:43:50.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.825+0000 7f6833200700 1 -- 172.21.15.45:0/2880122998 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f682c1a2a70 con 0x7f682c1043d0 2024-03-28T17:43:50.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.826+0000 7f6833200700 1 -- 172.21.15.45:0/2880122998 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f682c068e50 con 0x7f682c1043d0 2024-03-28T17:43:50.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.828+0000 7f6828c00700 1 -- 172.21.15.45:0/2880122998 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f681c00c7c0 con 0x7f682c1043d0 2024-03-28T17:43:50.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.829+0000 7f6828c00700 1 --2- 172.21.15.45:0/2880122998 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f68100a2910 0x7f68100a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:43:50.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.829+0000 7f6828c00700 1 -- 172.21.15.45:0/2880122998 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5829+0+0 (secure 0 0 0) 0x7f681c0ce2e0 con 0x7f682c1043d0 2024-03-28T17:43:50.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.830+0000 7f682b400700 1 --2- 172.21.15.45:0/2880122998 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f68100a2910 0x7f68100a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:43:50.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.830+0000 7f682b400700 1 --2- 172.21.15.45:0/2880122998 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f68100a2910 0x7f68100a4dd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f68180038c0 tx=0x7f6818005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:43:50.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:50.833+0000 7f6828c00700 1 -- 172.21.15.45:0/2880122998 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f681c09bc90 con 0x7f682c1043d0 2024-03-28T17:43:50.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:50 smithi045 ceph-mon[31773]: pgmap v224: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:43:51.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:43:51.076+0000 7f6833200700 1 -- 172.21.15.45:0/2880122998 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f682c0625d0 con 0x7f68100a2910 2024-03-28T17:43:51.224 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:50 smithi151 ceph-mon[36481]: pgmap v224: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:43:51.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:50 smithi071 ceph-mon[36597]: pgmap v224: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:43:52.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:51 smithi045 ceph-mon[31773]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:43:52.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:51 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:43:52.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:51 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:43:52.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:51 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:51 smithi071 ceph-mon[36597]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:43:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:51 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:43:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:51 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:43:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:51 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:51 smithi151 ceph-mon[36481]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:43:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:51 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:43:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:51 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:43:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:51 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:43:53.127 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:52 smithi045 ceph-mon[31773]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-28T17:43:53.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:52 smithi071 ceph-mon[36597]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-28T17:43:53.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:52 smithi151 ceph-mon[36481]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-28T17:43:55.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:54 smithi045 ceph-mon[31773]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-28T17:43:55.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:54 smithi071 ceph-mon[36597]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-28T17:43:55.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:54 smithi151 ceph-mon[36481]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-28T17:43:56.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:55 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3975888581' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed3fc096-961e-4392-abc2-c5aa4d050117"}]: dispatch 2024-03-28T17:43:56.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:55 smithi045 ceph-mon[31773]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed3fc096-961e-4392-abc2-c5aa4d050117"}]: dispatch 2024-03-28T17:43:56.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:55 smithi045 ceph-mon[31773]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed3fc096-961e-4392-abc2-c5aa4d050117"}]': finished 2024-03-28T17:43:56.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:55 smithi045 ceph-mon[31773]: osdmap e64: 10 total, 9 up, 10 in 2024-03-28T17:43:56.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:55 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T17:43:56.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:55 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3975888581' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed3fc096-961e-4392-abc2-c5aa4d050117"}]: dispatch 2024-03-28T17:43:56.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:55 smithi071 ceph-mon[36597]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed3fc096-961e-4392-abc2-c5aa4d050117"}]: dispatch 2024-03-28T17:43:56.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:55 smithi071 ceph-mon[36597]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed3fc096-961e-4392-abc2-c5aa4d050117"}]': finished 2024-03-28T17:43:56.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:55 smithi071 ceph-mon[36597]: osdmap e64: 10 total, 9 up, 10 in 2024-03-28T17:43:56.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:55 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T17:43:56.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:55 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3975888581' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed3fc096-961e-4392-abc2-c5aa4d050117"}]: dispatch 2024-03-28T17:43:56.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:55 smithi151 ceph-mon[36481]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed3fc096-961e-4392-abc2-c5aa4d050117"}]: dispatch 2024-03-28T17:43:56.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:55 smithi151 ceph-mon[36481]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed3fc096-961e-4392-abc2-c5aa4d050117"}]': finished 2024-03-28T17:43:56.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:55 smithi151 ceph-mon[36481]: osdmap e64: 10 total, 9 up, 10 in 2024-03-28T17:43:56.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:55 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T17:43:57.154 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:56 smithi045 ceph-mon[31773]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-28T17:43:57.155 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:56 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1754206922' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:43:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:56 smithi071 ceph-mon[36597]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-28T17:43:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:56 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1754206922' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:43:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:56 smithi151 ceph-mon[36481]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-28T17:43:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:56 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1754206922' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:43:59.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:43:58 smithi045 ceph-mon[31773]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-28T17:43:59.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:43:58 smithi071 ceph-mon[36597]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-28T17:43:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:43:58 smithi151 ceph-mon[36481]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-28T17:44:01.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:00 smithi045 ceph-mon[31773]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T17:44:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:00 smithi071 ceph-mon[36597]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T17:44:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:00 smithi151 ceph-mon[36481]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T17:44:03.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:02 smithi045 ceph-mon[31773]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:02 smithi071 ceph-mon[36597]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:03.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:02 smithi151 ceph-mon[36481]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:05.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:04 smithi071 ceph-mon[36597]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:05.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:04 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T17:44:05.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:04 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:05.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:04 smithi071 ceph-mon[36597]: Deploying daemon osd.9 on smithi045 2024-03-28T17:44:05.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:04 smithi151 ceph-mon[36481]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:05.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T17:44:05.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:04 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:05.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:04 smithi151 ceph-mon[36481]: Deploying daemon osd.9 on smithi045 2024-03-28T17:44:05.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:04 smithi045 ceph-mon[31773]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:05.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:04 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T17:44:05.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:04 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:05.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:04 smithi045 ceph-mon[31773]: Deploying daemon osd.9 on smithi045 2024-03-28T17:44:07.256 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:06 smithi045 ceph-mon[31773]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:06 smithi071 ceph-mon[36597]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:06 smithi151 ceph-mon[36481]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:09.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:08 smithi071 ceph-mon[36597]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:09.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:08 smithi151 ceph-mon[36481]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:09.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:08 smithi045 ceph-mon[31773]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:11.207 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:10 smithi045 ceph-mon[31773]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:11.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:10 smithi071 ceph-mon[36597]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:11.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:10 smithi151 ceph-mon[36481]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:12 smithi045 ceph-mon[31773]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:13.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:12 smithi071 ceph-mon[36597]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:13.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:12 smithi151 ceph-mon[36481]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:14.187 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:13 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:14.187 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:13 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:14.187 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:13 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:44:14.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:14.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:14.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:44:14.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:13 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:14.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:13 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:14.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:13 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:44:15.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:15 smithi045 ceph-mon[31773]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:15.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:15 smithi071 ceph-mon[36597]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:15.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:15 smithi151 ceph-mon[36481]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:17.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:17 smithi071 ceph-mon[36597]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:17.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:17 smithi071 ceph-mon[36597]: from='osd.9 [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T17:44:17.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:17 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:17.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:17 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:17 smithi151 ceph-mon[36481]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:17 smithi151 ceph-mon[36481]: from='osd.9 [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T17:44:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:17 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:17 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:17 smithi045 ceph-mon[31773]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:17 smithi045 ceph-mon[31773]: from='osd.9 [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T17:44:17.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:17 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:17.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:17 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:18.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:44:18.080+0000 7f6828c00700 1 -- 172.21.15.45:0/2880122998 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f682c0625d0 con 0x7f68100a2910 2024-03-28T17:44:18.081 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 9 on host 'smithi045' 2024-03-28T17:44:18.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:44:18.085+0000 7f6833200700 1 -- 172.21.15.45:0/2880122998 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f68100a2910 msgr2=0x7f68100a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:44:18.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:44:18.085+0000 7f6833200700 1 --2- 172.21.15.45:0/2880122998 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f68100a2910 0x7f68100a4dd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f68180038c0 tx=0x7f6818005fb0 comp rx=0 tx=0).stop 2024-03-28T17:44:18.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:44:18.085+0000 7f6833200700 1 -- 172.21.15.45:0/2880122998 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f682c1043d0 msgr2=0x7f682c19a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:44:18.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:44:18.085+0000 7f6833200700 1 --2- 172.21.15.45:0/2880122998 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f682c1043d0 0x7f682c19a590 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f681c00e9f0 tx=0x7f681c00ed00 comp rx=0 tx=0).stop 2024-03-28T17:44:18.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:44:18.085+0000 7f6833200700 1 -- 172.21.15.45:0/2880122998 shutdown_connections 2024-03-28T17:44:18.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:44:18.085+0000 7f6833200700 1 --2- 172.21.15.45:0/2880122998 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f68100a2910 0x7f68100a4dd0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:44:18.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:44:18.085+0000 7f6833200700 1 --2- 172.21.15.45:0/2880122998 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f682c1043d0 0x7f682c19a590 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f681c00e9f0 tx=0x7f681c00ed00 comp rx=0 tx=0).stop 2024-03-28T17:44:18.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:44:18.085+0000 7f6833200700 1 --2- 172.21.15.45:0/2880122998 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f682c105f70 0x7f682c1a1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:44:18.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:44:18.085+0000 7f6833200700 1 --2- 172.21.15.45:0/2880122998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f682c1055d0 0x7f682c19aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:44:18.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:44:18.085+0000 7f6833200700 1 -- 172.21.15.45:0/2880122998 >> 172.21.15.45:0/2880122998 conn(0x7f682c0ff950 msgr2=0x7f682c1088d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:44:18.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:44:18.086+0000 7f6833200700 1 -- 172.21.15.45:0/2880122998 shutdown_connections 2024-03-28T17:44:18.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:44:18.086+0000 7f6833200700 1 -- 172.21.15.45:0/2880122998 wait complete. 2024-03-28T17:44:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:18 smithi045 ceph-mon[31773]: from='osd.9 [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T17:44:18.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:18 smithi045 ceph-mon[31773]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T17:44:18.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:18 smithi045 ceph-mon[31773]: from='osd.9 [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-28T17:44:18.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:18 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T17:44:18.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:18 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:18.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:18 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:18.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:18 smithi045 ceph-mon[31773]: from='osd.9 [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-28T17:44:18.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:18 smithi071 ceph-mon[36597]: from='osd.9 [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T17:44:18.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:18 smithi071 ceph-mon[36597]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T17:44:18.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:18 smithi071 ceph-mon[36597]: from='osd.9 [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-28T17:44:18.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:18 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T17:44:18.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:18 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:18.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:18 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:18.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:18 smithi071 ceph-mon[36597]: from='osd.9 [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-28T17:44:18.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:18 smithi151 ceph-mon[36481]: from='osd.9 [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T17:44:18.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:18 smithi151 ceph-mon[36481]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T17:44:18.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:18 smithi151 ceph-mon[36481]: from='osd.9 [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-28T17:44:18.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:18 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T17:44:18.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:18 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:18.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:18 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:18.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:18 smithi151 ceph-mon[36481]: from='osd.9 [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-28T17:44:18.863 DEBUG:teuthology.orchestra.run.smithi045:osd.9> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@osd.9.service 2024-03-28T17:44:18.869 INFO:tasks.cephadm:Deploying osd.10 on smithi071 with /dev/vg_nvme/lv_1... 2024-03-28T17:44:18.869 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T17:44:19.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:19 smithi045 ceph-mon[31773]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:19.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:19 smithi045 ceph-mon[31773]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T17:44:19.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:19 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T17:44:19.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:19 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T17:44:19.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:19 smithi045 ceph-mon[31773]: from='osd.9 [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684]' entity='osd.9' 2024-03-28T17:44:19.114 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T17:44:19.236 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:19 smithi071 ceph-mon[36597]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:19.236 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:19 smithi071 ceph-mon[36597]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T17:44:19.236 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:19 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T17:44:19.236 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:19 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T17:44:19.236 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:19 smithi071 ceph-mon[36597]: from='osd.9 [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684]' entity='osd.9' 2024-03-28T17:44:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:19 smithi151 ceph-mon[36481]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:19 smithi151 ceph-mon[36481]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T17:44:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:19 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T17:44:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:19 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T17:44:19.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:19 smithi151 ceph-mon[36481]: from='osd.9 [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684]' entity='osd.9' 2024-03-28T17:44:20.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:20 smithi045 ceph-mon[31773]: purged_snaps scrub starts 2024-03-28T17:44:20.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:20 smithi045 ceph-mon[31773]: purged_snaps scrub ok 2024-03-28T17:44:20.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:20 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T17:44:20.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:20 smithi045 ceph-mon[31773]: osd.9 [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684] boot 2024-03-28T17:44:20.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:20 smithi045 ceph-mon[31773]: osdmap e67: 10 total, 10 up, 10 in 2024-03-28T17:44:20.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:20 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T17:44:20.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:20 smithi071 ceph-mon[36597]: purged_snaps scrub starts 2024-03-28T17:44:20.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:20 smithi071 ceph-mon[36597]: purged_snaps scrub ok 2024-03-28T17:44:20.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T17:44:20.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:20 smithi071 ceph-mon[36597]: osd.9 [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684] boot 2024-03-28T17:44:20.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:20 smithi071 ceph-mon[36597]: osdmap e67: 10 total, 10 up, 10 in 2024-03-28T17:44:20.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:20 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T17:44:20.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:20 smithi151 ceph-mon[36481]: purged_snaps scrub starts 2024-03-28T17:44:20.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:20 smithi151 ceph-mon[36481]: purged_snaps scrub ok 2024-03-28T17:44:20.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:20 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T17:44:20.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:20 smithi151 ceph-mon[36481]: osd.9 [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684] boot 2024-03-28T17:44:20.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:20 smithi151 ceph-mon[36481]: osdmap e67: 10 total, 10 up, 10 in 2024-03-28T17:44:20.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:20 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T17:44:21.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:21 smithi045 ceph-mon[31773]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:21.240 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:21 smithi071 ceph-mon[36597]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:21.240 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T17:44:21.241 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T17:44:21.241 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10+0 records in 2024-03-28T17:44:21.241 INFO:teuthology.orchestra.run.smithi071.stderr:10+0 records out 2024-03-28T17:44:21.241 INFO:teuthology.orchestra.run.smithi071.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113424 s, 924 MB/s 2024-03-28T17:44:21.241 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping successful for: 2024-03-28T17:44:21.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:21 smithi151 ceph-mon[36481]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-28T17:44:21.945 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi071:vg_nvme/lv_1 2024-03-28T17:44:22.187 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T17:44:22.308 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:22 smithi071 ceph-mon[36597]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T17:44:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:22 smithi151 ceph-mon[36481]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T17:44:22.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:22 smithi045 ceph-mon[31773]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T17:44:23.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:23 smithi071 ceph-mon[36597]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:23.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:23 smithi151 ceph-mon[36481]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:23.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:23 smithi045 ceph-mon[31773]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:23.925 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.924+0000 7fc5d6aae700 1 -- 172.21.15.71:0/1056689954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5d0102cd0 msgr2=0x7fc5d0109580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:44:23.925 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.924+0000 7fc5d6aae700 1 --2- 172.21.15.71:0/1056689954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5d0102cd0 0x7fc5d0109580 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc5c800b0d0 tx=0x7fc5c800b3e0 comp rx=0 tx=0).stop 2024-03-28T17:44:23.925 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.924+0000 7fc5d6aae700 1 -- 172.21.15.71:0/1056689954 shutdown_connections 2024-03-28T17:44:23.925 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.924+0000 7fc5d6aae700 1 --2- 172.21.15.71:0/1056689954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5d0102cd0 0x7fc5d0109580 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:44:23.925 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.924+0000 7fc5d6aae700 1 --2- 172.21.15.71:0/1056689954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d0101130 0x7fc5d0101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:44:23.925 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.924+0000 7fc5d6aae700 1 --2- 172.21.15.71:0/1056689954 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5d0102330 0x7fc5d0102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:44:23.925 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.924+0000 7fc5d6aae700 1 -- 172.21.15.71:0/1056689954 >> 172.21.15.71:0/1056689954 conn(0x7fc5d00fc6f0 msgr2=0x7fc5d00feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:44:23.925 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.924+0000 7fc5d6aae700 1 -- 172.21.15.71:0/1056689954 shutdown_connections 2024-03-28T17:44:23.925 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.924+0000 7fc5d6aae700 1 -- 172.21.15.71:0/1056689954 wait complete. 2024-03-28T17:44:23.927 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.925+0000 7fc5d6aae700 1 Processor -- start 2024-03-28T17:44:23.928 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.925+0000 7fc5d6aae700 1 -- start start 2024-03-28T17:44:23.928 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.925+0000 7fc5d6aae700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d0101130 0x7fc5d0105190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:44:23.928 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.925+0000 7fc5d6aae700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5d0102330 0x7fc5d01037e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:44:23.928 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.925+0000 7fc5d6aae700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5d0102cd0 0x7fc5d0103d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:44:23.928 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.925+0000 7fc5d6aae700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d010cd50 con 0x7fc5d0101130 2024-03-28T17:44:23.928 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.925+0000 7fc5d6aae700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d010cbd0 con 0x7fc5d0102330 2024-03-28T17:44:23.928 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.925+0000 7fc5d6aae700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d010ced0 con 0x7fc5d0102cd0 2024-03-28T17:44:23.928 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.925+0000 7fc5cffff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5d0102330 0x7fc5d01037e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:44:23.928 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.925+0000 7fc5cffff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5d0102330 0x7fc5d01037e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51246/0 (socket says 172.21.15.71:51246) 2024-03-28T17:44:23.928 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.925+0000 7fc5cffff700 1 -- 172.21.15.71:0/852913401 learned_addr learned my addr 172.21.15.71:0/852913401 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-03-28T17:44:23.928 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.925+0000 7fc5d504b700 1 --2- 172.21.15.71:0/852913401 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5d0102cd0 0x7fc5d0103d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:44:23.929 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.926+0000 7fc5cffff700 1 -- 172.21.15.71:0/852913401 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5d0102cd0 msgr2=0x7fc5d0103d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:44:23.929 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.926+0000 7fc5cffff700 1 --2- 172.21.15.71:0/852913401 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5d0102cd0 0x7fc5d0103d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:44:23.929 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.926+0000 7fc5cffff700 1 -- 172.21.15.71:0/852913401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d0101130 msgr2=0x7fc5d0105190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:44:23.929 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.926+0000 7fc5cffff700 1 --2- 172.21.15.71:0/852913401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d0101130 0x7fc5d0105190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:44:23.929 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.926+0000 7fc5cffff700 1 -- 172.21.15.71:0/852913401 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d0104260 con 0x7fc5d0102330 2024-03-28T17:44:23.929 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.926+0000 7fc5cffff700 1 --2- 172.21.15.71:0/852913401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5d0102330 0x7fc5d01037e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc00d980 tx=0x7fc5bc00dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:44:23.929 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.926+0000 7fc5cdffb700 1 -- 172.21.15.71:0/852913401 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc017070 con 0x7fc5d0102330 2024-03-28T17:44:23.929 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.926+0000 7fc5cdffb700 1 -- 172.21.15.71:0/852913401 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc5bc004d10 con 0x7fc5d0102330 2024-03-28T17:44:23.929 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.926+0000 7fc5d6aae700 1 -- 172.21.15.71:0/852913401 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d0104550 con 0x7fc5d0102330 2024-03-28T17:44:23.929 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.926+0000 7fc5cdffb700 1 -- 172.21.15.71:0/852913401 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc0055c0 con 0x7fc5d0102330 2024-03-28T17:44:23.929 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.926+0000 7fc5d6aae700 1 -- 172.21.15.71:0/852913401 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d01a6f40 con 0x7fc5d0102330 2024-03-28T17:44:23.929 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.927+0000 7fc5d6aae700 1 -- 172.21.15.71:0/852913401 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d01a7110 con 0x7fc5d0102330 2024-03-28T17:44:23.930 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.928+0000 7fc5cdffb700 1 -- 172.21.15.71:0/852913401 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc5bc004750 con 0x7fc5d0102330 2024-03-28T17:44:23.930 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.929+0000 7fc5cdffb700 1 --2- 172.21.15.71:0/852913401 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc5c00a2890 0x7fc5c00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:44:23.930 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.929+0000 7fc5cdffb700 1 -- 172.21.15.71:0/852913401 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6261+0+0 (secure 0 0 0) 0x7fc5bc0ce1b0 con 0x7fc5d0102330 2024-03-28T17:44:23.930 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.929+0000 7fc5d484a700 1 --2- 172.21.15.71:0/852913401 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc5c00a2890 0x7fc5c00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:44:23.931 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.930+0000 7fc5d484a700 1 --2- 172.21.15.71:0/852913401 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc5c00a2890 0x7fc5c00a4d50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0102190 tx=0x7fc5c4006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:44:23.933 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:23.933+0000 7fc5cdffb700 1 -- 172.21.15.71:0/852913401 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc5bc020020 con 0x7fc5d0102330 2024-03-28T17:44:24.137 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:24.137+0000 7fc5d6aae700 1 -- 172.21.15.71:0/852913401 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5d006afa0 con 0x7fc5c00a2890 2024-03-28T17:44:25.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:25 smithi151 ceph-mon[36481]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:25.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:25 smithi151 ceph-mon[36481]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:44:25.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:44:25.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:44:25.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:25.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:25 smithi151 ceph-mon[36481]: Detected new or changed devices on smithi045 2024-03-28T17:44:25.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:25.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:25.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:25.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:25.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:25.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:25.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:25 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi045 to 4273M 2024-03-28T17:44:25.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:25.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:25.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:44:25.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:25.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:44:25.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:25.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:44:25.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:25.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:25 smithi045 ceph-mon[31773]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:25.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:25 smithi045 ceph-mon[31773]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:44:25.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:44:25.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:44:25.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:25.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:25 smithi045 ceph-mon[31773]: Detected new or changed devices on smithi045 2024-03-28T17:44:25.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:25.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:25.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:25.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:25.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:25.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:25.418 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:25 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi045 to 4273M 2024-03-28T17:44:25.418 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:25.418 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:25.418 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:44:25.418 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:25.418 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:44:25.418 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:25.418 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:44:25.418 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:25 smithi071 ceph-mon[36597]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:25 smithi071 ceph-mon[36597]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:44:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:44:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:44:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:25 smithi071 ceph-mon[36597]: Detected new or changed devices on smithi045 2024-03-28T17:44:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:25.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:25.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:25 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi045 to 4273M 2024-03-28T17:44:25.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:25.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:25.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:44:25.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:25.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:44:25.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:25.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:44:25.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:27.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:27 smithi045 ceph-mon[31773]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:27.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:27 smithi045 ceph-mon[31773]: from='client.? 172.21.15.71:0/3739365881' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba94762b-bc5c-4c97-9cc1-17e5651c14b9"}]: dispatch 2024-03-28T17:44:27.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:27 smithi045 ceph-mon[31773]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba94762b-bc5c-4c97-9cc1-17e5651c14b9"}]: dispatch 2024-03-28T17:44:27.420 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:27 smithi071 ceph-mon[36597]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:27.420 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:27 smithi071 ceph-mon[36597]: from='client.? 172.21.15.71:0/3739365881' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba94762b-bc5c-4c97-9cc1-17e5651c14b9"}]: dispatch 2024-03-28T17:44:27.420 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:27 smithi071 ceph-mon[36597]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba94762b-bc5c-4c97-9cc1-17e5651c14b9"}]: dispatch 2024-03-28T17:44:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:27 smithi151 ceph-mon[36481]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:27 smithi151 ceph-mon[36481]: from='client.? 172.21.15.71:0/3739365881' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba94762b-bc5c-4c97-9cc1-17e5651c14b9"}]: dispatch 2024-03-28T17:44:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:27 smithi151 ceph-mon[36481]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba94762b-bc5c-4c97-9cc1-17e5651c14b9"}]: dispatch 2024-03-28T17:44:28.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:28 smithi045 ceph-mon[31773]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba94762b-bc5c-4c97-9cc1-17e5651c14b9"}]': finished 2024-03-28T17:44:28.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:28 smithi045 ceph-mon[31773]: osdmap e69: 11 total, 10 up, 11 in 2024-03-28T17:44:28.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:28 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T17:44:28.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:28 smithi045 ceph-mon[31773]: from='client.? 172.21.15.71:0/4253841100' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:44:28.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:28 smithi071 ceph-mon[36597]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba94762b-bc5c-4c97-9cc1-17e5651c14b9"}]': finished 2024-03-28T17:44:28.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:28 smithi071 ceph-mon[36597]: osdmap e69: 11 total, 10 up, 11 in 2024-03-28T17:44:28.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:28 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T17:44:28.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:28 smithi071 ceph-mon[36597]: from='client.? 172.21.15.71:0/4253841100' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:44:28.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:28 smithi151 ceph-mon[36481]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba94762b-bc5c-4c97-9cc1-17e5651c14b9"}]': finished 2024-03-28T17:44:28.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:28 smithi151 ceph-mon[36481]: osdmap e69: 11 total, 10 up, 11 in 2024-03-28T17:44:28.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:28 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T17:44:28.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:28 smithi151 ceph-mon[36481]: from='client.? 172.21.15.71:0/4253841100' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:44:29.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:29 smithi045 ceph-mon[31773]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:29.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:29 smithi071 ceph-mon[36597]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:29.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:29 smithi151 ceph-mon[36481]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:31.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:31 smithi045 ceph-mon[31773]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:31.476 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:31 smithi071 ceph-mon[36597]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:31 smithi151 ceph-mon[36481]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:33.144 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:33 smithi071 ceph-mon[36597]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:33.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:33 smithi045 ceph-mon[31773]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:33 smithi151 ceph-mon[36481]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:35.258 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:35 smithi071 ceph-mon[36597]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:35.258 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:35 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T17:44:35.258 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:35 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:35.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:35 smithi045 ceph-mon[31773]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:35.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:35 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T17:44:35.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:35 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:35.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:35 smithi151 ceph-mon[36481]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:35.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:35 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T17:44:35.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:35 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:36.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:36 smithi045 ceph-mon[31773]: Deploying daemon osd.10 on smithi071 2024-03-28T17:44:36.424 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:36 smithi071 ceph-mon[36597]: Deploying daemon osd.10 on smithi071 2024-03-28T17:44:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:36 smithi151 ceph-mon[36481]: Deploying daemon osd.10 on smithi071 2024-03-28T17:44:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:37 smithi045 ceph-mon[31773]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:37.514 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:37 smithi071 ceph-mon[36597]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:37 smithi151 ceph-mon[36481]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:39.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:39 smithi045 ceph-mon[31773]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:39.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:39 smithi071 ceph-mon[36597]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:39.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:39 smithi151 ceph-mon[36481]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:41.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:41 smithi071 ceph-mon[36597]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:41.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:41 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:41.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:41 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:41.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:41 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:44:41.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:41 smithi045 ceph-mon[31773]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:41.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:41 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:41.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:41 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:41.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:41 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:44:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:41 smithi151 ceph-mon[36481]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:41.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:41 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:41.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:41 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:41.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:41 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:44:43.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:43 smithi071 ceph-mon[36597]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:43.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:43 smithi045 ceph-mon[31773]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:43.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:43 smithi151 ceph-mon[36481]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:44.674 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:44.674+0000 7fc5cdffb700 1 -- 172.21.15.71:0/852913401 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fc5d006afa0 con 0x7fc5c00a2890 2024-03-28T17:44:44.675 INFO:teuthology.orchestra.run.smithi071.stdout:Created osd(s) 10 on host 'smithi071' 2024-03-28T17:44:44.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:44.676+0000 7fc5d6aae700 1 -- 172.21.15.71:0/852913401 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc5c00a2890 msgr2=0x7fc5c00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:44:44.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:44.676+0000 7fc5d6aae700 1 --2- 172.21.15.71:0/852913401 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc5c00a2890 0x7fc5c00a4d50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0102190 tx=0x7fc5c4006d20 comp rx=0 tx=0).stop 2024-03-28T17:44:44.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:44.676+0000 7fc5d6aae700 1 -- 172.21.15.71:0/852913401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5d0102330 msgr2=0x7fc5d01037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:44:44.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:44.676+0000 7fc5d6aae700 1 --2- 172.21.15.71:0/852913401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5d0102330 0x7fc5d01037e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc00d980 tx=0x7fc5bc00dd40 comp rx=0 tx=0).stop 2024-03-28T17:44:44.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:44.676+0000 7fc5d6aae700 1 -- 172.21.15.71:0/852913401 shutdown_connections 2024-03-28T17:44:44.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:44.676+0000 7fc5d6aae700 1 --2- 172.21.15.71:0/852913401 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc5c00a2890 0x7fc5c00a4d50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:44:44.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:44.676+0000 7fc5d6aae700 1 --2- 172.21.15.71:0/852913401 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5d0102cd0 0x7fc5d0103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:44:44.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:44.676+0000 7fc5d6aae700 1 --2- 172.21.15.71:0/852913401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d0101130 0x7fc5d0105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:44:44.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:44.676+0000 7fc5d6aae700 1 --2- 172.21.15.71:0/852913401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5d0102330 0x7fc5d01037e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:44:44.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:44.676+0000 7fc5d6aae700 1 -- 172.21.15.71:0/852913401 >> 172.21.15.71:0/852913401 conn(0x7fc5d00fc6f0 msgr2=0x7fc5d006a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:44:44.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:44.676+0000 7fc5d6aae700 1 -- 172.21.15.71:0/852913401 shutdown_connections 2024-03-28T17:44:44.678 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-28T17:44:44.676+0000 7fc5d6aae700 1 -- 172.21.15.71:0/852913401 wait complete. 2024-03-28T17:44:45.049 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:44 smithi071 ceph-mon[36597]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:45.049 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:45.049 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:45.049 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:45.049 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:44:45.049 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:45.049 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:44 smithi071 ceph-mon[36597]: from='osd.10 [v2:172.21.15.71:6824/1099494037,v1:172.21.15.71:6825/1099494037]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T17:44:45.049 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:44 smithi071 ceph-mon[36597]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T17:44:45.049 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:45.049 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:45.049 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:44:45.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:44 smithi151 ceph-mon[36481]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:45.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:45.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:45.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:45.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:44:45.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:45.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:44 smithi151 ceph-mon[36481]: from='osd.10 [v2:172.21.15.71:6824/1099494037,v1:172.21.15.71:6825/1099494037]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T17:44:45.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:44 smithi151 ceph-mon[36481]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T17:44:45.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:45.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:45.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:44:45.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:44 smithi045 ceph-mon[31773]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:45.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:45.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:45.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:45.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:44:45.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:45.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:44 smithi045 ceph-mon[31773]: from='osd.10 [v2:172.21.15.71:6824/1099494037,v1:172.21.15.71:6825/1099494037]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T17:44:45.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:44 smithi045 ceph-mon[31773]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T17:44:45.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:45.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:45.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:44:45.357 DEBUG:teuthology.orchestra.run.smithi071:osd.10> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@osd.10.service 2024-03-28T17:44:45.360 INFO:tasks.cephadm:Deploying osd.11 on smithi151 with /dev/vg_nvme/lv_1... 2024-03-28T17:44:45.360 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T17:44:45.610 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T17:44:46.153 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:46 smithi071 ceph-mon[36597]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T17:44:46.153 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:46 smithi071 ceph-mon[36597]: from='osd.10 [v2:172.21.15.71:6824/1099494037,v1:172.21.15.71:6825/1099494037]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:44:46.154 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:46 smithi071 ceph-mon[36597]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T17:44:46.154 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:46 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T17:44:46.154 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:46 smithi071 ceph-mon[36597]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:44:46.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:46 smithi151 ceph-mon[36481]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T17:44:46.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:46 smithi151 ceph-mon[36481]: from='osd.10 [v2:172.21.15.71:6824/1099494037,v1:172.21.15.71:6825/1099494037]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:44:46.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:46 smithi151 ceph-mon[36481]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T17:44:46.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:46 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T17:44:46.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:46 smithi151 ceph-mon[36481]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:44:46.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:46 smithi045 ceph-mon[31773]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T17:44:46.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:46 smithi045 ceph-mon[31773]: from='osd.10 [v2:172.21.15.71:6824/1099494037,v1:172.21.15.71:6825/1099494037]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:44:46.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:46 smithi045 ceph-mon[31773]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T17:44:46.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:46 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T17:44:46.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:46 smithi045 ceph-mon[31773]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-28T17:44:46.560 INFO:journalctl@ceph.osd.10.smithi071.stdout:Mar 28 17:44:46 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-osd-10[61669]: 2024-03-28T17:44:46.153+0000 7f3e7db07700 -1 osd.10 0 waiting for initial osdmap 2024-03-28T17:44:46.561 INFO:journalctl@ceph.osd.10.smithi071.stdout:Mar 28 17:44:46 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-osd-10[61669]: 2024-03-28T17:44:46.164+0000 7f3e798fa700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T17:44:47.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:47 smithi071 ceph-mon[36597]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:47.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:47 smithi071 ceph-mon[36597]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-28T17:44:47.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:47 smithi071 ceph-mon[36597]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T17:44:47.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:47 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T17:44:47.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:47 smithi045 ceph-mon[31773]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:47 smithi045 ceph-mon[31773]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-28T17:44:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:47 smithi045 ceph-mon[31773]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T17:44:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:47 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T17:44:47.444 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:47 smithi151 ceph-mon[36481]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T17:44:47.444 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:47 smithi151 ceph-mon[36481]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-28T17:44:47.444 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:47 smithi151 ceph-mon[36481]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T17:44:47.444 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:47 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T17:44:47.798 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T17:44:47.798 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T17:44:47.798 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2024-03-28T17:44:47.798 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2024-03-28T17:44:47.798 INFO:teuthology.orchestra.run.smithi151.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112949 s, 928 MB/s 2024-03-28T17:44:47.798 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 2024-03-28T17:44:47.799 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2024-03-28T17:44:48.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:48 smithi045 ceph-mon[31773]: purged_snaps scrub starts 2024-03-28T17:44:48.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:48 smithi045 ceph-mon[31773]: purged_snaps scrub ok 2024-03-28T17:44:48.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:48 smithi045 ceph-mon[31773]: osd.10 [v2:172.21.15.71:6824/1099494037,v1:172.21.15.71:6825/1099494037] boot 2024-03-28T17:44:48.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:48 smithi045 ceph-mon[31773]: osdmap e72: 11 total, 11 up, 11 in 2024-03-28T17:44:48.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:48 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T17:44:48.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:48 smithi045 ceph-mon[31773]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T17:44:48.436 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:48 smithi071 ceph-mon[36597]: purged_snaps scrub starts 2024-03-28T17:44:48.437 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:48 smithi071 ceph-mon[36597]: purged_snaps scrub ok 2024-03-28T17:44:48.437 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:48 smithi071 ceph-mon[36597]: osd.10 [v2:172.21.15.71:6824/1099494037,v1:172.21.15.71:6825/1099494037] boot 2024-03-28T17:44:48.437 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:48 smithi071 ceph-mon[36597]: osdmap e72: 11 total, 11 up, 11 in 2024-03-28T17:44:48.437 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:48 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T17:44:48.437 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:48 smithi071 ceph-mon[36597]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T17:44:48.481 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:48 smithi151 ceph-mon[36481]: purged_snaps scrub starts 2024-03-28T17:44:48.482 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:48 smithi151 ceph-mon[36481]: purged_snaps scrub ok 2024-03-28T17:44:48.482 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:48 smithi151 ceph-mon[36481]: osd.10 [v2:172.21.15.71:6824/1099494037,v1:172.21.15.71:6825/1099494037] boot 2024-03-28T17:44:48.482 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:48 smithi151 ceph-mon[36481]: osdmap e72: 11 total, 11 up, 11 in 2024-03-28T17:44:48.482 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:48 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T17:44:48.482 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:48 smithi151 ceph-mon[36481]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T17:44:48.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi151:vg_nvme/lv_1 2024-03-28T17:44:48.756 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T17:44:49.742 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:49 smithi151 ceph-mon[36481]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:44:49.742 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:49 smithi151 ceph-mon[36481]: Detected new or changed devices on smithi071 2024-03-28T17:44:49.742 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:49 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:49.742 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:49 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:49.742 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:49 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:49.742 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:49 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:49.742 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:49 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:49.743 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:49 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:49.743 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:49 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi071 to 4270M 2024-03-28T17:44:49.743 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:49 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:49.743 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:49 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:49.743 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:49 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:44:49.743 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:49 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:49.743 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:49 smithi151 ceph-mon[36481]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T17:44:49.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:49 smithi071 ceph-mon[36597]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:44:49.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:49 smithi071 ceph-mon[36597]: Detected new or changed devices on smithi071 2024-03-28T17:44:49.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:49 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:49.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:49 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:49.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:49 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:49.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:49 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:49.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:49 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:49.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:49 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:49.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:49 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi071 to 4270M 2024-03-28T17:44:49.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:49 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:49.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:49 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:49.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:49 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:44:49.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:49 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:49.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:49 smithi071 ceph-mon[36597]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T17:44:49.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:49 smithi045 ceph-mon[31773]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:44:49.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:49 smithi045 ceph-mon[31773]: Detected new or changed devices on smithi071 2024-03-28T17:44:49.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:49 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:49.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:49 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:49.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:49 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:49.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:49 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:49.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:49 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:49.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:49 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:44:49.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:49 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi071 to 4270M 2024-03-28T17:44:49.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:49 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:49.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:49 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:49.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:49 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:44:49.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:49 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:44:49.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:49 smithi045 ceph-mon[31773]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T17:44:50.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.589+0000 7fdef7fbc700 1 -- 172.21.15.151:0/3897486677 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdef0102c60 msgr2=0x7fdef0109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:44:50.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.589+0000 7fdef7fbc700 1 --2- 172.21.15.151:0/3897486677 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdef0102c60 0x7fdef0109510 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdee800b0a0 tx=0x7fdee800b3b0 comp rx=0 tx=0).stop 2024-03-28T17:44:50.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.589+0000 7fdef7fbc700 1 -- 172.21.15.151:0/3897486677 shutdown_connections 2024-03-28T17:44:50.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.589+0000 7fdef7fbc700 1 --2- 172.21.15.151:0/3897486677 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdef0102c60 0x7fdef0109510 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:44:50.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.589+0000 7fdef7fbc700 1 --2- 172.21.15.151:0/3897486677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdef01010c0 0x7fdef01014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:44:50.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.589+0000 7fdef7fbc700 1 --2- 172.21.15.151:0/3897486677 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdef01022c0 0x7fdef0102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:44:50.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.589+0000 7fdef7fbc700 1 -- 172.21.15.151:0/3897486677 >> 172.21.15.151:0/3897486677 conn(0x7fdef00fc660 msgr2=0x7fdef00feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:44:50.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.589+0000 7fdef7fbc700 1 -- 172.21.15.151:0/3897486677 shutdown_connections 2024-03-28T17:44:50.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.589+0000 7fdef7fbc700 1 -- 172.21.15.151:0/3897486677 wait complete. 2024-03-28T17:44:50.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.590+0000 7fdef7fbc700 1 Processor -- start 2024-03-28T17:44:50.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.590+0000 7fdef7fbc700 1 -- start start 2024-03-28T17:44:50.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.590+0000 7fdef7fbc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdef01010c0 0x7fdef006b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:44:50.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.590+0000 7fdef7fbc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdef01022c0 0x7fdef019d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:44:50.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.590+0000 7fdef7fbc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdef0102c60 0x7fdef019f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:44:50.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.590+0000 7fdef7fbc700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef010cd50 con 0x7fdef01022c0 2024-03-28T17:44:50.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.590+0000 7fdef7fbc700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef010cbd0 con 0x7fdef01010c0 2024-03-28T17:44:50.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.590+0000 7fdef7fbc700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef010ced0 con 0x7fdef0102c60 2024-03-28T17:44:50.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.590+0000 7fdef6559700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdef0102c60 0x7fdef019f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:44:50.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.590+0000 7fdef6559700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdef0102c60 0x7fdef019f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:37798/0 (socket says 172.21.15.151:37798) 2024-03-28T17:44:50.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.590+0000 7fdef6559700 1 -- 172.21.15.151:0/3067364981 learned_addr learned my addr 172.21.15.151:0/3067364981 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-03-28T17:44:50.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.590+0000 7fdef5d58700 1 --2- 172.21.15.151:0/3067364981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdef01010c0 0x7fdef006b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:44:50.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.590+0000 7fdef6559700 1 -- 172.21.15.151:0/3067364981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdef01010c0 msgr2=0x7fdef006b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:44:50.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.591+0000 7fdef6559700 1 --2- 172.21.15.151:0/3067364981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdef01010c0 0x7fdef006b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:44:50.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.591+0000 7fdef6559700 1 -- 172.21.15.151:0/3067364981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdef01022c0 msgr2=0x7fdef019d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:44:50.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.591+0000 7fdef6559700 1 --2- 172.21.15.151:0/3067364981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdef01022c0 0x7fdef019d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:44:50.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.591+0000 7fdef6559700 1 -- 172.21.15.151:0/3067364981 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef01a0060 con 0x7fdef0102c60 2024-03-28T17:44:50.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.591+0000 7fdef5d58700 1 --2- 172.21.15.151:0/3067364981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdef01010c0 0x7fdef006b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:44:50.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.591+0000 7fdef6559700 1 --2- 172.21.15.151:0/3067364981 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdef0102c60 0x7fdef019f790 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdee800b070 tx=0x7fdee800be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:44:50.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.591+0000 7fdee6ffd700 1 -- 172.21.15.151:0/3067364981 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee8021070 con 0x7fdef0102c60 2024-03-28T17:44:50.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.591+0000 7fdee6ffd700 1 -- 172.21.15.151:0/3067364981 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdee8005520 con 0x7fdef0102c60 2024-03-28T17:44:50.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.591+0000 7fdef7fbc700 1 -- 172.21.15.151:0/3067364981 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef01a02f0 con 0x7fdef0102c60 2024-03-28T17:44:50.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.591+0000 7fdee6ffd700 1 -- 172.21.15.151:0/3067364981 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee801b900 con 0x7fdef0102c60 2024-03-28T17:44:50.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.591+0000 7fdef7fbc700 1 -- 172.21.15.151:0/3067364981 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef01a07c0 con 0x7fdef0102c60 2024-03-28T17:44:50.594 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.592+0000 7fdef7fbc700 1 -- 172.21.15.151:0/3067364981 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef002da30 con 0x7fdef0102c60 2024-03-28T17:44:50.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.593+0000 7fdee6ffd700 1 -- 172.21.15.151:0/3067364981 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdee8024070 con 0x7fdef0102c60 2024-03-28T17:44:50.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.593+0000 7fdee6ffd700 1 --2- 172.21.15.151:0/3067364981 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdedc0a2890 0x7fdedc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:44:50.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.594+0000 7fdee6ffd700 1 -- 172.21.15.151:0/3067364981 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6694+0+0 (secure 0 0 0) 0x7fdee800f020 con 0x7fdef0102c60 2024-03-28T17:44:50.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.594+0000 7fdef5d58700 1 --2- 172.21.15.151:0/3067364981 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdedc0a2890 0x7fdedc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:44:50.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.595+0000 7fdef5d58700 1 --2- 172.21.15.151:0/3067364981 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdedc0a2890 0x7fdedc0a4d50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdef0102120 tx=0x7fdeec008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:44:50.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.597+0000 7fdee6ffd700 1 -- 172.21.15.151:0/3067364981 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdee802b020 con 0x7fdef0102c60 2024-03-28T17:44:50.801 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:44:50.801+0000 7fdef7fbc700 1 -- 172.21.15.151:0/3067364981 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fdef00625d0 con 0x7fdedc0a2890 2024-03-28T17:44:51.438 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:51 smithi151 ceph-mon[36481]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:44:51.438 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:51 smithi151 ceph-mon[36481]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T17:44:51.438 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:51 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:44:51.438 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:51 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:44:51.438 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:51 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:51 smithi071 ceph-mon[36597]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:44:51.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:51 smithi071 ceph-mon[36597]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T17:44:51.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:51 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:44:51.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:51 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:44:51.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:51 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:51.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:51 smithi045 ceph-mon[31773]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:44:51.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:51 smithi045 ceph-mon[31773]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T17:44:51.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:51 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T17:44:51.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:51 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T17:44:51.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:51 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:44:52.465 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:52 smithi151 ceph-mon[36481]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:44:52.465 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:52 smithi151 ceph-mon[36481]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T17:44:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:52 smithi071 ceph-mon[36597]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:44:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:52 smithi071 ceph-mon[36597]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T17:44:52.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:52 smithi045 ceph-mon[31773]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:44:52.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:52 smithi045 ceph-mon[31773]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T17:44:53.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:53 smithi071 ceph-mon[36597]: pgmap v268: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T17:44:53.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:53 smithi151 ceph-mon[36481]: pgmap v268: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T17:44:53.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:53 smithi045 ceph-mon[31773]: pgmap v268: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T17:44:54.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:54 smithi151 ceph-mon[36481]: from='client.? 172.21.15.151:0/1867772400' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d178378-b9f5-40f6-99ef-3d3d445da640"}]: dispatch 2024-03-28T17:44:54.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:54 smithi151 ceph-mon[36481]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d178378-b9f5-40f6-99ef-3d3d445da640"}]: dispatch 2024-03-28T17:44:54.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:54 smithi151 ceph-mon[36481]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d178378-b9f5-40f6-99ef-3d3d445da640"}]': finished 2024-03-28T17:44:54.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:54 smithi151 ceph-mon[36481]: osdmap e77: 12 total, 11 up, 12 in 2024-03-28T17:44:54.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:54 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T17:44:54.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:54 smithi071 ceph-mon[36597]: from='client.? 172.21.15.151:0/1867772400' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d178378-b9f5-40f6-99ef-3d3d445da640"}]: dispatch 2024-03-28T17:44:54.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:54 smithi071 ceph-mon[36597]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d178378-b9f5-40f6-99ef-3d3d445da640"}]: dispatch 2024-03-28T17:44:54.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:54 smithi071 ceph-mon[36597]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d178378-b9f5-40f6-99ef-3d3d445da640"}]': finished 2024-03-28T17:44:54.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:54 smithi071 ceph-mon[36597]: osdmap e77: 12 total, 11 up, 12 in 2024-03-28T17:44:54.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:54 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T17:44:54.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:54 smithi045 ceph-mon[31773]: from='client.? 172.21.15.151:0/1867772400' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d178378-b9f5-40f6-99ef-3d3d445da640"}]: dispatch 2024-03-28T17:44:54.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:54 smithi045 ceph-mon[31773]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d178378-b9f5-40f6-99ef-3d3d445da640"}]: dispatch 2024-03-28T17:44:54.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:54 smithi045 ceph-mon[31773]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d178378-b9f5-40f6-99ef-3d3d445da640"}]': finished 2024-03-28T17:44:54.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:54 smithi045 ceph-mon[31773]: osdmap e77: 12 total, 11 up, 12 in 2024-03-28T17:44:54.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:54 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T17:44:55.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:55 smithi045 ceph-mon[31773]: pgmap v269: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-28T17:44:55.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:55 smithi045 ceph-mon[31773]: from='client.? 172.21.15.151:0/4236561970' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:44:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:55 smithi071 ceph-mon[36597]: pgmap v269: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-28T17:44:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:55 smithi071 ceph-mon[36597]: from='client.? 172.21.15.151:0/4236561970' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:44:55.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:55 smithi151 ceph-mon[36481]: pgmap v269: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-28T17:44:55.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:55 smithi151 ceph-mon[36481]: from='client.? 172.21.15.151:0/4236561970' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T17:44:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:57 smithi071 ceph-mon[36597]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T17:44:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:57 smithi151 ceph-mon[36481]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T17:44:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:57 smithi045 ceph-mon[31773]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T17:44:59.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:44:59 smithi071 ceph-mon[36597]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-28T17:44:59.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:44:59 smithi151 ceph-mon[36481]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-28T17:44:59.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:44:59 smithi045 ceph-mon[31773]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-28T17:45:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:01 smithi071 ceph-mon[36597]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:01.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:01 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T17:45:01.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:01 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:01 smithi151 ceph-mon[36481]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:01 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T17:45:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:01 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:01.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:01 smithi045 ceph-mon[31773]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:01 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T17:45:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:01 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:02 smithi151 ceph-mon[36481]: Deploying daemon osd.11 on smithi151 2024-03-28T17:45:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:02 smithi071 ceph-mon[36597]: Deploying daemon osd.11 on smithi151 2024-03-28T17:45:02.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:02 smithi045 ceph-mon[31773]: Deploying daemon osd.11 on smithi151 2024-03-28T17:45:03.413 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:03 smithi151 ceph-mon[36481]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:03.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:03 smithi071 ceph-mon[36597]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:03.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:03 smithi045 ceph-mon[31773]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:05.307 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:05 smithi151 ceph-mon[36481]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:05.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:05 smithi071 ceph-mon[36597]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:05.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:05 smithi045 ceph-mon[31773]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:07 smithi071 ceph-mon[36597]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:07 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:07.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:07 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:45:07.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:07 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:07 smithi151 ceph-mon[36481]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:07 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:07 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:45:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:07 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:07 smithi045 ceph-mon[31773]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:07 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:07 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:45:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:07 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:09.429 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:09 smithi151 ceph-mon[36481]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:09.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:09 smithi071 ceph-mon[36597]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:09.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:09 smithi045 ceph-mon[31773]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:10.613 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 11 on host 'smithi151' 2024-03-28T17:45:10.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:10.613+0000 7fdee6ffd700 1 -- 172.21.15.151:0/3067364981 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fdef00625d0 con 0x7fdedc0a2890 2024-03-28T17:45:10.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:10.616+0000 7fdef7fbc700 1 -- 172.21.15.151:0/3067364981 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdedc0a2890 msgr2=0x7fdedc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:10.615 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:10.616+0000 7fdef7fbc700 1 --2- 172.21.15.151:0/3067364981 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdedc0a2890 0x7fdedc0a4d50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdef0102120 tx=0x7fdeec008040 comp rx=0 tx=0).stop 2024-03-28T17:45:10.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:10.616+0000 7fdef7fbc700 1 -- 172.21.15.151:0/3067364981 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdef0102c60 msgr2=0x7fdef019f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:10.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:10.616+0000 7fdef7fbc700 1 --2- 172.21.15.151:0/3067364981 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdef0102c60 0x7fdef019f790 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdee800b070 tx=0x7fdee800be20 comp rx=0 tx=0).stop 2024-03-28T17:45:10.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:10.616+0000 7fdef7fbc700 1 -- 172.21.15.151:0/3067364981 shutdown_connections 2024-03-28T17:45:10.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:10.616+0000 7fdef7fbc700 1 --2- 172.21.15.151:0/3067364981 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdedc0a2890 0x7fdedc0a4d50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:10.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:10.616+0000 7fdef7fbc700 1 --2- 172.21.15.151:0/3067364981 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdef0102c60 0x7fdef019f790 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:10.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:10.616+0000 7fdef7fbc700 1 --2- 172.21.15.151:0/3067364981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdef01010c0 0x7fdef006b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:10.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:10.616+0000 7fdef7fbc700 1 --2- 172.21.15.151:0/3067364981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdef01022c0 0x7fdef019d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:10.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:10.616+0000 7fdef7fbc700 1 -- 172.21.15.151:0/3067364981 >> 172.21.15.151:0/3067364981 conn(0x7fdef00fc660 msgr2=0x7fdef0105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:10.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:10.616+0000 7fdef7fbc700 1 -- 172.21.15.151:0/3067364981 shutdown_connections 2024-03-28T17:45:10.616 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:10.616+0000 7fdef7fbc700 1 -- 172.21.15.151:0/3067364981 wait complete. 2024-03-28T17:45:11.310 DEBUG:teuthology.orchestra.run.smithi151:osd.11> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@osd.11.service 2024-03-28T17:45:11.312 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-28T17:45:11.312 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-28T17:45:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:11 smithi045 ceph-mon[31773]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:11 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:11 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:11 smithi045 ceph-mon[31773]: from='osd.11 [v2:172.21.15.151:6824/2941055364,v1:172.21.15.151:6825/2941055364]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T17:45:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:11 smithi045 ceph-mon[31773]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T17:45:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:11 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:11 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:11.798 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:45:11.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:11 smithi151 ceph-mon[36481]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:11.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:11 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:11.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:11 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:11.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:11 smithi151 ceph-mon[36481]: from='osd.11 [v2:172.21.15.151:6824/2941055364,v1:172.21.15.151:6825/2941055364]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T17:45:11.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:11 smithi151 ceph-mon[36481]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T17:45:11.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:11 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:11.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:11 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:11 smithi071 ceph-mon[36597]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:12.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:11 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:12.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:11 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:12.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:11 smithi071 ceph-mon[36597]: from='osd.11 [v2:172.21.15.151:6824/2941055364,v1:172.21.15.151:6825/2941055364]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T17:45:12.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:11 smithi071 ceph-mon[36597]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T17:45:12.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:11 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:12.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:11 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:12.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:12 smithi045 ceph-mon[31773]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T17:45:12.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:12 smithi045 ceph-mon[31773]: from='osd.11 [v2:172.21.15.151:6824/2941055364,v1:172.21.15.151:6825/2941055364]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:45:12.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:12 smithi045 ceph-mon[31773]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T17:45:12.855 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:12 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T17:45:12.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:12 smithi045 ceph-mon[31773]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:45:12.899 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:12 smithi151 ceph-mon[36481]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T17:45:12.899 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:12 smithi151 ceph-mon[36481]: from='osd.11 [v2:172.21.15.151:6824/2941055364,v1:172.21.15.151:6825/2941055364]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:45:12.899 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:12 smithi151 ceph-mon[36481]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T17:45:12.899 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:12 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T17:45:12.899 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:12 smithi151 ceph-mon[36481]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:45:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:12 smithi071 ceph-mon[36597]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T17:45:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:12 smithi071 ceph-mon[36597]: from='osd.11 [v2:172.21.15.151:6824/2941055364,v1:172.21.15.151:6825/2941055364]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:45:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:12 smithi071 ceph-mon[36597]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T17:45:13.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:12 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T17:45:13.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:12 smithi071 ceph-mon[36597]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-28T17:45:13.323 INFO:journalctl@ceph.osd.11.smithi151.stdout:Mar 28 17:45:13 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-osd-11[61843]: 2024-03-28T17:45:13.038+0000 7fd2c28bd700 -1 osd.11 0 waiting for initial osdmap 2024-03-28T17:45:13.323 INFO:journalctl@ceph.osd.11.smithi151.stdout:Mar 28 17:45:13 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-osd-11[61843]: 2024-03-28T17:45:13.055+0000 7fd2ba6a8700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T17:45:13.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:13 smithi045 ceph-mon[31773]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:13.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:13 smithi045 ceph-mon[31773]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-28T17:45:13.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:13 smithi045 ceph-mon[31773]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T17:45:13.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:13 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T17:45:13.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:13 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T17:45:13.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:13 smithi045 ceph-mon[31773]: from='osd.11 ' entity='osd.11' 2024-03-28T17:45:13.956 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:13 smithi151 ceph-mon[36481]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:13.956 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:13 smithi151 ceph-mon[36481]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-28T17:45:13.956 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:13 smithi151 ceph-mon[36481]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T17:45:13.956 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:13 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T17:45:13.956 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:13 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T17:45:13.956 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:13 smithi151 ceph-mon[36481]: from='osd.11 ' entity='osd.11' 2024-03-28T17:45:14.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:13 smithi071 ceph-mon[36597]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:14.087 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:13 smithi071 ceph-mon[36597]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-28T17:45:14.087 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:13 smithi071 ceph-mon[36597]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T17:45:14.087 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T17:45:14.087 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T17:45:14.088 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:13 smithi071 ceph-mon[36597]: from='osd.11 ' entity='osd.11' 2024-03-28T17:45:14.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.132+0000 7f88c2c00700 1 -- 172.21.15.45:0/4173209568 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88bc104410 msgr2=0x7f88bc104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:14.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.132+0000 7f88c2c00700 1 --2- 172.21.15.45:0/4173209568 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88bc104410 0x7f88bc104830 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f88ac009920 tx=0x7f88ac009c30 comp rx=0 tx=0).stop 2024-03-28T17:45:14.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.133+0000 7f88c2c00700 1 -- 172.21.15.45:0/4173209568 shutdown_connections 2024-03-28T17:45:14.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.133+0000 7f88c2c00700 1 --2- 172.21.15.45:0/4173209568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88bc105fb0 0x7f88bc10c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:14.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.133+0000 7f88c2c00700 1 --2- 172.21.15.45:0/4173209568 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88bc104410 0x7f88bc104830 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:14.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.133+0000 7f88c2c00700 1 --2- 172.21.15.45:0/4173209568 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f88bc105610 0x7f88bc105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:14.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.133+0000 7f88c2c00700 1 -- 172.21.15.45:0/4173209568 >> 172.21.15.45:0/4173209568 conn(0x7f88bc0ff9b0 msgr2=0x7f88bc101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:14.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.133+0000 7f88c2c00700 1 -- 172.21.15.45:0/4173209568 shutdown_connections 2024-03-28T17:45:14.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.133+0000 7f88c2c00700 1 -- 172.21.15.45:0/4173209568 wait complete. 2024-03-28T17:45:14.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.134+0000 7f88c2c00700 1 Processor -- start 2024-03-28T17:45:14.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.134+0000 7f88c2c00700 1 -- start start 2024-03-28T17:45:14.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.134+0000 7f88c2c00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f88bc104410 0x7f88bc079f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:14.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.134+0000 7f88c2c00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88bc105610 0x7f88bc07a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:14.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.134+0000 7f88c2c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88bc105fb0 0x7f88bc076ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:14.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.134+0000 7f88c2c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc110030 con 0x7f88bc105fb0 2024-03-28T17:45:14.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.134+0000 7f88c2c00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc10feb0 con 0x7f88bc105610 2024-03-28T17:45:14.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.134+0000 7f88c2c00700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc1101b0 con 0x7f88bc104410 2024-03-28T17:45:14.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.135+0000 7f88bb400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88bc105610 0x7f88bc07a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:14.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.135+0000 7f88bb400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88bc105610 0x7f88bc07a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:51438/0 (socket says 172.21.15.45:51438) 2024-03-28T17:45:14.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.135+0000 7f88bb400700 1 -- 172.21.15.45:0/2053040520 learned_addr learned my addr 172.21.15.45:0/2053040520 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:45:14.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.135+0000 7f88bbe00700 1 --2- 172.21.15.45:0/2053040520 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f88bc104410 0x7f88bc079f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:14.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.135+0000 7f88c0c00700 1 --2- 172.21.15.45:0/2053040520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88bc105fb0 0x7f88bc076ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:14.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.135+0000 7f88bb400700 1 -- 172.21.15.45:0/2053040520 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f88bc104410 msgr2=0x7f88bc079f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:14.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.135+0000 7f88bb400700 1 --2- 172.21.15.45:0/2053040520 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f88bc104410 0x7f88bc079f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:14.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.135+0000 7f88bb400700 1 -- 172.21.15.45:0/2053040520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88bc105fb0 msgr2=0x7f88bc076ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:14.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.135+0000 7f88bb400700 1 --2- 172.21.15.45:0/2053040520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88bc105fb0 0x7f88bc076ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:14.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.135+0000 7f88bb400700 1 -- 172.21.15.45:0/2053040520 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88bc077390 con 0x7f88bc105610 2024-03-28T17:45:14.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.135+0000 7f88bb400700 1 --2- 172.21.15.45:0/2053040520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88bc105610 0x7f88bc07a480 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f88a800e9c0 tx=0x7f88a800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:14.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.136+0000 7f88b8c00700 1 -- 172.21.15.45:0/2053040520 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a8004a30 con 0x7f88bc105610 2024-03-28T17:45:14.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.136+0000 7f88b8c00700 1 -- 172.21.15.45:0/2053040520 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88a800cca0 con 0x7f88bc105610 2024-03-28T17:45:14.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.136+0000 7f88b8c00700 1 -- 172.21.15.45:0/2053040520 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a8005680 con 0x7f88bc105610 2024-03-28T17:45:14.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.136+0000 7f88c2c00700 1 -- 172.21.15.45:0/2053040520 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88bc1a6b20 con 0x7f88bc105610 2024-03-28T17:45:14.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.136+0000 7f88c2c00700 1 -- 172.21.15.45:0/2053040520 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88bc1a6fd0 con 0x7f88bc105610 2024-03-28T17:45:14.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.137+0000 7f88c2c00700 1 -- 172.21.15.45:0/2053040520 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88bc068e50 con 0x7f88bc105610 2024-03-28T17:45:14.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.139+0000 7f88b8c00700 1 -- 172.21.15.45:0/2053040520 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f88a800c7c0 con 0x7f88bc105610 2024-03-28T17:45:14.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.140+0000 7f88b8c00700 1 --2- 172.21.15.45:0/2053040520 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f88a00a2910 0x7f88a00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:14.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.140+0000 7f88b8c00700 1 -- 172.21.15.45:0/2053040520 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f88a80ce690 con 0x7f88bc105610 2024-03-28T17:45:14.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.140+0000 7f88bbe00700 1 --2- 172.21.15.45:0/2053040520 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f88a00a2910 0x7f88a00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:14.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.141+0000 7f88bbe00700 1 --2- 172.21.15.45:0/2053040520 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f88a00a2910 0x7f88a00a4dd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f88ac00fb70 tx=0x7f88ac019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:14.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.145+0000 7f88b8c00700 1 -- 172.21.15.45:0/2053040520 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f88a809bb30 con 0x7f88bc105610 2024-03-28T17:45:14.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.373+0000 7f88c2c00700 1 -- 172.21.15.45:0/2053040520 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f88bc062da0 con 0x7f88bc105610 2024-03-28T17:45:14.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.374+0000 7f88b8c00700 1 -- 172.21.15.45:0/2053040520 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) v1 ==== 74+0+133 (secure 0 0 0) 0x7f88a809b430 con 0x7f88bc105610 2024-03-28T17:45:14.375 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:45:14.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.378+0000 7f88c2c00700 1 -- 172.21.15.45:0/2053040520 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f88a00a2910 msgr2=0x7f88a00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:14.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.378+0000 7f88c2c00700 1 --2- 172.21.15.45:0/2053040520 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f88a00a2910 0x7f88a00a4dd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f88ac00fb70 tx=0x7f88ac019040 comp rx=0 tx=0).stop 2024-03-28T17:45:14.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.378+0000 7f88c2c00700 1 -- 172.21.15.45:0/2053040520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88bc105610 msgr2=0x7f88bc07a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:14.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.378+0000 7f88c2c00700 1 --2- 172.21.15.45:0/2053040520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88bc105610 0x7f88bc07a480 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f88a800e9c0 tx=0x7f88a800ecd0 comp rx=0 tx=0).stop 2024-03-28T17:45:14.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.379+0000 7f88c2c00700 1 -- 172.21.15.45:0/2053040520 shutdown_connections 2024-03-28T17:45:14.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.379+0000 7f88c2c00700 1 --2- 172.21.15.45:0/2053040520 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f88a00a2910 0x7f88a00a4dd0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:14.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.379+0000 7f88c2c00700 1 --2- 172.21.15.45:0/2053040520 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f88bc104410 0x7f88bc079f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:14.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.379+0000 7f88c2c00700 1 --2- 172.21.15.45:0/2053040520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88bc105fb0 0x7f88bc076ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:14.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.379+0000 7f88c2c00700 1 --2- 172.21.15.45:0/2053040520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88bc105610 0x7f88bc07a480 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:14.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.379+0000 7f88c2c00700 1 -- 172.21.15.45:0/2053040520 >> 172.21.15.45:0/2053040520 conn(0x7f88bc0ff9b0 msgr2=0x7f88bc108910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:14.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.379+0000 7f88c2c00700 1 -- 172.21.15.45:0/2053040520 shutdown_connections 2024-03-28T17:45:14.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:14.380+0000 7f88c2c00700 1 -- 172.21.15.45:0/2053040520 wait complete. 2024-03-28T17:45:14.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:14 smithi045 ceph-mon[31773]: purged_snaps scrub starts 2024-03-28T17:45:14.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:14 smithi045 ceph-mon[31773]: purged_snaps scrub ok 2024-03-28T17:45:14.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:14 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T17:45:14.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:14 smithi045 ceph-mon[31773]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:14.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:14 smithi045 ceph-mon[31773]: osd.11 [v2:172.21.15.151:6824/2941055364,v1:172.21.15.151:6825/2941055364] boot 2024-03-28T17:45:14.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:14 smithi045 ceph-mon[31773]: osdmap e80: 12 total, 12 up, 12 in 2024-03-28T17:45:14.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:14 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T17:45:14.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:14 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/2053040520' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T17:45:14.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:14 smithi151 ceph-mon[36481]: purged_snaps scrub starts 2024-03-28T17:45:14.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:14 smithi151 ceph-mon[36481]: purged_snaps scrub ok 2024-03-28T17:45:14.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:14 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T17:45:14.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:14 smithi151 ceph-mon[36481]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:14.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:14 smithi151 ceph-mon[36481]: osd.11 [v2:172.21.15.151:6824/2941055364,v1:172.21.15.151:6825/2941055364] boot 2024-03-28T17:45:14.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:14 smithi151 ceph-mon[36481]: osdmap e80: 12 total, 12 up, 12 in 2024-03-28T17:45:14.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:14 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T17:45:14.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:14 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/2053040520' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T17:45:15.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:14 smithi071 ceph-mon[36597]: purged_snaps scrub starts 2024-03-28T17:45:15.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:14 smithi071 ceph-mon[36597]: purged_snaps scrub ok 2024-03-28T17:45:15.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:14 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T17:45:15.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:14 smithi071 ceph-mon[36597]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T17:45:15.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:14 smithi071 ceph-mon[36597]: osd.11 [v2:172.21.15.151:6824/2941055364,v1:172.21.15.151:6825/2941055364] boot 2024-03-28T17:45:15.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:14 smithi071 ceph-mon[36597]: osdmap e80: 12 total, 12 up, 12 in 2024-03-28T17:45:15.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:14 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T17:45:15.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:14 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/2053040520' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T17:45:15.222 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1711647914,"num_in_osds":12,"osd_in_since":1711647893,"num_remapped_pgs":0} 2024-03-28T17:45:15.223 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T17:45:15.675 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:45:15.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:15 smithi151 ceph-mon[36481]: Detected new or changed devices on smithi151 2024-03-28T17:45:15.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:15 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:15.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:15 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:15.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:15 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:45:15.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:15 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:45:15.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:15 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:45:15.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:15 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:45:15.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:15 smithi045 ceph-mon[31773]: Detected new or changed devices on smithi151 2024-03-28T17:45:15.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:15 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:15.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:15 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:15.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:15 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:45:15.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:15 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:45:15.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:15 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:45:15.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:15 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:45:16.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:15 smithi071 ceph-mon[36597]: Detected new or changed devices on smithi151 2024-03-28T17:45:16.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:15 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:16.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:15 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:16.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:15 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:45:16.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:15 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:45:16.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:15 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:45:16.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:15 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:45:16.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:16 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi151 to 4270M 2024-03-28T17:45:16.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:16 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:16.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:16 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:16.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:16 smithi045 ceph-mon[31773]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T17:45:16.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:16 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:45:16.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:16 smithi045 ceph-mon[31773]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:16.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:16 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:16.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:16 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:45:16.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:16 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:16.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:16 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:45:16.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:16 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:16 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi151 to 4270M 2024-03-28T17:45:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:16 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:16 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:16 smithi071 ceph-mon[36597]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T17:45:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:16 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:45:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:16 smithi071 ceph-mon[36597]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:16 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:16 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:45:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:16 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:16 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:45:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:16 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:16 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi151 to 4270M 2024-03-28T17:45:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:16 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:16 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:16 smithi151 ceph-mon[36481]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T17:45:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:16 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:45:17.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:16 smithi151 ceph-mon[36481]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:17.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:16 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:17.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:16 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:45:17.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:16 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:17.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:16 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:45:17.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:16 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:17.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.825+0000 7f4e23800700 1 -- 172.21.15.45:0/83466116 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e1c076fe0 msgr2=0x7f4e1c112b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:17.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.825+0000 7f4e23800700 1 --2- 172.21.15.45:0/83466116 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e1c076fe0 0x7f4e1c112b00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4e080098e0 tx=0x7f4e08009bf0 comp rx=0 tx=0).stop 2024-03-28T17:45:17.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.825+0000 7f4e23800700 1 -- 172.21.15.45:0/83466116 shutdown_connections 2024-03-28T17:45:17.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.825+0000 7f4e23800700 1 --2- 172.21.15.45:0/83466116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e1c113040 0x7f4e1c115430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:17.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.825+0000 7f4e23800700 1 --2- 172.21.15.45:0/83466116 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e1c0766c0 0x7f4e1c076aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:17.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.825+0000 7f4e23800700 1 --2- 172.21.15.45:0/83466116 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e1c076fe0 0x7f4e1c112b00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:17.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.825+0000 7f4e23800700 1 -- 172.21.15.45:0/83466116 >> 172.21.15.45:0/83466116 conn(0x7f4e1c0ffe70 msgr2=0x7f4e1c102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:17.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.826+0000 7f4e23800700 1 -- 172.21.15.45:0/83466116 shutdown_connections 2024-03-28T17:45:17.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.826+0000 7f4e23800700 1 -- 172.21.15.45:0/83466116 wait complete. 2024-03-28T17:45:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.827+0000 7f4e23800700 1 Processor -- start 2024-03-28T17:45:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.827+0000 7f4e23800700 1 -- start start 2024-03-28T17:45:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.827+0000 7f4e23800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e1c0766c0 0x7f4e1c19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.827+0000 7f4e23800700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e1c076fe0 0x7f4e1c19f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.827+0000 7f4e23800700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e1c113040 0x7f4e1c1a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.827+0000 7f4e23800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1c118b50 con 0x7f4e1c0766c0 2024-03-28T17:45:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.827+0000 7f4e23800700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1c1189d0 con 0x7f4e1c076fe0 2024-03-28T17:45:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.827+0000 7f4e23800700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1c118cd0 con 0x7f4e1c113040 2024-03-28T17:45:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.828+0000 7f4e20e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e1c0766c0 0x7f4e1c19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.828+0000 7f4e20e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e1c0766c0 0x7f4e1c19f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42314/0 (socket says 172.21.15.45:42314) 2024-03-28T17:45:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.828+0000 7f4e20e00700 1 -- 172.21.15.45:0/523853992 learned_addr learned my addr 172.21.15.45:0/523853992 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:45:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.828+0000 7f4e1be00700 1 --2- 172.21.15.45:0/523853992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e1c076fe0 0x7f4e1c19f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.828+0000 7f4e21800700 1 --2- 172.21.15.45:0/523853992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e1c113040 0x7f4e1c1a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.828+0000 7f4e20e00700 1 -- 172.21.15.45:0/523853992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e1c113040 msgr2=0x7f4e1c1a38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.828+0000 7f4e20e00700 1 --2- 172.21.15.45:0/523853992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e1c113040 0x7f4e1c1a38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.828+0000 7f4e20e00700 1 -- 172.21.15.45:0/523853992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e1c076fe0 msgr2=0x7f4e1c19f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.828+0000 7f4e20e00700 1 --2- 172.21.15.45:0/523853992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e1c076fe0 0x7f4e1c19f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.828+0000 7f4e20e00700 1 -- 172.21.15.45:0/523853992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e1c1a4000 con 0x7f4e1c0766c0 2024-03-28T17:45:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.828+0000 7f4e1be00700 1 --2- 172.21.15.45:0/523853992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e1c076fe0 0x7f4e1c19f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:45:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.829+0000 7f4e20e00700 1 --2- 172.21.15.45:0/523853992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e1c0766c0 0x7f4e1c19f040 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4e1000ef10 tx=0x7f4e1000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.829+0000 7f4e19600700 1 -- 172.21.15.45:0/523853992 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1000c9c0 con 0x7f4e1c0766c0 2024-03-28T17:45:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.829+0000 7f4e23800700 1 -- 172.21.15.45:0/523853992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e1c1a4290 con 0x7f4e1c0766c0 2024-03-28T17:45:17.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.829+0000 7f4e19600700 1 -- 172.21.15.45:0/523853992 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e10019070 con 0x7f4e1c0766c0 2024-03-28T17:45:17.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.829+0000 7f4e19600700 1 -- 172.21.15.45:0/523853992 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e10007e60 con 0x7f4e1c0766c0 2024-03-28T17:45:17.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.829+0000 7f4e23800700 1 -- 172.21.15.45:0/523853992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e1c1abb70 con 0x7f4e1c0766c0 2024-03-28T17:45:17.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.831+0000 7f4e23800700 1 -- 172.21.15.45:0/523853992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e1c068e50 con 0x7f4e1c0766c0 2024-03-28T17:45:17.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.832+0000 7f4e19600700 1 -- 172.21.15.45:0/523853992 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4e10012050 con 0x7f4e1c0766c0 2024-03-28T17:45:17.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.834+0000 7f4e19600700 1 --2- 172.21.15.45:0/523853992 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4e040a2890 0x7f4e040a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:17.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.834+0000 7f4e19600700 1 -- 172.21.15.45:0/523853992 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4e100ce600 con 0x7f4e1c0766c0 2024-03-28T17:45:17.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.834+0000 7f4e1be00700 1 --2- 172.21.15.45:0/523853992 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4e040a2890 0x7f4e040a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:17.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.835+0000 7f4e1be00700 1 --2- 172.21.15.45:0/523853992 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4e040a2890 0x7f4e040a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c0795c0 tx=0x7f4e08009f80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:17.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:17.838+0000 7f4e19600700 1 -- 172.21.15.45:0/523853992 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4e10010040 con 0x7f4e1c0766c0 2024-03-28T17:45:18.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:18.066+0000 7f4e23800700 1 -- 172.21.15.45:0/523853992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e1c02da30 con 0x7f4e1c0766c0 2024-03-28T17:45:18.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:18.067+0000 7f4e19600700 1 -- 172.21.15.45:0/523853992 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18445 (secure 0 0 0) 0x7f4e1009baa0 con 0x7f4e1c0766c0 2024-03-28T17:45:18.068 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:45:18.068 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":81,"fsid":"9f4888ec-ed29-11ee-95d0-87774f69a715","created":"2024-03-28T17:36:51.623185+0000","modified":"2024-03-28T17:45:15.608581+0000","last_up_change":"2024-03-28T17:45:14.043432+0000","last_in_change":"2024-03-28T17:44:53.654489+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T17:40:54.208375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"74c56bc2-4169-4919-afd4-2ebd236f23e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6803","nonce":3795754023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6805","nonce":3795754023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6809","nonce":3795754023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6807","nonce":3795754023}]},"public_addr":"172.21.15.45:6803/3795754023","cluster_addr":"172.21.15.45:6805/3795754023","heartbeat_back_addr":"172.21.15.45:6809/3795754023","heartbeat_front_addr":"172.21.15.45:6807/3795754023","state":["exists","up"]},{"osd":1,"uuid":"78dcc21b-75b2-453d-b690-17b55b8ed1a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6801","nonce":2014529906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6803","nonce":2014529906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6807","nonce":2014529906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6805","nonce":2014529906}]},"public_addr":"172.21.15.71:6801/2014529906","cluster_addr":"172.21.15.71:6803/2014529906","heartbeat_back_addr":"172.21.15.71:6807/2014529906","heartbeat_front_addr":"172.21.15.71:6805/2014529906","state":["exists","up"]},{"osd":2,"uuid":"ca9f3f36-b71e-4962-a94f-cedf2f12eb7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6801","nonce":1850933775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6803","nonce":1850933775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6807","nonce":1850933775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6805","nonce":1850933775}]},"public_addr":"172.21.15.151:6801/1850933775","cluster_addr":"172.21.15.151:6803/1850933775","heartbeat_back_addr":"172.21.15.151:6807/1850933775","heartbeat_front_addr":"172.21.15.151:6805/1850933775","state":["exists","up"]},{"osd":3,"uuid":"d3f21f01-2973-413e-8daa-9974b8a44af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6811","nonce":3021569975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6813","nonce":3021569975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6817","nonce":3021569975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6815","nonce":3021569975}]},"public_addr":"172.21.15.45:6811/3021569975","cluster_addr":"172.21.15.45:6813/3021569975","heartbeat_back_addr":"172.21.15.45:6817/3021569975","heartbeat_front_addr":"172.21.15.45:6815/3021569975","state":["exists","up"]},{"osd":4,"uuid":"4a336f42-0605-492b-ad88-56cb02acdcf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6809","nonce":2069578333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6811","nonce":2069578333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6815","nonce":2069578333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6813","nonce":2069578333}]},"public_addr":"172.21.15.71:6809/2069578333","cluster_addr":"172.21.15.71:6811/2069578333","heartbeat_back_addr":"172.21.15.71:6815/2069578333","heartbeat_front_addr":"172.21.15.71:6813/2069578333","state":["exists","up"]},{"osd":5,"uuid":"c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6809","nonce":2629217227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6811","nonce":2629217227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6815","nonce":2629217227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6813","nonce":2629217227}]},"public_addr":"172.21.15.151:6809/2629217227","cluster_addr":"172.21.15.151:6811/2629217227","heartbeat_back_addr":"172.21.15.151:6815/2629217227","heartbeat_front_addr":"172.21.15.151:6813/2629217227","state":["exists","up"]},{"osd":6,"uuid":"58a6c5bd-a8b7-4892-b346-bcd7994bcb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6819","nonce":2077907039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6821","nonce":2077907039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6825","nonce":2077907039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6823","nonce":2077907039}]},"public_addr":"172.21.15.45:6819/2077907039","cluster_addr":"172.21.15.45:6821/2077907039","heartbeat_back_addr":"172.21.15.45:6825/2077907039","heartbeat_front_addr":"172.21.15.45:6823/2077907039","state":["exists","up"]},{"osd":7,"uuid":"f7a45d52-807b-4407-bb04-787b2a52a0ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6817","nonce":2000211251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6819","nonce":2000211251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6823","nonce":2000211251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6821","nonce":2000211251}]},"public_addr":"172.21.15.71:6817/2000211251","cluster_addr":"172.21.15.71:6819/2000211251","heartbeat_back_addr":"172.21.15.71:6823/2000211251","heartbeat_front_addr":"172.21.15.71:6821/2000211251","state":["exists","up"]},{"osd":8,"uuid":"88abcb0f-f189-49be-8315-c700c77369f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6817","nonce":3665460137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6819","nonce":3665460137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6823","nonce":3665460137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6821","nonce":3665460137}]},"public_addr":"172.21.15.151:6817/3665460137","cluster_addr":"172.21.15.151:6819/3665460137","heartbeat_back_addr":"172.21.15.151:6823/3665460137","heartbeat_front_addr":"172.21.15.151:6821/3665460137","state":["exists","up"]},{"osd":9,"uuid":"ed3fc096-961e-4392-abc2-c5aa4d050117","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6827","nonce":2122322684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6829","nonce":2122322684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6833","nonce":2122322684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6831","nonce":2122322684}]},"public_addr":"172.21.15.45:6827/2122322684","cluster_addr":"172.21.15.45:6829/2122322684","heartbeat_back_addr":"172.21.15.45:6833/2122322684","heartbeat_front_addr":"172.21.15.45:6831/2122322684","state":["exists","up"]},{"osd":10,"uuid":"ba94762b-bc5c-4c97-9cc1-17e5651c14b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6825","nonce":1099494037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6827","nonce":1099494037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6831","nonce":1099494037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6829","nonce":1099494037}]},"public_addr":"172.21.15.71:6825/1099494037","cluster_addr":"172.21.15.71:6827/1099494037","heartbeat_back_addr":"172.21.15.71:6831/1099494037","heartbeat_front_addr":"172.21.15.71:6829/1099494037","state":["exists","up"]},{"osd":11,"uuid":"1d178378-b9f5-40f6-99ef-3d3d445da640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6825","nonce":2941055364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6827","nonce":2941055364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6831","nonce":2941055364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6829","nonce":2941055364}]},"public_addr":"172.21.15.151:6825/2941055364","cluster_addr":"172.21.15.151:6827/2941055364","heartbeat_back_addr":"172.21.15.151:6831/2941055364","heartbeat_front_addr":"172.21.15.151:6829/2941055364","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:02.269586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:27.819386+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:51.552277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:25.052674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:51.873529+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:17.127867+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:51.588249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:18.381450+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:44.140534+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:17.634362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:45.469340+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:45:12.003433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3533611957":"2024-03-29T17:38:07.505611+0000","172.21.15.45:0/3625922430":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6801/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/482610468":"2024-03-29T17:37:32.424229+0000","172.21.15.45:6801/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3649804714":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/1198364079":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3211303958":"2024-03-29T17:38:07.505611+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T17:45:18.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:18.070+0000 7f4e23800700 1 -- 172.21.15.45:0/523853992 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4e040a2890 msgr2=0x7f4e040a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:18.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:18.070+0000 7f4e23800700 1 --2- 172.21.15.45:0/523853992 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4e040a2890 0x7f4e040a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c0795c0 tx=0x7f4e08009f80 comp rx=0 tx=0).stop 2024-03-28T17:45:18.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:18.070+0000 7f4e23800700 1 -- 172.21.15.45:0/523853992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e1c0766c0 msgr2=0x7f4e1c19f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:18.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:18.070+0000 7f4e23800700 1 --2- 172.21.15.45:0/523853992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e1c0766c0 0x7f4e1c19f040 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4e1000ef10 tx=0x7f4e1000c3a0 comp rx=0 tx=0).stop 2024-03-28T17:45:18.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:18.070+0000 7f4e23800700 1 -- 172.21.15.45:0/523853992 shutdown_connections 2024-03-28T17:45:18.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:18.070+0000 7f4e23800700 1 --2- 172.21.15.45:0/523853992 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4e040a2890 0x7f4e040a4d50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:18.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:18.070+0000 7f4e23800700 1 --2- 172.21.15.45:0/523853992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e1c113040 0x7f4e1c1a38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:18.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:18.070+0000 7f4e23800700 1 --2- 172.21.15.45:0/523853992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e1c0766c0 0x7f4e1c19f040 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:18.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:18.070+0000 7f4e23800700 1 --2- 172.21.15.45:0/523853992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e1c076fe0 0x7f4e1c19f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:18.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:18.070+0000 7f4e23800700 1 -- 172.21.15.45:0/523853992 >> 172.21.15.45:0/523853992 conn(0x7f4e1c0ffe70 msgr2=0x7f4e1c102260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:18.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:18.071+0000 7f4e23800700 1 -- 172.21.15.45:0/523853992 shutdown_connections 2024-03-28T17:45:18.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:18.071+0000 7f4e23800700 1 -- 172.21.15.45:0/523853992 wait complete. 2024-03-28T17:45:18.820 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T17:40:54.208375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T17:45:18.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-28T17:45:19.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:18 smithi071 ceph-mon[36597]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:19.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:18 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/523853992' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:45:19.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:18 smithi151 ceph-mon[36481]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:19.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:18 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/523853992' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:45:19.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:18 smithi045 ceph-mon[31773]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:19.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:18 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/523853992' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:45:19.276 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:45:20.971 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:20 smithi045 ceph-mon[31773]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:21.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:20 smithi071 ceph-mon[36597]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:21.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:20 smithi151 ceph-mon[36481]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.527+0000 7f53b6a00700 1 -- 172.21.15.45:0/3436276537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53b0102ba0 msgr2=0x7f53b010efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.527+0000 7f53b6a00700 1 --2- 172.21.15.45:0/3436276537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53b0102ba0 0x7f53b010efe0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f53a800b0d0 tx=0x7f53a800b3e0 comp rx=0 tx=0).stop 2024-03-28T17:45:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.527+0000 7f53b6a00700 1 -- 172.21.15.45:0/3436276537 shutdown_connections 2024-03-28T17:45:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.527+0000 7f53b6a00700 1 --2- 172.21.15.45:0/3436276537 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53b01081c0 0x7f53b01085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.527+0000 7f53b6a00700 1 --2- 172.21.15.45:0/3436276537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53b0102ba0 0x7f53b010efe0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.527+0000 7f53b6a00700 1 --2- 172.21.15.45:0/3436276537 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53b01021e0 0x7f53b0102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.527+0000 7f53b6a00700 1 -- 172.21.15.45:0/3436276537 >> 172.21.15.45:0/3436276537 conn(0x7f53b00792c0 msgr2=0x7f53b00796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.527+0000 7f53b6a00700 1 -- 172.21.15.45:0/3436276537 shutdown_connections 2024-03-28T17:45:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.527+0000 7f53b6a00700 1 -- 172.21.15.45:0/3436276537 wait complete. 2024-03-28T17:45:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.528+0000 7f53b6a00700 1 Processor -- start 2024-03-28T17:45:21.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.528+0000 7f53b6a00700 1 -- start start 2024-03-28T17:45:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.529+0000 7f53b6a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53b01021e0 0x7f53b0074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.529+0000 7f53b6a00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53b0102ba0 0x7f53b00749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.529+0000 7f53b6a00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53b01081c0 0x7f53b0074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.529+0000 7f53b6a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b0112460 con 0x7f53b01021e0 2024-03-28T17:45:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.529+0000 7f53b6a00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b01122e0 con 0x7f53b0102ba0 2024-03-28T17:45:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.529+0000 7f53b6a00700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b01125e0 con 0x7f53b01081c0 2024-03-28T17:45:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.529+0000 7f53af400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53b0102ba0 0x7f53b00749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.529+0000 7f53afe00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53b01021e0 0x7f53b0074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.529+0000 7f53af400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53b0102ba0 0x7f53b00749d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53578/0 (socket says 172.21.15.45:53578) 2024-03-28T17:45:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.529+0000 7f53afe00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53b01021e0 0x7f53b0074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42332/0 (socket says 172.21.15.45:42332) 2024-03-28T17:45:21.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.529+0000 7f53af400700 1 -- 172.21.15.45:0/1817362680 learned_addr learned my addr 172.21.15.45:0/1817362680 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:45:21.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.530+0000 7f53b4a00700 1 --2- 172.21.15.45:0/1817362680 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53b01081c0 0x7f53b0074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:21.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.530+0000 7f53af400700 1 -- 172.21.15.45:0/1817362680 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53b01081c0 msgr2=0x7f53b0074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:21.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.530+0000 7f53af400700 1 --2- 172.21.15.45:0/1817362680 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53b01081c0 0x7f53b0074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:21.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.530+0000 7f53af400700 1 -- 172.21.15.45:0/1817362680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53b01021e0 msgr2=0x7f53b0074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:21.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.530+0000 7f53af400700 1 --2- 172.21.15.45:0/1817362680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53b01021e0 0x7f53b0074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:21.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.530+0000 7f53af400700 1 -- 172.21.15.45:0/1817362680 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b01a6d80 con 0x7f53b0102ba0 2024-03-28T17:45:21.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.530+0000 7f53afe00700 1 --2- 172.21.15.45:0/1817362680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53b01021e0 0x7f53b0074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:45:21.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.530+0000 7f53af400700 1 --2- 172.21.15.45:0/1817362680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53b0102ba0 0x7f53b00749d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f539c00e9a0 tx=0x7f539c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:21.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.531+0000 7f53acc00700 1 -- 172.21.15.45:0/1817362680 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c00ca10 con 0x7f53b0102ba0 2024-03-28T17:45:21.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.531+0000 7f53acc00700 1 -- 172.21.15.45:0/1817362680 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f539c00cbb0 con 0x7f53b0102ba0 2024-03-28T17:45:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.531+0000 7f53b6a00700 1 -- 172.21.15.45:0/1817362680 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b01a7070 con 0x7f53b0102ba0 2024-03-28T17:45:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.531+0000 7f53acc00700 1 -- 172.21.15.45:0/1817362680 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c018680 con 0x7f53b0102ba0 2024-03-28T17:45:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.531+0000 7f53b6a00700 1 -- 172.21.15.45:0/1817362680 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b01a7600 con 0x7f53b0102ba0 2024-03-28T17:45:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.532+0000 7f53b6a00700 1 -- 172.21.15.45:0/1817362680 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b0068e50 con 0x7f53b0102ba0 2024-03-28T17:45:21.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.535+0000 7f53acc00700 1 -- 172.21.15.45:0/1817362680 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f539c010430 con 0x7f53b0102ba0 2024-03-28T17:45:21.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.536+0000 7f53acc00700 1 --2- 172.21.15.45:0/1817362680 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f53940a2910 0x7f53940a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:21.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.536+0000 7f53acc00700 1 -- 172.21.15.45:0/1817362680 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f539c0cec90 con 0x7f53b0102ba0 2024-03-28T17:45:21.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.536+0000 7f53afe00700 1 --2- 172.21.15.45:0/1817362680 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f53940a2910 0x7f53940a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:21.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.537+0000 7f53afe00700 1 --2- 172.21.15.45:0/1817362680 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f53940a2910 0x7f53940a4dd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009fd0 tx=0x7f53a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:21.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.538+0000 7f53acc00700 1 -- 172.21.15.45:0/1817362680 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f539c09c130 con 0x7f53b0102ba0 2024-03-28T17:45:21.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.764+0000 7f53b6a00700 1 -- 172.21.15.45:0/1817362680 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f53b002da30 con 0x7f53b0102ba0 2024-03-28T17:45:21.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.765+0000 7f53acc00700 1 -- 172.21.15.45:0/1817362680 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f539c09ba30 con 0x7f53b0102ba0 2024-03-28T17:45:21.766 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-28T17:45:21.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.768+0000 7f53b6a00700 1 -- 172.21.15.45:0/1817362680 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f53940a2910 msgr2=0x7f53940a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:21.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.768+0000 7f53b6a00700 1 --2- 172.21.15.45:0/1817362680 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f53940a2910 0x7f53940a4dd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009fd0 tx=0x7f53a0009260 comp rx=0 tx=0).stop 2024-03-28T17:45:21.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.768+0000 7f53b6a00700 1 -- 172.21.15.45:0/1817362680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53b0102ba0 msgr2=0x7f53b00749d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:21.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.768+0000 7f53b6a00700 1 --2- 172.21.15.45:0/1817362680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53b0102ba0 0x7f53b00749d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f539c00e9a0 tx=0x7f539c00ecb0 comp rx=0 tx=0).stop 2024-03-28T17:45:21.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.769+0000 7f53b6a00700 1 -- 172.21.15.45:0/1817362680 shutdown_connections 2024-03-28T17:45:21.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.769+0000 7f53b6a00700 1 --2- 172.21.15.45:0/1817362680 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f53940a2910 0x7f53940a4dd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:21.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.769+0000 7f53b6a00700 1 --2- 172.21.15.45:0/1817362680 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53b01081c0 0x7f53b0074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:21.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.769+0000 7f53b6a00700 1 --2- 172.21.15.45:0/1817362680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53b01021e0 0x7f53b0074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:21.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.769+0000 7f53b6a00700 1 --2- 172.21.15.45:0/1817362680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53b0102ba0 0x7f53b00749d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:21.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.769+0000 7f53b6a00700 1 -- 172.21.15.45:0/1817362680 >> 172.21.15.45:0/1817362680 conn(0x7f53b00792c0 msgr2=0x7f53b0100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:21.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.769+0000 7f53b6a00700 1 -- 172.21.15.45:0/1817362680 shutdown_connections 2024-03-28T17:45:21.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:21.769+0000 7f53b6a00700 1 -- 172.21.15.45:0/1817362680 wait complete. 2024-03-28T17:45:22.629 INFO:tasks.cephadm:Adding mds.a on smithi045 2024-03-28T17:45:22.629 INFO:tasks.cephadm:Adding mds.d on smithi045 2024-03-28T17:45:22.630 INFO:tasks.cephadm:Adding mds.g on smithi045 2024-03-28T17:45:22.630 INFO:tasks.cephadm:Adding mds.j on smithi045 2024-03-28T17:45:22.630 INFO:tasks.cephadm:Adding mds.b on smithi071 2024-03-28T17:45:22.630 INFO:tasks.cephadm:Adding mds.e on smithi071 2024-03-28T17:45:22.630 INFO:tasks.cephadm:Adding mds.h on smithi071 2024-03-28T17:45:22.630 INFO:tasks.cephadm:Adding mds.k on smithi071 2024-03-28T17:45:22.630 INFO:tasks.cephadm:Adding mds.c on smithi151 2024-03-28T17:45:22.630 INFO:tasks.cephadm:Adding mds.f on smithi151 2024-03-28T17:45:22.630 INFO:tasks.cephadm:Adding mds.i on smithi151 2024-03-28T17:45:22.630 INFO:tasks.cephadm:Adding mds.l on smithi151 2024-03-28T17:45:22.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph orch apply mds all '12;smithi045=a;smithi045=d;smithi045=g;smithi045=j;smithi071=b;smithi071=e;smithi071=h;smithi071=k;smithi151=c;smithi151=f;smithi151=i;smithi151=l' 2024-03-28T17:45:22.892 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T17:45:22.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:22 smithi045 ceph-mon[31773]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:22.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:22 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1817362680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:45:23.015 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:22 smithi151 ceph-mon[36481]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:23.015 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:22 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1817362680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:45:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:22 smithi071 ceph-mon[36597]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:22 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1817362680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:45:24.619 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.615+0000 7f74d6661700 1 -- 172.21.15.151:0/303489761 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74d0102cd0 msgr2=0x7f74d0109580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:24.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.615+0000 7f74d6661700 1 --2- 172.21.15.151:0/303489761 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74d0102cd0 0x7f74d0109580 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f74c800b0a0 tx=0x7f74c800b3b0 comp rx=0 tx=0).stop 2024-03-28T17:45:24.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.615+0000 7f74d6661700 1 -- 172.21.15.151:0/303489761 shutdown_connections 2024-03-28T17:45:24.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.615+0000 7f74d6661700 1 --2- 172.21.15.151:0/303489761 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74d0102cd0 0x7f74d0109580 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:24.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.615+0000 7f74d6661700 1 --2- 172.21.15.151:0/303489761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74d0101130 0x7f74d0101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:24.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.615+0000 7f74d6661700 1 --2- 172.21.15.151:0/303489761 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74d0102330 0x7f74d0102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:24.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.615+0000 7f74d6661700 1 -- 172.21.15.151:0/303489761 >> 172.21.15.151:0/303489761 conn(0x7f74d00fc6f0 msgr2=0x7f74d00feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:24.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.615+0000 7f74d6661700 1 -- 172.21.15.151:0/303489761 shutdown_connections 2024-03-28T17:45:24.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.615+0000 7f74d6661700 1 -- 172.21.15.151:0/303489761 wait complete. 2024-03-28T17:45:24.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.616+0000 7f74d6661700 1 Processor -- start 2024-03-28T17:45:24.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.616+0000 7f74d6661700 1 -- start start 2024-03-28T17:45:24.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.616+0000 7f74d6661700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74d0101130 0x7f74d01050c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:24.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.616+0000 7f74d6661700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74d0102330 0x7f74d0103710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:24.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.616+0000 7f74d6661700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74d0102cd0 0x7f74d0103c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:24.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.616+0000 7f74d6661700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d010cd30 con 0x7f74d0102cd0 2024-03-28T17:45:24.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.616+0000 7f74d6661700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d010cbb0 con 0x7f74d0101130 2024-03-28T17:45:24.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.616+0000 7f74d6661700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d010ceb0 con 0x7f74d0102330 2024-03-28T17:45:24.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.616+0000 7f74cf7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74d0102330 0x7f74d0103710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:24.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.616+0000 7f74cf7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74d0102330 0x7f74d0103710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:34578/0 (socket says 172.21.15.151:34578) 2024-03-28T17:45:24.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.616+0000 7f74cf7fe700 1 -- 172.21.15.151:0/4052330228 learned_addr learned my addr 172.21.15.151:0/4052330228 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-03-28T17:45:24.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.616+0000 7f74cffff700 1 --2- 172.21.15.151:0/4052330228 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74d0101130 0x7f74d01050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:24.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.617+0000 7f74cf7fe700 1 -- 172.21.15.151:0/4052330228 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74d0101130 msgr2=0x7f74d01050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:24.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.617+0000 7f74cf7fe700 1 --2- 172.21.15.151:0/4052330228 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74d0101130 0x7f74d01050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:24.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.617+0000 7f74cf7fe700 1 -- 172.21.15.151:0/4052330228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74d0102cd0 msgr2=0x7f74d0103c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:24.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.617+0000 7f74cf7fe700 1 --2- 172.21.15.151:0/4052330228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74d0102cd0 0x7f74d0103c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:24.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.617+0000 7f74cf7fe700 1 -- 172.21.15.151:0/4052330228 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d0104500 con 0x7f74d0102330 2024-03-28T17:45:24.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.617+0000 7f74cf7fe700 1 --2- 172.21.15.151:0/4052330228 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74d0102330 0x7f74d0103710 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f74c400c9b0 tx=0x7f74c400ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:24.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.617+0000 7f74cd7fa700 1 -- 172.21.15.151:0/4052330228 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c4007b00 con 0x7f74d0102330 2024-03-28T17:45:24.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.617+0000 7f74d6661700 1 -- 172.21.15.151:0/4052330228 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d01a2510 con 0x7f74d0102330 2024-03-28T17:45:24.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.617+0000 7f74cd7fa700 1 -- 172.21.15.151:0/4052330228 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f74c4007ca0 con 0x7f74d0102330 2024-03-28T17:45:24.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.617+0000 7f74cd7fa700 1 -- 172.21.15.151:0/4052330228 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c4018730 con 0x7f74d0102330 2024-03-28T17:45:24.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.617+0000 7f74d6661700 1 -- 172.21.15.151:0/4052330228 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d01a2ad0 con 0x7f74d0102330 2024-03-28T17:45:24.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.618+0000 7f74d6661700 1 -- 172.21.15.151:0/4052330228 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d0068e50 con 0x7f74d0102330 2024-03-28T17:45:24.622 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.619+0000 7f74cd7fa700 1 -- 172.21.15.151:0/4052330228 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f74c40188d0 con 0x7f74d0102330 2024-03-28T17:45:24.623 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.620+0000 7f74cd7fa700 1 --2- 172.21.15.151:0/4052330228 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f74b80a2890 0x7f74b80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:24.623 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.620+0000 7f74cd7fa700 1 -- 172.21.15.151:0/4052330228 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f74c40ce850 con 0x7f74d0102330 2024-03-28T17:45:24.623 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.620+0000 7f74cffff700 1 --2- 172.21.15.151:0/4052330228 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f74b80a2890 0x7f74b80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:24.623 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.621+0000 7f74cffff700 1 --2- 172.21.15.151:0/4052330228 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f74b80a2890 0x7f74b80a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f74d0102190 tx=0x7f74c0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:24.624 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.624+0000 7f74cd7fa700 1 -- 172.21.15.151:0/4052330228 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f74c4014030 con 0x7f74d0102330 2024-03-28T17:45:24.829 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.829+0000 7f74d6661700 1 -- 172.21.15.151:0/4052330228 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi045=a;smithi045=d;smithi045=g;smithi045=j;smithi071=b;smithi071=e;smithi071=h;smithi071=k;smithi151=c;smithi151=f;smithi151=i;smithi151=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f74d00fe840 con 0x7f74b80a2890 2024-03-28T17:45:24.846 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.845+0000 7f74cd7fa700 1 -- 172.21.15.151:0/4052330228 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f74d00fe840 con 0x7f74b80a2890 2024-03-28T17:45:24.846 INFO:teuthology.orchestra.run.smithi151.stdout:Scheduled mds.all update... 2024-03-28T17:45:24.848 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.848+0000 7f74d6661700 1 -- 172.21.15.151:0/4052330228 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f74b80a2890 msgr2=0x7f74b80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:24.848 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.848+0000 7f74d6661700 1 --2- 172.21.15.151:0/4052330228 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f74b80a2890 0x7f74b80a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f74d0102190 tx=0x7f74c0009190 comp rx=0 tx=0).stop 2024-03-28T17:45:24.848 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.848+0000 7f74d6661700 1 -- 172.21.15.151:0/4052330228 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74d0102330 msgr2=0x7f74d0103710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:24.848 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.848+0000 7f74d6661700 1 --2- 172.21.15.151:0/4052330228 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74d0102330 0x7f74d0103710 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f74c400c9b0 tx=0x7f74c400ccc0 comp rx=0 tx=0).stop 2024-03-28T17:45:24.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.848+0000 7f74d6661700 1 -- 172.21.15.151:0/4052330228 shutdown_connections 2024-03-28T17:45:24.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.848+0000 7f74d6661700 1 --2- 172.21.15.151:0/4052330228 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f74b80a2890 0x7f74b80a4d50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:24.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.848+0000 7f74d6661700 1 --2- 172.21.15.151:0/4052330228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74d0102cd0 0x7f74d0103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:24.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.848+0000 7f74d6661700 1 --2- 172.21.15.151:0/4052330228 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74d0101130 0x7f74d01050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:24.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.848+0000 7f74d6661700 1 --2- 172.21.15.151:0/4052330228 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74d0102330 0x7f74d0103710 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:24.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.848+0000 7f74d6661700 1 -- 172.21.15.151:0/4052330228 >> 172.21.15.151:0/4052330228 conn(0x7f74d00fc6f0 msgr2=0x7f74d00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:24.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.848+0000 7f74d6661700 1 -- 172.21.15.151:0/4052330228 shutdown_connections 2024-03-28T17:45:24.849 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-28T17:45:24.848+0000 7f74d6661700 1 -- 172.21.15.151:0/4052330228 wait complete. 2024-03-28T17:45:25.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:24 smithi071 ceph-mon[36597]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:25.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:24 smithi151 ceph-mon[36481]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:25.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:24 smithi045 ceph-mon[31773]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:25.517 DEBUG:teuthology.orchestra.run.smithi045:mds.a> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service 2024-03-28T17:45:25.525 DEBUG:teuthology.orchestra.run.smithi045:mds.d> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service 2024-03-28T17:45:25.571 DEBUG:teuthology.orchestra.run.smithi045:mds.g> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service 2024-03-28T17:45:25.617 DEBUG:teuthology.orchestra.run.smithi045:mds.j> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service 2024-03-28T17:45:25.662 DEBUG:teuthology.orchestra.run.smithi071:mds.b> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.b.service 2024-03-28T17:45:25.665 DEBUG:teuthology.orchestra.run.smithi071:mds.e> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.e.service 2024-03-28T17:45:25.708 DEBUG:teuthology.orchestra.run.smithi071:mds.h> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.h.service 2024-03-28T17:45:25.751 DEBUG:teuthology.orchestra.run.smithi071:mds.k> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.k.service 2024-03-28T17:45:25.794 DEBUG:teuthology.orchestra.run.smithi151:mds.c> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.c.service 2024-03-28T17:45:25.796 DEBUG:teuthology.orchestra.run.smithi151:mds.f> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.f.service 2024-03-28T17:45:25.839 DEBUG:teuthology.orchestra.run.smithi151:mds.i> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.i.service 2024-03-28T17:45:25.882 DEBUG:teuthology.orchestra.run.smithi151:mds.l> sudo journalctl -f -n 0 -u ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.l.service 2024-03-28T17:45:25.925 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-28T17:45:25.925 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T17:45:26.012 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:25 smithi045 ceph-mon[31773]: from='client.24409 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi045=a;smithi045=d;smithi045=g;smithi045=j;smithi071=b;smithi071=e;smithi071=h;smithi071=k;smithi151=c;smithi151=f;smithi151=i;smithi151=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:45:26.012 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:25 smithi045 ceph-mon[31773]: Saving service mds.all spec with placement smithi045=a;smithi045=d;smithi045=g;smithi045=j;smithi071=b;smithi071=e;smithi071=h;smithi071=k;smithi151=c;smithi151=f;smithi151=i;smithi151=l;count:12 2024-03-28T17:45:26.012 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:26.013 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:25 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:45:26.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:25 smithi071 ceph-mon[36597]: from='client.24409 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi045=a;smithi045=d;smithi045=g;smithi045=j;smithi071=b;smithi071=e;smithi071=h;smithi071=k;smithi151=c;smithi151=f;smithi151=i;smithi151=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:45:26.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:25 smithi071 ceph-mon[36597]: Saving service mds.all spec with placement smithi045=a;smithi045=d;smithi045=g;smithi045=j;smithi071=b;smithi071=e;smithi071=h;smithi071=k;smithi151=c;smithi151=f;smithi151=i;smithi151=l;count:12 2024-03-28T17:45:26.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:26.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:25 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:45:26.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:25 smithi151 ceph-mon[36481]: from='client.24409 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi045=a;smithi045=d;smithi045=g;smithi045=j;smithi071=b;smithi071=e;smithi071=h;smithi071=k;smithi151=c;smithi151=f;smithi151=i;smithi151=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:45:26.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:25 smithi151 ceph-mon[36481]: Saving service mds.all spec with placement smithi045=a;smithi045=d;smithi045=g;smithi045=j;smithi071=b;smithi071=e;smithi071=h;smithi071=k;smithi151=c;smithi151=f;smithi151=i;smithi151=l;count:12 2024-03-28T17:45:26.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:26.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:25 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:45:26.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.151+0000 7fcaeca00640 1 -- 172.21.15.45:0/2704627615 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcae81026f0 msgr2=0x7fcae8102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:26.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.151+0000 7fcaeca00640 1 --2- 172.21.15.45:0/2704627615 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcae81026f0 0x7fcae8102b70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcad4009860 tx=0x7fcad402bb90 comp rx=0 tx=0).stop 2024-03-28T17:45:26.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.151+0000 7fcaeca00640 1 -- 172.21.15.45:0/2704627615 shutdown_connections 2024-03-28T17:45:26.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.151+0000 7fcaeca00640 1 --2- 172.21.15.45:0/2704627615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcae81030b0 0x7fcae8109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:26.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.151+0000 7fcaeca00640 1 --2- 172.21.15.45:0/2704627615 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcae81026f0 0x7fcae8102b70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:26.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.151+0000 7fcaeca00640 1 --2- 172.21.15.45:0/2704627615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcae8101500 0x7fcae8101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:26.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.151+0000 7fcaeca00640 1 -- 172.21.15.45:0/2704627615 >> 172.21.15.45:0/2704627615 conn(0x7fcae80fca70 msgr2=0x7fcae80feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:26.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.151+0000 7fcaeca00640 1 -- 172.21.15.45:0/2704627615 shutdown_connections 2024-03-28T17:45:26.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.151+0000 7fcaeca00640 1 -- 172.21.15.45:0/2704627615 wait complete. 2024-03-28T17:45:26.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.152+0000 7fcaeca00640 1 Processor -- start 2024-03-28T17:45:26.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.152+0000 7fcaeca00640 1 -- start start 2024-03-28T17:45:26.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.153+0000 7fcaeca00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcae8101500 0x7fcae81999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:26.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.153+0000 7fcaeca00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcae81026f0 0x7fcae8199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:26.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.153+0000 7fcaeca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcae81030b0 0x7fcae81a0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:26.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.153+0000 7fcaeca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae810d190 con 0x7fcae81030b0 2024-03-28T17:45:26.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.153+0000 7fcaeca00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae810d010 con 0x7fcae81026f0 2024-03-28T17:45:26.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.153+0000 7fcaeca00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae810d310 con 0x7fcae8101500 2024-03-28T17:45:26.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.153+0000 7fcae7400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcae8101500 0x7fcae81999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:26.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.153+0000 7fcae7400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcae8101500 0x7fcae81999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:44238/0 (socket says 172.21.15.45:44238) 2024-03-28T17:45:26.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.153+0000 7fcae7400640 1 -- 172.21.15.45:0/2395688626 learned_addr learned my addr 172.21.15.45:0/2395688626 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:45:26.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.153+0000 7fcae6a00640 1 --2- 172.21.15.45:0/2395688626 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcae81026f0 0x7fcae8199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:26.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.153+0000 7fcae7e00640 1 --2- 172.21.15.45:0/2395688626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcae81030b0 0x7fcae81a0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:26.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.153+0000 7fcae7400640 1 -- 172.21.15.45:0/2395688626 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcae81026f0 msgr2=0x7fcae8199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:26.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.153+0000 7fcae7400640 1 --2- 172.21.15.45:0/2395688626 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcae81026f0 0x7fcae8199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:26.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.153+0000 7fcae7400640 1 -- 172.21.15.45:0/2395688626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcae81030b0 msgr2=0x7fcae81a0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:26.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.153+0000 7fcae7400640 1 --2- 172.21.15.45:0/2395688626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcae81030b0 0x7fcae81a0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:26.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.153+0000 7fcae7400640 1 -- 172.21.15.45:0/2395688626 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae81a15c0 con 0x7fcae8101500 2024-03-28T17:45:26.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.153+0000 7fcae7e00640 1 --2- 172.21.15.45:0/2395688626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcae81030b0 0x7fcae81a0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:45:26.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.154+0000 7fcae7400640 1 --2- 172.21.15.45:0/2395688626 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcae8101500 0x7fcae81999a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcad800e9a0 tx=0x7fcad800ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:26.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.154+0000 7fcadfe00640 1 -- 172.21.15.45:0/2395688626 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad800cd80 con 0x7fcae8101500 2024-03-28T17:45:26.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.154+0000 7fcadfe00640 1 -- 172.21.15.45:0/2395688626 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcad8010910 con 0x7fcae8101500 2024-03-28T17:45:26.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.154+0000 7fcaeca00640 1 -- 172.21.15.45:0/2395688626 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcae81a18b0 con 0x7fcae8101500 2024-03-28T17:45:26.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.154+0000 7fcadfe00640 1 -- 172.21.15.45:0/2395688626 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad8018a30 con 0x7fcae8101500 2024-03-28T17:45:26.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.154+0000 7fcaeca00640 1 -- 172.21.15.45:0/2395688626 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcae81a1e20 con 0x7fcae8101500 2024-03-28T17:45:26.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.156+0000 7fcaeca00640 1 -- 172.21.15.45:0/2395688626 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcae81a20b0 con 0x7fcae8101500 2024-03-28T17:45:26.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.158+0000 7fcadfe00640 1 -- 172.21.15.45:0/2395688626 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcad8010ac0 con 0x7fcae8101500 2024-03-28T17:45:26.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.159+0000 7fcadfe00640 1 --2- 172.21.15.45:0/2395688626 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcacc0a2880 0x7fcacc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:26.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.159+0000 7fcadfe00640 1 -- 172.21.15.45:0/2395688626 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcad80ce9d0 con 0x7fcae8101500 2024-03-28T17:45:26.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.160+0000 7fcae6a00640 1 --2- 172.21.15.45:0/2395688626 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcacc0a2880 0x7fcacc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:26.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.160+0000 7fcae6a00640 1 --2- 172.21.15.45:0/2395688626 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcacc0a2880 0x7fcacc0a4d40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcad402c1a0 tx=0x7fcad4034ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:26.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.163+0000 7fcadfe00640 1 -- 172.21.15.45:0/2395688626 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcad8053740 con 0x7fcae8101500 2024-03-28T17:45:26.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.367+0000 7fcaeca00640 1 -- 172.21.15.45:0/2395688626 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcae81a2760 con 0x7fcae8101500 2024-03-28T17:45:26.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.369+0000 7fcadfe00640 1 -- 172.21.15.45:0/2395688626 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18445 (secure 0 0 0) 0x7fcad809be70 con 0x7fcae8101500 2024-03-28T17:45:26.369 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:45:26.370 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":81,"fsid":"9f4888ec-ed29-11ee-95d0-87774f69a715","created":"2024-03-28T17:36:51.623185+0000","modified":"2024-03-28T17:45:15.608581+0000","last_up_change":"2024-03-28T17:45:14.043432+0000","last_in_change":"2024-03-28T17:44:53.654489+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T17:40:54.208375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"74c56bc2-4169-4919-afd4-2ebd236f23e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6803","nonce":3795754023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6805","nonce":3795754023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6809","nonce":3795754023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6807","nonce":3795754023}]},"public_addr":"172.21.15.45:6803/3795754023","cluster_addr":"172.21.15.45:6805/3795754023","heartbeat_back_addr":"172.21.15.45:6809/3795754023","heartbeat_front_addr":"172.21.15.45:6807/3795754023","state":["exists","up"]},{"osd":1,"uuid":"78dcc21b-75b2-453d-b690-17b55b8ed1a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6801","nonce":2014529906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6803","nonce":2014529906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6807","nonce":2014529906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6805","nonce":2014529906}]},"public_addr":"172.21.15.71:6801/2014529906","cluster_addr":"172.21.15.71:6803/2014529906","heartbeat_back_addr":"172.21.15.71:6807/2014529906","heartbeat_front_addr":"172.21.15.71:6805/2014529906","state":["exists","up"]},{"osd":2,"uuid":"ca9f3f36-b71e-4962-a94f-cedf2f12eb7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6801","nonce":1850933775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6803","nonce":1850933775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6807","nonce":1850933775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6805","nonce":1850933775}]},"public_addr":"172.21.15.151:6801/1850933775","cluster_addr":"172.21.15.151:6803/1850933775","heartbeat_back_addr":"172.21.15.151:6807/1850933775","heartbeat_front_addr":"172.21.15.151:6805/1850933775","state":["exists","up"]},{"osd":3,"uuid":"d3f21f01-2973-413e-8daa-9974b8a44af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6811","nonce":3021569975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6813","nonce":3021569975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6817","nonce":3021569975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6815","nonce":3021569975}]},"public_addr":"172.21.15.45:6811/3021569975","cluster_addr":"172.21.15.45:6813/3021569975","heartbeat_back_addr":"172.21.15.45:6817/3021569975","heartbeat_front_addr":"172.21.15.45:6815/3021569975","state":["exists","up"]},{"osd":4,"uuid":"4a336f42-0605-492b-ad88-56cb02acdcf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6809","nonce":2069578333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6811","nonce":2069578333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6815","nonce":2069578333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6813","nonce":2069578333}]},"public_addr":"172.21.15.71:6809/2069578333","cluster_addr":"172.21.15.71:6811/2069578333","heartbeat_back_addr":"172.21.15.71:6815/2069578333","heartbeat_front_addr":"172.21.15.71:6813/2069578333","state":["exists","up"]},{"osd":5,"uuid":"c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6809","nonce":2629217227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6811","nonce":2629217227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6815","nonce":2629217227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6813","nonce":2629217227}]},"public_addr":"172.21.15.151:6809/2629217227","cluster_addr":"172.21.15.151:6811/2629217227","heartbeat_back_addr":"172.21.15.151:6815/2629217227","heartbeat_front_addr":"172.21.15.151:6813/2629217227","state":["exists","up"]},{"osd":6,"uuid":"58a6c5bd-a8b7-4892-b346-bcd7994bcb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6819","nonce":2077907039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6821","nonce":2077907039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6825","nonce":2077907039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6823","nonce":2077907039}]},"public_addr":"172.21.15.45:6819/2077907039","cluster_addr":"172.21.15.45:6821/2077907039","heartbeat_back_addr":"172.21.15.45:6825/2077907039","heartbeat_front_addr":"172.21.15.45:6823/2077907039","state":["exists","up"]},{"osd":7,"uuid":"f7a45d52-807b-4407-bb04-787b2a52a0ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6817","nonce":2000211251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6819","nonce":2000211251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6823","nonce":2000211251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6821","nonce":2000211251}]},"public_addr":"172.21.15.71:6817/2000211251","cluster_addr":"172.21.15.71:6819/2000211251","heartbeat_back_addr":"172.21.15.71:6823/2000211251","heartbeat_front_addr":"172.21.15.71:6821/2000211251","state":["exists","up"]},{"osd":8,"uuid":"88abcb0f-f189-49be-8315-c700c77369f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6817","nonce":3665460137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6819","nonce":3665460137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6823","nonce":3665460137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6821","nonce":3665460137}]},"public_addr":"172.21.15.151:6817/3665460137","cluster_addr":"172.21.15.151:6819/3665460137","heartbeat_back_addr":"172.21.15.151:6823/3665460137","heartbeat_front_addr":"172.21.15.151:6821/3665460137","state":["exists","up"]},{"osd":9,"uuid":"ed3fc096-961e-4392-abc2-c5aa4d050117","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6827","nonce":2122322684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6829","nonce":2122322684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6833","nonce":2122322684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6831","nonce":2122322684}]},"public_addr":"172.21.15.45:6827/2122322684","cluster_addr":"172.21.15.45:6829/2122322684","heartbeat_back_addr":"172.21.15.45:6833/2122322684","heartbeat_front_addr":"172.21.15.45:6831/2122322684","state":["exists","up"]},{"osd":10,"uuid":"ba94762b-bc5c-4c97-9cc1-17e5651c14b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6825","nonce":1099494037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6827","nonce":1099494037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6831","nonce":1099494037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6829","nonce":1099494037}]},"public_addr":"172.21.15.71:6825/1099494037","cluster_addr":"172.21.15.71:6827/1099494037","heartbeat_back_addr":"172.21.15.71:6831/1099494037","heartbeat_front_addr":"172.21.15.71:6829/1099494037","state":["exists","up"]},{"osd":11,"uuid":"1d178378-b9f5-40f6-99ef-3d3d445da640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6825","nonce":2941055364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6827","nonce":2941055364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6831","nonce":2941055364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6829","nonce":2941055364}]},"public_addr":"172.21.15.151:6825/2941055364","cluster_addr":"172.21.15.151:6827/2941055364","heartbeat_back_addr":"172.21.15.151:6831/2941055364","heartbeat_front_addr":"172.21.15.151:6829/2941055364","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:02.269586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:27.819386+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:51.552277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:25.052674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:51.873529+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:17.127867+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:51.588249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:18.381450+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:44.140534+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:17.634362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:45.469340+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:45:12.003433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3533611957":"2024-03-29T17:38:07.505611+0000","172.21.15.45:0/3625922430":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6801/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/482610468":"2024-03-29T17:37:32.424229+0000","172.21.15.45:6801/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3649804714":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/1198364079":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3211303958":"2024-03-29T17:38:07.505611+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T17:45:26.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.371+0000 7fcaeca00640 1 -- 172.21.15.45:0/2395688626 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcacc0a2880 msgr2=0x7fcacc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:26.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.371+0000 7fcaeca00640 1 --2- 172.21.15.45:0/2395688626 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcacc0a2880 0x7fcacc0a4d40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcad402c1a0 tx=0x7fcad4034ad0 comp rx=0 tx=0).stop 2024-03-28T17:45:26.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.371+0000 7fcaeca00640 1 -- 172.21.15.45:0/2395688626 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcae8101500 msgr2=0x7fcae81999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:26.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.371+0000 7fcaeca00640 1 --2- 172.21.15.45:0/2395688626 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcae8101500 0x7fcae81999a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcad800e9a0 tx=0x7fcad800ee60 comp rx=0 tx=0).stop 2024-03-28T17:45:26.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.372+0000 7fcaeca00640 1 -- 172.21.15.45:0/2395688626 shutdown_connections 2024-03-28T17:45:26.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.372+0000 7fcaeca00640 1 --2- 172.21.15.45:0/2395688626 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcacc0a2880 0x7fcacc0a4d40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:26.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.372+0000 7fcaeca00640 1 --2- 172.21.15.45:0/2395688626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcae81030b0 0x7fcae81a0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:26.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.372+0000 7fcaeca00640 1 --2- 172.21.15.45:0/2395688626 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcae81026f0 0x7fcae8199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:26.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.372+0000 7fcaeca00640 1 --2- 172.21.15.45:0/2395688626 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcae8101500 0x7fcae81999a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:26.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.372+0000 7fcaeca00640 1 -- 172.21.15.45:0/2395688626 >> 172.21.15.45:0/2395688626 conn(0x7fcae80fca70 msgr2=0x7fcae80738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:26.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.372+0000 7fcaeca00640 1 -- 172.21.15.45:0/2395688626 shutdown_connections 2024-03-28T17:45:26.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.372+0000 7fcaeca00640 1 -- 172.21.15.45:0/2395688626 wait complete. 2024-03-28T17:45:26.393 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T17:40:54.208375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T17:45:26.393 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T17:45:26.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.626+0000 7f5a22200640 1 -- 172.21.15.45:0/2769774408 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a1c1049b0 msgr2=0x7f5a1c104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:26.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.626+0000 7f5a22200640 1 --2- 172.21.15.45:0/2769774408 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a1c1049b0 0x7f5a1c104e30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5a08009900 tx=0x7f5a0802bc30 comp rx=0 tx=0).stop 2024-03-28T17:45:26.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.626+0000 7f5a22200640 1 -- 172.21.15.45:0/2769774408 shutdown_connections 2024-03-28T17:45:26.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.626+0000 7f5a22200640 1 --2- 172.21.15.45:0/2769774408 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a1c105370 0x7f5a1c10bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:26.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.626+0000 7f5a22200640 1 --2- 172.21.15.45:0/2769774408 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a1c1049b0 0x7f5a1c104e30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:26.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.626+0000 7f5a22200640 1 --2- 172.21.15.45:0/2769774408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a1c1037c0 0x7f5a1c103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:26.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.626+0000 7f5a22200640 1 -- 172.21.15.45:0/2769774408 >> 172.21.15.45:0/2769774408 conn(0x7f5a1c0fed50 msgr2=0x7f5a1c101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:26.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.626+0000 7f5a22200640 1 -- 172.21.15.45:0/2769774408 shutdown_connections 2024-03-28T17:45:26.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.626+0000 7f5a22200640 1 -- 172.21.15.45:0/2769774408 wait complete. 2024-03-28T17:45:26.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.628+0000 7f5a22200640 1 Processor -- start 2024-03-28T17:45:26.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.628+0000 7f5a22200640 1 -- start start 2024-03-28T17:45:26.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.628+0000 7f5a22200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a1c1037c0 0x7f5a1c199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:26.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.628+0000 7f5a22200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a1c1049b0 0x7f5a1c19a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:26.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.628+0000 7f5a22200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a1c105370 0x7f5a1c1a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:26.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.628+0000 7f5a22200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c10f490 con 0x7f5a1c1037c0 2024-03-28T17:45:26.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.628+0000 7f5a22200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c10f310 con 0x7f5a1c105370 2024-03-28T17:45:26.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.628+0000 7f5a22200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c10f610 con 0x7f5a1c1049b0 2024-03-28T17:45:26.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.629+0000 7f5a20e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a1c1037c0 0x7f5a1c199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:26.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.629+0000 7f5a1be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a1c1049b0 0x7f5a1c19a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:26.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.629+0000 7f5a20e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a1c1037c0 0x7f5a1c199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55602/0 (socket says 172.21.15.45:55602) 2024-03-28T17:45:26.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.629+0000 7f5a1be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a1c1049b0 0x7f5a1c19a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:44262/0 (socket says 172.21.15.45:44262) 2024-03-28T17:45:26.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.629+0000 7f5a20e00640 1 -- 172.21.15.45:0/3072545488 learned_addr learned my addr 172.21.15.45:0/3072545488 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:45:26.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.629+0000 7f5a21800640 1 --2- 172.21.15.45:0/3072545488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a1c105370 0x7f5a1c1a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:26.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.629+0000 7f5a1be00640 1 -- 172.21.15.45:0/3072545488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a1c105370 msgr2=0x7f5a1c1a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:26.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.629+0000 7f5a1be00640 1 --2- 172.21.15.45:0/3072545488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a1c105370 0x7f5a1c1a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:26.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.629+0000 7f5a1be00640 1 -- 172.21.15.45:0/3072545488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a1c1037c0 msgr2=0x7f5a1c199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:26.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.629+0000 7f5a1be00640 1 --2- 172.21.15.45:0/3072545488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a1c1037c0 0x7f5a1c199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:26.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.629+0000 7f5a1be00640 1 -- 172.21.15.45:0/3072545488 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1c1a1710 con 0x7f5a1c1049b0 2024-03-28T17:45:26.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.629+0000 7f5a1be00640 1 --2- 172.21.15.45:0/3072545488 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a1c1049b0 0x7f5a1c19a030 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5a0802c170 tx=0x7f5a0802c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:26.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.630+0000 7f5a19600640 1 -- 172.21.15.45:0/3072545488 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a08043020 con 0x7f5a1c1049b0 2024-03-28T17:45:26.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.630+0000 7f5a19600640 1 -- 172.21.15.45:0/3072545488 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a080054e0 con 0x7f5a1c1049b0 2024-03-28T17:45:26.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.630+0000 7f5a19600640 1 -- 172.21.15.45:0/3072545488 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a08003c80 con 0x7f5a1c1049b0 2024-03-28T17:45:26.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.630+0000 7f5a22200640 1 -- 172.21.15.45:0/3072545488 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a1c1a19a0 con 0x7f5a1c1049b0 2024-03-28T17:45:26.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.630+0000 7f5a22200640 1 -- 172.21.15.45:0/3072545488 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a1c1a1e70 con 0x7f5a1c1049b0 2024-03-28T17:45:26.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.631+0000 7f5a22200640 1 -- 172.21.15.45:0/3072545488 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a1c10aa80 con 0x7f5a1c1049b0 2024-03-28T17:45:26.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.633+0000 7f5a19600640 1 -- 172.21.15.45:0/3072545488 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5a08046020 con 0x7f5a1c1049b0 2024-03-28T17:45:26.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.635+0000 7f5a19600640 1 --2- 172.21.15.45:0/3072545488 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f5a040a2800 0x7f5a040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:26.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.635+0000 7f5a19600640 1 -- 172.21.15.45:0/3072545488 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5a080f06f0 con 0x7f5a1c1049b0 2024-03-28T17:45:26.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.635+0000 7f5a20e00640 1 --2- 172.21.15.45:0/3072545488 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f5a040a2800 0x7f5a040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:26.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.636+0000 7f5a20e00640 1 --2- 172.21.15.45:0/3072545488 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f5a040a2800 0x7f5a040a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5a10005fd0 tx=0x7f5a1000b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:26.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.639+0000 7f5a19600640 1 -- 172.21.15.45:0/3072545488 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5a0804d020 con 0x7f5a1c1049b0 2024-03-28T17:45:26.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.840+0000 7f5a22200640 1 -- 172.21.15.45:0/3072545488 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5a1c0622a0 con 0x7f5a1c1049b0 2024-03-28T17:45:26.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.841+0000 7f5a19600640 1 -- 172.21.15.45:0/3072545488 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5a080bdb90 con 0x7f5a1c1049b0 2024-03-28T17:45:26.841 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-28T17:45:26.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.843+0000 7f5a22200640 1 -- 172.21.15.45:0/3072545488 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f5a040a2800 msgr2=0x7f5a040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:26.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.843+0000 7f5a22200640 1 --2- 172.21.15.45:0/3072545488 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f5a040a2800 0x7f5a040a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5a10005fd0 tx=0x7f5a1000b4a0 comp rx=0 tx=0).stop 2024-03-28T17:45:26.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.843+0000 7f5a22200640 1 -- 172.21.15.45:0/3072545488 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a1c1049b0 msgr2=0x7f5a1c19a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:26.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.843+0000 7f5a22200640 1 --2- 172.21.15.45:0/3072545488 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a1c1049b0 0x7f5a1c19a030 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5a0802c170 tx=0x7f5a0802c6c0 comp rx=0 tx=0).stop 2024-03-28T17:45:26.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.844+0000 7f5a22200640 1 -- 172.21.15.45:0/3072545488 shutdown_connections 2024-03-28T17:45:26.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.844+0000 7f5a22200640 1 --2- 172.21.15.45:0/3072545488 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f5a040a2800 0x7f5a040a4cc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:26.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.844+0000 7f5a22200640 1 --2- 172.21.15.45:0/3072545488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a1c105370 0x7f5a1c1a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:26.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.844+0000 7f5a22200640 1 --2- 172.21.15.45:0/3072545488 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a1c1049b0 0x7f5a1c19a030 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:26.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.844+0000 7f5a22200640 1 --2- 172.21.15.45:0/3072545488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a1c1037c0 0x7f5a1c199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:26.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.844+0000 7f5a22200640 1 -- 172.21.15.45:0/3072545488 >> 172.21.15.45:0/3072545488 conn(0x7f5a1c0fed50 msgr2=0x7f5a1c107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:26.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.844+0000 7f5a22200640 1 -- 172.21.15.45:0/3072545488 shutdown_connections 2024-03-28T17:45:26.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:26.844+0000 7f5a22200640 1 -- 172.21.15.45:0/3072545488 wait complete. 2024-03-28T17:45:26.865 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:45:27.025 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:26 smithi151 ceph-mon[36481]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:27.025 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:26 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:27.025 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:26 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:45:27.025 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:26 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:27.025 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:26 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:27.025 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:26 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:27.025 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:26 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:27.025 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:26 smithi151 ceph-mon[36481]: Deploying daemon mds.l on smithi151 2024-03-28T17:45:27.025 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:26 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/2395688626' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:45:27.025 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:26 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3072545488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:45:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.080+0000 7f3b23c00640 1 -- 172.21.15.45:0/977621012 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b1c104ad0 msgr2=0x7f3b1c104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.080+0000 7f3b23c00640 1 --2- 172.21.15.45:0/977621012 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b1c104ad0 0x7f3b1c104f50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3b08009860 tx=0x7f3b0802bb70 comp rx=0 tx=0).stop 2024-03-28T17:45:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.081+0000 7f3b23c00640 1 -- 172.21.15.45:0/977621012 shutdown_connections 2024-03-28T17:45:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.081+0000 7f3b23c00640 1 --2- 172.21.15.45:0/977621012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b1c105490 0x7f3b1c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.081+0000 7f3b23c00640 1 --2- 172.21.15.45:0/977621012 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b1c104ad0 0x7f3b1c104f50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.081+0000 7f3b23c00640 1 --2- 172.21.15.45:0/977621012 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b1c1038e0 0x7f3b1c103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.081+0000 7f3b23c00640 1 -- 172.21.15.45:0/977621012 >> 172.21.15.45:0/977621012 conn(0x7f3b1c0fee50 msgr2=0x7f3b1c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:27.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.081+0000 7f3b23c00640 1 -- 172.21.15.45:0/977621012 shutdown_connections 2024-03-28T17:45:27.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.081+0000 7f3b23c00640 1 -- 172.21.15.45:0/977621012 wait complete. 2024-03-28T17:45:27.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.082+0000 7f3b23c00640 1 Processor -- start 2024-03-28T17:45:27.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.082+0000 7f3b23c00640 1 -- start start 2024-03-28T17:45:27.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.082+0000 7f3b23c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b1c1038e0 0x7f3b1c199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:27.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.082+0000 7f3b23c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b1c104ad0 0x7f3b1c19a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:27.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.082+0000 7f3b23c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b1c105490 0x7f3b1c1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:27.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.082+0000 7f3b23c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c10f540 con 0x7f3b1c104ad0 2024-03-28T17:45:27.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.082+0000 7f3b23c00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c10f3c0 con 0x7f3b1c105490 2024-03-28T17:45:27.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.082+0000 7f3b23c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c10f6c0 con 0x7f3b1c1038e0 2024-03-28T17:45:27.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.083+0000 7f3b21c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b1c105490 0x7f3b1c1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:27.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.083+0000 7f3b21c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b1c105490 0x7f3b1c1a1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:38710/0 (socket says 172.21.15.45:38710) 2024-03-28T17:45:27.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.083+0000 7f3b21c00640 1 -- 172.21.15.45:0/431303251 learned_addr learned my addr 172.21.15.45:0/431303251 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:45:27.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.083+0000 7f3b21200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b1c1038e0 0x7f3b1c199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:27.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.083+0000 7f3b1be00640 1 --2- 172.21.15.45:0/431303251 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b1c104ad0 0x7f3b1c19a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:27.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.083+0000 7f3b21c00640 1 -- 172.21.15.45:0/431303251 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b1c1038e0 msgr2=0x7f3b1c199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:27.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.083+0000 7f3b21c00640 1 --2- 172.21.15.45:0/431303251 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b1c1038e0 0x7f3b1c199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:27.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.083+0000 7f3b21c00640 1 -- 172.21.15.45:0/431303251 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b1c104ad0 msgr2=0x7f3b1c19a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:27.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.083+0000 7f3b21c00640 1 --2- 172.21.15.45:0/431303251 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b1c104ad0 0x7f3b1c19a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:27.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.083+0000 7f3b21c00640 1 -- 172.21.15.45:0/431303251 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1c1a1800 con 0x7f3b1c105490 2024-03-28T17:45:27.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.083+0000 7f3b21c00640 1 --2- 172.21.15.45:0/431303251 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b1c105490 0x7f3b1c1a1190 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400eb10 tx=0x7f3b1400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:27.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.084+0000 7f3b19600640 1 -- 172.21.15.45:0/431303251 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b14004b20 con 0x7f3b1c105490 2024-03-28T17:45:27.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.084+0000 7f3b19600640 1 -- 172.21.15.45:0/431303251 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b14004cc0 con 0x7f3b1c105490 2024-03-28T17:45:27.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.084+0000 7f3b23c00640 1 -- 172.21.15.45:0/431303251 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b1c1a1af0 con 0x7f3b1c105490 2024-03-28T17:45:27.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.084+0000 7f3b19600640 1 -- 172.21.15.45:0/431303251 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b14010760 con 0x7f3b1c105490 2024-03-28T17:45:27.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.084+0000 7f3b23c00640 1 -- 172.21.15.45:0/431303251 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b1c1a2010 con 0x7f3b1c105490 2024-03-28T17:45:27.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.085+0000 7f3b23c00640 1 -- 172.21.15.45:0/431303251 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b1c10ac00 con 0x7f3b1c105490 2024-03-28T17:45:27.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.087+0000 7f3b19600640 1 -- 172.21.15.45:0/431303251 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3b14005020 con 0x7f3b1c105490 2024-03-28T17:45:27.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.088+0000 7f3b19600640 1 --2- 172.21.15.45:0/431303251 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3b040a2880 0x7f3b040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:27.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.088+0000 7f3b19600640 1 -- 172.21.15.45:0/431303251 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3b140ceb40 con 0x7f3b1c105490 2024-03-28T17:45:27.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.088+0000 7f3b21200640 1 --2- 172.21.15.45:0/431303251 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3b040a2880 0x7f3b040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:27.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.089+0000 7f3b21200640 1 --2- 172.21.15.45:0/431303251 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3b040a2880 0x7f3b040a4d40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3b10009530 tx=0x7f3b10009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:27.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.093+0000 7f3b19600640 1 -- 172.21.15.45:0/431303251 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3b1409bfe0 con 0x7f3b1c105490 2024-03-28T17:45:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:26 smithi045 ceph-mon[31773]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:26 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:26 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:45:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:26 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:26 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:26 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:26 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:26 smithi045 ceph-mon[31773]: Deploying daemon mds.l on smithi151 2024-03-28T17:45:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:26 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/2395688626' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:45:27.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:26 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3072545488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:45:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:26 smithi071 ceph-mon[36597]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:26 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:26 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:45:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:26 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:26 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:26 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:26 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:26 smithi071 ceph-mon[36597]: Deploying daemon mds.l on smithi151 2024-03-28T17:45:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:26 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/2395688626' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:45:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:26 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3072545488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:45:27.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.339+0000 7f3b23c00640 1 -- 172.21.15.45:0/431303251 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b1c02d070 con 0x7f3b1c105490 2024-03-28T17:45:27.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.341+0000 7f3b19600640 1 -- 172.21.15.45:0/431303251 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7f3b1409b8e0 con 0x7f3b1c105490 2024-03-28T17:45:27.341 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:45:27.341 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-28T17:45:27.341 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 1 2024-03-28T17:45:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.343+0000 7f3b23c00640 1 -- 172.21.15.45:0/431303251 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3b040a2880 msgr2=0x7f3b040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.343+0000 7f3b23c00640 1 --2- 172.21.15.45:0/431303251 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3b040a2880 0x7f3b040a4d40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3b10009530 tx=0x7f3b10009260 comp rx=0 tx=0).stop 2024-03-28T17:45:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.343+0000 7f3b23c00640 1 -- 172.21.15.45:0/431303251 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b1c105490 msgr2=0x7f3b1c1a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.343+0000 7f3b23c00640 1 --2- 172.21.15.45:0/431303251 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b1c105490 0x7f3b1c1a1190 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400eb10 tx=0x7f3b1400c2d0 comp rx=0 tx=0).stop 2024-03-28T17:45:27.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.343+0000 7f3b23c00640 1 -- 172.21.15.45:0/431303251 shutdown_connections 2024-03-28T17:45:27.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.343+0000 7f3b23c00640 1 --2- 172.21.15.45:0/431303251 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3b040a2880 0x7f3b040a4d40 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:27.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.343+0000 7f3b23c00640 1 --2- 172.21.15.45:0/431303251 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b1c105490 0x7f3b1c1a1190 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:27.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.343+0000 7f3b23c00640 1 --2- 172.21.15.45:0/431303251 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b1c104ad0 0x7f3b1c19a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:27.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.343+0000 7f3b23c00640 1 --2- 172.21.15.45:0/431303251 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b1c1038e0 0x7f3b1c199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:27.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.343+0000 7f3b23c00640 1 -- 172.21.15.45:0/431303251 >> 172.21.15.45:0/431303251 conn(0x7f3b1c0fee50 msgr2=0x7f3b1c107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:27.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.343+0000 7f3b23c00640 1 -- 172.21.15.45:0/431303251 shutdown_connections 2024-03-28T17:45:27.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:27.343+0000 7f3b23c00640 1 -- 172.21.15.45:0/431303251 wait complete. 2024-03-28T17:45:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:27 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/431303251' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:28.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:27 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/431303251' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:28.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:27 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/431303251' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:28.492 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:45:28 smithi151 systemd[1]: Starting Ceph mds.l for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:45:28.823 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:45:28 smithi151 podman[66260]: 2024-03-28 17:45:28.612777842 +0000 UTC m=+0.018327071 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:45:28.823 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:45:28 smithi151 podman[66260]: 2024-03-28 17:45:28.72794932 +0000 UTC m=+0.133498544 container create 3ac9123a664969c9ebb0f696fb14b9252ce0f4a7de41fbfcd3cc78ef0548ad47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD) 2024-03-28T17:45:29.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:28 smithi045 ceph-mon[31773]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:29.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:28 smithi071 ceph-mon[36597]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:29.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:28 smithi151 ceph-mon[36481]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:29.323 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:45:28 smithi151 podman[66260]: 2024-03-28 17:45:28.964048967 +0000 UTC m=+0.369598197 container init 3ac9123a664969c9ebb0f696fb14b9252ce0f4a7de41fbfcd3cc78ef0548ad47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-28T17:45:29.323 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:45:28 smithi151 podman[66260]: 2024-03-28 17:45:28.966931795 +0000 UTC m=+0.372481013 container start 3ac9123a664969c9ebb0f696fb14b9252ce0f4a7de41fbfcd3cc78ef0548ad47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-28T17:45:29.323 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:45:28 smithi151 ceph-mds[66280]: -- 172.21.15.151:0/2973333524 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564f5a61ba40 con 0x564f5a5ee800 2024-03-28T17:45:29.323 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:45:28 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: starting mds.l at 2024-03-28T17:45:29.323 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:45:29 smithi151 bash[66260]: 3ac9123a664969c9ebb0f696fb14b9252ce0f4a7de41fbfcd3cc78ef0548ad47 2024-03-28T17:45:29.324 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:45:29 smithi151 systemd[1]: Started Ceph mds.l for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:45:29.364 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:45:29.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.595+0000 7f4b81800640 1 -- 172.21.15.45:0/2609561834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b7c101f70 msgr2=0x7f4b7c104360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:29.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.595+0000 7f4b81800640 1 --2- 172.21.15.45:0/2609561834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b7c101f70 0x7f4b7c104360 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c009900 tx=0x7f4b6c02bc30 comp rx=0 tx=0).stop 2024-03-28T17:45:29.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.595+0000 7f4b81800640 1 -- 172.21.15.45:0/2609561834 shutdown_connections 2024-03-28T17:45:29.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.595+0000 7f4b81800640 1 --2- 172.21.15.45:0/2609561834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b7c1071d0 0x7f4b7c109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:29.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.595+0000 7f4b81800640 1 --2- 172.21.15.45:0/2609561834 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b7c1048a0 0x7f4b7c106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:29.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.595+0000 7f4b81800640 1 --2- 172.21.15.45:0/2609561834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b7c101f70 0x7f4b7c104360 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:29.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.595+0000 7f4b81800640 1 -- 172.21.15.45:0/2609561834 >> 172.21.15.45:0/2609561834 conn(0x7f4b7c0fb980 msgr2=0x7f4b7c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:29.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.595+0000 7f4b81800640 1 -- 172.21.15.45:0/2609561834 shutdown_connections 2024-03-28T17:45:29.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.595+0000 7f4b81800640 1 -- 172.21.15.45:0/2609561834 wait complete. 2024-03-28T17:45:29.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.596+0000 7f4b81800640 1 Processor -- start 2024-03-28T17:45:29.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.596+0000 7f4b81800640 1 -- start start 2024-03-28T17:45:29.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.597+0000 7f4b81800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b7c101f70 0x7f4b7c1018e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:29.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.597+0000 7f4b81800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b7c1048a0 0x7f4b7c0fff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:29.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.597+0000 7f4b81800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b7c1071d0 0x7f4b7c100460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:29.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.597+0000 7f4b81800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c10d1d0 con 0x7f4b7c101f70 2024-03-28T17:45:29.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.597+0000 7f4b81800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c10d050 con 0x7f4b7c1048a0 2024-03-28T17:45:29.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.597+0000 7f4b81800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c10d350 con 0x7f4b7c1071d0 2024-03-28T17:45:29.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.597+0000 7f4b7be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b7c101f70 0x7f4b7c1018e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:29.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.597+0000 7f4b7b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b7c1048a0 0x7f4b7c0fff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:29.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.597+0000 7f4b80e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b7c1071d0 0x7f4b7c100460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:29.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.597+0000 7f4b7b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b7c1048a0 0x7f4b7c0fff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:38722/0 (socket says 172.21.15.45:38722) 2024-03-28T17:45:29.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.597+0000 7f4b7b400640 1 -- 172.21.15.45:0/216070893 learned_addr learned my addr 172.21.15.45:0/216070893 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:45:29.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.597+0000 7f4b7b400640 1 -- 172.21.15.45:0/216070893 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b7c1071d0 msgr2=0x7f4b7c100460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:29.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.597+0000 7f4b7b400640 1 --2- 172.21.15.45:0/216070893 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b7c1071d0 0x7f4b7c100460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:29.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.597+0000 7f4b7b400640 1 -- 172.21.15.45:0/216070893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b7c101f70 msgr2=0x7f4b7c1018e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:29.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.597+0000 7f4b7b400640 1 --2- 172.21.15.45:0/216070893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b7c101f70 0x7f4b7c1018e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:29.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.597+0000 7f4b7b400640 1 -- 172.21.15.45:0/216070893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7c100cf0 con 0x7f4b7c1048a0 2024-03-28T17:45:29.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.597+0000 7f4b80e00640 1 --2- 172.21.15.45:0/216070893 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b7c1071d0 0x7f4b7c100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:45:29.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.598+0000 7f4b7b400640 1 --2- 172.21.15.45:0/216070893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b7c1048a0 0x7f4b7c0fff20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4b6800e9f0 tx=0x7f4b6800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:29.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.598+0000 7f4b78c00640 1 -- 172.21.15.45:0/216070893 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6800ccc0 con 0x7f4b7c1048a0 2024-03-28T17:45:29.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.598+0000 7f4b78c00640 1 -- 172.21.15.45:0/216070893 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b68005500 con 0x7f4b7c1048a0 2024-03-28T17:45:29.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.598+0000 7f4b78c00640 1 -- 172.21.15.45:0/216070893 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b68010690 con 0x7f4b7c1048a0 2024-03-28T17:45:29.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.598+0000 7f4b81800640 1 -- 172.21.15.45:0/216070893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b7c1a1940 con 0x7f4b7c1048a0 2024-03-28T17:45:29.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.598+0000 7f4b81800640 1 -- 172.21.15.45:0/216070893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b7c1a1e40 con 0x7f4b7c1048a0 2024-03-28T17:45:29.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.599+0000 7f4b81800640 1 -- 172.21.15.45:0/216070893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b7c0fd470 con 0x7f4b7c1048a0 2024-03-28T17:45:29.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.602+0000 7f4b78c00640 1 -- 172.21.15.45:0/216070893 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4b680046c0 con 0x7f4b7c1048a0 2024-03-28T17:45:29.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.603+0000 7f4b78c00640 1 --2- 172.21.15.45:0/216070893 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4b600a2880 0x7f4b600a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:29.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.603+0000 7f4b78c00640 1 -- 172.21.15.45:0/216070893 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4b680ce990 con 0x7f4b7c1048a0 2024-03-28T17:45:29.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.604+0000 7f4b7be00640 1 --2- 172.21.15.45:0/216070893 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4b600a2880 0x7f4b600a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:29.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.604+0000 7f4b7be00640 1 --2- 172.21.15.45:0/216070893 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4b600a2880 0x7f4b600a4d40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c02c240 tx=0x7f4b6c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:29.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.607+0000 7f4b78c00640 1 -- 172.21.15.45:0/216070893 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4b6809be30 con 0x7f4b7c1048a0 2024-03-28T17:45:29.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.872+0000 7f4b81800640 1 -- 172.21.15.45:0/216070893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b7c0622a0 con 0x7f4b7c1048a0 2024-03-28T17:45:29.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.873+0000 7f4b78c00640 1 -- 172.21.15.45:0/216070893 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7f4b6809b730 con 0x7f4b7c1048a0 2024-03-28T17:45:29.874 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:45:29.874 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T17:45:29.874 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 2 2024-03-28T17:45:29.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.876+0000 7f4b81800640 1 -- 172.21.15.45:0/216070893 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4b600a2880 msgr2=0x7f4b600a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:29.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.876+0000 7f4b81800640 1 --2- 172.21.15.45:0/216070893 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4b600a2880 0x7f4b600a4d40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c02c240 tx=0x7f4b6c02df90 comp rx=0 tx=0).stop 2024-03-28T17:45:29.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.876+0000 7f4b81800640 1 -- 172.21.15.45:0/216070893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b7c1048a0 msgr2=0x7f4b7c0fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:29.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.876+0000 7f4b81800640 1 --2- 172.21.15.45:0/216070893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b7c1048a0 0x7f4b7c0fff20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4b6800e9f0 tx=0x7f4b6800eeb0 comp rx=0 tx=0).stop 2024-03-28T17:45:29.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.877+0000 7f4b81800640 1 -- 172.21.15.45:0/216070893 shutdown_connections 2024-03-28T17:45:29.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.877+0000 7f4b81800640 1 --2- 172.21.15.45:0/216070893 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4b600a2880 0x7f4b600a4d40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:29.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.877+0000 7f4b81800640 1 --2- 172.21.15.45:0/216070893 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b7c1071d0 0x7f4b7c100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:29.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.877+0000 7f4b81800640 1 --2- 172.21.15.45:0/216070893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b7c1048a0 0x7f4b7c0fff20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:29.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.877+0000 7f4b81800640 1 --2- 172.21.15.45:0/216070893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b7c101f70 0x7f4b7c1018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:29.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.877+0000 7f4b81800640 1 -- 172.21.15.45:0/216070893 >> 172.21.15.45:0/216070893 conn(0x7f4b7c0fb980 msgr2=0x7f4b7c1029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:29.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.877+0000 7f4b81800640 1 -- 172.21.15.45:0/216070893 shutdown_connections 2024-03-28T17:45:29.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:29.877+0000 7f4b81800640 1 -- 172.21.15.45:0/216070893 wait complete. 2024-03-28T17:45:30.103 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:30 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/1898596562,v1:172.21.15.151:6833/1898596562] up:boot 2024-03-28T17:45:30.103 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:30 smithi071 ceph-mon[36597]: fsmap 1 up:standby 2024-03-28T17:45:30.103 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:30 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T17:45:30.103 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:30 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:30.103 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:30 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:30.103 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:30 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:30.103 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:30 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:30.103 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:30 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:30.103 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:30 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:30.103 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:30 smithi071 ceph-mon[36597]: Deploying daemon mds.k on smithi071 2024-03-28T17:45:30.103 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:30 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/216070893' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:30.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:30 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/1898596562,v1:172.21.15.151:6833/1898596562] up:boot 2024-03-28T17:45:30.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:30 smithi045 ceph-mon[31773]: fsmap 1 up:standby 2024-03-28T17:45:30.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:30 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T17:45:30.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:30 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:30.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:30 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:30.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:30 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:30.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:30 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:30.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:30 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:30.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:30 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:30.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:30 smithi045 ceph-mon[31773]: Deploying daemon mds.k on smithi071 2024-03-28T17:45:30.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:30 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/216070893' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:30.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:30 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/1898596562,v1:172.21.15.151:6833/1898596562] up:boot 2024-03-28T17:45:30.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:30 smithi151 ceph-mon[36481]: fsmap 1 up:standby 2024-03-28T17:45:30.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:30 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T17:45:30.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:30 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:30.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:30 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:30.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:30 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:30.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:30 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:30.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:30 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:30.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:30 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:30.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:30 smithi151 ceph-mon[36481]: Deploying daemon mds.k on smithi071 2024-03-28T17:45:30.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:30 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/216070893' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:31.457 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:31 smithi071 ceph-mon[36597]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:31 smithi151 ceph-mon[36481]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:31.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:31 smithi045 ceph-mon[31773]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:31.811 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:45:31 smithi071 podman[65941]: 2024-03-28 17:45:31.695708475 +0000 UTC m=+0.138461315 container create 9f17d11e5016d688c6a7b93b43eebf0b6103668fb7a28d69cd5763c3f363bca0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD) 2024-03-28T17:45:31.899 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:45:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.138+0000 7fcc42200640 1 -- 172.21.15.45:0/4004328988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc3c104ad0 msgr2=0x7fcc3c104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.138+0000 7fcc42200640 1 --2- 172.21.15.45:0/4004328988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc3c104ad0 0x7fcc3c104f50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcc28009900 tx=0x7fcc2802bc30 comp rx=0 tx=0).stop 2024-03-28T17:45:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.138+0000 7fcc42200640 1 -- 172.21.15.45:0/4004328988 shutdown_connections 2024-03-28T17:45:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.138+0000 7fcc42200640 1 --2- 172.21.15.45:0/4004328988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc3c105490 0x7fcc3c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.138+0000 7fcc42200640 1 --2- 172.21.15.45:0/4004328988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc3c104ad0 0x7fcc3c104f50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.138+0000 7fcc42200640 1 --2- 172.21.15.45:0/4004328988 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc3c1038e0 0x7fcc3c103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.138+0000 7fcc42200640 1 -- 172.21.15.45:0/4004328988 >> 172.21.15.45:0/4004328988 conn(0x7fcc3c0fee50 msgr2=0x7fcc3c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.138+0000 7fcc42200640 1 -- 172.21.15.45:0/4004328988 shutdown_connections 2024-03-28T17:45:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.138+0000 7fcc42200640 1 -- 172.21.15.45:0/4004328988 wait complete. 2024-03-28T17:45:32.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.139+0000 7fcc42200640 1 Processor -- start 2024-03-28T17:45:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.139+0000 7fcc42200640 1 -- start start 2024-03-28T17:45:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.140+0000 7fcc42200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc3c1038e0 0x7fcc3c199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.140+0000 7fcc42200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc3c104ad0 0x7fcc3c19a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.140+0000 7fcc42200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc3c105490 0x7fcc3c1a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.140+0000 7fcc42200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c10f590 con 0x7fcc3c105490 2024-03-28T17:45:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.140+0000 7fcc42200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c10f410 con 0x7fcc3c104ad0 2024-03-28T17:45:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.140+0000 7fcc42200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c10f710 con 0x7fcc3c1038e0 2024-03-28T17:45:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.140+0000 7fcc3b400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc3c1038e0 0x7fcc3c199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.140+0000 7fcc3b400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc3c1038e0 0x7fcc3c199bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:44308/0 (socket says 172.21.15.45:44308) 2024-03-28T17:45:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.140+0000 7fcc3b400640 1 -- 172.21.15.45:0/1806018282 learned_addr learned my addr 172.21.15.45:0/1806018282 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:45:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.140+0000 7fcc3aa00640 1 --2- 172.21.15.45:0/1806018282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc3c104ad0 0x7fcc3c19a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.140+0000 7fcc3be00640 1 --2- 172.21.15.45:0/1806018282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc3c105490 0x7fcc3c1a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.141+0000 7fcc3b400640 1 -- 172.21.15.45:0/1806018282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc3c104ad0 msgr2=0x7fcc3c19a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.141+0000 7fcc3b400640 1 --2- 172.21.15.45:0/1806018282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc3c104ad0 0x7fcc3c19a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.141+0000 7fcc3b400640 1 -- 172.21.15.45:0/1806018282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc3c105490 msgr2=0x7fcc3c1a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.141+0000 7fcc3b400640 1 --2- 172.21.15.45:0/1806018282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc3c105490 0x7fcc3c1a1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.141+0000 7fcc3b400640 1 -- 172.21.15.45:0/1806018282 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3c1a17d0 con 0x7fcc3c1038e0 2024-03-28T17:45:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.141+0000 7fcc3aa00640 1 --2- 172.21.15.45:0/1806018282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc3c104ad0 0x7fcc3c19a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:45:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.141+0000 7fcc3be00640 1 --2- 172.21.15.45:0/1806018282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc3c105490 0x7fcc3c1a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T17:45:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.141+0000 7fcc3b400640 1 --2- 172.21.15.45:0/1806018282 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc3c1038e0 0x7fcc3c199bb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c00e9c0 tx=0x7fcc2c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.141+0000 7fcc33e00640 1 -- 172.21.15.45:0/1806018282 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c00cde0 con 0x7fcc3c1038e0 2024-03-28T17:45:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.142+0000 7fcc33e00640 1 -- 172.21.15.45:0/1806018282 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc2c010910 con 0x7fcc3c1038e0 2024-03-28T17:45:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.142+0000 7fcc33e00640 1 -- 172.21.15.45:0/1806018282 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c018a70 con 0x7fcc3c1038e0 2024-03-28T17:45:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.142+0000 7fcc42200640 1 -- 172.21.15.45:0/1806018282 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc3c1a1ac0 con 0x7fcc3c1038e0 2024-03-28T17:45:32.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.142+0000 7fcc42200640 1 -- 172.21.15.45:0/1806018282 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc3c1a1f90 con 0x7fcc3c1038e0 2024-03-28T17:45:32.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.143+0000 7fcc42200640 1 -- 172.21.15.45:0/1806018282 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc3c10ac00 con 0x7fcc3c1038e0 2024-03-28T17:45:32.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.145+0000 7fcc33e00640 1 -- 172.21.15.45:0/1806018282 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcc2c01f020 con 0x7fcc3c1038e0 2024-03-28T17:45:32.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.146+0000 7fcc33e00640 1 --2- 172.21.15.45:0/1806018282 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcc200a2880 0x7fcc200a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:32.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.147+0000 7fcc33e00640 1 -- 172.21.15.45:0/1806018282 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcc2c0cebc0 con 0x7fcc3c1038e0 2024-03-28T17:45:32.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.147+0000 7fcc3aa00640 1 --2- 172.21.15.45:0/1806018282 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcc200a2880 0x7fcc200a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:32.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.148+0000 7fcc3aa00640 1 --2- 172.21.15.45:0/1806018282 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcc200a2880 0x7fcc200a4d40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcc2802c240 tx=0x7fcc2802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:32.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.149+0000 7fcc33e00640 1 -- 172.21.15.45:0/1806018282 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcc2c09c060 con 0x7fcc3c1038e0 2024-03-28T17:45:32.310 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:45:31 smithi071 podman[65941]: 2024-03-28 17:45:31.932131557 +0000 UTC m=+0.374884397 container init 9f17d11e5016d688c6a7b93b43eebf0b6103668fb7a28d69cd5763c3f363bca0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-28T17:45:32.310 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:45:31 smithi071 podman[65941]: 2024-03-28 17:45:31.934759527 +0000 UTC m=+0.377512367 container start 9f17d11e5016d688c6a7b93b43eebf0b6103668fb7a28d69cd5763c3f363bca0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-28T17:45:32.311 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:45:31 smithi071 ceph-mds[65960]: -- 172.21.15.71:0/2169031747 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558e57799a40 con 0x558e5776d000 2024-03-28T17:45:32.311 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:45:31 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: starting mds.k at 2024-03-28T17:45:32.311 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:45:32 smithi071 bash[65941]: 9f17d11e5016d688c6a7b93b43eebf0b6103668fb7a28d69cd5763c3f363bca0 2024-03-28T17:45:32.311 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:45:32 smithi071 systemd[1]: Started Ceph mds.k for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:45:32.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.410+0000 7fcc42200640 1 -- 172.21.15.45:0/1806018282 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc3c02d070 con 0x7fcc3c1038e0 2024-03-28T17:45:32.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.411+0000 7fcc33e00640 1 -- 172.21.15.45:0/1806018282 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2119 (secure 0 0 0) 0x7fcc2c09b960 con 0x7fcc3c1038e0 2024-03-28T17:45:32.411 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:45:32.411 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-28T17:45:32.411 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 3 2024-03-28T17:45:32.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.413+0000 7fcc42200640 1 -- 172.21.15.45:0/1806018282 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcc200a2880 msgr2=0x7fcc200a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:32.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.413+0000 7fcc42200640 1 --2- 172.21.15.45:0/1806018282 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcc200a2880 0x7fcc200a4d40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcc2802c240 tx=0x7fcc2802df90 comp rx=0 tx=0).stop 2024-03-28T17:45:32.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.414+0000 7fcc42200640 1 -- 172.21.15.45:0/1806018282 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc3c1038e0 msgr2=0x7fcc3c199bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:32.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.414+0000 7fcc42200640 1 --2- 172.21.15.45:0/1806018282 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc3c1038e0 0x7fcc3c199bb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c00e9c0 tx=0x7fcc2c00ee80 comp rx=0 tx=0).stop 2024-03-28T17:45:32.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.414+0000 7fcc42200640 1 -- 172.21.15.45:0/1806018282 shutdown_connections 2024-03-28T17:45:32.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.414+0000 7fcc42200640 1 --2- 172.21.15.45:0/1806018282 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcc200a2880 0x7fcc200a4d40 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:32.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.414+0000 7fcc42200640 1 --2- 172.21.15.45:0/1806018282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc3c105490 0x7fcc3c1a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:32.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.414+0000 7fcc42200640 1 --2- 172.21.15.45:0/1806018282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc3c104ad0 0x7fcc3c19a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:32.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.414+0000 7fcc42200640 1 --2- 172.21.15.45:0/1806018282 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc3c1038e0 0x7fcc3c199bb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:32.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.414+0000 7fcc42200640 1 -- 172.21.15.45:0/1806018282 >> 172.21.15.45:0/1806018282 conn(0x7fcc3c0fee50 msgr2=0x7fcc3c107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:32.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.414+0000 7fcc42200640 1 -- 172.21.15.45:0/1806018282 shutdown_connections 2024-03-28T17:45:32.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:32.414+0000 7fcc42200640 1 -- 172.21.15.45:0/1806018282 wait complete. 2024-03-28T17:45:33.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:33 smithi045 ceph-mon[31773]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:33.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:33 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:33.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:33 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:33.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:33 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:33.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:33 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:33.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:33 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:33.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:33 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:33.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:33 smithi045 ceph-mon[31773]: Deploying daemon mds.j on smithi045 2024-03-28T17:45:33.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:33 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/3145454285,v1:172.21.15.71:6833/3145454285] up:boot 2024-03-28T17:45:33.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:33 smithi045 ceph-mon[31773]: fsmap 2 up:standby 2024-03-28T17:45:33.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:33 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T17:45:33.338 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:33 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1806018282' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:33.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:33 smithi071 ceph-mon[36597]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:33.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:33 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:33.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:33 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:33.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:33 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:33.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:33 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:33.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:33 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:33.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:33 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:33.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:33 smithi071 ceph-mon[36597]: Deploying daemon mds.j on smithi045 2024-03-28T17:45:33.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:33 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/3145454285,v1:172.21.15.71:6833/3145454285] up:boot 2024-03-28T17:45:33.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:33 smithi071 ceph-mon[36597]: fsmap 2 up:standby 2024-03-28T17:45:33.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:33 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T17:45:33.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:33 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1806018282' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:33 smithi151 ceph-mon[36481]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:33 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:33 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:33 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:33.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:33 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:33.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:33 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:33.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:33 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:33.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:33 smithi151 ceph-mon[36481]: Deploying daemon mds.j on smithi045 2024-03-28T17:45:33.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:33 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/3145454285,v1:172.21.15.71:6833/3145454285] up:boot 2024-03-28T17:45:33.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:33 smithi151 ceph-mon[36481]: fsmap 2 up:standby 2024-03-28T17:45:33.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:33 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T17:45:33.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:33 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1806018282' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:34.434 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:45:34.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.868+0000 7f1997e00640 1 -- 172.21.15.45:0/2501829876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19900772a0 msgr2=0x7f1990077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:34.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.868+0000 7f1997e00640 1 --2- 172.21.15.45:0/2501829876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19900772a0 0x7f1990077700 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f197c009860 tx=0x7f197c02bb70 comp rx=0 tx=0).stop 2024-03-28T17:45:34.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.868+0000 7f1997e00640 1 -- 172.21.15.45:0/2501829876 shutdown_connections 2024-03-28T17:45:34.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.868+0000 7f1997e00640 1 --2- 172.21.15.45:0/2501829876 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1990077c40 0x7f19900784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:34.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.868+0000 7f1997e00640 1 --2- 172.21.15.45:0/2501829876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19900772a0 0x7f1990077700 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:34.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.868+0000 7f1997e00640 1 --2- 172.21.15.45:0/2501829876 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1990076060 0x7f1990076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:34.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.868+0000 7f1997e00640 1 -- 172.21.15.45:0/2501829876 >> 172.21.15.45:0/2501829876 conn(0x7f19900fee50 msgr2=0x7f1990101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:34.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.868+0000 7f1997e00640 1 -- 172.21.15.45:0/2501829876 shutdown_connections 2024-03-28T17:45:34.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.868+0000 7f1997e00640 1 -- 172.21.15.45:0/2501829876 wait complete. 2024-03-28T17:45:34.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.870+0000 7f1997e00640 1 Processor -- start 2024-03-28T17:45:34.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.870+0000 7f1997e00640 1 -- start start 2024-03-28T17:45:34.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.871+0000 7f1997e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1990076060 0x7f1990072dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:34.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.871+0000 7f1997e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1990077c40 0x7f1990073310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:34.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.871+0000 7f1997e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1990071410 0x7f1990071890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:34.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.873+0000 7f1995400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1990076060 0x7f1990072dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:34.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.873+0000 7f1995400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1990076060 0x7f1990072dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:44332/0 (socket says 172.21.15.45:44332) 2024-03-28T17:45:34.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.874+0000 7f1995e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1990071410 0x7f1990071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:34.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.874+0000 7f1995e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1990071410 0x7f1990071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55700/0 (socket says 172.21.15.45:55700) 2024-03-28T17:45:34.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.874+0000 7f1994a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1990077c40 0x7f1990073310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:34.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.874+0000 7f1997e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199006bad0 con 0x7f1990071410 2024-03-28T17:45:34.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.874+0000 7f1997e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199006b950 con 0x7f1990077c40 2024-03-28T17:45:34.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.874+0000 7f1997e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199006bc50 con 0x7f1990076060 2024-03-28T17:45:34.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.874+0000 7f1995400640 1 -- 172.21.15.45:0/521629731 learned_addr learned my addr 172.21.15.45:0/521629731 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:45:34.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.874+0000 7f1995400640 1 -- 172.21.15.45:0/521629731 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1990077c40 msgr2=0x7f1990073310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:34.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.874+0000 7f1995400640 1 --2- 172.21.15.45:0/521629731 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1990077c40 0x7f1990073310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:34.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.875+0000 7f1995400640 1 -- 172.21.15.45:0/521629731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1990071410 msgr2=0x7f1990071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:34.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.875+0000 7f1995400640 1 --2- 172.21.15.45:0/521629731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1990071410 0x7f1990071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:34.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.875+0000 7f1995400640 1 -- 172.21.15.45:0/521629731 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1990072150 con 0x7f1990076060 2024-03-28T17:45:34.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.875+0000 7f1995e00640 1 --2- 172.21.15.45:0/521629731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1990071410 0x7f1990071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:45:34.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.875+0000 7f1995400640 1 --2- 172.21.15.45:0/521629731 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1990076060 0x7f1990072dd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f198800c420 tx=0x7f1988009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:34.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.877+0000 7f198e000640 1 -- 172.21.15.45:0/521629731 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1988003970 con 0x7f1990076060 2024-03-28T17:45:34.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.877+0000 7f198e000640 1 -- 172.21.15.45:0/521629731 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1988004020 con 0x7f1990076060 2024-03-28T17:45:34.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.877+0000 7f198e000640 1 -- 172.21.15.45:0/521629731 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198800bad0 con 0x7f1990076060 2024-03-28T17:45:34.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.877+0000 7f1997e00640 1 -- 172.21.15.45:0/521629731 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19901aecd0 con 0x7f1990076060 2024-03-28T17:45:34.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.877+0000 7f1997e00640 1 -- 172.21.15.45:0/521629731 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19901af1a0 con 0x7f1990076060 2024-03-28T17:45:34.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.879+0000 7f1997e00640 1 -- 172.21.15.45:0/521629731 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1990110490 con 0x7f1990076060 2024-03-28T17:45:34.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.883+0000 7f198e000640 1 -- 172.21.15.45:0/521629731 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f198801d020 con 0x7f1990076060 2024-03-28T17:45:34.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.885+0000 7f198e000640 1 --2- 172.21.15.45:0/521629731 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f19780a2880 0x7f19780a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:34.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.886+0000 7f198e000640 1 -- 172.21.15.45:0/521629731 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f19880ce5d0 con 0x7f1990076060 2024-03-28T17:45:34.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.886+0000 7f1994a00640 1 --2- 172.21.15.45:0/521629731 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f19780a2880 0x7f19780a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:34.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.886+0000 7f198e000640 1 -- 172.21.15.45:0/521629731 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f19880cea50 con 0x7f1990076060 2024-03-28T17:45:34.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:34.887+0000 7f1994a00640 1 --2- 172.21.15.45:0/521629731 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f19780a2880 0x7f19780a4d40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f197c009780 tx=0x7f197c034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:35.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:35.139+0000 7f1997e00640 1 -- 172.21.15.45:0/521629731 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f199002d1b0 con 0x7f1990076060 2024-03-28T17:45:35.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:35.140+0000 7f198e000640 1 -- 172.21.15.45:0/521629731 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2119 (secure 0 0 0) 0x7f198809ba70 con 0x7f1990076060 2024-03-28T17:45:35.140 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:45:35.140 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-28T17:45:35.140 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 3 2024-03-28T17:45:35.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:35.142+0000 7f1997e00640 1 -- 172.21.15.45:0/521629731 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f19780a2880 msgr2=0x7f19780a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:35.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:35.142+0000 7f1997e00640 1 --2- 172.21.15.45:0/521629731 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f19780a2880 0x7f19780a4d40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f197c009780 tx=0x7f197c034ab0 comp rx=0 tx=0).stop 2024-03-28T17:45:35.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:35.142+0000 7f1997e00640 1 -- 172.21.15.45:0/521629731 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1990076060 msgr2=0x7f1990072dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:35.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:35.142+0000 7f1997e00640 1 --2- 172.21.15.45:0/521629731 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1990076060 0x7f1990072dd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f198800c420 tx=0x7f1988009f90 comp rx=0 tx=0).stop 2024-03-28T17:45:35.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:35.142+0000 7f1997e00640 1 -- 172.21.15.45:0/521629731 shutdown_connections 2024-03-28T17:45:35.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:35.142+0000 7f1997e00640 1 --2- 172.21.15.45:0/521629731 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f19780a2880 0x7f19780a4d40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:35.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:35.142+0000 7f1997e00640 1 --2- 172.21.15.45:0/521629731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1990071410 0x7f1990071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:35.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:35.142+0000 7f1997e00640 1 --2- 172.21.15.45:0/521629731 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1990077c40 0x7f1990073310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:35.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:35.142+0000 7f1997e00640 1 --2- 172.21.15.45:0/521629731 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1990076060 0x7f1990072dd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:35.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:35.142+0000 7f1997e00640 1 -- 172.21.15.45:0/521629731 >> 172.21.15.45:0/521629731 conn(0x7f19900fee50 msgr2=0x7f19901005d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:35.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:35.142+0000 7f1997e00640 1 -- 172.21.15.45:0/521629731 shutdown_connections 2024-03-28T17:45:35.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:35.143+0000 7f1997e00640 1 -- 172.21.15.45:0/521629731 wait complete. 2024-03-28T17:45:35.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:35 smithi045 ceph-mon[31773]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:35.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:35 smithi071 ceph-mon[36597]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:35.565 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:35 smithi151 ceph-mon[36481]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:35.813 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 17:45:35 smithi045 podman[72411]: 2024-03-28 17:45:35.498820244 +0000 UTC m=+0.175314117 container create 5768fb00f3eaf9dfef9cf13875ae9a09bb016a213aeb0bd17d960838939136a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T17:45:35.813 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 17:45:35 smithi045 podman[72411]: 2024-03-28 17:45:35.811709168 +0000 UTC m=+0.488203051 container init 5768fb00f3eaf9dfef9cf13875ae9a09bb016a213aeb0bd17d960838939136a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T17:45:36.102 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 17:45:35 smithi045 podman[72411]: 2024-03-28 17:45:35.822119974 +0000 UTC m=+0.498613821 container start 5768fb00f3eaf9dfef9cf13875ae9a09bb016a213aeb0bd17d960838939136a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T17:45:36.102 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 17:45:35 smithi045 ceph-mds[72429]: -- 172.21.15.45:0/3359800194 <== mon.0 v2:172.21.15.45:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c448837a40 con 0x55c44880ac00 2024-03-28T17:45:36.102 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 17:45:35 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: starting mds.j at 2024-03-28T17:45:36.102 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 17:45:35 smithi045 bash[72411]: 5768fb00f3eaf9dfef9cf13875ae9a09bb016a213aeb0bd17d960838939136a6 2024-03-28T17:45:36.102 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 17:45:35 smithi045 systemd[1]: Started Ceph mds.j for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:45:36.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:36 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/521629731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:36.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:36 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:36.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:36 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:36.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:36 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:36.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:36 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:36.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:36 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:36.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:36 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:36 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/521629731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:36 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:36.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:36 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:36.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:36 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:36.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:36 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:36.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:36 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:36.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:36 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:36 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/521629731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:36 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:36 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:36 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:36 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:36 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:36.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:36 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:37.164 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:45:37.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.411+0000 7f449e600640 1 -- 172.21.15.45:0/4102896692 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44981016c0 msgr2=0x7f4498101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:37.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.411+0000 7f449e600640 1 --2- 172.21.15.45:0/4102896692 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44981016c0 0x7f4498101ae0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4488009900 tx=0x7f448802bc30 comp rx=0 tx=0).stop 2024-03-28T17:45:37.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.411+0000 7f449e600640 1 -- 172.21.15.45:0/4102896692 shutdown_connections 2024-03-28T17:45:37.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.411+0000 7f449e600640 1 --2- 172.21.15.45:0/4102896692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4498103270 0x7f4498109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:37.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.411+0000 7f449e600640 1 --2- 172.21.15.45:0/4102896692 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44981028b0 0x7f4498102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:37.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.411+0000 7f449e600640 1 --2- 172.21.15.45:0/4102896692 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f44981016c0 0x7f4498101ae0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:37.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.411+0000 7f449e600640 1 -- 172.21.15.45:0/4102896692 >> 172.21.15.45:0/4102896692 conn(0x7f44980fcc50 msgr2=0x7f44980ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:37.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.411+0000 7f449e600640 1 -- 172.21.15.45:0/4102896692 shutdown_connections 2024-03-28T17:45:37.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.411+0000 7f449e600640 1 -- 172.21.15.45:0/4102896692 wait complete. 2024-03-28T17:45:37.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.412+0000 7f449e600640 1 Processor -- start 2024-03-28T17:45:37.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.412+0000 7f449e600640 1 -- start start 2024-03-28T17:45:37.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.413+0000 7f449e600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44981016c0 0x7f4498199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:37.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.413+0000 7f449e600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44981028b0 0x7f449819a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:37.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.413+0000 7f449e600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4498103270 0x7f44981a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:37.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.413+0000 7f449e600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449810d380 con 0x7f44981028b0 2024-03-28T17:45:37.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.413+0000 7f449e600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449810d200 con 0x7f44981016c0 2024-03-28T17:45:37.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.413+0000 7f449e600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449810d500 con 0x7f4498103270 2024-03-28T17:45:37.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.413+0000 7f448fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44981028b0 0x7f449819a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:37.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.413+0000 7f4497400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44981016c0 0x7f4498199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:37.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.413+0000 7f4497e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4498103270 0x7f44981a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:37.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.413+0000 7f4497400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44981016c0 0x7f4498199bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:41544/0 (socket says 172.21.15.45:41544) 2024-03-28T17:45:37.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.413+0000 7f4497400640 1 -- 172.21.15.45:0/3441039217 learned_addr learned my addr 172.21.15.45:0/3441039217 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:45:37.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.413+0000 7f448fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44981028b0 0x7f449819a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35346/0 (socket says 172.21.15.45:35346) 2024-03-28T17:45:37.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.413+0000 7f4497400640 1 -- 172.21.15.45:0/3441039217 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4498103270 msgr2=0x7f44981a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:37.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.413+0000 7f4497400640 1 --2- 172.21.15.45:0/3441039217 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4498103270 0x7f44981a1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:37.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.413+0000 7f4497400640 1 -- 172.21.15.45:0/3441039217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44981028b0 msgr2=0x7f449819a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:37.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.413+0000 7f4497400640 1 --2- 172.21.15.45:0/3441039217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44981028b0 0x7f449819a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:37.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.413+0000 7f4497400640 1 -- 172.21.15.45:0/3441039217 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44981a17d0 con 0x7f44981016c0 2024-03-28T17:45:37.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.414+0000 7f4497400640 1 --2- 172.21.15.45:0/3441039217 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44981016c0 0x7f4498199bb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4488000c00 tx=0x7f448802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:37.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.414+0000 7f4494c00640 1 -- 172.21.15.45:0/3441039217 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4488043020 con 0x7f44981016c0 2024-03-28T17:45:37.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.414+0000 7f4494c00640 1 -- 172.21.15.45:0/3441039217 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44880054e0 con 0x7f44981016c0 2024-03-28T17:45:37.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.414+0000 7f449e600640 1 -- 172.21.15.45:0/3441039217 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44981a1a60 con 0x7f44981016c0 2024-03-28T17:45:37.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.414+0000 7f4494c00640 1 -- 172.21.15.45:0/3441039217 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4488003c40 con 0x7f44981016c0 2024-03-28T17:45:37.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.414+0000 7f449e600640 1 -- 172.21.15.45:0/3441039217 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44981a1fb0 con 0x7f44981016c0 2024-03-28T17:45:37.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:37 smithi045 ceph-mon[31773]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:37.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:37 smithi045 ceph-mon[31773]: Deploying daemon mds.d on smithi045 2024-03-28T17:45:37.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:37 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/333952790,v1:172.21.15.45:6835/333952790] up:boot 2024-03-28T17:45:37.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:37 smithi045 ceph-mon[31773]: fsmap 3 up:standby 2024-03-28T17:45:37.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:37 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T17:45:37.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.415+0000 7f449e600640 1 -- 172.21.15.45:0/3441039217 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44981a2180 con 0x7f44981016c0 2024-03-28T17:45:37.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.417+0000 7f4494c00640 1 -- 172.21.15.45:0/3441039217 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4488046020 con 0x7f44981016c0 2024-03-28T17:45:37.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.418+0000 7f4494c00640 1 --2- 172.21.15.45:0/3441039217 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f447c0a2800 0x7f447c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:37.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.419+0000 7f4494c00640 1 -- 172.21.15.45:0/3441039217 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f44880f0600 con 0x7f44981016c0 2024-03-28T17:45:37.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.419+0000 7f448fe00640 1 --2- 172.21.15.45:0/3441039217 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f447c0a2800 0x7f447c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:37.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.420+0000 7f448fe00640 1 --2- 172.21.15.45:0/3441039217 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f447c0a2800 0x7f447c0a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f449819b0c0 tx=0x7f4480005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:37.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.423+0000 7f4494c00640 1 -- 172.21.15.45:0/3441039217 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f448804d020 con 0x7f44981016c0 2024-03-28T17:45:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:37 smithi071 ceph-mon[36597]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:37 smithi071 ceph-mon[36597]: Deploying daemon mds.d on smithi045 2024-03-28T17:45:37.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:37 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/333952790,v1:172.21.15.45:6835/333952790] up:boot 2024-03-28T17:45:37.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:37 smithi071 ceph-mon[36597]: fsmap 3 up:standby 2024-03-28T17:45:37.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:37 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T17:45:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:37 smithi151 ceph-mon[36481]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:37 smithi151 ceph-mon[36481]: Deploying daemon mds.d on smithi045 2024-03-28T17:45:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:37 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/333952790,v1:172.21.15.45:6835/333952790] up:boot 2024-03-28T17:45:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:37 smithi151 ceph-mon[36481]: fsmap 3 up:standby 2024-03-28T17:45:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:37 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T17:45:37.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.681+0000 7f449e600640 1 -- 172.21.15.45:0/3441039217 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44981a2510 con 0x7f44981016c0 2024-03-28T17:45:37.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.682+0000 7f4494c00640 1 -- 172.21.15.45:0/3441039217 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2907 (secure 0 0 0) 0x7f44880bdaa0 con 0x7f44981016c0 2024-03-28T17:45:37.682 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:45:37.682 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-28T17:45:37.683 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 4 2024-03-28T17:45:37.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.685+0000 7f449e600640 1 -- 172.21.15.45:0/3441039217 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f447c0a2800 msgr2=0x7f447c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:37.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.685+0000 7f449e600640 1 --2- 172.21.15.45:0/3441039217 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f447c0a2800 0x7f447c0a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f449819b0c0 tx=0x7f4480005f90 comp rx=0 tx=0).stop 2024-03-28T17:45:37.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.685+0000 7f449e600640 1 -- 172.21.15.45:0/3441039217 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44981016c0 msgr2=0x7f4498199bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:37.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.685+0000 7f449e600640 1 --2- 172.21.15.45:0/3441039217 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44981016c0 0x7f4498199bb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4488000c00 tx=0x7f448802c650 comp rx=0 tx=0).stop 2024-03-28T17:45:37.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.685+0000 7f449e600640 1 -- 172.21.15.45:0/3441039217 shutdown_connections 2024-03-28T17:45:37.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.685+0000 7f449e600640 1 --2- 172.21.15.45:0/3441039217 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f447c0a2800 0x7f447c0a4cc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:37.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.685+0000 7f449e600640 1 --2- 172.21.15.45:0/3441039217 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4498103270 0x7f44981a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:37.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.685+0000 7f449e600640 1 --2- 172.21.15.45:0/3441039217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44981028b0 0x7f449819a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:37.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.685+0000 7f449e600640 1 --2- 172.21.15.45:0/3441039217 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44981016c0 0x7f4498199bb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:37.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.685+0000 7f449e600640 1 -- 172.21.15.45:0/3441039217 >> 172.21.15.45:0/3441039217 conn(0x7f44980fcc50 msgr2=0x7f44980738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:37.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.685+0000 7f449e600640 1 -- 172.21.15.45:0/3441039217 shutdown_connections 2024-03-28T17:45:37.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:37.685+0000 7f449e600640 1 -- 172.21.15.45:0/3441039217 wait complete. 2024-03-28T17:45:38.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:38 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3441039217' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:38.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:38 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3441039217' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:38.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:38 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3441039217' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:39.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:39 smithi045 ceph-mon[31773]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:39.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:39 smithi071 ceph-mon[36597]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:39.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:39 smithi151 ceph-mon[36481]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:39.666 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:45:39 smithi045 systemd[1]: Starting Ceph mds.d for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:45:39.708 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:45:39.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.947+0000 7faf5be00640 1 -- 172.21.15.45:0/3669172688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf5c1048a0 msgr2=0x7faf5c106c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:39.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.947+0000 7faf5be00640 1 --2- 172.21.15.45:0/3669172688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf5c1048a0 0x7faf5c106c90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7faf44009900 tx=0x7faf4402bc30 comp rx=0 tx=0).stop 2024-03-28T17:45:39.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.947+0000 7faf5be00640 1 -- 172.21.15.45:0/3669172688 shutdown_connections 2024-03-28T17:45:39.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.947+0000 7faf5be00640 1 --2- 172.21.15.45:0/3669172688 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf5c1071d0 0x7faf5c109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:39.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.947+0000 7faf5be00640 1 --2- 172.21.15.45:0/3669172688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf5c1048a0 0x7faf5c106c90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:39.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.947+0000 7faf5be00640 1 --2- 172.21.15.45:0/3669172688 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf5c101f70 0x7faf5c104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:39.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.947+0000 7faf5be00640 1 -- 172.21.15.45:0/3669172688 >> 172.21.15.45:0/3669172688 conn(0x7faf5c0fb980 msgr2=0x7faf5c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:39.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.947+0000 7faf5be00640 1 -- 172.21.15.45:0/3669172688 shutdown_connections 2024-03-28T17:45:39.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.947+0000 7faf5be00640 1 -- 172.21.15.45:0/3669172688 wait complete. 2024-03-28T17:45:39.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.948+0000 7faf5be00640 1 Processor -- start 2024-03-28T17:45:39.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.948+0000 7faf5be00640 1 -- start start 2024-03-28T17:45:39.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.949+0000 7faf5be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf5c101f70 0x7faf5c199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:39.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.949+0000 7faf5be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf5c1048a0 0x7faf5c199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:39.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.949+0000 7faf5be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf5c1071d0 0x7faf5c1a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:39.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.949+0000 7faf5be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5c10d180 con 0x7faf5c1048a0 2024-03-28T17:45:39.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.949+0000 7faf5be00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5c10d000 con 0x7faf5c101f70 2024-03-28T17:45:39.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.949+0000 7faf5be00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5c10d300 con 0x7faf5c1071d0 2024-03-28T17:45:39.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.949+0000 7faf5a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf5c1048a0 0x7faf5c199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:39.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.949+0000 7faf5a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf5c1048a0 0x7faf5c199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35374/0 (socket says 172.21.15.45:35374) 2024-03-28T17:45:39.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.949+0000 7faf5a000640 1 -- 172.21.15.45:0/130166610 learned_addr learned my addr 172.21.15.45:0/130166610 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:45:39.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.949+0000 7faf5aa00640 1 --2- 172.21.15.45:0/130166610 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf5c101f70 0x7faf5c199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:39.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.949+0000 7faf5b400640 1 --2- 172.21.15.45:0/130166610 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf5c1071d0 0x7faf5c1a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:39.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.949+0000 7faf5aa00640 1 -- 172.21.15.45:0/130166610 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf5c1071d0 msgr2=0x7faf5c1a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:39.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.949+0000 7faf5aa00640 1 --2- 172.21.15.45:0/130166610 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf5c1071d0 0x7faf5c1a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:39.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.949+0000 7faf5aa00640 1 -- 172.21.15.45:0/130166610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf5c1048a0 msgr2=0x7faf5c199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:39.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.949+0000 7faf5aa00640 1 --2- 172.21.15.45:0/130166610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf5c1048a0 0x7faf5c199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:39.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.949+0000 7faf5aa00640 1 -- 172.21.15.45:0/130166610 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf5c1a16d0 con 0x7faf5c101f70 2024-03-28T17:45:39.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.950+0000 7faf5aa00640 1 --2- 172.21.15.45:0/130166610 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf5c101f70 0x7faf5c199ab0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009fd0 tx=0x7faf4c00eed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:39.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.950+0000 7faf53400640 1 -- 172.21.15.45:0/130166610 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c00cde0 con 0x7faf5c101f70 2024-03-28T17:45:39.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.950+0000 7faf53400640 1 -- 172.21.15.45:0/130166610 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faf4c005020 con 0x7faf5c101f70 2024-03-28T17:45:39.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.950+0000 7faf5be00640 1 -- 172.21.15.45:0/130166610 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf5c1a19c0 con 0x7faf5c101f70 2024-03-28T17:45:39.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.950+0000 7faf53400640 1 -- 172.21.15.45:0/130166610 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c0106d0 con 0x7faf5c101f70 2024-03-28T17:45:39.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.950+0000 7faf5be00640 1 -- 172.21.15.45:0/130166610 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf5c1a1ee0 con 0x7faf5c101f70 2024-03-28T17:45:39.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.951+0000 7faf5be00640 1 -- 172.21.15.45:0/130166610 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf5c108d20 con 0x7faf5c101f70 2024-03-28T17:45:39.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.953+0000 7faf53400640 1 -- 172.21.15.45:0/130166610 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7faf4c005540 con 0x7faf5c101f70 2024-03-28T17:45:39.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.955+0000 7faf53400640 1 --2- 172.21.15.45:0/130166610 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7faf400a2880 0x7faf400a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:39.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.955+0000 7faf53400640 1 -- 172.21.15.45:0/130166610 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7faf4c0cf100 con 0x7faf5c101f70 2024-03-28T17:45:39.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.955+0000 7faf5a000640 1 --2- 172.21.15.45:0/130166610 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7faf400a2880 0x7faf400a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:39.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.956+0000 7faf5a000640 1 --2- 172.21.15.45:0/130166610 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7faf400a2880 0x7faf400a4d40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7faf4402c8e0 tx=0x7faf44005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:39.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:39.957+0000 7faf53400640 1 -- 172.21.15.45:0/130166610 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faf4c09c5a0 con 0x7faf5c101f70 2024-03-28T17:45:40.042 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:45:39 smithi045 podman[72755]: 2024-03-28 17:45:39.620224847 +0000 UTC m=+0.034961679 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:45:40.042 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:45:39 smithi045 podman[72755]: 2024-03-28 17:45:39.728344581 +0000 UTC m=+0.143081433 container create 1902e980607f67272121c61ead4881d3d01e1ccc1d681ee589bef689ff93ef2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-28T17:45:40.042 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:45:40 smithi045 podman[72755]: 2024-03-28 17:45:40.041768538 +0000 UTC m=+0.456505443 container init 1902e980607f67272121c61ead4881d3d01e1ccc1d681ee589bef689ff93ef2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-28T17:45:40.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:40.209+0000 7faf5be00640 1 -- 172.21.15.45:0/130166610 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf5c02d070 con 0x7faf5c101f70 2024-03-28T17:45:40.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:40.210+0000 7faf53400640 1 -- 172.21.15.45:0/130166610 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3698 (secure 0 0 0) 0x7faf4c09bea0 con 0x7faf5c101f70 2024-03-28T17:45:40.211 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:45:40.211 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[]} 2024-03-28T17:45:40.211 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 5 2024-03-28T17:45:40.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:40.213+0000 7faf5be00640 1 -- 172.21.15.45:0/130166610 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7faf400a2880 msgr2=0x7faf400a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:40.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:40.213+0000 7faf5be00640 1 --2- 172.21.15.45:0/130166610 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7faf400a2880 0x7faf400a4d40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7faf4402c8e0 tx=0x7faf44005b50 comp rx=0 tx=0).stop 2024-03-28T17:45:40.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:40.213+0000 7faf5be00640 1 -- 172.21.15.45:0/130166610 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf5c101f70 msgr2=0x7faf5c199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:40.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:40.213+0000 7faf5be00640 1 --2- 172.21.15.45:0/130166610 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf5c101f70 0x7faf5c199ab0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009fd0 tx=0x7faf4c00eed0 comp rx=0 tx=0).stop 2024-03-28T17:45:40.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:40.213+0000 7faf5be00640 1 -- 172.21.15.45:0/130166610 shutdown_connections 2024-03-28T17:45:40.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:40.213+0000 7faf5be00640 1 --2- 172.21.15.45:0/130166610 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7faf400a2880 0x7faf400a4d40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:40.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:40.213+0000 7faf5be00640 1 --2- 172.21.15.45:0/130166610 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf5c1071d0 0x7faf5c1a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:40.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:40.213+0000 7faf5be00640 1 --2- 172.21.15.45:0/130166610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf5c1048a0 0x7faf5c199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:40.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:40.213+0000 7faf5be00640 1 --2- 172.21.15.45:0/130166610 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf5c101f70 0x7faf5c199ab0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:40.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:40.213+0000 7faf5be00640 1 -- 172.21.15.45:0/130166610 >> 172.21.15.45:0/130166610 conn(0x7faf5c0fb980 msgr2=0x7faf5c1016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:40.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:40.213+0000 7faf5be00640 1 -- 172.21.15.45:0/130166610 shutdown_connections 2024-03-28T17:45:40.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:40.213+0000 7faf5be00640 1 -- 172.21.15.45:0/130166610 wait complete. 2024-03-28T17:45:40.416 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:45:40 smithi045 podman[72755]: 2024-03-28 17:45:40.053826156 +0000 UTC m=+0.468562978 container start 1902e980607f67272121c61ead4881d3d01e1ccc1d681ee589bef689ff93ef2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-28T17:45:40.416 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:45:40 smithi045 ceph-mds[72813]: -- 172.21.15.45:0/3144544015 <== mon.0 v2:172.21.15.45:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563fd40e5a40 con 0x563fd40b9000 2024-03-28T17:45:40.416 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:45:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: starting mds.d at 2024-03-28T17:45:40.417 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:45:40 smithi045 bash[72755]: 1902e980607f67272121c61ead4881d3d01e1ccc1d681ee589bef689ff93ef2b 2024-03-28T17:45:40.417 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:45:40 smithi045 systemd[1]: Started Ceph mds.d for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:45:41.208 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:41 smithi071 ceph-mon[36597]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:41.208 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:41 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/1262971887,v1:172.21.15.45:6837/1262971887] up:boot 2024-03-28T17:45:41.208 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:41 smithi071 ceph-mon[36597]: fsmap 4 up:standby 2024-03-28T17:45:41.208 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:41 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T17:45:41.209 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:41 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/130166610' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:41.209 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:41 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:41.209 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:41 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:41.209 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:41 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:41.209 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:41 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:41.209 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:41 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:41.209 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:41 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:41.209 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:41 smithi071 ceph-mon[36597]: Deploying daemon mds.e on smithi071 2024-03-28T17:45:41.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:41 smithi045 ceph-mon[31773]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:41.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:41 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/1262971887,v1:172.21.15.45:6837/1262971887] up:boot 2024-03-28T17:45:41.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:41 smithi045 ceph-mon[31773]: fsmap 4 up:standby 2024-03-28T17:45:41.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:41 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T17:45:41.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:41 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/130166610' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:41.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:41 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:41.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:41 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:41.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:41 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:41.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:41 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:41.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:41 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:41.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:41 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:41.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:41 smithi045 ceph-mon[31773]: Deploying daemon mds.e on smithi071 2024-03-28T17:45:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:41 smithi151 ceph-mon[36481]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:41.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:41 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/1262971887,v1:172.21.15.45:6837/1262971887] up:boot 2024-03-28T17:45:41.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:41 smithi151 ceph-mon[36481]: fsmap 4 up:standby 2024-03-28T17:45:41.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:41 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T17:45:41.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:41 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/130166610' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:41.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:41 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:41.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:41 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:41.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:41 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:41.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:41 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:41.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:41 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:41.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:41 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:41.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:41 smithi151 ceph-mon[36481]: Deploying daemon mds.e on smithi071 2024-03-28T17:45:42.234 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:45:42.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.466+0000 7f5891e00640 1 -- 172.21.15.45:0/3087892310 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f588c103270 msgr2=0x7f588c109b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:42.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.466+0000 7f5891e00640 1 --2- 172.21.15.45:0/3087892310 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f588c103270 0x7f588c109b00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5884009ca0 tx=0x7f588402c100 comp rx=0 tx=0).stop 2024-03-28T17:45:42.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.466+0000 7f5891e00640 1 -- 172.21.15.45:0/3087892310 shutdown_connections 2024-03-28T17:45:42.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.466+0000 7f5891e00640 1 --2- 172.21.15.45:0/3087892310 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f588c103270 0x7f588c109b00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:42.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.466+0000 7f5891e00640 1 --2- 172.21.15.45:0/3087892310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f588c1028b0 0x7f588c102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:42.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.466+0000 7f5891e00640 1 --2- 172.21.15.45:0/3087892310 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f588c1016c0 0x7f588c101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:42.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.466+0000 7f5891e00640 1 -- 172.21.15.45:0/3087892310 >> 172.21.15.45:0/3087892310 conn(0x7f588c0fcc50 msgr2=0x7f588c0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:42.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.466+0000 7f5891e00640 1 -- 172.21.15.45:0/3087892310 shutdown_connections 2024-03-28T17:45:42.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.466+0000 7f5891e00640 1 -- 172.21.15.45:0/3087892310 wait complete. 2024-03-28T17:45:42.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.467+0000 7f5891e00640 1 Processor -- start 2024-03-28T17:45:42.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.467+0000 7f5891e00640 1 -- start start 2024-03-28T17:45:42.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.468+0000 7f5891e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f588c1016c0 0x7f588c199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:42.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.468+0000 7f5891e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f588c1028b0 0x7f588c19a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:42.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.468+0000 7f5891e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f588c103270 0x7f588c1a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:42.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.468+0000 7f5891e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c10d320 con 0x7f588c1028b0 2024-03-28T17:45:42.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.468+0000 7f5891e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c10d1a0 con 0x7f588c1016c0 2024-03-28T17:45:42.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.468+0000 7f5891e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c10d4a0 con 0x7f588c103270 2024-03-28T17:45:42.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.468+0000 7f588aa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f588c1016c0 0x7f588c199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:42.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.468+0000 7f588a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f588c1028b0 0x7f588c19a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:42.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.468+0000 7f588aa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f588c1016c0 0x7f588c199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:41614/0 (socket says 172.21.15.45:41614) 2024-03-28T17:45:42.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.468+0000 7f588aa00640 1 -- 172.21.15.45:0/3150434954 learned_addr learned my addr 172.21.15.45:0/3150434954 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:45:42.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.468+0000 7f588b400640 1 --2- 172.21.15.45:0/3150434954 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f588c103270 0x7f588c1a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:42.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.469+0000 7f588aa00640 1 -- 172.21.15.45:0/3150434954 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f588c103270 msgr2=0x7f588c1a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:42.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.469+0000 7f588aa00640 1 --2- 172.21.15.45:0/3150434954 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f588c103270 0x7f588c1a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:42.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.469+0000 7f588aa00640 1 -- 172.21.15.45:0/3150434954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f588c1028b0 msgr2=0x7f588c19a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:42.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.469+0000 7f588aa00640 1 --2- 172.21.15.45:0/3150434954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f588c1028b0 0x7f588c19a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:42.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.469+0000 7f588aa00640 1 -- 172.21.15.45:0/3150434954 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588c1a1770 con 0x7f588c1016c0 2024-03-28T17:45:42.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.469+0000 7f588a000640 1 --2- 172.21.15.45:0/3150434954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f588c1028b0 0x7f588c19a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:45:42.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.469+0000 7f588aa00640 1 --2- 172.21.15.45:0/3150434954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f588c1016c0 0x7f588c199b50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f587c00d950 tx=0x7f587c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:42.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.469+0000 7f5890a00640 1 -- 172.21.15.45:0/3150434954 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c017020 con 0x7f588c1016c0 2024-03-28T17:45:42.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.470+0000 7f5890a00640 1 -- 172.21.15.45:0/3150434954 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f587c009c30 con 0x7f588c1016c0 2024-03-28T17:45:42.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.470+0000 7f5891e00640 1 -- 172.21.15.45:0/3150434954 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f588c1a1a60 con 0x7f588c1016c0 2024-03-28T17:45:42.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.470+0000 7f5890a00640 1 -- 172.21.15.45:0/3150434954 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c00f740 con 0x7f588c1016c0 2024-03-28T17:45:42.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.470+0000 7f5891e00640 1 -- 172.21.15.45:0/3150434954 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f588c1a1fb0 con 0x7f588c1016c0 2024-03-28T17:45:42.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.471+0000 7f5891e00640 1 -- 172.21.15.45:0/3150434954 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f588c0688e0 con 0x7f588c1016c0 2024-03-28T17:45:42.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.474+0000 7f5890a00640 1 -- 172.21.15.45:0/3150434954 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f587c01a020 con 0x7f588c1016c0 2024-03-28T17:45:42.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.475+0000 7f5890a00640 1 --2- 172.21.15.45:0/3150434954 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f58700a2800 0x7f58700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:42.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.475+0000 7f5890a00640 1 -- 172.21.15.45:0/3150434954 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f587c0cdfe0 con 0x7f588c1016c0 2024-03-28T17:45:42.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.475+0000 7f588a000640 1 --2- 172.21.15.45:0/3150434954 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f58700a2800 0x7f58700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:42.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.476+0000 7f588a000640 1 --2- 172.21.15.45:0/3150434954 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f58700a2800 0x7f58700a4cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f588c19b060 tx=0x7f5878005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:42.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.479+0000 7f5890a00640 1 -- 172.21.15.45:0/3150434954 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f587c021070 con 0x7f588c1016c0 2024-03-28T17:45:42.562 INFO:journalctl@ceph.mds.e.smithi071.stdout:Mar 28 17:45:42 smithi071 systemd[1]: Starting Ceph mds.e for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:45:42.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.721+0000 7f5891e00640 1 -- 172.21.15.45:0/3150434954 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f588c1a2240 con 0x7f588c1016c0 2024-03-28T17:45:42.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.722+0000 7f5890a00640 1 -- 172.21.15.45:0/3150434954 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3698 (secure 0 0 0) 0x7f587c009de0 con 0x7f588c1016c0 2024-03-28T17:45:42.723 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:45:42.723 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[]} 2024-03-28T17:45:42.723 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 5 2024-03-28T17:45:42.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.725+0000 7f5891e00640 1 -- 172.21.15.45:0/3150434954 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f58700a2800 msgr2=0x7f58700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:42.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.725+0000 7f5891e00640 1 --2- 172.21.15.45:0/3150434954 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f58700a2800 0x7f58700a4cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f588c19b060 tx=0x7f5878005f90 comp rx=0 tx=0).stop 2024-03-28T17:45:42.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.725+0000 7f5891e00640 1 -- 172.21.15.45:0/3150434954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f588c1016c0 msgr2=0x7f588c199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:42.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.725+0000 7f5891e00640 1 --2- 172.21.15.45:0/3150434954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f588c1016c0 0x7f588c199b50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f587c00d950 tx=0x7f587c00de10 comp rx=0 tx=0).stop 2024-03-28T17:45:42.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.725+0000 7f5891e00640 1 -- 172.21.15.45:0/3150434954 shutdown_connections 2024-03-28T17:45:42.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.725+0000 7f5891e00640 1 --2- 172.21.15.45:0/3150434954 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f58700a2800 0x7f58700a4cc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:42.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.725+0000 7f5891e00640 1 --2- 172.21.15.45:0/3150434954 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f588c103270 0x7f588c1a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:42.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.725+0000 7f5891e00640 1 --2- 172.21.15.45:0/3150434954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f588c1028b0 0x7f588c19a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:42.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.725+0000 7f5891e00640 1 --2- 172.21.15.45:0/3150434954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f588c1016c0 0x7f588c199b50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:42.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.725+0000 7f5891e00640 1 -- 172.21.15.45:0/3150434954 >> 172.21.15.45:0/3150434954 conn(0x7f588c0fcc50 msgr2=0x7f588c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:42.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.725+0000 7f5891e00640 1 -- 172.21.15.45:0/3150434954 shutdown_connections 2024-03-28T17:45:42.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:42.725+0000 7f5891e00640 1 -- 172.21.15.45:0/3150434954 wait complete. 2024-03-28T17:45:42.954 INFO:journalctl@ceph.mds.e.smithi071.stdout:Mar 28 17:45:42 smithi071 podman[66259]: 2024-03-28 17:45:42.593128346 +0000 UTC m=+0.017777466 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:45:42.954 INFO:journalctl@ceph.mds.e.smithi071.stdout:Mar 28 17:45:42 smithi071 podman[66259]: 2024-03-28 17:45:42.710159401 +0000 UTC m=+0.134808520 container create 605a78da2aac0e3a5fc5f6e12469ee7e257c2b9cefb5d0f6e96d10d4399b67f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-e, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T17:45:42.954 INFO:journalctl@ceph.mds.e.smithi071.stdout:Mar 28 17:45:42 smithi071 podman[66259]: 2024-03-28 17:45:42.95487395 +0000 UTC m=+0.379523063 container init 605a78da2aac0e3a5fc5f6e12469ee7e257c2b9cefb5d0f6e96d10d4399b67f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-28T17:45:43.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:43 smithi071 ceph-mon[36597]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:43.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:43 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3150434954' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:43.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:43 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:43.311 INFO:journalctl@ceph.mds.e.smithi071.stdout:Mar 28 17:45:42 smithi071 podman[66259]: 2024-03-28 17:45:42.957417914 +0000 UTC m=+0.382067029 container start 605a78da2aac0e3a5fc5f6e12469ee7e257c2b9cefb5d0f6e96d10d4399b67f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-28T17:45:43.311 INFO:journalctl@ceph.mds.e.smithi071.stdout:Mar 28 17:45:42 smithi071 ceph-mds[66278]: -- 172.21.15.71:0/1553796425 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a169967a40 con 0x55a16993b000 2024-03-28T17:45:43.311 INFO:journalctl@ceph.mds.e.smithi071.stdout:Mar 28 17:45:42 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-e[66274]: starting mds.e at 2024-03-28T17:45:43.311 INFO:journalctl@ceph.mds.e.smithi071.stdout:Mar 28 17:45:43 smithi071 bash[66259]: 605a78da2aac0e3a5fc5f6e12469ee7e257c2b9cefb5d0f6e96d10d4399b67f4 2024-03-28T17:45:43.311 INFO:journalctl@ceph.mds.e.smithi071.stdout:Mar 28 17:45:43 smithi071 systemd[1]: Started Ceph mds.e for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:45:43.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:43 smithi045 ceph-mon[31773]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:43.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:43 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3150434954' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:43.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:43 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:43.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:43 smithi151 ceph-mon[36481]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:43.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:43 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3150434954' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:43.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:43 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:44.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:44.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:44 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6834/4234998385,v1:172.21.15.71:6835/4234998385] up:boot 2024-03-28T17:45:44.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:44 smithi071 ceph-mon[36597]: fsmap 5 up:standby 2024-03-28T17:45:44.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T17:45:44.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:44.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:44.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:44.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:44.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:44 smithi071 ceph-mon[36597]: Deploying daemon mds.f on smithi151 2024-03-28T17:45:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:44 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6834/4234998385,v1:172.21.15.71:6835/4234998385] up:boot 2024-03-28T17:45:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:44 smithi045 ceph-mon[31773]: fsmap 5 up:standby 2024-03-28T17:45:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T17:45:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:44.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:44.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:44.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:44.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:44 smithi045 ceph-mon[31773]: Deploying daemon mds.f on smithi151 2024-03-28T17:45:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:44 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6834/4234998385,v1:172.21.15.71:6835/4234998385] up:boot 2024-03-28T17:45:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:44 smithi151 ceph-mon[36481]: fsmap 5 up:standby 2024-03-28T17:45:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T17:45:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:44.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:44.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:44 smithi151 ceph-mon[36481]: Deploying daemon mds.f on smithi151 2024-03-28T17:45:44.744 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:45:44.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.972+0000 7f7cb5400640 1 -- 172.21.15.45:0/3854039101 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7cb0105490 msgr2=0x7f7cb010bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:44.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.972+0000 7f7cb5400640 1 --2- 172.21.15.45:0/3854039101 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7cb0105490 0x7f7cb010bd20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8009be0 tx=0x7f7ca802bef0 comp rx=0 tx=0).stop 2024-03-28T17:45:44.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.972+0000 7f7cb5400640 1 -- 172.21.15.45:0/3854039101 shutdown_connections 2024-03-28T17:45:44.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.972+0000 7f7cb5400640 1 --2- 172.21.15.45:0/3854039101 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7cb0105490 0x7f7cb010bd20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:44.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.972+0000 7f7cb5400640 1 --2- 172.21.15.45:0/3854039101 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb0104ad0 0x7f7cb0104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:44.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.972+0000 7f7cb5400640 1 --2- 172.21.15.45:0/3854039101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cb01038e0 0x7f7cb0103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:44.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.972+0000 7f7cb5400640 1 -- 172.21.15.45:0/3854039101 >> 172.21.15.45:0/3854039101 conn(0x7f7cb00fee50 msgr2=0x7f7cb0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:44.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.972+0000 7f7cb5400640 1 -- 172.21.15.45:0/3854039101 shutdown_connections 2024-03-28T17:45:44.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.972+0000 7f7cb5400640 1 -- 172.21.15.45:0/3854039101 wait complete. 2024-03-28T17:45:44.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.973+0000 7f7cb5400640 1 Processor -- start 2024-03-28T17:45:44.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.973+0000 7f7cb5400640 1 -- start start 2024-03-28T17:45:44.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.974+0000 7f7cb5400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7cb01038e0 0x7f7cb0199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:44.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.974+0000 7f7cb5400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb0104ad0 0x7f7cb019a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:44.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.974+0000 7f7cb5400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cb0105490 0x7f7cb01a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:44.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.974+0000 7f7cb5400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb010f560 con 0x7f7cb0105490 2024-03-28T17:45:44.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.974+0000 7f7cb5400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb010f3e0 con 0x7f7cb01038e0 2024-03-28T17:45:44.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.974+0000 7f7cb5400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb010f6e0 con 0x7f7cb0104ad0 2024-03-28T17:45:44.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.974+0000 7f7cade00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb0104ad0 0x7f7cb019a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:44.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.974+0000 7f7caf200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cb0105490 0x7f7cb01a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:44.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.974+0000 7f7cade00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb0104ad0 0x7f7cb019a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:45342/0 (socket says 172.21.15.45:45342) 2024-03-28T17:45:44.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.974+0000 7f7cade00640 1 -- 172.21.15.45:0/4105391684 learned_addr learned my addr 172.21.15.45:0/4105391684 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:45:44.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.974+0000 7f7cae800640 1 --2- 172.21.15.45:0/4105391684 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7cb01038e0 0x7f7cb0199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:44.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.975+0000 7f7cade00640 1 -- 172.21.15.45:0/4105391684 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7cb01038e0 msgr2=0x7f7cb0199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:44.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.975+0000 7f7cade00640 1 --2- 172.21.15.45:0/4105391684 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7cb01038e0 0x7f7cb0199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:44.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.975+0000 7f7cade00640 1 -- 172.21.15.45:0/4105391684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cb0105490 msgr2=0x7f7cb01a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:44.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.975+0000 7f7cade00640 1 --2- 172.21.15.45:0/4105391684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cb0105490 0x7f7cb01a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:44.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.975+0000 7f7cade00640 1 -- 172.21.15.45:0/4105391684 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb01a17c0 con 0x7f7cb0104ad0 2024-03-28T17:45:44.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.975+0000 7f7cade00640 1 --2- 172.21.15.45:0/4105391684 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb0104ad0 0x7f7cb019a0e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c00e9f0 tx=0x7f7c9c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:44.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.975+0000 7f7ca7400640 1 -- 172.21.15.45:0/4105391684 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c00ccc0 con 0x7f7cb0104ad0 2024-03-28T17:45:44.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.975+0000 7f7cb5400640 1 -- 172.21.15.45:0/4105391684 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb01a1ab0 con 0x7f7cb0104ad0 2024-03-28T17:45:44.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.975+0000 7f7ca7400640 1 -- 172.21.15.45:0/4105391684 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c9c00ce60 con 0x7f7cb0104ad0 2024-03-28T17:45:44.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.975+0000 7f7ca7400640 1 -- 172.21.15.45:0/4105391684 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c010690 con 0x7f7cb0104ad0 2024-03-28T17:45:44.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.975+0000 7f7cb5400640 1 -- 172.21.15.45:0/4105391684 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb01a1fb0 con 0x7f7cb0104ad0 2024-03-28T17:45:44.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.977+0000 7f7cb5400640 1 -- 172.21.15.45:0/4105391684 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cb010ac00 con 0x7f7cb0104ad0 2024-03-28T17:45:44.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.979+0000 7f7ca7400640 1 -- 172.21.15.45:0/4105391684 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7c9c0046c0 con 0x7f7cb0104ad0 2024-03-28T17:45:44.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.981+0000 7f7ca7400640 1 --2- 172.21.15.45:0/4105391684 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f7c940a2800 0x7f7c940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:44.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.981+0000 7f7ca7400640 1 -- 172.21.15.45:0/4105391684 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7c9c0cea10 con 0x7f7cb0104ad0 2024-03-28T17:45:44.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.981+0000 7f7cae800640 1 --2- 172.21.15.45:0/4105391684 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f7c940a2800 0x7f7c940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:44.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.982+0000 7f7cae800640 1 --2- 172.21.15.45:0/4105391684 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f7c940a2800 0x7f7c940a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0104930 tx=0x7f7ca0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:44.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:44.985+0000 7f7ca7400640 1 -- 172.21.15.45:0/4105391684 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7c9c09beb0 con 0x7f7cb0104ad0 2024-03-28T17:45:45.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:45.233+0000 7f7cb5400640 1 -- 172.21.15.45:0/4105391684 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cb002d070 con 0x7f7cb0104ad0 2024-03-28T17:45:45.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:45.234+0000 7f7ca7400640 1 -- 172.21.15.45:0/4105391684 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4489 (secure 0 0 0) 0x7f7c9c09b7b0 con 0x7f7cb0104ad0 2024-03-28T17:45:45.235 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:45:45.235 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6835/4234998385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":4234998385},{"type":"v1","addr":"172.21.15.71:6835","nonce":4234998385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-28T17:45:45.236 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 6 2024-03-28T17:45:45.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:45.236+0000 7f7cb5400640 1 -- 172.21.15.45:0/4105391684 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f7c940a2800 msgr2=0x7f7c940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:45.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:45.236+0000 7f7cb5400640 1 --2- 172.21.15.45:0/4105391684 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f7c940a2800 0x7f7c940a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0104930 tx=0x7f7ca0009260 comp rx=0 tx=0).stop 2024-03-28T17:45:45.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:45.236+0000 7f7cb5400640 1 -- 172.21.15.45:0/4105391684 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb0104ad0 msgr2=0x7f7cb019a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:45.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:45.236+0000 7f7cb5400640 1 --2- 172.21.15.45:0/4105391684 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb0104ad0 0x7f7cb019a0e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c00e9f0 tx=0x7f7c9c00edf0 comp rx=0 tx=0).stop 2024-03-28T17:45:45.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:45.237+0000 7f7cb5400640 1 -- 172.21.15.45:0/4105391684 shutdown_connections 2024-03-28T17:45:45.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:45.237+0000 7f7cb5400640 1 --2- 172.21.15.45:0/4105391684 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f7c940a2800 0x7f7c940a4cc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:45.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:45.237+0000 7f7cb5400640 1 --2- 172.21.15.45:0/4105391684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cb0105490 0x7f7cb01a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:45.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:45.237+0000 7f7cb5400640 1 --2- 172.21.15.45:0/4105391684 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cb0104ad0 0x7f7cb019a0e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:45.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:45.237+0000 7f7cb5400640 1 --2- 172.21.15.45:0/4105391684 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7cb01038e0 0x7f7cb0199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:45.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:45.237+0000 7f7cb5400640 1 -- 172.21.15.45:0/4105391684 >> 172.21.15.45:0/4105391684 conn(0x7f7cb00fee50 msgr2=0x7f7cb0107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:45.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:45.237+0000 7f7cb5400640 1 -- 172.21.15.45:0/4105391684 shutdown_connections 2024-03-28T17:45:45.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:45.237+0000 7f7cb5400640 1 -- 172.21.15.45:0/4105391684 wait complete. 2024-03-28T17:45:45.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:45 smithi045 ceph-mon[31773]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:45.437 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:45 smithi151 ceph-mon[36481]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:45.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:45 smithi071 ceph-mon[36597]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:46.175 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:45:45 smithi151 systemd[1]: Started Ceph mds.f for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:45:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:46 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/4105391684' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:46.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:46 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:46.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:46 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:46.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:46 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:46.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:46 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:46.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:46 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:46.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:46 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:46 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/4105391684' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:46 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:46 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:46 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:46 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:46 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:46 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:46.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:46 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/4105391684' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:46.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:46 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:46.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:46 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:46.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:46 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:46.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:46 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:46.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:46 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:46.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:46 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:47.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:45:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:47 smithi151 ceph-mon[36481]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:47 smithi151 ceph-mon[36481]: Deploying daemon mds.c on smithi151 2024-03-28T17:45:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:47 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/2876354410,v1:172.21.15.151:6835/2876354410] up:boot 2024-03-28T17:45:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:47 smithi151 ceph-mon[36481]: fsmap 6 up:standby 2024-03-28T17:45:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:47 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T17:45:47.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.492+0000 7f80d5600640 1 -- 172.21.15.45:0/1053491748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80d01036f0 msgr2=0x7f80d0103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:47.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.492+0000 7f80d5600640 1 --2- 172.21.15.45:0/1053491748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80d01036f0 0x7f80d0103b10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f80c0009900 tx=0x7f80c002bc30 comp rx=0 tx=0).stop 2024-03-28T17:45:47.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.492+0000 7f80d5600640 1 -- 172.21.15.45:0/1053491748 shutdown_connections 2024-03-28T17:45:47.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.492+0000 7f80d5600640 1 --2- 172.21.15.45:0/1053491748 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80d01052a0 0x7f80d010bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:47.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.492+0000 7f80d5600640 1 --2- 172.21.15.45:0/1053491748 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80d01048e0 0x7f80d0104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:47.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.492+0000 7f80d5600640 1 --2- 172.21.15.45:0/1053491748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80d01036f0 0x7f80d0103b10 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:47.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.492+0000 7f80d5600640 1 -- 172.21.15.45:0/1053491748 >> 172.21.15.45:0/1053491748 conn(0x7f80d00fec80 msgr2=0x7f80d01010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:47.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.492+0000 7f80d5600640 1 -- 172.21.15.45:0/1053491748 shutdown_connections 2024-03-28T17:45:47.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.492+0000 7f80d5600640 1 -- 172.21.15.45:0/1053491748 wait complete. 2024-03-28T17:45:47.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.493+0000 7f80d5600640 1 Processor -- start 2024-03-28T17:45:47.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.493+0000 7f80d5600640 1 -- start start 2024-03-28T17:45:47.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.493+0000 7f80d5600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80d01036f0 0x7f80d0199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:47.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.493+0000 7f80d5600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80d01048e0 0x7f80d0199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:47.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.493+0000 7f80d5600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80d01052a0 0x7f80d01a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:47.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.493+0000 7f80d5600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d010f390 con 0x7f80d01036f0 2024-03-28T17:45:47.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.493+0000 7f80d5600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d010f210 con 0x7f80d01052a0 2024-03-28T17:45:47.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.493+0000 7f80d5600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d010f510 con 0x7f80d01048e0 2024-03-28T17:45:47.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.494+0000 7f80cfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80d01036f0 0x7f80d0199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:47.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.494+0000 7f80c7e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80d01048e0 0x7f80d0199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:47.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.494+0000 7f80cfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80d01036f0 0x7f80d0199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53368/0 (socket says 172.21.15.45:53368) 2024-03-28T17:45:47.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.494+0000 7f80cfe00640 1 -- 172.21.15.45:0/1979428951 learned_addr learned my addr 172.21.15.45:0/1979428951 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:45:47.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.494+0000 7f80d4c00640 1 --2- 172.21.15.45:0/1979428951 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80d01052a0 0x7f80d01a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:47.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.494+0000 7f80c7e00640 1 -- 172.21.15.45:0/1979428951 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80d01052a0 msgr2=0x7f80d01a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:47.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.494+0000 7f80c7e00640 1 --2- 172.21.15.45:0/1979428951 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80d01052a0 0x7f80d01a0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:47.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.494+0000 7f80c7e00640 1 -- 172.21.15.45:0/1979428951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80d01036f0 msgr2=0x7f80d0199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:47.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.494+0000 7f80c7e00640 1 --2- 172.21.15.45:0/1979428951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80d01036f0 0x7f80d0199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:47.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.494+0000 7f80c7e00640 1 -- 172.21.15.45:0/1979428951 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d01a1660 con 0x7f80d01048e0 2024-03-28T17:45:47.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.494+0000 7f80c7e00640 1 --2- 172.21.15.45:0/1979428951 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80d01048e0 0x7f80d0199f80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f80b800e9f0 tx=0x7f80b800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:47.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.495+0000 7f80cd600640 1 -- 172.21.15.45:0/1979428951 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b800ccc0 con 0x7f80d01048e0 2024-03-28T17:45:47.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.495+0000 7f80cd600640 1 -- 172.21.15.45:0/1979428951 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80b8010910 con 0x7f80d01048e0 2024-03-28T17:45:47.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.495+0000 7f80cd600640 1 -- 172.21.15.45:0/1979428951 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b8018960 con 0x7f80d01048e0 2024-03-28T17:45:47.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.495+0000 7f80d5600640 1 -- 172.21.15.45:0/1979428951 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80d01a1950 con 0x7f80d01048e0 2024-03-28T17:45:47.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.495+0000 7f80d5600640 1 -- 172.21.15.45:0/1979428951 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80d01a1ef0 con 0x7f80d01048e0 2024-03-28T17:45:47.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.496+0000 7f80d5600640 1 -- 172.21.15.45:0/1979428951 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80d010a9b0 con 0x7f80d01048e0 2024-03-28T17:45:47.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.498+0000 7f80cd600640 1 -- 172.21.15.45:0/1979428951 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f80b8010ac0 con 0x7f80d01048e0 2024-03-28T17:45:47.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.499+0000 7f80cd600640 1 --2- 172.21.15.45:0/1979428951 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80b40a2880 0x7f80b40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:47.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.499+0000 7f80cd600640 1 -- 172.21.15.45:0/1979428951 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f80b80cea20 con 0x7f80d01048e0 2024-03-28T17:45:47.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.499+0000 7f80cfe00640 1 --2- 172.21.15.45:0/1979428951 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80b40a2880 0x7f80b40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:47.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.500+0000 7f80cfe00640 1 --2- 172.21.15.45:0/1979428951 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80b40a2880 0x7f80b40a4d40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f80c002c8e0 tx=0x7f80c002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:47.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.503+0000 7f80cd600640 1 -- 172.21.15.45:0/1979428951 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f80b809bec0 con 0x7f80d01048e0 2024-03-28T17:45:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:47 smithi071 ceph-mon[36597]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:47 smithi071 ceph-mon[36597]: Deploying daemon mds.c on smithi151 2024-03-28T17:45:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:47 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/2876354410,v1:172.21.15.151:6835/2876354410] up:boot 2024-03-28T17:45:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:47 smithi071 ceph-mon[36597]: fsmap 6 up:standby 2024-03-28T17:45:47.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:47 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T17:45:47.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:47 smithi045 ceph-mon[31773]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:47.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:47 smithi045 ceph-mon[31773]: Deploying daemon mds.c on smithi151 2024-03-28T17:45:47.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:47 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/2876354410,v1:172.21.15.151:6835/2876354410] up:boot 2024-03-28T17:45:47.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:47 smithi045 ceph-mon[31773]: fsmap 6 up:standby 2024-03-28T17:45:47.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:47 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T17:45:47.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.753+0000 7f80d5600640 1 -- 172.21.15.45:0/1979428951 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80d002d070 con 0x7f80d01048e0 2024-03-28T17:45:47.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.754+0000 7f80cd600640 1 -- 172.21.15.45:0/1979428951 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5283 (secure 0 0 0) 0x7f80b809b7c0 con 0x7f80d01048e0 2024-03-28T17:45:47.754 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:45:47.754 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6835/2876354410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":2876354410},{"type":"v1","addr":"172.21.15.151:6835","nonce":2876354410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6835/4234998385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":4234998385},{"type":"v1","addr":"172.21.15.71:6835","nonce":4234998385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-28T17:45:47.755 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 7 2024-03-28T17:45:47.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.757+0000 7f80d5600640 1 -- 172.21.15.45:0/1979428951 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80b40a2880 msgr2=0x7f80b40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:47.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.757+0000 7f80d5600640 1 --2- 172.21.15.45:0/1979428951 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80b40a2880 0x7f80b40a4d40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f80c002c8e0 tx=0x7f80c002df90 comp rx=0 tx=0).stop 2024-03-28T17:45:47.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.757+0000 7f80d5600640 1 -- 172.21.15.45:0/1979428951 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80d01048e0 msgr2=0x7f80d0199f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:47.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.757+0000 7f80d5600640 1 --2- 172.21.15.45:0/1979428951 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80d01048e0 0x7f80d0199f80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f80b800e9f0 tx=0x7f80b800eeb0 comp rx=0 tx=0).stop 2024-03-28T17:45:47.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.757+0000 7f80d5600640 1 -- 172.21.15.45:0/1979428951 shutdown_connections 2024-03-28T17:45:47.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.757+0000 7f80d5600640 1 --2- 172.21.15.45:0/1979428951 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80b40a2880 0x7f80b40a4d40 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:47.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.757+0000 7f80d5600640 1 --2- 172.21.15.45:0/1979428951 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80d01052a0 0x7f80d01a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:47.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.757+0000 7f80d5600640 1 --2- 172.21.15.45:0/1979428951 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80d01048e0 0x7f80d0199f80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:47.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.757+0000 7f80d5600640 1 --2- 172.21.15.45:0/1979428951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80d01036f0 0x7f80d0199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:47.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.757+0000 7f80d5600640 1 -- 172.21.15.45:0/1979428951 >> 172.21.15.45:0/1979428951 conn(0x7f80d00fec80 msgr2=0x7f80d0107b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:47.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.757+0000 7f80d5600640 1 -- 172.21.15.45:0/1979428951 shutdown_connections 2024-03-28T17:45:47.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:47.757+0000 7f80d5600640 1 -- 172.21.15.45:0/1979428951 wait complete. 2024-03-28T17:45:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:48 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1979428951' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:48.323 INFO:journalctl@ceph.mds.c.smithi151.stdout:Mar 28 17:45:48 smithi151 systemd[1]: Starting Ceph mds.c for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:45:48.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:48 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1979428951' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:48.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:48 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1979428951' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:48.709 INFO:journalctl@ceph.mds.c.smithi151.stdout:Mar 28 17:45:48 smithi151 podman[66880]: 2024-03-28 17:45:48.349482876 +0000 UTC m=+0.018011892 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:45:48.709 INFO:journalctl@ceph.mds.c.smithi151.stdout:Mar 28 17:45:48 smithi151 podman[66880]: 2024-03-28 17:45:48.46495968 +0000 UTC m=+0.133488697 container create 89a8ca9efd55521cd78fe8fc007e7c254d281fcb19e1a26e4d3fb31544099546 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-c, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-28T17:45:48.709 INFO:journalctl@ceph.mds.c.smithi151.stdout:Mar 28 17:45:48 smithi151 podman[66880]: 2024-03-28 17:45:48.684336567 +0000 UTC m=+0.352865596 container init 89a8ca9efd55521cd78fe8fc007e7c254d281fcb19e1a26e4d3fb31544099546 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-c, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-28T17:45:48.710 INFO:journalctl@ceph.mds.c.smithi151.stdout:Mar 28 17:45:48 smithi151 podman[66880]: 2024-03-28 17:45:48.686959738 +0000 UTC m=+0.355488755 container start 89a8ca9efd55521cd78fe8fc007e7c254d281fcb19e1a26e4d3fb31544099546 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-c, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-28T17:45:49.073 INFO:journalctl@ceph.mds.c.smithi151.stdout:Mar 28 17:45:48 smithi151 ceph-mds[66899]: -- 172.21.15.151:0/1747144946 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55586bccfa40 con 0x55586bca2c00 2024-03-28T17:45:49.073 INFO:journalctl@ceph.mds.c.smithi151.stdout:Mar 28 17:45:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-c[66895]: starting mds.c at 2024-03-28T17:45:49.073 INFO:journalctl@ceph.mds.c.smithi151.stdout:Mar 28 17:45:48 smithi151 bash[66880]: 89a8ca9efd55521cd78fe8fc007e7c254d281fcb19e1a26e4d3fb31544099546 2024-03-28T17:45:49.074 INFO:journalctl@ceph.mds.c.smithi151.stdout:Mar 28 17:45:48 smithi151 systemd[1]: Started Ceph mds.c for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:45:49.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:49 smithi071 ceph-mon[36597]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:49.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:49 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:49.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:49 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:49.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:49 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:49.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:49 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:49.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:49 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:49.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:49 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:49 smithi151 ceph-mon[36481]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:49 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:49 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:49 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:49 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:49 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:49.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:49 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:49.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:49 smithi045 ceph-mon[31773]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:49.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:49 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:49.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:49 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:49.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:49 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:49.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:49 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:49.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:49 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:49.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:49 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:49.777 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:45:50.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.009+0000 7fc1cb200640 1 -- 172.21.15.45:0/972017194 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1cc1048e0 msgr2=0x7fc1cc104d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:50.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.009+0000 7fc1cb200640 1 --2- 172.21.15.45:0/972017194 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1cc1048e0 0x7fc1cc104d60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8009900 tx=0x7fc1b802bc40 comp rx=0 tx=0).stop 2024-03-28T17:45:50.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.009+0000 7fc1cb200640 1 -- 172.21.15.45:0/972017194 shutdown_connections 2024-03-28T17:45:50.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.009+0000 7fc1cb200640 1 --2- 172.21.15.45:0/972017194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1cc1052a0 0x7fc1cc10bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:50.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.009+0000 7fc1cb200640 1 --2- 172.21.15.45:0/972017194 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1cc1048e0 0x7fc1cc104d60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:50.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.009+0000 7fc1cb200640 1 --2- 172.21.15.45:0/972017194 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc1cc1036f0 0x7fc1cc103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:50.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.009+0000 7fc1cb200640 1 -- 172.21.15.45:0/972017194 >> 172.21.15.45:0/972017194 conn(0x7fc1cc0fec80 msgr2=0x7fc1cc1010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:50.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.009+0000 7fc1cb200640 1 -- 172.21.15.45:0/972017194 shutdown_connections 2024-03-28T17:45:50.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.009+0000 7fc1cb200640 1 -- 172.21.15.45:0/972017194 wait complete. 2024-03-28T17:45:50.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.010+0000 7fc1cb200640 1 Processor -- start 2024-03-28T17:45:50.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.010+0000 7fc1cb200640 1 -- start start 2024-03-28T17:45:50.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.011+0000 7fc1cb200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc1cc1036f0 0x7fc1cc1999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:50.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.011+0000 7fc1cb200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1cc1048e0 0x7fc1cc199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:50.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.011+0000 7fc1cb200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1cc1052a0 0x7fc1cc1a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:50.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.011+0000 7fc1cb200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc10f3c0 con 0x7fc1cc1048e0 2024-03-28T17:45:50.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.011+0000 7fc1cb200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc10f240 con 0x7fc1cc1036f0 2024-03-28T17:45:50.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.011+0000 7fc1cb200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc10f540 con 0x7fc1cc1052a0 2024-03-28T17:45:50.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.011+0000 7fc1c9400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1cc1048e0 0x7fc1cc199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:50.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.011+0000 7fc1c9400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1cc1048e0 0x7fc1cc199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53382/0 (socket says 172.21.15.45:53382) 2024-03-28T17:45:50.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.011+0000 7fc1c9400640 1 -- 172.21.15.45:0/587241236 learned_addr learned my addr 172.21.15.45:0/587241236 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:45:50.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.011+0000 7fc1c9e00640 1 --2- 172.21.15.45:0/587241236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc1cc1036f0 0x7fc1cc1999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:50.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.011+0000 7fc1ca800640 1 --2- 172.21.15.45:0/587241236 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1cc1052a0 0x7fc1cc1a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:50.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.012+0000 7fc1c9e00640 1 -- 172.21.15.45:0/587241236 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1cc1052a0 msgr2=0x7fc1cc1a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:50.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.012+0000 7fc1c9e00640 1 --2- 172.21.15.45:0/587241236 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1cc1052a0 0x7fc1cc1a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:50.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.012+0000 7fc1c9e00640 1 -- 172.21.15.45:0/587241236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1cc1048e0 msgr2=0x7fc1cc199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:50.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.012+0000 7fc1c9e00640 1 --2- 172.21.15.45:0/587241236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1cc1048e0 0x7fc1cc199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:50.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.012+0000 7fc1c9e00640 1 -- 172.21.15.45:0/587241236 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1cc1a1600 con 0x7fc1cc1036f0 2024-03-28T17:45:50.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.012+0000 7fc1c9e00640 1 --2- 172.21.15.45:0/587241236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc1cc1036f0 0x7fc1cc1999e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00e9c0 tx=0x7fc1bc00edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:50.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.012+0000 7fc1c2a00640 1 -- 172.21.15.45:0/587241236 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc00ccc0 con 0x7fc1cc1036f0 2024-03-28T17:45:50.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.012+0000 7fc1c2a00640 1 -- 172.21.15.45:0/587241236 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc1bc0035f0 con 0x7fc1cc1036f0 2024-03-28T17:45:50.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.012+0000 7fc1c2a00640 1 -- 172.21.15.45:0/587241236 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc00ce60 con 0x7fc1cc1036f0 2024-03-28T17:45:50.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.012+0000 7fc1cb200640 1 -- 172.21.15.45:0/587241236 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1cc1a18f0 con 0x7fc1cc1036f0 2024-03-28T17:45:50.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.012+0000 7fc1cb200640 1 -- 172.21.15.45:0/587241236 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1cc1a1dc0 con 0x7fc1cc1036f0 2024-03-28T17:45:50.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.013+0000 7fc1cb200640 1 -- 172.21.15.45:0/587241236 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1cc10aa10 con 0x7fc1cc1036f0 2024-03-28T17:45:50.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.015+0000 7fc1c2a00640 1 -- 172.21.15.45:0/587241236 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc1bc0037a0 con 0x7fc1cc1036f0 2024-03-28T17:45:50.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.017+0000 7fc1c2a00640 1 --2- 172.21.15.45:0/587241236 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc1b00a2880 0x7fc1b00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:50.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.017+0000 7fc1c2a00640 1 -- 172.21.15.45:0/587241236 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc1bc0ce910 con 0x7fc1cc1036f0 2024-03-28T17:45:50.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.017+0000 7fc1c9400640 1 --2- 172.21.15.45:0/587241236 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc1b00a2880 0x7fc1b00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:50.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.018+0000 7fc1c9400640 1 --2- 172.21.15.45:0/587241236 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc1b00a2880 0x7fc1b00a4d40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8006010 tx=0x7fc1b802cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:50.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.021+0000 7fc1c2a00640 1 -- 172.21.15.45:0/587241236 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc1bc09bdb0 con 0x7fc1cc1036f0 2024-03-28T17:45:50.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.268+0000 7fc1cb200640 1 -- 172.21.15.45:0/587241236 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1cc02d070 con 0x7fc1cc1036f0 2024-03-28T17:45:50.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.269+0000 7fc1c2a00640 1 -- 172.21.15.45:0/587241236 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6077 (secure 0 0 0) 0x7fc1bc0bb030 con 0x7fc1cc1036f0 2024-03-28T17:45:50.270 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:45:50.270 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6835/2876354410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":2876354410},{"type":"v1","addr":"172.21.15.151:6835","nonce":2876354410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6835/4234998385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":4234998385},{"type":"v1","addr":"172.21.15.71:6835","nonce":4234998385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/1238964845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":1238964845},{"type":"v1","addr":"172.21.15.151:6837","nonce":1238964845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-28T17:45:50.271 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 8 2024-03-28T17:45:50.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.272+0000 7fc1cb200640 1 -- 172.21.15.45:0/587241236 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc1b00a2880 msgr2=0x7fc1b00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:50.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.272+0000 7fc1cb200640 1 --2- 172.21.15.45:0/587241236 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc1b00a2880 0x7fc1b00a4d40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8006010 tx=0x7fc1b802cf90 comp rx=0 tx=0).stop 2024-03-28T17:45:50.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.272+0000 7fc1cb200640 1 -- 172.21.15.45:0/587241236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc1cc1036f0 msgr2=0x7fc1cc1999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:50.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.272+0000 7fc1cb200640 1 --2- 172.21.15.45:0/587241236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc1cc1036f0 0x7fc1cc1999e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00e9c0 tx=0x7fc1bc00edc0 comp rx=0 tx=0).stop 2024-03-28T17:45:50.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.272+0000 7fc1cb200640 1 -- 172.21.15.45:0/587241236 shutdown_connections 2024-03-28T17:45:50.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.272+0000 7fc1cb200640 1 --2- 172.21.15.45:0/587241236 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc1b00a2880 0x7fc1b00a4d40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:50.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.272+0000 7fc1cb200640 1 --2- 172.21.15.45:0/587241236 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc1cc1052a0 0x7fc1cc1a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:50.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.272+0000 7fc1cb200640 1 --2- 172.21.15.45:0/587241236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1cc1048e0 0x7fc1cc199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:50.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.272+0000 7fc1cb200640 1 --2- 172.21.15.45:0/587241236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc1cc1036f0 0x7fc1cc1999e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:50.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.272+0000 7fc1cb200640 1 -- 172.21.15.45:0/587241236 >> 172.21.15.45:0/587241236 conn(0x7fc1cc0fec80 msgr2=0x7fc1cc107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:50.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.272+0000 7fc1cb200640 1 -- 172.21.15.45:0/587241236 shutdown_connections 2024-03-28T17:45:50.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:50.272+0000 7fc1cb200640 1 -- 172.21.15.45:0/587241236 wait complete. 2024-03-28T17:45:50.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:50 smithi071 ceph-mon[36597]: Deploying daemon mds.h on smithi071 2024-03-28T17:45:50.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:50 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6836/1238964845,v1:172.21.15.151:6837/1238964845] up:boot 2024-03-28T17:45:50.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:50 smithi071 ceph-mon[36597]: fsmap 7 up:standby 2024-03-28T17:45:50.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:50 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T17:45:50.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:50 smithi151 ceph-mon[36481]: Deploying daemon mds.h on smithi071 2024-03-28T17:45:50.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:50 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6836/1238964845,v1:172.21.15.151:6837/1238964845] up:boot 2024-03-28T17:45:50.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:50 smithi151 ceph-mon[36481]: fsmap 7 up:standby 2024-03-28T17:45:50.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:50 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T17:45:50.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:50 smithi045 ceph-mon[31773]: Deploying daemon mds.h on smithi071 2024-03-28T17:45:50.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:50 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6836/1238964845,v1:172.21.15.151:6837/1238964845] up:boot 2024-03-28T17:45:50.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:50 smithi045 ceph-mon[31773]: fsmap 7 up:standby 2024-03-28T17:45:50.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:50 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T17:45:51.207 INFO:journalctl@ceph.mds.h.smithi071.stdout:Mar 28 17:45:51 smithi071 systemd[1]: Starting Ceph mds.h for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:45:51.465 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:51 smithi071 ceph-mon[36597]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:51.465 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:51 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/587241236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:51.466 INFO:journalctl@ceph.mds.h.smithi071.stdout:Mar 28 17:45:51 smithi071 podman[66566]: 2024-03-28 17:45:51.205907382 +0000 UTC m=+0.017759162 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:45:51.466 INFO:journalctl@ceph.mds.h.smithi071.stdout:Mar 28 17:45:51 smithi071 podman[66566]: 2024-03-28 17:45:51.333798079 +0000 UTC m=+0.145649804 container create 4961e27f07c59e6b2b285fb7ba1d8115d88c731b7ddd3736ab55992644f6f461 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-h, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T17:45:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:51 smithi151 ceph-mon[36481]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:51 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/587241236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:51.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:51 smithi045 ceph-mon[31773]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:51.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:51 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/587241236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:52.061 INFO:journalctl@ceph.mds.h.smithi071.stdout:Mar 28 17:45:51 smithi071 podman[66566]: 2024-03-28 17:45:51.578572042 +0000 UTC m=+0.390423773 container init 4961e27f07c59e6b2b285fb7ba1d8115d88c731b7ddd3736ab55992644f6f461 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-28T17:45:52.061 INFO:journalctl@ceph.mds.h.smithi071.stdout:Mar 28 17:45:51 smithi071 podman[66566]: 2024-03-28 17:45:51.581398373 +0000 UTC m=+0.393250099 container start 4961e27f07c59e6b2b285fb7ba1d8115d88c731b7ddd3736ab55992644f6f461 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-h, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T17:45:52.061 INFO:journalctl@ceph.mds.h.smithi071.stdout:Mar 28 17:45:51 smithi071 ceph-mds[66585]: -- 172.21.15.71:0/1918740109 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5644ebd29a40 con 0x5644ebcfcc00 2024-03-28T17:45:52.061 INFO:journalctl@ceph.mds.h.smithi071.stdout:Mar 28 17:45:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-h[66581]: starting mds.h at 2024-03-28T17:45:52.061 INFO:journalctl@ceph.mds.h.smithi071.stdout:Mar 28 17:45:51 smithi071 bash[66566]: 4961e27f07c59e6b2b285fb7ba1d8115d88c731b7ddd3736ab55992644f6f461 2024-03-28T17:45:52.061 INFO:journalctl@ceph.mds.h.smithi071.stdout:Mar 28 17:45:51 smithi071 systemd[1]: Started Ceph mds.h for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:45:52.292 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:45:52.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.526+0000 7fd150c00640 1 -- 172.21.15.45:0/3703106688 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd14c1038e0 msgr2=0x7fd14c103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:52.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.526+0000 7fd150c00640 1 --2- 172.21.15.45:0/3703106688 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd14c1038e0 0x7fd14c103d00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009900 tx=0x7fd13c02bc30 comp rx=0 tx=0).stop 2024-03-28T17:45:52.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.526+0000 7fd150c00640 1 -- 172.21.15.45:0/3703106688 shutdown_connections 2024-03-28T17:45:52.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.526+0000 7fd150c00640 1 --2- 172.21.15.45:0/3703106688 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd14c105490 0x7fd14c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:52.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.526+0000 7fd150c00640 1 --2- 172.21.15.45:0/3703106688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd14c104ad0 0x7fd14c104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:52.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.526+0000 7fd150c00640 1 --2- 172.21.15.45:0/3703106688 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd14c1038e0 0x7fd14c103d00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:52.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.526+0000 7fd150c00640 1 -- 172.21.15.45:0/3703106688 >> 172.21.15.45:0/3703106688 conn(0x7fd14c0fee50 msgr2=0x7fd14c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:52.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.526+0000 7fd150c00640 1 -- 172.21.15.45:0/3703106688 shutdown_connections 2024-03-28T17:45:52.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.526+0000 7fd150c00640 1 -- 172.21.15.45:0/3703106688 wait complete. 2024-03-28T17:45:52.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.527+0000 7fd150c00640 1 Processor -- start 2024-03-28T17:45:52.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.528+0000 7fd150c00640 1 -- start start 2024-03-28T17:45:52.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.528+0000 7fd150c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd14c1038e0 0x7fd14c199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:52.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.528+0000 7fd150c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd14c104ad0 0x7fd14c19a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:52.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.528+0000 7fd150c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd14c105490 0x7fd14c1a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:52.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.528+0000 7fd150c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c10f580 con 0x7fd14c105490 2024-03-28T17:45:52.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.528+0000 7fd150c00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c10f400 con 0x7fd14c104ad0 2024-03-28T17:45:52.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.528+0000 7fd150c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c10f700 con 0x7fd14c1038e0 2024-03-28T17:45:52.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.528+0000 7fd149e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd14c1038e0 0x7fd14c199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:52.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.528+0000 7fd149e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd14c1038e0 0x7fd14c199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:54208/0 (socket says 172.21.15.45:54208) 2024-03-28T17:45:52.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.528+0000 7fd149e00640 1 -- 172.21.15.45:0/3811230617 learned_addr learned my addr 172.21.15.45:0/3811230617 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:45:52.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.528+0000 7fd149400640 1 --2- 172.21.15.45:0/3811230617 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd14c104ad0 0x7fd14c19a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:52.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.528+0000 7fd14a800640 1 --2- 172.21.15.45:0/3811230617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd14c105490 0x7fd14c1a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:52.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.528+0000 7fd149e00640 1 -- 172.21.15.45:0/3811230617 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd14c104ad0 msgr2=0x7fd14c19a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:52.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.528+0000 7fd149e00640 1 --2- 172.21.15.45:0/3811230617 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd14c104ad0 0x7fd14c19a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:52.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.528+0000 7fd149e00640 1 -- 172.21.15.45:0/3811230617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd14c105490 msgr2=0x7fd14c1a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:52.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.528+0000 7fd149e00640 1 --2- 172.21.15.45:0/3811230617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd14c105490 0x7fd14c1a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:52.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.528+0000 7fd149e00640 1 -- 172.21.15.45:0/3811230617 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd14c1a17f0 con 0x7fd14c1038e0 2024-03-28T17:45:52.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.528+0000 7fd14a800640 1 --2- 172.21.15.45:0/3811230617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd14c105490 0x7fd14c1a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:45:52.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.529+0000 7fd149e00640 1 --2- 172.21.15.45:0/3811230617 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd14c1038e0 0x7fd14c199bd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd13c000c00 tx=0x7fd13c02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:52.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.529+0000 7fd142a00640 1 -- 172.21.15.45:0/3811230617 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c043020 con 0x7fd14c1038e0 2024-03-28T17:45:52.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.529+0000 7fd142a00640 1 -- 172.21.15.45:0/3811230617 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd13c0054e0 con 0x7fd14c1038e0 2024-03-28T17:45:52.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.529+0000 7fd150c00640 1 -- 172.21.15.45:0/3811230617 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd14c1a1a80 con 0x7fd14c1038e0 2024-03-28T17:45:52.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.529+0000 7fd142a00640 1 -- 172.21.15.45:0/3811230617 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c003c40 con 0x7fd14c1038e0 2024-03-28T17:45:52.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.529+0000 7fd150c00640 1 -- 172.21.15.45:0/3811230617 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd14c1a2020 con 0x7fd14c1038e0 2024-03-28T17:45:52.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.531+0000 7fd150c00640 1 -- 172.21.15.45:0/3811230617 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd14c10ac00 con 0x7fd14c1038e0 2024-03-28T17:45:52.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.532+0000 7fd142a00640 1 -- 172.21.15.45:0/3811230617 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd13c046020 con 0x7fd14c1038e0 2024-03-28T17:45:52.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.534+0000 7fd142a00640 1 --2- 172.21.15.45:0/3811230617 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd1300a2880 0x7fd1300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:52.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.534+0000 7fd142a00640 1 -- 172.21.15.45:0/3811230617 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd13c0f0670 con 0x7fd14c1038e0 2024-03-28T17:45:52.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.534+0000 7fd149400640 1 --2- 172.21.15.45:0/3811230617 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd1300a2880 0x7fd1300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:52.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.534+0000 7fd149400640 1 --2- 172.21.15.45:0/3811230617 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd1300a2880 0x7fd1300a4d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd1380095f0 tx=0x7fd138005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:52.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.539+0000 7fd142a00640 1 -- 172.21.15.45:0/3811230617 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd13c04d020 con 0x7fd14c1038e0 2024-03-28T17:45:52.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.786+0000 7fd150c00640 1 -- 172.21.15.45:0/3811230617 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd14c02d070 con 0x7fd14c1038e0 2024-03-28T17:45:52.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.787+0000 7fd142a00640 1 -- 172.21.15.45:0/3811230617 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6868 (secure 0 0 0) 0x7fd13c0bdb10 con 0x7fd14c1038e0 2024-03-28T17:45:52.788 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:45:52.788 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6835/2876354410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":2876354410},{"type":"v1","addr":"172.21.15.151:6835","nonce":2876354410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6835/4234998385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":4234998385},{"type":"v1","addr":"172.21.15.71:6835","nonce":4234998385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/1238964845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":1238964845},{"type":"v1","addr":"172.21.15.151:6837","nonce":1238964845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/2008752655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2008752655},{"type":"v1","addr":"172.21.15.71:6837","nonce":2008752655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-28T17:45:52.788 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 9 2024-03-28T17:45:52.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.790+0000 7fd150c00640 1 -- 172.21.15.45:0/3811230617 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd1300a2880 msgr2=0x7fd1300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:52.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.790+0000 7fd150c00640 1 --2- 172.21.15.45:0/3811230617 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd1300a2880 0x7fd1300a4d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd1380095f0 tx=0x7fd138005f90 comp rx=0 tx=0).stop 2024-03-28T17:45:52.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.790+0000 7fd150c00640 1 -- 172.21.15.45:0/3811230617 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd14c1038e0 msgr2=0x7fd14c199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:52.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.790+0000 7fd150c00640 1 --2- 172.21.15.45:0/3811230617 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd14c1038e0 0x7fd14c199bd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd13c000c00 tx=0x7fd13c02c650 comp rx=0 tx=0).stop 2024-03-28T17:45:52.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.790+0000 7fd150c00640 1 -- 172.21.15.45:0/3811230617 shutdown_connections 2024-03-28T17:45:52.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.790+0000 7fd150c00640 1 --2- 172.21.15.45:0/3811230617 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd1300a2880 0x7fd1300a4d40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:52.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.790+0000 7fd150c00640 1 --2- 172.21.15.45:0/3811230617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd14c105490 0x7fd14c1a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:52.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.790+0000 7fd150c00640 1 --2- 172.21.15.45:0/3811230617 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd14c104ad0 0x7fd14c19a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:52.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.790+0000 7fd150c00640 1 --2- 172.21.15.45:0/3811230617 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd14c1038e0 0x7fd14c199bd0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:52.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.790+0000 7fd150c00640 1 -- 172.21.15.45:0/3811230617 >> 172.21.15.45:0/3811230617 conn(0x7fd14c0fee50 msgr2=0x7fd14c107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:52.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.790+0000 7fd150c00640 1 -- 172.21.15.45:0/3811230617 shutdown_connections 2024-03-28T17:45:52.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:52.790+0000 7fd150c00640 1 -- 172.21.15.45:0/3811230617 wait complete. 2024-03-28T17:45:53.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:52 smithi071 ceph-mon[36597]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:53.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:52 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:53.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:52 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:53.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:52 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:53.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:52 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:53.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:52 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:53.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:52 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:53.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:52 smithi071 ceph-mon[36597]: Deploying daemon mds.i on smithi151 2024-03-28T17:45:53.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:52 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] up:boot 2024-03-28T17:45:53.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:52 smithi071 ceph-mon[36597]: fsmap 8 up:standby 2024-03-28T17:45:53.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:52 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T17:45:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:52 smithi151 ceph-mon[36481]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:52 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:52 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:52 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:52 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:52 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:52 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:53.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:52 smithi151 ceph-mon[36481]: Deploying daemon mds.i on smithi151 2024-03-28T17:45:53.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:52 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] up:boot 2024-03-28T17:45:53.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:52 smithi151 ceph-mon[36481]: fsmap 8 up:standby 2024-03-28T17:45:53.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:52 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T17:45:53.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:52 smithi045 ceph-mon[31773]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:53.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:52 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:53.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:52 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:53.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:52 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:53.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:52 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:53.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:52 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:53.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:52 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:53.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:52 smithi045 ceph-mon[31773]: Deploying daemon mds.i on smithi151 2024-03-28T17:45:53.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:52 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] up:boot 2024-03-28T17:45:53.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:52 smithi045 ceph-mon[31773]: fsmap 8 up:standby 2024-03-28T17:45:53.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:52 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T17:45:54.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:53 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3811230617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:54.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:53 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3811230617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:54.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:53 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3811230617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:54.812 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:45:54.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:54 smithi151 ceph-mon[36481]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:55.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.031+0000 7f807e000640 1 -- 172.21.15.45:0/1932564982 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80780fff50 msgr2=0x7f8078100370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:55.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.031+0000 7f807e000640 1 --2- 172.21.15.45:0/1932564982 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80780fff50 0x7f8078100370 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8068009900 tx=0x7f806802bc30 comp rx=0 tx=0).stop 2024-03-28T17:45:55.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.031+0000 7f807e000640 1 -- 172.21.15.45:0/1932564982 shutdown_connections 2024-03-28T17:45:55.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.031+0000 7f807e000640 1 --2- 172.21.15.45:0/1932564982 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8078101250 0x7f80781016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:55.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.031+0000 7f807e000640 1 --2- 172.21.15.45:0/1932564982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80781008b0 0x7f8078100d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:55.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.031+0000 7f807e000640 1 --2- 172.21.15.45:0/1932564982 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80780fff50 0x7f8078100370 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:55.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.031+0000 7f807e000640 1 -- 172.21.15.45:0/1932564982 >> 172.21.15.45:0/1932564982 conn(0x7f80780fbb50 msgr2=0x7f80780fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:55.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.032+0000 7f807e000640 1 -- 172.21.15.45:0/1932564982 shutdown_connections 2024-03-28T17:45:55.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.032+0000 7f807e000640 1 -- 172.21.15.45:0/1932564982 wait complete. 2024-03-28T17:45:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.033+0000 7f807e000640 1 Processor -- start 2024-03-28T17:45:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.033+0000 7f807e000640 1 -- start start 2024-03-28T17:45:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.033+0000 7f807e000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80780fff50 0x7f8078199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.033+0000 7f807e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80781008b0 0x7f807819a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.033+0000 7f807e000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8078101250 0x7f80781a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:55.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.033+0000 7f807e000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807810f590 con 0x7f80781008b0 2024-03-28T17:45:55.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.033+0000 7f807e000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807810f410 con 0x7f80780fff50 2024-03-28T17:45:55.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.033+0000 7f807e000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807810f710 con 0x7f8078101250 2024-03-28T17:45:55.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.033+0000 7f806fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80781008b0 0x7f807819a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:55.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.033+0000 7f8077e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8078101250 0x7f80781a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:55.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.033+0000 7f806fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80781008b0 0x7f807819a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53398/0 (socket says 172.21.15.45:53398) 2024-03-28T17:45:55.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.033+0000 7f806fe00640 1 -- 172.21.15.45:0/2809926696 learned_addr learned my addr 172.21.15.45:0/2809926696 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:45:55.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.033+0000 7f8077400640 1 --2- 172.21.15.45:0/2809926696 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80780fff50 0x7f8078199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:55.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.034+0000 7f8077e00640 1 -- 172.21.15.45:0/2809926696 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80780fff50 msgr2=0x7f8078199bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:55.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.034+0000 7f8077e00640 1 --2- 172.21.15.45:0/2809926696 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80780fff50 0x7f8078199bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:55.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.034+0000 7f8077e00640 1 -- 172.21.15.45:0/2809926696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80781008b0 msgr2=0x7f807819a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:55.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.034+0000 7f8077e00640 1 --2- 172.21.15.45:0/2809926696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80781008b0 0x7f807819a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:55.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.034+0000 7f8077e00640 1 -- 172.21.15.45:0/2809926696 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80781a17d0 con 0x7f8078101250 2024-03-28T17:45:55.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.034+0000 7f8077e00640 1 --2- 172.21.15.45:0/2809926696 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8078101250 0x7f80781a1160 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8070009fd0 tx=0x7f807000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:55.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.034+0000 7f8074c00640 1 -- 172.21.15.45:0/2809926696 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807000ce20 con 0x7f8078101250 2024-03-28T17:45:55.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.034+0000 7f8074c00640 1 -- 172.21.15.45:0/2809926696 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8070005020 con 0x7f8078101250 2024-03-28T17:45:55.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.034+0000 7f8074c00640 1 -- 172.21.15.45:0/2809926696 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8070010720 con 0x7f8078101250 2024-03-28T17:45:55.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.034+0000 7f807e000640 1 -- 172.21.15.45:0/2809926696 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80781a1ac0 con 0x7f8078101250 2024-03-28T17:45:55.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.034+0000 7f807e000640 1 -- 172.21.15.45:0/2809926696 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80781a2060 con 0x7f8078101250 2024-03-28T17:45:55.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.036+0000 7f807e000640 1 -- 172.21.15.45:0/2809926696 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f807810adf0 con 0x7f8078101250 2024-03-28T17:45:55.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.037+0000 7f8074c00640 1 -- 172.21.15.45:0/2809926696 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8070005540 con 0x7f8078101250 2024-03-28T17:45:55.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.038+0000 7f8074c00640 1 --2- 172.21.15.45:0/2809926696 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f805c0a2880 0x7f805c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:55.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.038+0000 7f8074c00640 1 -- 172.21.15.45:0/2809926696 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f80700ceb40 con 0x7f8078101250 2024-03-28T17:45:55.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.039+0000 7f8077400640 1 --2- 172.21.15.45:0/2809926696 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f805c0a2880 0x7f805c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:55.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.039+0000 7f8077400640 1 --2- 172.21.15.45:0/2809926696 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f805c0a2880 0x7f805c0a4d40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f806802c240 tx=0x7f806802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:55.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.042+0000 7f8074c00640 1 -- 172.21.15.45:0/2809926696 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f807009c080 con 0x7f8078101250 2024-03-28T17:45:55.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:54 smithi071 ceph-mon[36597]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:55.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:54 smithi045 ceph-mon[31773]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:55.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.287+0000 7f807e000640 1 -- 172.21.15.45:0/2809926696 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f807802d220 con 0x7f8078101250 2024-03-28T17:45:55.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.288+0000 7f8074c00640 1 -- 172.21.15.45:0/2809926696 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7664 (secure 0 0 0) 0x7f807009b980 con 0x7f8078101250 2024-03-28T17:45:55.288 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:45:55.288 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6835/2876354410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":2876354410},{"type":"v1","addr":"172.21.15.151:6835","nonce":2876354410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6835/4234998385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":4234998385},{"type":"v1","addr":"172.21.15.71:6835","nonce":4234998385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/1238964845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":1238964845},{"type":"v1","addr":"172.21.15.151:6837","nonce":1238964845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24529,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6839/3003564901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6838","nonce":3003564901},{"type":"v1","addr":"172.21.15.151:6839","nonce":3003564901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/2008752655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2008752655},{"type":"v1","addr":"172.21.15.71:6837","nonce":2008752655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-28T17:45:55.289 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 10 2024-03-28T17:45:55.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.290+0000 7f807e000640 1 -- 172.21.15.45:0/2809926696 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f805c0a2880 msgr2=0x7f805c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:55.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.290+0000 7f807e000640 1 --2- 172.21.15.45:0/2809926696 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f805c0a2880 0x7f805c0a4d40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f806802c240 tx=0x7f806802df90 comp rx=0 tx=0).stop 2024-03-28T17:45:55.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.290+0000 7f807e000640 1 -- 172.21.15.45:0/2809926696 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8078101250 msgr2=0x7f80781a1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:55.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.290+0000 7f807e000640 1 --2- 172.21.15.45:0/2809926696 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8078101250 0x7f80781a1160 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8070009fd0 tx=0x7f807000ef10 comp rx=0 tx=0).stop 2024-03-28T17:45:55.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.291+0000 7f807e000640 1 -- 172.21.15.45:0/2809926696 shutdown_connections 2024-03-28T17:45:55.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.291+0000 7f807e000640 1 --2- 172.21.15.45:0/2809926696 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f805c0a2880 0x7f805c0a4d40 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:55.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.291+0000 7f807e000640 1 --2- 172.21.15.45:0/2809926696 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8078101250 0x7f80781a1160 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:55.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.291+0000 7f807e000640 1 --2- 172.21.15.45:0/2809926696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80781008b0 0x7f807819a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:55.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.291+0000 7f807e000640 1 --2- 172.21.15.45:0/2809926696 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80780fff50 0x7f8078199bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:55.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.291+0000 7f807e000640 1 -- 172.21.15.45:0/2809926696 >> 172.21.15.45:0/2809926696 conn(0x7f80780fbb50 msgr2=0x7f80780fd230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:55.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.291+0000 7f807e000640 1 -- 172.21.15.45:0/2809926696 shutdown_connections 2024-03-28T17:45:55.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:55.291+0000 7f807e000640 1 -- 172.21.15.45:0/2809926696 wait complete. 2024-03-28T17:45:56.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:55 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] up:boot 2024-03-28T17:45:56.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:55 smithi045 ceph-mon[31773]: fsmap 9 up:standby 2024-03-28T17:45:56.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:55 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T17:45:56.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:55 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:56.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:55 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:56.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:55 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:56.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:55 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:56.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:55 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:56.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:55 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:56.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:55 smithi045 ceph-mon[31773]: Deploying daemon mds.g on smithi045 2024-03-28T17:45:56.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:55 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/2809926696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:56.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:55 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] up:boot 2024-03-28T17:45:56.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:55 smithi071 ceph-mon[36597]: fsmap 9 up:standby 2024-03-28T17:45:56.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:55 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T17:45:56.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:55 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:56.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:55 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:56.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:55 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:56.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:55 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:56.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:55 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:56.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:55 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:56.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:55 smithi071 ceph-mon[36597]: Deploying daemon mds.g on smithi045 2024-03-28T17:45:56.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:55 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/2809926696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:56.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:55 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] up:boot 2024-03-28T17:45:56.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:55 smithi151 ceph-mon[36481]: fsmap 9 up:standby 2024-03-28T17:45:56.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:55 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T17:45:56.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:55 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:56.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:55 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:56.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:55 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:56.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:55 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:56.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:55 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:56.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:55 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:56.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:55 smithi151 ceph-mon[36481]: Deploying daemon mds.g on smithi045 2024-03-28T17:45:56.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:55 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/2809926696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:56 smithi151 ceph-mon[36481]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:57.140 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:56 smithi045 ceph-mon[31773]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:56 smithi071 ceph-mon[36597]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:57.321 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:45:57.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.728+0000 7f2680e00640 1 -- 172.21.15.45:0/3532154003 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f267c105490 msgr2=0x7f267c10bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:57.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.728+0000 7f2680e00640 1 --2- 172.21.15.45:0/3532154003 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f267c105490 0x7f267c10bd20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2674009ca0 tx=0x7f267402c140 comp rx=0 tx=0).stop 2024-03-28T17:45:57.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.728+0000 7f2680e00640 1 -- 172.21.15.45:0/3532154003 shutdown_connections 2024-03-28T17:45:57.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.728+0000 7f2680e00640 1 --2- 172.21.15.45:0/3532154003 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f267c105490 0x7f267c10bd20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:57.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.728+0000 7f2680e00640 1 --2- 172.21.15.45:0/3532154003 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f267c104ad0 0x7f267c104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:57.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.728+0000 7f2680e00640 1 --2- 172.21.15.45:0/3532154003 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f267c1038e0 0x7f267c103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:57.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.728+0000 7f2680e00640 1 -- 172.21.15.45:0/3532154003 >> 172.21.15.45:0/3532154003 conn(0x7f267c0fee50 msgr2=0x7f267c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:57.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.729+0000 7f2680e00640 1 -- 172.21.15.45:0/3532154003 shutdown_connections 2024-03-28T17:45:57.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.729+0000 7f2680e00640 1 -- 172.21.15.45:0/3532154003 wait complete. 2024-03-28T17:45:57.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.730+0000 7f2680e00640 1 Processor -- start 2024-03-28T17:45:57.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.730+0000 7f2680e00640 1 -- start start 2024-03-28T17:45:57.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.731+0000 7f2680e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f267c1038e0 0x7f267c19df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:57.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.731+0000 7f2680e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f267c104ad0 0x7f267c19e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:57.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.731+0000 7f2680e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f267c105490 0x7f267c1a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:57.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.731+0000 7f2680e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267c10f540 con 0x7f267c105490 2024-03-28T17:45:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.731+0000 7f2680e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267c10f3c0 con 0x7f267c1038e0 2024-03-28T17:45:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.731+0000 7f2680e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267c10f6c0 con 0x7f267c104ad0 2024-03-28T17:45:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.731+0000 7f267a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f267c105490 0x7f267c1a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.731+0000 7f267a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f267c105490 0x7f267c1a5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34832/0 (socket says 172.21.15.45:34832) 2024-03-28T17:45:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.731+0000 7f267a800640 1 -- 172.21.15.45:0/2459323034 learned_addr learned my addr 172.21.15.45:0/2459323034 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:45:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.732+0000 7f2679e00640 1 --2- 172.21.15.45:0/2459323034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f267c1038e0 0x7f267c19df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.732+0000 7f267a800640 1 -- 172.21.15.45:0/2459323034 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f267c104ad0 msgr2=0x7f267c19e4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T17:45:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.732+0000 7f267a800640 1 --2- 172.21.15.45:0/2459323034 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f267c104ad0 0x7f267c19e4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.732+0000 7f267a800640 1 -- 172.21.15.45:0/2459323034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f267c1038e0 msgr2=0x7f267c19df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:57.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.734+0000 7f267a800640 1 --2- 172.21.15.45:0/2459323034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f267c1038e0 0x7f267c19df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:57.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.734+0000 7f267a800640 1 -- 172.21.15.45:0/2459323034 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f267c1a5bb0 con 0x7f267c105490 2024-03-28T17:45:57.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.735+0000 7f2679e00640 1 --2- 172.21.15.45:0/2459323034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f267c1038e0 0x7f267c19df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:45:57.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.735+0000 7f267a800640 1 --2- 172.21.15.45:0/2459323034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f267c105490 0x7f267c1a5540 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2674037040 tx=0x7f2674032340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:57.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.736+0000 7f2672a00640 1 -- 172.21.15.45:0/2459323034 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2674041020 con 0x7f267c105490 2024-03-28T17:45:57.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.736+0000 7f2680e00640 1 -- 172.21.15.45:0/2459323034 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f267c1a5e40 con 0x7f267c105490 2024-03-28T17:45:57.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.736+0000 7f2680e00640 1 -- 172.21.15.45:0/2459323034 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f267c1a6300 con 0x7f267c105490 2024-03-28T17:45:57.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.736+0000 7f2672a00640 1 -- 172.21.15.45:0/2459323034 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26740329b0 con 0x7f267c105490 2024-03-28T17:45:57.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.736+0000 7f2672a00640 1 -- 172.21.15.45:0/2459323034 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2674039040 con 0x7f267c105490 2024-03-28T17:45:57.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.738+0000 7f2680e00640 1 -- 172.21.15.45:0/2459323034 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f265c0052c0 con 0x7f267c105490 2024-03-28T17:45:57.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.742+0000 7f2672a00640 1 -- 172.21.15.45:0/2459323034 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2674044020 con 0x7f267c105490 2024-03-28T17:45:57.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.744+0000 7f2672a00640 1 --2- 172.21.15.45:0/2459323034 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f26600a2880 0x7f26600a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:45:57.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.744+0000 7f2672a00640 1 -- 172.21.15.45:0/2459323034 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f26740f0720 con 0x7f267c105490 2024-03-28T17:45:57.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.745+0000 7f2679e00640 1 --2- 172.21.15.45:0/2459323034 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f26600a2880 0x7f26600a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:45:57.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.746+0000 7f2679e00640 1 --2- 172.21.15.45:0/2459323034 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f26600a2880 0x7f26600a4d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f267c104930 tx=0x7f266c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:45:57.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.751+0000 7f2672a00640 1 -- 172.21.15.45:0/2459323034 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f267404b020 con 0x7f267c105490 2024-03-28T17:45:57.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.998+0000 7f2680e00640 1 -- 172.21.15.45:0/2459323034 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f265c0055b0 con 0x7f267c105490 2024-03-28T17:45:57.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:57.999+0000 7f2672a00640 1 -- 172.21.15.45:0/2459323034 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7664 (secure 0 0 0) 0x7f267403b2f0 con 0x7f267c105490 2024-03-28T17:45:57.999 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:45:57.999 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6835/2876354410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":2876354410},{"type":"v1","addr":"172.21.15.151:6835","nonce":2876354410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6835/4234998385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":4234998385},{"type":"v1","addr":"172.21.15.71:6835","nonce":4234998385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/1238964845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":1238964845},{"type":"v1","addr":"172.21.15.151:6837","nonce":1238964845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24529,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6839/3003564901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6838","nonce":3003564901},{"type":"v1","addr":"172.21.15.151:6839","nonce":3003564901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/2008752655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2008752655},{"type":"v1","addr":"172.21.15.71:6837","nonce":2008752655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-28T17:45:57.999 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 10 2024-03-28T17:45:58.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:58.001+0000 7f2680e00640 1 -- 172.21.15.45:0/2459323034 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f26600a2880 msgr2=0x7f26600a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:58.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:58.001+0000 7f2680e00640 1 --2- 172.21.15.45:0/2459323034 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f26600a2880 0x7f26600a4d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f267c104930 tx=0x7f266c005f90 comp rx=0 tx=0).stop 2024-03-28T17:45:58.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:58.001+0000 7f2680e00640 1 -- 172.21.15.45:0/2459323034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f267c105490 msgr2=0x7f267c1a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:45:58.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:58.001+0000 7f2680e00640 1 --2- 172.21.15.45:0/2459323034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f267c105490 0x7f267c1a5540 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2674037040 tx=0x7f2674032340 comp rx=0 tx=0).stop 2024-03-28T17:45:58.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:58.001+0000 7f2680e00640 1 -- 172.21.15.45:0/2459323034 shutdown_connections 2024-03-28T17:45:58.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:58.001+0000 7f2680e00640 1 --2- 172.21.15.45:0/2459323034 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f26600a2880 0x7f26600a4d40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:58.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:58.001+0000 7f2680e00640 1 --2- 172.21.15.45:0/2459323034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f267c105490 0x7f267c1a5540 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:58.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:58.001+0000 7f2680e00640 1 --2- 172.21.15.45:0/2459323034 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f267c104ad0 0x7f267c19e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:58.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:58.001+0000 7f2680e00640 1 --2- 172.21.15.45:0/2459323034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f267c1038e0 0x7f267c19df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:45:58.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:58.001+0000 7f2680e00640 1 -- 172.21.15.45:0/2459323034 >> 172.21.15.45:0/2459323034 conn(0x7f267c0fee50 msgr2=0x7f267c077af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:45:58.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:58.001+0000 7f2680e00640 1 -- 172.21.15.45:0/2459323034 shutdown_connections 2024-03-28T17:45:58.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:45:58.001+0000 7f2680e00640 1 -- 172.21.15.45:0/2459323034 wait complete. 2024-03-28T17:45:59.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:58 smithi045 ceph-mon[31773]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:59.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:58 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/2459323034' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:59.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:58 smithi071 ceph-mon[36597]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:59.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:58 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/2459323034' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:58 smithi151 ceph-mon[36481]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:45:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:58 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/2459323034' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:45:59.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:59 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6838/2313500150,v1:172.21.15.45:6839/2313500150] up:boot 2024-03-28T17:45:59.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:59 smithi045 ceph-mon[31773]: fsmap 10 up:standby 2024-03-28T17:45:59.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T17:45:59.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:59.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:59.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:45:59.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:45:59.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:45:59.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:59 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:45:59.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:45:59 smithi045 ceph-mon[31773]: Deploying daemon mds.a on smithi045 2024-03-28T17:46:00.022 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:46:00.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.255+0000 7ff874800640 1 -- 172.21.15.45:0/2098130212 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff86c0772a0 msgr2=0x7ff86c077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:00.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.255+0000 7ff874800640 1 --2- 172.21.15.45:0/2098130212 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff86c0772a0 0x7ff86c077700 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff85c009900 tx=0x7ff85c02bc40 comp rx=0 tx=0).stop 2024-03-28T17:46:00.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.255+0000 7ff874800640 1 -- 172.21.15.45:0/2098130212 shutdown_connections 2024-03-28T17:46:00.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.255+0000 7ff874800640 1 --2- 172.21.15.45:0/2098130212 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff86c077c40 0x7ff86c0784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:00.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.255+0000 7ff874800640 1 --2- 172.21.15.45:0/2098130212 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff86c0772a0 0x7ff86c077700 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:00.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.255+0000 7ff874800640 1 --2- 172.21.15.45:0/2098130212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff86c076060 0x7ff86c076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:00.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.255+0000 7ff874800640 1 -- 172.21.15.45:0/2098130212 >> 172.21.15.45:0/2098130212 conn(0x7ff86c0fee90 msgr2=0x7ff86c1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:00.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.255+0000 7ff874800640 1 -- 172.21.15.45:0/2098130212 shutdown_connections 2024-03-28T17:46:00.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.255+0000 7ff874800640 1 -- 172.21.15.45:0/2098130212 wait complete. 2024-03-28T17:46:00.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.257+0000 7ff874800640 1 Processor -- start 2024-03-28T17:46:00.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.257+0000 7ff874800640 1 -- start start 2024-03-28T17:46:00.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.257+0000 7ff874800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff86c076060 0x7ff86c19df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:00.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.257+0000 7ff874800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff86c0772a0 0x7ff86c19e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:00.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.257+0000 7ff874800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff86c077c40 0x7ff86c1a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:00.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.257+0000 7ff874800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c1139b0 con 0x7ff86c0772a0 2024-03-28T17:46:00.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.257+0000 7ff874800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c113830 con 0x7ff86c077c40 2024-03-28T17:46:00.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.257+0000 7ff874800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c113b30 con 0x7ff86c076060 2024-03-28T17:46:00.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.258+0000 7ff871400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff86c0772a0 0x7ff86c19e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:00.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.258+0000 7ff871e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff86c076060 0x7ff86c19df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:00.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.258+0000 7ff872800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff86c077c40 0x7ff86c1a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:00.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.258+0000 7ff871e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff86c076060 0x7ff86c19df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:52958/0 (socket says 172.21.15.45:52958) 2024-03-28T17:46:00.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.258+0000 7ff871e00640 1 -- 172.21.15.45:0/621353243 learned_addr learned my addr 172.21.15.45:0/621353243 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:00.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.258+0000 7ff872800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff86c077c40 0x7ff86c1a5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53076/0 (socket says 172.21.15.45:53076) 2024-03-28T17:46:00.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.258+0000 7ff872800640 1 -- 172.21.15.45:0/621353243 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff86c076060 msgr2=0x7ff86c19df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:00.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.258+0000 7ff872800640 1 --2- 172.21.15.45:0/621353243 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff86c076060 0x7ff86c19df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:00.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.258+0000 7ff872800640 1 -- 172.21.15.45:0/621353243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff86c0772a0 msgr2=0x7ff86c19e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:00.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.258+0000 7ff872800640 1 --2- 172.21.15.45:0/621353243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff86c0772a0 0x7ff86c19e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:00.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.258+0000 7ff872800640 1 -- 172.21.15.45:0/621353243 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86c1a5ba0 con 0x7ff86c077c40 2024-03-28T17:46:00.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.259+0000 7ff872800640 1 --2- 172.21.15.45:0/621353243 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff86c077c40 0x7ff86c1a5530 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff86800e940 tx=0x7ff86800ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:00.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.259+0000 7ff866a00640 1 -- 172.21.15.45:0/621353243 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff868004d20 con 0x7ff86c077c40 2024-03-28T17:46:00.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.259+0000 7ff866a00640 1 -- 172.21.15.45:0/621353243 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff868005020 con 0x7ff86c077c40 2024-03-28T17:46:00.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.259+0000 7ff866a00640 1 -- 172.21.15.45:0/621353243 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8680116b0 con 0x7ff86c077c40 2024-03-28T17:46:00.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.259+0000 7ff874800640 1 -- 172.21.15.45:0/621353243 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff86c1a5e90 con 0x7ff86c077c40 2024-03-28T17:46:00.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.259+0000 7ff874800640 1 -- 172.21.15.45:0/621353243 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff86c1a6350 con 0x7ff86c077c40 2024-03-28T17:46:00.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.260+0000 7ff874800640 1 -- 172.21.15.45:0/621353243 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff86c110390 con 0x7ff86c077c40 2024-03-28T17:46:00.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.262+0000 7ff866a00640 1 -- 172.21.15.45:0/621353243 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff8680054d0 con 0x7ff86c077c40 2024-03-28T17:46:00.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.263+0000 7ff866a00640 1 --2- 172.21.15.45:0/621353243 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff8540a2880 0x7ff8540a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:00.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.263+0000 7ff866a00640 1 -- 172.21.15.45:0/621353243 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff8680ce600 con 0x7ff86c077c40 2024-03-28T17:46:00.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.263+0000 7ff871e00640 1 --2- 172.21.15.45:0/621353243 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff8540a2880 0x7ff8540a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:00.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.264+0000 7ff871e00640 1 --2- 172.21.15.45:0/621353243 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff8540a2880 0x7ff8540a4d40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff860005fd0 tx=0x7ff86000b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:00.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.268+0000 7ff866a00640 1 -- 172.21.15.45:0/621353243 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff868016030 con 0x7ff86c077c40 2024-03-28T17:46:00.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:59 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6838/2313500150,v1:172.21.15.45:6839/2313500150] up:boot 2024-03-28T17:46:00.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:59 smithi071 ceph-mon[36597]: fsmap 10 up:standby 2024-03-28T17:46:00.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:59 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T17:46:00.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:59 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:00.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:59 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:00.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:59 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:00.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:59 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:46:00.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:59 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:46:00.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:59 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:46:00.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:45:59 smithi071 ceph-mon[36597]: Deploying daemon mds.a on smithi045 2024-03-28T17:46:00.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:59 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6838/2313500150,v1:172.21.15.45:6839/2313500150] up:boot 2024-03-28T17:46:00.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:59 smithi151 ceph-mon[36481]: fsmap 10 up:standby 2024-03-28T17:46:00.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:59 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T17:46:00.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:59 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:00.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:59 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:00.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:59 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:00.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:59 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:46:00.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:59 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:46:00.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:59 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:46:00.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:45:59 smithi151 ceph-mon[36481]: Deploying daemon mds.a on smithi045 2024-03-28T17:46:00.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.517+0000 7ff874800640 1 -- 172.21.15.45:0/621353243 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff86c062340 con 0x7ff86c077c40 2024-03-28T17:46:00.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.519+0000 7ff866a00640 1 -- 172.21.15.45:0/621353243 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8456 (secure 0 0 0) 0x7ff86809b450 con 0x7ff86c077c40 2024-03-28T17:46:00.519 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:46:00.520 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6839/2313500150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":2313500150},{"type":"v1","addr":"172.21.15.45:6839","nonce":2313500150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6835/2876354410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":2876354410},{"type":"v1","addr":"172.21.15.151:6835","nonce":2876354410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6835/4234998385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":4234998385},{"type":"v1","addr":"172.21.15.71:6835","nonce":4234998385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/1238964845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":1238964845},{"type":"v1","addr":"172.21.15.151:6837","nonce":1238964845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24529,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6839/3003564901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6838","nonce":3003564901},{"type":"v1","addr":"172.21.15.151:6839","nonce":3003564901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/2008752655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2008752655},{"type":"v1","addr":"172.21.15.71:6837","nonce":2008752655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-28T17:46:00.520 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 11 2024-03-28T17:46:00.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.522+0000 7ff874800640 1 -- 172.21.15.45:0/621353243 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff8540a2880 msgr2=0x7ff8540a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:00.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.522+0000 7ff874800640 1 --2- 172.21.15.45:0/621353243 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff8540a2880 0x7ff8540a4d40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff860005fd0 tx=0x7ff86000b4a0 comp rx=0 tx=0).stop 2024-03-28T17:46:00.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.522+0000 7ff874800640 1 -- 172.21.15.45:0/621353243 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff86c077c40 msgr2=0x7ff86c1a5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:00.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.522+0000 7ff874800640 1 --2- 172.21.15.45:0/621353243 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff86c077c40 0x7ff86c1a5530 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff86800e940 tx=0x7ff86800ee00 comp rx=0 tx=0).stop 2024-03-28T17:46:00.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.522+0000 7ff874800640 1 -- 172.21.15.45:0/621353243 shutdown_connections 2024-03-28T17:46:00.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.522+0000 7ff874800640 1 --2- 172.21.15.45:0/621353243 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff8540a2880 0x7ff8540a4d40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:00.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.522+0000 7ff874800640 1 --2- 172.21.15.45:0/621353243 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff86c077c40 0x7ff86c1a5530 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:00.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.522+0000 7ff874800640 1 --2- 172.21.15.45:0/621353243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff86c0772a0 0x7ff86c19e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:00.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.522+0000 7ff874800640 1 --2- 172.21.15.45:0/621353243 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff86c076060 0x7ff86c19df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:00.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.522+0000 7ff874800640 1 -- 172.21.15.45:0/621353243 >> 172.21.15.45:0/621353243 conn(0x7ff86c0fee90 msgr2=0x7ff86c100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:00.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.522+0000 7ff874800640 1 -- 172.21.15.45:0/621353243 shutdown_connections 2024-03-28T17:46:00.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:00.523+0000 7ff874800640 1 -- 172.21.15.45:0/621353243 wait complete. 2024-03-28T17:46:01.247 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:00 smithi045 ceph-mon[31773]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:01.247 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:00 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/621353243' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:46:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:00 smithi071 ceph-mon[36597]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:00 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/621353243' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:46:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:00 smithi151 ceph-mon[36481]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:00 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/621353243' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:46:02.416 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:46:02 smithi045 systemd[1]: Starting Ceph mds.a for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:46:02.540 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:46:02.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.773+0000 7fd263600640 1 -- 172.21.15.45:0/3697638230 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd25c105250 msgr2=0x7fd25c1056d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:02.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.773+0000 7fd263600640 1 --2- 172.21.15.45:0/3697638230 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd25c105250 0x7fd25c1056d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd254009ca0 tx=0x7fd25402c100 comp rx=0 tx=0).stop 2024-03-28T17:46:02.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.773+0000 7fd263600640 1 -- 172.21.15.45:0/3697638230 shutdown_connections 2024-03-28T17:46:02.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.773+0000 7fd263600640 1 --2- 172.21.15.45:0/3697638230 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd25c105250 0x7fd25c1056d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:02.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.773+0000 7fd263600640 1 --2- 172.21.15.45:0/3697638230 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd25c1028a0 0x7fd25c104d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:02.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.773+0000 7fd263600640 1 --2- 172.21.15.45:0/3697638230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd25c0fff30 0x7fd25c102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:02.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.773+0000 7fd263600640 1 -- 172.21.15.45:0/3697638230 >> 172.21.15.45:0/3697638230 conn(0x7fd25c0fbb50 msgr2=0x7fd25c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:02.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.773+0000 7fd263600640 1 -- 172.21.15.45:0/3697638230 shutdown_connections 2024-03-28T17:46:02.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.773+0000 7fd263600640 1 -- 172.21.15.45:0/3697638230 wait complete. 2024-03-28T17:46:02.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.774+0000 7fd263600640 1 Processor -- start 2024-03-28T17:46:02.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.774+0000 7fd263600640 1 -- start start 2024-03-28T17:46:02.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.774+0000 7fd263600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd25c0fff30 0x7fd25c19df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:02.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.774+0000 7fd263600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd25c1028a0 0x7fd25c19e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.774+0000 7fd263600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd25c105250 0x7fd25c1a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.774+0000 7fd263600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25c113910 con 0x7fd25c105250 2024-03-28T17:46:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.774+0000 7fd263600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25c113790 con 0x7fd25c1028a0 2024-03-28T17:46:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.774+0000 7fd263600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25c113a90 con 0x7fd25c0fff30 2024-03-28T17:46:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.775+0000 7fd260c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd25c0fff30 0x7fd25c19df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.775+0000 7fd260c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd25c0fff30 0x7fd25c19df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:52984/0 (socket says 172.21.15.45:52984) 2024-03-28T17:46:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.775+0000 7fd260c00640 1 -- 172.21.15.45:0/2303301870 learned_addr learned my addr 172.21.15.45:0/2303301870 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.775+0000 7fd261600640 1 --2- 172.21.15.45:0/2303301870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd25c105250 0x7fd25c1a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.775+0000 7fd25be00640 1 --2- 172.21.15.45:0/2303301870 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd25c1028a0 0x7fd25c19e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.775+0000 7fd260c00640 1 -- 172.21.15.45:0/2303301870 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd25c1028a0 msgr2=0x7fd25c19e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.775+0000 7fd260c00640 1 --2- 172.21.15.45:0/2303301870 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd25c1028a0 0x7fd25c19e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.775+0000 7fd260c00640 1 -- 172.21.15.45:0/2303301870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd25c105250 msgr2=0x7fd25c1a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.775+0000 7fd260c00640 1 --2- 172.21.15.45:0/2303301870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd25c105250 0x7fd25c1a54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.775+0000 7fd260c00640 1 -- 172.21.15.45:0/2303301870 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd25c1a5b50 con 0x7fd25c0fff30 2024-03-28T17:46:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.775+0000 7fd261600640 1 --2- 172.21.15.45:0/2303301870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd25c105250 0x7fd25c1a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:46:02.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.776+0000 7fd260c00640 1 --2- 172.21.15.45:0/2303301870 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd25c0fff30 0x7fd25c19df30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd24c00e9f0 tx=0x7fd24c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:02.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.776+0000 7fd259600640 1 -- 172.21.15.45:0/2303301870 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24c00cd80 con 0x7fd25c0fff30 2024-03-28T17:46:02.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.776+0000 7fd259600640 1 -- 172.21.15.45:0/2303301870 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd24c010910 con 0x7fd25c0fff30 2024-03-28T17:46:02.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.776+0000 7fd259600640 1 -- 172.21.15.45:0/2303301870 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24c018990 con 0x7fd25c0fff30 2024-03-28T17:46:02.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.776+0000 7fd263600640 1 -- 172.21.15.45:0/2303301870 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd25c1a5e40 con 0x7fd25c0fff30 2024-03-28T17:46:02.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.776+0000 7fd263600640 1 -- 172.21.15.45:0/2303301870 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd25c1a61f0 con 0x7fd25c0fff30 2024-03-28T17:46:02.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.777+0000 7fd263600640 1 -- 172.21.15.45:0/2303301870 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd25c0fd640 con 0x7fd25c0fff30 2024-03-28T17:46:02.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.779+0000 7fd259600640 1 -- 172.21.15.45:0/2303301870 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd24c010ac0 con 0x7fd25c0fff30 2024-03-28T17:46:02.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.780+0000 7fd259600640 1 --2- 172.21.15.45:0/2303301870 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd2400a2800 0x7fd2400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:02.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.781+0000 7fd259600640 1 -- 172.21.15.45:0/2303301870 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd24c0ce990 con 0x7fd25c0fff30 2024-03-28T17:46:02.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.781+0000 7fd25be00640 1 --2- 172.21.15.45:0/2303301870 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd2400a2800 0x7fd2400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:02.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.781+0000 7fd25be00640 1 --2- 172.21.15.45:0/2303301870 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd2400a2800 0x7fd2400a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd25c19f440 tx=0x7fd244005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:02.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:02.785+0000 7fd259600640 1 -- 172.21.15.45:0/2303301870 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd24c09b7e0 con 0x7fd25c0fff30 2024-03-28T17:46:02.811 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:46:02 smithi045 podman[73745]: 2024-03-28 17:46:02.393784689 +0000 UTC m=+0.040692559 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:46:02.812 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:46:02 smithi045 podman[73745]: 2024-03-28 17:46:02.523073375 +0000 UTC m=+0.169981268 container create 6c14419b3a77dcc47a370deeeb01fe0f2017e4cd220f8eabe3be703ce25e36bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-28T17:46:02.812 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:46:02 smithi045 podman[73745]: 2024-03-28 17:46:02.810132103 +0000 UTC m=+0.457039973 container init 6c14419b3a77dcc47a370deeeb01fe0f2017e4cd220f8eabe3be703ce25e36bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-28T17:46:03.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:03.049+0000 7fd263600640 1 -- 172.21.15.45:0/2303301870 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd25c02d2f0 con 0x7fd25c0fff30 2024-03-28T17:46:03.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:03.050+0000 7fd259600640 1 -- 172.21.15.45:0/2303301870 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9248 (secure 0 0 0) 0x7fd24c09f880 con 0x7fd25c0fff30 2024-03-28T17:46:03.050 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:46:03.051 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6839/2313500150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":2313500150},{"type":"v1","addr":"172.21.15.45:6839","nonce":2313500150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6841/2647972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":2647972571},{"type":"v1","addr":"172.21.15.45:6841","nonce":2647972571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6835/2876354410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":2876354410},{"type":"v1","addr":"172.21.15.151:6835","nonce":2876354410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6835/4234998385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":4234998385},{"type":"v1","addr":"172.21.15.71:6835","nonce":4234998385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/1238964845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":1238964845},{"type":"v1","addr":"172.21.15.151:6837","nonce":1238964845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24529,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6839/3003564901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6838","nonce":3003564901},{"type":"v1","addr":"172.21.15.151:6839","nonce":3003564901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/2008752655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2008752655},{"type":"v1","addr":"172.21.15.71:6837","nonce":2008752655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-28T17:46:03.051 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 12 2024-03-28T17:46:03.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:03.053+0000 7fd263600640 1 -- 172.21.15.45:0/2303301870 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd2400a2800 msgr2=0x7fd2400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:03.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:03.053+0000 7fd263600640 1 --2- 172.21.15.45:0/2303301870 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd2400a2800 0x7fd2400a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd25c19f440 tx=0x7fd244005f90 comp rx=0 tx=0).stop 2024-03-28T17:46:03.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:03.053+0000 7fd263600640 1 -- 172.21.15.45:0/2303301870 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd25c0fff30 msgr2=0x7fd25c19df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:03.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:03.053+0000 7fd263600640 1 --2- 172.21.15.45:0/2303301870 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd25c0fff30 0x7fd25c19df30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd24c00e9f0 tx=0x7fd24c00eeb0 comp rx=0 tx=0).stop 2024-03-28T17:46:03.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:03.053+0000 7fd263600640 1 -- 172.21.15.45:0/2303301870 shutdown_connections 2024-03-28T17:46:03.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:03.053+0000 7fd263600640 1 --2- 172.21.15.45:0/2303301870 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd2400a2800 0x7fd2400a4cc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:03.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:03.053+0000 7fd263600640 1 --2- 172.21.15.45:0/2303301870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd25c105250 0x7fd25c1a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:03.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:03.053+0000 7fd263600640 1 --2- 172.21.15.45:0/2303301870 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd25c1028a0 0x7fd25c19e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:03.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:03.053+0000 7fd263600640 1 --2- 172.21.15.45:0/2303301870 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd25c0fff30 0x7fd25c19df30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:03.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:03.053+0000 7fd263600640 1 -- 172.21.15.45:0/2303301870 >> 172.21.15.45:0/2303301870 conn(0x7fd25c0fbb50 msgr2=0x7fd25c1008b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:03.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:03.053+0000 7fd263600640 1 -- 172.21.15.45:0/2303301870 shutdown_connections 2024-03-28T17:46:03.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:03.053+0000 7fd263600640 1 -- 172.21.15.45:0/2303301870 wait complete. 2024-03-28T17:46:03.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:02 smithi045 ceph-mon[31773]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:03.072 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:46:02 smithi045 podman[73745]: 2024-03-28 17:46:02.822608391 +0000 UTC m=+0.469516272 container start 6c14419b3a77dcc47a370deeeb01fe0f2017e4cd220f8eabe3be703ce25e36bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, ceph=True) 2024-03-28T17:46:03.072 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:46:02 smithi045 ceph-mds[73802]: -- 172.21.15.45:0/3342306582 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557244495a40 con 0x557244468800 2024-03-28T17:46:03.072 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:46:02 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: starting mds.a at 2024-03-28T17:46:03.072 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:46:02 smithi045 bash[73745]: 6c14419b3a77dcc47a370deeeb01fe0f2017e4cd220f8eabe3be703ce25e36bd 2024-03-28T17:46:03.072 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:46:02 smithi045 systemd[1]: Started Ceph mds.a for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:46:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:02 smithi071 ceph-mon[36597]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:03.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:02 smithi151 ceph-mon[36481]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:04.015 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:03 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6840/2647972571,v1:172.21.15.45:6841/2647972571] up:boot 2024-03-28T17:46:04.015 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:03 smithi071 ceph-mon[36597]: fsmap 11 up:standby 2024-03-28T17:46:04.015 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:03 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T17:46:04.016 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:03 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:04.016 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:03 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:04.016 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:03 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/2303301870' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:46:04.016 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:03 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:04.016 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:03 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:46:04.016 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:03 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:46:04.016 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:03 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:46:04.016 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:03 smithi071 ceph-mon[36597]: Deploying daemon mds.b on smithi071 2024-03-28T17:46:04.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:03 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6840/2647972571,v1:172.21.15.45:6841/2647972571] up:boot 2024-03-28T17:46:04.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:03 smithi151 ceph-mon[36481]: fsmap 11 up:standby 2024-03-28T17:46:04.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:03 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T17:46:04.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:03 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:04.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:03 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:04.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:03 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/2303301870' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:46:04.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:03 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:04.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:03 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:46:04.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:03 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:46:04.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:03 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:46:04.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:03 smithi151 ceph-mon[36481]: Deploying daemon mds.b on smithi071 2024-03-28T17:46:04.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:03 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6840/2647972571,v1:172.21.15.45:6841/2647972571] up:boot 2024-03-28T17:46:04.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:03 smithi045 ceph-mon[31773]: fsmap 11 up:standby 2024-03-28T17:46:04.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:03 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T17:46:04.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:03 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:04.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:03 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:04.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:03 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/2303301870' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:46:04.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:03 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:04.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:03 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T17:46:04.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:03 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T17:46:04.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:03 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:46:04.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:03 smithi045 ceph-mon[31773]: Deploying daemon mds.b on smithi071 2024-03-28T17:46:05.078 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:46:05.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.314+0000 7f416d200640 1 -- 172.21.15.45:0/2930019736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41681028b0 msgr2=0x7f4168102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:05.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.314+0000 7f416d200640 1 --2- 172.21.15.45:0/2930019736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41681028b0 0x7f4168102d30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4154009900 tx=0x7f415402bc40 comp rx=0 tx=0).stop 2024-03-28T17:46:05.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.314+0000 7f416d200640 1 -- 172.21.15.45:0/2930019736 shutdown_connections 2024-03-28T17:46:05.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.314+0000 7f416d200640 1 --2- 172.21.15.45:0/2930019736 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4168103270 0x7f4168109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:05.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.314+0000 7f416d200640 1 --2- 172.21.15.45:0/2930019736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41681028b0 0x7f4168102d30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:05.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.314+0000 7f416d200640 1 --2- 172.21.15.45:0/2930019736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41681016c0 0x7f4168101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:05.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.314+0000 7f416d200640 1 -- 172.21.15.45:0/2930019736 >> 172.21.15.45:0/2930019736 conn(0x7f41680fcc50 msgr2=0x7f41680ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:05.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.314+0000 7f416d200640 1 -- 172.21.15.45:0/2930019736 shutdown_connections 2024-03-28T17:46:05.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.314+0000 7f416d200640 1 -- 172.21.15.45:0/2930019736 wait complete. 2024-03-28T17:46:05.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.315+0000 7f416d200640 1 Processor -- start 2024-03-28T17:46:05.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.315+0000 7f416d200640 1 -- start start 2024-03-28T17:46:05.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.316+0000 7f416d200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41681016c0 0x7f4168197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:05.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.316+0000 7f416d200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41681028b0 0x7f4168197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:05.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.316+0000 7f416d200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4168103270 0x7f416819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:05.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.316+0000 7f416d200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416810d370 con 0x7f41681028b0 2024-03-28T17:46:05.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.316+0000 7f416d200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416810d1f0 con 0x7f4168103270 2024-03-28T17:46:05.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.316+0000 7f416d200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416810d4f0 con 0x7f41681016c0 2024-03-28T17:46:05.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.316+0000 7f4165400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41681028b0 0x7f4168197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:05.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.316+0000 7f4165400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41681028b0 0x7f4168197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34916/0 (socket says 172.21.15.45:34916) 2024-03-28T17:46:05.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.316+0000 7f4165400640 1 -- 172.21.15.45:0/2366597804 learned_addr learned my addr 172.21.15.45:0/2366597804 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:05.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.316+0000 7f4165e00640 1 --2- 172.21.15.45:0/2366597804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41681016c0 0x7f4168197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:05.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.316+0000 7f4166800640 1 --2- 172.21.15.45:0/2366597804 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4168103270 0x7f416819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:05.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.317+0000 7f4165400640 1 -- 172.21.15.45:0/2366597804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41681016c0 msgr2=0x7f4168197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:05.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.317+0000 7f4165400640 1 --2- 172.21.15.45:0/2366597804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41681016c0 0x7f4168197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:05.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.317+0000 7f4165400640 1 -- 172.21.15.45:0/2366597804 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4168103270 msgr2=0x7f416819ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:05.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.317+0000 7f4165400640 1 --2- 172.21.15.45:0/2366597804 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4168103270 0x7f416819ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:05.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.317+0000 7f4165400640 1 -- 172.21.15.45:0/2366597804 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416819f5b0 con 0x7f41681028b0 2024-03-28T17:46:05.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.317+0000 7f4165e00640 1 --2- 172.21.15.45:0/2366597804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41681016c0 0x7f4168197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:46:05.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.317+0000 7f4166800640 1 --2- 172.21.15.45:0/2366597804 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4168103270 0x7f416819ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:46:05.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.317+0000 7f4165400640 1 --2- 172.21.15.45:0/2366597804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41681028b0 0x7f4168197ed0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f415402c8f0 tx=0x7f4154005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:05.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.317+0000 7f415ea00640 1 -- 172.21.15.45:0/2366597804 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4154041070 con 0x7f41681028b0 2024-03-28T17:46:05.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.317+0000 7f415ea00640 1 -- 172.21.15.45:0/2366597804 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f41540044d0 con 0x7f41681028b0 2024-03-28T17:46:05.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.317+0000 7f416d200640 1 -- 172.21.15.45:0/2366597804 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416819f840 con 0x7f41681028b0 2024-03-28T17:46:05.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.317+0000 7f415ea00640 1 -- 172.21.15.45:0/2366597804 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4154034440 con 0x7f41681028b0 2024-03-28T17:46:05.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.317+0000 7f416d200640 1 -- 172.21.15.45:0/2366597804 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f416819fd30 con 0x7f41681028b0 2024-03-28T17:46:05.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.319+0000 7f416d200640 1 -- 172.21.15.45:0/2366597804 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41681089e0 con 0x7f41681028b0 2024-03-28T17:46:05.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.321+0000 7f415ea00640 1 -- 172.21.15.45:0/2366597804 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4154044070 con 0x7f41681028b0 2024-03-28T17:46:05.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.322+0000 7f415ea00640 1 --2- 172.21.15.45:0/2366597804 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f414c0a2800 0x7f414c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:05.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.322+0000 7f415ea00640 1 -- 172.21.15.45:0/2366597804 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f41540f01f0 con 0x7f41681028b0 2024-03-28T17:46:05.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.323+0000 7f4165e00640 1 --2- 172.21.15.45:0/2366597804 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f414c0a2800 0x7f414c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:05.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.323+0000 7f4165e00640 1 --2- 172.21.15.45:0/2366597804 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f414c0a2800 0x7f414c0a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4168102710 tx=0x7f4158005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:05.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.327+0000 7f415ea00640 1 -- 172.21.15.45:0/2366597804 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f41540bd040 con 0x7f41681028b0 2024-03-28T17:46:05.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:05 smithi045 ceph-mon[31773]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:05.483 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:05 smithi071 ceph-mon[36597]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:05.483 INFO:journalctl@ceph.mds.b.smithi071.stdout:Mar 28 17:46:05 smithi071 systemd[1]: Starting Ceph mds.b for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:46:05.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.572+0000 7f416d200640 1 -- 172.21.15.45:0/2366597804 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f416802d070 con 0x7f41681028b0 2024-03-28T17:46:05.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:05 smithi151 ceph-mon[36481]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:05.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.574+0000 7f415ea00640 1 -- 172.21.15.45:0/2366597804 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9248 (secure 0 0 0) 0x7f41540c10e0 con 0x7f41681028b0 2024-03-28T17:46:05.574 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:46:05.575 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6839/2313500150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":2313500150},{"type":"v1","addr":"172.21.15.45:6839","nonce":2313500150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6841/2647972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":2647972571},{"type":"v1","addr":"172.21.15.45:6841","nonce":2647972571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6835/2876354410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":2876354410},{"type":"v1","addr":"172.21.15.151:6835","nonce":2876354410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6835/4234998385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":4234998385},{"type":"v1","addr":"172.21.15.71:6835","nonce":4234998385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/1238964845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":1238964845},{"type":"v1","addr":"172.21.15.151:6837","nonce":1238964845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24529,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6839/3003564901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6838","nonce":3003564901},{"type":"v1","addr":"172.21.15.151:6839","nonce":3003564901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/2008752655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2008752655},{"type":"v1","addr":"172.21.15.71:6837","nonce":2008752655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-28T17:46:05.575 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 12 2024-03-28T17:46:05.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.577+0000 7f416d200640 1 -- 172.21.15.45:0/2366597804 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f414c0a2800 msgr2=0x7f414c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:05.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.577+0000 7f416d200640 1 --2- 172.21.15.45:0/2366597804 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f414c0a2800 0x7f414c0a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4168102710 tx=0x7f4158005f90 comp rx=0 tx=0).stop 2024-03-28T17:46:05.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.577+0000 7f416d200640 1 -- 172.21.15.45:0/2366597804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41681028b0 msgr2=0x7f4168197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:05.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.577+0000 7f416d200640 1 --2- 172.21.15.45:0/2366597804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41681028b0 0x7f4168197ed0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f415402c8f0 tx=0x7f4154005d50 comp rx=0 tx=0).stop 2024-03-28T17:46:05.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.577+0000 7f416d200640 1 -- 172.21.15.45:0/2366597804 shutdown_connections 2024-03-28T17:46:05.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.577+0000 7f416d200640 1 --2- 172.21.15.45:0/2366597804 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f414c0a2800 0x7f414c0a4cc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:05.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.577+0000 7f416d200640 1 --2- 172.21.15.45:0/2366597804 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4168103270 0x7f416819ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:05.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.577+0000 7f416d200640 1 --2- 172.21.15.45:0/2366597804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41681028b0 0x7f4168197ed0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:05.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.577+0000 7f416d200640 1 --2- 172.21.15.45:0/2366597804 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41681016c0 0x7f4168197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:05.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.577+0000 7f416d200640 1 -- 172.21.15.45:0/2366597804 >> 172.21.15.45:0/2366597804 conn(0x7f41680fcc50 msgr2=0x7f4168105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:05.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.577+0000 7f416d200640 1 -- 172.21.15.45:0/2366597804 shutdown_connections 2024-03-28T17:46:05.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:05.577+0000 7f416d200640 1 -- 172.21.15.45:0/2366597804 wait complete. 2024-03-28T17:46:05.769 INFO:journalctl@ceph.mds.b.smithi071.stdout:Mar 28 17:46:05 smithi071 podman[66885]: 2024-03-28 17:46:05.386230846 +0000 UTC m=+0.017581948 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:46:05.769 INFO:journalctl@ceph.mds.b.smithi071.stdout:Mar 28 17:46:05 smithi071 podman[66885]: 2024-03-28 17:46:05.512331162 +0000 UTC m=+0.143682254 container create 1963dfdb35f491bd484bea0154f14231c659b426fda17a38e703a458113eccec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T17:46:05.769 INFO:journalctl@ceph.mds.b.smithi071.stdout:Mar 28 17:46:05 smithi071 podman[66885]: 2024-03-28 17:46:05.769384322 +0000 UTC m=+0.400735424 container init 1963dfdb35f491bd484bea0154f14231c659b426fda17a38e703a458113eccec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T17:46:06.061 INFO:journalctl@ceph.mds.b.smithi071.stdout:Mar 28 17:46:05 smithi071 podman[66885]: 2024-03-28 17:46:05.772179226 +0000 UTC m=+0.403530328 container start 1963dfdb35f491bd484bea0154f14231c659b426fda17a38e703a458113eccec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T17:46:06.061 INFO:journalctl@ceph.mds.b.smithi071.stdout:Mar 28 17:46:05 smithi071 ceph-mds[66904]: -- 172.21.15.71:0/3598461363 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5569ffbbfa40 con 0x5569ffb92c00 2024-03-28T17:46:06.061 INFO:journalctl@ceph.mds.b.smithi071.stdout:Mar 28 17:46:05 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-b[66900]: starting mds.b at 2024-03-28T17:46:06.061 INFO:journalctl@ceph.mds.b.smithi071.stdout:Mar 28 17:46:05 smithi071 bash[66885]: 1963dfdb35f491bd484bea0154f14231c659b426fda17a38e703a458113eccec 2024-03-28T17:46:06.061 INFO:journalctl@ceph.mds.b.smithi071.stdout:Mar 28 17:46:05 smithi071 systemd[1]: Started Ceph mds.b for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:46:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:06 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/2366597804' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:46:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:06 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:06 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:06 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:06 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:06 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:06 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/2366597804' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:46:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:06 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:06 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:06.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:06 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:06.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:06 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:06.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:06 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:06 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/2366597804' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:46:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:06 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:06.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:06 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:06.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:06 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:06.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:06 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:06.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:06 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:07.355 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:07 smithi071 ceph-mon[36597]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:07.355 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:07 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] up:boot 2024-03-28T17:46:07.356 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:07 smithi071 ceph-mon[36597]: fsmap 12 up:standby 2024-03-28T17:46:07.356 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:07 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T17:46:07.356 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:07 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:46:07.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:07 smithi045 ceph-mon[31773]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:07.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:07 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] up:boot 2024-03-28T17:46:07.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:07 smithi045 ceph-mon[31773]: fsmap 12 up:standby 2024-03-28T17:46:07.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:07 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T17:46:07.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:07 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:46:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:07 smithi151 ceph-mon[36481]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:07 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] up:boot 2024-03-28T17:46:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:07 smithi151 ceph-mon[36481]: fsmap 12 up:standby 2024-03-28T17:46:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:07 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T17:46:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:07 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:46:07.600 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:46:07.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.822+0000 7f2a12000640 1 -- 172.21.15.45:0/1178923392 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a0c1038e0 msgr2=0x7f2a0c103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:07.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.822+0000 7f2a12000640 1 --2- 172.21.15.45:0/1178923392 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a0c1038e0 0x7f2a0c103d00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f29fc009900 tx=0x7f29fc02bc30 comp rx=0 tx=0).stop 2024-03-28T17:46:07.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.822+0000 7f2a12000640 1 -- 172.21.15.45:0/1178923392 shutdown_connections 2024-03-28T17:46:07.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.822+0000 7f2a12000640 1 --2- 172.21.15.45:0/1178923392 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a0c105490 0x7f2a0c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:07.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.822+0000 7f2a12000640 1 --2- 172.21.15.45:0/1178923392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a0c104ad0 0x7f2a0c104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:07.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.822+0000 7f2a12000640 1 --2- 172.21.15.45:0/1178923392 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a0c1038e0 0x7f2a0c103d00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:07.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.822+0000 7f2a12000640 1 -- 172.21.15.45:0/1178923392 >> 172.21.15.45:0/1178923392 conn(0x7f2a0c0fee50 msgr2=0x7f2a0c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:07.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.822+0000 7f2a12000640 1 -- 172.21.15.45:0/1178923392 shutdown_connections 2024-03-28T17:46:07.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.823+0000 7f2a12000640 1 -- 172.21.15.45:0/1178923392 wait complete. 2024-03-28T17:46:07.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.824+0000 7f2a12000640 1 Processor -- start 2024-03-28T17:46:07.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.824+0000 7f2a12000640 1 -- start start 2024-03-28T17:46:07.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.824+0000 7f2a12000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a0c1038e0 0x7f2a0c199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:07.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.824+0000 7f2a12000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a0c104ad0 0x7f2a0c19a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:07.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.824+0000 7f2a12000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a0c105490 0x7f2a0c1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:07.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.824+0000 7f2a12000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c10f530 con 0x7f2a0c105490 2024-03-28T17:46:07.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.824+0000 7f2a12000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c10f3b0 con 0x7f2a0c104ad0 2024-03-28T17:46:07.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.824+0000 7f2a12000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c10f6b0 con 0x7f2a0c1038e0 2024-03-28T17:46:07.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.825+0000 7f2a0b400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a0c1038e0 0x7f2a0c199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:07.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.825+0000 7f2a0aa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a0c104ad0 0x7f2a0c19a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:07.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.825+0000 7f2a0aa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a0c104ad0 0x7f2a0c19a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:41132/0 (socket says 172.21.15.45:41132) 2024-03-28T17:46:07.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.825+0000 7f2a0aa00640 1 -- 172.21.15.45:0/1406545019 learned_addr learned my addr 172.21.15.45:0/1406545019 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:07.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.825+0000 7f2a0b400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a0c1038e0 0x7f2a0c199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:38552/0 (socket says 172.21.15.45:38552) 2024-03-28T17:46:07.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.825+0000 7f2a0be00640 1 --2- 172.21.15.45:0/1406545019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a0c105490 0x7f2a0c1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:07.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.825+0000 7f2a0aa00640 1 -- 172.21.15.45:0/1406545019 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a0c1038e0 msgr2=0x7f2a0c199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:07.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.825+0000 7f2a0aa00640 1 --2- 172.21.15.45:0/1406545019 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a0c1038e0 0x7f2a0c199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:07.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.825+0000 7f2a0aa00640 1 -- 172.21.15.45:0/1406545019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a0c105490 msgr2=0x7f2a0c1a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:07.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.825+0000 7f2a0aa00640 1 --2- 172.21.15.45:0/1406545019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a0c105490 0x7f2a0c1a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:07.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.825+0000 7f2a0aa00640 1 -- 172.21.15.45:0/1406545019 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0c1a1830 con 0x7f2a0c104ad0 2024-03-28T17:46:07.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.825+0000 7f2a0b400640 1 --2- 172.21.15.45:0/1406545019 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a0c1038e0 0x7f2a0c199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:46:07.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.825+0000 7f2a0aa00640 1 --2- 172.21.15.45:0/1406545019 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a0c104ad0 0x7f2a0c19a160 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f29f800e9f0 tx=0x7f29f800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:07.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.826+0000 7f2a03e00640 1 -- 172.21.15.45:0/1406545019 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f800cd80 con 0x7f2a0c104ad0 2024-03-28T17:46:07.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.826+0000 7f2a03e00640 1 -- 172.21.15.45:0/1406545019 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29f8010910 con 0x7f2a0c104ad0 2024-03-28T17:46:07.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.826+0000 7f2a12000640 1 -- 172.21.15.45:0/1406545019 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0c1a1b20 con 0x7f2a0c104ad0 2024-03-28T17:46:07.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.826+0000 7f2a03e00640 1 -- 172.21.15.45:0/1406545019 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f8018990 con 0x7f2a0c104ad0 2024-03-28T17:46:07.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.826+0000 7f2a12000640 1 -- 172.21.15.45:0/1406545019 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0c1a2040 con 0x7f2a0c104ad0 2024-03-28T17:46:07.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.827+0000 7f2a12000640 1 -- 172.21.15.45:0/1406545019 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a0c10acd0 con 0x7f2a0c104ad0 2024-03-28T17:46:07.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.830+0000 7f2a03e00640 1 -- 172.21.15.45:0/1406545019 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f29f8010ac0 con 0x7f2a0c104ad0 2024-03-28T17:46:07.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.831+0000 7f2a03e00640 1 --2- 172.21.15.45:0/1406545019 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f29f00a2800 0x7f29f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:07.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.831+0000 7f2a03e00640 1 -- 172.21.15.45:0/1406545019 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f29f80cea40 con 0x7f2a0c104ad0 2024-03-28T17:46:07.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.831+0000 7f2a0b400640 1 --2- 172.21.15.45:0/1406545019 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f29f00a2800 0x7f29f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:07.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.832+0000 7f2a0b400640 1 --2- 172.21.15.45:0/1406545019 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f29f00a2800 0x7f29f00a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f29fc000c00 tx=0x7f29fc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:07.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:07.835+0000 7f2a03e00640 1 -- 172.21.15.45:0/1406545019 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f29f809b890 con 0x7f2a0c104ad0 2024-03-28T17:46:08.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.121+0000 7f2a12000640 1 -- 172.21.15.45:0/1406545019 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a0c0622a0 con 0x7f2a0c104ad0 2024-03-28T17:46:08.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.122+0000 7f2a03e00640 1 -- 172.21.15.45:0/1406545019 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10040 (secure 0 0 0) 0x7f29f809f930 con 0x7f2a0c104ad0 2024-03-28T17:46:08.123 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:46:08.123 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6839/2313500150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":2313500150},{"type":"v1","addr":"172.21.15.45:6839","nonce":2313500150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6841/2647972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":2647972571},{"type":"v1","addr":"172.21.15.45:6841","nonce":2647972571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6835/2876354410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":2876354410},{"type":"v1","addr":"172.21.15.151:6835","nonce":2876354410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6835/4234998385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":4234998385},{"type":"v1","addr":"172.21.15.71:6835","nonce":4234998385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/1238964845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":1238964845},{"type":"v1","addr":"172.21.15.151:6837","nonce":1238964845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24529,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6839/3003564901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6838","nonce":3003564901},{"type":"v1","addr":"172.21.15.151:6839","nonce":3003564901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/2008752655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2008752655},{"type":"v1","addr":"172.21.15.71:6837","nonce":2008752655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24584,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6839/2709938800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":2709938800},{"type":"v1","addr":"172.21.15.71:6839","nonce":2709938800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2024-03-28T17:46:08.123 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 13 2024-03-28T17:46:08.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.125+0000 7f2a12000640 1 -- 172.21.15.45:0/1406545019 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f29f00a2800 msgr2=0x7f29f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:08.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.125+0000 7f2a12000640 1 --2- 172.21.15.45:0/1406545019 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f29f00a2800 0x7f29f00a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f29fc000c00 tx=0x7f29fc005fd0 comp rx=0 tx=0).stop 2024-03-28T17:46:08.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.125+0000 7f2a12000640 1 -- 172.21.15.45:0/1406545019 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a0c104ad0 msgr2=0x7f2a0c19a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:08.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.125+0000 7f2a12000640 1 --2- 172.21.15.45:0/1406545019 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a0c104ad0 0x7f2a0c19a160 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f29f800e9f0 tx=0x7f29f800eeb0 comp rx=0 tx=0).stop 2024-03-28T17:46:08.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.126+0000 7f2a12000640 1 -- 172.21.15.45:0/1406545019 shutdown_connections 2024-03-28T17:46:08.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.126+0000 7f2a12000640 1 --2- 172.21.15.45:0/1406545019 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f29f00a2800 0x7f29f00a4cc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:08.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.126+0000 7f2a12000640 1 --2- 172.21.15.45:0/1406545019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a0c105490 0x7f2a0c1a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:08.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.126+0000 7f2a12000640 1 --2- 172.21.15.45:0/1406545019 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a0c104ad0 0x7f2a0c19a160 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:08.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.126+0000 7f2a12000640 1 --2- 172.21.15.45:0/1406545019 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a0c1038e0 0x7f2a0c199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:08.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.126+0000 7f2a12000640 1 -- 172.21.15.45:0/1406545019 >> 172.21.15.45:0/1406545019 conn(0x7f2a0c0fee50 msgr2=0x7f2a0c107ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:08.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.126+0000 7f2a12000640 1 -- 172.21.15.45:0/1406545019 shutdown_connections 2024-03-28T17:46:08.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.126+0000 7f2a12000640 1 -- 172.21.15.45:0/1406545019 wait complete. 2024-03-28T17:46:08.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T17:46:08.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.356+0000 7f2f0cc00640 1 -- 172.21.15.45:0/2054802886 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f08077c40 msgr2=0x7f2f080784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:08.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.356+0000 7f2f0cc00640 1 --2- 172.21.15.45:0/2054802886 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f08077c40 0x7f2f080784e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2f00009ca0 tx=0x7f2f0002c0f0 comp rx=0 tx=0).stop 2024-03-28T17:46:08.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.356+0000 7f2f0cc00640 1 -- 172.21.15.45:0/2054802886 shutdown_connections 2024-03-28T17:46:08.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.356+0000 7f2f0cc00640 1 --2- 172.21.15.45:0/2054802886 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f08077c40 0x7f2f080784e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:08.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.356+0000 7f2f0cc00640 1 --2- 172.21.15.45:0/2054802886 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f080772a0 0x7f2f08077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:08.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.356+0000 7f2f0cc00640 1 --2- 172.21.15.45:0/2054802886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f08076060 0x7f2f08076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:08.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.356+0000 7f2f0cc00640 1 -- 172.21.15.45:0/2054802886 >> 172.21.15.45:0/2054802886 conn(0x7f2f080fee50 msgr2=0x7f2f08101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:08.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.356+0000 7f2f0cc00640 1 -- 172.21.15.45:0/2054802886 shutdown_connections 2024-03-28T17:46:08.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.356+0000 7f2f0cc00640 1 -- 172.21.15.45:0/2054802886 wait complete. 2024-03-28T17:46:08.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.358+0000 7f2f0cc00640 1 Processor -- start 2024-03-28T17:46:08.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.358+0000 7f2f0cc00640 1 -- start start 2024-03-28T17:46:08.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.358+0000 7f2f0cc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f08076060 0x7f2f0819df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:08.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.358+0000 7f2f0cc00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f080772a0 0x7f2f0819e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:08.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.358+0000 7f2f0cc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f08077c40 0x7f2f081a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:08.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.358+0000 7f2f0cc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f08113970 con 0x7f2f08077c40 2024-03-28T17:46:08.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.358+0000 7f2f0cc00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f081137f0 con 0x7f2f08076060 2024-03-28T17:46:08.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.358+0000 7f2f0cc00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f08113af0 con 0x7f2f080772a0 2024-03-28T17:46:08.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.358+0000 7f2f05e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f08076060 0x7f2f0819df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:08.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.359+0000 7f2f05e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f08076060 0x7f2f0819df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:41150/0 (socket says 172.21.15.45:41150) 2024-03-28T17:46:08.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.359+0000 7f2f05e00640 1 -- 172.21.15.45:0/1726939918 learned_addr learned my addr 172.21.15.45:0/1726939918 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:08.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.359+0000 7f2f05400640 1 --2- 172.21.15.45:0/1726939918 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f080772a0 0x7f2f0819e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:08.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.359+0000 7f2f06800640 1 --2- 172.21.15.45:0/1726939918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f08077c40 0x7f2f081a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:08.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.359+0000 7f2f05e00640 1 -- 172.21.15.45:0/1726939918 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f080772a0 msgr2=0x7f2f0819e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:08.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.359+0000 7f2f05e00640 1 --2- 172.21.15.45:0/1726939918 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f080772a0 0x7f2f0819e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:08.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.359+0000 7f2f05e00640 1 -- 172.21.15.45:0/1726939918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f08077c40 msgr2=0x7f2f081a54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:08.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.359+0000 7f2f05e00640 1 --2- 172.21.15.45:0/1726939918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f08077c40 0x7f2f081a54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:08.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.359+0000 7f2f05e00640 1 -- 172.21.15.45:0/1726939918 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f081a5b60 con 0x7f2f08076060 2024-03-28T17:46:08.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.359+0000 7f2f05e00640 1 --2- 172.21.15.45:0/1726939918 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f08076060 0x7f2f0819df40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2ef800e9f0 tx=0x7f2ef800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:08.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.360+0000 7f2efea00640 1 -- 172.21.15.45:0/1726939918 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef800cd80 con 0x7f2f08076060 2024-03-28T17:46:08.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.360+0000 7f2efea00640 1 -- 172.21.15.45:0/1726939918 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ef8010910 con 0x7f2f08076060 2024-03-28T17:46:08.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.360+0000 7f2efea00640 1 -- 172.21.15.45:0/1726939918 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8018990 con 0x7f2f08076060 2024-03-28T17:46:08.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.360+0000 7f2f0cc00640 1 -- 172.21.15.45:0/1726939918 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f081a5e50 con 0x7f2f08076060 2024-03-28T17:46:08.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.360+0000 7f2f0cc00640 1 -- 172.21.15.45:0/1726939918 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f081a6260 con 0x7f2f08076060 2024-03-28T17:46:08.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.361+0000 7f2f0cc00640 1 -- 172.21.15.45:0/1726939918 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f08110390 con 0x7f2f08076060 2024-03-28T17:46:08.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.363+0000 7f2efea00640 1 -- 172.21.15.45:0/1726939918 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2ef8010ac0 con 0x7f2f08076060 2024-03-28T17:46:08.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.365+0000 7f2efea00640 1 --2- 172.21.15.45:0/1726939918 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2eec0a2880 0x7f2eec0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:08.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.365+0000 7f2efea00640 1 -- 172.21.15.45:0/1726939918 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2ef80ce910 con 0x7f2f08076060 2024-03-28T17:46:08.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.365+0000 7f2f05400640 1 --2- 172.21.15.45:0/1726939918 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2eec0a2880 0x7f2eec0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:08.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.366+0000 7f2f05400640 1 --2- 172.21.15.45:0/1726939918 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2eec0a2880 0x7f2eec0a4d40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2ef40095f0 tx=0x7f2ef4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:08.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.368+0000 7f2efea00640 1 -- 172.21.15.45:0/1726939918 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2ef809b760 con 0x7f2f08076060 2024-03-28T17:46:08.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.596+0000 7f2f0cc00640 1 -- 172.21.15.45:0/1726939918 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f08062340 con 0x7f2f08076060 2024-03-28T17:46:08.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.597+0000 7f2efea00640 1 -- 172.21.15.45:0/1726939918 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18445 (secure 0 0 0) 0x7f2ef80bb020 con 0x7f2f08076060 2024-03-28T17:46:08.598 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:46:08.598 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":81,"fsid":"9f4888ec-ed29-11ee-95d0-87774f69a715","created":"2024-03-28T17:36:51.623185+0000","modified":"2024-03-28T17:45:15.608581+0000","last_up_change":"2024-03-28T17:45:14.043432+0000","last_in_change":"2024-03-28T17:44:53.654489+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T17:40:54.208375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"74c56bc2-4169-4919-afd4-2ebd236f23e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6803","nonce":3795754023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6805","nonce":3795754023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6809","nonce":3795754023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6807","nonce":3795754023}]},"public_addr":"172.21.15.45:6803/3795754023","cluster_addr":"172.21.15.45:6805/3795754023","heartbeat_back_addr":"172.21.15.45:6809/3795754023","heartbeat_front_addr":"172.21.15.45:6807/3795754023","state":["exists","up"]},{"osd":1,"uuid":"78dcc21b-75b2-453d-b690-17b55b8ed1a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6801","nonce":2014529906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6803","nonce":2014529906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6807","nonce":2014529906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6805","nonce":2014529906}]},"public_addr":"172.21.15.71:6801/2014529906","cluster_addr":"172.21.15.71:6803/2014529906","heartbeat_back_addr":"172.21.15.71:6807/2014529906","heartbeat_front_addr":"172.21.15.71:6805/2014529906","state":["exists","up"]},{"osd":2,"uuid":"ca9f3f36-b71e-4962-a94f-cedf2f12eb7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6801","nonce":1850933775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6803","nonce":1850933775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6807","nonce":1850933775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6805","nonce":1850933775}]},"public_addr":"172.21.15.151:6801/1850933775","cluster_addr":"172.21.15.151:6803/1850933775","heartbeat_back_addr":"172.21.15.151:6807/1850933775","heartbeat_front_addr":"172.21.15.151:6805/1850933775","state":["exists","up"]},{"osd":3,"uuid":"d3f21f01-2973-413e-8daa-9974b8a44af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6811","nonce":3021569975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6813","nonce":3021569975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6817","nonce":3021569975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6815","nonce":3021569975}]},"public_addr":"172.21.15.45:6811/3021569975","cluster_addr":"172.21.15.45:6813/3021569975","heartbeat_back_addr":"172.21.15.45:6817/3021569975","heartbeat_front_addr":"172.21.15.45:6815/3021569975","state":["exists","up"]},{"osd":4,"uuid":"4a336f42-0605-492b-ad88-56cb02acdcf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6809","nonce":2069578333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6811","nonce":2069578333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6815","nonce":2069578333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6813","nonce":2069578333}]},"public_addr":"172.21.15.71:6809/2069578333","cluster_addr":"172.21.15.71:6811/2069578333","heartbeat_back_addr":"172.21.15.71:6815/2069578333","heartbeat_front_addr":"172.21.15.71:6813/2069578333","state":["exists","up"]},{"osd":5,"uuid":"c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6809","nonce":2629217227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6811","nonce":2629217227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6815","nonce":2629217227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6813","nonce":2629217227}]},"public_addr":"172.21.15.151:6809/2629217227","cluster_addr":"172.21.15.151:6811/2629217227","heartbeat_back_addr":"172.21.15.151:6815/2629217227","heartbeat_front_addr":"172.21.15.151:6813/2629217227","state":["exists","up"]},{"osd":6,"uuid":"58a6c5bd-a8b7-4892-b346-bcd7994bcb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6819","nonce":2077907039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6821","nonce":2077907039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6825","nonce":2077907039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6823","nonce":2077907039}]},"public_addr":"172.21.15.45:6819/2077907039","cluster_addr":"172.21.15.45:6821/2077907039","heartbeat_back_addr":"172.21.15.45:6825/2077907039","heartbeat_front_addr":"172.21.15.45:6823/2077907039","state":["exists","up"]},{"osd":7,"uuid":"f7a45d52-807b-4407-bb04-787b2a52a0ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6817","nonce":2000211251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6819","nonce":2000211251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6823","nonce":2000211251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6821","nonce":2000211251}]},"public_addr":"172.21.15.71:6817/2000211251","cluster_addr":"172.21.15.71:6819/2000211251","heartbeat_back_addr":"172.21.15.71:6823/2000211251","heartbeat_front_addr":"172.21.15.71:6821/2000211251","state":["exists","up"]},{"osd":8,"uuid":"88abcb0f-f189-49be-8315-c700c77369f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6817","nonce":3665460137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6819","nonce":3665460137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6823","nonce":3665460137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6821","nonce":3665460137}]},"public_addr":"172.21.15.151:6817/3665460137","cluster_addr":"172.21.15.151:6819/3665460137","heartbeat_back_addr":"172.21.15.151:6823/3665460137","heartbeat_front_addr":"172.21.15.151:6821/3665460137","state":["exists","up"]},{"osd":9,"uuid":"ed3fc096-961e-4392-abc2-c5aa4d050117","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6827","nonce":2122322684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6829","nonce":2122322684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6833","nonce":2122322684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6831","nonce":2122322684}]},"public_addr":"172.21.15.45:6827/2122322684","cluster_addr":"172.21.15.45:6829/2122322684","heartbeat_back_addr":"172.21.15.45:6833/2122322684","heartbeat_front_addr":"172.21.15.45:6831/2122322684","state":["exists","up"]},{"osd":10,"uuid":"ba94762b-bc5c-4c97-9cc1-17e5651c14b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6825","nonce":1099494037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6827","nonce":1099494037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6831","nonce":1099494037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6829","nonce":1099494037}]},"public_addr":"172.21.15.71:6825/1099494037","cluster_addr":"172.21.15.71:6827/1099494037","heartbeat_back_addr":"172.21.15.71:6831/1099494037","heartbeat_front_addr":"172.21.15.71:6829/1099494037","state":["exists","up"]},{"osd":11,"uuid":"1d178378-b9f5-40f6-99ef-3d3d445da640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6825","nonce":2941055364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6827","nonce":2941055364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6831","nonce":2941055364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6829","nonce":2941055364}]},"public_addr":"172.21.15.151:6825/2941055364","cluster_addr":"172.21.15.151:6827/2941055364","heartbeat_back_addr":"172.21.15.151:6831/2941055364","heartbeat_front_addr":"172.21.15.151:6829/2941055364","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:02.269586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:27.819386+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:51.552277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:25.052674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:51.873529+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:17.127867+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:51.588249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:18.381450+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:44.140534+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:17.634362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:45.469340+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:45:12.003433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3533611957":"2024-03-29T17:38:07.505611+0000","172.21.15.45:0/3625922430":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6801/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/482610468":"2024-03-29T17:37:32.424229+0000","172.21.15.45:6801/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3649804714":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/1198364079":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3211303958":"2024-03-29T17:38:07.505611+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T17:46:08.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.600+0000 7f2f0cc00640 1 -- 172.21.15.45:0/1726939918 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2eec0a2880 msgr2=0x7f2eec0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:08.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.600+0000 7f2f0cc00640 1 --2- 172.21.15.45:0/1726939918 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2eec0a2880 0x7f2eec0a4d40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2ef40095f0 tx=0x7f2ef4005f90 comp rx=0 tx=0).stop 2024-03-28T17:46:08.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.600+0000 7f2f0cc00640 1 -- 172.21.15.45:0/1726939918 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f08076060 msgr2=0x7f2f0819df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:08.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.600+0000 7f2f0cc00640 1 --2- 172.21.15.45:0/1726939918 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f08076060 0x7f2f0819df40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2ef800e9f0 tx=0x7f2ef800eeb0 comp rx=0 tx=0).stop 2024-03-28T17:46:08.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.600+0000 7f2f0cc00640 1 -- 172.21.15.45:0/1726939918 shutdown_connections 2024-03-28T17:46:08.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.600+0000 7f2f0cc00640 1 --2- 172.21.15.45:0/1726939918 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2eec0a2880 0x7f2eec0a4d40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:08.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.600+0000 7f2f0cc00640 1 --2- 172.21.15.45:0/1726939918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f08077c40 0x7f2f081a54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:08.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.600+0000 7f2f0cc00640 1 --2- 172.21.15.45:0/1726939918 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f080772a0 0x7f2f0819e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:08.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.600+0000 7f2f0cc00640 1 --2- 172.21.15.45:0/1726939918 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f08076060 0x7f2f0819df40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:08.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.600+0000 7f2f0cc00640 1 -- 172.21.15.45:0/1726939918 >> 172.21.15.45:0/1726939918 conn(0x7f2f080fee50 msgr2=0x7f2f08100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:08.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.600+0000 7f2f0cc00640 1 -- 172.21.15.45:0/1726939918 shutdown_connections 2024-03-28T17:46:08.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.600+0000 7f2f0cc00640 1 -- 172.21.15.45:0/1726939918 wait complete. 2024-03-28T17:46:08.624 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T17:40:54.208375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T17:46:08.625 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T17:46:08.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.824+0000 7f8744a00640 1 -- 172.21.15.45:0/1290779863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8740107390 msgr2=0x7f87401097e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:08.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.824+0000 7f8744a00640 1 --2- 172.21.15.45:0/1290779863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8740107390 0x7f87401097e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8738009ca0 tx=0x7f873802c0f0 comp rx=0 tx=0).stop 2024-03-28T17:46:08.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.824+0000 7f8744a00640 1 -- 172.21.15.45:0/1290779863 shutdown_connections 2024-03-28T17:46:08.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.824+0000 7f8744a00640 1 --2- 172.21.15.45:0/1290779863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8740107390 0x7f87401097e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:08.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.824+0000 7f8744a00640 1 --2- 172.21.15.45:0/1290779863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8740104a60 0x7f8740106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:08.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.824+0000 7f8744a00640 1 --2- 172.21.15.45:0/1290779863 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8740102130 0x7f8740104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:08.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.824+0000 7f8744a00640 1 -- 172.21.15.45:0/1290779863 >> 172.21.15.45:0/1290779863 conn(0x7f87400fbb40 msgr2=0x7f87400fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:08.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.824+0000 7f8744a00640 1 -- 172.21.15.45:0/1290779863 shutdown_connections 2024-03-28T17:46:08.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.824+0000 7f8744a00640 1 -- 172.21.15.45:0/1290779863 wait complete. 2024-03-28T17:46:08.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.825+0000 7f8744a00640 1 Processor -- start 2024-03-28T17:46:08.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.825+0000 7f8744a00640 1 -- start start 2024-03-28T17:46:08.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.825+0000 7f8744a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8740102130 0x7f8740101aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:08.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.825+0000 7f8744a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8740104a60 0x7f87401000e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:08.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.825+0000 7f8744a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8740107390 0x7f8740100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:08.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.825+0000 7f8744a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874010d350 con 0x7f8740107390 2024-03-28T17:46:08.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.825+0000 7f8744a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874010d1d0 con 0x7f8740104a60 2024-03-28T17:46:08.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.825+0000 7f8744a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874010d4d0 con 0x7f8740102130 2024-03-28T17:46:08.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.826+0000 7f873d400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8740104a60 0x7f87401000e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:08.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.826+0000 7f873e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8740107390 0x7f8740100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:08.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.826+0000 7f873de00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8740102130 0x7f8740101aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:08.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.826+0000 7f873d400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8740104a60 0x7f87401000e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:41166/0 (socket says 172.21.15.45:41166) 2024-03-28T17:46:08.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.826+0000 7f873d400640 1 -- 172.21.15.45:0/1400661396 learned_addr learned my addr 172.21.15.45:0/1400661396 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:08.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.826+0000 7f873d400640 1 -- 172.21.15.45:0/1400661396 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8740102130 msgr2=0x7f8740101aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:08.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.826+0000 7f873d400640 1 --2- 172.21.15.45:0/1400661396 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8740102130 0x7f8740101aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:08.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.826+0000 7f873d400640 1 -- 172.21.15.45:0/1400661396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8740107390 msgr2=0x7f8740100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:08.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.826+0000 7f873d400640 1 --2- 172.21.15.45:0/1400661396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8740107390 0x7f8740100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:08.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.826+0000 7f873d400640 1 -- 172.21.15.45:0/1400661396 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8740100eb0 con 0x7f8740104a60 2024-03-28T17:46:08.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.826+0000 7f873d400640 1 --2- 172.21.15.45:0/1400661396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8740104a60 0x7f87401000e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f872c00e9a0 tx=0x7f872c00ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:08.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.826+0000 7f8736a00640 1 -- 172.21.15.45:0/1400661396 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c00cd80 con 0x7f8740104a60 2024-03-28T17:46:08.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.826+0000 7f8736a00640 1 -- 172.21.15.45:0/1400661396 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f872c0035f0 con 0x7f8740104a60 2024-03-28T17:46:08.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.826+0000 7f8736a00640 1 -- 172.21.15.45:0/1400661396 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c010670 con 0x7f8740104a60 2024-03-28T17:46:08.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.827+0000 7f8744a00640 1 -- 172.21.15.45:0/1400661396 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87401a1ab0 con 0x7f8740104a60 2024-03-28T17:46:08.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.827+0000 7f8744a00640 1 -- 172.21.15.45:0/1400661396 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87401a2000 con 0x7f8740104a60 2024-03-28T17:46:08.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.828+0000 7f8744a00640 1 -- 172.21.15.45:0/1400661396 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87400fd630 con 0x7f8740104a60 2024-03-28T17:46:08.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.830+0000 7f8736a00640 1 -- 172.21.15.45:0/1400661396 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f872c004020 con 0x7f8740104a60 2024-03-28T17:46:08.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.831+0000 7f8736a00640 1 --2- 172.21.15.45:0/1400661396 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f87240a2800 0x7f87240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:08.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.831+0000 7f8736a00640 1 -- 172.21.15.45:0/1400661396 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f872c0ce950 con 0x7f8740104a60 2024-03-28T17:46:08.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.832+0000 7f873de00640 1 --2- 172.21.15.45:0/1400661396 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f87240a2800 0x7f87240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:08.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.832+0000 7f873de00640 1 --2- 172.21.15.45:0/1400661396 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f87240a2800 0x7f87240a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8730009bd0 tx=0x7f8730005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:08.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:08.835+0000 7f8736a00640 1 -- 172.21.15.45:0/1400661396 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f872c09bdf0 con 0x7f8740104a60 2024-03-28T17:46:08.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:08.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:08 smithi045 ceph-mon[31773]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:08.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:08.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:08.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:08.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:08.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:08 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1406545019' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:46:08.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:08 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1726939918' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:46:09.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:09.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:08 smithi071 ceph-mon[36597]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:09.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:09.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:09.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:09.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:09.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:08 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1406545019' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:46:09.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:08 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1726939918' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:46:09.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:09.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:08 smithi151 ceph-mon[36481]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:09.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:09.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:09.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:09.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:09.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:08 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1406545019' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:46:09.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:08 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1726939918' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:46:09.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.077+0000 7f8744a00640 1 -- 172.21.15.45:0/1400661396 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f87400622a0 con 0x7f8740104a60 2024-03-28T17:46:09.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.077+0000 7f8736a00640 1 -- 172.21.15.45:0/1400661396 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f872c09b6f0 con 0x7f8740104a60 2024-03-28T17:46:09.078 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-28T17:46:09.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.080+0000 7f8744a00640 1 -- 172.21.15.45:0/1400661396 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f87240a2800 msgr2=0x7f87240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:09.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.080+0000 7f8744a00640 1 --2- 172.21.15.45:0/1400661396 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f87240a2800 0x7f87240a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8730009bd0 tx=0x7f8730005f90 comp rx=0 tx=0).stop 2024-03-28T17:46:09.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.080+0000 7f8744a00640 1 -- 172.21.15.45:0/1400661396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8740104a60 msgr2=0x7f87401000e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:09.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.080+0000 7f8744a00640 1 --2- 172.21.15.45:0/1400661396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8740104a60 0x7f87401000e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f872c00e9a0 tx=0x7f872c00ee60 comp rx=0 tx=0).stop 2024-03-28T17:46:09.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.080+0000 7f8744a00640 1 -- 172.21.15.45:0/1400661396 shutdown_connections 2024-03-28T17:46:09.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.080+0000 7f8744a00640 1 --2- 172.21.15.45:0/1400661396 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f87240a2800 0x7f87240a4cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:09.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.080+0000 7f8744a00640 1 --2- 172.21.15.45:0/1400661396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8740107390 0x7f8740100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:09.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.080+0000 7f8744a00640 1 --2- 172.21.15.45:0/1400661396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8740104a60 0x7f87401000e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:09.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.080+0000 7f8744a00640 1 --2- 172.21.15.45:0/1400661396 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8740102130 0x7f8740101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:09.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.081+0000 7f8744a00640 1 -- 172.21.15.45:0/1400661396 >> 172.21.15.45:0/1400661396 conn(0x7f87400fbb40 msgr2=0x7f8740102a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:09.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.081+0000 7f8744a00640 1 -- 172.21.15.45:0/1400661396 shutdown_connections 2024-03-28T17:46:09.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.081+0000 7f8744a00640 1 -- 172.21.15.45:0/1400661396 wait complete. 2024-03-28T17:46:09.107 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T17:46:09.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.323+0000 7f71eae00640 1 -- 172.21.15.45:0/3536737451 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71e40ffd60 msgr2=0x7f71e4102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:09.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.323+0000 7f71eae00640 1 --2- 172.21.15.45:0/3536737451 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71e40ffd60 0x7f71e4102190 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f71d8009ca0 tx=0x7f71d802c100 comp rx=0 tx=0).stop 2024-03-28T17:46:09.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.323+0000 7f71eae00640 1 -- 172.21.15.45:0/3536737451 shutdown_connections 2024-03-28T17:46:09.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.323+0000 7f71eae00640 1 --2- 172.21.15.45:0/3536737451 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71e4105080 0x7f71e4105500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:09.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.323+0000 7f71eae00640 1 --2- 172.21.15.45:0/3536737451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71e41026d0 0x7f71e4104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:09.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.323+0000 7f71eae00640 1 --2- 172.21.15.45:0/3536737451 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71e40ffd60 0x7f71e4102190 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:09.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.323+0000 7f71eae00640 1 -- 172.21.15.45:0/3536737451 >> 172.21.15.45:0/3536737451 conn(0x7f71e40fb980 msgr2=0x7f71e40fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:09.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.323+0000 7f71eae00640 1 -- 172.21.15.45:0/3536737451 shutdown_connections 2024-03-28T17:46:09.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.323+0000 7f71eae00640 1 -- 172.21.15.45:0/3536737451 wait complete. 2024-03-28T17:46:09.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.324+0000 7f71eae00640 1 Processor -- start 2024-03-28T17:46:09.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.324+0000 7f71eae00640 1 -- start start 2024-03-28T17:46:09.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.325+0000 7f71eae00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71e40ffd60 0x7f71e419dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:09.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.325+0000 7f71eae00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71e41026d0 0x7f71e419e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:09.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.325+0000 7f71eae00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71e4105080 0x7f71e41a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:09.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.325+0000 7f71eae00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e4113740 con 0x7f71e4105080 2024-03-28T17:46:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.325+0000 7f71eae00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e41135c0 con 0x7f71e40ffd60 2024-03-28T17:46:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.325+0000 7f71eae00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e41138c0 con 0x7f71e41026d0 2024-03-28T17:46:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.325+0000 7f71ea400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71e4105080 0x7f71e41a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.325+0000 7f71ea400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71e4105080 0x7f71e41a5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59504/0 (socket says 172.21.15.45:59504) 2024-03-28T17:46:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.325+0000 7f71ea400640 1 -- 172.21.15.45:0/3588198006 learned_addr learned my addr 172.21.15.45:0/3588198006 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.325+0000 7f71e9000640 1 --2- 172.21.15.45:0/3588198006 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71e41026d0 0x7f71e419e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.325+0000 7f71e9a00640 1 --2- 172.21.15.45:0/3588198006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71e40ffd60 0x7f71e419dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.326+0000 7f71e9000640 1 -- 172.21.15.45:0/3588198006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71e40ffd60 msgr2=0x7f71e419dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.326+0000 7f71e9000640 1 --2- 172.21.15.45:0/3588198006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71e40ffd60 0x7f71e419dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.326+0000 7f71e9000640 1 -- 172.21.15.45:0/3588198006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71e4105080 msgr2=0x7f71e41a5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.326+0000 7f71e9000640 1 --2- 172.21.15.45:0/3588198006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71e4105080 0x7f71e41a5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.326+0000 7f71e9000640 1 -- 172.21.15.45:0/3588198006 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e41a59b0 con 0x7f71e41026d0 2024-03-28T17:46:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.326+0000 7f71ea400640 1 --2- 172.21.15.45:0/3588198006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71e4105080 0x7f71e41a5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:46:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.326+0000 7f71e9000640 1 --2- 172.21.15.45:0/3588198006 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71e41026d0 0x7f71e419e2e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f71d0005fd0 tx=0x7f71d000dd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:09.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.326+0000 7f71e2000640 1 -- 172.21.15.45:0/3588198006 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d0009c30 con 0x7f71e41026d0 2024-03-28T17:46:09.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.326+0000 7f71e2000640 1 -- 172.21.15.45:0/3588198006 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f71d000be30 con 0x7f71e41026d0 2024-03-28T17:46:09.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.326+0000 7f71eae00640 1 -- 172.21.15.45:0/3588198006 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e41a5ca0 con 0x7f71e41026d0 2024-03-28T17:46:09.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.326+0000 7f71e2000640 1 -- 172.21.15.45:0/3588198006 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d0010460 con 0x7f71e41026d0 2024-03-28T17:46:09.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.327+0000 7f71eae00640 1 -- 172.21.15.45:0/3588198006 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e41a61c0 con 0x7f71e41026d0 2024-03-28T17:46:09.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.328+0000 7f71eae00640 1 -- 172.21.15.45:0/3588198006 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e40fd470 con 0x7f71e41026d0 2024-03-28T17:46:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.330+0000 7f71e2000640 1 -- 172.21.15.45:0/3588198006 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f71d0003e70 con 0x7f71e41026d0 2024-03-28T17:46:09.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.331+0000 7f71e2000640 1 --2- 172.21.15.45:0/3588198006 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f71cc0a2880 0x7f71cc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:09.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.332+0000 7f71e2000640 1 -- 172.21.15.45:0/3588198006 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f71d00ce490 con 0x7f71e41026d0 2024-03-28T17:46:09.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.332+0000 7f71e9a00640 1 --2- 172.21.15.45:0/3588198006 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f71cc0a2880 0x7f71cc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:09.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.333+0000 7f71e9a00640 1 --2- 172.21.15.45:0/3588198006 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f71cc0a2880 0x7f71cc0a4d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f71d802c710 tx=0x7f71d80323e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:09.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.336+0000 7f71e2000640 1 -- 172.21.15.45:0/3588198006 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f71d0017020 con 0x7f71e41026d0 2024-03-28T17:46:09.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.552+0000 7f71eae00640 1 -- 172.21.15.45:0/3588198006 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f71e402d2e0 con 0x7f71e41026d0 2024-03-28T17:46:09.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.552+0000 7f71e2000640 1 -- 172.21.15.45:0/3588198006 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) v1 ==== 84+0+65 (secure 0 0 0) 0x7f71d009b930 con 0x7f71e41026d0 2024-03-28T17:46:09.553 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:46:09.553 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2024-03-28T17:46:09.553 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-28T17:46:09.553 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2024-03-28T17:46:09.553 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2024-03-28T17:46:09.553 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:46:09.553 INFO:teuthology.orchestra.run.smithi045.stdout:] 2024-03-28T17:46:09.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.555+0000 7f71eae00640 1 -- 172.21.15.45:0/3588198006 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f71cc0a2880 msgr2=0x7f71cc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:09.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.555+0000 7f71eae00640 1 --2- 172.21.15.45:0/3588198006 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f71cc0a2880 0x7f71cc0a4d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f71d802c710 tx=0x7f71d80323e0 comp rx=0 tx=0).stop 2024-03-28T17:46:09.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.555+0000 7f71eae00640 1 -- 172.21.15.45:0/3588198006 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71e41026d0 msgr2=0x7f71e419e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:09.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.555+0000 7f71eae00640 1 --2- 172.21.15.45:0/3588198006 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71e41026d0 0x7f71e419e2e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f71d0005fd0 tx=0x7f71d000dd20 comp rx=0 tx=0).stop 2024-03-28T17:46:09.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.555+0000 7f71eae00640 1 -- 172.21.15.45:0/3588198006 shutdown_connections 2024-03-28T17:46:09.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.555+0000 7f71eae00640 1 --2- 172.21.15.45:0/3588198006 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f71cc0a2880 0x7f71cc0a4d40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:09.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.555+0000 7f71eae00640 1 --2- 172.21.15.45:0/3588198006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71e4105080 0x7f71e41a5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:09.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.555+0000 7f71eae00640 1 --2- 172.21.15.45:0/3588198006 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71e41026d0 0x7f71e419e2e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:09.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.555+0000 7f71eae00640 1 --2- 172.21.15.45:0/3588198006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71e40ffd60 0x7f71e419dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:09.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.555+0000 7f71eae00640 1 -- 172.21.15.45:0/3588198006 >> 172.21.15.45:0/3588198006 conn(0x7f71e40fb980 msgr2=0x7f71e4103410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:09.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.555+0000 7f71eae00640 1 -- 172.21.15.45:0/3588198006 shutdown_connections 2024-03-28T17:46:09.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.556+0000 7f71eae00640 1 -- 172.21.15.45:0/3588198006 wait complete. 2024-03-28T17:46:09.574 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-28T17:46:09.574 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-28T17:46:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.774+0000 7f3c19a00640 1 -- 172.21.15.45:0/2675277805 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c14104ad0 msgr2=0x7f3c14104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.774+0000 7f3c19a00640 1 --2- 172.21.15.45:0/2675277805 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c14104ad0 0x7f3c14104f50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3c00009900 tx=0x7f3c0002bc30 comp rx=0 tx=0).stop 2024-03-28T17:46:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.774+0000 7f3c19a00640 1 -- 172.21.15.45:0/2675277805 shutdown_connections 2024-03-28T17:46:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.774+0000 7f3c19a00640 1 --2- 172.21.15.45:0/2675277805 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c14105490 0x7f3c1410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.774+0000 7f3c19a00640 1 --2- 172.21.15.45:0/2675277805 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c14104ad0 0x7f3c14104f50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.774+0000 7f3c19a00640 1 --2- 172.21.15.45:0/2675277805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c141038e0 0x7f3c14103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.774+0000 7f3c19a00640 1 -- 172.21.15.45:0/2675277805 >> 172.21.15.45:0/2675277805 conn(0x7f3c140fee50 msgr2=0x7f3c14101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.775+0000 7f3c19a00640 1 -- 172.21.15.45:0/2675277805 shutdown_connections 2024-03-28T17:46:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.775+0000 7f3c19a00640 1 -- 172.21.15.45:0/2675277805 wait complete. 2024-03-28T17:46:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.775+0000 7f3c19a00640 1 Processor -- start 2024-03-28T17:46:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.776+0000 7f3c19a00640 1 -- start start 2024-03-28T17:46:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.776+0000 7f3c19a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c141038e0 0x7f3c14199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.776+0000 7f3c19a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c14104ad0 0x7f3c1419a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.776+0000 7f3c19a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c14105490 0x7f3c141a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.776+0000 7f3c19a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1410f590 con 0x7f3c141038e0 2024-03-28T17:46:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.776+0000 7f3c19a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1410f410 con 0x7f3c14105490 2024-03-28T17:46:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.776+0000 7f3c19a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1410f710 con 0x7f3c14104ad0 2024-03-28T17:46:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.776+0000 7f3c13400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c14105490 0x7f3c141a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.776+0000 7f3c13400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c14105490 0x7f3c141a11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:41198/0 (socket says 172.21.15.45:41198) 2024-03-28T17:46:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.776+0000 7f3c12000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c14104ad0 0x7f3c1419a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.776+0000 7f3c13400640 1 -- 172.21.15.45:0/2615214305 learned_addr learned my addr 172.21.15.45:0/2615214305 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.776+0000 7f3c12a00640 1 --2- 172.21.15.45:0/2615214305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c141038e0 0x7f3c14199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.777+0000 7f3c13400640 1 -- 172.21.15.45:0/2615214305 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c14104ad0 msgr2=0x7f3c1419a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.777+0000 7f3c13400640 1 --2- 172.21.15.45:0/2615214305 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c14104ad0 0x7f3c1419a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.777+0000 7f3c13400640 1 -- 172.21.15.45:0/2615214305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c141038e0 msgr2=0x7f3c14199c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.777+0000 7f3c13400640 1 --2- 172.21.15.45:0/2615214305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c141038e0 0x7f3c14199c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.777+0000 7f3c13400640 1 -- 172.21.15.45:0/2615214305 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c141a1860 con 0x7f3c14105490 2024-03-28T17:46:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.777+0000 7f3c13400640 1 --2- 172.21.15.45:0/2615214305 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c14105490 0x7f3c141a11c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c00efd0 tx=0x7f3c0c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.777+0000 7f3c0b400640 1 -- 172.21.15.45:0/2615214305 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c003b70 con 0x7f3c14105490 2024-03-28T17:46:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.777+0000 7f3c19a00640 1 -- 172.21.15.45:0/2615214305 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c141a1b50 con 0x7f3c14105490 2024-03-28T17:46:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.777+0000 7f3c0b400640 1 -- 172.21.15.45:0/2615214305 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c0c003d10 con 0x7f3c14105490 2024-03-28T17:46:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.777+0000 7f3c0b400640 1 -- 172.21.15.45:0/2615214305 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c0109f0 con 0x7f3c14105490 2024-03-28T17:46:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.778+0000 7f3c19a00640 1 -- 172.21.15.45:0/2615214305 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c141a20a0 con 0x7f3c14105490 2024-03-28T17:46:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.779+0000 7f3c19a00640 1 -- 172.21.15.45:0/2615214305 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c1410ac50 con 0x7f3c14105490 2024-03-28T17:46:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.780+0000 7f3c0b400640 1 -- 172.21.15.45:0/2615214305 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3c0c004020 con 0x7f3c14105490 2024-03-28T17:46:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.782+0000 7f3c0b400640 1 --2- 172.21.15.45:0/2615214305 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3bf80a2800 0x7f3bf80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.782+0000 7f3c0b400640 1 -- 172.21.15.45:0/2615214305 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3c0c0ceaf0 con 0x7f3c14105490 2024-03-28T17:46:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.782+0000 7f3c12a00640 1 --2- 172.21.15.45:0/2615214305 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3bf80a2800 0x7f3bf80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.783+0000 7f3c12a00640 1 --2- 172.21.15.45:0/2615214305 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3bf80a2800 0x7f3bf80a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3c1419b120 tx=0x7f3c04005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:09.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.790+0000 7f3c0b400640 1 -- 172.21.15.45:0/2615214305 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3c0c09bf90 con 0x7f3c14105490 2024-03-28T17:46:09.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:09 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1400661396' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:46:09.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:09.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:09.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:46:09.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:46:09.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:09.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:46:09.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:09 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3588198006' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T17:46:09.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:09.994+0000 7f3c19a00640 1 -- 172.21.15.45:0/2615214305 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f3c140622d0 con 0x7f3c14105490 2024-03-28T17:46:10.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:09 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1400661396' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:46:10.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:09 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:10.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:09 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:10.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:09 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:46:10.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:09 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:46:10.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:09 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:10.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:09 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:46:10.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:09 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3588198006' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T17:46:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:09 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1400661396' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:46:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:46:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:46:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:46:10.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:09 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3588198006' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T17:46:10.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.652+0000 7f3c0b400640 1 -- 172.21.15.45:0/2615214305 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3c0c09b890 con 0x7f3c14105490 2024-03-28T17:46:10.653 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' created 2024-03-28T17:46:10.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.656+0000 7f3c19a00640 1 -- 172.21.15.45:0/2615214305 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3bf80a2800 msgr2=0x7f3bf80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:10.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.656+0000 7f3c19a00640 1 --2- 172.21.15.45:0/2615214305 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3bf80a2800 0x7f3bf80a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3c1419b120 tx=0x7f3c04005f90 comp rx=0 tx=0).stop 2024-03-28T17:46:10.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.656+0000 7f3c19a00640 1 -- 172.21.15.45:0/2615214305 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c14105490 msgr2=0x7f3c141a11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:10.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.656+0000 7f3c19a00640 1 --2- 172.21.15.45:0/2615214305 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c14105490 0x7f3c141a11c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c00efd0 tx=0x7f3c0c00c6a0 comp rx=0 tx=0).stop 2024-03-28T17:46:10.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.656+0000 7f3c19a00640 1 -- 172.21.15.45:0/2615214305 shutdown_connections 2024-03-28T17:46:10.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.656+0000 7f3c19a00640 1 --2- 172.21.15.45:0/2615214305 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3bf80a2800 0x7f3bf80a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:10.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.656+0000 7f3c19a00640 1 --2- 172.21.15.45:0/2615214305 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c14105490 0x7f3c141a11c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:10.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.656+0000 7f3c19a00640 1 --2- 172.21.15.45:0/2615214305 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c14104ad0 0x7f3c1419a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:10.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.656+0000 7f3c19a00640 1 --2- 172.21.15.45:0/2615214305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c141038e0 0x7f3c14199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:10.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.656+0000 7f3c19a00640 1 -- 172.21.15.45:0/2615214305 >> 172.21.15.45:0/2615214305 conn(0x7f3c140fee50 msgr2=0x7f3c14107e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:10.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.657+0000 7f3c19a00640 1 -- 172.21.15.45:0/2615214305 shutdown_connections 2024-03-28T17:46:10.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.657+0000 7f3c19a00640 1 -- 172.21.15.45:0/2615214305 wait complete. 2024-03-28T17:46:10.682 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T17:46:10.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.876+0000 7fb33c400640 1 -- 172.21.15.45:0/3022062184 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb334102120 msgr2=0x7fb334104510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:10.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.876+0000 7fb33c400640 1 --2- 172.21.15.45:0/3022062184 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb334102120 0x7fb334104510 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb328009900 tx=0x7fb32802bc30 comp rx=0 tx=0).stop 2024-03-28T17:46:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.876+0000 7fb33c400640 1 -- 172.21.15.45:0/3022062184 shutdown_connections 2024-03-28T17:46:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.876+0000 7fb33c400640 1 --2- 172.21.15.45:0/3022062184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb334107380 0x7fb3341097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.876+0000 7fb33c400640 1 --2- 172.21.15.45:0/3022062184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb334104a50 0x7fb334106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.876+0000 7fb33c400640 1 --2- 172.21.15.45:0/3022062184 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb334102120 0x7fb334104510 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.876+0000 7fb33c400640 1 -- 172.21.15.45:0/3022062184 >> 172.21.15.45:0/3022062184 conn(0x7fb3340fbb30 msgr2=0x7fb3340fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.876+0000 7fb33c400640 1 -- 172.21.15.45:0/3022062184 shutdown_connections 2024-03-28T17:46:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.876+0000 7fb33c400640 1 -- 172.21.15.45:0/3022062184 wait complete. 2024-03-28T17:46:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.877+0000 7fb33c400640 1 Processor -- start 2024-03-28T17:46:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.877+0000 7fb33c400640 1 -- start start 2024-03-28T17:46:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.877+0000 7fb33c400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb334102120 0x7fb334101a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.877+0000 7fb33c400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb334104a50 0x7fb3341000d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.877+0000 7fb33c400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb334107380 0x7fb334100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.877+0000 7fb33c400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33410d380 con 0x7fb334104a50 2024-03-28T17:46:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.877+0000 7fb33c400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33410d200 con 0x7fb334102120 2024-03-28T17:46:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.877+0000 7fb33c400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33410d500 con 0x7fb334107380 2024-03-28T17:46:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.877+0000 7fb339000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb334104a50 0x7fb3341000d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.877+0000 7fb33a400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb334107380 0x7fb334100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.877+0000 7fb339000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb334104a50 0x7fb3341000d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59528/0 (socket says 172.21.15.45:59528) 2024-03-28T17:46:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.878+0000 7fb339000640 1 -- 172.21.15.45:0/4226097123 learned_addr learned my addr 172.21.15.45:0/4226097123 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.878+0000 7fb33a400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb334107380 0x7fb334100610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:38636/0 (socket says 172.21.15.45:38636) 2024-03-28T17:46:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.878+0000 7fb339a00640 1 --2- 172.21.15.45:0/4226097123 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb334102120 0x7fb334101a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.878+0000 7fb339000640 1 -- 172.21.15.45:0/4226097123 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb334107380 msgr2=0x7fb334100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.878+0000 7fb339000640 1 --2- 172.21.15.45:0/4226097123 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb334107380 0x7fb334100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.878+0000 7fb339000640 1 -- 172.21.15.45:0/4226097123 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb334102120 msgr2=0x7fb334101a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.878+0000 7fb339000640 1 --2- 172.21.15.45:0/4226097123 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb334102120 0x7fb334101a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.878+0000 7fb339000640 1 -- 172.21.15.45:0/4226097123 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb334100ea0 con 0x7fb334104a50 2024-03-28T17:46:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.878+0000 7fb339a00640 1 --2- 172.21.15.45:0/4226097123 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb334102120 0x7fb334101a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:46:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.878+0000 7fb339000640 1 --2- 172.21.15.45:0/4226097123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb334104a50 0x7fb3341000d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb32000e9c0 tx=0x7fb32000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.878+0000 7fb332000640 1 -- 172.21.15.45:0/4226097123 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32000ccc0 con 0x7fb334104a50 2024-03-28T17:46:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.878+0000 7fb332000640 1 -- 172.21.15.45:0/4226097123 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb320005500 con 0x7fb334104a50 2024-03-28T17:46:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.878+0000 7fb332000640 1 -- 172.21.15.45:0/4226097123 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb320010430 con 0x7fb334104a50 2024-03-28T17:46:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.878+0000 7fb33c400640 1 -- 172.21.15.45:0/4226097123 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3341a1b20 con 0x7fb334104a50 2024-03-28T17:46:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.879+0000 7fb33c400640 1 -- 172.21.15.45:0/4226097123 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3341a2070 con 0x7fb334104a50 2024-03-28T17:46:10.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.880+0000 7fb33c400640 1 -- 172.21.15.45:0/4226097123 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3340fd620 con 0x7fb334104a50 2024-03-28T17:46:10.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.882+0000 7fb332000640 1 -- 172.21.15.45:0/4226097123 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb320005000 con 0x7fb334104a50 2024-03-28T17:46:10.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.884+0000 7fb332000640 1 --2- 172.21.15.45:0/4226097123 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb31c0a2800 0x7fb31c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:10.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.884+0000 7fb332000640 1 -- 172.21.15.45:0/4226097123 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7497+0+0 (secure 0 0 0) 0x7fb3200cedf0 con 0x7fb334104a50 2024-03-28T17:46:10.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.884+0000 7fb339a00640 1 --2- 172.21.15.45:0/4226097123 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb31c0a2800 0x7fb31c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:10.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.885+0000 7fb339a00640 1 --2- 172.21.15.45:0/4226097123 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb31c0a2800 0x7fb31c0a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb32802c240 tx=0x7fb32802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:10.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:10.886+0000 7fb332000640 1 -- 172.21.15.45:0/4226097123 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb320053740 con 0x7fb334104a50 2024-03-28T17:46:10.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:10 smithi045 ceph-mon[31773]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:10.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:10 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/2615214305' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T17:46:10.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:10 smithi045 ceph-mon[31773]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T17:46:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:10 smithi071 ceph-mon[36597]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:10 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/2615214305' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T17:46:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:10 smithi071 ceph-mon[36597]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T17:46:11.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:10 smithi151 ceph-mon[36481]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:11.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:10 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/2615214305' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T17:46:11.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:10 smithi151 ceph-mon[36481]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T17:46:11.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.088+0000 7fb33c400640 1 -- 172.21.15.45:0/4226097123 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb33402d070 con 0x7fb334104a50 2024-03-28T17:46:11.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.745+0000 7fb332000640 1 -- 172.21.15.45:0/4226097123 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v83) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb32009c120 con 0x7fb334104a50 2024-03-28T17:46:11.746 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' created 2024-03-28T17:46:11.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.748+0000 7fb33c400640 1 -- 172.21.15.45:0/4226097123 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb31c0a2800 msgr2=0x7fb31c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:11.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.748+0000 7fb33c400640 1 --2- 172.21.15.45:0/4226097123 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb31c0a2800 0x7fb31c0a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb32802c240 tx=0x7fb32802df90 comp rx=0 tx=0).stop 2024-03-28T17:46:11.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.748+0000 7fb33c400640 1 -- 172.21.15.45:0/4226097123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb334104a50 msgr2=0x7fb3341000d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:11.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.748+0000 7fb33c400640 1 --2- 172.21.15.45:0/4226097123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb334104a50 0x7fb3341000d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb32000e9c0 tx=0x7fb32000edc0 comp rx=0 tx=0).stop 2024-03-28T17:46:11.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.749+0000 7fb33c400640 1 -- 172.21.15.45:0/4226097123 shutdown_connections 2024-03-28T17:46:11.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.749+0000 7fb33c400640 1 --2- 172.21.15.45:0/4226097123 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb31c0a2800 0x7fb31c0a4cc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:11.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.749+0000 7fb33c400640 1 --2- 172.21.15.45:0/4226097123 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb334107380 0x7fb334100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:11.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.749+0000 7fb33c400640 1 --2- 172.21.15.45:0/4226097123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb334104a50 0x7fb3341000d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:11.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.749+0000 7fb33c400640 1 --2- 172.21.15.45:0/4226097123 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb334102120 0x7fb334101a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:11.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.749+0000 7fb33c400640 1 -- 172.21.15.45:0/4226097123 >> 172.21.15.45:0/4226097123 conn(0x7fb3340fbb30 msgr2=0x7fb334102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:11.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.749+0000 7fb33c400640 1 -- 172.21.15.45:0/4226097123 shutdown_connections 2024-03-28T17:46:11.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.749+0000 7fb33c400640 1 -- 172.21.15.45:0/4226097123 wait complete. 2024-03-28T17:46:11.797 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-28T17:46:11.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.978+0000 7f89afc00640 1 -- 172.21.15.45:0/2927112926 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89a8105490 msgr2=0x7f89a810bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:11.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.978+0000 7f89afc00640 1 --2- 172.21.15.45:0/2927112926 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89a8105490 0x7f89a810bd20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f89a0009ca0 tx=0x7f89a002c100 comp rx=0 tx=0).stop 2024-03-28T17:46:11.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.978+0000 7f89afc00640 1 -- 172.21.15.45:0/2927112926 shutdown_connections 2024-03-28T17:46:11.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.978+0000 7f89afc00640 1 --2- 172.21.15.45:0/2927112926 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89a8105490 0x7f89a810bd20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:11.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.978+0000 7f89afc00640 1 --2- 172.21.15.45:0/2927112926 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89a8104ad0 0x7f89a8104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:11.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.978+0000 7f89afc00640 1 --2- 172.21.15.45:0/2927112926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89a81038e0 0x7f89a8103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:11.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.978+0000 7f89afc00640 1 -- 172.21.15.45:0/2927112926 >> 172.21.15.45:0/2927112926 conn(0x7f89a80fee50 msgr2=0x7f89a8101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:11.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.978+0000 7f89afc00640 1 -- 172.21.15.45:0/2927112926 shutdown_connections 2024-03-28T17:46:11.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.978+0000 7f89afc00640 1 -- 172.21.15.45:0/2927112926 wait complete. 2024-03-28T17:46:11.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.979+0000 7f89afc00640 1 Processor -- start 2024-03-28T17:46:11.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.980+0000 7f89afc00640 1 -- start start 2024-03-28T17:46:11.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.980+0000 7f89afc00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89a81038e0 0x7f89a8199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:11.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.980+0000 7f89afc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89a8104ad0 0x7f89a819a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:11.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.980+0000 7f89afc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89a8105490 0x7f89a81a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:11.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.980+0000 7f89afc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a810f540 con 0x7f89a8104ad0 2024-03-28T17:46:11.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.980+0000 7f89afc00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a810f3c0 con 0x7f89a8105490 2024-03-28T17:46:11.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.980+0000 7f89afc00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a810f6c0 con 0x7f89a81038e0 2024-03-28T17:46:11.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.980+0000 7f899fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89a8104ad0 0x7f89a819a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:11.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.980+0000 7f89ad200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89a81038e0 0x7f89a8199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:11.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.980+0000 7f89ad200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89a81038e0 0x7f89a8199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:38654/0 (socket says 172.21.15.45:38654) 2024-03-28T17:46:11.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.980+0000 7f89ad200640 1 -- 172.21.15.45:0/466855696 learned_addr learned my addr 172.21.15.45:0/466855696 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:11.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.981+0000 7f89adc00640 1 --2- 172.21.15.45:0/466855696 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89a8105490 0x7f89a81a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:11.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.981+0000 7f899fe00640 1 -- 172.21.15.45:0/466855696 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89a81038e0 msgr2=0x7f89a8199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:11.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.981+0000 7f899fe00640 1 --2- 172.21.15.45:0/466855696 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89a81038e0 0x7f89a8199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:11.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.981+0000 7f899fe00640 1 -- 172.21.15.45:0/466855696 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89a8105490 msgr2=0x7f89a81a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:11.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.981+0000 7f899fe00640 1 --2- 172.21.15.45:0/466855696 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89a8105490 0x7f89a81a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:11.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.981+0000 7f899fe00640 1 -- 172.21.15.45:0/466855696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a81a17a0 con 0x7f89a8104ad0 2024-03-28T17:46:11.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.981+0000 7f89adc00640 1 --2- 172.21.15.45:0/466855696 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89a8105490 0x7f89a81a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:46:11.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.981+0000 7f89ad200640 1 --2- 172.21.15.45:0/466855696 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89a81038e0 0x7f89a8199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:46:11.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.981+0000 7f899fe00640 1 --2- 172.21.15.45:0/466855696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89a8104ad0 0x7f89a819a0c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f899000e9f0 tx=0x7f899000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:11.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.982+0000 7f89a6000640 1 -- 172.21.15.45:0/466855696 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899000ccc0 con 0x7f89a8104ad0 2024-03-28T17:46:11.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.982+0000 7f89afc00640 1 -- 172.21.15.45:0/466855696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a81a1a90 con 0x7f89a8104ad0 2024-03-28T17:46:11.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.982+0000 7f89a6000640 1 -- 172.21.15.45:0/466855696 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8990010910 con 0x7f89a8104ad0 2024-03-28T17:46:11.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.982+0000 7f89a6000640 1 -- 172.21.15.45:0/466855696 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8990018960 con 0x7f89a8104ad0 2024-03-28T17:46:11.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.982+0000 7f89afc00640 1 -- 172.21.15.45:0/466855696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a8075ae0 con 0x7f89a8104ad0 2024-03-28T17:46:11.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.983+0000 7f89afc00640 1 -- 172.21.15.45:0/466855696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89a810ac00 con 0x7f89a8104ad0 2024-03-28T17:46:11.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.985+0000 7f89a6000640 1 -- 172.21.15.45:0/466855696 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8990010ac0 con 0x7f89a8104ad0 2024-03-28T17:46:11.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.986+0000 7f89a6000640 1 --2- 172.21.15.45:0/466855696 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f898c0a2880 0x7f898c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:11.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.987+0000 7f89a6000640 1 -- 172.21.15.45:0/466855696 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f89900cec60 con 0x7f89a8104ad0 2024-03-28T17:46:11.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.987+0000 7f89ad200640 1 --2- 172.21.15.45:0/466855696 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f898c0a2880 0x7f898c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:11.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.987+0000 7f89ad200640 1 --2- 172.21.15.45:0/466855696 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f898c0a2880 0x7f898c0a4d40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f89980058c0 tx=0x7f8998005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:11.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:11.992+0000 7f89a6000640 1 -- 172.21.15.45:0/466855696 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f899009b7c0 con 0x7f89a8104ad0 2024-03-28T17:46:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:11 smithi071 ceph-mon[36597]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T17:46:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:11 smithi071 ceph-mon[36597]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T17:46:12.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:11 smithi151 ceph-mon[36481]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T17:46:12.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:11 smithi151 ceph-mon[36481]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T17:46:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:11 smithi045 ceph-mon[31773]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T17:46:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:11 smithi045 ceph-mon[31773]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T17:46:12.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:12.258+0000 7f89afc00640 1 -- 172.21.15.45:0/466855696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f89a802d070 con 0x7f89a8104ad0 2024-03-28T17:46:13.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:12 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/4226097123' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T17:46:13.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:12 smithi071 ceph-mon[36597]: pgmap v314: 33 pgs: 1 creating+peering, 1 active+clean, 31 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:13.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:12 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/4226097123' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T17:46:13.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:12 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:13.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:12 smithi071 ceph-mon[36597]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T17:46:13.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:12 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:13.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:12 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:13.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:12 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:13.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:12 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:13.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:12 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:13.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:12 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:46:13.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:12 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:46:13.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:12 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:13.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:12 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/466855696' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T17:46:13.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:12 smithi071 ceph-mon[36597]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T17:46:13.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:12 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/4226097123' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T17:46:13.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:12 smithi045 ceph-mon[31773]: pgmap v314: 33 pgs: 1 creating+peering, 1 active+clean, 31 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:13.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:12 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/4226097123' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T17:46:13.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:12 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:13.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:12 smithi045 ceph-mon[31773]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T17:46:13.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:12 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:13.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:12 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:13.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:12 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:13.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:12 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:13.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:12 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:13.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:12 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:46:13.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:12 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:46:13.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:12 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:13.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:12 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/466855696' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T17:46:13.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:12 smithi045 ceph-mon[31773]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T17:46:13.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:12 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/4226097123' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T17:46:13.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:12 smithi151 ceph-mon[36481]: pgmap v314: 33 pgs: 1 creating+peering, 1 active+clean, 31 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:13.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:12 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/4226097123' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T17:46:13.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:12 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:13.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:12 smithi151 ceph-mon[36481]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T17:46:13.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:12 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:13.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:12 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:13.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:12 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:13.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:12 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:13.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:12 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:13.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:12 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:46:13.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:12 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:46:13.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:12 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:46:13.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:12 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/466855696' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T17:46:13.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:12 smithi151 ceph-mon[36481]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T17:46:13.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.242+0000 7f89a6000640 1 -- 172.21.15.45:0/466855696 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f899009f860 con 0x7f89a8104ad0 2024-03-28T17:46:13.243 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 2 and data pool 3 2024-03-28T17:46:13.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.246+0000 7f89afc00640 1 -- 172.21.15.45:0/466855696 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f898c0a2880 msgr2=0x7f898c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:13.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.246+0000 7f89afc00640 1 --2- 172.21.15.45:0/466855696 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f898c0a2880 0x7f898c0a4d40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f89980058c0 tx=0x7f8998005f90 comp rx=0 tx=0).stop 2024-03-28T17:46:13.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.246+0000 7f89afc00640 1 -- 172.21.15.45:0/466855696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89a8104ad0 msgr2=0x7f89a819a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:13.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.246+0000 7f89afc00640 1 --2- 172.21.15.45:0/466855696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89a8104ad0 0x7f89a819a0c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f899000e9f0 tx=0x7f899000eeb0 comp rx=0 tx=0).stop 2024-03-28T17:46:13.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.246+0000 7f89afc00640 1 -- 172.21.15.45:0/466855696 shutdown_connections 2024-03-28T17:46:13.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.246+0000 7f89afc00640 1 --2- 172.21.15.45:0/466855696 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f898c0a2880 0x7f898c0a4d40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:13.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.246+0000 7f89afc00640 1 --2- 172.21.15.45:0/466855696 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89a8105490 0x7f89a81a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:13.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.246+0000 7f89afc00640 1 --2- 172.21.15.45:0/466855696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89a8104ad0 0x7f89a819a0c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:13.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.246+0000 7f89afc00640 1 --2- 172.21.15.45:0/466855696 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89a81038e0 0x7f89a8199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:13.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.246+0000 7f89afc00640 1 -- 172.21.15.45:0/466855696 >> 172.21.15.45:0/466855696 conn(0x7f89a80fee50 msgr2=0x7f89a8107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:13.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.247+0000 7f89afc00640 1 -- 172.21.15.45:0/466855696 shutdown_connections 2024-03-28T17:46:13.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.247+0000 7f89afc00640 1 -- 172.21.15.45:0/466855696 wait complete. 2024-03-28T17:46:13.273 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T17:46:13.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.474+0000 7fbf43000640 1 -- 172.21.15.45:0/3383492077 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf3c1005c0 msgr2=0x7fbf3c1009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:13.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.474+0000 7fbf43000640 1 --2- 172.21.15.45:0/3383492077 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf3c1005c0 0x7fbf3c1009e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c009860 tx=0x7fbf2c02bb70 comp rx=0 tx=0).stop 2024-03-28T17:46:13.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.474+0000 7fbf43000640 1 -- 172.21.15.45:0/3383492077 shutdown_connections 2024-03-28T17:46:13.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.474+0000 7fbf43000640 1 --2- 172.21.15.45:0/3383492077 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf3c102170 0x7fbf3c108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:13.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.474+0000 7fbf43000640 1 --2- 172.21.15.45:0/3383492077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf3c1017b0 0x7fbf3c101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:13.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.474+0000 7fbf43000640 1 --2- 172.21.15.45:0/3383492077 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf3c1005c0 0x7fbf3c1009e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:13.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.474+0000 7fbf43000640 1 -- 172.21.15.45:0/3383492077 >> 172.21.15.45:0/3383492077 conn(0x7fbf3c0fbb30 msgr2=0x7fbf3c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:13.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.474+0000 7fbf43000640 1 -- 172.21.15.45:0/3383492077 shutdown_connections 2024-03-28T17:46:13.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.475+0000 7fbf43000640 1 -- 172.21.15.45:0/3383492077 wait complete. 2024-03-28T17:46:13.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.476+0000 7fbf43000640 1 Processor -- start 2024-03-28T17:46:13.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.476+0000 7fbf43000640 1 -- start start 2024-03-28T17:46:13.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.476+0000 7fbf43000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf3c1005c0 0x7fbf3c104550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:13.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.476+0000 7fbf43000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf3c1017b0 0x7fbf3c102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:13.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.476+0000 7fbf43000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf3c102170 0x7fbf3c1030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:13.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.477+0000 7fbf43000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c10c210 con 0x7fbf3c1017b0 2024-03-28T17:46:13.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.477+0000 7fbf43000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c10c090 con 0x7fbf3c1005c0 2024-03-28T17:46:13.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.477+0000 7fbf43000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c10c390 con 0x7fbf3c102170 2024-03-28T17:46:13.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.477+0000 7fbf33e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf3c1017b0 0x7fbf3c102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:13.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.477+0000 7fbf3be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf3c1005c0 0x7fbf3c104550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:13.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.477+0000 7fbf3be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf3c1005c0 0x7fbf3c104550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:41248/0 (socket says 172.21.15.45:41248) 2024-03-28T17:46:13.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.477+0000 7fbf3be00640 1 -- 172.21.15.45:0/1496130219 learned_addr learned my addr 172.21.15.45:0/1496130219 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:13.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.477+0000 7fbf41000640 1 --2- 172.21.15.45:0/1496130219 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf3c102170 0x7fbf3c1030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.477+0000 7fbf3be00640 1 -- 172.21.15.45:0/1496130219 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf3c102170 msgr2=0x7fbf3c1030d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.477+0000 7fbf3be00640 1 --2- 172.21.15.45:0/1496130219 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf3c102170 0x7fbf3c1030d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.477+0000 7fbf3be00640 1 -- 172.21.15.45:0/1496130219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf3c1017b0 msgr2=0x7fbf3c102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.477+0000 7fbf3be00640 1 --2- 172.21.15.45:0/1496130219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf3c1017b0 0x7fbf3c102b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.477+0000 7fbf3be00640 1 -- 172.21.15.45:0/1496130219 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3c103960 con 0x7fbf3c1005c0 2024-03-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.478+0000 7fbf3be00640 1 --2- 172.21.15.45:0/1496130219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf3c1005c0 0x7fbf3c104550 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c02c0b0 tx=0x7fbf2c032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.478+0000 7fbf39600640 1 -- 172.21.15.45:0/1496130219 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c041020 con 0x7fbf3c1005c0 2024-03-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.478+0000 7fbf39600640 1 -- 172.21.15.45:0/1496130219 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf2c02c9f0 con 0x7fbf3c1005c0 2024-03-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.478+0000 7fbf43000640 1 -- 172.21.15.45:0/1496130219 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf3c1a1aa0 con 0x7fbf3c1005c0 2024-03-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.478+0000 7fbf39600640 1 -- 172.21.15.45:0/1496130219 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c039040 con 0x7fbf3c1005c0 2024-03-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.478+0000 7fbf43000640 1 -- 172.21.15.45:0/1496130219 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf3c1a1ff0 con 0x7fbf3c1005c0 2024-03-28T17:46:13.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.479+0000 7fbf43000640 1 -- 172.21.15.45:0/1496130219 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf3c1a21c0 con 0x7fbf3c1005c0 2024-03-28T17:46:13.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.481+0000 7fbf39600640 1 -- 172.21.15.45:0/1496130219 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbf2c044020 con 0x7fbf3c1005c0 2024-03-28T17:46:13.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.482+0000 7fbf39600640 1 --2- 172.21.15.45:0/1496130219 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fbf200a2800 0x7fbf200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:13.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.483+0000 7fbf39600640 1 -- 172.21.15.45:0/1496130219 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fbf2c0f02d0 con 0x7fbf3c1005c0 2024-03-28T17:46:13.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.483+0000 7fbf33e00640 1 --2- 172.21.15.45:0/1496130219 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fbf200a2800 0x7fbf200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:13.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.483+0000 7fbf33e00640 1 --2- 172.21.15.45:0/1496130219 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fbf200a2800 0x7fbf200a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c1042c0 tx=0x7fbf24009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:13.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.487+0000 7fbf39600640 1 -- 172.21.15.45:0/1496130219 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbf2c04b020 con 0x7fbf3c1005c0 2024-03-28T17:46:13.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.686+0000 7fbf43000640 1 -- 172.21.15.45:0/1496130219 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf3c1a2550 con 0x7fbf3c1005c0 2024-03-28T17:46:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.688+0000 7fbf39600640 1 -- 172.21.15.45:0/1496130219 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+22221 (secure 0 0 0) 0x7fbf2c0bd420 con 0x7fbf3c1005c0 2024-03-28T17:46:13.689 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:46:13.689 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":84,"fsid":"9f4888ec-ed29-11ee-95d0-87774f69a715","created":"2024-03-28T17:36:51.623185+0000","modified":"2024-03-28T17:46:12.260478+0000","last_up_change":"2024-03-28T17:45:14.043432+0000","last_in_change":"2024-03-28T17:44:53.654489+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T17:40:54.208375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T17:46:10.564519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T17:46:11.734367+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"74c56bc2-4169-4919-afd4-2ebd236f23e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6803","nonce":3795754023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6805","nonce":3795754023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6809","nonce":3795754023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6807","nonce":3795754023}]},"public_addr":"172.21.15.45:6803/3795754023","cluster_addr":"172.21.15.45:6805/3795754023","heartbeat_back_addr":"172.21.15.45:6809/3795754023","heartbeat_front_addr":"172.21.15.45:6807/3795754023","state":["exists","up"]},{"osd":1,"uuid":"78dcc21b-75b2-453d-b690-17b55b8ed1a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6801","nonce":2014529906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6803","nonce":2014529906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6807","nonce":2014529906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6805","nonce":2014529906}]},"public_addr":"172.21.15.71:6801/2014529906","cluster_addr":"172.21.15.71:6803/2014529906","heartbeat_back_addr":"172.21.15.71:6807/2014529906","heartbeat_front_addr":"172.21.15.71:6805/2014529906","state":["exists","up"]},{"osd":2,"uuid":"ca9f3f36-b71e-4962-a94f-cedf2f12eb7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6801","nonce":1850933775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6803","nonce":1850933775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6807","nonce":1850933775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6805","nonce":1850933775}]},"public_addr":"172.21.15.151:6801/1850933775","cluster_addr":"172.21.15.151:6803/1850933775","heartbeat_back_addr":"172.21.15.151:6807/1850933775","heartbeat_front_addr":"172.21.15.151:6805/1850933775","state":["exists","up"]},{"osd":3,"uuid":"d3f21f01-2973-413e-8daa-9974b8a44af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6811","nonce":3021569975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6813","nonce":3021569975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6817","nonce":3021569975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6815","nonce":3021569975}]},"public_addr":"172.21.15.45:6811/3021569975","cluster_addr":"172.21.15.45:6813/3021569975","heartbeat_back_addr":"172.21.15.45:6817/3021569975","heartbeat_front_addr":"172.21.15.45:6815/3021569975","state":["exists","up"]},{"osd":4,"uuid":"4a336f42-0605-492b-ad88-56cb02acdcf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6809","nonce":2069578333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6811","nonce":2069578333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6815","nonce":2069578333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6813","nonce":2069578333}]},"public_addr":"172.21.15.71:6809/2069578333","cluster_addr":"172.21.15.71:6811/2069578333","heartbeat_back_addr":"172.21.15.71:6815/2069578333","heartbeat_front_addr":"172.21.15.71:6813/2069578333","state":["exists","up"]},{"osd":5,"uuid":"c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6809","nonce":2629217227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6811","nonce":2629217227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6815","nonce":2629217227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6813","nonce":2629217227}]},"public_addr":"172.21.15.151:6809/2629217227","cluster_addr":"172.21.15.151:6811/2629217227","heartbeat_back_addr":"172.21.15.151:6815/2629217227","heartbeat_front_addr":"172.21.15.151:6813/2629217227","state":["exists","up"]},{"osd":6,"uuid":"58a6c5bd-a8b7-4892-b346-bcd7994bcb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6819","nonce":2077907039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6821","nonce":2077907039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6825","nonce":2077907039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6823","nonce":2077907039}]},"public_addr":"172.21.15.45:6819/2077907039","cluster_addr":"172.21.15.45:6821/2077907039","heartbeat_back_addr":"172.21.15.45:6825/2077907039","heartbeat_front_addr":"172.21.15.45:6823/2077907039","state":["exists","up"]},{"osd":7,"uuid":"f7a45d52-807b-4407-bb04-787b2a52a0ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6817","nonce":2000211251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6819","nonce":2000211251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6823","nonce":2000211251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6821","nonce":2000211251}]},"public_addr":"172.21.15.71:6817/2000211251","cluster_addr":"172.21.15.71:6819/2000211251","heartbeat_back_addr":"172.21.15.71:6823/2000211251","heartbeat_front_addr":"172.21.15.71:6821/2000211251","state":["exists","up"]},{"osd":8,"uuid":"88abcb0f-f189-49be-8315-c700c77369f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6817","nonce":3665460137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6819","nonce":3665460137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6823","nonce":3665460137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6821","nonce":3665460137}]},"public_addr":"172.21.15.151:6817/3665460137","cluster_addr":"172.21.15.151:6819/3665460137","heartbeat_back_addr":"172.21.15.151:6823/3665460137","heartbeat_front_addr":"172.21.15.151:6821/3665460137","state":["exists","up"]},{"osd":9,"uuid":"ed3fc096-961e-4392-abc2-c5aa4d050117","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6827","nonce":2122322684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6829","nonce":2122322684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6833","nonce":2122322684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6831","nonce":2122322684}]},"public_addr":"172.21.15.45:6827/2122322684","cluster_addr":"172.21.15.45:6829/2122322684","heartbeat_back_addr":"172.21.15.45:6833/2122322684","heartbeat_front_addr":"172.21.15.45:6831/2122322684","state":["exists","up"]},{"osd":10,"uuid":"ba94762b-bc5c-4c97-9cc1-17e5651c14b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6825","nonce":1099494037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6827","nonce":1099494037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6831","nonce":1099494037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6829","nonce":1099494037}]},"public_addr":"172.21.15.71:6825/1099494037","cluster_addr":"172.21.15.71:6827/1099494037","heartbeat_back_addr":"172.21.15.71:6831/1099494037","heartbeat_front_addr":"172.21.15.71:6829/1099494037","state":["exists","up"]},{"osd":11,"uuid":"1d178378-b9f5-40f6-99ef-3d3d445da640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6825","nonce":2941055364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6827","nonce":2941055364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6831","nonce":2941055364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6829","nonce":2941055364}]},"public_addr":"172.21.15.151:6825/2941055364","cluster_addr":"172.21.15.151:6827/2941055364","heartbeat_back_addr":"172.21.15.151:6831/2941055364","heartbeat_front_addr":"172.21.15.151:6829/2941055364","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:02.269586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:27.819386+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:51.552277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:25.052674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:51.873529+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:17.127867+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:51.588249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:18.381450+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:44.140534+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:17.634362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:45.469340+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:45:12.003433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3533611957":"2024-03-29T17:38:07.505611+0000","172.21.15.45:0/3625922430":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6801/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/482610468":"2024-03-29T17:37:32.424229+0000","172.21.15.45:6801/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3649804714":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/1198364079":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3211303958":"2024-03-29T17:38:07.505611+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T17:46:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.691+0000 7fbf43000640 1 -- 172.21.15.45:0/1496130219 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fbf200a2800 msgr2=0x7fbf200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.691+0000 7fbf43000640 1 --2- 172.21.15.45:0/1496130219 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fbf200a2800 0x7fbf200a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c1042c0 tx=0x7fbf24009260 comp rx=0 tx=0).stop 2024-03-28T17:46:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.691+0000 7fbf43000640 1 -- 172.21.15.45:0/1496130219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf3c1005c0 msgr2=0x7fbf3c104550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.691+0000 7fbf43000640 1 --2- 172.21.15.45:0/1496130219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf3c1005c0 0x7fbf3c104550 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c02c0b0 tx=0x7fbf2c032430 comp rx=0 tx=0).stop 2024-03-28T17:46:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.691+0000 7fbf43000640 1 -- 172.21.15.45:0/1496130219 shutdown_connections 2024-03-28T17:46:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.691+0000 7fbf43000640 1 --2- 172.21.15.45:0/1496130219 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fbf200a2800 0x7fbf200a4cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.691+0000 7fbf43000640 1 --2- 172.21.15.45:0/1496130219 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf3c102170 0x7fbf3c1030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.691+0000 7fbf43000640 1 --2- 172.21.15.45:0/1496130219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf3c1017b0 0x7fbf3c102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.691+0000 7fbf43000640 1 --2- 172.21.15.45:0/1496130219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf3c1005c0 0x7fbf3c104550 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.691+0000 7fbf43000640 1 -- 172.21.15.45:0/1496130219 >> 172.21.15.45:0/1496130219 conn(0x7fbf3c0fbb30 msgr2=0x7fbf3c0fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.692+0000 7fbf43000640 1 -- 172.21.15.45:0/1496130219 shutdown_connections 2024-03-28T17:46:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.692+0000 7fbf43000640 1 -- 172.21.15.45:0/1496130219 wait complete. 2024-03-28T17:46:13.711 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T17:46:13.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:13 smithi045 ceph-mon[31773]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T17:46:13.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:13 smithi045 ceph-mon[31773]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T17:46:13.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:13 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/466855696' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T17:46:13.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:13 smithi045 ceph-mon[31773]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T17:46:13.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:13 smithi045 ceph-mon[31773]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T17:46:13.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:13 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1496130219' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:46:13.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:13 smithi045 ceph-mon[31773]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T17:46:13.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.915+0000 7fa00d200640 1 -- 172.21.15.45:0/2012761175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0081048a0 msgr2=0x7fa008106c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:13.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.915+0000 7fa00d200640 1 --2- 172.21.15.45:0/2012761175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0081048a0 0x7fa008106c90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4009900 tx=0x7f9ff402bc30 comp rx=0 tx=0).stop 2024-03-28T17:46:13.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.915+0000 7fa00d200640 1 -- 172.21.15.45:0/2012761175 shutdown_connections 2024-03-28T17:46:13.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.915+0000 7fa00d200640 1 --2- 172.21.15.45:0/2012761175 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0081071d0 0x7fa008109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:13.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.915+0000 7fa00d200640 1 --2- 172.21.15.45:0/2012761175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0081048a0 0x7fa008106c90 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:13.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.915+0000 7fa00d200640 1 --2- 172.21.15.45:0/2012761175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa008101f70 0x7fa008104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:13.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.915+0000 7fa00d200640 1 -- 172.21.15.45:0/2012761175 >> 172.21.15.45:0/2012761175 conn(0x7fa0080fb980 msgr2=0x7fa0080fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.916+0000 7fa00d200640 1 -- 172.21.15.45:0/2012761175 shutdown_connections 2024-03-28T17:46:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.916+0000 7fa00d200640 1 -- 172.21.15.45:0/2012761175 wait complete. 2024-03-28T17:46:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.916+0000 7fa00d200640 1 Processor -- start 2024-03-28T17:46:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.917+0000 7fa00d200640 1 -- start start 2024-03-28T17:46:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.917+0000 7fa00d200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa008101f70 0x7fa008101890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.917+0000 7fa00d200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0081048a0 0x7fa0080fff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.917+0000 7fa00d200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0081071d0 0x7fa008100460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.917+0000 7fa00d200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00810d1d0 con 0x7fa008101f70 2024-03-28T17:46:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.917+0000 7fa00d200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00810d050 con 0x7fa0081071d0 2024-03-28T17:46:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.917+0000 7fa00d200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00810d350 con 0x7fa0081048a0 2024-03-28T17:46:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.917+0000 7fa007400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa008101f70 0x7fa008101890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.917+0000 7fa007400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa008101f70 0x7fa008101890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59572/0 (socket says 172.21.15.45:59572) 2024-03-28T17:46:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.917+0000 7fa007400640 1 -- 172.21.15.45:0/3492491350 learned_addr learned my addr 172.21.15.45:0/3492491350 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.917+0000 7fa007e00640 1 --2- 172.21.15.45:0/3492491350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0081071d0 0x7fa008100460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:13.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.917+0000 7fa006a00640 1 --2- 172.21.15.45:0/3492491350 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0081048a0 0x7fa0080fff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.918+0000 7fa007400640 1 -- 172.21.15.45:0/3492491350 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0081048a0 msgr2=0x7fa0080fff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.918+0000 7fa007400640 1 --2- 172.21.15.45:0/3492491350 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0081048a0 0x7fa0080fff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.918+0000 7fa007400640 1 -- 172.21.15.45:0/3492491350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0081071d0 msgr2=0x7fa008100460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.918+0000 7fa007400640 1 --2- 172.21.15.45:0/3492491350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0081071d0 0x7fa008100460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.918+0000 7fa007400640 1 -- 172.21.15.45:0/3492491350 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa008100d20 con 0x7fa008101f70 2024-03-28T17:46:13.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.918+0000 7fa007400640 1 --2- 172.21.15.45:0/3492491350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa008101f70 0x7fa008101890 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9ff800e9f0 tx=0x7f9ff800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:13.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.918+0000 7f9fffe00640 1 -- 172.21.15.45:0/3492491350 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff800cd80 con 0x7fa008101f70 2024-03-28T17:46:13.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.918+0000 7f9fffe00640 1 -- 172.21.15.45:0/3492491350 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ff8010910 con 0x7fa008101f70 2024-03-28T17:46:13.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.918+0000 7fa00d200640 1 -- 172.21.15.45:0/3492491350 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0081a1960 con 0x7fa008101f70 2024-03-28T17:46:13.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.918+0000 7f9fffe00640 1 -- 172.21.15.45:0/3492491350 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff8018990 con 0x7fa008101f70 2024-03-28T17:46:13.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.919+0000 7fa00d200640 1 -- 172.21.15.45:0/3492491350 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0081a1e80 con 0x7fa008101f70 2024-03-28T17:46:13.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.920+0000 7fa00d200640 1 -- 172.21.15.45:0/3492491350 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0080683b0 con 0x7fa008101f70 2024-03-28T17:46:13.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.921+0000 7f9fffe00640 1 -- 172.21.15.45:0/3492491350 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9ff8010ac0 con 0x7fa008101f70 2024-03-28T17:46:13.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.923+0000 7f9fffe00640 1 --2- 172.21.15.45:0/3492491350 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9fec0a2800 0x7f9fec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:13.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.924+0000 7f9fffe00640 1 -- 172.21.15.45:0/3492491350 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f9ff80cece0 con 0x7fa008101f70 2024-03-28T17:46:13.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.924+0000 7fa006a00640 1 --2- 172.21.15.45:0/3492491350 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9fec0a2800 0x7f9fec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:13.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.924+0000 7fa006a00640 1 --2- 172.21.15.45:0/3492491350 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9fec0a2800 0x7f9fec0a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9ff402c170 tx=0x7f9ff402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:13.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:13.927+0000 7f9fffe00640 1 -- 172.21.15.45:0/3492491350 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9ff809b7e0 con 0x7fa008101f70 2024-03-28T17:46:14.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:13 smithi071 ceph-mon[36597]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T17:46:14.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:13 smithi071 ceph-mon[36597]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T17:46:14.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:13 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/466855696' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T17:46:14.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:13 smithi071 ceph-mon[36597]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T17:46:14.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:13 smithi071 ceph-mon[36597]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T17:46:14.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:13 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1496130219' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:46:14.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:13 smithi071 ceph-mon[36597]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T17:46:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:13 smithi151 ceph-mon[36481]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T17:46:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:13 smithi151 ceph-mon[36481]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T17:46:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:13 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/466855696' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T17:46:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:13 smithi151 ceph-mon[36481]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T17:46:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:13 smithi151 ceph-mon[36481]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T17:46:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:13 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1496130219' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:46:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:13 smithi151 ceph-mon[36481]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T17:46:14.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.136+0000 7fa00d200640 1 -- 172.21.15.45:0/3492491350 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0080fd470 con 0x7fa008101f70 2024-03-28T17:46:14.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.139+0000 7f9fffe00640 1 -- 172.21.15.45:0/3492491350 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22208 (secure 0 0 0) 0x7f9ff809f880 con 0x7fa008101f70 2024-03-28T17:46:14.140 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:46:14.140 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":85,"fsid":"9f4888ec-ed29-11ee-95d0-87774f69a715","created":"2024-03-28T17:36:51.623185+0000","modified":"2024-03-28T17:46:13.741486+0000","last_up_change":"2024-03-28T17:45:14.043432+0000","last_in_change":"2024-03-28T17:44:53.654489+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T17:40:54.208375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T17:46:10.564519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T17:46:11.734367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"74c56bc2-4169-4919-afd4-2ebd236f23e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6803","nonce":3795754023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6805","nonce":3795754023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6809","nonce":3795754023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6807","nonce":3795754023}]},"public_addr":"172.21.15.45:6803/3795754023","cluster_addr":"172.21.15.45:6805/3795754023","heartbeat_back_addr":"172.21.15.45:6809/3795754023","heartbeat_front_addr":"172.21.15.45:6807/3795754023","state":["exists","up"]},{"osd":1,"uuid":"78dcc21b-75b2-453d-b690-17b55b8ed1a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6801","nonce":2014529906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6803","nonce":2014529906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6807","nonce":2014529906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6805","nonce":2014529906}]},"public_addr":"172.21.15.71:6801/2014529906","cluster_addr":"172.21.15.71:6803/2014529906","heartbeat_back_addr":"172.21.15.71:6807/2014529906","heartbeat_front_addr":"172.21.15.71:6805/2014529906","state":["exists","up"]},{"osd":2,"uuid":"ca9f3f36-b71e-4962-a94f-cedf2f12eb7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6801","nonce":1850933775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6803","nonce":1850933775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6807","nonce":1850933775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6805","nonce":1850933775}]},"public_addr":"172.21.15.151:6801/1850933775","cluster_addr":"172.21.15.151:6803/1850933775","heartbeat_back_addr":"172.21.15.151:6807/1850933775","heartbeat_front_addr":"172.21.15.151:6805/1850933775","state":["exists","up"]},{"osd":3,"uuid":"d3f21f01-2973-413e-8daa-9974b8a44af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6811","nonce":3021569975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6813","nonce":3021569975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6817","nonce":3021569975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6815","nonce":3021569975}]},"public_addr":"172.21.15.45:6811/3021569975","cluster_addr":"172.21.15.45:6813/3021569975","heartbeat_back_addr":"172.21.15.45:6817/3021569975","heartbeat_front_addr":"172.21.15.45:6815/3021569975","state":["exists","up"]},{"osd":4,"uuid":"4a336f42-0605-492b-ad88-56cb02acdcf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6809","nonce":2069578333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6811","nonce":2069578333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6815","nonce":2069578333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6813","nonce":2069578333}]},"public_addr":"172.21.15.71:6809/2069578333","cluster_addr":"172.21.15.71:6811/2069578333","heartbeat_back_addr":"172.21.15.71:6815/2069578333","heartbeat_front_addr":"172.21.15.71:6813/2069578333","state":["exists","up"]},{"osd":5,"uuid":"c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6809","nonce":2629217227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6811","nonce":2629217227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6815","nonce":2629217227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6813","nonce":2629217227}]},"public_addr":"172.21.15.151:6809/2629217227","cluster_addr":"172.21.15.151:6811/2629217227","heartbeat_back_addr":"172.21.15.151:6815/2629217227","heartbeat_front_addr":"172.21.15.151:6813/2629217227","state":["exists","up"]},{"osd":6,"uuid":"58a6c5bd-a8b7-4892-b346-bcd7994bcb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6819","nonce":2077907039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6821","nonce":2077907039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6825","nonce":2077907039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6823","nonce":2077907039}]},"public_addr":"172.21.15.45:6819/2077907039","cluster_addr":"172.21.15.45:6821/2077907039","heartbeat_back_addr":"172.21.15.45:6825/2077907039","heartbeat_front_addr":"172.21.15.45:6823/2077907039","state":["exists","up"]},{"osd":7,"uuid":"f7a45d52-807b-4407-bb04-787b2a52a0ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6817","nonce":2000211251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6819","nonce":2000211251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6823","nonce":2000211251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6821","nonce":2000211251}]},"public_addr":"172.21.15.71:6817/2000211251","cluster_addr":"172.21.15.71:6819/2000211251","heartbeat_back_addr":"172.21.15.71:6823/2000211251","heartbeat_front_addr":"172.21.15.71:6821/2000211251","state":["exists","up"]},{"osd":8,"uuid":"88abcb0f-f189-49be-8315-c700c77369f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6817","nonce":3665460137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6819","nonce":3665460137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6823","nonce":3665460137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6821","nonce":3665460137}]},"public_addr":"172.21.15.151:6817/3665460137","cluster_addr":"172.21.15.151:6819/3665460137","heartbeat_back_addr":"172.21.15.151:6823/3665460137","heartbeat_front_addr":"172.21.15.151:6821/3665460137","state":["exists","up"]},{"osd":9,"uuid":"ed3fc096-961e-4392-abc2-c5aa4d050117","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6827","nonce":2122322684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6829","nonce":2122322684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6833","nonce":2122322684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6831","nonce":2122322684}]},"public_addr":"172.21.15.45:6827/2122322684","cluster_addr":"172.21.15.45:6829/2122322684","heartbeat_back_addr":"172.21.15.45:6833/2122322684","heartbeat_front_addr":"172.21.15.45:6831/2122322684","state":["exists","up"]},{"osd":10,"uuid":"ba94762b-bc5c-4c97-9cc1-17e5651c14b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6825","nonce":1099494037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6827","nonce":1099494037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6831","nonce":1099494037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6829","nonce":1099494037}]},"public_addr":"172.21.15.71:6825/1099494037","cluster_addr":"172.21.15.71:6827/1099494037","heartbeat_back_addr":"172.21.15.71:6831/1099494037","heartbeat_front_addr":"172.21.15.71:6829/1099494037","state":["exists","up"]},{"osd":11,"uuid":"1d178378-b9f5-40f6-99ef-3d3d445da640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6825","nonce":2941055364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6827","nonce":2941055364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6831","nonce":2941055364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6829","nonce":2941055364}]},"public_addr":"172.21.15.151:6825/2941055364","cluster_addr":"172.21.15.151:6827/2941055364","heartbeat_back_addr":"172.21.15.151:6831/2941055364","heartbeat_front_addr":"172.21.15.151:6829/2941055364","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:02.269586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:27.819386+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:51.552277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:25.052674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:51.873529+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:17.127867+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:51.588249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:18.381450+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:44.140534+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:17.634362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:45.469340+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:45:12.003433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3533611957":"2024-03-29T17:38:07.505611+0000","172.21.15.45:0/3625922430":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6801/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/482610468":"2024-03-29T17:37:32.424229+0000","172.21.15.45:6801/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3649804714":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/1198364079":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3211303958":"2024-03-29T17:38:07.505611+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T17:46:14.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.143+0000 7fa00d200640 1 -- 172.21.15.45:0/3492491350 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9fec0a2800 msgr2=0x7f9fec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:14.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.143+0000 7fa00d200640 1 --2- 172.21.15.45:0/3492491350 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9fec0a2800 0x7f9fec0a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9ff402c170 tx=0x7f9ff402df90 comp rx=0 tx=0).stop 2024-03-28T17:46:14.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.143+0000 7fa00d200640 1 -- 172.21.15.45:0/3492491350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa008101f70 msgr2=0x7fa008101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:14.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.143+0000 7fa00d200640 1 --2- 172.21.15.45:0/3492491350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa008101f70 0x7fa008101890 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9ff800e9f0 tx=0x7f9ff800eeb0 comp rx=0 tx=0).stop 2024-03-28T17:46:14.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.144+0000 7fa00d200640 1 -- 172.21.15.45:0/3492491350 shutdown_connections 2024-03-28T17:46:14.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.144+0000 7fa00d200640 1 --2- 172.21.15.45:0/3492491350 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9fec0a2800 0x7f9fec0a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:14.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.144+0000 7fa00d200640 1 --2- 172.21.15.45:0/3492491350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0081071d0 0x7fa008100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:14.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.144+0000 7fa00d200640 1 --2- 172.21.15.45:0/3492491350 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0081048a0 0x7fa0080fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:14.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.144+0000 7fa00d200640 1 --2- 172.21.15.45:0/3492491350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa008101f70 0x7fa008101890 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:14.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.144+0000 7fa00d200640 1 -- 172.21.15.45:0/3492491350 >> 172.21.15.45:0/3492491350 conn(0x7fa0080fb980 msgr2=0x7fa008102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:14.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.144+0000 7fa00d200640 1 -- 172.21.15.45:0/3492491350 shutdown_connections 2024-03-28T17:46:14.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.144+0000 7fa00d200640 1 -- 172.21.15.45:0/3492491350 wait complete. 2024-03-28T17:46:14.165 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-28T17:46:14.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.381+0000 7f9fcec00640 1 -- 172.21.15.45:0/1932440716 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fc8102000 msgr2=0x7f9fc81043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:14.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.381+0000 7f9fcec00640 1 --2- 172.21.15.45:0/1932440716 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fc8102000 0x7f9fc81043f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc009ca0 tx=0x7f9fbc02c0f0 comp rx=0 tx=0).stop 2024-03-28T17:46:14.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.381+0000 7f9fcec00640 1 -- 172.21.15.45:0/1932440716 shutdown_connections 2024-03-28T17:46:14.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.381+0000 7f9fcec00640 1 --2- 172.21.15.45:0/1932440716 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9fc8107260 0x7f9fc81096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:14.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.381+0000 7f9fcec00640 1 --2- 172.21.15.45:0/1932440716 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fc8104930 0x7f9fc8106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:14.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.381+0000 7f9fcec00640 1 --2- 172.21.15.45:0/1932440716 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fc8102000 0x7f9fc81043f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:14.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.381+0000 7f9fcec00640 1 -- 172.21.15.45:0/1932440716 >> 172.21.15.45:0/1932440716 conn(0x7f9fc80fba10 msgr2=0x7f9fc80fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:14.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.381+0000 7f9fcec00640 1 -- 172.21.15.45:0/1932440716 shutdown_connections 2024-03-28T17:46:14.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.381+0000 7f9fcec00640 1 -- 172.21.15.45:0/1932440716 wait complete. 2024-03-28T17:46:14.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.382+0000 7f9fcec00640 1 Processor -- start 2024-03-28T17:46:14.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.382+0000 7f9fcec00640 1 -- start start 2024-03-28T17:46:14.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.383+0000 7f9fcec00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fc8102000 0x7f9fc8199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:14.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.383+0000 7f9fcec00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fc8104930 0x7f9fc8199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:14.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.383+0000 7f9fcec00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9fc8107260 0x7f9fc81a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:14.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.383+0000 7f9fcec00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc810d210 con 0x7f9fc8102000 2024-03-28T17:46:14.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.383+0000 7f9fcec00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc810d090 con 0x7f9fc8104930 2024-03-28T17:46:14.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.383+0000 7f9fcec00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc810d390 con 0x7f9fc8107260 2024-03-28T17:46:14.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.383+0000 7f9fcd800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fc8102000 0x7f9fc8199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:14.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.383+0000 7f9fcd800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fc8102000 0x7f9fc8199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59594/0 (socket says 172.21.15.45:59594) 2024-03-28T17:46:14.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.383+0000 7f9fcd800640 1 -- 172.21.15.45:0/697438393 learned_addr learned my addr 172.21.15.45:0/697438393 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:14.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.383+0000 7f9fcce00640 1 --2- 172.21.15.45:0/697438393 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fc8104930 0x7f9fc8199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:14.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.383+0000 7f9fce200640 1 --2- 172.21.15.45:0/697438393 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9fc8107260 0x7f9fc81a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:14.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.383+0000 7f9fcd800640 1 -- 172.21.15.45:0/697438393 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9fc8107260 msgr2=0x7f9fc81a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:14.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.383+0000 7f9fcd800640 1 --2- 172.21.15.45:0/697438393 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9fc8107260 0x7f9fc81a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:14.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.383+0000 7f9fcd800640 1 -- 172.21.15.45:0/697438393 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fc8104930 msgr2=0x7f9fc8199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:14.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.383+0000 7f9fcd800640 1 --2- 172.21.15.45:0/697438393 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fc8104930 0x7f9fc8199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:14.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.383+0000 7f9fcd800640 1 -- 172.21.15.45:0/697438393 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc81a16a0 con 0x7f9fc8102000 2024-03-28T17:46:14.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.384+0000 7f9fcd800640 1 --2- 172.21.15.45:0/697438393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fc8102000 0x7f9fc8199a80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc0325f0 tx=0x7f9fbc032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:14.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.384+0000 7f9fc6000640 1 -- 172.21.15.45:0/697438393 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc041020 con 0x7f9fc8102000 2024-03-28T17:46:14.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.384+0000 7f9fc6000640 1 -- 172.21.15.45:0/697438393 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9fbc034cd0 con 0x7f9fc8102000 2024-03-28T17:46:14.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.384+0000 7f9fcec00640 1 -- 172.21.15.45:0/697438393 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fc81a1930 con 0x7f9fc8102000 2024-03-28T17:46:14.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.384+0000 7f9fc6000640 1 -- 172.21.15.45:0/697438393 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc039040 con 0x7f9fc8102000 2024-03-28T17:46:14.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.384+0000 7f9fcec00640 1 -- 172.21.15.45:0/697438393 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fc81a1df0 con 0x7f9fc8102000 2024-03-28T17:46:14.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.386+0000 7f9fcec00640 1 -- 172.21.15.45:0/697438393 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fc8108db0 con 0x7f9fc8102000 2024-03-28T17:46:14.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.388+0000 7f9fc6000640 1 -- 172.21.15.45:0/697438393 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9fbc044020 con 0x7f9fc8102000 2024-03-28T17:46:14.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.390+0000 7f9fc6000640 1 --2- 172.21.15.45:0/697438393 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9fb00a2800 0x7f9fb00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:14.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.390+0000 7f9fc6000640 1 -- 172.21.15.45:0/697438393 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f9fbc0f0860 con 0x7f9fc8102000 2024-03-28T17:46:14.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.390+0000 7f9fcce00640 1 --2- 172.21.15.45:0/697438393 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9fb00a2800 0x7f9fb00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:14.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.391+0000 7f9fcce00640 1 --2- 172.21.15.45:0/697438393 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9fb00a2800 0x7f9fb00a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9fb40095f0 tx=0x7f9fb4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:14.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.394+0000 7f9fc6000640 1 -- 172.21.15.45:0/697438393 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9fbc04b020 con 0x7f9fc8102000 2024-03-28T17:46:14.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:14.631+0000 7f9fcec00640 1 -- 172.21.15.45:0/697438393 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9fc80622a0 con 0x7f9fc8102000 2024-03-28T17:46:15.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:14 smithi071 ceph-mon[36597]: pgmap v317: 97 pgs: 5 creating+peering, 14 active+clean, 78 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:15.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:14 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3492491350' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:46:15.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:14 smithi071 ceph-mon[36597]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T17:46:15.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:14 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:46:15.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:14 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] up:active 2024-03-28T17:46:15.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:14 smithi071 ceph-mon[36597]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T17:46:15.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:14 smithi071 ceph-mon[36597]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T17:46:15.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:14 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/697438393' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T17:46:15.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:14 smithi151 ceph-mon[36481]: pgmap v317: 97 pgs: 5 creating+peering, 14 active+clean, 78 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:15.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:14 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3492491350' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:46:15.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:14 smithi151 ceph-mon[36481]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T17:46:15.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:14 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:46:15.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:14 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] up:active 2024-03-28T17:46:15.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:14 smithi151 ceph-mon[36481]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T17:46:15.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:14 smithi151 ceph-mon[36481]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T17:46:15.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:14 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/697438393' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T17:46:15.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:14 smithi045 ceph-mon[31773]: pgmap v317: 97 pgs: 5 creating+peering, 14 active+clean, 78 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:46:15.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:14 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3492491350' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:46:15.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:14 smithi045 ceph-mon[31773]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T17:46:15.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:14 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:46:15.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:14 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] up:active 2024-03-28T17:46:15.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:14 smithi045 ceph-mon[31773]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T17:46:15.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:14 smithi045 ceph-mon[31773]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T17:46:15.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:14 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/697438393' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T17:46:15.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.265+0000 7f9fc6000640 1 -- 172.21.15.45:0/697438393 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9fbc0bda40 con 0x7f9fc8102000 2024-03-28T17:46:15.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.269+0000 7f9fcec00640 1 -- 172.21.15.45:0/697438393 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9fb00a2800 msgr2=0x7f9fb00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:15.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.269+0000 7f9fcec00640 1 --2- 172.21.15.45:0/697438393 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9fb00a2800 0x7f9fb00a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9fb40095f0 tx=0x7f9fb4005f90 comp rx=0 tx=0).stop 2024-03-28T17:46:15.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.269+0000 7f9fcec00640 1 -- 172.21.15.45:0/697438393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fc8102000 msgr2=0x7f9fc8199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:15.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.269+0000 7f9fcec00640 1 --2- 172.21.15.45:0/697438393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fc8102000 0x7f9fc8199a80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc0325f0 tx=0x7f9fbc032ec0 comp rx=0 tx=0).stop 2024-03-28T17:46:15.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.269+0000 7f9fcec00640 1 -- 172.21.15.45:0/697438393 shutdown_connections 2024-03-28T17:46:15.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.269+0000 7f9fcec00640 1 --2- 172.21.15.45:0/697438393 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9fb00a2800 0x7f9fb00a4cc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:15.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.269+0000 7f9fcec00640 1 --2- 172.21.15.45:0/697438393 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9fc8107260 0x7f9fc81a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:15.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.269+0000 7f9fcec00640 1 --2- 172.21.15.45:0/697438393 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fc8104930 0x7f9fc8199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:15.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.269+0000 7f9fcec00640 1 --2- 172.21.15.45:0/697438393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fc8102000 0x7f9fc8199a80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:15.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.269+0000 7f9fcec00640 1 -- 172.21.15.45:0/697438393 >> 172.21.15.45:0/697438393 conn(0x7f9fc80fba10 msgr2=0x7f9fc8101720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:15.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.269+0000 7f9fcec00640 1 -- 172.21.15.45:0/697438393 shutdown_connections 2024-03-28T17:46:15.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.269+0000 7f9fcec00640 1 -- 172.21.15.45:0/697438393 wait complete. 2024-03-28T17:46:15.289 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-03-28T17:46:15.289 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-28T17:46:15.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.515+0000 7f375be00640 1 -- 172.21.15.45:0/3132391097 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f375c108db0 msgr2=0x7f375c109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:15.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.515+0000 7f375be00640 1 --2- 172.21.15.45:0/3132391097 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f375c108db0 0x7f375c109230 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3748009900 tx=0x7f374802bc40 comp rx=0 tx=0).stop 2024-03-28T17:46:15.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.515+0000 7f375be00640 1 -- 172.21.15.45:0/3132391097 shutdown_connections 2024-03-28T17:46:15.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.515+0000 7f375be00640 1 --2- 172.21.15.45:0/3132391097 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f375c109770 0x7f375c110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:15.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.515+0000 7f375be00640 1 --2- 172.21.15.45:0/3132391097 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f375c108db0 0x7f375c109230 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:15.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.515+0000 7f375be00640 1 --2- 172.21.15.45:0/3132391097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f375c107bc0 0x7f375c107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:15.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.515+0000 7f375be00640 1 -- 172.21.15.45:0/3132391097 >> 172.21.15.45:0/3132391097 conn(0x7f375c076b70 msgr2=0x7f375c078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:15.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.515+0000 7f375be00640 1 -- 172.21.15.45:0/3132391097 shutdown_connections 2024-03-28T17:46:15.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.515+0000 7f375be00640 1 -- 172.21.15.45:0/3132391097 wait complete. 2024-03-28T17:46:15.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.516+0000 7f375be00640 1 Processor -- start 2024-03-28T17:46:15.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.516+0000 7f375be00640 1 -- start start 2024-03-28T17:46:15.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.517+0000 7f375be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f375c107bc0 0x7f375c19deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:15.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.517+0000 7f375be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f375c108db0 0x7f375c19e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:15.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.517+0000 7f375be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f375c109770 0x7f375c1a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:15.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.517+0000 7f375be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375c113890 con 0x7f375c108db0 2024-03-28T17:46:15.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.517+0000 7f375be00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375c113710 con 0x7f375c109770 2024-03-28T17:46:15.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.517+0000 7f375be00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375c113a10 con 0x7f375c107bc0 2024-03-28T17:46:15.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.517+0000 7f375a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f375c108db0 0x7f375c19e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:15.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.517+0000 7f375a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f375c108db0 0x7f375c19e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59614/0 (socket says 172.21.15.45:59614) 2024-03-28T17:46:15.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.517+0000 7f375a000640 1 -- 172.21.15.45:0/3341284081 learned_addr learned my addr 172.21.15.45:0/3341284081 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:15.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.517+0000 7f375b400640 1 --2- 172.21.15.45:0/3341284081 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f375c109770 0x7f375c1a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:15.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.517+0000 7f375aa00640 1 --2- 172.21.15.45:0/3341284081 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f375c107bc0 0x7f375c19deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.518+0000 7f375b400640 1 -- 172.21.15.45:0/3341284081 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f375c107bc0 msgr2=0x7f375c19deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.518+0000 7f375b400640 1 --2- 172.21.15.45:0/3341284081 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f375c107bc0 0x7f375c19deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.518+0000 7f375b400640 1 -- 172.21.15.45:0/3341284081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f375c108db0 msgr2=0x7f375c19e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.518+0000 7f375b400640 1 --2- 172.21.15.45:0/3341284081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f375c108db0 0x7f375c19e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.518+0000 7f375b400640 1 -- 172.21.15.45:0/3341284081 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375c1a5ad0 con 0x7f375c109770 2024-03-28T17:46:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.518+0000 7f375b400640 1 --2- 172.21.15.45:0/3341284081 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f375c109770 0x7f375c1a5460 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3754009fd0 tx=0x7f375400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.518+0000 7f3760a00640 1 -- 172.21.15.45:0/3341284081 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375400ce20 con 0x7f375c109770 2024-03-28T17:46:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.518+0000 7f3760a00640 1 -- 172.21.15.45:0/3341284081 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3754005020 con 0x7f375c109770 2024-03-28T17:46:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.518+0000 7f3760a00640 1 -- 172.21.15.45:0/3341284081 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3754010720 con 0x7f375c109770 2024-03-28T17:46:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.518+0000 7f375be00640 1 -- 172.21.15.45:0/3341284081 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f375c1a5dc0 con 0x7f375c109770 2024-03-28T17:46:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.518+0000 7f375be00640 1 -- 172.21.15.45:0/3341284081 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f375c1a6230 con 0x7f375c109770 2024-03-28T17:46:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.520+0000 7f375be00640 1 -- 172.21.15.45:0/3341284081 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f375c10eee0 con 0x7f375c109770 2024-03-28T17:46:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.521+0000 7f3760a00640 1 -- 172.21.15.45:0/3341284081 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3754005540 con 0x7f375c109770 2024-03-28T17:46:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.522+0000 7f3760a00640 1 --2- 172.21.15.45:0/3341284081 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f37400a2880 0x7f37400a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.523+0000 7f3760a00640 1 -- 172.21.15.45:0/3341284081 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f37540ceea0 con 0x7f375c109770 2024-03-28T17:46:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.523+0000 7f375aa00640 1 --2- 172.21.15.45:0/3341284081 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f37400a2880 0x7f37400a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:15.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.523+0000 7f375aa00640 1 --2- 172.21.15.45:0/3341284081 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f37400a2880 0x7f37400a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f374c005fd0 tx=0x7f374c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:15.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.526+0000 7f3760a00640 1 -- 172.21.15.45:0/3341284081 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f375409bff0 con 0x7f375c109770 2024-03-28T17:46:15.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.772+0000 7f375be00640 1 -- 172.21.15.45:0/3341284081 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f375c02d070 con 0x7f375c109770 2024-03-28T17:46:15.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.931+0000 7f3760a00640 1 -- 172.21.15.45:0/3341284081 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) v1 ==== 105+0+0 (secure 0 0 0) 0x7f375409b8f0 con 0x7f375c109770 2024-03-28T17:46:15.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.937+0000 7f375be00640 1 -- 172.21.15.45:0/3341284081 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f37400a2880 msgr2=0x7f37400a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:15.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.937+0000 7f375be00640 1 --2- 172.21.15.45:0/3341284081 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f37400a2880 0x7f37400a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f374c005fd0 tx=0x7f374c00b410 comp rx=0 tx=0).stop 2024-03-28T17:46:15.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.937+0000 7f375be00640 1 -- 172.21.15.45:0/3341284081 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f375c109770 msgr2=0x7f375c1a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:15.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.937+0000 7f375be00640 1 --2- 172.21.15.45:0/3341284081 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f375c109770 0x7f375c1a5460 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3754009fd0 tx=0x7f375400ef10 comp rx=0 tx=0).stop 2024-03-28T17:46:15.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.939+0000 7f375be00640 1 -- 172.21.15.45:0/3341284081 shutdown_connections 2024-03-28T17:46:15.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.939+0000 7f375be00640 1 --2- 172.21.15.45:0/3341284081 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f37400a2880 0x7f37400a4d40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:15.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.939+0000 7f375be00640 1 --2- 172.21.15.45:0/3341284081 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f375c109770 0x7f375c1a5460 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:15.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.939+0000 7f375be00640 1 --2- 172.21.15.45:0/3341284081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f375c108db0 0x7f375c19e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:15.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.939+0000 7f375be00640 1 --2- 172.21.15.45:0/3341284081 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f375c107bc0 0x7f375c19deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:15.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.939+0000 7f375be00640 1 -- 172.21.15.45:0/3341284081 >> 172.21.15.45:0/3341284081 conn(0x7f375c076b70 msgr2=0x7f375c10c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:15.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.941+0000 7f375be00640 1 -- 172.21.15.45:0/3341284081 shutdown_connections 2024-03-28T17:46:15.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:15.941+0000 7f375be00640 1 -- 172.21.15.45:0/3341284081 wait complete. 2024-03-28T17:46:15.966 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-28T17:46:16.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.171+0000 7fdc4f200640 1 -- 172.21.15.45:0/3940260890 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc50109770 msgr2=0x7fdc50110000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:16.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.171+0000 7fdc4f200640 1 --2- 172.21.15.45:0/3940260890 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc50109770 0x7fdc50110000 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdc48009ca0 tx=0x7fdc4802c100 comp rx=0 tx=0).stop 2024-03-28T17:46:16.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.172+0000 7fdc4f200640 1 -- 172.21.15.45:0/3940260890 shutdown_connections 2024-03-28T17:46:16.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.172+0000 7fdc4f200640 1 --2- 172.21.15.45:0/3940260890 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc50109770 0x7fdc50110000 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:16.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.172+0000 7fdc4f200640 1 --2- 172.21.15.45:0/3940260890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc50108db0 0x7fdc50109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:16.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.172+0000 7fdc4f200640 1 --2- 172.21.15.45:0/3940260890 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc50107bc0 0x7fdc50107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:16.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.172+0000 7fdc4f200640 1 -- 172.21.15.45:0/3940260890 >> 172.21.15.45:0/3940260890 conn(0x7fdc50076b70 msgr2=0x7fdc50078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:16.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.172+0000 7fdc4f200640 1 -- 172.21.15.45:0/3940260890 shutdown_connections 2024-03-28T17:46:16.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.172+0000 7fdc4f200640 1 -- 172.21.15.45:0/3940260890 wait complete. 2024-03-28T17:46:16.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.173+0000 7fdc4f200640 1 Processor -- start 2024-03-28T17:46:16.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.173+0000 7fdc4f200640 1 -- start start 2024-03-28T17:46:16.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.174+0000 7fdc4f200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc50107bc0 0x7fdc5019de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:16.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.174+0000 7fdc4f200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc50108db0 0x7fdc5019e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:16.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.174+0000 7fdc4f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc50109770 0x7fdc501a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:16.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.174+0000 7fdc4f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc50113820 con 0x7fdc50109770 2024-03-28T17:46:16.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.174+0000 7fdc4f200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc501136a0 con 0x7fdc50107bc0 2024-03-28T17:46:16.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.174+0000 7fdc4f200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc501139a0 con 0x7fdc50108db0 2024-03-28T17:46:16.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.174+0000 7fdc4de00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc50107bc0 0x7fdc5019de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:16.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.174+0000 7fdc4de00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc50107bc0 0x7fdc5019de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:57364/0 (socket says 172.21.15.45:57364) 2024-03-28T17:46:16.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.174+0000 7fdc4de00640 1 -- 172.21.15.45:0/2740394872 learned_addr learned my addr 172.21.15.45:0/2740394872 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:16.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.174+0000 7fdc4e800640 1 --2- 172.21.15.45:0/2740394872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc50109770 0x7fdc501a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:16.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.174+0000 7fdc4d400640 1 --2- 172.21.15.45:0/2740394872 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc50108db0 0x7fdc5019e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:16.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.174+0000 7fdc4de00640 1 -- 172.21.15.45:0/2740394872 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc50108db0 msgr2=0x7fdc5019e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:16.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.174+0000 7fdc4de00640 1 --2- 172.21.15.45:0/2740394872 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc50108db0 0x7fdc5019e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:16.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.174+0000 7fdc4de00640 1 -- 172.21.15.45:0/2740394872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc50109770 msgr2=0x7fdc501a53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:16.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.174+0000 7fdc4de00640 1 --2- 172.21.15.45:0/2740394872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc50109770 0x7fdc501a53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:16.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.174+0000 7fdc4de00640 1 -- 172.21.15.45:0/2740394872 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc501a5a20 con 0x7fdc50107bc0 2024-03-28T17:46:16.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.174+0000 7fdc4d400640 1 --2- 172.21.15.45:0/2740394872 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc50108db0 0x7fdc5019e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:46:16.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.175+0000 7fdc4de00640 1 --2- 172.21.15.45:0/2740394872 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc50107bc0 0x7fdc5019de00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdc4000d770 tx=0x7fdc4000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:16.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.175+0000 7fdc46a00640 1 -- 172.21.15.45:0/2740394872 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc40017070 con 0x7fdc50107bc0 2024-03-28T17:46:16.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.175+0000 7fdc46a00640 1 -- 172.21.15.45:0/2740394872 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc40009c30 con 0x7fdc50107bc0 2024-03-28T17:46:16.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.175+0000 7fdc46a00640 1 -- 172.21.15.45:0/2740394872 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc40003dd0 con 0x7fdc50107bc0 2024-03-28T17:46:16.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.175+0000 7fdc4f200640 1 -- 172.21.15.45:0/2740394872 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc501a5d10 con 0x7fdc50107bc0 2024-03-28T17:46:16.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.175+0000 7fdc4f200640 1 -- 172.21.15.45:0/2740394872 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc501a61b0 con 0x7fdc50107bc0 2024-03-28T17:46:16.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.176+0000 7fdc4f200640 1 -- 172.21.15.45:0/2740394872 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc5010eee0 con 0x7fdc50107bc0 2024-03-28T17:46:16.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.178+0000 7fdc46a00640 1 -- 172.21.15.45:0/2740394872 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdc40003f70 con 0x7fdc50107bc0 2024-03-28T17:46:16.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.179+0000 7fdc46a00640 1 --2- 172.21.15.45:0/2740394872 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdc340a2880 0x7fdc340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:16.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.180+0000 7fdc46a00640 1 -- 172.21.15.45:0/2740394872 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fdc400ce690 con 0x7fdc50107bc0 2024-03-28T17:46:16.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.180+0000 7fdc4d400640 1 --2- 172.21.15.45:0/2740394872 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdc340a2880 0x7fdc340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:16.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.180+0000 7fdc4d400640 1 --2- 172.21.15.45:0/2740394872 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdc340a2880 0x7fdc340a4d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdc5019f310 tx=0x7fdc3c00b5f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:16.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.182+0000 7fdc46a00640 1 -- 172.21.15.45:0/2740394872 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdc40009de0 con 0x7fdc50107bc0 2024-03-28T17:46:16.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.433+0000 7fdc4f200640 1 -- 172.21.15.45:0/2740394872 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fdc5002d070 con 0x7fdc50107bc0 2024-03-28T17:46:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:16 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/697438393' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T17:46:16.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:16 smithi071 ceph-mon[36597]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T17:46:16.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:16 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3341284081' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-28T17:46:16.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:16 smithi071 ceph-mon[36597]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-28T17:46:16.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:16 smithi071 ceph-mon[36597]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T17:46:16.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:16 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T17:46:16.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:16 smithi071 ceph-mon[36597]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-28T17:46:16.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:16 smithi071 ceph-mon[36597]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-28T17:46:16.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:16 smithi071 ceph-mon[36597]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-28T17:46:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:16 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/697438393' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T17:46:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:16 smithi151 ceph-mon[36481]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T17:46:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:16 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3341284081' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-28T17:46:16.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:16 smithi151 ceph-mon[36481]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-28T17:46:16.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:16 smithi151 ceph-mon[36481]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T17:46:16.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:16 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T17:46:16.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:16 smithi151 ceph-mon[36481]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-28T17:46:16.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:16 smithi151 ceph-mon[36481]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-28T17:46:16.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:16 smithi151 ceph-mon[36481]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-28T17:46:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:16 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/697438393' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T17:46:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:16 smithi045 ceph-mon[31773]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T17:46:16.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:16 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3341284081' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-28T17:46:16.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:16 smithi045 ceph-mon[31773]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-28T17:46:16.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:16 smithi045 ceph-mon[31773]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T17:46:16.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:16 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T17:46:16.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:16 smithi045 ceph-mon[31773]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-28T17:46:16.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:16 smithi045 ceph-mon[31773]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-28T17:46:16.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:16 smithi045 ceph-mon[31773]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-28T17:46:16.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.950+0000 7fdc46a00640 1 -- 172.21.15.45:0/2740394872 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v19) v1 ==== 121+0+0 (secure 0 0 0) 0x7fdc4009b860 con 0x7fdc50107bc0 2024-03-28T17:46:16.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.953+0000 7fdc4f200640 1 -- 172.21.15.45:0/2740394872 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdc340a2880 msgr2=0x7fdc340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:16.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.953+0000 7fdc4f200640 1 --2- 172.21.15.45:0/2740394872 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdc340a2880 0x7fdc340a4d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdc5019f310 tx=0x7fdc3c00b5f0 comp rx=0 tx=0).stop 2024-03-28T17:46:16.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.953+0000 7fdc4f200640 1 -- 172.21.15.45:0/2740394872 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc50107bc0 msgr2=0x7fdc5019de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:16.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.953+0000 7fdc4f200640 1 --2- 172.21.15.45:0/2740394872 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc50107bc0 0x7fdc5019de00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdc4000d770 tx=0x7fdc4000dc30 comp rx=0 tx=0).stop 2024-03-28T17:46:16.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.953+0000 7fdc4f200640 1 -- 172.21.15.45:0/2740394872 shutdown_connections 2024-03-28T17:46:16.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.953+0000 7fdc4f200640 1 --2- 172.21.15.45:0/2740394872 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdc340a2880 0x7fdc340a4d40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:16.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.953+0000 7fdc4f200640 1 --2- 172.21.15.45:0/2740394872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc50109770 0x7fdc501a53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:16.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.953+0000 7fdc4f200640 1 --2- 172.21.15.45:0/2740394872 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc50108db0 0x7fdc5019e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:16.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.953+0000 7fdc4f200640 1 --2- 172.21.15.45:0/2740394872 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc50107bc0 0x7fdc5019de00 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:16.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.953+0000 7fdc4f200640 1 -- 172.21.15.45:0/2740394872 >> 172.21.15.45:0/2740394872 conn(0x7fdc50076b70 msgr2=0x7fdc5010c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:16.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.954+0000 7fdc4f200640 1 -- 172.21.15.45:0/2740394872 shutdown_connections 2024-03-28T17:46:16.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:16.954+0000 7fdc4f200640 1 -- 172.21.15.45:0/2740394872 wait complete. 2024-03-28T17:46:16.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-28T17:46:17.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.173+0000 7f8032a00640 1 -- 172.21.15.45:0/3778533703 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f802c0ffdf0 msgr2=0x7f802c102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:17.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.173+0000 7f8032a00640 1 --2- 172.21.15.45:0/3778533703 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f802c0ffdf0 0x7f802c102220 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8020009ca0 tx=0x7f802002c0f0 comp rx=0 tx=0).stop 2024-03-28T17:46:17.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.173+0000 7f8032a00640 1 -- 172.21.15.45:0/3778533703 shutdown_connections 2024-03-28T17:46:17.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.173+0000 7f8032a00640 1 --2- 172.21.15.45:0/3778533703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f802c105110 0x7f802c105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:17.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.173+0000 7f8032a00640 1 --2- 172.21.15.45:0/3778533703 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f802c102760 0x7f802c104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:17.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.173+0000 7f8032a00640 1 --2- 172.21.15.45:0/3778533703 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f802c0ffdf0 0x7f802c102220 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:17.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.173+0000 7f8032a00640 1 -- 172.21.15.45:0/3778533703 >> 172.21.15.45:0/3778533703 conn(0x7f802c0fba10 msgr2=0x7f802c0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:17.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.173+0000 7f8032a00640 1 -- 172.21.15.45:0/3778533703 shutdown_connections 2024-03-28T17:46:17.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.173+0000 7f8032a00640 1 -- 172.21.15.45:0/3778533703 wait complete. 2024-03-28T17:46:17.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.174+0000 7f8032a00640 1 Processor -- start 2024-03-28T17:46:17.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.174+0000 7f8032a00640 1 -- start start 2024-03-28T17:46:17.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.175+0000 7f8032a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f802c0ffdf0 0x7f802c19df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:17.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.175+0000 7f8032a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f802c102760 0x7f802c19e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:17.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.175+0000 7f8032a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f802c105110 0x7f802c1a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:17.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.175+0000 7f8032a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802c1137f0 con 0x7f802c0ffdf0 2024-03-28T17:46:17.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.175+0000 7f8032a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802c113670 con 0x7f802c102760 2024-03-28T17:46:17.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.175+0000 7f8032a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802c113970 con 0x7f802c105110 2024-03-28T17:46:17.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.175+0000 7f8031600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f802c0ffdf0 0x7f802c19df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:17.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.175+0000 7f8030c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f802c102760 0x7f802c19e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:17.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.175+0000 7f8031600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f802c0ffdf0 0x7f802c19df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41360/0 (socket says 172.21.15.45:41360) 2024-03-28T17:46:17.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.175+0000 7f8030c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f802c102760 0x7f802c19e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:57368/0 (socket says 172.21.15.45:57368) 2024-03-28T17:46:17.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.175+0000 7f8031600640 1 -- 172.21.15.45:0/2175064051 learned_addr learned my addr 172.21.15.45:0/2175064051 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:17.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.176+0000 7f8032000640 1 --2- 172.21.15.45:0/2175064051 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f802c105110 0x7f802c1a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:17.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.176+0000 7f8030c00640 1 -- 172.21.15.45:0/2175064051 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f802c105110 msgr2=0x7f802c1a5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:17.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.176+0000 7f8030c00640 1 --2- 172.21.15.45:0/2175064051 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f802c105110 0x7f802c1a5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:17.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.176+0000 7f8030c00640 1 -- 172.21.15.45:0/2175064051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f802c0ffdf0 msgr2=0x7f802c19df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:17.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.176+0000 7f8030c00640 1 --2- 172.21.15.45:0/2175064051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f802c0ffdf0 0x7f802c19df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:17.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.176+0000 7f8030c00640 1 -- 172.21.15.45:0/2175064051 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f802c1a5ae0 con 0x7f802c102760 2024-03-28T17:46:17.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.176+0000 7f8032000640 1 --2- 172.21.15.45:0/2175064051 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f802c105110 0x7f802c1a5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:46:17.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.176+0000 7f8030c00640 1 --2- 172.21.15.45:0/2175064051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f802c102760 0x7f802c19e450 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f801800e9f0 tx=0x7f801800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:17.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.176+0000 7f802a000640 1 -- 172.21.15.45:0/2175064051 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801800ccc0 con 0x7f802c102760 2024-03-28T17:46:17.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.176+0000 7f802a000640 1 -- 172.21.15.45:0/2175064051 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8018005020 con 0x7f802c102760 2024-03-28T17:46:17.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.176+0000 7f8032a00640 1 -- 172.21.15.45:0/2175064051 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f802c1a5dd0 con 0x7f802c102760 2024-03-28T17:46:17.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.176+0000 7f802a000640 1 -- 172.21.15.45:0/2175064051 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80180049e0 con 0x7f802c102760 2024-03-28T17:46:17.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.177+0000 7f8032a00640 1 -- 172.21.15.45:0/2175064051 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f802c1a6290 con 0x7f802c102760 2024-03-28T17:46:17.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.178+0000 7f8032a00640 1 -- 172.21.15.45:0/2175064051 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f802c0fd500 con 0x7f802c102760 2024-03-28T17:46:17.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.180+0000 7f802a000640 1 -- 172.21.15.45:0/2175064051 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8018005560 con 0x7f802c102760 2024-03-28T17:46:17.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.182+0000 7f802a000640 1 --2- 172.21.15.45:0/2175064051 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80140a2880 0x7f80140a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:17.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.182+0000 7f802a000640 1 -- 172.21.15.45:0/2175064051 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f80180ce900 con 0x7f802c102760 2024-03-28T17:46:17.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.182+0000 7f8031600640 1 --2- 172.21.15.45:0/2175064051 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80140a2880 0x7f80140a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:17.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.183+0000 7f8031600640 1 --2- 172.21.15.45:0/2175064051 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80140a2880 0x7f80140a4d40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8020006010 tx=0x7f8020004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:17.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.183+0000 7f802a000640 1 -- 172.21.15.45:0/2175064051 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f80180051d0 con 0x7f802c102760 2024-03-28T17:46:17.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.433+0000 7f8032a00640 1 -- 172.21.15.45:0/2175064051 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f802c02d070 con 0x7f802c102760 2024-03-28T17:46:17.560 INFO:journalctl@ceph.mds.b.smithi071.stdout:Mar 28 17:46:17 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-b[66900]: 2024-03-28T17:46:17.091+0000 7ff9f838b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T17:46:17.560 INFO:journalctl@ceph.mds.b.smithi071.stdout:Mar 28 17:46:17 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-b[66900]: 2024-03-28T17:46:17.091+0000 7ff9f838b700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T17:46:17.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:17 smithi071 ceph-mon[36597]: pgmap v319: 97 pgs: 5 creating+peering, 76 active+clean, 16 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 3 op/s 2024-03-28T17:46:17.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:17 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/2740394872' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T17:46:17.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:17 smithi071 ceph-mon[36597]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T17:46:17.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:17 smithi071 ceph-mon[36597]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T17:46:17.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:17 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] up:active 2024-03-28T17:46:17.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:17 smithi071 ceph-mon[36597]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T17:46:17.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:17 smithi071 ceph-mon[36597]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T17:46:17.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:17 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:46:17.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:17 smithi071 ceph-mon[36597]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-03-28T17:46:17.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:17 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-03-28T17:46:17.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:17 smithi071 ceph-mon[36597]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-28T17:46:17.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:17 smithi151 ceph-mon[36481]: pgmap v319: 97 pgs: 5 creating+peering, 76 active+clean, 16 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 3 op/s 2024-03-28T17:46:17.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:17 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/2740394872' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T17:46:17.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:17 smithi151 ceph-mon[36481]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T17:46:17.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:17 smithi151 ceph-mon[36481]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T17:46:17.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:17 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] up:active 2024-03-28T17:46:17.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:17 smithi151 ceph-mon[36481]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T17:46:17.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:17 smithi151 ceph-mon[36481]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T17:46:17.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:17 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:46:17.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:17 smithi151 ceph-mon[36481]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-03-28T17:46:17.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:17 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-03-28T17:46:17.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:17 smithi151 ceph-mon[36481]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-28T17:46:17.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:17 smithi045 ceph-mon[31773]: pgmap v319: 97 pgs: 5 creating+peering, 76 active+clean, 16 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 3 op/s 2024-03-28T17:46:17.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:17 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/2740394872' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T17:46:17.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:17 smithi045 ceph-mon[31773]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T17:46:17.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:17 smithi045 ceph-mon[31773]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T17:46:17.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:17 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] up:active 2024-03-28T17:46:17.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:17 smithi045 ceph-mon[31773]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T17:46:17.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:17 smithi045 ceph-mon[31773]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T17:46:17.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:17 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:46:17.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:17 smithi045 ceph-mon[31773]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-03-28T17:46:17.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:17 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-03-28T17:46:17.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:17 smithi045 ceph-mon[31773]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-28T17:46:17.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.963+0000 7f802a000640 1 -- 172.21.15.45:0/2175064051 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v21) v1 ==== 115+0+0 (secure 0 0 0) 0x7f801809ba50 con 0x7f802c102760 2024-03-28T17:46:17.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.968+0000 7f8032a00640 1 -- 172.21.15.45:0/2175064051 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80140a2880 msgr2=0x7f80140a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:17.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.968+0000 7f8032a00640 1 --2- 172.21.15.45:0/2175064051 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80140a2880 0x7f80140a4d40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8020006010 tx=0x7f8020004a40 comp rx=0 tx=0).stop 2024-03-28T17:46:17.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.968+0000 7f8032a00640 1 -- 172.21.15.45:0/2175064051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f802c102760 msgr2=0x7f802c19e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:17.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.968+0000 7f8032a00640 1 --2- 172.21.15.45:0/2175064051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f802c102760 0x7f802c19e450 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f801800e9f0 tx=0x7f801800eeb0 comp rx=0 tx=0).stop 2024-03-28T17:46:17.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.968+0000 7f8032a00640 1 -- 172.21.15.45:0/2175064051 shutdown_connections 2024-03-28T17:46:17.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.968+0000 7f8032a00640 1 --2- 172.21.15.45:0/2175064051 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80140a2880 0x7f80140a4d40 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:17.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.968+0000 7f8032a00640 1 --2- 172.21.15.45:0/2175064051 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f802c105110 0x7f802c1a5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:17.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.968+0000 7f8032a00640 1 --2- 172.21.15.45:0/2175064051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f802c102760 0x7f802c19e450 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:17.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.968+0000 7f8032a00640 1 --2- 172.21.15.45:0/2175064051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f802c0ffdf0 0x7f802c19df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:17.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.968+0000 7f8032a00640 1 -- 172.21.15.45:0/2175064051 >> 172.21.15.45:0/2175064051 conn(0x7f802c0fba10 msgr2=0x7f802c1030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:17.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.968+0000 7f8032a00640 1 -- 172.21.15.45:0/2175064051 shutdown_connections 2024-03-28T17:46:17.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:17.968+0000 7f8032a00640 1 -- 172.21.15.45:0/2175064051 wait complete. 2024-03-28T17:46:17.994 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-03-28T17:46:17.994 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-28T17:46:18.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.226+0000 7ff219200640 1 -- 172.21.15.45:0/245957617 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff214104ab0 msgr2=0x7ff214106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:18.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.226+0000 7ff219200640 1 --2- 172.21.15.45:0/245957617 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff214104ab0 0x7ff214106ea0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc009900 tx=0x7ff1fc02bc40 comp rx=0 tx=0).stop 2024-03-28T17:46:18.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.226+0000 7ff219200640 1 -- 172.21.15.45:0/245957617 shutdown_connections 2024-03-28T17:46:18.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.226+0000 7ff219200640 1 --2- 172.21.15.45:0/245957617 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff2141073e0 0x7ff2141097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:18.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.226+0000 7ff219200640 1 --2- 172.21.15.45:0/245957617 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff214104ab0 0x7ff214106ea0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:18.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.226+0000 7ff219200640 1 --2- 172.21.15.45:0/245957617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff214102180 0x7ff214104570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:18.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.226+0000 7ff219200640 1 -- 172.21.15.45:0/245957617 >> 172.21.15.45:0/245957617 conn(0x7ff2140fbb10 msgr2=0x7ff2140fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:18.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.226+0000 7ff219200640 1 -- 172.21.15.45:0/245957617 shutdown_connections 2024-03-28T17:46:18.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.226+0000 7ff219200640 1 -- 172.21.15.45:0/245957617 wait complete. 2024-03-28T17:46:18.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.227+0000 7ff219200640 1 Processor -- start 2024-03-28T17:46:18.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.228+0000 7ff219200640 1 -- start start 2024-03-28T17:46:18.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.228+0000 7ff219200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff214102180 0x7ff214101af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:18.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.228+0000 7ff219200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff214104ab0 0x7ff214100130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:18.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.228+0000 7ff219200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2141073e0 0x7ff214100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:18.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.228+0000 7ff219200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21410d370 con 0x7ff2141073e0 2024-03-28T17:46:18.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.228+0000 7ff219200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21410d1f0 con 0x7ff214104ab0 2024-03-28T17:46:18.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.228+0000 7ff219200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21410d4f0 con 0x7ff214102180 2024-03-28T17:46:18.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.228+0000 7ff211e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff214102180 0x7ff214101af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:18.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.228+0000 7ff211e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff214102180 0x7ff214101af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:49908/0 (socket says 172.21.15.45:49908) 2024-03-28T17:46:18.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.228+0000 7ff211e00640 1 -- 172.21.15.45:0/1324435624 learned_addr learned my addr 172.21.15.45:0/1324435624 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:18.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.228+0000 7ff20be00640 1 --2- 172.21.15.45:0/1324435624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff214104ab0 0x7ff214100130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:18.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.229+0000 7ff212800640 1 --2- 172.21.15.45:0/1324435624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2141073e0 0x7ff214100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:18.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.229+0000 7ff211e00640 1 -- 172.21.15.45:0/1324435624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff214104ab0 msgr2=0x7ff214100130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:18.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.229+0000 7ff211e00640 1 --2- 172.21.15.45:0/1324435624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff214104ab0 0x7ff214100130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:18.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.229+0000 7ff211e00640 1 -- 172.21.15.45:0/1324435624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2141073e0 msgr2=0x7ff214100670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:18.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.229+0000 7ff211e00640 1 --2- 172.21.15.45:0/1324435624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2141073e0 0x7ff214100670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:18.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.229+0000 7ff211e00640 1 -- 172.21.15.45:0/1324435624 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff214100f00 con 0x7ff214102180 2024-03-28T17:46:18.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.229+0000 7ff20be00640 1 --2- 172.21.15.45:0/1324435624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff214104ab0 0x7ff214100130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:46:18.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.229+0000 7ff211e00640 1 --2- 172.21.15.45:0/1324435624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff214102180 0x7ff214101af0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff20400e9f0 tx=0x7ff20400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:18.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.229+0000 7ff20aa00640 1 -- 172.21.15.45:0/1324435624 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20400cd80 con 0x7ff214102180 2024-03-28T17:46:18.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.229+0000 7ff20aa00640 1 -- 172.21.15.45:0/1324435624 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff204010910 con 0x7ff214102180 2024-03-28T17:46:18.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.229+0000 7ff20aa00640 1 -- 172.21.15.45:0/1324435624 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff204018990 con 0x7ff214102180 2024-03-28T17:46:18.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.230+0000 7ff219200640 1 -- 172.21.15.45:0/1324435624 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2141a1b00 con 0x7ff214102180 2024-03-28T17:46:18.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.230+0000 7ff219200640 1 -- 172.21.15.45:0/1324435624 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2141a1fd0 con 0x7ff214102180 2024-03-28T17:46:18.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.231+0000 7ff219200640 1 -- 172.21.15.45:0/1324435624 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2140fd600 con 0x7ff214102180 2024-03-28T17:46:18.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.233+0000 7ff20aa00640 1 -- 172.21.15.45:0/1324435624 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff204010ac0 con 0x7ff214102180 2024-03-28T17:46:18.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.234+0000 7ff20aa00640 1 --2- 172.21.15.45:0/1324435624 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff1f80a2880 0x7ff1f80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:18.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.234+0000 7ff20aa00640 1 -- 172.21.15.45:0/1324435624 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff2040cee00 con 0x7ff214102180 2024-03-28T17:46:18.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.234+0000 7ff20be00640 1 --2- 172.21.15.45:0/1324435624 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff1f80a2880 0x7ff1f80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:18.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.235+0000 7ff20be00640 1 --2- 172.21.15.45:0/1324435624 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff1f80a2880 0x7ff1f80a4d40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc02c180 tx=0x7ff1fc02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:18.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.236+0000 7ff20aa00640 1 -- 172.21.15.45:0/1324435624 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff20409b900 con 0x7ff214102180 2024-03-28T17:46:18.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.468+0000 7ff219200640 1 -- 172.21.15.45:0/1324435624 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7ff214061ae0 con 0x7ff1f80a2880 2024-03-28T17:46:18.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.499+0000 7ff20aa00640 1 -- 172.21.15.45:0/1324435624 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff214061ae0 con 0x7ff1f80a2880 2024-03-28T17:46:18.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.501+0000 7ff219200640 1 -- 172.21.15.45:0/1324435624 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff1f80a2880 msgr2=0x7ff1f80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:18.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.501+0000 7ff219200640 1 --2- 172.21.15.45:0/1324435624 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff1f80a2880 0x7ff1f80a4d40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc02c180 tx=0x7ff1fc02cf90 comp rx=0 tx=0).stop 2024-03-28T17:46:18.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.501+0000 7ff219200640 1 -- 172.21.15.45:0/1324435624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff214102180 msgr2=0x7ff214101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:18.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.501+0000 7ff219200640 1 --2- 172.21.15.45:0/1324435624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff214102180 0x7ff214101af0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff20400e9f0 tx=0x7ff20400eeb0 comp rx=0 tx=0).stop 2024-03-28T17:46:18.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.501+0000 7ff219200640 1 -- 172.21.15.45:0/1324435624 shutdown_connections 2024-03-28T17:46:18.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.501+0000 7ff219200640 1 --2- 172.21.15.45:0/1324435624 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff1f80a2880 0x7ff1f80a4d40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:18.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.501+0000 7ff219200640 1 --2- 172.21.15.45:0/1324435624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2141073e0 0x7ff214100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:18.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.501+0000 7ff219200640 1 --2- 172.21.15.45:0/1324435624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff214104ab0 0x7ff214100130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:18.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.501+0000 7ff219200640 1 --2- 172.21.15.45:0/1324435624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff214102180 0x7ff214101af0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:18.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.501+0000 7ff219200640 1 -- 172.21.15.45:0/1324435624 >> 172.21.15.45:0/1324435624 conn(0x7ff2140fbb10 msgr2=0x7ff214102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:18.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.502+0000 7ff219200640 1 -- 172.21.15.45:0/1324435624 shutdown_connections 2024-03-28T17:46:18.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.502+0000 7ff219200640 1 -- 172.21.15.45:0/1324435624 wait complete. 2024-03-28T17:46:18.521 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-28T17:46:18.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:18 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/2175064051' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T17:46:18.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:18 smithi071 ceph-mon[36597]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T17:46:18.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:18 smithi071 ceph-mon[36597]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T17:46:18.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:18 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] up:active 2024-03-28T17:46:18.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:18 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-03-28T17:46:18.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:18 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:46:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:18 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/2175064051' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T17:46:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:18 smithi151 ceph-mon[36481]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T17:46:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:18 smithi151 ceph-mon[36481]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T17:46:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:18 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] up:active 2024-03-28T17:46:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:18 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-03-28T17:46:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:18 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:46:18.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:18 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/2175064051' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T17:46:18.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:18 smithi045 ceph-mon[31773]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T17:46:18.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:18 smithi045 ceph-mon[31773]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T17:46:18.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:18 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] up:active 2024-03-28T17:46:18.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:18 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-03-28T17:46:18.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:18 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:46:18.600 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:46:18 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:46:18.477+0000 7ffa32c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:46:18.600 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:46:18 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:46:18.477+0000 7ffa32c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:46:18.600 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:46:18 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:46:18.477+0000 7ffa32c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:46:18.600 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:46:18 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:46:18.477+0000 7ffa32c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:46:18.601 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:46:18 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:46:18.477+0000 7ffa32c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:46:18.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.744+0000 7f378fc00640 1 -- 172.21.15.45:0/847133571 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37881038c0 msgr2=0x7f3788103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:18.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.744+0000 7f378fc00640 1 --2- 172.21.15.45:0/847133571 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37881038c0 0x7f3788103ce0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3778009ca0 tx=0x7f377802c0f0 comp rx=0 tx=0).stop 2024-03-28T17:46:18.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.744+0000 7f378fc00640 1 -- 172.21.15.45:0/847133571 shutdown_connections 2024-03-28T17:46:18.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.744+0000 7f378fc00640 1 --2- 172.21.15.45:0/847133571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3788105470 0x7f378810bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:18.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.744+0000 7f378fc00640 1 --2- 172.21.15.45:0/847133571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3788104ab0 0x7f3788104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:18.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.744+0000 7f378fc00640 1 --2- 172.21.15.45:0/847133571 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37881038c0 0x7f3788103ce0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:18.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.744+0000 7f378fc00640 1 -- 172.21.15.45:0/847133571 >> 172.21.15.45:0/847133571 conn(0x7f37880fee50 msgr2=0x7f3788101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:18.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.744+0000 7f378fc00640 1 -- 172.21.15.45:0/847133571 shutdown_connections 2024-03-28T17:46:18.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.745+0000 7f378fc00640 1 -- 172.21.15.45:0/847133571 wait complete. 2024-03-28T17:46:18.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.745+0000 7f378fc00640 1 Processor -- start 2024-03-28T17:46:18.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.746+0000 7f378fc00640 1 -- start start 2024-03-28T17:46:18.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.746+0000 7f378fc00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37881038c0 0x7f3788199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:18.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.746+0000 7f378fc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3788104ab0 0x7f378819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:18.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.746+0000 7f378fc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3788105470 0x7f37881a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:18.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.746+0000 7f378fc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378810f530 con 0x7f3788105470 2024-03-28T17:46:18.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.746+0000 7f378fc00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378810f3b0 con 0x7f3788104ab0 2024-03-28T17:46:18.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.746+0000 7f378fc00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378810f6b0 con 0x7f37881038c0 2024-03-28T17:46:18.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.746+0000 7f378d200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37881038c0 0x7f3788199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:18.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.746+0000 7f377fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3788104ab0 0x7f378819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:18.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.746+0000 7f377fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3788104ab0 0x7f378819a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:57430/0 (socket says 172.21.15.45:57430) 2024-03-28T17:46:18.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.746+0000 7f378d200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37881038c0 0x7f3788199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:49942/0 (socket says 172.21.15.45:49942) 2024-03-28T17:46:18.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.746+0000 7f378dc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3788105470 0x7f37881a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:18.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.746+0000 7f377fe00640 1 -- 172.21.15.45:0/2891529640 learned_addr learned my addr 172.21.15.45:0/2891529640 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:18.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.747+0000 7f377fe00640 1 -- 172.21.15.45:0/2891529640 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37881038c0 msgr2=0x7f3788199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:18.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.747+0000 7f377fe00640 1 --2- 172.21.15.45:0/2891529640 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37881038c0 0x7f3788199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:18.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.747+0000 7f377fe00640 1 -- 172.21.15.45:0/2891529640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3788105470 msgr2=0x7f37881a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:18.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.747+0000 7f377fe00640 1 --2- 172.21.15.45:0/2891529640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3788105470 0x7f37881a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:18.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.747+0000 7f377fe00640 1 -- 172.21.15.45:0/2891529640 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37881a1870 con 0x7f3788104ab0 2024-03-28T17:46:18.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.747+0000 7f378d200640 1 --2- 172.21.15.45:0/2891529640 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37881038c0 0x7f3788199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:46:18.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.747+0000 7f377fe00640 1 --2- 172.21.15.45:0/2891529640 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3788104ab0 0x7f378819a1f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f377000e9f0 tx=0x7f377000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:18.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.747+0000 7f3786000640 1 -- 172.21.15.45:0/2891529640 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377000cd80 con 0x7f3788104ab0 2024-03-28T17:46:18.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.747+0000 7f3786000640 1 -- 172.21.15.45:0/2891529640 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3770005500 con 0x7f3788104ab0 2024-03-28T17:46:18.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.747+0000 7f378fc00640 1 -- 172.21.15.45:0/2891529640 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37881a1b60 con 0x7f3788104ab0 2024-03-28T17:46:18.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.747+0000 7f3786000640 1 -- 172.21.15.45:0/2891529640 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3770010690 con 0x7f3788104ab0 2024-03-28T17:46:18.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.747+0000 7f378fc00640 1 -- 172.21.15.45:0/2891529640 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37881a2080 con 0x7f3788104ab0 2024-03-28T17:46:18.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.749+0000 7f378fc00640 1 -- 172.21.15.45:0/2891529640 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f378810ad00 con 0x7f3788104ab0 2024-03-28T17:46:18.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.750+0000 7f3786000640 1 -- 172.21.15.45:0/2891529640 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f37700046c0 con 0x7f3788104ab0 2024-03-28T17:46:18.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.751+0000 7f3786000640 1 --2- 172.21.15.45:0/2891529640 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f376c0a2880 0x7f376c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:18.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.752+0000 7f3786000640 1 -- 172.21.15.45:0/2891529640 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f37700cedc0 con 0x7f3788104ab0 2024-03-28T17:46:18.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.752+0000 7f378d200640 1 --2- 172.21.15.45:0/2891529640 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f376c0a2880 0x7f376c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:18.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.753+0000 7f378d200640 1 --2- 172.21.15.45:0/2891529640 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f376c0a2880 0x7f376c0a4d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f377802c630 tx=0x7f3778009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:18.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.756+0000 7f3786000640 1 -- 172.21.15.45:0/2891529640 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f377009bf10 con 0x7f3788104ab0 2024-03-28T17:46:18.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:18.987+0000 7f378fc00640 1 -- 172.21.15.45:0/2891529640 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f3788061ae0 con 0x7f376c0a2880 2024-03-28T17:46:19.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.021+0000 7f3786000640 1 -- 172.21.15.45:0/2891529640 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3788061ae0 con 0x7f376c0a2880 2024-03-28T17:46:19.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.024+0000 7f378fc00640 1 -- 172.21.15.45:0/2891529640 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f376c0a2880 msgr2=0x7f376c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:19.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.024+0000 7f378fc00640 1 --2- 172.21.15.45:0/2891529640 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f376c0a2880 0x7f376c0a4d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f377802c630 tx=0x7f3778009360 comp rx=0 tx=0).stop 2024-03-28T17:46:19.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.024+0000 7f378fc00640 1 -- 172.21.15.45:0/2891529640 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3788104ab0 msgr2=0x7f378819a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:19.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.024+0000 7f378fc00640 1 --2- 172.21.15.45:0/2891529640 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3788104ab0 0x7f378819a1f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f377000e9f0 tx=0x7f377000edf0 comp rx=0 tx=0).stop 2024-03-28T17:46:19.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.024+0000 7f378fc00640 1 -- 172.21.15.45:0/2891529640 shutdown_connections 2024-03-28T17:46:19.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.024+0000 7f378fc00640 1 --2- 172.21.15.45:0/2891529640 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f376c0a2880 0x7f376c0a4d40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:19.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.024+0000 7f378fc00640 1 --2- 172.21.15.45:0/2891529640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3788105470 0x7f37881a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:19.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.024+0000 7f378fc00640 1 --2- 172.21.15.45:0/2891529640 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3788104ab0 0x7f378819a1f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:19.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.024+0000 7f378fc00640 1 --2- 172.21.15.45:0/2891529640 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37881038c0 0x7f3788199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:19.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.024+0000 7f378fc00640 1 -- 172.21.15.45:0/2891529640 >> 172.21.15.45:0/2891529640 conn(0x7f37880fee50 msgr2=0x7f3788107ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:19.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.024+0000 7f378fc00640 1 -- 172.21.15.45:0/2891529640 shutdown_connections 2024-03-28T17:46:19.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.024+0000 7f378fc00640 1 -- 172.21.15.45:0/2891529640 wait complete. 2024-03-28T17:46:19.043 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-28T17:46:19.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.276+0000 7f6658000640 1 -- 172.21.15.45:0/4036163469 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66501028a0 msgr2=0x7f6650102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:19.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.276+0000 7f6658000640 1 --2- 172.21.15.45:0/4036163469 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66501028a0 0x7f6650102d20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f663c009900 tx=0x7f663c02bc30 comp rx=0 tx=0).stop 2024-03-28T17:46:19.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.276+0000 7f6658000640 1 -- 172.21.15.45:0/4036163469 shutdown_connections 2024-03-28T17:46:19.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.276+0000 7f6658000640 1 --2- 172.21.15.45:0/4036163469 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6650103260 0x7f6650109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:19.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.276+0000 7f6658000640 1 --2- 172.21.15.45:0/4036163469 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66501028a0 0x7f6650102d20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:19.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.276+0000 7f6658000640 1 --2- 172.21.15.45:0/4036163469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66501016b0 0x7f6650101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:19.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.276+0000 7f6658000640 1 -- 172.21.15.45:0/4036163469 >> 172.21.15.45:0/4036163469 conn(0x7f66500fcc40 msgr2=0x7f66500ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:19.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.276+0000 7f6658000640 1 -- 172.21.15.45:0/4036163469 shutdown_connections 2024-03-28T17:46:19.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.276+0000 7f6658000640 1 -- 172.21.15.45:0/4036163469 wait complete. 2024-03-28T17:46:19.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.277+0000 7f6658000640 1 Processor -- start 2024-03-28T17:46:19.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.277+0000 7f6658000640 1 -- start start 2024-03-28T17:46:19.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.277+0000 7f6658000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66501016b0 0x7f6650199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:19.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.277+0000 7f6658000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66501028a0 0x7f665019a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:19.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.277+0000 7f6658000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6650103260 0x7f66501a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:19.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.277+0000 7f6658000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665010d360 con 0x7f66501028a0 2024-03-28T17:46:19.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.277+0000 7f6658000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665010d1e0 con 0x7f66501016b0 2024-03-28T17:46:19.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.277+0000 7f6658000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665010d4e0 con 0x7f6650103260 2024-03-28T17:46:19.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.278+0000 7f6655600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66501016b0 0x7f6650199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:19.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.278+0000 7f6654c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66501028a0 0x7f665019a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:19.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.278+0000 7f6655600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66501016b0 0x7f6650199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:57442/0 (socket says 172.21.15.45:57442) 2024-03-28T17:46:19.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.278+0000 7f6655600640 1 -- 172.21.15.45:0/769452280 learned_addr learned my addr 172.21.15.45:0/769452280 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:19.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.278+0000 7f6656000640 1 --2- 172.21.15.45:0/769452280 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6650103260 0x7f66501a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:19.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.278+0000 7f6654c00640 1 -- 172.21.15.45:0/769452280 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6650103260 msgr2=0x7f66501a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:19.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.278+0000 7f6654c00640 1 --2- 172.21.15.45:0/769452280 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6650103260 0x7f66501a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:19.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.278+0000 7f6654c00640 1 -- 172.21.15.45:0/769452280 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66501016b0 msgr2=0x7f6650199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:19.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.278+0000 7f6654c00640 1 --2- 172.21.15.45:0/769452280 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66501016b0 0x7f6650199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:19.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.278+0000 7f6654c00640 1 -- 172.21.15.45:0/769452280 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66501a1830 con 0x7f66501028a0 2024-03-28T17:46:19.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.278+0000 7f6654c00640 1 --2- 172.21.15.45:0/769452280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66501028a0 0x7f665019a160 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f663c036040 tx=0x7f663c02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:19.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.279+0000 7f664e000640 1 -- 172.21.15.45:0/769452280 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c043020 con 0x7f66501028a0 2024-03-28T17:46:19.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.279+0000 7f6658000640 1 -- 172.21.15.45:0/769452280 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66501a1ac0 con 0x7f66501028a0 2024-03-28T17:46:19.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.279+0000 7f664e000640 1 -- 172.21.15.45:0/769452280 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f663c0054e0 con 0x7f66501028a0 2024-03-28T17:46:19.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.279+0000 7f664e000640 1 -- 172.21.15.45:0/769452280 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c003c30 con 0x7f66501028a0 2024-03-28T17:46:19.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.279+0000 7f6658000640 1 -- 172.21.15.45:0/769452280 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66501a2030 con 0x7f66501028a0 2024-03-28T17:46:19.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.280+0000 7f6658000640 1 -- 172.21.15.45:0/769452280 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66501a2260 con 0x7f66501028a0 2024-03-28T17:46:19.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.282+0000 7f664e000640 1 -- 172.21.15.45:0/769452280 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f663c046020 con 0x7f66501028a0 2024-03-28T17:46:19.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.283+0000 7f664e000640 1 --2- 172.21.15.45:0/769452280 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f66380a2800 0x7f66380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:19.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.283+0000 7f664e000640 1 -- 172.21.15.45:0/769452280 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f663c0f0980 con 0x7f66501028a0 2024-03-28T17:46:19.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.283+0000 7f6655600640 1 --2- 172.21.15.45:0/769452280 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f66380a2800 0x7f66380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:19.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.284+0000 7f6655600640 1 --2- 172.21.15.45:0/769452280 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f66380a2800 0x7f66380a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f665019b0f0 tx=0x7f6644005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:19.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.287+0000 7f664e000640 1 -- 172.21.15.45:0/769452280 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f663c04d020 con 0x7f66501028a0 2024-03-28T17:46:19.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:19 smithi045 ceph-mon[31773]: pgmap v320: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2024-03-28T17:46:19.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:19 smithi045 ceph-mon[31773]: from='client.24649 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:46:19.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.526+0000 7f6658000640 1 -- 172.21.15.45:0/769452280 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f6650074ff0 con 0x7f66380a2800 2024-03-28T17:46:19.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.538+0000 7f664e000640 1 -- 172.21.15.45:0/769452280 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f6650074ff0 con 0x7f66380a2800 2024-03-28T17:46:19.538 INFO:teuthology.orchestra.run.smithi045.stdout:/volumes/qa/sv_0/35307f97-70a9-4f80-80a8-f1595be79aa2 2024-03-28T17:46:19.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.540+0000 7f6658000640 1 -- 172.21.15.45:0/769452280 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f66380a2800 msgr2=0x7f66380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:19.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.540+0000 7f6658000640 1 --2- 172.21.15.45:0/769452280 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f66380a2800 0x7f66380a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f665019b0f0 tx=0x7f6644005f90 comp rx=0 tx=0).stop 2024-03-28T17:46:19.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.540+0000 7f6658000640 1 -- 172.21.15.45:0/769452280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66501028a0 msgr2=0x7f665019a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:19.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.540+0000 7f6658000640 1 --2- 172.21.15.45:0/769452280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66501028a0 0x7f665019a160 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f663c036040 tx=0x7f663c02c650 comp rx=0 tx=0).stop 2024-03-28T17:46:19.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.541+0000 7f6658000640 1 -- 172.21.15.45:0/769452280 shutdown_connections 2024-03-28T17:46:19.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.541+0000 7f6658000640 1 --2- 172.21.15.45:0/769452280 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f66380a2800 0x7f66380a4cc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:19.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.541+0000 7f6658000640 1 --2- 172.21.15.45:0/769452280 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6650103260 0x7f66501a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:19.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.541+0000 7f6658000640 1 --2- 172.21.15.45:0/769452280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66501028a0 0x7f665019a160 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:19.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.541+0000 7f6658000640 1 --2- 172.21.15.45:0/769452280 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66501016b0 0x7f6650199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:19.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.541+0000 7f6658000640 1 -- 172.21.15.45:0/769452280 >> 172.21.15.45:0/769452280 conn(0x7f66500fcc40 msgr2=0x7f66500738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:19.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.541+0000 7f6658000640 1 -- 172.21.15.45:0/769452280 shutdown_connections 2024-03-28T17:46:19.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.541+0000 7f6658000640 1 -- 172.21.15.45:0/769452280 wait complete. 2024-03-28T17:46:19.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:19 smithi071 ceph-mon[36597]: pgmap v320: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2024-03-28T17:46:19.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:19 smithi071 ceph-mon[36597]: from='client.24649 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:46:19.561 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-28T17:46:19.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:19 smithi151 ceph-mon[36481]: pgmap v320: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2024-03-28T17:46:19.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:19 smithi151 ceph-mon[36481]: from='client.24649 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:46:19.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.781+0000 7fafab000640 1 -- 172.21.15.45:0/352206297 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafa4104990 msgr2=0x7fafa4104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:19.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.781+0000 7fafab000640 1 --2- 172.21.15.45:0/352206297 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafa4104990 0x7fafa4104e10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7faf90009900 tx=0x7faf9002bc30 comp rx=0 tx=0).stop 2024-03-28T17:46:19.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.781+0000 7fafab000640 1 -- 172.21.15.45:0/352206297 shutdown_connections 2024-03-28T17:46:19.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.781+0000 7fafab000640 1 --2- 172.21.15.45:0/352206297 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fafa4105350 0x7fafa410bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:19.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.781+0000 7fafab000640 1 --2- 172.21.15.45:0/352206297 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafa4104990 0x7fafa4104e10 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:19.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.781+0000 7fafab000640 1 --2- 172.21.15.45:0/352206297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafa41037a0 0x7fafa4103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:19.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.781+0000 7fafab000640 1 -- 172.21.15.45:0/352206297 >> 172.21.15.45:0/352206297 conn(0x7fafa40fed10 msgr2=0x7fafa4101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:19.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.781+0000 7fafab000640 1 -- 172.21.15.45:0/352206297 shutdown_connections 2024-03-28T17:46:19.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.781+0000 7fafab000640 1 -- 172.21.15.45:0/352206297 wait complete. 2024-03-28T17:46:19.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.783+0000 7fafab000640 1 Processor -- start 2024-03-28T17:46:19.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.783+0000 7fafab000640 1 -- start start 2024-03-28T17:46:19.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.783+0000 7fafab000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fafa41037a0 0x7fafa4199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:19.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.783+0000 7fafab000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafa4104990 0x7fafa4199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:19.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.783+0000 7fafab000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafa4105350 0x7fafa41a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:19.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.783+0000 7fafab000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa410f450 con 0x7fafa4105350 2024-03-28T17:46:19.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.783+0000 7fafab000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa410f2d0 con 0x7fafa4104990 2024-03-28T17:46:19.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.783+0000 7fafab000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa410f5d0 con 0x7fafa41037a0 2024-03-28T17:46:19.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.783+0000 7fafa9200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafa4104990 0x7fafa4199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:19.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.783+0000 7fafa9c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fafa41037a0 0x7fafa4199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:19.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.783+0000 7fafaa600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafa4105350 0x7fafa41a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:19.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.783+0000 7fafa9c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fafa41037a0 0x7fafa4199a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:49992/0 (socket says 172.21.15.45:49992) 2024-03-28T17:46:19.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.783+0000 7fafa9200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafa4104990 0x7fafa4199f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:57462/0 (socket says 172.21.15.45:57462) 2024-03-28T17:46:19.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.783+0000 7fafa9c00640 1 -- 172.21.15.45:0/4177564490 learned_addr learned my addr 172.21.15.45:0/4177564490 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:19.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.784+0000 7fafa9200640 1 -- 172.21.15.45:0/4177564490 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fafa41037a0 msgr2=0x7fafa4199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:19.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.784+0000 7fafa9200640 1 --2- 172.21.15.45:0/4177564490 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fafa41037a0 0x7fafa4199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:19.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.784+0000 7fafa9200640 1 -- 172.21.15.45:0/4177564490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafa4105350 msgr2=0x7fafa41a0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:19.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.784+0000 7fafa9200640 1 --2- 172.21.15.45:0/4177564490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafa4105350 0x7fafa41a0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:19.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.784+0000 7fafa9200640 1 -- 172.21.15.45:0/4177564490 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa41a1650 con 0x7fafa4104990 2024-03-28T17:46:19.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.784+0000 7fafa9200640 1 --2- 172.21.15.45:0/4177564490 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafa4104990 0x7fafa4199f70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7faf90036040 tx=0x7faf90005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:19.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.784+0000 7fafa2000640 1 -- 172.21.15.45:0/4177564490 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf90043020 con 0x7fafa4104990 2024-03-28T17:46:19.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.784+0000 7fafa2000640 1 -- 172.21.15.45:0/4177564490 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faf90004410 con 0x7fafa4104990 2024-03-28T17:46:19.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.784+0000 7fafa2000640 1 -- 172.21.15.45:0/4177564490 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf90003df0 con 0x7fafa4104990 2024-03-28T17:46:19.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.784+0000 7fafab000640 1 -- 172.21.15.45:0/4177564490 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa41a18e0 con 0x7fafa4104990 2024-03-28T17:46:19.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.785+0000 7fafab000640 1 -- 172.21.15.45:0/4177564490 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa41a1e00 con 0x7fafa4104990 2024-03-28T17:46:19.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.786+0000 7fafab000640 1 -- 172.21.15.45:0/4177564490 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa410aac0 con 0x7fafa4104990 2024-03-28T17:46:19.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.788+0000 7fafa2000640 1 -- 172.21.15.45:0/4177564490 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7faf90046020 con 0x7fafa4104990 2024-03-28T17:46:19.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.789+0000 7fafa2000640 1 --2- 172.21.15.45:0/4177564490 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7faf8c0a2880 0x7faf8c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:19.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.789+0000 7fafa2000640 1 -- 172.21.15.45:0/4177564490 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7faf900f0a10 con 0x7fafa4104990 2024-03-28T17:46:19.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.789+0000 7fafa9c00640 1 --2- 172.21.15.45:0/4177564490 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7faf8c0a2880 0x7faf8c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:19.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.790+0000 7fafa9c00640 1 --2- 172.21.15.45:0/4177564490 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7faf8c0a2880 0x7faf8c0a4d40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faf9800bc80 tx=0x7faf9800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:19.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:19.793+0000 7fafa2000640 1 -- 172.21.15.45:0/4177564490 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faf9004d020 con 0x7fafa4104990 2024-03-28T17:46:20.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.026+0000 7fafab000640 1 -- 172.21.15.45:0/4177564490 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fafa4061ae0 con 0x7faf8c0a2880 2024-03-28T17:46:20.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.062+0000 7fafa2000640 1 -- 172.21.15.45:0/4177564490 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fafa4061ae0 con 0x7faf8c0a2880 2024-03-28T17:46:20.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.066+0000 7fafab000640 1 -- 172.21.15.45:0/4177564490 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7faf8c0a2880 msgr2=0x7faf8c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:20.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.066+0000 7fafab000640 1 --2- 172.21.15.45:0/4177564490 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7faf8c0a2880 0x7faf8c0a4d40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faf9800bc80 tx=0x7faf9800b410 comp rx=0 tx=0).stop 2024-03-28T17:46:20.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.066+0000 7fafab000640 1 -- 172.21.15.45:0/4177564490 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafa4104990 msgr2=0x7fafa4199f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:20.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.066+0000 7fafab000640 1 --2- 172.21.15.45:0/4177564490 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafa4104990 0x7fafa4199f70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7faf90036040 tx=0x7faf90005dd0 comp rx=0 tx=0).stop 2024-03-28T17:46:20.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.066+0000 7fafab000640 1 -- 172.21.15.45:0/4177564490 shutdown_connections 2024-03-28T17:46:20.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.066+0000 7fafab000640 1 --2- 172.21.15.45:0/4177564490 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7faf8c0a2880 0x7faf8c0a4d40 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:20.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.066+0000 7fafab000640 1 --2- 172.21.15.45:0/4177564490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafa4105350 0x7fafa41a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:20.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.066+0000 7fafab000640 1 --2- 172.21.15.45:0/4177564490 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafa4104990 0x7fafa4199f70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:20.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.066+0000 7fafab000640 1 --2- 172.21.15.45:0/4177564490 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fafa41037a0 0x7fafa4199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:20.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.066+0000 7fafab000640 1 -- 172.21.15.45:0/4177564490 >> 172.21.15.45:0/4177564490 conn(0x7fafa40fed10 msgr2=0x7fafa4107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:20.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.066+0000 7fafab000640 1 -- 172.21.15.45:0/4177564490 shutdown_connections 2024-03-28T17:46:20.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.066+0000 7fafab000640 1 -- 172.21.15.45:0/4177564490 wait complete. 2024-03-28T17:46:20.085 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-28T17:46:20.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.318+0000 7f84aba00640 1 -- 172.21.15.45:0/3428266883 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84a4105490 msgr2=0x7f84a410bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:20.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.318+0000 7f84aba00640 1 --2- 172.21.15.45:0/3428266883 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84a4105490 0x7f84a410bd20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f849c009ca0 tx=0x7f849c02c0f0 comp rx=0 tx=0).stop 2024-03-28T17:46:20.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.319+0000 7f84aba00640 1 -- 172.21.15.45:0/3428266883 shutdown_connections 2024-03-28T17:46:20.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.319+0000 7f84aba00640 1 --2- 172.21.15.45:0/3428266883 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84a4105490 0x7f84a410bd20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:20.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.319+0000 7f84aba00640 1 --2- 172.21.15.45:0/3428266883 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84a4104ad0 0x7f84a4104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:20.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.319+0000 7f84aba00640 1 --2- 172.21.15.45:0/3428266883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84a41038e0 0x7f84a4103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:20.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.319+0000 7f84aba00640 1 -- 172.21.15.45:0/3428266883 >> 172.21.15.45:0/3428266883 conn(0x7f84a40fee50 msgr2=0x7f84a4101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:20.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.319+0000 7f84aba00640 1 -- 172.21.15.45:0/3428266883 shutdown_connections 2024-03-28T17:46:20.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.319+0000 7f84aba00640 1 -- 172.21.15.45:0/3428266883 wait complete. 2024-03-28T17:46:20.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.320+0000 7f84aba00640 1 Processor -- start 2024-03-28T17:46:20.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.320+0000 7f84aba00640 1 -- start start 2024-03-28T17:46:20.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.320+0000 7f84aba00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84a41038e0 0x7f84a4199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:20.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.320+0000 7f84aba00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84a4104ad0 0x7f84a419a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:20.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.320+0000 7f84aba00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84a4105490 0x7f84a41a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:20.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.320+0000 7f84aba00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a410f540 con 0x7f84a41038e0 2024-03-28T17:46:20.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.320+0000 7f84aba00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a410f3c0 con 0x7f84a4105490 2024-03-28T17:46:20.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.320+0000 7f84aba00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a410f6c0 con 0x7f84a4104ad0 2024-03-28T17:46:20.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.320+0000 7f84a9000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84a41038e0 0x7f84a4199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:20.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.320+0000 7f84a3e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84a4104ad0 0x7f84a419a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:20.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.320+0000 7f84a9a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84a4105490 0x7f84a41a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:20.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.320+0000 7f84a9000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84a41038e0 0x7f84a4199bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41470/0 (socket says 172.21.15.45:41470) 2024-03-28T17:46:20.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.320+0000 7f84a9a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84a4105490 0x7f84a41a1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:57468/0 (socket says 172.21.15.45:57468) 2024-03-28T17:46:20.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.321+0000 7f84a9a00640 1 -- 172.21.15.45:0/308645069 learned_addr learned my addr 172.21.15.45:0/308645069 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:20.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.320+0000 7f84a3e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84a4104ad0 0x7f84a419a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:50022/0 (socket says 172.21.15.45:50022) 2024-03-28T17:46:20.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.321+0000 7f84a9a00640 1 -- 172.21.15.45:0/308645069 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84a4104ad0 msgr2=0x7f84a419a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:20.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.321+0000 7f84a9a00640 1 --2- 172.21.15.45:0/308645069 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84a4104ad0 0x7f84a419a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:20.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.321+0000 7f84a9a00640 1 -- 172.21.15.45:0/308645069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84a41038e0 msgr2=0x7f84a4199bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:20.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.321+0000 7f84a9a00640 1 --2- 172.21.15.45:0/308645069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84a41038e0 0x7f84a4199bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:20.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.321+0000 7f84a9a00640 1 -- 172.21.15.45:0/308645069 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a41a17d0 con 0x7f84a4105490 2024-03-28T17:46:20.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.321+0000 7f84a9000640 1 --2- 172.21.15.45:0/308645069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84a41038e0 0x7f84a4199bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:46:20.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.321+0000 7f84a9a00640 1 --2- 172.21.15.45:0/308645069 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84a4105490 0x7f84a41a1160 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f849c02c630 tx=0x7f849c032ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:20.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.321+0000 7f84a1600640 1 -- 172.21.15.45:0/308645069 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849c041020 con 0x7f84a4105490 2024-03-28T17:46:20.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.321+0000 7f84a1600640 1 -- 172.21.15.45:0/308645069 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f849c034cd0 con 0x7f84a4105490 2024-03-28T17:46:20.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.321+0000 7f84a1600640 1 -- 172.21.15.45:0/308645069 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849c039040 con 0x7f84a4105490 2024-03-28T17:46:20.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.321+0000 7f84aba00640 1 -- 172.21.15.45:0/308645069 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a41a1a60 con 0x7f84a4105490 2024-03-28T17:46:20.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.322+0000 7f84aba00640 1 -- 172.21.15.45:0/308645069 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a41a1f50 con 0x7f84a4105490 2024-03-28T17:46:20.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.323+0000 7f84aba00640 1 -- 172.21.15.45:0/308645069 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a410ac00 con 0x7f84a4105490 2024-03-28T17:46:20.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.324+0000 7f84a1600640 1 -- 172.21.15.45:0/308645069 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f849c044020 con 0x7f84a4105490 2024-03-28T17:46:20.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.326+0000 7f84a1600640 1 --2- 172.21.15.45:0/308645069 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f848c0a2890 0x7f848c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:20.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.326+0000 7f84a1600640 1 -- 172.21.15.45:0/308645069 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f849c0f0850 con 0x7f84a4105490 2024-03-28T17:46:20.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.326+0000 7f84a9000640 1 --2- 172.21.15.45:0/308645069 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f848c0a2890 0x7f848c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:20.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.327+0000 7f84a9000640 1 --2- 172.21.15.45:0/308645069 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f848c0a2890 0x7f848c0a4d50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f84a4104930 tx=0x7f8498005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:20.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.330+0000 7f84a1600640 1 -- 172.21.15.45:0/308645069 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f849c04b020 con 0x7f84a4105490 2024-03-28T17:46:20.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:20 smithi045 ceph-mon[31773]: from='client.24680 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:46:20.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:20 smithi045 ceph-mon[31773]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:46:20.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.563+0000 7f84aba00640 1 -- 172.21.15.45:0/308645069 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f84a4061b70 con 0x7f848c0a2890 2024-03-28T17:46:20.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:20 smithi151 ceph-mon[36481]: from='client.24680 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:46:20.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:20 smithi151 ceph-mon[36481]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:46:20.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.573+0000 7f84a1600640 1 -- 172.21.15.45:0/308645069 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f84a4061b70 con 0x7f848c0a2890 2024-03-28T17:46:20.573 INFO:teuthology.orchestra.run.smithi045.stdout:/volumes/qa/sv_1/e789f540-3642-4eb0-b4d2-e1b55b454a1f 2024-03-28T17:46:20.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.575+0000 7f84aba00640 1 -- 172.21.15.45:0/308645069 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f848c0a2890 msgr2=0x7f848c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:20.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.575+0000 7f84aba00640 1 --2- 172.21.15.45:0/308645069 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f848c0a2890 0x7f848c0a4d50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f84a4104930 tx=0x7f8498005f90 comp rx=0 tx=0).stop 2024-03-28T17:46:20.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.575+0000 7f84aba00640 1 -- 172.21.15.45:0/308645069 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84a4105490 msgr2=0x7f84a41a1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:20.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.575+0000 7f84aba00640 1 --2- 172.21.15.45:0/308645069 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84a4105490 0x7f84a41a1160 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f849c02c630 tx=0x7f849c032ea0 comp rx=0 tx=0).stop 2024-03-28T17:46:20.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.576+0000 7f84aba00640 1 -- 172.21.15.45:0/308645069 shutdown_connections 2024-03-28T17:46:20.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.576+0000 7f84aba00640 1 --2- 172.21.15.45:0/308645069 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f848c0a2890 0x7f848c0a4d50 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:20.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.576+0000 7f84aba00640 1 --2- 172.21.15.45:0/308645069 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84a4105490 0x7f84a41a1160 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:20.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.576+0000 7f84aba00640 1 --2- 172.21.15.45:0/308645069 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84a4104ad0 0x7f84a419a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:20.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.576+0000 7f84aba00640 1 --2- 172.21.15.45:0/308645069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84a41038e0 0x7f84a4199bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:20.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.576+0000 7f84aba00640 1 -- 172.21.15.45:0/308645069 >> 172.21.15.45:0/308645069 conn(0x7f84a40fee50 msgr2=0x7f84a4107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:20.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.576+0000 7f84aba00640 1 -- 172.21.15.45:0/308645069 shutdown_connections 2024-03-28T17:46:20.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.576+0000 7f84aba00640 1 -- 172.21.15.45:0/308645069 wait complete. 2024-03-28T17:46:20.595 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:46:20.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:20 smithi071 ceph-mon[36597]: from='client.24680 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:46:20.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:20 smithi071 ceph-mon[36597]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:46:20.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.827+0000 7f893f200640 1 -- 172.21.15.45:0/3020075019 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8940077c40 msgr2=0x7f89400784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:20.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.827+0000 7f893f200640 1 --2- 172.21.15.45:0/3020075019 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8940077c40 0x7f89400784e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8938009ca0 tx=0x7f893802c0f0 comp rx=0 tx=0).stop 2024-03-28T17:46:20.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.827+0000 7f893f200640 1 -- 172.21.15.45:0/3020075019 shutdown_connections 2024-03-28T17:46:20.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.827+0000 7f893f200640 1 --2- 172.21.15.45:0/3020075019 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8940077c40 0x7f89400784e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:20.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.827+0000 7f893f200640 1 --2- 172.21.15.45:0/3020075019 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89400772a0 0x7f8940077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:20.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.827+0000 7f893f200640 1 --2- 172.21.15.45:0/3020075019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8940076060 0x7f8940076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:20.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.827+0000 7f893f200640 1 -- 172.21.15.45:0/3020075019 >> 172.21.15.45:0/3020075019 conn(0x7f89400fec80 msgr2=0x7f89401010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:20.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.827+0000 7f893f200640 1 -- 172.21.15.45:0/3020075019 shutdown_connections 2024-03-28T17:46:20.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.827+0000 7f893f200640 1 -- 172.21.15.45:0/3020075019 wait complete. 2024-03-28T17:46:20.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.828+0000 7f893f200640 1 Processor -- start 2024-03-28T17:46:20.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.829+0000 7f893f200640 1 -- start start 2024-03-28T17:46:20.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.829+0000 7f893f200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8940076060 0x7f894019ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:20.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.829+0000 7f893f200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89400772a0 0x7f894019e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:20.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.829+0000 7f893f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8940077c40 0x7f89401a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:20.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.829+0000 7f893f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8940113780 con 0x7f8940077c40 2024-03-28T17:46:20.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.829+0000 7f893f200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8940113600 con 0x7f89400772a0 2024-03-28T17:46:20.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.829+0000 7f893f200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8940113900 con 0x7f8940076060 2024-03-28T17:46:20.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.829+0000 7f893d400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89400772a0 0x7f894019e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:20.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.829+0000 7f893de00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8940076060 0x7f894019ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:20.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.829+0000 7f893de00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8940076060 0x7f894019ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:50032/0 (socket says 172.21.15.45:50032) 2024-03-28T17:46:20.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.829+0000 7f893de00640 1 -- 172.21.15.45:0/587241111 learned_addr learned my addr 172.21.15.45:0/587241111 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:20.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.829+0000 7f893d400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89400772a0 0x7f894019e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:57484/0 (socket says 172.21.15.45:57484) 2024-03-28T17:46:20.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.829+0000 7f893e800640 1 --2- 172.21.15.45:0/587241111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8940077c40 0x7f89401a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:20.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.830+0000 7f893de00640 1 -- 172.21.15.45:0/587241111 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89400772a0 msgr2=0x7f894019e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:20.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.830+0000 7f893de00640 1 --2- 172.21.15.45:0/587241111 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89400772a0 0x7f894019e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:20.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.830+0000 7f893de00640 1 -- 172.21.15.45:0/587241111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8940077c40 msgr2=0x7f89401a5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:20.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.830+0000 7f893de00640 1 --2- 172.21.15.45:0/587241111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8940077c40 0x7f89401a5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:20.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.830+0000 7f893de00640 1 -- 172.21.15.45:0/587241111 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89401a59d0 con 0x7f8940076060 2024-03-28T17:46:20.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.830+0000 7f893d400640 1 --2- 172.21.15.45:0/587241111 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89400772a0 0x7f894019e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:46:20.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.830+0000 7f893de00640 1 --2- 172.21.15.45:0/587241111 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8940076060 0x7f894019ddc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f893000e9f0 tx=0x7f893000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:20.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.830+0000 7f8936a00640 1 -- 172.21.15.45:0/587241111 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893000cd80 con 0x7f8940076060 2024-03-28T17:46:20.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.830+0000 7f893f200640 1 -- 172.21.15.45:0/587241111 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89401a5cc0 con 0x7f8940076060 2024-03-28T17:46:20.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.830+0000 7f8936a00640 1 -- 172.21.15.45:0/587241111 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8930010910 con 0x7f8940076060 2024-03-28T17:46:20.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.830+0000 7f8936a00640 1 -- 172.21.15.45:0/587241111 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89300188f0 con 0x7f8940076060 2024-03-28T17:46:20.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.831+0000 7f893f200640 1 -- 172.21.15.45:0/587241111 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89401a61b0 con 0x7f8940076060 2024-03-28T17:46:20.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.832+0000 7f893f200640 1 -- 172.21.15.45:0/587241111 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89401101c0 con 0x7f8940076060 2024-03-28T17:46:20.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.834+0000 7f8936a00640 1 -- 172.21.15.45:0/587241111 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f8930010ac0 con 0x7f8940076060 2024-03-28T17:46:20.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.835+0000 7f8936a00640 1 --2- 172.21.15.45:0/587241111 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f89240a2910 0x7f89240a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:20.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.835+0000 7f8936a00640 1 -- 172.21.15.45:0/587241111 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f89300ceff0 con 0x7f8940076060 2024-03-28T17:46:20.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.835+0000 7f893d400640 1 --2- 172.21.15.45:0/587241111 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f89240a2910 0x7f89240a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:20.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.836+0000 7f893d400640 1 --2- 172.21.15.45:0/587241111 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f89240a2910 0x7f89240a4dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f892c009fc0 tx=0x7f892c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:20.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:20.837+0000 7f8936a00640 1 -- 172.21.15.45:0/587241111 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f893009b9f0 con 0x7f8940076060 2024-03-28T17:46:21.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.081+0000 7f893f200640 1 -- 172.21.15.45:0/587241111 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f894002d070 con 0x7f8940076060 2024-03-28T17:46:21.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.082+0000 7f8936a00640 1 -- 172.21.15.45:0/587241111 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f893009fa90 con 0x7f8940076060 2024-03-28T17:46:21.082 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:46:21.082 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6839/2313500150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":2313500150},{"type":"v1","addr":"172.21.15.45:6839","nonce":2313500150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6841/2647972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":2647972571},{"type":"v1","addr":"172.21.15.45:6841","nonce":2647972571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T17:46:12.261135+0000","modified":"2024-03-28T17:46:17.967524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24584,"mds_1":24530,"mds_2":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6835/2876354410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":2876354410},{"type":"v1","addr":"172.21.15.151:6835","nonce":2876354410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24500":{"gid":24500,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/4234998385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":4234998385},{"type":"v1","addr":"172.21.15.71:6835","nonce":4234998385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6837/1238964845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":1238964845},{"type":"v1","addr":"172.21.15.151:6837","nonce":1238964845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.151:6839/3003564901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6838","nonce":3003564901},{"type":"v1","addr":"172.21.15.151:6839","nonce":3003564901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6837/2008752655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2008752655},{"type":"v1","addr":"172.21.15.71:6837","nonce":2008752655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.71:6839/2709938800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":2709938800},{"type":"v1","addr":"172.21.15.71:6839","nonce":2709938800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24584,"qdb_cluster":[24529,24530,24584]},"id":1}]} 2024-03-28T17:46:21.083 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 22 2024-03-28T17:46:21.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.084+0000 7f893f200640 1 -- 172.21.15.45:0/587241111 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f89240a2910 msgr2=0x7f89240a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:21.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.084+0000 7f893f200640 1 --2- 172.21.15.45:0/587241111 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f89240a2910 0x7f89240a4dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f892c009fc0 tx=0x7f892c005f90 comp rx=0 tx=0).stop 2024-03-28T17:46:21.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.084+0000 7f893f200640 1 -- 172.21.15.45:0/587241111 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8940076060 msgr2=0x7f894019ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:21.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.084+0000 7f893f200640 1 --2- 172.21.15.45:0/587241111 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8940076060 0x7f894019ddc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f893000e9f0 tx=0x7f893000eeb0 comp rx=0 tx=0).stop 2024-03-28T17:46:21.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.084+0000 7f893f200640 1 -- 172.21.15.45:0/587241111 shutdown_connections 2024-03-28T17:46:21.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.085+0000 7f893f200640 1 --2- 172.21.15.45:0/587241111 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f89240a2910 0x7f89240a4dd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:21.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.085+0000 7f893f200640 1 --2- 172.21.15.45:0/587241111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8940077c40 0x7f89401a5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:21.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.085+0000 7f893f200640 1 --2- 172.21.15.45:0/587241111 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89400772a0 0x7f894019e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:21.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.085+0000 7f893f200640 1 --2- 172.21.15.45:0/587241111 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8940076060 0x7f894019ddc0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:21.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.085+0000 7f893f200640 1 -- 172.21.15.45:0/587241111 >> 172.21.15.45:0/587241111 conn(0x7f89400fec80 msgr2=0x7f8940100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:21.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.085+0000 7f893f200640 1 -- 172.21.15.45:0/587241111 shutdown_connections 2024-03-28T17:46:21.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.085+0000 7f893f200640 1 -- 172.21.15.45:0/587241111 wait complete. 2024-03-28T17:46:21.105 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T17:46:21.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.326+0000 7fb8bf400640 1 -- 172.21.15.45:0/183963906 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8b81016c0 msgr2=0x7fb8b8101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:21.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.326+0000 7fb8bf400640 1 --2- 172.21.15.45:0/183963906 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8b81016c0 0x7fb8b8101ae0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009900 tx=0x7fb8ac02bc30 comp rx=0 tx=0).stop 2024-03-28T17:46:21.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.326+0000 7fb8bf400640 1 -- 172.21.15.45:0/183963906 shutdown_connections 2024-03-28T17:46:21.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.326+0000 7fb8bf400640 1 --2- 172.21.15.45:0/183963906 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb8b8103270 0x7fb8b8109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:21.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.326+0000 7fb8bf400640 1 --2- 172.21.15.45:0/183963906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8b81028b0 0x7fb8b8102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:21.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.326+0000 7fb8bf400640 1 --2- 172.21.15.45:0/183963906 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8b81016c0 0x7fb8b8101ae0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:21.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.326+0000 7fb8bf400640 1 -- 172.21.15.45:0/183963906 >> 172.21.15.45:0/183963906 conn(0x7fb8b80fcc50 msgr2=0x7fb8b80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:21.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.326+0000 7fb8bf400640 1 -- 172.21.15.45:0/183963906 shutdown_connections 2024-03-28T17:46:21.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.326+0000 7fb8bf400640 1 -- 172.21.15.45:0/183963906 wait complete. 2024-03-28T17:46:21.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.327+0000 7fb8bf400640 1 Processor -- start 2024-03-28T17:46:21.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.327+0000 7fb8bf400640 1 -- start start 2024-03-28T17:46:21.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.328+0000 7fb8bf400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb8b81016c0 0x7fb8b8075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:21.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.328+0000 7fb8bf400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8b81028b0 0x7fb8b8073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:21.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.328+0000 7fb8bf400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8b8103270 0x7fb8b8073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:21.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.328+0000 7fb8bf400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b810d390 con 0x7fb8b81028b0 2024-03-28T17:46:21.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.328+0000 7fb8bf400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b810d210 con 0x7fb8b8103270 2024-03-28T17:46:21.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.328+0000 7fb8bf400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b810d510 con 0x7fb8b81016c0 2024-03-28T17:46:21.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.328+0000 7fb8bca00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb8b81016c0 0x7fb8b8075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:21.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.328+0000 7fb8bca00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb8b81016c0 0x7fb8b8075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:50052/0 (socket says 172.21.15.45:50052) 2024-03-28T17:46:21.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.328+0000 7fb8bca00640 1 -- 172.21.15.45:0/3527279728 learned_addr learned my addr 172.21.15.45:0/3527279728 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:21.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.328+0000 7fb8bd400640 1 --2- 172.21.15.45:0/3527279728 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8b8103270 0x7fb8b8073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:21.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.328+0000 7fb8b7e00640 1 --2- 172.21.15.45:0/3527279728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8b81028b0 0x7fb8b8073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:21.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.328+0000 7fb8bca00640 1 -- 172.21.15.45:0/3527279728 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8b8103270 msgr2=0x7fb8b8073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:21.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.329+0000 7fb8bca00640 1 --2- 172.21.15.45:0/3527279728 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8b8103270 0x7fb8b8073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:21.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.329+0000 7fb8bca00640 1 -- 172.21.15.45:0/3527279728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8b81028b0 msgr2=0x7fb8b8073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:21.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.329+0000 7fb8bca00640 1 --2- 172.21.15.45:0/3527279728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8b81028b0 0x7fb8b8073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:21.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.329+0000 7fb8bca00640 1 -- 172.21.15.45:0/3527279728 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b8074860 con 0x7fb8b81016c0 2024-03-28T17:46:21.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.329+0000 7fb8bca00640 1 --2- 172.21.15.45:0/3527279728 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb8b81016c0 0x7fb8b8075450 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac000c00 tx=0x7fb8ac02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:21.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.329+0000 7fb8b5600640 1 -- 172.21.15.45:0/3527279728 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac043020 con 0x7fb8b81016c0 2024-03-28T17:46:21.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.329+0000 7fb8b5600640 1 -- 172.21.15.45:0/3527279728 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb8ac0054e0 con 0x7fb8b81016c0 2024-03-28T17:46:21.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.329+0000 7fb8bf400640 1 -- 172.21.15.45:0/3527279728 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b81a1b20 con 0x7fb8b81016c0 2024-03-28T17:46:21.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.329+0000 7fb8b5600640 1 -- 172.21.15.45:0/3527279728 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac003c40 con 0x7fb8b81016c0 2024-03-28T17:46:21.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.329+0000 7fb8bf400640 1 -- 172.21.15.45:0/3527279728 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b81a2070 con 0x7fb8b81016c0 2024-03-28T17:46:21.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.330+0000 7fb8bf400640 1 -- 172.21.15.45:0/3527279728 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8b81089e0 con 0x7fb8b81016c0 2024-03-28T17:46:21.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.332+0000 7fb8b5600640 1 -- 172.21.15.45:0/3527279728 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fb8ac046020 con 0x7fb8b81016c0 2024-03-28T17:46:21.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.333+0000 7fb8b5600640 1 --2- 172.21.15.45:0/3527279728 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb8a00a2890 0x7fb8a00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:21.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.334+0000 7fb8b5600640 1 -- 172.21.15.45:0/3527279728 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb8ac0f09f0 con 0x7fb8b81016c0 2024-03-28T17:46:21.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.334+0000 7fb8b7e00640 1 --2- 172.21.15.45:0/3527279728 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb8a00a2890 0x7fb8a00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:21.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.334+0000 7fb8b7e00640 1 --2- 172.21.15.45:0/3527279728 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb8a00a2890 0x7fb8a00a4d50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb8b80751c0 tx=0x7fb8a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:21.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.337+0000 7fb8b5600640 1 -- 172.21.15.45:0/3527279728 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb8ac04d020 con 0x7fb8b81016c0 2024-03-28T17:46:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:21 smithi045 ceph-mon[31773]: pgmap v321: 97 pgs: 97 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-03-28T17:46:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:21 smithi045 ceph-mon[31773]: from='client.24692 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:46:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:21 smithi045 ceph-mon[31773]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-28T17:46:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:21 smithi045 ceph-mon[31773]: from='client.24698 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:46:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:21 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/587241111' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:46:21.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.543+0000 7fb8bf400640 1 -- 172.21.15.45:0/3527279728 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8b80622a0 con 0x7fb8b81016c0 2024-03-28T17:46:21.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.545+0000 7fb8b5600640 1 -- 172.21.15.45:0/3527279728 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22208 (secure 0 0 0) 0x7fb8ac0bdaf0 con 0x7fb8b81016c0 2024-03-28T17:46:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:46:21.546 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":85,"fsid":"9f4888ec-ed29-11ee-95d0-87774f69a715","created":"2024-03-28T17:36:51.623185+0000","modified":"2024-03-28T17:46:13.741486+0000","last_up_change":"2024-03-28T17:45:14.043432+0000","last_in_change":"2024-03-28T17:44:53.654489+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T17:40:54.208375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T17:46:10.564519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T17:46:11.734367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"74c56bc2-4169-4919-afd4-2ebd236f23e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6803","nonce":3795754023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6805","nonce":3795754023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6809","nonce":3795754023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6807","nonce":3795754023}]},"public_addr":"172.21.15.45:6803/3795754023","cluster_addr":"172.21.15.45:6805/3795754023","heartbeat_back_addr":"172.21.15.45:6809/3795754023","heartbeat_front_addr":"172.21.15.45:6807/3795754023","state":["exists","up"]},{"osd":1,"uuid":"78dcc21b-75b2-453d-b690-17b55b8ed1a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6801","nonce":2014529906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6803","nonce":2014529906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6807","nonce":2014529906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6805","nonce":2014529906}]},"public_addr":"172.21.15.71:6801/2014529906","cluster_addr":"172.21.15.71:6803/2014529906","heartbeat_back_addr":"172.21.15.71:6807/2014529906","heartbeat_front_addr":"172.21.15.71:6805/2014529906","state":["exists","up"]},{"osd":2,"uuid":"ca9f3f36-b71e-4962-a94f-cedf2f12eb7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6801","nonce":1850933775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6803","nonce":1850933775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6807","nonce":1850933775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6805","nonce":1850933775}]},"public_addr":"172.21.15.151:6801/1850933775","cluster_addr":"172.21.15.151:6803/1850933775","heartbeat_back_addr":"172.21.15.151:6807/1850933775","heartbeat_front_addr":"172.21.15.151:6805/1850933775","state":["exists","up"]},{"osd":3,"uuid":"d3f21f01-2973-413e-8daa-9974b8a44af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6811","nonce":3021569975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6813","nonce":3021569975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6817","nonce":3021569975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6815","nonce":3021569975}]},"public_addr":"172.21.15.45:6811/3021569975","cluster_addr":"172.21.15.45:6813/3021569975","heartbeat_back_addr":"172.21.15.45:6817/3021569975","heartbeat_front_addr":"172.21.15.45:6815/3021569975","state":["exists","up"]},{"osd":4,"uuid":"4a336f42-0605-492b-ad88-56cb02acdcf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6809","nonce":2069578333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6811","nonce":2069578333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6815","nonce":2069578333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6813","nonce":2069578333}]},"public_addr":"172.21.15.71:6809/2069578333","cluster_addr":"172.21.15.71:6811/2069578333","heartbeat_back_addr":"172.21.15.71:6815/2069578333","heartbeat_front_addr":"172.21.15.71:6813/2069578333","state":["exists","up"]},{"osd":5,"uuid":"c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6809","nonce":2629217227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6811","nonce":2629217227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6815","nonce":2629217227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6813","nonce":2629217227}]},"public_addr":"172.21.15.151:6809/2629217227","cluster_addr":"172.21.15.151:6811/2629217227","heartbeat_back_addr":"172.21.15.151:6815/2629217227","heartbeat_front_addr":"172.21.15.151:6813/2629217227","state":["exists","up"]},{"osd":6,"uuid":"58a6c5bd-a8b7-4892-b346-bcd7994bcb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6819","nonce":2077907039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6821","nonce":2077907039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6825","nonce":2077907039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6823","nonce":2077907039}]},"public_addr":"172.21.15.45:6819/2077907039","cluster_addr":"172.21.15.45:6821/2077907039","heartbeat_back_addr":"172.21.15.45:6825/2077907039","heartbeat_front_addr":"172.21.15.45:6823/2077907039","state":["exists","up"]},{"osd":7,"uuid":"f7a45d52-807b-4407-bb04-787b2a52a0ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6817","nonce":2000211251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6819","nonce":2000211251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6823","nonce":2000211251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6821","nonce":2000211251}]},"public_addr":"172.21.15.71:6817/2000211251","cluster_addr":"172.21.15.71:6819/2000211251","heartbeat_back_addr":"172.21.15.71:6823/2000211251","heartbeat_front_addr":"172.21.15.71:6821/2000211251","state":["exists","up"]},{"osd":8,"uuid":"88abcb0f-f189-49be-8315-c700c77369f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6817","nonce":3665460137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6819","nonce":3665460137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6823","nonce":3665460137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6821","nonce":3665460137}]},"public_addr":"172.21.15.151:6817/3665460137","cluster_addr":"172.21.15.151:6819/3665460137","heartbeat_back_addr":"172.21.15.151:6823/3665460137","heartbeat_front_addr":"172.21.15.151:6821/3665460137","state":["exists","up"]},{"osd":9,"uuid":"ed3fc096-961e-4392-abc2-c5aa4d050117","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6827","nonce":2122322684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6829","nonce":2122322684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6833","nonce":2122322684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6831","nonce":2122322684}]},"public_addr":"172.21.15.45:6827/2122322684","cluster_addr":"172.21.15.45:6829/2122322684","heartbeat_back_addr":"172.21.15.45:6833/2122322684","heartbeat_front_addr":"172.21.15.45:6831/2122322684","state":["exists","up"]},{"osd":10,"uuid":"ba94762b-bc5c-4c97-9cc1-17e5651c14b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6825","nonce":1099494037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6827","nonce":1099494037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6831","nonce":1099494037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6829","nonce":1099494037}]},"public_addr":"172.21.15.71:6825/1099494037","cluster_addr":"172.21.15.71:6827/1099494037","heartbeat_back_addr":"172.21.15.71:6831/1099494037","heartbeat_front_addr":"172.21.15.71:6829/1099494037","state":["exists","up"]},{"osd":11,"uuid":"1d178378-b9f5-40f6-99ef-3d3d445da640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6825","nonce":2941055364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6827","nonce":2941055364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6831","nonce":2941055364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6829","nonce":2941055364}]},"public_addr":"172.21.15.151:6825/2941055364","cluster_addr":"172.21.15.151:6827/2941055364","heartbeat_back_addr":"172.21.15.151:6831/2941055364","heartbeat_front_addr":"172.21.15.151:6829/2941055364","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:02.269586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:27.819386+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:51.552277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:25.052674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:51.873529+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:17.127867+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:51.588249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:18.381450+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:44.140534+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:17.634362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:45.469340+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:45:12.003433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3533611957":"2024-03-29T17:38:07.505611+0000","172.21.15.45:0/3625922430":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6801/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/482610468":"2024-03-29T17:37:32.424229+0000","172.21.15.45:6801/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3649804714":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/1198364079":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3211303958":"2024-03-29T17:38:07.505611+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T17:46:21.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.547+0000 7fb8bf400640 1 -- 172.21.15.45:0/3527279728 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb8a00a2890 msgr2=0x7fb8a00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:21.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.547+0000 7fb8bf400640 1 --2- 172.21.15.45:0/3527279728 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb8a00a2890 0x7fb8a00a4d50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb8b80751c0 tx=0x7fb8a4005f90 comp rx=0 tx=0).stop 2024-03-28T17:46:21.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.547+0000 7fb8bf400640 1 -- 172.21.15.45:0/3527279728 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb8b81016c0 msgr2=0x7fb8b8075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:21.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.547+0000 7fb8bf400640 1 --2- 172.21.15.45:0/3527279728 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb8b81016c0 0x7fb8b8075450 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac000c00 tx=0x7fb8ac02c650 comp rx=0 tx=0).stop 2024-03-28T17:46:21.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.548+0000 7fb8bf400640 1 -- 172.21.15.45:0/3527279728 shutdown_connections 2024-03-28T17:46:21.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.548+0000 7fb8bf400640 1 --2- 172.21.15.45:0/3527279728 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb8a00a2890 0x7fb8a00a4d50 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:21.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.548+0000 7fb8bf400640 1 --2- 172.21.15.45:0/3527279728 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8b8103270 0x7fb8b8073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:21.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.548+0000 7fb8bf400640 1 --2- 172.21.15.45:0/3527279728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8b81028b0 0x7fb8b8073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:21.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.548+0000 7fb8bf400640 1 --2- 172.21.15.45:0/3527279728 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb8b81016c0 0x7fb8b8075450 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:21.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.548+0000 7fb8bf400640 1 -- 172.21.15.45:0/3527279728 >> 172.21.15.45:0/3527279728 conn(0x7fb8b80fcc50 msgr2=0x7fb8b8105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:21.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.548+0000 7fb8bf400640 1 -- 172.21.15.45:0/3527279728 shutdown_connections 2024-03-28T17:46:21.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.548+0000 7fb8bf400640 1 -- 172.21.15.45:0/3527279728 wait complete. 2024-03-28T17:46:21.568 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T17:46:21.569 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T17:46:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:21 smithi151 ceph-mon[36481]: pgmap v321: 97 pgs: 97 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-03-28T17:46:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:21 smithi151 ceph-mon[36481]: from='client.24692 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:46:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:21 smithi151 ceph-mon[36481]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-28T17:46:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:21 smithi151 ceph-mon[36481]: from='client.24698 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:46:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:21 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/587241111' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:46:21.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.789+0000 7ff99ea00640 1 -- 172.21.15.45:0/1954664382 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9981048e0 msgr2=0x7ff998104d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:21.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.789+0000 7ff99ea00640 1 --2- 172.21.15.45:0/1954664382 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9981048e0 0x7ff998104d60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff984009900 tx=0x7ff98402bc30 comp rx=0 tx=0).stop 2024-03-28T17:46:21.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.789+0000 7ff99ea00640 1 -- 172.21.15.45:0/1954664382 shutdown_connections 2024-03-28T17:46:21.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.789+0000 7ff99ea00640 1 --2- 172.21.15.45:0/1954664382 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9981052a0 0x7ff99810bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:21.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.789+0000 7ff99ea00640 1 --2- 172.21.15.45:0/1954664382 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9981048e0 0x7ff998104d60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:21.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.789+0000 7ff99ea00640 1 --2- 172.21.15.45:0/1954664382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9981036f0 0x7ff998103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:21.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.789+0000 7ff99ea00640 1 -- 172.21.15.45:0/1954664382 >> 172.21.15.45:0/1954664382 conn(0x7ff9980fec80 msgr2=0x7ff9981010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:21.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.789+0000 7ff99ea00640 1 -- 172.21.15.45:0/1954664382 shutdown_connections 2024-03-28T17:46:21.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.789+0000 7ff99ea00640 1 -- 172.21.15.45:0/1954664382 wait complete. 2024-03-28T17:46:21.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.790+0000 7ff99ea00640 1 Processor -- start 2024-03-28T17:46:21.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.790+0000 7ff99ea00640 1 -- start start 2024-03-28T17:46:21.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.790+0000 7ff99ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9981036f0 0x7ff9981999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:21.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.790+0000 7ff99ea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9981048e0 0x7ff998199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:21.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.790+0000 7ff99ea00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9981052a0 0x7ff9981a0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:21.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.790+0000 7ff99ea00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99810f3c0 con 0x7ff9981036f0 2024-03-28T17:46:21.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.790+0000 7ff99ea00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99810f240 con 0x7ff9981048e0 2024-03-28T17:46:21.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.790+0000 7ff99ea00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99810f540 con 0x7ff9981052a0 2024-03-28T17:46:21.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.791+0000 7ff99cc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9981048e0 0x7ff998199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:21.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.791+0000 7ff99d600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9981036f0 0x7ff9981999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:21.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.791+0000 7ff99cc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9981048e0 0x7ff998199ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:57524/0 (socket says 172.21.15.45:57524) 2024-03-28T17:46:21.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.791+0000 7ff99cc00640 1 -- 172.21.15.45:0/1263176271 learned_addr learned my addr 172.21.15.45:0/1263176271 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:21.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.791+0000 7ff99e000640 1 --2- 172.21.15.45:0/1263176271 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9981052a0 0x7ff9981a0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:21.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.791+0000 7ff99cc00640 1 -- 172.21.15.45:0/1263176271 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9981052a0 msgr2=0x7ff9981a0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:21.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.791+0000 7ff99cc00640 1 --2- 172.21.15.45:0/1263176271 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9981052a0 0x7ff9981a0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:21.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.791+0000 7ff99cc00640 1 -- 172.21.15.45:0/1263176271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9981036f0 msgr2=0x7ff9981999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:21.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.791+0000 7ff99cc00640 1 --2- 172.21.15.45:0/1263176271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9981036f0 0x7ff9981999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:21.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.791+0000 7ff99cc00640 1 -- 172.21.15.45:0/1263176271 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9981a15c0 con 0x7ff9981048e0 2024-03-28T17:46:21.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.791+0000 7ff99d600640 1 --2- 172.21.15.45:0/1263176271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9981036f0 0x7ff9981999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:46:21.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.791+0000 7ff99cc00640 1 --2- 172.21.15.45:0/1263176271 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9981048e0 0x7ff998199ee0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff984000c00 tx=0x7ff98402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:21.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.792+0000 7ff996000640 1 -- 172.21.15.45:0/1263176271 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984043020 con 0x7ff9981048e0 2024-03-28T17:46:21.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.792+0000 7ff996000640 1 -- 172.21.15.45:0/1263176271 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9840054e0 con 0x7ff9981048e0 2024-03-28T17:46:21.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.792+0000 7ff996000640 1 -- 172.21.15.45:0/1263176271 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984003c40 con 0x7ff9981048e0 2024-03-28T17:46:21.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.792+0000 7ff99ea00640 1 -- 172.21.15.45:0/1263176271 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9981a1850 con 0x7ff9981048e0 2024-03-28T17:46:21.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.792+0000 7ff99ea00640 1 -- 172.21.15.45:0/1263176271 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9981a1d50 con 0x7ff9981048e0 2024-03-28T17:46:21.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.793+0000 7ff99ea00640 1 -- 172.21.15.45:0/1263176271 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff99810aa10 con 0x7ff9981048e0 2024-03-28T17:46:21.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.795+0000 7ff996000640 1 -- 172.21.15.45:0/1263176271 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7ff984046020 con 0x7ff9981048e0 2024-03-28T17:46:21.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.796+0000 7ff996000640 1 --2- 172.21.15.45:0/1263176271 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff9800a2890 0x7ff9800a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:21.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.796+0000 7ff996000640 1 -- 172.21.15.45:0/1263176271 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff9840f0a70 con 0x7ff9981048e0 2024-03-28T17:46:21.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.796+0000 7ff99d600640 1 --2- 172.21.15.45:0/1263176271 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff9800a2890 0x7ff9800a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:21.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.797+0000 7ff99d600640 1 --2- 172.21.15.45:0/1263176271 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff9800a2890 0x7ff9800a4d50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff998104740 tx=0x7ff98c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:21.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.800+0000 7ff996000640 1 -- 172.21.15.45:0/1263176271 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff98404d020 con 0x7ff9981048e0 2024-03-28T17:46:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:21 smithi071 ceph-mon[36597]: pgmap v321: 97 pgs: 97 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-03-28T17:46:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:21 smithi071 ceph-mon[36597]: from='client.24692 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:46:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:21 smithi071 ceph-mon[36597]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-28T17:46:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:21 smithi071 ceph-mon[36597]: from='client.24698 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:46:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:21 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/587241111' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:46:21.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:21.999+0000 7ff99ea00640 1 -- 172.21.15.45:0/1263176271 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff998061ae0 con 0x7ff9800a2890 2024-03-28T17:46:22.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.015+0000 7ff996000640 1 -- 172.21.15.45:0/1263176271 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258895 (secure 0 0 0) 0x7ff998061ae0 con 0x7ff9800a2890 2024-03-28T17:46:22.019 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:46:22.020 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-28T17:46:22.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.021+0000 7ff99ea00640 1 -- 172.21.15.45:0/1263176271 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff9800a2890 msgr2=0x7ff9800a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:22.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.021+0000 7ff99ea00640 1 --2- 172.21.15.45:0/1263176271 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff9800a2890 0x7ff9800a4d50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff998104740 tx=0x7ff98c005f90 comp rx=0 tx=0).stop 2024-03-28T17:46:22.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.021+0000 7ff99ea00640 1 -- 172.21.15.45:0/1263176271 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9981048e0 msgr2=0x7ff998199ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:22.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.021+0000 7ff99ea00640 1 --2- 172.21.15.45:0/1263176271 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9981048e0 0x7ff998199ee0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff984000c00 tx=0x7ff98402c650 comp rx=0 tx=0).stop 2024-03-28T17:46:22.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.021+0000 7ff99ea00640 1 -- 172.21.15.45:0/1263176271 shutdown_connections 2024-03-28T17:46:22.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.021+0000 7ff99ea00640 1 --2- 172.21.15.45:0/1263176271 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff9800a2890 0x7ff9800a4d50 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:22.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.021+0000 7ff99ea00640 1 --2- 172.21.15.45:0/1263176271 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9981052a0 0x7ff9981a0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:22.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.021+0000 7ff99ea00640 1 --2- 172.21.15.45:0/1263176271 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9981048e0 0x7ff998199ee0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:22.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.021+0000 7ff99ea00640 1 --2- 172.21.15.45:0/1263176271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9981036f0 0x7ff9981999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:22.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.021+0000 7ff99ea00640 1 -- 172.21.15.45:0/1263176271 >> 172.21.15.45:0/1263176271 conn(0x7ff9980fec80 msgr2=0x7ff998107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:22.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.021+0000 7ff99ea00640 1 -- 172.21.15.45:0/1263176271 shutdown_connections 2024-03-28T17:46:22.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.021+0000 7ff99ea00640 1 -- 172.21.15.45:0/1263176271 wait complete. 2024-03-28T17:46:22.040 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-03-28T17:46:21.647753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596489,"num_objects":63,"num_object_clones":0,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":162,"num_read_kb":141,"num_write":311,"num_write_kb":2980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":250,"ondisk_log_size":250,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360284,"kb_used_data":11132,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581540,"statfs":{"total":1151940427776,"available":1151571496960,"internally_reserved":0,"allocated":11399168,"data_stored":7107771,"data_compressed":110469,"data_compressed_allocated":139264,"data_compressed_original":1777664,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6121,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":26,"num_read_kb":22,"num_write":130,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.904669"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140522+0000","last_change":"2024-03-28T17:46:12.783038+0000","last_active":"2024-03-28T17:46:14.140522+0000","last_peered":"2024-03-28T17:46:14.140522+0000","last_clean":"2024-03-28T17:46:14.140522+0000","last_became_active":"2024-03-28T17:46:12.782668+0000","last_became_peered":"2024-03-28T17:46:12.782668+0000","last_unstale":"2024-03-28T17:46:14.140522+0000","last_undegraded":"2024-03-28T17:46:14.140522+0000","last_fullsized":"2024-03-28T17:46:14.140522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:03.424959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.143291+0000","last_change":"2024-03-28T17:46:12.781828+0000","last_active":"2024-03-28T17:46:14.143291+0000","last_peered":"2024-03-28T17:46:14.143291+0000","last_clean":"2024-03-28T17:46:14.143291+0000","last_became_active":"2024-03-28T17:46:12.781424+0000","last_became_peered":"2024-03-28T17:46:12.781424+0000","last_unstale":"2024-03-28T17:46:14.143291+0000","last_undegraded":"2024-03-28T17:46:14.143291+0000","last_fullsized":"2024-03-28T17:46:14.143291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:15:16.639473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.142731+0000","last_change":"2024-03-28T17:46:12.785431+0000","last_active":"2024-03-28T17:46:14.142731+0000","last_peered":"2024-03-28T17:46:14.142731+0000","last_clean":"2024-03-28T17:46:14.142731+0000","last_became_active":"2024-03-28T17:46:12.784992+0000","last_became_peered":"2024-03-28T17:46:12.784992+0000","last_unstale":"2024-03-28T17:46:14.142731+0000","last_undegraded":"2024-03-28T17:46:14.142731+0000","last_fullsized":"2024-03-28T17:46:14.142731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:03:36.926613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140143+0000","last_change":"2024-03-28T17:46:12.765274+0000","last_active":"2024-03-28T17:46:14.140143+0000","last_peered":"2024-03-28T17:46:14.140143+0000","last_clean":"2024-03-28T17:46:14.140143+0000","last_became_active":"2024-03-28T17:46:12.764890+0000","last_became_peered":"2024-03-28T17:46:12.764890+0000","last_unstale":"2024-03-28T17:46:14.140143+0000","last_undegraded":"2024-03-28T17:46:14.140143+0000","last_fullsized":"2024-03-28T17:46:14.140143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:58:37.149756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139637+0000","last_change":"2024-03-28T17:46:12.782791+0000","last_active":"2024-03-28T17:46:14.139637+0000","last_peered":"2024-03-28T17:46:14.139637+0000","last_clean":"2024-03-28T17:46:14.139637+0000","last_became_active":"2024-03-28T17:46:12.782420+0000","last_became_peered":"2024-03-28T17:46:12.782420+0000","last_unstale":"2024-03-28T17:46:14.139637+0000","last_undegraded":"2024-03-28T17:46:14.139637+0000","last_fullsized":"2024-03-28T17:46:14.139637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:30.512863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140112+0000","last_change":"2024-03-28T17:46:12.780713+0000","last_active":"2024-03-28T17:46:14.140112+0000","last_peered":"2024-03-28T17:46:14.140112+0000","last_clean":"2024-03-28T17:46:14.140112+0000","last_became_active":"2024-03-28T17:46:12.780305+0000","last_became_peered":"2024-03-28T17:46:12.780305+0000","last_unstale":"2024-03-28T17:46:14.140112+0000","last_undegraded":"2024-03-28T17:46:14.140112+0000","last_fullsized":"2024-03-28T17:46:14.140112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:08:09.840491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139743+0000","last_change":"2024-03-28T17:46:12.778120+0000","last_active":"2024-03-28T17:46:14.139743+0000","last_peered":"2024-03-28T17:46:14.139743+0000","last_clean":"2024-03-28T17:46:14.139743+0000","last_became_active":"2024-03-28T17:46:12.777724+0000","last_became_peered":"2024-03-28T17:46:12.777724+0000","last_unstale":"2024-03-28T17:46:14.139743+0000","last_undegraded":"2024-03-28T17:46:14.139743+0000","last_fullsized":"2024-03-28T17:46:14.139743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:29:50.980366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140014+0000","last_change":"2024-03-28T17:46:12.793447+0000","last_active":"2024-03-28T17:46:14.140014+0000","last_peered":"2024-03-28T17:46:14.140014+0000","last_clean":"2024-03-28T17:46:14.140014+0000","last_became_active":"2024-03-28T17:46:12.793070+0000","last_became_peered":"2024-03-28T17:46:12.793070+0000","last_unstale":"2024-03-28T17:46:14.140014+0000","last_undegraded":"2024-03-28T17:46:14.140014+0000","last_fullsized":"2024-03-28T17:46:14.140014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:01.364577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.749303+0000","last_change":"2024-03-28T17:46:12.763908+0000","last_active":"2024-03-28T17:46:13.749303+0000","last_peered":"2024-03-28T17:46:13.749303+0000","last_clean":"2024-03-28T17:46:13.749303+0000","last_became_active":"2024-03-28T17:46:12.763267+0000","last_became_peered":"2024-03-28T17:46:12.763267+0000","last_unstale":"2024-03-28T17:46:13.749303+0000","last_undegraded":"2024-03-28T17:46:13.749303+0000","last_fullsized":"2024-03-28T17:46:13.749303+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:10.692974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140573+0000","last_change":"2024-03-28T17:46:12.762648+0000","last_active":"2024-03-28T17:46:14.140573+0000","last_peered":"2024-03-28T17:46:14.140573+0000","last_clean":"2024-03-28T17:46:14.140573+0000","last_became_active":"2024-03-28T17:46:12.762300+0000","last_became_peered":"2024-03-28T17:46:12.762300+0000","last_unstale":"2024-03-28T17:46:14.140573+0000","last_undegraded":"2024-03-28T17:46:14.140573+0000","last_fullsized":"2024-03-28T17:46:14.140573+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:02.210828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140429+0000","last_change":"2024-03-28T17:46:12.766482+0000","last_active":"2024-03-28T17:46:14.140429+0000","last_peered":"2024-03-28T17:46:14.140429+0000","last_clean":"2024-03-28T17:46:14.140429+0000","last_became_active":"2024-03-28T17:46:12.766155+0000","last_became_peered":"2024-03-28T17:46:12.766155+0000","last_unstale":"2024-03-28T17:46:14.140429+0000","last_undegraded":"2024-03-28T17:46:14.140429+0000","last_fullsized":"2024-03-28T17:46:14.140429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:46:24.261744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139882+0000","last_change":"2024-03-28T17:46:12.766100+0000","last_active":"2024-03-28T17:46:14.139882+0000","last_peered":"2024-03-28T17:46:14.139882+0000","last_clean":"2024-03-28T17:46:14.139882+0000","last_became_active":"2024-03-28T17:46:12.765762+0000","last_became_peered":"2024-03-28T17:46:12.765762+0000","last_unstale":"2024-03-28T17:46:14.139882+0000","last_undegraded":"2024-03-28T17:46:14.139882+0000","last_fullsized":"2024-03-28T17:46:14.139882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:11.666653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.746273+0000","last_change":"2024-03-28T17:46:12.753465+0000","last_active":"2024-03-28T17:46:13.746273+0000","last_peered":"2024-03-28T17:46:13.746273+0000","last_clean":"2024-03-28T17:46:13.746273+0000","last_became_active":"2024-03-28T17:46:12.751931+0000","last_became_peered":"2024-03-28T17:46:12.751931+0000","last_unstale":"2024-03-28T17:46:13.746273+0000","last_undegraded":"2024-03-28T17:46:13.746273+0000","last_fullsized":"2024-03-28T17:46:13.746273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:39:38.757878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.142180+0000","last_change":"2024-03-28T17:46:12.784177+0000","last_active":"2024-03-28T17:46:14.142180+0000","last_peered":"2024-03-28T17:46:14.142180+0000","last_clean":"2024-03-28T17:46:14.142180+0000","last_became_active":"2024-03-28T17:46:12.783823+0000","last_became_peered":"2024-03-28T17:46:12.783823+0000","last_unstale":"2024-03-28T17:46:14.142180+0000","last_undegraded":"2024-03-28T17:46:14.142180+0000","last_fullsized":"2024-03-28T17:46:14.142180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:11:33.221095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.143014+0000","last_change":"2024-03-28T17:46:12.777152+0000","last_active":"2024-03-28T17:46:14.143014+0000","last_peered":"2024-03-28T17:46:14.143014+0000","last_clean":"2024-03-28T17:46:14.143014+0000","last_became_active":"2024-03-28T17:46:12.776758+0000","last_became_peered":"2024-03-28T17:46:12.776758+0000","last_unstale":"2024-03-28T17:46:14.143014+0000","last_undegraded":"2024-03-28T17:46:14.143014+0000","last_fullsized":"2024-03-28T17:46:14.143014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:20:56.587470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.748726+0000","last_change":"2024-03-28T17:46:12.782384+0000","last_active":"2024-03-28T17:46:13.748726+0000","last_peered":"2024-03-28T17:46:13.748726+0000","last_clean":"2024-03-28T17:46:13.748726+0000","last_became_active":"2024-03-28T17:46:12.781971+0000","last_became_peered":"2024-03-28T17:46:12.781971+0000","last_unstale":"2024-03-28T17:46:13.748726+0000","last_undegraded":"2024-03-28T17:46:13.748726+0000","last_fullsized":"2024-03-28T17:46:13.748726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:19:32.652629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140636+0000","last_change":"2024-03-28T17:46:12.789364+0000","last_active":"2024-03-28T17:46:14.140636+0000","last_peered":"2024-03-28T17:46:14.140636+0000","last_clean":"2024-03-28T17:46:14.140636+0000","last_became_active":"2024-03-28T17:46:12.788976+0000","last_became_peered":"2024-03-28T17:46:12.788976+0000","last_unstale":"2024-03-28T17:46:14.140636+0000","last_undegraded":"2024-03-28T17:46:14.140636+0000","last_fullsized":"2024-03-28T17:46:14.140636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:43:37.183705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.143756+0000","last_change":"2024-03-28T17:46:12.782664+0000","last_active":"2024-03-28T17:46:14.143756+0000","last_peered":"2024-03-28T17:46:14.143756+0000","last_clean":"2024-03-28T17:46:14.143756+0000","last_became_active":"2024-03-28T17:46:12.782305+0000","last_became_peered":"2024-03-28T17:46:12.782305+0000","last_unstale":"2024-03-28T17:46:14.143756+0000","last_undegraded":"2024-03-28T17:46:14.143756+0000","last_fullsized":"2024-03-28T17:46:14.143756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:47:18.373517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.746900+0000","last_change":"2024-03-28T17:46:12.794494+0000","last_active":"2024-03-28T17:46:13.746900+0000","last_peered":"2024-03-28T17:46:13.746900+0000","last_clean":"2024-03-28T17:46:13.746900+0000","last_became_active":"2024-03-28T17:46:12.794151+0000","last_became_peered":"2024-03-28T17:46:12.794151+0000","last_unstale":"2024-03-28T17:46:13.746900+0000","last_undegraded":"2024-03-28T17:46:13.746900+0000","last_fullsized":"2024-03-28T17:46:13.746900+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:17:24.596333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:16.975617+0000","last_change":"2024-03-28T17:46:11.759643+0000","last_active":"2024-03-28T17:46:16.975617+0000","last_peered":"2024-03-28T17:46:16.975617+0000","last_clean":"2024-03-28T17:46:16.975617+0000","last_became_active":"2024-03-28T17:46:11.759273+0000","last_became_peered":"2024-03-28T17:46:11.759273+0000","last_unstale":"2024-03-28T17:46:16.975617+0000","last_undegraded":"2024-03-28T17:46:16.975617+0000","last_fullsized":"2024-03-28T17:46:16.975617+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:09:51.764471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.141043+0000","last_change":"2024-03-28T17:46:12.786503+0000","last_active":"2024-03-28T17:46:14.141043+0000","last_peered":"2024-03-28T17:46:14.141043+0000","last_clean":"2024-03-28T17:46:14.141043+0000","last_became_active":"2024-03-28T17:46:12.784651+0000","last_became_peered":"2024-03-28T17:46:12.784651+0000","last_unstale":"2024-03-28T17:46:14.141043+0000","last_undegraded":"2024-03-28T17:46:14.141043+0000","last_fullsized":"2024-03-28T17:46:14.141043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:30:22.401767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.141439+0000","last_change":"2024-03-28T17:46:11.780340+0000","last_active":"2024-03-28T17:46:14.141439+0000","last_peered":"2024-03-28T17:46:14.141439+0000","last_clean":"2024-03-28T17:46:14.141439+0000","last_became_active":"2024-03-28T17:46:11.779917+0000","last_became_peered":"2024-03-28T17:46:11.779917+0000","last_unstale":"2024-03-28T17:46:14.141439+0000","last_undegraded":"2024-03-28T17:46:14.141439+0000","last_fullsized":"2024-03-28T17:46:14.141439+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:22:45.617444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.142365+0000","last_change":"2024-03-28T17:46:12.795086+0000","last_active":"2024-03-28T17:46:14.142365+0000","last_peered":"2024-03-28T17:46:14.142365+0000","last_clean":"2024-03-28T17:46:14.142365+0000","last_became_active":"2024-03-28T17:46:12.794124+0000","last_became_peered":"2024-03-28T17:46:12.794124+0000","last_unstale":"2024-03-28T17:46:14.142365+0000","last_undegraded":"2024-03-28T17:46:14.142365+0000","last_fullsized":"2024-03-28T17:46:14.142365+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:11:05.347047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:16.970109+0000","last_change":"2024-03-28T17:46:11.766541+0000","last_active":"2024-03-28T17:46:16.970109+0000","last_peered":"2024-03-28T17:46:16.970109+0000","last_clean":"2024-03-28T17:46:16.970109+0000","last_became_active":"2024-03-28T17:46:11.766230+0000","last_became_peered":"2024-03-28T17:46:11.766230+0000","last_unstale":"2024-03-28T17:46:16.970109+0000","last_undegraded":"2024-03-28T17:46:16.970109+0000","last_fullsized":"2024-03-28T17:46:16.970109+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:05:23.657666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.748968+0000","last_change":"2024-03-28T17:46:12.768118+0000","last_active":"2024-03-28T17:46:13.748968+0000","last_peered":"2024-03-28T17:46:13.748968+0000","last_clean":"2024-03-28T17:46:13.748968+0000","last_became_active":"2024-03-28T17:46:12.767765+0000","last_became_peered":"2024-03-28T17:46:12.767765+0000","last_unstale":"2024-03-28T17:46:13.748968+0000","last_undegraded":"2024-03-28T17:46:13.748968+0000","last_fullsized":"2024-03-28T17:46:13.748968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:16:12.268968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140881+0000","last_change":"2024-03-28T17:46:11.760204+0000","last_active":"2024-03-28T17:46:14.140881+0000","last_peered":"2024-03-28T17:46:14.140881+0000","last_clean":"2024-03-28T17:46:14.140881+0000","last_became_active":"2024-03-28T17:46:11.759829+0000","last_became_peered":"2024-03-28T17:46:11.759829+0000","last_unstale":"2024-03-28T17:46:14.140881+0000","last_undegraded":"2024-03-28T17:46:14.140881+0000","last_fullsized":"2024-03-28T17:46:14.140881+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:29:02.075424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140896+0000","last_change":"2024-03-28T17:46:12.762229+0000","last_active":"2024-03-28T17:46:14.140896+0000","last_peered":"2024-03-28T17:46:14.140896+0000","last_clean":"2024-03-28T17:46:14.140896+0000","last_became_active":"2024-03-28T17:46:12.761889+0000","last_became_peered":"2024-03-28T17:46:12.761889+0000","last_unstale":"2024-03-28T17:46:14.140896+0000","last_undegraded":"2024-03-28T17:46:14.140896+0000","last_fullsized":"2024-03-28T17:46:14.140896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:15.023966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:18.978434+0000","last_change":"2024-03-28T17:46:11.763152+0000","last_active":"2024-03-28T17:46:18.978434+0000","last_peered":"2024-03-28T17:46:18.978434+0000","last_clean":"2024-03-28T17:46:18.978434+0000","last_became_active":"2024-03-28T17:46:11.762754+0000","last_became_peered":"2024-03-28T17:46:11.762754+0000","last_unstale":"2024-03-28T17:46:18.978434+0000","last_undegraded":"2024-03-28T17:46:18.978434+0000","last_fullsized":"2024-03-28T17:46:18.978434+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:17:06.659908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139414+0000","last_change":"2024-03-28T17:46:12.780701+0000","last_active":"2024-03-28T17:46:14.139414+0000","last_peered":"2024-03-28T17:46:14.139414+0000","last_clean":"2024-03-28T17:46:14.139414+0000","last_became_active":"2024-03-28T17:46:12.780339+0000","last_became_peered":"2024-03-28T17:46:12.780339+0000","last_unstale":"2024-03-28T17:46:14.139414+0000","last_undegraded":"2024-03-28T17:46:14.139414+0000","last_fullsized":"2024-03-28T17:46:14.139414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:59.774888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:16.976152+0000","last_change":"2024-03-28T17:46:11.763387+0000","last_active":"2024-03-28T17:46:16.976152+0000","last_peered":"2024-03-28T17:46:16.976152+0000","last_clean":"2024-03-28T17:46:16.976152+0000","last_became_active":"2024-03-28T17:46:11.763028+0000","last_became_peered":"2024-03-28T17:46:11.763028+0000","last_unstale":"2024-03-28T17:46:16.976152+0000","last_undegraded":"2024-03-28T17:46:16.976152+0000","last_fullsized":"2024-03-28T17:46:16.976152+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:21:31.585047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140361+0000","last_change":"2024-03-28T17:46:12.794167+0000","last_active":"2024-03-28T17:46:14.140361+0000","last_peered":"2024-03-28T17:46:14.140361+0000","last_clean":"2024-03-28T17:46:14.140361+0000","last_became_active":"2024-03-28T17:46:12.793830+0000","last_became_peered":"2024-03-28T17:46:12.793830+0000","last_unstale":"2024-03-28T17:46:14.140361+0000","last_undegraded":"2024-03-28T17:46:14.140361+0000","last_fullsized":"2024-03-28T17:46:14.140361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:20.907752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:16.978907+0000","last_change":"2024-03-28T17:46:11.769322+0000","last_active":"2024-03-28T17:46:16.978907+0000","last_peered":"2024-03-28T17:46:16.978907+0000","last_clean":"2024-03-28T17:46:16.978907+0000","last_became_active":"2024-03-28T17:46:11.768931+0000","last_became_peered":"2024-03-28T17:46:11.768931+0000","last_unstale":"2024-03-28T17:46:16.978907+0000","last_undegraded":"2024-03-28T17:46:16.978907+0000","last_fullsized":"2024-03-28T17:46:16.978907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:29:34.372947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139977+0000","last_change":"2024-03-28T17:46:12.781306+0000","last_active":"2024-03-28T17:46:14.139977+0000","last_peered":"2024-03-28T17:46:14.139977+0000","last_clean":"2024-03-28T17:46:14.139977+0000","last_became_active":"2024-03-28T17:46:12.779161+0000","last_became_peered":"2024-03-28T17:46:12.779161+0000","last_unstale":"2024-03-28T17:46:14.139977+0000","last_undegraded":"2024-03-28T17:46:14.139977+0000","last_fullsized":"2024-03-28T17:46:14.139977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:06:10.302067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139462+0000","last_change":"2024-03-28T17:46:11.762852+0000","last_active":"2024-03-28T17:46:14.139462+0000","last_peered":"2024-03-28T17:46:14.139462+0000","last_clean":"2024-03-28T17:46:14.139462+0000","last_became_active":"2024-03-28T17:46:11.762428+0000","last_became_peered":"2024-03-28T17:46:11.762428+0000","last_unstale":"2024-03-28T17:46:14.139462+0000","last_undegraded":"2024-03-28T17:46:14.139462+0000","last_fullsized":"2024-03-28T17:46:14.139462+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:38:20.433408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.143193+0000","last_change":"2024-03-28T17:46:12.786295+0000","last_active":"2024-03-28T17:46:14.143193+0000","last_peered":"2024-03-28T17:46:14.143193+0000","last_clean":"2024-03-28T17:46:14.143193+0000","last_became_active":"2024-03-28T17:46:12.785894+0000","last_became_peered":"2024-03-28T17:46:12.785894+0000","last_unstale":"2024-03-28T17:46:14.143193+0000","last_undegraded":"2024-03-28T17:46:14.143193+0000","last_fullsized":"2024-03-28T17:46:14.143193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:06.271660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.945393+0000","last_change":"2024-03-28T17:46:11.766787+0000","last_active":"2024-03-28T17:46:15.945393+0000","last_peered":"2024-03-28T17:46:15.945393+0000","last_clean":"2024-03-28T17:46:15.945393+0000","last_became_active":"2024-03-28T17:46:11.766352+0000","last_became_peered":"2024-03-28T17:46:11.766352+0000","last_unstale":"2024-03-28T17:46:15.945393+0000","last_undegraded":"2024-03-28T17:46:15.945393+0000","last_fullsized":"2024-03-28T17:46:15.945393+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:54:48.157597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.749226+0000","last_change":"2024-03-28T17:46:12.785724+0000","last_active":"2024-03-28T17:46:13.749226+0000","last_peered":"2024-03-28T17:46:13.749226+0000","last_clean":"2024-03-28T17:46:13.749226+0000","last_became_active":"2024-03-28T17:46:12.785353+0000","last_became_peered":"2024-03-28T17:46:12.785353+0000","last_unstale":"2024-03-28T17:46:13.749226+0000","last_undegraded":"2024-03-28T17:46:13.749226+0000","last_fullsized":"2024-03-28T17:46:13.749226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:21:22.692899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.936732+0000","last_change":"2024-03-28T17:46:11.763507+0000","last_active":"2024-03-28T17:46:15.936732+0000","last_peered":"2024-03-28T17:46:15.936732+0000","last_clean":"2024-03-28T17:46:15.936732+0000","last_became_active":"2024-03-28T17:46:11.763177+0000","last_became_peered":"2024-03-28T17:46:11.763177+0000","last_unstale":"2024-03-28T17:46:15.936732+0000","last_undegraded":"2024-03-28T17:46:15.936732+0000","last_fullsized":"2024-03-28T17:46:15.936732+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:56:39.696699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.142583+0000","last_change":"2024-03-28T17:46:12.793693+0000","last_active":"2024-03-28T17:46:14.142583+0000","last_peered":"2024-03-28T17:46:14.142583+0000","last_clean":"2024-03-28T17:46:14.142583+0000","last_became_active":"2024-03-28T17:46:12.792981+0000","last_became_peered":"2024-03-28T17:46:12.792981+0000","last_unstale":"2024-03-28T17:46:14.142583+0000","last_undegraded":"2024-03-28T17:46:14.142583+0000","last_fullsized":"2024-03-28T17:46:14.142583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:33:50.727663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.942364+0000","last_change":"2024-03-28T17:46:11.774961+0000","last_active":"2024-03-28T17:46:15.942364+0000","last_peered":"2024-03-28T17:46:15.942364+0000","last_clean":"2024-03-28T17:46:15.942364+0000","last_became_active":"2024-03-28T17:46:11.774645+0000","last_became_peered":"2024-03-28T17:46:11.774645+0000","last_unstale":"2024-03-28T17:46:15.942364+0000","last_undegraded":"2024-03-28T17:46:15.942364+0000","last_fullsized":"2024-03-28T17:46:15.942364+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:38:23.983739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.749179+0000","last_change":"2024-03-28T17:46:12.763698+0000","last_active":"2024-03-28T17:46:13.749179+0000","last_peered":"2024-03-28T17:46:13.749179+0000","last_clean":"2024-03-28T17:46:13.749179+0000","last_became_active":"2024-03-28T17:46:12.763238+0000","last_became_peered":"2024-03-28T17:46:12.763238+0000","last_unstale":"2024-03-28T17:46:13.749179+0000","last_undegraded":"2024-03-28T17:46:13.749179+0000","last_fullsized":"2024-03-28T17:46:13.749179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:12:58.252855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'8","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.938002+0000","last_change":"2024-03-28T17:46:11.763096+0000","last_active":"2024-03-28T17:46:15.938002+0000","last_peered":"2024-03-28T17:46:15.938002+0000","last_clean":"2024-03-28T17:46:15.938002+0000","last_became_active":"2024-03-28T17:46:11.762730+0000","last_became_peered":"2024-03-28T17:46:11.762730+0000","last_unstale":"2024-03-28T17:46:15.938002+0000","last_undegraded":"2024-03-28T17:46:15.938002+0000","last_fullsized":"2024-03-28T17:46:15.938002+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:45:59.469096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140275+0000","last_change":"2024-03-28T17:46:12.760366+0000","last_active":"2024-03-28T17:46:14.140275+0000","last_peered":"2024-03-28T17:46:14.140275+0000","last_clean":"2024-03-28T17:46:14.140275+0000","last_became_active":"2024-03-28T17:46:12.759957+0000","last_became_peered":"2024-03-28T17:46:12.759957+0000","last_unstale":"2024-03-28T17:46:14.140275+0000","last_undegraded":"2024-03-28T17:46:14.140275+0000","last_fullsized":"2024-03-28T17:46:14.140275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:08:02.579659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:16.971643+0000","last_change":"2024-03-28T17:46:11.787108+0000","last_active":"2024-03-28T17:46:16.971643+0000","last_peered":"2024-03-28T17:46:16.971643+0000","last_clean":"2024-03-28T17:46:16.971643+0000","last_became_active":"2024-03-28T17:46:11.786734+0000","last_became_peered":"2024-03-28T17:46:11.786734+0000","last_unstale":"2024-03-28T17:46:16.971643+0000","last_undegraded":"2024-03-28T17:46:16.971643+0000","last_fullsized":"2024-03-28T17:46:16.971643+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:04:00.480545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.745847+0000","last_change":"2024-03-28T17:46:12.786362+0000","last_active":"2024-03-28T17:46:13.745847+0000","last_peered":"2024-03-28T17:46:13.745847+0000","last_clean":"2024-03-28T17:46:13.745847+0000","last_became_active":"2024-03-28T17:46:12.785946+0000","last_became_peered":"2024-03-28T17:46:12.785946+0000","last_unstale":"2024-03-28T17:46:13.745847+0000","last_undegraded":"2024-03-28T17:46:13.745847+0000","last_fullsized":"2024-03-28T17:46:13.745847+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:13:57.857818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.940782+0000","last_change":"2024-03-28T17:46:11.786292+0000","last_active":"2024-03-28T17:46:15.940782+0000","last_peered":"2024-03-28T17:46:15.940782+0000","last_clean":"2024-03-28T17:46:15.940782+0000","last_became_active":"2024-03-28T17:46:11.785953+0000","last_became_peered":"2024-03-28T17:46:11.785953+0000","last_unstale":"2024-03-28T17:46:15.940782+0000","last_undegraded":"2024-03-28T17:46:15.940782+0000","last_fullsized":"2024-03-28T17:46:15.940782+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:02:18.093585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140190+0000","last_change":"2024-03-28T17:46:12.770669+0000","last_active":"2024-03-28T17:46:14.140190+0000","last_peered":"2024-03-28T17:46:14.140190+0000","last_clean":"2024-03-28T17:46:14.140190+0000","last_became_active":"2024-03-28T17:46:12.770301+0000","last_became_peered":"2024-03-28T17:46:12.770301+0000","last_unstale":"2024-03-28T17:46:14.140190+0000","last_undegraded":"2024-03-28T17:46:14.140190+0000","last_fullsized":"2024-03-28T17:46:14.140190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:59:00.601357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'4","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:18.982721+0000","last_change":"2024-03-28T17:46:11.759686+0000","last_active":"2024-03-28T17:46:18.982721+0000","last_peered":"2024-03-28T17:46:18.982721+0000","last_clean":"2024-03-28T17:46:18.982721+0000","last_became_active":"2024-03-28T17:46:11.759309+0000","last_became_peered":"2024-03-28T17:46:11.759309+0000","last_unstale":"2024-03-28T17:46:18.982721+0000","last_undegraded":"2024-03-28T17:46:18.982721+0000","last_fullsized":"2024-03-28T17:46:18.982721+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:31:09.999279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139507+0000","last_change":"2024-03-28T17:46:12.760484+0000","last_active":"2024-03-28T17:46:14.139507+0000","last_peered":"2024-03-28T17:46:14.139507+0000","last_clean":"2024-03-28T17:46:14.139507+0000","last_became_active":"2024-03-28T17:46:12.760121+0000","last_became_peered":"2024-03-28T17:46:12.760121+0000","last_unstale":"2024-03-28T17:46:14.139507+0000","last_undegraded":"2024-03-28T17:46:14.139507+0000","last_fullsized":"2024-03-28T17:46:14.139507+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:33:51.457426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:16.979825+0000","last_change":"2024-03-28T17:46:11.769133+0000","last_active":"2024-03-28T17:46:16.979825+0000","last_peered":"2024-03-28T17:46:16.979825+0000","last_clean":"2024-03-28T17:46:16.979825+0000","last_became_active":"2024-03-28T17:46:11.768857+0000","last_became_peered":"2024-03-28T17:46:11.768857+0000","last_unstale":"2024-03-28T17:46:16.979825+0000","last_undegraded":"2024-03-28T17:46:16.979825+0000","last_fullsized":"2024-03-28T17:46:16.979825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:19:26.065368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139541+0000","last_change":"2024-03-28T17:46:12.765246+0000","last_active":"2024-03-28T17:46:14.139541+0000","last_peered":"2024-03-28T17:46:14.139541+0000","last_clean":"2024-03-28T17:46:14.139541+0000","last_became_active":"2024-03-28T17:46:12.764788+0000","last_became_peered":"2024-03-28T17:46:12.764788+0000","last_unstale":"2024-03-28T17:46:14.139541+0000","last_undegraded":"2024-03-28T17:46:14.139541+0000","last_fullsized":"2024-03-28T17:46:14.139541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:36:22.864337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:17.966415+0000","last_change":"2024-03-28T17:46:11.761005+0000","last_active":"2024-03-28T17:46:17.966415+0000","last_peered":"2024-03-28T17:46:17.966415+0000","last_clean":"2024-03-28T17:46:17.966415+0000","last_became_active":"2024-03-28T17:46:11.760660+0000","last_became_peered":"2024-03-28T17:46:11.760660+0000","last_unstale":"2024-03-28T17:46:17.966415+0000","last_undegraded":"2024-03-28T17:46:17.966415+0000","last_fullsized":"2024-03-28T17:46:17.966415+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:01:10.408853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140954+0000","last_change":"2024-03-28T17:46:12.762851+0000","last_active":"2024-03-28T17:46:14.140954+0000","last_peered":"2024-03-28T17:46:14.140954+0000","last_clean":"2024-03-28T17:46:14.140954+0000","last_became_active":"2024-03-28T17:46:12.762508+0000","last_became_peered":"2024-03-28T17:46:12.762508+0000","last_unstale":"2024-03-28T17:46:14.140954+0000","last_undegraded":"2024-03-28T17:46:14.140954+0000","last_fullsized":"2024-03-28T17:46:14.140954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:55:29.389985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.143754+0000","last_change":"2024-03-28T17:46:11.776919+0000","last_active":"2024-03-28T17:46:14.143754+0000","last_peered":"2024-03-28T17:46:14.143754+0000","last_clean":"2024-03-28T17:46:14.143754+0000","last_became_active":"2024-03-28T17:46:11.776607+0000","last_became_peered":"2024-03-28T17:46:11.776607+0000","last_unstale":"2024-03-28T17:46:14.143754+0000","last_undegraded":"2024-03-28T17:46:14.143754+0000","last_fullsized":"2024-03-28T17:46:14.143754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:20:56.365757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140495+0000","last_change":"2024-03-28T17:46:12.794290+0000","last_active":"2024-03-28T17:46:14.140495+0000","last_peered":"2024-03-28T17:46:14.140495+0000","last_clean":"2024-03-28T17:46:14.140495+0000","last_became_active":"2024-03-28T17:46:12.793906+0000","last_became_peered":"2024-03-28T17:46:12.793906+0000","last_unstale":"2024-03-28T17:46:14.140495+0000","last_undegraded":"2024-03-28T17:46:14.140495+0000","last_fullsized":"2024-03-28T17:46:14.140495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:26:04.996019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.371419+0000","last_change":"2024-03-28T17:46:11.762676+0000","last_active":"2024-03-28T17:46:15.371419+0000","last_peered":"2024-03-28T17:46:15.371419+0000","last_clean":"2024-03-28T17:46:15.371419+0000","last_became_active":"2024-03-28T17:46:11.761458+0000","last_became_peered":"2024-03-28T17:46:11.761458+0000","last_unstale":"2024-03-28T17:46:15.371419+0000","last_undegraded":"2024-03-28T17:46:15.371419+0000","last_fullsized":"2024-03-28T17:46:15.371419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:22:26.642798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140562+0000","last_change":"2024-03-28T17:46:12.787107+0000","last_active":"2024-03-28T17:46:14.140562+0000","last_peered":"2024-03-28T17:46:14.140562+0000","last_clean":"2024-03-28T17:46:14.140562+0000","last_became_active":"2024-03-28T17:46:12.786610+0000","last_became_peered":"2024-03-28T17:46:12.786610+0000","last_unstale":"2024-03-28T17:46:14.140562+0000","last_undegraded":"2024-03-28T17:46:14.140562+0000","last_fullsized":"2024-03-28T17:46:14.140562+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:30:14.880483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139789+0000","last_change":"2024-03-28T17:46:12.763861+0000","last_active":"2024-03-28T17:46:14.139789+0000","last_peered":"2024-03-28T17:46:14.139789+0000","last_clean":"2024-03-28T17:46:14.139789+0000","last_became_active":"2024-03-28T17:46:12.763411+0000","last_became_peered":"2024-03-28T17:46:12.763411+0000","last_unstale":"2024-03-28T17:46:14.139789+0000","last_undegraded":"2024-03-28T17:46:14.139789+0000","last_fullsized":"2024-03-28T17:46:14.139789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:52:02.657945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:17.979059+0000","last_change":"2024-03-28T17:46:11.784261+0000","last_active":"2024-03-28T17:46:17.979059+0000","last_peered":"2024-03-28T17:46:17.979059+0000","last_clean":"2024-03-28T17:46:17.979059+0000","last_became_active":"2024-03-28T17:46:11.782957+0000","last_became_peered":"2024-03-28T17:46:11.782957+0000","last_unstale":"2024-03-28T17:46:17.979059+0000","last_undegraded":"2024-03-28T17:46:17.979059+0000","last_fullsized":"2024-03-28T17:46:17.979059+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:08:48.009774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140767+0000","last_change":"2024-03-28T17:46:12.784024+0000","last_active":"2024-03-28T17:46:14.140767+0000","last_peered":"2024-03-28T17:46:14.140767+0000","last_clean":"2024-03-28T17:46:14.140767+0000","last_became_active":"2024-03-28T17:46:12.782157+0000","last_became_peered":"2024-03-28T17:46:12.782157+0000","last_unstale":"2024-03-28T17:46:14.140767+0000","last_undegraded":"2024-03-28T17:46:14.140767+0000","last_fullsized":"2024-03-28T17:46:14.140767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:33.679894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:17.973716+0000","last_change":"2024-03-28T17:46:11.759602+0000","last_active":"2024-03-28T17:46:17.973716+0000","last_peered":"2024-03-28T17:46:17.973716+0000","last_clean":"2024-03-28T17:46:17.973716+0000","last_became_active":"2024-03-28T17:46:11.759248+0000","last_became_peered":"2024-03-28T17:46:11.759248+0000","last_unstale":"2024-03-28T17:46:17.973716+0000","last_undegraded":"2024-03-28T17:46:17.973716+0000","last_fullsized":"2024-03-28T17:46:17.973716+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:15:59.997948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":347,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139608+0000","last_change":"2024-03-28T17:44:51.172493+0000","last_active":"2024-03-28T17:46:14.139608+0000","last_peered":"2024-03-28T17:46:14.139608+0000","last_clean":"2024-03-28T17:46:14.139608+0000","last_became_active":"2024-03-28T17:44:51.172175+0000","last_became_peered":"2024-03-28T17:44:51.172175+0000","last_unstale":"2024-03-28T17:46:14.139608+0000","last_undegraded":"2024-03-28T17:46:14.139608+0000","last_fullsized":"2024-03-28T17:46:14.139608+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:40:54.580958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:40:54.580958+0000","last_clean_scrub_stamp":"2024-03-28T17:40:54.580958+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:34.942490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.745598+0000","last_change":"2024-03-28T17:46:12.766728+0000","last_active":"2024-03-28T17:46:13.745598+0000","last_peered":"2024-03-28T17:46:13.745598+0000","last_clean":"2024-03-28T17:46:13.745598+0000","last_became_active":"2024-03-28T17:46:12.763671+0000","last_became_peered":"2024-03-28T17:46:12.763671+0000","last_unstale":"2024-03-28T17:46:13.745598+0000","last_undegraded":"2024-03-28T17:46:13.745598+0000","last_fullsized":"2024-03-28T17:46:13.745598+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:25:39.802551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:16.970580+0000","last_change":"2024-03-28T17:46:11.763361+0000","last_active":"2024-03-28T17:46:16.970580+0000","last_peered":"2024-03-28T17:46:16.970580+0000","last_clean":"2024-03-28T17:46:16.970580+0000","last_became_active":"2024-03-28T17:46:11.763049+0000","last_became_peered":"2024-03-28T17:46:11.763049+0000","last_unstale":"2024-03-28T17:46:16.970580+0000","last_undegraded":"2024-03-28T17:46:16.970580+0000","last_fullsized":"2024-03-28T17:46:16.970580+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:01.767964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.946355+0000","last_change":"2024-03-28T17:46:11.774957+0000","last_active":"2024-03-28T17:46:15.946355+0000","last_peered":"2024-03-28T17:46:15.946355+0000","last_clean":"2024-03-28T17:46:15.946355+0000","last_became_active":"2024-03-28T17:46:11.774658+0000","last_became_peered":"2024-03-28T17:46:11.774658+0000","last_unstale":"2024-03-28T17:46:15.946355+0000","last_undegraded":"2024-03-28T17:46:15.946355+0000","last_fullsized":"2024-03-28T17:46:15.946355+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:40.254318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.746510+0000","last_change":"2024-03-28T17:46:12.778206+0000","last_active":"2024-03-28T17:46:13.746510+0000","last_peered":"2024-03-28T17:46:13.746510+0000","last_clean":"2024-03-28T17:46:13.746510+0000","last_became_active":"2024-03-28T17:46:12.777780+0000","last_became_peered":"2024-03-28T17:46:12.777780+0000","last_unstale":"2024-03-28T17:46:13.746510+0000","last_undegraded":"2024-03-28T17:46:13.746510+0000","last_fullsized":"2024-03-28T17:46:13.746510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:13.241794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:16.979312+0000","last_change":"2024-03-28T17:46:11.787480+0000","last_active":"2024-03-28T17:46:16.979312+0000","last_peered":"2024-03-28T17:46:16.979312+0000","last_clean":"2024-03-28T17:46:16.979312+0000","last_became_active":"2024-03-28T17:46:11.786808+0000","last_became_peered":"2024-03-28T17:46:11.786808+0000","last_unstale":"2024-03-28T17:46:16.979312+0000","last_undegraded":"2024-03-28T17:46:16.979312+0000","last_fullsized":"2024-03-28T17:46:16.979312+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:35.236554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140851+0000","last_change":"2024-03-28T17:46:12.790489+0000","last_active":"2024-03-28T17:46:14.140851+0000","last_peered":"2024-03-28T17:46:14.140851+0000","last_clean":"2024-03-28T17:46:14.140851+0000","last_became_active":"2024-03-28T17:46:12.790108+0000","last_became_peered":"2024-03-28T17:46:12.790108+0000","last_unstale":"2024-03-28T17:46:14.140851+0000","last_undegraded":"2024-03-28T17:46:14.140851+0000","last_fullsized":"2024-03-28T17:46:14.140851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:16:21.210182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.943430+0000","last_change":"2024-03-28T17:46:11.762658+0000","last_active":"2024-03-28T17:46:15.943430+0000","last_peered":"2024-03-28T17:46:15.943430+0000","last_clean":"2024-03-28T17:46:15.943430+0000","last_became_active":"2024-03-28T17:46:11.762296+0000","last_became_peered":"2024-03-28T17:46:11.762296+0000","last_unstale":"2024-03-28T17:46:15.943430+0000","last_undegraded":"2024-03-28T17:46:15.943430+0000","last_fullsized":"2024-03-28T17:46:15.943430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:42:52.768285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.143561+0000","last_change":"2024-03-28T17:46:12.790462+0000","last_active":"2024-03-28T17:46:14.143561+0000","last_peered":"2024-03-28T17:46:14.143561+0000","last_clean":"2024-03-28T17:46:14.143561+0000","last_became_active":"2024-03-28T17:46:12.788783+0000","last_became_peered":"2024-03-28T17:46:12.788783+0000","last_unstale":"2024-03-28T17:46:14.143561+0000","last_undegraded":"2024-03-28T17:46:14.143561+0000","last_fullsized":"2024-03-28T17:46:14.143561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:47:09.130970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.942122+0000","last_change":"2024-03-28T17:46:11.769261+0000","last_active":"2024-03-28T17:46:15.942122+0000","last_peered":"2024-03-28T17:46:15.942122+0000","last_clean":"2024-03-28T17:46:15.942122+0000","last_became_active":"2024-03-28T17:46:11.768918+0000","last_became_peered":"2024-03-28T17:46:11.768918+0000","last_unstale":"2024-03-28T17:46:15.942122+0000","last_undegraded":"2024-03-28T17:46:15.942122+0000","last_fullsized":"2024-03-28T17:46:15.942122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:26:38.276425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139702+0000","last_change":"2024-03-28T17:46:12.762617+0000","last_active":"2024-03-28T17:46:14.139702+0000","last_peered":"2024-03-28T17:46:14.139702+0000","last_clean":"2024-03-28T17:46:14.139702+0000","last_became_active":"2024-03-28T17:46:12.762273+0000","last_became_peered":"2024-03-28T17:46:12.762273+0000","last_unstale":"2024-03-28T17:46:14.139702+0000","last_undegraded":"2024-03-28T17:46:14.139702+0000","last_fullsized":"2024-03-28T17:46:14.139702+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:45.848454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.946385+0000","last_change":"2024-03-28T17:46:11.770831+0000","last_active":"2024-03-28T17:46:15.946385+0000","last_peered":"2024-03-28T17:46:15.946385+0000","last_clean":"2024-03-28T17:46:15.946385+0000","last_became_active":"2024-03-28T17:46:11.770303+0000","last_became_peered":"2024-03-28T17:46:11.770303+0000","last_unstale":"2024-03-28T17:46:15.946385+0000","last_undegraded":"2024-03-28T17:46:15.946385+0000","last_fullsized":"2024-03-28T17:46:15.946385+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:48:25.566922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140248+0000","last_change":"2024-03-28T17:46:12.768885+0000","last_active":"2024-03-28T17:46:14.140248+0000","last_peered":"2024-03-28T17:46:14.140248+0000","last_clean":"2024-03-28T17:46:14.140248+0000","last_became_active":"2024-03-28T17:46:12.768525+0000","last_became_peered":"2024-03-28T17:46:12.768525+0000","last_unstale":"2024-03-28T17:46:14.140248+0000","last_undegraded":"2024-03-28T17:46:14.140248+0000","last_fullsized":"2024-03-28T17:46:14.140248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:35:30.779279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.582388+0000","last_change":"2024-03-28T17:46:11.786661+0000","last_active":"2024-03-28T17:46:14.582388+0000","last_peered":"2024-03-28T17:46:14.582388+0000","last_clean":"2024-03-28T17:46:14.582388+0000","last_became_active":"2024-03-28T17:46:11.786226+0000","last_became_peered":"2024-03-28T17:46:11.786226+0000","last_unstale":"2024-03-28T17:46:14.582388+0000","last_undegraded":"2024-03-28T17:46:14.582388+0000","last_fullsized":"2024-03-28T17:46:14.582388+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:51:24.781368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140181+0000","last_change":"2024-03-28T17:46:12.797531+0000","last_active":"2024-03-28T17:46:14.140181+0000","last_peered":"2024-03-28T17:46:14.140181+0000","last_clean":"2024-03-28T17:46:14.140181+0000","last_became_active":"2024-03-28T17:46:12.797161+0000","last_became_peered":"2024-03-28T17:46:12.797161+0000","last_unstale":"2024-03-28T17:46:14.140181+0000","last_undegraded":"2024-03-28T17:46:14.140181+0000","last_fullsized":"2024-03-28T17:46:14.140181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:01.534666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.938194+0000","last_change":"2024-03-28T17:46:11.766589+0000","last_active":"2024-03-28T17:46:15.938194+0000","last_peered":"2024-03-28T17:46:15.938194+0000","last_clean":"2024-03-28T17:46:15.938194+0000","last_became_active":"2024-03-28T17:46:11.766269+0000","last_became_peered":"2024-03-28T17:46:11.766269+0000","last_unstale":"2024-03-28T17:46:15.938194+0000","last_undegraded":"2024-03-28T17:46:15.938194+0000","last_fullsized":"2024-03-28T17:46:15.938194+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:12:26.055262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140141+0000","last_change":"2024-03-28T17:46:12.765723+0000","last_active":"2024-03-28T17:46:14.140141+0000","last_peered":"2024-03-28T17:46:14.140141+0000","last_clean":"2024-03-28T17:46:14.140141+0000","last_became_active":"2024-03-28T17:46:12.765366+0000","last_became_peered":"2024-03-28T17:46:12.765366+0000","last_unstale":"2024-03-28T17:46:14.140141+0000","last_undegraded":"2024-03-28T17:46:14.140141+0000","last_fullsized":"2024-03-28T17:46:14.140141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:39:55.103246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:16.978290+0000","last_change":"2024-03-28T17:46:11.767686+0000","last_active":"2024-03-28T17:46:16.978290+0000","last_peered":"2024-03-28T17:46:16.978290+0000","last_clean":"2024-03-28T17:46:16.978290+0000","last_became_active":"2024-03-28T17:46:11.767365+0000","last_became_peered":"2024-03-28T17:46:11.767365+0000","last_unstale":"2024-03-28T17:46:16.978290+0000","last_undegraded":"2024-03-28T17:46:16.978290+0000","last_fullsized":"2024-03-28T17:46:16.978290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:49.909554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140531+0000","last_change":"2024-03-28T17:46:12.764129+0000","last_active":"2024-03-28T17:46:14.140531+0000","last_peered":"2024-03-28T17:46:14.140531+0000","last_clean":"2024-03-28T17:46:14.140531+0000","last_became_active":"2024-03-28T17:46:12.763716+0000","last_became_peered":"2024-03-28T17:46:12.763716+0000","last_unstale":"2024-03-28T17:46:14.140531+0000","last_undegraded":"2024-03-28T17:46:14.140531+0000","last_fullsized":"2024-03-28T17:46:14.140531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:08.365578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.141019+0000","last_change":"2024-03-28T17:46:11.787374+0000","last_active":"2024-03-28T17:46:14.141019+0000","last_peered":"2024-03-28T17:46:14.141019+0000","last_clean":"2024-03-28T17:46:14.141019+0000","last_became_active":"2024-03-28T17:46:11.787047+0000","last_became_peered":"2024-03-28T17:46:11.787047+0000","last_unstale":"2024-03-28T17:46:14.141019+0000","last_undegraded":"2024-03-28T17:46:14.141019+0000","last_fullsized":"2024-03-28T17:46:14.141019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:33.657488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.371446+0000","last_change":"2024-03-28T17:46:12.794069+0000","last_active":"2024-03-28T17:46:15.371446+0000","last_peered":"2024-03-28T17:46:15.371446+0000","last_clean":"2024-03-28T17:46:15.371446+0000","last_became_active":"2024-03-28T17:46:12.793640+0000","last_became_peered":"2024-03-28T17:46:12.793640+0000","last_unstale":"2024-03-28T17:46:15.371446+0000","last_undegraded":"2024-03-28T17:46:15.371446+0000","last_fullsized":"2024-03-28T17:46:15.371446+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:23:15.065887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.746450+0000","last_change":"2024-03-28T17:46:11.780388+0000","last_active":"2024-03-28T17:46:13.746450+0000","last_peered":"2024-03-28T17:46:13.746450+0000","last_clean":"2024-03-28T17:46:13.746450+0000","last_became_active":"2024-03-28T17:46:11.780016+0000","last_became_peered":"2024-03-28T17:46:11.780016+0000","last_unstale":"2024-03-28T17:46:13.746450+0000","last_undegraded":"2024-03-28T17:46:13.746450+0000","last_fullsized":"2024-03-28T17:46:13.746450+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:51:06.181772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140809+0000","last_change":"2024-03-28T17:46:12.764748+0000","last_active":"2024-03-28T17:46:14.140809+0000","last_peered":"2024-03-28T17:46:14.140809+0000","last_clean":"2024-03-28T17:46:14.140809+0000","last_became_active":"2024-03-28T17:46:12.764407+0000","last_became_peered":"2024-03-28T17:46:12.764407+0000","last_unstale":"2024-03-28T17:46:14.140809+0000","last_undegraded":"2024-03-28T17:46:14.140809+0000","last_fullsized":"2024-03-28T17:46:14.140809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:41:54.185079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139691+0000","last_change":"2024-03-28T17:46:12.762486+0000","last_active":"2024-03-28T17:46:14.139691+0000","last_peered":"2024-03-28T17:46:14.139691+0000","last_clean":"2024-03-28T17:46:14.139691+0000","last_became_active":"2024-03-28T17:46:12.762107+0000","last_became_peered":"2024-03-28T17:46:12.762107+0000","last_unstale":"2024-03-28T17:46:14.139691+0000","last_undegraded":"2024-03-28T17:46:14.139691+0000","last_fullsized":"2024-03-28T17:46:14.139691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:31.062456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.746525+0000","last_change":"2024-03-28T17:46:12.764569+0000","last_active":"2024-03-28T17:46:13.746525+0000","last_peered":"2024-03-28T17:46:13.746525+0000","last_clean":"2024-03-28T17:46:13.746525+0000","last_became_active":"2024-03-28T17:46:12.764112+0000","last_became_peered":"2024-03-28T17:46:12.764112+0000","last_unstale":"2024-03-28T17:46:13.746525+0000","last_undegraded":"2024-03-28T17:46:13.746525+0000","last_fullsized":"2024-03-28T17:46:13.746525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:39:55.526432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.746595+0000","last_change":"2024-03-28T17:46:12.786031+0000","last_active":"2024-03-28T17:46:13.746595+0000","last_peered":"2024-03-28T17:46:13.746595+0000","last_clean":"2024-03-28T17:46:13.746595+0000","last_became_active":"2024-03-28T17:46:12.785679+0000","last_became_peered":"2024-03-28T17:46:12.785679+0000","last_unstale":"2024-03-28T17:46:13.746595+0000","last_undegraded":"2024-03-28T17:46:13.746595+0000","last_fullsized":"2024-03-28T17:46:13.746595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:41:22.537979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.142231+0000","last_change":"2024-03-28T17:46:12.793285+0000","last_active":"2024-03-28T17:46:14.142231+0000","last_peered":"2024-03-28T17:46:14.142231+0000","last_clean":"2024-03-28T17:46:14.142231+0000","last_became_active":"2024-03-28T17:46:12.788935+0000","last_became_peered":"2024-03-28T17:46:12.788935+0000","last_unstale":"2024-03-28T17:46:14.142231+0000","last_undegraded":"2024-03-28T17:46:14.142231+0000","last_fullsized":"2024-03-28T17:46:14.142231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:53:58.886607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.746357+0000","last_change":"2024-03-28T17:46:12.791572+0000","last_active":"2024-03-28T17:46:13.746357+0000","last_peered":"2024-03-28T17:46:13.746357+0000","last_clean":"2024-03-28T17:46:13.746357+0000","last_became_active":"2024-03-28T17:46:12.791081+0000","last_became_peered":"2024-03-28T17:46:12.791081+0000","last_unstale":"2024-03-28T17:46:13.746357+0000","last_undegraded":"2024-03-28T17:46:13.746357+0000","last_fullsized":"2024-03-28T17:46:13.746357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:51:38.955784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140041+0000","last_change":"2024-03-28T17:46:12.765140+0000","last_active":"2024-03-28T17:46:14.140041+0000","last_peered":"2024-03-28T17:46:14.140041+0000","last_clean":"2024-03-28T17:46:14.140041+0000","last_became_active":"2024-03-28T17:46:12.764760+0000","last_became_peered":"2024-03-28T17:46:12.764760+0000","last_unstale":"2024-03-28T17:46:14.140041+0000","last_undegraded":"2024-03-28T17:46:14.140041+0000","last_fullsized":"2024-03-28T17:46:14.140041+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:41.576437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.747038+0000","last_change":"2024-03-28T17:46:12.760271+0000","last_active":"2024-03-28T17:46:13.747038+0000","last_peered":"2024-03-28T17:46:13.747038+0000","last_clean":"2024-03-28T17:46:13.747038+0000","last_became_active":"2024-03-28T17:46:12.759939+0000","last_became_peered":"2024-03-28T17:46:12.759939+0000","last_unstale":"2024-03-28T17:46:13.747038+0000","last_undegraded":"2024-03-28T17:46:13.747038+0000","last_fullsized":"2024-03-28T17:46:13.747038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:06:42.529826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140627+0000","last_change":"2024-03-28T17:46:12.792360+0000","last_active":"2024-03-28T17:46:14.140627+0000","last_peered":"2024-03-28T17:46:14.140627+0000","last_clean":"2024-03-28T17:46:14.140627+0000","last_became_active":"2024-03-28T17:46:12.791979+0000","last_became_peered":"2024-03-28T17:46:12.791979+0000","last_unstale":"2024-03-28T17:46:14.140627+0000","last_undegraded":"2024-03-28T17:46:14.140627+0000","last_fullsized":"2024-03-28T17:46:14.140627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:39.866969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139611+0000","last_change":"2024-03-28T17:46:12.784228+0000","last_active":"2024-03-28T17:46:14.139611+0000","last_peered":"2024-03-28T17:46:14.139611+0000","last_clean":"2024-03-28T17:46:14.139611+0000","last_became_active":"2024-03-28T17:46:12.783890+0000","last_became_peered":"2024-03-28T17:46:12.783890+0000","last_unstale":"2024-03-28T17:46:14.139611+0000","last_undegraded":"2024-03-28T17:46:14.139611+0000","last_fullsized":"2024-03-28T17:46:14.139611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:58:13.388705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.141830+0000","last_change":"2024-03-28T17:46:12.782675+0000","last_active":"2024-03-28T17:46:14.141830+0000","last_peered":"2024-03-28T17:46:14.141830+0000","last_clean":"2024-03-28T17:46:14.141830+0000","last_became_active":"2024-03-28T17:46:12.779255+0000","last_became_peered":"2024-03-28T17:46:12.779255+0000","last_unstale":"2024-03-28T17:46:14.141830+0000","last_undegraded":"2024-03-28T17:46:14.141830+0000","last_fullsized":"2024-03-28T17:46:14.141830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:04:31.447512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.748959+0000","last_change":"2024-03-28T17:46:12.786308+0000","last_active":"2024-03-28T17:46:13.748959+0000","last_peered":"2024-03-28T17:46:13.748959+0000","last_clean":"2024-03-28T17:46:13.748959+0000","last_became_active":"2024-03-28T17:46:12.785948+0000","last_became_peered":"2024-03-28T17:46:12.785948+0000","last_unstale":"2024-03-28T17:46:13.748959+0000","last_undegraded":"2024-03-28T17:46:13.748959+0000","last_fullsized":"2024-03-28T17:46:13.748959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:43.292925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.746289+0000","last_change":"2024-03-28T17:46:12.797278+0000","last_active":"2024-03-28T17:46:13.746289+0000","last_peered":"2024-03-28T17:46:13.746289+0000","last_clean":"2024-03-28T17:46:13.746289+0000","last_became_active":"2024-03-28T17:46:12.796593+0000","last_became_peered":"2024-03-28T17:46:12.796593+0000","last_unstale":"2024-03-28T17:46:13.746289+0000","last_undegraded":"2024-03-28T17:46:13.746289+0000","last_fullsized":"2024-03-28T17:46:13.746289+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:07:34.433680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140860+0000","last_change":"2024-03-28T17:46:12.764249+0000","last_active":"2024-03-28T17:46:14.140860+0000","last_peered":"2024-03-28T17:46:14.140860+0000","last_clean":"2024-03-28T17:46:14.140860+0000","last_became_active":"2024-03-28T17:46:12.763909+0000","last_became_peered":"2024-03-28T17:46:12.763909+0000","last_unstale":"2024-03-28T17:46:14.140860+0000","last_undegraded":"2024-03-28T17:46:14.140860+0000","last_fullsized":"2024-03-28T17:46:14.140860+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:40:51.780187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250098,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6004,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":26,"num_read_kb":22,"num_write":129,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":74920,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383696,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":439103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645332,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":1031457,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808859,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":439339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070496,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":439259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332133,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":1031580,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495659,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":438611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574920,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":478149,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738446,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":439999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411395,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":461157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182460,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":438805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920823,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":925696,"data_stored":439556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":1030756,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34554,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34554,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":39628,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":22636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T17:46:22.044 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T17:46:22.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.303+0000 7ff781a00640 1 -- 172.21.15.45:0/3801495455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff77c107b00 msgr2=0x7ff77c107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:22.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.303+0000 7ff781a00640 1 --2- 172.21.15.45:0/3801495455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff77c107b00 0x7ff77c107f20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff76c009900 tx=0x7ff76c02bc30 comp rx=0 tx=0).stop 2024-03-28T17:46:22.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.303+0000 7ff781a00640 1 -- 172.21.15.45:0/3801495455 shutdown_connections 2024-03-28T17:46:22.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.303+0000 7ff781a00640 1 --2- 172.21.15.45:0/3801495455 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff77c1096b0 0x7ff77c10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:22.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.303+0000 7ff781a00640 1 --2- 172.21.15.45:0/3801495455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff77c108cf0 0x7ff77c109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:22.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.303+0000 7ff781a00640 1 --2- 172.21.15.45:0/3801495455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff77c107b00 0x7ff77c107f20 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:22.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.303+0000 7ff781a00640 1 -- 172.21.15.45:0/3801495455 >> 172.21.15.45:0/3801495455 conn(0x7ff77c076ab0 msgr2=0x7ff77c078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:22.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.303+0000 7ff781a00640 1 -- 172.21.15.45:0/3801495455 shutdown_connections 2024-03-28T17:46:22.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.303+0000 7ff781a00640 1 -- 172.21.15.45:0/3801495455 wait complete. 2024-03-28T17:46:22.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.304+0000 7ff781a00640 1 Processor -- start 2024-03-28T17:46:22.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.304+0000 7ff781a00640 1 -- start start 2024-03-28T17:46:22.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.305+0000 7ff781a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff77c107b00 0x7ff77c19ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:22.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.305+0000 7ff781a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff77c108cf0 0x7ff77c19e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:22.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.305+0000 7ff781a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff77c1096b0 0x7ff77c1a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:22.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.305+0000 7ff781a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c1137a0 con 0x7ff77c108cf0 2024-03-28T17:46:22.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.305+0000 7ff781a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c113620 con 0x7ff77c1096b0 2024-03-28T17:46:22.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.305+0000 7ff781a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c113920 con 0x7ff77c107b00 2024-03-28T17:46:22.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.305+0000 7ff77be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff77c107b00 0x7ff77c19ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:22.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.305+0000 7ff781000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff77c1096b0 0x7ff77c1a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:22.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.305+0000 7ff773e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff77c108cf0 0x7ff77c19e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:22.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.305+0000 7ff77be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff77c107b00 0x7ff77c19ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:50080/0 (socket says 172.21.15.45:50080) 2024-03-28T17:46:22.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.305+0000 7ff77be00640 1 -- 172.21.15.45:0/1640998784 learned_addr learned my addr 172.21.15.45:0/1640998784 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:22.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.305+0000 7ff781000640 1 -- 172.21.15.45:0/1640998784 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff77c107b00 msgr2=0x7ff77c19ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:22.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.305+0000 7ff781000640 1 --2- 172.21.15.45:0/1640998784 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff77c107b00 0x7ff77c19ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:22.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.305+0000 7ff781000640 1 -- 172.21.15.45:0/1640998784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff77c108cf0 msgr2=0x7ff77c19e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:22.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.305+0000 7ff781000640 1 --2- 172.21.15.45:0/1640998784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff77c108cf0 0x7ff77c19e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:22.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.305+0000 7ff781000640 1 -- 172.21.15.45:0/1640998784 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77c1a5a10 con 0x7ff77c1096b0 2024-03-28T17:46:22.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.306+0000 7ff781000640 1 --2- 172.21.15.45:0/1640998784 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff77c1096b0 0x7ff77c1a53a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff77400e940 tx=0x7ff77400ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:22.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.306+0000 7ff779600640 1 -- 172.21.15.45:0/1640998784 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff774004d20 con 0x7ff77c1096b0 2024-03-28T17:46:22.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.306+0000 7ff779600640 1 -- 172.21.15.45:0/1640998784 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff774005020 con 0x7ff77c1096b0 2024-03-28T17:46:22.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.306+0000 7ff779600640 1 -- 172.21.15.45:0/1640998784 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7740116b0 con 0x7ff77c1096b0 2024-03-28T17:46:22.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.306+0000 7ff781a00640 1 -- 172.21.15.45:0/1640998784 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff77c1a5d00 con 0x7ff77c1096b0 2024-03-28T17:46:22.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.306+0000 7ff781a00640 1 -- 172.21.15.45:0/1640998784 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff77c1a61a0 con 0x7ff77c1096b0 2024-03-28T17:46:22.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.307+0000 7ff781a00640 1 -- 172.21.15.45:0/1640998784 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff77c10ee20 con 0x7ff77c1096b0 2024-03-28T17:46:22.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.309+0000 7ff779600640 1 -- 172.21.15.45:0/1640998784 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7ff7740054d0 con 0x7ff77c1096b0 2024-03-28T17:46:22.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.310+0000 7ff779600640 1 --2- 172.21.15.45:0/1640998784 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff7600a2910 0x7ff7600a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:22.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.310+0000 7ff779600640 1 -- 172.21.15.45:0/1640998784 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff7740ceaf0 con 0x7ff77c1096b0 2024-03-28T17:46:22.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.310+0000 7ff77be00640 1 --2- 172.21.15.45:0/1640998784 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff7600a2910 0x7ff7600a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:22.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.311+0000 7ff77be00640 1 --2- 172.21.15.45:0/1640998784 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff7600a2910 0x7ff7600a4dd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff76c006010 tx=0x7ff76c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:22.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.314+0000 7ff779600640 1 -- 172.21.15.45:0/1640998784 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff774016030 con 0x7ff77c1096b0 2024-03-28T17:46:22.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.515+0000 7ff781a00640 1 -- 172.21.15.45:0/1640998784 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff77c061ae0 con 0x7ff7600a2910 2024-03-28T17:46:22.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.525+0000 7ff779600640 1 -- 172.21.15.45:0/1640998784 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258895 (secure 0 0 0) 0x7ff77c061ae0 con 0x7ff7600a2910 2024-03-28T17:46:22.529 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:46:22.530 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-28T17:46:22.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.531+0000 7ff781a00640 1 -- 172.21.15.45:0/1640998784 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff7600a2910 msgr2=0x7ff7600a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:22.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.531+0000 7ff781a00640 1 --2- 172.21.15.45:0/1640998784 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff7600a2910 0x7ff7600a4dd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff76c006010 tx=0x7ff76c02df90 comp rx=0 tx=0).stop 2024-03-28T17:46:22.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.531+0000 7ff781a00640 1 -- 172.21.15.45:0/1640998784 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff77c1096b0 msgr2=0x7ff77c1a53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:22.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.531+0000 7ff781a00640 1 --2- 172.21.15.45:0/1640998784 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff77c1096b0 0x7ff77c1a53a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff77400e940 tx=0x7ff77400ee00 comp rx=0 tx=0).stop 2024-03-28T17:46:22.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.531+0000 7ff781a00640 1 -- 172.21.15.45:0/1640998784 shutdown_connections 2024-03-28T17:46:22.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.531+0000 7ff781a00640 1 --2- 172.21.15.45:0/1640998784 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff7600a2910 0x7ff7600a4dd0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:22.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.531+0000 7ff781a00640 1 --2- 172.21.15.45:0/1640998784 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff77c1096b0 0x7ff77c1a53a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:22.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.531+0000 7ff781a00640 1 --2- 172.21.15.45:0/1640998784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff77c108cf0 0x7ff77c19e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:22.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.531+0000 7ff781a00640 1 --2- 172.21.15.45:0/1640998784 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff77c107b00 0x7ff77c19ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:22.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.531+0000 7ff781a00640 1 -- 172.21.15.45:0/1640998784 >> 172.21.15.45:0/1640998784 conn(0x7ff77c076ab0 msgr2=0x7ff77c10bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:22.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.531+0000 7ff781a00640 1 -- 172.21.15.45:0/1640998784 shutdown_connections 2024-03-28T17:46:22.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:22.531+0000 7ff781a00640 1 -- 172.21.15.45:0/1640998784 wait complete. 2024-03-28T17:46:22.551 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-03-28T17:46:21.647753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596489,"num_objects":63,"num_object_clones":0,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":162,"num_read_kb":141,"num_write":311,"num_write_kb":2980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":250,"ondisk_log_size":250,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360284,"kb_used_data":11132,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581540,"statfs":{"total":1151940427776,"available":1151571496960,"internally_reserved":0,"allocated":11399168,"data_stored":7107771,"data_compressed":110469,"data_compressed_allocated":139264,"data_compressed_original":1777664,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6121,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":26,"num_read_kb":22,"num_write":130,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.904669"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140522+0000","last_change":"2024-03-28T17:46:12.783038+0000","last_active":"2024-03-28T17:46:14.140522+0000","last_peered":"2024-03-28T17:46:14.140522+0000","last_clean":"2024-03-28T17:46:14.140522+0000","last_became_active":"2024-03-28T17:46:12.782668+0000","last_became_peered":"2024-03-28T17:46:12.782668+0000","last_unstale":"2024-03-28T17:46:14.140522+0000","last_undegraded":"2024-03-28T17:46:14.140522+0000","last_fullsized":"2024-03-28T17:46:14.140522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:03.424959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.143291+0000","last_change":"2024-03-28T17:46:12.781828+0000","last_active":"2024-03-28T17:46:14.143291+0000","last_peered":"2024-03-28T17:46:14.143291+0000","last_clean":"2024-03-28T17:46:14.143291+0000","last_became_active":"2024-03-28T17:46:12.781424+0000","last_became_peered":"2024-03-28T17:46:12.781424+0000","last_unstale":"2024-03-28T17:46:14.143291+0000","last_undegraded":"2024-03-28T17:46:14.143291+0000","last_fullsized":"2024-03-28T17:46:14.143291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:15:16.639473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.142731+0000","last_change":"2024-03-28T17:46:12.785431+0000","last_active":"2024-03-28T17:46:14.142731+0000","last_peered":"2024-03-28T17:46:14.142731+0000","last_clean":"2024-03-28T17:46:14.142731+0000","last_became_active":"2024-03-28T17:46:12.784992+0000","last_became_peered":"2024-03-28T17:46:12.784992+0000","last_unstale":"2024-03-28T17:46:14.142731+0000","last_undegraded":"2024-03-28T17:46:14.142731+0000","last_fullsized":"2024-03-28T17:46:14.142731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:03:36.926613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140143+0000","last_change":"2024-03-28T17:46:12.765274+0000","last_active":"2024-03-28T17:46:14.140143+0000","last_peered":"2024-03-28T17:46:14.140143+0000","last_clean":"2024-03-28T17:46:14.140143+0000","last_became_active":"2024-03-28T17:46:12.764890+0000","last_became_peered":"2024-03-28T17:46:12.764890+0000","last_unstale":"2024-03-28T17:46:14.140143+0000","last_undegraded":"2024-03-28T17:46:14.140143+0000","last_fullsized":"2024-03-28T17:46:14.140143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:58:37.149756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139637+0000","last_change":"2024-03-28T17:46:12.782791+0000","last_active":"2024-03-28T17:46:14.139637+0000","last_peered":"2024-03-28T17:46:14.139637+0000","last_clean":"2024-03-28T17:46:14.139637+0000","last_became_active":"2024-03-28T17:46:12.782420+0000","last_became_peered":"2024-03-28T17:46:12.782420+0000","last_unstale":"2024-03-28T17:46:14.139637+0000","last_undegraded":"2024-03-28T17:46:14.139637+0000","last_fullsized":"2024-03-28T17:46:14.139637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:30.512863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140112+0000","last_change":"2024-03-28T17:46:12.780713+0000","last_active":"2024-03-28T17:46:14.140112+0000","last_peered":"2024-03-28T17:46:14.140112+0000","last_clean":"2024-03-28T17:46:14.140112+0000","last_became_active":"2024-03-28T17:46:12.780305+0000","last_became_peered":"2024-03-28T17:46:12.780305+0000","last_unstale":"2024-03-28T17:46:14.140112+0000","last_undegraded":"2024-03-28T17:46:14.140112+0000","last_fullsized":"2024-03-28T17:46:14.140112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:08:09.840491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139743+0000","last_change":"2024-03-28T17:46:12.778120+0000","last_active":"2024-03-28T17:46:14.139743+0000","last_peered":"2024-03-28T17:46:14.139743+0000","last_clean":"2024-03-28T17:46:14.139743+0000","last_became_active":"2024-03-28T17:46:12.777724+0000","last_became_peered":"2024-03-28T17:46:12.777724+0000","last_unstale":"2024-03-28T17:46:14.139743+0000","last_undegraded":"2024-03-28T17:46:14.139743+0000","last_fullsized":"2024-03-28T17:46:14.139743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:29:50.980366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140014+0000","last_change":"2024-03-28T17:46:12.793447+0000","last_active":"2024-03-28T17:46:14.140014+0000","last_peered":"2024-03-28T17:46:14.140014+0000","last_clean":"2024-03-28T17:46:14.140014+0000","last_became_active":"2024-03-28T17:46:12.793070+0000","last_became_peered":"2024-03-28T17:46:12.793070+0000","last_unstale":"2024-03-28T17:46:14.140014+0000","last_undegraded":"2024-03-28T17:46:14.140014+0000","last_fullsized":"2024-03-28T17:46:14.140014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:01.364577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.749303+0000","last_change":"2024-03-28T17:46:12.763908+0000","last_active":"2024-03-28T17:46:13.749303+0000","last_peered":"2024-03-28T17:46:13.749303+0000","last_clean":"2024-03-28T17:46:13.749303+0000","last_became_active":"2024-03-28T17:46:12.763267+0000","last_became_peered":"2024-03-28T17:46:12.763267+0000","last_unstale":"2024-03-28T17:46:13.749303+0000","last_undegraded":"2024-03-28T17:46:13.749303+0000","last_fullsized":"2024-03-28T17:46:13.749303+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:10.692974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140573+0000","last_change":"2024-03-28T17:46:12.762648+0000","last_active":"2024-03-28T17:46:14.140573+0000","last_peered":"2024-03-28T17:46:14.140573+0000","last_clean":"2024-03-28T17:46:14.140573+0000","last_became_active":"2024-03-28T17:46:12.762300+0000","last_became_peered":"2024-03-28T17:46:12.762300+0000","last_unstale":"2024-03-28T17:46:14.140573+0000","last_undegraded":"2024-03-28T17:46:14.140573+0000","last_fullsized":"2024-03-28T17:46:14.140573+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:02.210828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140429+0000","last_change":"2024-03-28T17:46:12.766482+0000","last_active":"2024-03-28T17:46:14.140429+0000","last_peered":"2024-03-28T17:46:14.140429+0000","last_clean":"2024-03-28T17:46:14.140429+0000","last_became_active":"2024-03-28T17:46:12.766155+0000","last_became_peered":"2024-03-28T17:46:12.766155+0000","last_unstale":"2024-03-28T17:46:14.140429+0000","last_undegraded":"2024-03-28T17:46:14.140429+0000","last_fullsized":"2024-03-28T17:46:14.140429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:46:24.261744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139882+0000","last_change":"2024-03-28T17:46:12.766100+0000","last_active":"2024-03-28T17:46:14.139882+0000","last_peered":"2024-03-28T17:46:14.139882+0000","last_clean":"2024-03-28T17:46:14.139882+0000","last_became_active":"2024-03-28T17:46:12.765762+0000","last_became_peered":"2024-03-28T17:46:12.765762+0000","last_unstale":"2024-03-28T17:46:14.139882+0000","last_undegraded":"2024-03-28T17:46:14.139882+0000","last_fullsized":"2024-03-28T17:46:14.139882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:11.666653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.746273+0000","last_change":"2024-03-28T17:46:12.753465+0000","last_active":"2024-03-28T17:46:13.746273+0000","last_peered":"2024-03-28T17:46:13.746273+0000","last_clean":"2024-03-28T17:46:13.746273+0000","last_became_active":"2024-03-28T17:46:12.751931+0000","last_became_peered":"2024-03-28T17:46:12.751931+0000","last_unstale":"2024-03-28T17:46:13.746273+0000","last_undegraded":"2024-03-28T17:46:13.746273+0000","last_fullsized":"2024-03-28T17:46:13.746273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:39:38.757878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.142180+0000","last_change":"2024-03-28T17:46:12.784177+0000","last_active":"2024-03-28T17:46:14.142180+0000","last_peered":"2024-03-28T17:46:14.142180+0000","last_clean":"2024-03-28T17:46:14.142180+0000","last_became_active":"2024-03-28T17:46:12.783823+0000","last_became_peered":"2024-03-28T17:46:12.783823+0000","last_unstale":"2024-03-28T17:46:14.142180+0000","last_undegraded":"2024-03-28T17:46:14.142180+0000","last_fullsized":"2024-03-28T17:46:14.142180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:11:33.221095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.143014+0000","last_change":"2024-03-28T17:46:12.777152+0000","last_active":"2024-03-28T17:46:14.143014+0000","last_peered":"2024-03-28T17:46:14.143014+0000","last_clean":"2024-03-28T17:46:14.143014+0000","last_became_active":"2024-03-28T17:46:12.776758+0000","last_became_peered":"2024-03-28T17:46:12.776758+0000","last_unstale":"2024-03-28T17:46:14.143014+0000","last_undegraded":"2024-03-28T17:46:14.143014+0000","last_fullsized":"2024-03-28T17:46:14.143014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:20:56.587470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.748726+0000","last_change":"2024-03-28T17:46:12.782384+0000","last_active":"2024-03-28T17:46:13.748726+0000","last_peered":"2024-03-28T17:46:13.748726+0000","last_clean":"2024-03-28T17:46:13.748726+0000","last_became_active":"2024-03-28T17:46:12.781971+0000","last_became_peered":"2024-03-28T17:46:12.781971+0000","last_unstale":"2024-03-28T17:46:13.748726+0000","last_undegraded":"2024-03-28T17:46:13.748726+0000","last_fullsized":"2024-03-28T17:46:13.748726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:19:32.652629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140636+0000","last_change":"2024-03-28T17:46:12.789364+0000","last_active":"2024-03-28T17:46:14.140636+0000","last_peered":"2024-03-28T17:46:14.140636+0000","last_clean":"2024-03-28T17:46:14.140636+0000","last_became_active":"2024-03-28T17:46:12.788976+0000","last_became_peered":"2024-03-28T17:46:12.788976+0000","last_unstale":"2024-03-28T17:46:14.140636+0000","last_undegraded":"2024-03-28T17:46:14.140636+0000","last_fullsized":"2024-03-28T17:46:14.140636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:43:37.183705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.143756+0000","last_change":"2024-03-28T17:46:12.782664+0000","last_active":"2024-03-28T17:46:14.143756+0000","last_peered":"2024-03-28T17:46:14.143756+0000","last_clean":"2024-03-28T17:46:14.143756+0000","last_became_active":"2024-03-28T17:46:12.782305+0000","last_became_peered":"2024-03-28T17:46:12.782305+0000","last_unstale":"2024-03-28T17:46:14.143756+0000","last_undegraded":"2024-03-28T17:46:14.143756+0000","last_fullsized":"2024-03-28T17:46:14.143756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:47:18.373517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.746900+0000","last_change":"2024-03-28T17:46:12.794494+0000","last_active":"2024-03-28T17:46:13.746900+0000","last_peered":"2024-03-28T17:46:13.746900+0000","last_clean":"2024-03-28T17:46:13.746900+0000","last_became_active":"2024-03-28T17:46:12.794151+0000","last_became_peered":"2024-03-28T17:46:12.794151+0000","last_unstale":"2024-03-28T17:46:13.746900+0000","last_undegraded":"2024-03-28T17:46:13.746900+0000","last_fullsized":"2024-03-28T17:46:13.746900+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:17:24.596333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:16.975617+0000","last_change":"2024-03-28T17:46:11.759643+0000","last_active":"2024-03-28T17:46:16.975617+0000","last_peered":"2024-03-28T17:46:16.975617+0000","last_clean":"2024-03-28T17:46:16.975617+0000","last_became_active":"2024-03-28T17:46:11.759273+0000","last_became_peered":"2024-03-28T17:46:11.759273+0000","last_unstale":"2024-03-28T17:46:16.975617+0000","last_undegraded":"2024-03-28T17:46:16.975617+0000","last_fullsized":"2024-03-28T17:46:16.975617+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:09:51.764471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.141043+0000","last_change":"2024-03-28T17:46:12.786503+0000","last_active":"2024-03-28T17:46:14.141043+0000","last_peered":"2024-03-28T17:46:14.141043+0000","last_clean":"2024-03-28T17:46:14.141043+0000","last_became_active":"2024-03-28T17:46:12.784651+0000","last_became_peered":"2024-03-28T17:46:12.784651+0000","last_unstale":"2024-03-28T17:46:14.141043+0000","last_undegraded":"2024-03-28T17:46:14.141043+0000","last_fullsized":"2024-03-28T17:46:14.141043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:30:22.401767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.141439+0000","last_change":"2024-03-28T17:46:11.780340+0000","last_active":"2024-03-28T17:46:14.141439+0000","last_peered":"2024-03-28T17:46:14.141439+0000","last_clean":"2024-03-28T17:46:14.141439+0000","last_became_active":"2024-03-28T17:46:11.779917+0000","last_became_peered":"2024-03-28T17:46:11.779917+0000","last_unstale":"2024-03-28T17:46:14.141439+0000","last_undegraded":"2024-03-28T17:46:14.141439+0000","last_fullsized":"2024-03-28T17:46:14.141439+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:22:45.617444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.142365+0000","last_change":"2024-03-28T17:46:12.795086+0000","last_active":"2024-03-28T17:46:14.142365+0000","last_peered":"2024-03-28T17:46:14.142365+0000","last_clean":"2024-03-28T17:46:14.142365+0000","last_became_active":"2024-03-28T17:46:12.794124+0000","last_became_peered":"2024-03-28T17:46:12.794124+0000","last_unstale":"2024-03-28T17:46:14.142365+0000","last_undegraded":"2024-03-28T17:46:14.142365+0000","last_fullsized":"2024-03-28T17:46:14.142365+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:11:05.347047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:16.970109+0000","last_change":"2024-03-28T17:46:11.766541+0000","last_active":"2024-03-28T17:46:16.970109+0000","last_peered":"2024-03-28T17:46:16.970109+0000","last_clean":"2024-03-28T17:46:16.970109+0000","last_became_active":"2024-03-28T17:46:11.766230+0000","last_became_peered":"2024-03-28T17:46:11.766230+0000","last_unstale":"2024-03-28T17:46:16.970109+0000","last_undegraded":"2024-03-28T17:46:16.970109+0000","last_fullsized":"2024-03-28T17:46:16.970109+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:05:23.657666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.748968+0000","last_change":"2024-03-28T17:46:12.768118+0000","last_active":"2024-03-28T17:46:13.748968+0000","last_peered":"2024-03-28T17:46:13.748968+0000","last_clean":"2024-03-28T17:46:13.748968+0000","last_became_active":"2024-03-28T17:46:12.767765+0000","last_became_peered":"2024-03-28T17:46:12.767765+0000","last_unstale":"2024-03-28T17:46:13.748968+0000","last_undegraded":"2024-03-28T17:46:13.748968+0000","last_fullsized":"2024-03-28T17:46:13.748968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:16:12.268968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140881+0000","last_change":"2024-03-28T17:46:11.760204+0000","last_active":"2024-03-28T17:46:14.140881+0000","last_peered":"2024-03-28T17:46:14.140881+0000","last_clean":"2024-03-28T17:46:14.140881+0000","last_became_active":"2024-03-28T17:46:11.759829+0000","last_became_peered":"2024-03-28T17:46:11.759829+0000","last_unstale":"2024-03-28T17:46:14.140881+0000","last_undegraded":"2024-03-28T17:46:14.140881+0000","last_fullsized":"2024-03-28T17:46:14.140881+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:29:02.075424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140896+0000","last_change":"2024-03-28T17:46:12.762229+0000","last_active":"2024-03-28T17:46:14.140896+0000","last_peered":"2024-03-28T17:46:14.140896+0000","last_clean":"2024-03-28T17:46:14.140896+0000","last_became_active":"2024-03-28T17:46:12.761889+0000","last_became_peered":"2024-03-28T17:46:12.761889+0000","last_unstale":"2024-03-28T17:46:14.140896+0000","last_undegraded":"2024-03-28T17:46:14.140896+0000","last_fullsized":"2024-03-28T17:46:14.140896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:15.023966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:18.978434+0000","last_change":"2024-03-28T17:46:11.763152+0000","last_active":"2024-03-28T17:46:18.978434+0000","last_peered":"2024-03-28T17:46:18.978434+0000","last_clean":"2024-03-28T17:46:18.978434+0000","last_became_active":"2024-03-28T17:46:11.762754+0000","last_became_peered":"2024-03-28T17:46:11.762754+0000","last_unstale":"2024-03-28T17:46:18.978434+0000","last_undegraded":"2024-03-28T17:46:18.978434+0000","last_fullsized":"2024-03-28T17:46:18.978434+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:17:06.659908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139414+0000","last_change":"2024-03-28T17:46:12.780701+0000","last_active":"2024-03-28T17:46:14.139414+0000","last_peered":"2024-03-28T17:46:14.139414+0000","last_clean":"2024-03-28T17:46:14.139414+0000","last_became_active":"2024-03-28T17:46:12.780339+0000","last_became_peered":"2024-03-28T17:46:12.780339+0000","last_unstale":"2024-03-28T17:46:14.139414+0000","last_undegraded":"2024-03-28T17:46:14.139414+0000","last_fullsized":"2024-03-28T17:46:14.139414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:59.774888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:16.976152+0000","last_change":"2024-03-28T17:46:11.763387+0000","last_active":"2024-03-28T17:46:16.976152+0000","last_peered":"2024-03-28T17:46:16.976152+0000","last_clean":"2024-03-28T17:46:16.976152+0000","last_became_active":"2024-03-28T17:46:11.763028+0000","last_became_peered":"2024-03-28T17:46:11.763028+0000","last_unstale":"2024-03-28T17:46:16.976152+0000","last_undegraded":"2024-03-28T17:46:16.976152+0000","last_fullsized":"2024-03-28T17:46:16.976152+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:21:31.585047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140361+0000","last_change":"2024-03-28T17:46:12.794167+0000","last_active":"2024-03-28T17:46:14.140361+0000","last_peered":"2024-03-28T17:46:14.140361+0000","last_clean":"2024-03-28T17:46:14.140361+0000","last_became_active":"2024-03-28T17:46:12.793830+0000","last_became_peered":"2024-03-28T17:46:12.793830+0000","last_unstale":"2024-03-28T17:46:14.140361+0000","last_undegraded":"2024-03-28T17:46:14.140361+0000","last_fullsized":"2024-03-28T17:46:14.140361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:20.907752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:16.978907+0000","last_change":"2024-03-28T17:46:11.769322+0000","last_active":"2024-03-28T17:46:16.978907+0000","last_peered":"2024-03-28T17:46:16.978907+0000","last_clean":"2024-03-28T17:46:16.978907+0000","last_became_active":"2024-03-28T17:46:11.768931+0000","last_became_peered":"2024-03-28T17:46:11.768931+0000","last_unstale":"2024-03-28T17:46:16.978907+0000","last_undegraded":"2024-03-28T17:46:16.978907+0000","last_fullsized":"2024-03-28T17:46:16.978907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:29:34.372947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139977+0000","last_change":"2024-03-28T17:46:12.781306+0000","last_active":"2024-03-28T17:46:14.139977+0000","last_peered":"2024-03-28T17:46:14.139977+0000","last_clean":"2024-03-28T17:46:14.139977+0000","last_became_active":"2024-03-28T17:46:12.779161+0000","last_became_peered":"2024-03-28T17:46:12.779161+0000","last_unstale":"2024-03-28T17:46:14.139977+0000","last_undegraded":"2024-03-28T17:46:14.139977+0000","last_fullsized":"2024-03-28T17:46:14.139977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:06:10.302067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139462+0000","last_change":"2024-03-28T17:46:11.762852+0000","last_active":"2024-03-28T17:46:14.139462+0000","last_peered":"2024-03-28T17:46:14.139462+0000","last_clean":"2024-03-28T17:46:14.139462+0000","last_became_active":"2024-03-28T17:46:11.762428+0000","last_became_peered":"2024-03-28T17:46:11.762428+0000","last_unstale":"2024-03-28T17:46:14.139462+0000","last_undegraded":"2024-03-28T17:46:14.139462+0000","last_fullsized":"2024-03-28T17:46:14.139462+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:38:20.433408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.143193+0000","last_change":"2024-03-28T17:46:12.786295+0000","last_active":"2024-03-28T17:46:14.143193+0000","last_peered":"2024-03-28T17:46:14.143193+0000","last_clean":"2024-03-28T17:46:14.143193+0000","last_became_active":"2024-03-28T17:46:12.785894+0000","last_became_peered":"2024-03-28T17:46:12.785894+0000","last_unstale":"2024-03-28T17:46:14.143193+0000","last_undegraded":"2024-03-28T17:46:14.143193+0000","last_fullsized":"2024-03-28T17:46:14.143193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:06.271660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.945393+0000","last_change":"2024-03-28T17:46:11.766787+0000","last_active":"2024-03-28T17:46:15.945393+0000","last_peered":"2024-03-28T17:46:15.945393+0000","last_clean":"2024-03-28T17:46:15.945393+0000","last_became_active":"2024-03-28T17:46:11.766352+0000","last_became_peered":"2024-03-28T17:46:11.766352+0000","last_unstale":"2024-03-28T17:46:15.945393+0000","last_undegraded":"2024-03-28T17:46:15.945393+0000","last_fullsized":"2024-03-28T17:46:15.945393+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:54:48.157597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.749226+0000","last_change":"2024-03-28T17:46:12.785724+0000","last_active":"2024-03-28T17:46:13.749226+0000","last_peered":"2024-03-28T17:46:13.749226+0000","last_clean":"2024-03-28T17:46:13.749226+0000","last_became_active":"2024-03-28T17:46:12.785353+0000","last_became_peered":"2024-03-28T17:46:12.785353+0000","last_unstale":"2024-03-28T17:46:13.749226+0000","last_undegraded":"2024-03-28T17:46:13.749226+0000","last_fullsized":"2024-03-28T17:46:13.749226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:21:22.692899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.936732+0000","last_change":"2024-03-28T17:46:11.763507+0000","last_active":"2024-03-28T17:46:15.936732+0000","last_peered":"2024-03-28T17:46:15.936732+0000","last_clean":"2024-03-28T17:46:15.936732+0000","last_became_active":"2024-03-28T17:46:11.763177+0000","last_became_peered":"2024-03-28T17:46:11.763177+0000","last_unstale":"2024-03-28T17:46:15.936732+0000","last_undegraded":"2024-03-28T17:46:15.936732+0000","last_fullsized":"2024-03-28T17:46:15.936732+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:56:39.696699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.142583+0000","last_change":"2024-03-28T17:46:12.793693+0000","last_active":"2024-03-28T17:46:14.142583+0000","last_peered":"2024-03-28T17:46:14.142583+0000","last_clean":"2024-03-28T17:46:14.142583+0000","last_became_active":"2024-03-28T17:46:12.792981+0000","last_became_peered":"2024-03-28T17:46:12.792981+0000","last_unstale":"2024-03-28T17:46:14.142583+0000","last_undegraded":"2024-03-28T17:46:14.142583+0000","last_fullsized":"2024-03-28T17:46:14.142583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:33:50.727663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.942364+0000","last_change":"2024-03-28T17:46:11.774961+0000","last_active":"2024-03-28T17:46:15.942364+0000","last_peered":"2024-03-28T17:46:15.942364+0000","last_clean":"2024-03-28T17:46:15.942364+0000","last_became_active":"2024-03-28T17:46:11.774645+0000","last_became_peered":"2024-03-28T17:46:11.774645+0000","last_unstale":"2024-03-28T17:46:15.942364+0000","last_undegraded":"2024-03-28T17:46:15.942364+0000","last_fullsized":"2024-03-28T17:46:15.942364+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:38:23.983739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.749179+0000","last_change":"2024-03-28T17:46:12.763698+0000","last_active":"2024-03-28T17:46:13.749179+0000","last_peered":"2024-03-28T17:46:13.749179+0000","last_clean":"2024-03-28T17:46:13.749179+0000","last_became_active":"2024-03-28T17:46:12.763238+0000","last_became_peered":"2024-03-28T17:46:12.763238+0000","last_unstale":"2024-03-28T17:46:13.749179+0000","last_undegraded":"2024-03-28T17:46:13.749179+0000","last_fullsized":"2024-03-28T17:46:13.749179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:12:58.252855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'8","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.938002+0000","last_change":"2024-03-28T17:46:11.763096+0000","last_active":"2024-03-28T17:46:15.938002+0000","last_peered":"2024-03-28T17:46:15.938002+0000","last_clean":"2024-03-28T17:46:15.938002+0000","last_became_active":"2024-03-28T17:46:11.762730+0000","last_became_peered":"2024-03-28T17:46:11.762730+0000","last_unstale":"2024-03-28T17:46:15.938002+0000","last_undegraded":"2024-03-28T17:46:15.938002+0000","last_fullsized":"2024-03-28T17:46:15.938002+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:45:59.469096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140275+0000","last_change":"2024-03-28T17:46:12.760366+0000","last_active":"2024-03-28T17:46:14.140275+0000","last_peered":"2024-03-28T17:46:14.140275+0000","last_clean":"2024-03-28T17:46:14.140275+0000","last_became_active":"2024-03-28T17:46:12.759957+0000","last_became_peered":"2024-03-28T17:46:12.759957+0000","last_unstale":"2024-03-28T17:46:14.140275+0000","last_undegraded":"2024-03-28T17:46:14.140275+0000","last_fullsized":"2024-03-28T17:46:14.140275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:08:02.579659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:16.971643+0000","last_change":"2024-03-28T17:46:11.787108+0000","last_active":"2024-03-28T17:46:16.971643+0000","last_peered":"2024-03-28T17:46:16.971643+0000","last_clean":"2024-03-28T17:46:16.971643+0000","last_became_active":"2024-03-28T17:46:11.786734+0000","last_became_peered":"2024-03-28T17:46:11.786734+0000","last_unstale":"2024-03-28T17:46:16.971643+0000","last_undegraded":"2024-03-28T17:46:16.971643+0000","last_fullsized":"2024-03-28T17:46:16.971643+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:04:00.480545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.745847+0000","last_change":"2024-03-28T17:46:12.786362+0000","last_active":"2024-03-28T17:46:13.745847+0000","last_peered":"2024-03-28T17:46:13.745847+0000","last_clean":"2024-03-28T17:46:13.745847+0000","last_became_active":"2024-03-28T17:46:12.785946+0000","last_became_peered":"2024-03-28T17:46:12.785946+0000","last_unstale":"2024-03-28T17:46:13.745847+0000","last_undegraded":"2024-03-28T17:46:13.745847+0000","last_fullsized":"2024-03-28T17:46:13.745847+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:13:57.857818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.940782+0000","last_change":"2024-03-28T17:46:11.786292+0000","last_active":"2024-03-28T17:46:15.940782+0000","last_peered":"2024-03-28T17:46:15.940782+0000","last_clean":"2024-03-28T17:46:15.940782+0000","last_became_active":"2024-03-28T17:46:11.785953+0000","last_became_peered":"2024-03-28T17:46:11.785953+0000","last_unstale":"2024-03-28T17:46:15.940782+0000","last_undegraded":"2024-03-28T17:46:15.940782+0000","last_fullsized":"2024-03-28T17:46:15.940782+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:02:18.093585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140190+0000","last_change":"2024-03-28T17:46:12.770669+0000","last_active":"2024-03-28T17:46:14.140190+0000","last_peered":"2024-03-28T17:46:14.140190+0000","last_clean":"2024-03-28T17:46:14.140190+0000","last_became_active":"2024-03-28T17:46:12.770301+0000","last_became_peered":"2024-03-28T17:46:12.770301+0000","last_unstale":"2024-03-28T17:46:14.140190+0000","last_undegraded":"2024-03-28T17:46:14.140190+0000","last_fullsized":"2024-03-28T17:46:14.140190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:59:00.601357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'4","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:18.982721+0000","last_change":"2024-03-28T17:46:11.759686+0000","last_active":"2024-03-28T17:46:18.982721+0000","last_peered":"2024-03-28T17:46:18.982721+0000","last_clean":"2024-03-28T17:46:18.982721+0000","last_became_active":"2024-03-28T17:46:11.759309+0000","last_became_peered":"2024-03-28T17:46:11.759309+0000","last_unstale":"2024-03-28T17:46:18.982721+0000","last_undegraded":"2024-03-28T17:46:18.982721+0000","last_fullsized":"2024-03-28T17:46:18.982721+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:31:09.999279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139507+0000","last_change":"2024-03-28T17:46:12.760484+0000","last_active":"2024-03-28T17:46:14.139507+0000","last_peered":"2024-03-28T17:46:14.139507+0000","last_clean":"2024-03-28T17:46:14.139507+0000","last_became_active":"2024-03-28T17:46:12.760121+0000","last_became_peered":"2024-03-28T17:46:12.760121+0000","last_unstale":"2024-03-28T17:46:14.139507+0000","last_undegraded":"2024-03-28T17:46:14.139507+0000","last_fullsized":"2024-03-28T17:46:14.139507+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:33:51.457426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:16.979825+0000","last_change":"2024-03-28T17:46:11.769133+0000","last_active":"2024-03-28T17:46:16.979825+0000","last_peered":"2024-03-28T17:46:16.979825+0000","last_clean":"2024-03-28T17:46:16.979825+0000","last_became_active":"2024-03-28T17:46:11.768857+0000","last_became_peered":"2024-03-28T17:46:11.768857+0000","last_unstale":"2024-03-28T17:46:16.979825+0000","last_undegraded":"2024-03-28T17:46:16.979825+0000","last_fullsized":"2024-03-28T17:46:16.979825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:19:26.065368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139541+0000","last_change":"2024-03-28T17:46:12.765246+0000","last_active":"2024-03-28T17:46:14.139541+0000","last_peered":"2024-03-28T17:46:14.139541+0000","last_clean":"2024-03-28T17:46:14.139541+0000","last_became_active":"2024-03-28T17:46:12.764788+0000","last_became_peered":"2024-03-28T17:46:12.764788+0000","last_unstale":"2024-03-28T17:46:14.139541+0000","last_undegraded":"2024-03-28T17:46:14.139541+0000","last_fullsized":"2024-03-28T17:46:14.139541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:36:22.864337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:17.966415+0000","last_change":"2024-03-28T17:46:11.761005+0000","last_active":"2024-03-28T17:46:17.966415+0000","last_peered":"2024-03-28T17:46:17.966415+0000","last_clean":"2024-03-28T17:46:17.966415+0000","last_became_active":"2024-03-28T17:46:11.760660+0000","last_became_peered":"2024-03-28T17:46:11.760660+0000","last_unstale":"2024-03-28T17:46:17.966415+0000","last_undegraded":"2024-03-28T17:46:17.966415+0000","last_fullsized":"2024-03-28T17:46:17.966415+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:01:10.408853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140954+0000","last_change":"2024-03-28T17:46:12.762851+0000","last_active":"2024-03-28T17:46:14.140954+0000","last_peered":"2024-03-28T17:46:14.140954+0000","last_clean":"2024-03-28T17:46:14.140954+0000","last_became_active":"2024-03-28T17:46:12.762508+0000","last_became_peered":"2024-03-28T17:46:12.762508+0000","last_unstale":"2024-03-28T17:46:14.140954+0000","last_undegraded":"2024-03-28T17:46:14.140954+0000","last_fullsized":"2024-03-28T17:46:14.140954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:55:29.389985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.143754+0000","last_change":"2024-03-28T17:46:11.776919+0000","last_active":"2024-03-28T17:46:14.143754+0000","last_peered":"2024-03-28T17:46:14.143754+0000","last_clean":"2024-03-28T17:46:14.143754+0000","last_became_active":"2024-03-28T17:46:11.776607+0000","last_became_peered":"2024-03-28T17:46:11.776607+0000","last_unstale":"2024-03-28T17:46:14.143754+0000","last_undegraded":"2024-03-28T17:46:14.143754+0000","last_fullsized":"2024-03-28T17:46:14.143754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:20:56.365757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140495+0000","last_change":"2024-03-28T17:46:12.794290+0000","last_active":"2024-03-28T17:46:14.140495+0000","last_peered":"2024-03-28T17:46:14.140495+0000","last_clean":"2024-03-28T17:46:14.140495+0000","last_became_active":"2024-03-28T17:46:12.793906+0000","last_became_peered":"2024-03-28T17:46:12.793906+0000","last_unstale":"2024-03-28T17:46:14.140495+0000","last_undegraded":"2024-03-28T17:46:14.140495+0000","last_fullsized":"2024-03-28T17:46:14.140495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:26:04.996019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.371419+0000","last_change":"2024-03-28T17:46:11.762676+0000","last_active":"2024-03-28T17:46:15.371419+0000","last_peered":"2024-03-28T17:46:15.371419+0000","last_clean":"2024-03-28T17:46:15.371419+0000","last_became_active":"2024-03-28T17:46:11.761458+0000","last_became_peered":"2024-03-28T17:46:11.761458+0000","last_unstale":"2024-03-28T17:46:15.371419+0000","last_undegraded":"2024-03-28T17:46:15.371419+0000","last_fullsized":"2024-03-28T17:46:15.371419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:22:26.642798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140562+0000","last_change":"2024-03-28T17:46:12.787107+0000","last_active":"2024-03-28T17:46:14.140562+0000","last_peered":"2024-03-28T17:46:14.140562+0000","last_clean":"2024-03-28T17:46:14.140562+0000","last_became_active":"2024-03-28T17:46:12.786610+0000","last_became_peered":"2024-03-28T17:46:12.786610+0000","last_unstale":"2024-03-28T17:46:14.140562+0000","last_undegraded":"2024-03-28T17:46:14.140562+0000","last_fullsized":"2024-03-28T17:46:14.140562+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:30:14.880483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139789+0000","last_change":"2024-03-28T17:46:12.763861+0000","last_active":"2024-03-28T17:46:14.139789+0000","last_peered":"2024-03-28T17:46:14.139789+0000","last_clean":"2024-03-28T17:46:14.139789+0000","last_became_active":"2024-03-28T17:46:12.763411+0000","last_became_peered":"2024-03-28T17:46:12.763411+0000","last_unstale":"2024-03-28T17:46:14.139789+0000","last_undegraded":"2024-03-28T17:46:14.139789+0000","last_fullsized":"2024-03-28T17:46:14.139789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:52:02.657945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:17.979059+0000","last_change":"2024-03-28T17:46:11.784261+0000","last_active":"2024-03-28T17:46:17.979059+0000","last_peered":"2024-03-28T17:46:17.979059+0000","last_clean":"2024-03-28T17:46:17.979059+0000","last_became_active":"2024-03-28T17:46:11.782957+0000","last_became_peered":"2024-03-28T17:46:11.782957+0000","last_unstale":"2024-03-28T17:46:17.979059+0000","last_undegraded":"2024-03-28T17:46:17.979059+0000","last_fullsized":"2024-03-28T17:46:17.979059+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:08:48.009774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140767+0000","last_change":"2024-03-28T17:46:12.784024+0000","last_active":"2024-03-28T17:46:14.140767+0000","last_peered":"2024-03-28T17:46:14.140767+0000","last_clean":"2024-03-28T17:46:14.140767+0000","last_became_active":"2024-03-28T17:46:12.782157+0000","last_became_peered":"2024-03-28T17:46:12.782157+0000","last_unstale":"2024-03-28T17:46:14.140767+0000","last_undegraded":"2024-03-28T17:46:14.140767+0000","last_fullsized":"2024-03-28T17:46:14.140767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:33.679894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:17.973716+0000","last_change":"2024-03-28T17:46:11.759602+0000","last_active":"2024-03-28T17:46:17.973716+0000","last_peered":"2024-03-28T17:46:17.973716+0000","last_clean":"2024-03-28T17:46:17.973716+0000","last_became_active":"2024-03-28T17:46:11.759248+0000","last_became_peered":"2024-03-28T17:46:11.759248+0000","last_unstale":"2024-03-28T17:46:17.973716+0000","last_undegraded":"2024-03-28T17:46:17.973716+0000","last_fullsized":"2024-03-28T17:46:17.973716+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:15:59.997948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":347,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139608+0000","last_change":"2024-03-28T17:44:51.172493+0000","last_active":"2024-03-28T17:46:14.139608+0000","last_peered":"2024-03-28T17:46:14.139608+0000","last_clean":"2024-03-28T17:46:14.139608+0000","last_became_active":"2024-03-28T17:44:51.172175+0000","last_became_peered":"2024-03-28T17:44:51.172175+0000","last_unstale":"2024-03-28T17:46:14.139608+0000","last_undegraded":"2024-03-28T17:46:14.139608+0000","last_fullsized":"2024-03-28T17:46:14.139608+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:40:54.580958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:40:54.580958+0000","last_clean_scrub_stamp":"2024-03-28T17:40:54.580958+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:34.942490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.745598+0000","last_change":"2024-03-28T17:46:12.766728+0000","last_active":"2024-03-28T17:46:13.745598+0000","last_peered":"2024-03-28T17:46:13.745598+0000","last_clean":"2024-03-28T17:46:13.745598+0000","last_became_active":"2024-03-28T17:46:12.763671+0000","last_became_peered":"2024-03-28T17:46:12.763671+0000","last_unstale":"2024-03-28T17:46:13.745598+0000","last_undegraded":"2024-03-28T17:46:13.745598+0000","last_fullsized":"2024-03-28T17:46:13.745598+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:25:39.802551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:16.970580+0000","last_change":"2024-03-28T17:46:11.763361+0000","last_active":"2024-03-28T17:46:16.970580+0000","last_peered":"2024-03-28T17:46:16.970580+0000","last_clean":"2024-03-28T17:46:16.970580+0000","last_became_active":"2024-03-28T17:46:11.763049+0000","last_became_peered":"2024-03-28T17:46:11.763049+0000","last_unstale":"2024-03-28T17:46:16.970580+0000","last_undegraded":"2024-03-28T17:46:16.970580+0000","last_fullsized":"2024-03-28T17:46:16.970580+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:01.767964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.946355+0000","last_change":"2024-03-28T17:46:11.774957+0000","last_active":"2024-03-28T17:46:15.946355+0000","last_peered":"2024-03-28T17:46:15.946355+0000","last_clean":"2024-03-28T17:46:15.946355+0000","last_became_active":"2024-03-28T17:46:11.774658+0000","last_became_peered":"2024-03-28T17:46:11.774658+0000","last_unstale":"2024-03-28T17:46:15.946355+0000","last_undegraded":"2024-03-28T17:46:15.946355+0000","last_fullsized":"2024-03-28T17:46:15.946355+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:40.254318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.746510+0000","last_change":"2024-03-28T17:46:12.778206+0000","last_active":"2024-03-28T17:46:13.746510+0000","last_peered":"2024-03-28T17:46:13.746510+0000","last_clean":"2024-03-28T17:46:13.746510+0000","last_became_active":"2024-03-28T17:46:12.777780+0000","last_became_peered":"2024-03-28T17:46:12.777780+0000","last_unstale":"2024-03-28T17:46:13.746510+0000","last_undegraded":"2024-03-28T17:46:13.746510+0000","last_fullsized":"2024-03-28T17:46:13.746510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:13.241794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:16.979312+0000","last_change":"2024-03-28T17:46:11.787480+0000","last_active":"2024-03-28T17:46:16.979312+0000","last_peered":"2024-03-28T17:46:16.979312+0000","last_clean":"2024-03-28T17:46:16.979312+0000","last_became_active":"2024-03-28T17:46:11.786808+0000","last_became_peered":"2024-03-28T17:46:11.786808+0000","last_unstale":"2024-03-28T17:46:16.979312+0000","last_undegraded":"2024-03-28T17:46:16.979312+0000","last_fullsized":"2024-03-28T17:46:16.979312+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:35.236554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140851+0000","last_change":"2024-03-28T17:46:12.790489+0000","last_active":"2024-03-28T17:46:14.140851+0000","last_peered":"2024-03-28T17:46:14.140851+0000","last_clean":"2024-03-28T17:46:14.140851+0000","last_became_active":"2024-03-28T17:46:12.790108+0000","last_became_peered":"2024-03-28T17:46:12.790108+0000","last_unstale":"2024-03-28T17:46:14.140851+0000","last_undegraded":"2024-03-28T17:46:14.140851+0000","last_fullsized":"2024-03-28T17:46:14.140851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:16:21.210182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.943430+0000","last_change":"2024-03-28T17:46:11.762658+0000","last_active":"2024-03-28T17:46:15.943430+0000","last_peered":"2024-03-28T17:46:15.943430+0000","last_clean":"2024-03-28T17:46:15.943430+0000","last_became_active":"2024-03-28T17:46:11.762296+0000","last_became_peered":"2024-03-28T17:46:11.762296+0000","last_unstale":"2024-03-28T17:46:15.943430+0000","last_undegraded":"2024-03-28T17:46:15.943430+0000","last_fullsized":"2024-03-28T17:46:15.943430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:42:52.768285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.143561+0000","last_change":"2024-03-28T17:46:12.790462+0000","last_active":"2024-03-28T17:46:14.143561+0000","last_peered":"2024-03-28T17:46:14.143561+0000","last_clean":"2024-03-28T17:46:14.143561+0000","last_became_active":"2024-03-28T17:46:12.788783+0000","last_became_peered":"2024-03-28T17:46:12.788783+0000","last_unstale":"2024-03-28T17:46:14.143561+0000","last_undegraded":"2024-03-28T17:46:14.143561+0000","last_fullsized":"2024-03-28T17:46:14.143561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:47:09.130970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.942122+0000","last_change":"2024-03-28T17:46:11.769261+0000","last_active":"2024-03-28T17:46:15.942122+0000","last_peered":"2024-03-28T17:46:15.942122+0000","last_clean":"2024-03-28T17:46:15.942122+0000","last_became_active":"2024-03-28T17:46:11.768918+0000","last_became_peered":"2024-03-28T17:46:11.768918+0000","last_unstale":"2024-03-28T17:46:15.942122+0000","last_undegraded":"2024-03-28T17:46:15.942122+0000","last_fullsized":"2024-03-28T17:46:15.942122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:26:38.276425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139702+0000","last_change":"2024-03-28T17:46:12.762617+0000","last_active":"2024-03-28T17:46:14.139702+0000","last_peered":"2024-03-28T17:46:14.139702+0000","last_clean":"2024-03-28T17:46:14.139702+0000","last_became_active":"2024-03-28T17:46:12.762273+0000","last_became_peered":"2024-03-28T17:46:12.762273+0000","last_unstale":"2024-03-28T17:46:14.139702+0000","last_undegraded":"2024-03-28T17:46:14.139702+0000","last_fullsized":"2024-03-28T17:46:14.139702+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:45.848454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.946385+0000","last_change":"2024-03-28T17:46:11.770831+0000","last_active":"2024-03-28T17:46:15.946385+0000","last_peered":"2024-03-28T17:46:15.946385+0000","last_clean":"2024-03-28T17:46:15.946385+0000","last_became_active":"2024-03-28T17:46:11.770303+0000","last_became_peered":"2024-03-28T17:46:11.770303+0000","last_unstale":"2024-03-28T17:46:15.946385+0000","last_undegraded":"2024-03-28T17:46:15.946385+0000","last_fullsized":"2024-03-28T17:46:15.946385+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:48:25.566922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140248+0000","last_change":"2024-03-28T17:46:12.768885+0000","last_active":"2024-03-28T17:46:14.140248+0000","last_peered":"2024-03-28T17:46:14.140248+0000","last_clean":"2024-03-28T17:46:14.140248+0000","last_became_active":"2024-03-28T17:46:12.768525+0000","last_became_peered":"2024-03-28T17:46:12.768525+0000","last_unstale":"2024-03-28T17:46:14.140248+0000","last_undegraded":"2024-03-28T17:46:14.140248+0000","last_fullsized":"2024-03-28T17:46:14.140248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:35:30.779279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.582388+0000","last_change":"2024-03-28T17:46:11.786661+0000","last_active":"2024-03-28T17:46:14.582388+0000","last_peered":"2024-03-28T17:46:14.582388+0000","last_clean":"2024-03-28T17:46:14.582388+0000","last_became_active":"2024-03-28T17:46:11.786226+0000","last_became_peered":"2024-03-28T17:46:11.786226+0000","last_unstale":"2024-03-28T17:46:14.582388+0000","last_undegraded":"2024-03-28T17:46:14.582388+0000","last_fullsized":"2024-03-28T17:46:14.582388+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:51:24.781368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140181+0000","last_change":"2024-03-28T17:46:12.797531+0000","last_active":"2024-03-28T17:46:14.140181+0000","last_peered":"2024-03-28T17:46:14.140181+0000","last_clean":"2024-03-28T17:46:14.140181+0000","last_became_active":"2024-03-28T17:46:12.797161+0000","last_became_peered":"2024-03-28T17:46:12.797161+0000","last_unstale":"2024-03-28T17:46:14.140181+0000","last_undegraded":"2024-03-28T17:46:14.140181+0000","last_fullsized":"2024-03-28T17:46:14.140181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:01.534666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.938194+0000","last_change":"2024-03-28T17:46:11.766589+0000","last_active":"2024-03-28T17:46:15.938194+0000","last_peered":"2024-03-28T17:46:15.938194+0000","last_clean":"2024-03-28T17:46:15.938194+0000","last_became_active":"2024-03-28T17:46:11.766269+0000","last_became_peered":"2024-03-28T17:46:11.766269+0000","last_unstale":"2024-03-28T17:46:15.938194+0000","last_undegraded":"2024-03-28T17:46:15.938194+0000","last_fullsized":"2024-03-28T17:46:15.938194+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:12:26.055262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140141+0000","last_change":"2024-03-28T17:46:12.765723+0000","last_active":"2024-03-28T17:46:14.140141+0000","last_peered":"2024-03-28T17:46:14.140141+0000","last_clean":"2024-03-28T17:46:14.140141+0000","last_became_active":"2024-03-28T17:46:12.765366+0000","last_became_peered":"2024-03-28T17:46:12.765366+0000","last_unstale":"2024-03-28T17:46:14.140141+0000","last_undegraded":"2024-03-28T17:46:14.140141+0000","last_fullsized":"2024-03-28T17:46:14.140141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:39:55.103246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:16.978290+0000","last_change":"2024-03-28T17:46:11.767686+0000","last_active":"2024-03-28T17:46:16.978290+0000","last_peered":"2024-03-28T17:46:16.978290+0000","last_clean":"2024-03-28T17:46:16.978290+0000","last_became_active":"2024-03-28T17:46:11.767365+0000","last_became_peered":"2024-03-28T17:46:11.767365+0000","last_unstale":"2024-03-28T17:46:16.978290+0000","last_undegraded":"2024-03-28T17:46:16.978290+0000","last_fullsized":"2024-03-28T17:46:16.978290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:49.909554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140531+0000","last_change":"2024-03-28T17:46:12.764129+0000","last_active":"2024-03-28T17:46:14.140531+0000","last_peered":"2024-03-28T17:46:14.140531+0000","last_clean":"2024-03-28T17:46:14.140531+0000","last_became_active":"2024-03-28T17:46:12.763716+0000","last_became_peered":"2024-03-28T17:46:12.763716+0000","last_unstale":"2024-03-28T17:46:14.140531+0000","last_undegraded":"2024-03-28T17:46:14.140531+0000","last_fullsized":"2024-03-28T17:46:14.140531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:08.365578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.141019+0000","last_change":"2024-03-28T17:46:11.787374+0000","last_active":"2024-03-28T17:46:14.141019+0000","last_peered":"2024-03-28T17:46:14.141019+0000","last_clean":"2024-03-28T17:46:14.141019+0000","last_became_active":"2024-03-28T17:46:11.787047+0000","last_became_peered":"2024-03-28T17:46:11.787047+0000","last_unstale":"2024-03-28T17:46:14.141019+0000","last_undegraded":"2024-03-28T17:46:14.141019+0000","last_fullsized":"2024-03-28T17:46:14.141019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:33.657488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:15.371446+0000","last_change":"2024-03-28T17:46:12.794069+0000","last_active":"2024-03-28T17:46:15.371446+0000","last_peered":"2024-03-28T17:46:15.371446+0000","last_clean":"2024-03-28T17:46:15.371446+0000","last_became_active":"2024-03-28T17:46:12.793640+0000","last_became_peered":"2024-03-28T17:46:12.793640+0000","last_unstale":"2024-03-28T17:46:15.371446+0000","last_undegraded":"2024-03-28T17:46:15.371446+0000","last_fullsized":"2024-03-28T17:46:15.371446+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:23:15.065887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.746450+0000","last_change":"2024-03-28T17:46:11.780388+0000","last_active":"2024-03-28T17:46:13.746450+0000","last_peered":"2024-03-28T17:46:13.746450+0000","last_clean":"2024-03-28T17:46:13.746450+0000","last_became_active":"2024-03-28T17:46:11.780016+0000","last_became_peered":"2024-03-28T17:46:11.780016+0000","last_unstale":"2024-03-28T17:46:13.746450+0000","last_undegraded":"2024-03-28T17:46:13.746450+0000","last_fullsized":"2024-03-28T17:46:13.746450+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:51:06.181772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140809+0000","last_change":"2024-03-28T17:46:12.764748+0000","last_active":"2024-03-28T17:46:14.140809+0000","last_peered":"2024-03-28T17:46:14.140809+0000","last_clean":"2024-03-28T17:46:14.140809+0000","last_became_active":"2024-03-28T17:46:12.764407+0000","last_became_peered":"2024-03-28T17:46:12.764407+0000","last_unstale":"2024-03-28T17:46:14.140809+0000","last_undegraded":"2024-03-28T17:46:14.140809+0000","last_fullsized":"2024-03-28T17:46:14.140809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:41:54.185079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139691+0000","last_change":"2024-03-28T17:46:12.762486+0000","last_active":"2024-03-28T17:46:14.139691+0000","last_peered":"2024-03-28T17:46:14.139691+0000","last_clean":"2024-03-28T17:46:14.139691+0000","last_became_active":"2024-03-28T17:46:12.762107+0000","last_became_peered":"2024-03-28T17:46:12.762107+0000","last_unstale":"2024-03-28T17:46:14.139691+0000","last_undegraded":"2024-03-28T17:46:14.139691+0000","last_fullsized":"2024-03-28T17:46:14.139691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:31.062456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.746525+0000","last_change":"2024-03-28T17:46:12.764569+0000","last_active":"2024-03-28T17:46:13.746525+0000","last_peered":"2024-03-28T17:46:13.746525+0000","last_clean":"2024-03-28T17:46:13.746525+0000","last_became_active":"2024-03-28T17:46:12.764112+0000","last_became_peered":"2024-03-28T17:46:12.764112+0000","last_unstale":"2024-03-28T17:46:13.746525+0000","last_undegraded":"2024-03-28T17:46:13.746525+0000","last_fullsized":"2024-03-28T17:46:13.746525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:39:55.526432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.746595+0000","last_change":"2024-03-28T17:46:12.786031+0000","last_active":"2024-03-28T17:46:13.746595+0000","last_peered":"2024-03-28T17:46:13.746595+0000","last_clean":"2024-03-28T17:46:13.746595+0000","last_became_active":"2024-03-28T17:46:12.785679+0000","last_became_peered":"2024-03-28T17:46:12.785679+0000","last_unstale":"2024-03-28T17:46:13.746595+0000","last_undegraded":"2024-03-28T17:46:13.746595+0000","last_fullsized":"2024-03-28T17:46:13.746595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:41:22.537979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.142231+0000","last_change":"2024-03-28T17:46:12.793285+0000","last_active":"2024-03-28T17:46:14.142231+0000","last_peered":"2024-03-28T17:46:14.142231+0000","last_clean":"2024-03-28T17:46:14.142231+0000","last_became_active":"2024-03-28T17:46:12.788935+0000","last_became_peered":"2024-03-28T17:46:12.788935+0000","last_unstale":"2024-03-28T17:46:14.142231+0000","last_undegraded":"2024-03-28T17:46:14.142231+0000","last_fullsized":"2024-03-28T17:46:14.142231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:53:58.886607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.746357+0000","last_change":"2024-03-28T17:46:12.791572+0000","last_active":"2024-03-28T17:46:13.746357+0000","last_peered":"2024-03-28T17:46:13.746357+0000","last_clean":"2024-03-28T17:46:13.746357+0000","last_became_active":"2024-03-28T17:46:12.791081+0000","last_became_peered":"2024-03-28T17:46:12.791081+0000","last_unstale":"2024-03-28T17:46:13.746357+0000","last_undegraded":"2024-03-28T17:46:13.746357+0000","last_fullsized":"2024-03-28T17:46:13.746357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:51:38.955784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140041+0000","last_change":"2024-03-28T17:46:12.765140+0000","last_active":"2024-03-28T17:46:14.140041+0000","last_peered":"2024-03-28T17:46:14.140041+0000","last_clean":"2024-03-28T17:46:14.140041+0000","last_became_active":"2024-03-28T17:46:12.764760+0000","last_became_peered":"2024-03-28T17:46:12.764760+0000","last_unstale":"2024-03-28T17:46:14.140041+0000","last_undegraded":"2024-03-28T17:46:14.140041+0000","last_fullsized":"2024-03-28T17:46:14.140041+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:41.576437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.747038+0000","last_change":"2024-03-28T17:46:12.760271+0000","last_active":"2024-03-28T17:46:13.747038+0000","last_peered":"2024-03-28T17:46:13.747038+0000","last_clean":"2024-03-28T17:46:13.747038+0000","last_became_active":"2024-03-28T17:46:12.759939+0000","last_became_peered":"2024-03-28T17:46:12.759939+0000","last_unstale":"2024-03-28T17:46:13.747038+0000","last_undegraded":"2024-03-28T17:46:13.747038+0000","last_fullsized":"2024-03-28T17:46:13.747038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:06:42.529826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140627+0000","last_change":"2024-03-28T17:46:12.792360+0000","last_active":"2024-03-28T17:46:14.140627+0000","last_peered":"2024-03-28T17:46:14.140627+0000","last_clean":"2024-03-28T17:46:14.140627+0000","last_became_active":"2024-03-28T17:46:12.791979+0000","last_became_peered":"2024-03-28T17:46:12.791979+0000","last_unstale":"2024-03-28T17:46:14.140627+0000","last_undegraded":"2024-03-28T17:46:14.140627+0000","last_fullsized":"2024-03-28T17:46:14.140627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:39.866969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.139611+0000","last_change":"2024-03-28T17:46:12.784228+0000","last_active":"2024-03-28T17:46:14.139611+0000","last_peered":"2024-03-28T17:46:14.139611+0000","last_clean":"2024-03-28T17:46:14.139611+0000","last_became_active":"2024-03-28T17:46:12.783890+0000","last_became_peered":"2024-03-28T17:46:12.783890+0000","last_unstale":"2024-03-28T17:46:14.139611+0000","last_undegraded":"2024-03-28T17:46:14.139611+0000","last_fullsized":"2024-03-28T17:46:14.139611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:58:13.388705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.141830+0000","last_change":"2024-03-28T17:46:12.782675+0000","last_active":"2024-03-28T17:46:14.141830+0000","last_peered":"2024-03-28T17:46:14.141830+0000","last_clean":"2024-03-28T17:46:14.141830+0000","last_became_active":"2024-03-28T17:46:12.779255+0000","last_became_peered":"2024-03-28T17:46:12.779255+0000","last_unstale":"2024-03-28T17:46:14.141830+0000","last_undegraded":"2024-03-28T17:46:14.141830+0000","last_fullsized":"2024-03-28T17:46:14.141830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:04:31.447512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.748959+0000","last_change":"2024-03-28T17:46:12.786308+0000","last_active":"2024-03-28T17:46:13.748959+0000","last_peered":"2024-03-28T17:46:13.748959+0000","last_clean":"2024-03-28T17:46:13.748959+0000","last_became_active":"2024-03-28T17:46:12.785948+0000","last_became_peered":"2024-03-28T17:46:12.785948+0000","last_unstale":"2024-03-28T17:46:13.748959+0000","last_undegraded":"2024-03-28T17:46:13.748959+0000","last_fullsized":"2024-03-28T17:46:13.748959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:43.292925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:13.746289+0000","last_change":"2024-03-28T17:46:12.797278+0000","last_active":"2024-03-28T17:46:13.746289+0000","last_peered":"2024-03-28T17:46:13.746289+0000","last_clean":"2024-03-28T17:46:13.746289+0000","last_became_active":"2024-03-28T17:46:12.796593+0000","last_became_peered":"2024-03-28T17:46:12.796593+0000","last_unstale":"2024-03-28T17:46:13.746289+0000","last_undegraded":"2024-03-28T17:46:13.746289+0000","last_fullsized":"2024-03-28T17:46:13.746289+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:07:34.433680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-28T17:46:14.140860+0000","last_change":"2024-03-28T17:46:12.764249+0000","last_active":"2024-03-28T17:46:14.140860+0000","last_peered":"2024-03-28T17:46:14.140860+0000","last_clean":"2024-03-28T17:46:14.140860+0000","last_became_active":"2024-03-28T17:46:12.763909+0000","last_became_peered":"2024-03-28T17:46:12.763909+0000","last_unstale":"2024-03-28T17:46:14.140860+0000","last_undegraded":"2024-03-28T17:46:14.140860+0000","last_fullsized":"2024-03-28T17:46:14.140860+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:40:51.780187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250098,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6004,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":26,"num_read_kb":22,"num_write":129,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":74920,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383696,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":439103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645332,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":1031457,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808859,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":439339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070496,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":439259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332133,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":1031580,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495659,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":438611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574920,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":478149,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738446,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":439999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411395,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":461157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182460,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":438805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920823,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":925696,"data_stored":439556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":1030756,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34554,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34554,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":39628,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":22636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T17:46:22.556 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T17:46:22.556 INFO:tasks.cephadm:Setting up client nodes... 2024-03-28T17:46:22.556 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-28T17:46:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:22 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3527279728' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:46:22.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:22 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3527279728' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:46:22.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:22 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3527279728' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:46:23.054 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:46:23.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:23 smithi045 ceph-mon[31773]: pgmap v322: 97 pgs: 97 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 17 op/s 2024-03-28T17:46:23.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:23 smithi045 ceph-mon[31773]: from='client.24716 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:46:23.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:23 smithi045 ceph-mon[31773]: from='client.24722 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:46:23.744 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:23 smithi151 ceph-mon[36481]: pgmap v322: 97 pgs: 97 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 17 op/s 2024-03-28T17:46:23.744 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:23 smithi151 ceph-mon[36481]: from='client.24716 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:46:23.744 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:23 smithi151 ceph-mon[36481]: from='client.24722 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:46:23.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:23 smithi071 ceph-mon[36597]: pgmap v322: 97 pgs: 97 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 17 op/s 2024-03-28T17:46:23.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:23 smithi071 ceph-mon[36597]: from='client.24716 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:46:23.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:23 smithi071 ceph-mon[36597]: from='client.24722 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:46:25.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.307+0000 7f7461c00700 1 -- 172.21.15.45:0/3959550923 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f745c108810 msgr2=0x7f745c108c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:25.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.307+0000 7f7461c00700 1 --2- 172.21.15.45:0/3959550923 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f745c108810 0x7f745c108c30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f744c0099c0 tx=0x7f744c009cd0 comp rx=0 tx=0).stop 2024-03-28T17:46:25.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.307+0000 7f7461c00700 1 -- 172.21.15.45:0/3959550923 shutdown_connections 2024-03-28T17:46:25.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.307+0000 7f7461c00700 1 --2- 172.21.15.45:0/3959550923 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f745c10a3b0 0x7f745c110c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:25.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.307+0000 7f7461c00700 1 --2- 172.21.15.45:0/3959550923 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f745c108810 0x7f745c108c30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:25.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.307+0000 7f7461c00700 1 --2- 172.21.15.45:0/3959550923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f745c109a10 0x7f745c109e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:25.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.307+0000 7f7461c00700 1 -- 172.21.15.45:0/3959550923 >> 172.21.15.45:0/3959550923 conn(0x7f745c0777e0 msgr2=0x7f745c079c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:25.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.307+0000 7f7461c00700 1 -- 172.21.15.45:0/3959550923 shutdown_connections 2024-03-28T17:46:25.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.308+0000 7f7461c00700 1 -- 172.21.15.45:0/3959550923 wait complete. 2024-03-28T17:46:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.308+0000 7f7461c00700 1 Processor -- start 2024-03-28T17:46:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.309+0000 7f7461c00700 1 -- start start 2024-03-28T17:46:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.309+0000 7f7461c00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f745c108810 0x7f745c19e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.309+0000 7f7461c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f745c109a10 0x7f745c19ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.309+0000 7f7461c00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f745c10a3b0 0x7f745c1a5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.309+0000 7f7461c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c114420 con 0x7f745c109a10 2024-03-28T17:46:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.309+0000 7f7461c00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c1142a0 con 0x7f745c108810 2024-03-28T17:46:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.309+0000 7f7461c00700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c1145a0 con 0x7f745c10a3b0 2024-03-28T17:46:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.309+0000 7f745aa00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f745c108810 0x7f745c19e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.309+0000 7f745aa00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f745c108810 0x7f745c19e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:57544/0 (socket says 172.21.15.45:57544) 2024-03-28T17:46:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.309+0000 7f745aa00700 1 -- 172.21.15.45:0/3820534815 learned_addr learned my addr 172.21.15.45:0/3820534815 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.309+0000 7f745a000700 1 --2- 172.21.15.45:0/3820534815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f745c109a10 0x7f745c19ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.309+0000 7f745b400700 1 --2- 172.21.15.45:0/3820534815 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f745c10a3b0 0x7f745c1a5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:25.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.310+0000 7f745aa00700 1 -- 172.21.15.45:0/3820534815 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f745c10a3b0 msgr2=0x7f745c1a5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:25.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.310+0000 7f745aa00700 1 --2- 172.21.15.45:0/3820534815 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f745c10a3b0 0x7f745c1a5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:25.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.310+0000 7f745aa00700 1 -- 172.21.15.45:0/3820534815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f745c109a10 msgr2=0x7f745c19ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:25.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.310+0000 7f745aa00700 1 --2- 172.21.15.45:0/3820534815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f745c109a10 0x7f745c19ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:25.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.310+0000 7f745aa00700 1 -- 172.21.15.45:0/3820534815 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745c1a65a0 con 0x7f745c108810 2024-03-28T17:46:25.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.310+0000 7f745aa00700 1 --2- 172.21.15.45:0/3820534815 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f745c108810 0x7f745c19e940 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f744c0098e0 tx=0x7f744c009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:25.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.310+0000 7f7453400700 1 -- 172.21.15.45:0/3820534815 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744c027070 con 0x7f745c108810 2024-03-28T17:46:25.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.310+0000 7f7453400700 1 -- 172.21.15.45:0/3820534815 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f744c022470 con 0x7f745c108810 2024-03-28T17:46:25.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.310+0000 7f7453400700 1 -- 172.21.15.45:0/3820534815 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744c009e60 con 0x7f745c108810 2024-03-28T17:46:25.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.310+0000 7f7461c00700 1 -- 172.21.15.45:0/3820534815 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f745c1a6830 con 0x7f745c108810 2024-03-28T17:46:25.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.311+0000 7f7461c00700 1 -- 172.21.15.45:0/3820534815 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f745c1a6ce0 con 0x7f745c108810 2024-03-28T17:46:25.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.312+0000 7f7461c00700 1 -- 172.21.15.45:0/3820534815 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f745c0506a0 con 0x7f745c108810 2024-03-28T17:46:25.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.313+0000 7f7453400700 1 -- 172.21.15.45:0/3820534815 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f744c02a070 con 0x7f745c108810 2024-03-28T17:46:25.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.315+0000 7f7453400700 1 --2- 172.21.15.45:0/3820534815 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f74400a2a50 0x7f74400a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:25.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.315+0000 7f7453400700 1 -- 172.21.15.45:0/3820534815 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f744c0cf5a0 con 0x7f745c108810 2024-03-28T17:46:25.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.315+0000 7f745a000700 1 --2- 172.21.15.45:0/3820534815 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f74400a2a50 0x7f74400a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:25.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.316+0000 7f745a000700 1 --2- 172.21.15.45:0/3820534815 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f74400a2a50 0x7f74400a4f10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7448009c60 tx=0x7f7448009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:25.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.319+0000 7f7453400700 1 -- 172.21.15.45:0/3820534815 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f744c013070 con 0x7f745c108810 2024-03-28T17:46:25.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:25 smithi045 ceph-mon[31773]: pgmap v323: 97 pgs: 97 active+clean; 624 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 8.8 KiB/s wr, 18 op/s 2024-03-28T17:46:25.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.637+0000 7f7461c00700 1 -- 172.21.15.45:0/3820534815 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f745c02da30 con 0x7f745c108810 2024-03-28T17:46:25.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.645+0000 7f7453400700 1 -- 172.21.15.45:0/3820534815 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7f744c09c620 con 0x7f745c108810 2024-03-28T17:46:25.646 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2024-03-28T17:46:25.646 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQDxrAVmOYokJhAAB4Ln4SrPcc7D90c3w3eRQA== 2024-03-28T17:46:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.648+0000 7f7461c00700 1 -- 172.21.15.45:0/3820534815 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f74400a2a50 msgr2=0x7f74400a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.648+0000 7f7461c00700 1 --2- 172.21.15.45:0/3820534815 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f74400a2a50 0x7f74400a4f10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7448009c60 tx=0x7f7448009260 comp rx=0 tx=0).stop 2024-03-28T17:46:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.648+0000 7f7461c00700 1 -- 172.21.15.45:0/3820534815 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f745c108810 msgr2=0x7f745c19e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.648+0000 7f7461c00700 1 --2- 172.21.15.45:0/3820534815 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f745c108810 0x7f745c19e940 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f744c0098e0 tx=0x7f744c009580 comp rx=0 tx=0).stop 2024-03-28T17:46:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.648+0000 7f7461c00700 1 -- 172.21.15.45:0/3820534815 shutdown_connections 2024-03-28T17:46:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.648+0000 7f7461c00700 1 --2- 172.21.15.45:0/3820534815 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f74400a2a50 0x7f74400a4f10 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.648+0000 7f7461c00700 1 --2- 172.21.15.45:0/3820534815 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f745c10a3b0 0x7f745c1a5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.648+0000 7f7461c00700 1 --2- 172.21.15.45:0/3820534815 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f745c108810 0x7f745c19e940 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.648+0000 7f7461c00700 1 --2- 172.21.15.45:0/3820534815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f745c109a10 0x7f745c19ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.648+0000 7f7461c00700 1 -- 172.21.15.45:0/3820534815 >> 172.21.15.45:0/3820534815 conn(0x7f745c0777e0 msgr2=0x7f745c10cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.648+0000 7f7461c00700 1 -- 172.21.15.45:0/3820534815 shutdown_connections 2024-03-28T17:46:25.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:25.649+0000 7f7461c00700 1 -- 172.21.15.45:0/3820534815 wait complete. 2024-03-28T17:46:25.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:25 smithi071 ceph-mon[36597]: pgmap v323: 97 pgs: 97 active+clean; 624 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 8.8 KiB/s wr, 18 op/s 2024-03-28T17:46:25.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:25 smithi151 ceph-mon[36481]: pgmap v323: 97 pgs: 97 active+clean; 624 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 8.8 KiB/s wr, 18 op/s 2024-03-28T17:46:26.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:26 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3820534815' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T17:46:26.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:26 smithi045 ceph-mon[31773]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T17:46:26.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:26 smithi045 ceph-mon[31773]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T17:46:26.591 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-28T17:46:26.591 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-28T17:46:26.591 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-28T17:46:26.757 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-28T17:46:26.757 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-28T17:46:26.757 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-28T17:46:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:26 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3820534815' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T17:46:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:26 smithi071 ceph-mon[36597]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T17:46:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:26 smithi071 ceph-mon[36597]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T17:46:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:26 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3820534815' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T17:46:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:26 smithi151 ceph-mon[36481]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T17:46:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:26 smithi151 ceph-mon[36481]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T17:46:27.268 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:46:27.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:27 smithi045 ceph-mon[31773]: pgmap v324: 97 pgs: 97 active+clean; 624 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.6 KiB/s wr, 17 op/s 2024-03-28T17:46:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:27 smithi071 ceph-mon[36597]: pgmap v324: 97 pgs: 97 active+clean; 624 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.6 KiB/s wr, 17 op/s 2024-03-28T17:46:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:27 smithi151 ceph-mon[36481]: pgmap v324: 97 pgs: 97 active+clean; 624 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.6 KiB/s wr, 17 op/s 2024-03-28T17:46:29.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.541+0000 7f3f97c00700 1 -- 172.21.15.45:0/1190543731 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f9006a7e0 msgr2=0x7f3f9010f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:29.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.541+0000 7f3f97c00700 1 --2- 172.21.15.45:0/1190543731 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f9006a7e0 0x7f3f9010f020 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3f88009f90 tx=0x7f3f8800b330 comp rx=0 tx=0).stop 2024-03-28T17:46:29.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.541+0000 7f3f97c00700 1 -- 172.21.15.45:0/1190543731 shutdown_connections 2024-03-28T17:46:29.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.541+0000 7f3f97c00700 1 --2- 172.21.15.45:0/1190543731 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f901081e0 0x7f3f901085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:29.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.541+0000 7f3f97c00700 1 --2- 172.21.15.45:0/1190543731 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f9006a7e0 0x7f3f9010f020 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:29.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.541+0000 7f3f97c00700 1 --2- 172.21.15.45:0/1190543731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f90069e40 0x7f3f9006a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:29.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.541+0000 7f3f97c00700 1 -- 172.21.15.45:0/1190543731 >> 172.21.15.45:0/1190543731 conn(0x7f3f90077fc0 msgr2=0x7f3f900783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:29.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.541+0000 7f3f97c00700 1 -- 172.21.15.45:0/1190543731 shutdown_connections 2024-03-28T17:46:29.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.541+0000 7f3f97c00700 1 -- 172.21.15.45:0/1190543731 wait complete. 2024-03-28T17:46:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.542+0000 7f3f97c00700 1 Processor -- start 2024-03-28T17:46:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.542+0000 7f3f97c00700 1 -- start start 2024-03-28T17:46:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.543+0000 7f3f97c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f90069e40 0x7f3f901a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.543+0000 7f3f97c00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f9006a7e0 0x7f3f901a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.543+0000 7f3f97c00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f901081e0 0x7f3f9019a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.543+0000 7f3f97c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f90112480 con 0x7f3f90069e40 2024-03-28T17:46:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.543+0000 7f3f97c00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f90112300 con 0x7f3f9006a7e0 2024-03-28T17:46:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.543+0000 7f3f97c00700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f90112600 con 0x7f3f901081e0 2024-03-28T17:46:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.543+0000 7f3f95200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f90069e40 0x7f3f901a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.543+0000 7f3f8fe00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f9006a7e0 0x7f3f901a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:29.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.543+0000 7f3f8fe00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f9006a7e0 0x7f3f901a0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:50622/0 (socket says 172.21.15.45:50622) 2024-03-28T17:46:29.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.543+0000 7f3f8fe00700 1 -- 172.21.15.45:0/296080933 learned_addr learned my addr 172.21.15.45:0/296080933 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:29.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.543+0000 7f3f95c00700 1 --2- 172.21.15.45:0/296080933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f901081e0 0x7f3f9019a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:29.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.543+0000 7f3f8fe00700 1 -- 172.21.15.45:0/296080933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f901081e0 msgr2=0x7f3f9019a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:29.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.543+0000 7f3f8fe00700 1 --2- 172.21.15.45:0/296080933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f901081e0 0x7f3f9019a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:29.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.543+0000 7f3f8fe00700 1 -- 172.21.15.45:0/296080933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f90069e40 msgr2=0x7f3f901a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:29.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.543+0000 7f3f8fe00700 1 --2- 172.21.15.45:0/296080933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f90069e40 0x7f3f901a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:29.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.543+0000 7f3f8fe00700 1 -- 172.21.15.45:0/296080933 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f9019b070 con 0x7f3f9006a7e0 2024-03-28T17:46:29.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.543+0000 7f3f95c00700 1 --2- 172.21.15.45:0/296080933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f901081e0 0x7f3f9019a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:46:29.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.544+0000 7f3f8fe00700 1 --2- 172.21.15.45:0/296080933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f9006a7e0 0x7f3f901a0c90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c00e9a0 tx=0x7f3f7c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:29.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.544+0000 7f3f8d600700 1 -- 172.21.15.45:0/296080933 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c00cbd0 con 0x7f3f9006a7e0 2024-03-28T17:46:29.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.544+0000 7f3f8d600700 1 -- 172.21.15.45:0/296080933 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f7c00cd70 con 0x7f3f9006a7e0 2024-03-28T17:46:29.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.544+0000 7f3f8d600700 1 -- 172.21.15.45:0/296080933 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c018750 con 0x7f3f9006a7e0 2024-03-28T17:46:29.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.544+0000 7f3f97c00700 1 -- 172.21.15.45:0/296080933 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f9019b360 con 0x7f3f9006a7e0 2024-03-28T17:46:29.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.544+0000 7f3f97c00700 1 -- 172.21.15.45:0/296080933 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f901a7620 con 0x7f3f9006a7e0 2024-03-28T17:46:29.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.545+0000 7f3f97c00700 1 -- 172.21.15.45:0/296080933 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f90068e50 con 0x7f3f9006a7e0 2024-03-28T17:46:29.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.547+0000 7f3f8d600700 1 -- 172.21.15.45:0/296080933 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f3f7c010430 con 0x7f3f9006a7e0 2024-03-28T17:46:29.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.549+0000 7f3f8d600700 1 --2- 172.21.15.45:0/296080933 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3f780a29a0 0x7f3f780a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:29.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.549+0000 7f3f8d600700 1 -- 172.21.15.45:0/296080933 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f3f7c0ced90 con 0x7f3f9006a7e0 2024-03-28T17:46:29.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.549+0000 7f3f95200700 1 --2- 172.21.15.45:0/296080933 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3f780a29a0 0x7f3f780a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.550+0000 7f3f95200700 1 --2- 172.21.15.45:0/296080933 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3f780a29a0 0x7f3f780a4e60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3f84009750 tx=0x7f3f84009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:29.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.555+0000 7f3f8d600700 1 -- 172.21.15.45:0/296080933 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3f7c09bd80 con 0x7f3f9006a7e0 2024-03-28T17:46:29.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:29 smithi045 ceph-mon[31773]: pgmap v325: 97 pgs: 97 active+clean; 632 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 7.6 KiB/s wr, 17 op/s 2024-03-28T17:46:29.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:29 smithi071 ceph-mon[36597]: pgmap v325: 97 pgs: 97 active+clean; 632 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 7.6 KiB/s wr, 17 op/s 2024-03-28T17:46:29.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:29 smithi151 ceph-mon[36481]: pgmap v325: 97 pgs: 97 active+clean; 632 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 7.6 KiB/s wr, 17 op/s 2024-03-28T17:46:29.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.851+0000 7f3f97c00700 1 -- 172.21.15.45:0/296080933 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f3f901a7910 con 0x7f3f9006a7e0 2024-03-28T17:46:29.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.860+0000 7f3f8d600700 1 -- 172.21.15.45:0/296080933 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+262661 (secure 0 0 0) 0x7f3f7c09b680 con 0x7f3f9006a7e0 2024-03-28T17:46:29.865 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:46:29.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.867+0000 7f3f97c00700 1 -- 172.21.15.45:0/296080933 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3f780a29a0 msgr2=0x7f3f780a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:29.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.867+0000 7f3f97c00700 1 --2- 172.21.15.45:0/296080933 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3f780a29a0 0x7f3f780a4e60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3f84009750 tx=0x7f3f84009310 comp rx=0 tx=0).stop 2024-03-28T17:46:29.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.867+0000 7f3f97c00700 1 -- 172.21.15.45:0/296080933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f9006a7e0 msgr2=0x7f3f901a0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:29.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.867+0000 7f3f97c00700 1 --2- 172.21.15.45:0/296080933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f9006a7e0 0x7f3f901a0c90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c00e9a0 tx=0x7f3f7c00ecb0 comp rx=0 tx=0).stop 2024-03-28T17:46:29.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.867+0000 7f3f97c00700 1 -- 172.21.15.45:0/296080933 shutdown_connections 2024-03-28T17:46:29.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.867+0000 7f3f97c00700 1 --2- 172.21.15.45:0/296080933 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3f780a29a0 0x7f3f780a4e60 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:29.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.867+0000 7f3f97c00700 1 --2- 172.21.15.45:0/296080933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f901081e0 0x7f3f9019a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:29.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.867+0000 7f3f97c00700 1 --2- 172.21.15.45:0/296080933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f90069e40 0x7f3f901a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:29.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.867+0000 7f3f97c00700 1 --2- 172.21.15.45:0/296080933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f9006a7e0 0x7f3f901a0c90 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:29.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.867+0000 7f3f97c00700 1 -- 172.21.15.45:0/296080933 >> 172.21.15.45:0/296080933 conn(0x7f3f90077fc0 msgr2=0x7f3f9010d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:29.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.867+0000 7f3f97c00700 1 -- 172.21.15.45:0/296080933 shutdown_connections 2024-03-28T17:46:29.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:29.868+0000 7f3f97c00700 1 -- 172.21.15.45:0/296080933 wait complete. 2024-03-28T17:46:30.628 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:30 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/296080933' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T17:46:30.685 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1246060167},{"type":"v1","addr":"172.21.15.45:6801","nonce":1246060167}]},"active_addr":"172.21.15.45:6801/1246060167","active_change":"2024-03-28T17:38:07.505821+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.45:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":1279147680}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":4023553990}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":1305561640}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":2386239638}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":2266470219}]}]} 2024-03-28T17:46:30.689 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-28T17:46:30.690 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-28T17:46:30.690 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T17:46:30.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:30 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/296080933' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T17:46:30.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:30 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/296080933' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T17:46:31.172 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:46:31.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:31 smithi045 ceph-mon[31773]: pgmap v326: 97 pgs: 97 active+clean; 632 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 7.2 KiB/s wr, 15 op/s 2024-03-28T17:46:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:31 smithi071 ceph-mon[36597]: pgmap v326: 97 pgs: 97 active+clean; 632 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 7.2 KiB/s wr, 15 op/s 2024-03-28T17:46:31.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:31 smithi151 ceph-mon[36481]: pgmap v326: 97 pgs: 97 active+clean; 632 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 7.2 KiB/s wr, 15 op/s 2024-03-28T17:46:33.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.459+0000 7ff91f800700 1 -- 172.21.15.45:0/3404405733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff918104f90 msgr2=0x7ff918108fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:33.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.459+0000 7ff91f800700 1 --2- 172.21.15.45:0/3404405733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff918104f90 0x7ff918108fe0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff904009920 tx=0x7ff904009c30 comp rx=0 tx=0).stop 2024-03-28T17:46:33.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.459+0000 7ff91f800700 1 -- 172.21.15.45:0/3404405733 shutdown_connections 2024-03-28T17:46:33.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.459+0000 7ff91f800700 1 --2- 172.21.15.45:0/3404405733 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9181045e0 0x7ff9181049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:33.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.459+0000 7ff91f800700 1 --2- 172.21.15.45:0/3404405733 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9181096c0 0x7ff918111290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:33.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.459+0000 7ff91f800700 1 --2- 172.21.15.45:0/3404405733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff918104f90 0x7ff918108fe0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:33.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.459+0000 7ff91f800700 1 -- 172.21.15.45:0/3404405733 >> 172.21.15.45:0/3404405733 conn(0x7ff9180ffe70 msgr2=0x7ff918102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:33.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.460+0000 7ff91f800700 1 -- 172.21.15.45:0/3404405733 shutdown_connections 2024-03-28T17:46:33.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.460+0000 7ff91f800700 1 -- 172.21.15.45:0/3404405733 wait complete. 2024-03-28T17:46:33.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.461+0000 7ff91f800700 1 Processor -- start 2024-03-28T17:46:33.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.461+0000 7ff91f800700 1 -- start start 2024-03-28T17:46:33.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.461+0000 7ff91f800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9181045e0 0x7ff9181a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:33.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.461+0000 7ff91f800700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff918104f90 0x7ff9181a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:33.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.461+0000 7ff91f800700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9181096c0 0x7ff91819aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:33.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.461+0000 7ff91f800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff918114680 con 0x7ff9181045e0 2024-03-28T17:46:33.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.461+0000 7ff91f800700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff918114500 con 0x7ff9181096c0 2024-03-28T17:46:33.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.461+0000 7ff91f800700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff918114800 con 0x7ff918104f90 2024-03-28T17:46:33.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.462+0000 7ff91ce00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9181045e0 0x7ff9181a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:33.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.462+0000 7ff917e00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff918104f90 0x7ff9181a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:33.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.462+0000 7ff917e00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff918104f90 0x7ff9181a0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:48630/0 (socket says 172.21.15.45:48630) 2024-03-28T17:46:33.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.462+0000 7ff917e00700 1 -- 172.21.15.45:0/499185431 learned_addr learned my addr 172.21.15.45:0/499185431 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:33.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.462+0000 7ff91d800700 1 --2- 172.21.15.45:0/499185431 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9181096c0 0x7ff91819aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:33.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.462+0000 7ff917e00700 1 -- 172.21.15.45:0/499185431 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9181096c0 msgr2=0x7ff91819aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:33.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.462+0000 7ff917e00700 1 --2- 172.21.15.45:0/499185431 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9181096c0 0x7ff91819aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:33.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.462+0000 7ff917e00700 1 -- 172.21.15.45:0/499185431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9181045e0 msgr2=0x7ff9181a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:33.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.462+0000 7ff917e00700 1 --2- 172.21.15.45:0/499185431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9181045e0 0x7ff9181a0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:33.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.462+0000 7ff917e00700 1 -- 172.21.15.45:0/499185431 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91819b1a0 con 0x7ff918104f90 2024-03-28T17:46:33.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.462+0000 7ff91d800700 1 --2- 172.21.15.45:0/499185431 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9181096c0 0x7ff91819aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:46:33.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.462+0000 7ff917e00700 1 --2- 172.21.15.45:0/499185431 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff918104f90 0x7ff9181a0d70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff904009230 tx=0x7ff90400fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:33.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.462+0000 7ff915600700 1 -- 172.21.15.45:0/499185431 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff904027070 con 0x7ff918104f90 2024-03-28T17:46:33.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.463+0000 7ff915600700 1 -- 172.21.15.45:0/499185431 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff904005620 con 0x7ff918104f90 2024-03-28T17:46:33.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.463+0000 7ff915600700 1 -- 172.21.15.45:0/499185431 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9040049c0 con 0x7ff918104f90 2024-03-28T17:46:33.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.463+0000 7ff91f800700 1 -- 172.21.15.45:0/499185431 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff91819b430 con 0x7ff918104f90 2024-03-28T17:46:33.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.463+0000 7ff91f800700 1 -- 172.21.15.45:0/499185431 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9181a7700 con 0x7ff918104f90 2024-03-28T17:46:33.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.464+0000 7ff91f800700 1 -- 172.21.15.45:0/499185431 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff918068e50 con 0x7ff918104f90 2024-03-28T17:46:33.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.466+0000 7ff915600700 1 -- 172.21.15.45:0/499185431 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7ff90402a070 con 0x7ff918104f90 2024-03-28T17:46:33.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.467+0000 7ff915600700 1 --2- 172.21.15.45:0/499185431 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff9000a29a0 0x7ff9000a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:33.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.467+0000 7ff915600700 1 -- 172.21.15.45:0/499185431 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff9040cf4b0 con 0x7ff918104f90 2024-03-28T17:46:33.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.467+0000 7ff91ce00700 1 --2- 172.21.15.45:0/499185431 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff9000a29a0 0x7ff9000a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:33.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.468+0000 7ff91ce00700 1 --2- 172.21.15.45:0/499185431 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff9000a29a0 0x7ff9000a4e60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff9181002c0 tx=0x7ff90c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:33.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.469+0000 7ff915600700 1 -- 172.21.15.45:0/499185431 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff904013030 con 0x7ff918104f90 2024-03-28T17:46:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:33 smithi151 ceph-mon[36481]: pgmap v327: 97 pgs: 97 active+clean; 632 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 6.6 KiB/s wr, 12 op/s 2024-03-28T17:46:33.612 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:33 smithi071 ceph-mon[36597]: pgmap v327: 97 pgs: 97 active+clean; 632 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 6.6 KiB/s wr, 12 op/s 2024-03-28T17:46:33.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:33 smithi045 ceph-mon[31773]: pgmap v327: 97 pgs: 97 active+clean; 632 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 6.6 KiB/s wr, 12 op/s 2024-03-28T17:46:33.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.695+0000 7ff91f800700 1 -- 172.21.15.45:0/499185431 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff91802da30 con 0x7ff918104f90 2024-03-28T17:46:33.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.697+0000 7ff915600700 1 -- 172.21.15.45:0/499185431 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22208 (secure 0 0 0) 0x7ff90409c530 con 0x7ff918104f90 2024-03-28T17:46:33.698 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:46:33.698 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":85,"fsid":"9f4888ec-ed29-11ee-95d0-87774f69a715","created":"2024-03-28T17:36:51.623185+0000","modified":"2024-03-28T17:46:13.741486+0000","last_up_change":"2024-03-28T17:45:14.043432+0000","last_in_change":"2024-03-28T17:44:53.654489+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T17:40:54.208375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T17:46:10.564519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T17:46:11.734367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"74c56bc2-4169-4919-afd4-2ebd236f23e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6803","nonce":3795754023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6805","nonce":3795754023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6809","nonce":3795754023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6807","nonce":3795754023}]},"public_addr":"172.21.15.45:6803/3795754023","cluster_addr":"172.21.15.45:6805/3795754023","heartbeat_back_addr":"172.21.15.45:6809/3795754023","heartbeat_front_addr":"172.21.15.45:6807/3795754023","state":["exists","up"]},{"osd":1,"uuid":"78dcc21b-75b2-453d-b690-17b55b8ed1a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6801","nonce":2014529906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6803","nonce":2014529906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6807","nonce":2014529906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6805","nonce":2014529906}]},"public_addr":"172.21.15.71:6801/2014529906","cluster_addr":"172.21.15.71:6803/2014529906","heartbeat_back_addr":"172.21.15.71:6807/2014529906","heartbeat_front_addr":"172.21.15.71:6805/2014529906","state":["exists","up"]},{"osd":2,"uuid":"ca9f3f36-b71e-4962-a94f-cedf2f12eb7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6801","nonce":1850933775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6803","nonce":1850933775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6807","nonce":1850933775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6805","nonce":1850933775}]},"public_addr":"172.21.15.151:6801/1850933775","cluster_addr":"172.21.15.151:6803/1850933775","heartbeat_back_addr":"172.21.15.151:6807/1850933775","heartbeat_front_addr":"172.21.15.151:6805/1850933775","state":["exists","up"]},{"osd":3,"uuid":"d3f21f01-2973-413e-8daa-9974b8a44af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6811","nonce":3021569975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6813","nonce":3021569975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6817","nonce":3021569975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6815","nonce":3021569975}]},"public_addr":"172.21.15.45:6811/3021569975","cluster_addr":"172.21.15.45:6813/3021569975","heartbeat_back_addr":"172.21.15.45:6817/3021569975","heartbeat_front_addr":"172.21.15.45:6815/3021569975","state":["exists","up"]},{"osd":4,"uuid":"4a336f42-0605-492b-ad88-56cb02acdcf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6809","nonce":2069578333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6811","nonce":2069578333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6815","nonce":2069578333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6813","nonce":2069578333}]},"public_addr":"172.21.15.71:6809/2069578333","cluster_addr":"172.21.15.71:6811/2069578333","heartbeat_back_addr":"172.21.15.71:6815/2069578333","heartbeat_front_addr":"172.21.15.71:6813/2069578333","state":["exists","up"]},{"osd":5,"uuid":"c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6809","nonce":2629217227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6811","nonce":2629217227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6815","nonce":2629217227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6813","nonce":2629217227}]},"public_addr":"172.21.15.151:6809/2629217227","cluster_addr":"172.21.15.151:6811/2629217227","heartbeat_back_addr":"172.21.15.151:6815/2629217227","heartbeat_front_addr":"172.21.15.151:6813/2629217227","state":["exists","up"]},{"osd":6,"uuid":"58a6c5bd-a8b7-4892-b346-bcd7994bcb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6819","nonce":2077907039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6821","nonce":2077907039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6825","nonce":2077907039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6823","nonce":2077907039}]},"public_addr":"172.21.15.45:6819/2077907039","cluster_addr":"172.21.15.45:6821/2077907039","heartbeat_back_addr":"172.21.15.45:6825/2077907039","heartbeat_front_addr":"172.21.15.45:6823/2077907039","state":["exists","up"]},{"osd":7,"uuid":"f7a45d52-807b-4407-bb04-787b2a52a0ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6817","nonce":2000211251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6819","nonce":2000211251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6823","nonce":2000211251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6821","nonce":2000211251}]},"public_addr":"172.21.15.71:6817/2000211251","cluster_addr":"172.21.15.71:6819/2000211251","heartbeat_back_addr":"172.21.15.71:6823/2000211251","heartbeat_front_addr":"172.21.15.71:6821/2000211251","state":["exists","up"]},{"osd":8,"uuid":"88abcb0f-f189-49be-8315-c700c77369f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6817","nonce":3665460137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6819","nonce":3665460137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6823","nonce":3665460137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6821","nonce":3665460137}]},"public_addr":"172.21.15.151:6817/3665460137","cluster_addr":"172.21.15.151:6819/3665460137","heartbeat_back_addr":"172.21.15.151:6823/3665460137","heartbeat_front_addr":"172.21.15.151:6821/3665460137","state":["exists","up"]},{"osd":9,"uuid":"ed3fc096-961e-4392-abc2-c5aa4d050117","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6827","nonce":2122322684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6829","nonce":2122322684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6833","nonce":2122322684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6831","nonce":2122322684}]},"public_addr":"172.21.15.45:6827/2122322684","cluster_addr":"172.21.15.45:6829/2122322684","heartbeat_back_addr":"172.21.15.45:6833/2122322684","heartbeat_front_addr":"172.21.15.45:6831/2122322684","state":["exists","up"]},{"osd":10,"uuid":"ba94762b-bc5c-4c97-9cc1-17e5651c14b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6825","nonce":1099494037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6827","nonce":1099494037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6831","nonce":1099494037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6829","nonce":1099494037}]},"public_addr":"172.21.15.71:6825/1099494037","cluster_addr":"172.21.15.71:6827/1099494037","heartbeat_back_addr":"172.21.15.71:6831/1099494037","heartbeat_front_addr":"172.21.15.71:6829/1099494037","state":["exists","up"]},{"osd":11,"uuid":"1d178378-b9f5-40f6-99ef-3d3d445da640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6825","nonce":2941055364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6827","nonce":2941055364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6831","nonce":2941055364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6829","nonce":2941055364}]},"public_addr":"172.21.15.151:6825/2941055364","cluster_addr":"172.21.15.151:6827/2941055364","heartbeat_back_addr":"172.21.15.151:6831/2941055364","heartbeat_front_addr":"172.21.15.151:6829/2941055364","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:02.269586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:27.819386+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:51.552277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:25.052674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:51.873529+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:17.127867+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:51.588249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:18.381450+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:44.140534+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:17.634362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:45.469340+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:45:12.003433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3533611957":"2024-03-29T17:38:07.505611+0000","172.21.15.45:0/3625922430":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6801/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/482610468":"2024-03-29T17:37:32.424229+0000","172.21.15.45:6801/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3649804714":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/1198364079":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3211303958":"2024-03-29T17:38:07.505611+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T17:46:33.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.700+0000 7ff91f800700 1 -- 172.21.15.45:0/499185431 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff9000a29a0 msgr2=0x7ff9000a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:33.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.700+0000 7ff91f800700 1 --2- 172.21.15.45:0/499185431 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff9000a29a0 0x7ff9000a4e60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff9181002c0 tx=0x7ff90c009260 comp rx=0 tx=0).stop 2024-03-28T17:46:33.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.700+0000 7ff91f800700 1 -- 172.21.15.45:0/499185431 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff918104f90 msgr2=0x7ff9181a0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:33.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.700+0000 7ff91f800700 1 --2- 172.21.15.45:0/499185431 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff918104f90 0x7ff9181a0d70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff904009230 tx=0x7ff90400fa00 comp rx=0 tx=0).stop 2024-03-28T17:46:33.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.700+0000 7ff91f800700 1 -- 172.21.15.45:0/499185431 shutdown_connections 2024-03-28T17:46:33.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.700+0000 7ff91f800700 1 --2- 172.21.15.45:0/499185431 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff9000a29a0 0x7ff9000a4e60 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:33.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.700+0000 7ff91f800700 1 --2- 172.21.15.45:0/499185431 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9181096c0 0x7ff91819aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:33.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.700+0000 7ff91f800700 1 --2- 172.21.15.45:0/499185431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9181045e0 0x7ff9181a0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:33.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.700+0000 7ff91f800700 1 --2- 172.21.15.45:0/499185431 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff918104f90 0x7ff9181a0d70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:33.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.700+0000 7ff91f800700 1 -- 172.21.15.45:0/499185431 >> 172.21.15.45:0/499185431 conn(0x7ff9180ffe70 msgr2=0x7ff918102150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:33.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.700+0000 7ff91f800700 1 -- 172.21.15.45:0/499185431 shutdown_connections 2024-03-28T17:46:33.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:33.700+0000 7ff91f800700 1 -- 172.21.15.45:0/499185431 wait complete. 2024-03-28T17:46:34.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:34 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/499185431' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:46:34.563 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-28T17:46:34.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T17:46:34.624 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:34 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/499185431' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:46:34.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:34 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/499185431' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:46:35.029 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:46:35.539 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:35 smithi045 ceph-mon[31773]: pgmap v328: 97 pgs: 97 active+clean; 636 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.4 KiB/s wr, 9 op/s 2024-03-28T17:46:35.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:35 smithi071 ceph-mon[36597]: pgmap v328: 97 pgs: 97 active+clean; 636 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.4 KiB/s wr, 9 op/s 2024-03-28T17:46:35.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:35 smithi151 ceph-mon[36481]: pgmap v328: 97 pgs: 97 active+clean; 636 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.4 KiB/s wr, 9 op/s 2024-03-28T17:46:37.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.353+0000 7fb201e00700 1 -- 172.21.15.45:0/28998560 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1fc103230 msgr2=0x7fb1fc103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:37.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.353+0000 7fb201e00700 1 --2- 172.21.15.45:0/28998560 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1fc103230 0x7fb1fc103610 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec009990 tx=0x7fb1ec009ca0 comp rx=0 tx=0).stop 2024-03-28T17:46:37.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.353+0000 7fb201e00700 1 -- 172.21.15.45:0/28998560 shutdown_connections 2024-03-28T17:46:37.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.353+0000 7fb201e00700 1 --2- 172.21.15.45:0/28998560 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1fc10c660 0x7fb1fc10eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:37.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.353+0000 7fb201e00700 1 --2- 172.21.15.45:0/28998560 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1fc103230 0x7fb1fc103610 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:37.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.353+0000 7fb201e00700 1 --2- 172.21.15.45:0/28998560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1fc103b50 0x7fb1fc10c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:37.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.353+0000 7fb201e00700 1 -- 172.21.15.45:0/28998560 >> 172.21.15.45:0/28998560 conn(0x7fb1fc0fca60 msgr2=0x7fb1fc0fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:37.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.353+0000 7fb201e00700 1 -- 172.21.15.45:0/28998560 shutdown_connections 2024-03-28T17:46:37.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.353+0000 7fb201e00700 1 -- 172.21.15.45:0/28998560 wait complete. 2024-03-28T17:46:37.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.354+0000 7fb201e00700 1 Processor -- start 2024-03-28T17:46:37.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.355+0000 7fb201e00700 1 -- start start 2024-03-28T17:46:37.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.355+0000 7fb201e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1fc103230 0x7fb1fc1a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:37.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.355+0000 7fb201e00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1fc103b50 0x7fb1fc1a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:37.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.355+0000 7fb201e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1fc10c660 0x7fb1fc19a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:37.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.355+0000 7fb201e00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc112410 con 0x7fb1fc103230 2024-03-28T17:46:37.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.355+0000 7fb201e00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc112290 con 0x7fb1fc10c660 2024-03-28T17:46:37.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.355+0000 7fb201e00700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc112590 con 0x7fb1fc103b50 2024-03-28T17:46:37.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.356+0000 7fb1f3e00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1fc103b50 0x7fb1fc1a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:37.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.356+0000 7fb1f3e00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1fc103b50 0x7fb1fc1a0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:59592/0 (socket says 172.21.15.45:59592) 2024-03-28T17:46:37.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.356+0000 7fb1f3e00700 1 -- 172.21.15.45:0/3208266259 learned_addr learned my addr 172.21.15.45:0/3208266259 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:37.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.356+0000 7fb201400700 1 --2- 172.21.15.45:0/3208266259 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1fc10c660 0x7fb1fc19a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:37.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.356+0000 7fb200a00700 1 --2- 172.21.15.45:0/3208266259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1fc103230 0x7fb1fc1a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:37.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.356+0000 7fb1f3e00700 1 -- 172.21.15.45:0/3208266259 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1fc10c660 msgr2=0x7fb1fc19a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:37.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.356+0000 7fb1f3e00700 1 --2- 172.21.15.45:0/3208266259 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1fc10c660 0x7fb1fc19a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:37.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.356+0000 7fb1f3e00700 1 -- 172.21.15.45:0/3208266259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1fc103230 msgr2=0x7fb1fc1a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:37.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.356+0000 7fb1f3e00700 1 --2- 172.21.15.45:0/3208266259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1fc103230 0x7fb1fc1a06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:37.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.356+0000 7fb1f3e00700 1 -- 172.21.15.45:0/3208266259 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1fc19b000 con 0x7fb1fc103b50 2024-03-28T17:46:37.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.356+0000 7fb201400700 1 --2- 172.21.15.45:0/3208266259 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1fc10c660 0x7fb1fc19a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:46:37.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.356+0000 7fb1f3e00700 1 --2- 172.21.15.45:0/3208266259 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1fc103b50 0x7fb1fc1a0c20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb1e400e9c0 tx=0x7fb1e400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:37.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.357+0000 7fb1fa000700 1 -- 172.21.15.45:0/3208266259 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4004a30 con 0x7fb1fc103b50 2024-03-28T17:46:37.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.357+0000 7fb1fa000700 1 -- 172.21.15.45:0/3208266259 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb1e400cca0 con 0x7fb1fc103b50 2024-03-28T17:46:37.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.357+0000 7fb1fa000700 1 -- 172.21.15.45:0/3208266259 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4005610 con 0x7fb1fc103b50 2024-03-28T17:46:37.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.357+0000 7fb201e00700 1 -- 172.21.15.45:0/3208266259 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1fc19b2f0 con 0x7fb1fc103b50 2024-03-28T17:46:37.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.357+0000 7fb201e00700 1 -- 172.21.15.45:0/3208266259 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1fc1a75b0 con 0x7fb1fc103b50 2024-03-28T17:46:37.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.358+0000 7fb201e00700 1 -- 172.21.15.45:0/3208266259 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1fc068e50 con 0x7fb1fc103b50 2024-03-28T17:46:37.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.360+0000 7fb1fa000700 1 -- 172.21.15.45:0/3208266259 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fb1e400c7c0 con 0x7fb1fc103b50 2024-03-28T17:46:37.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.362+0000 7fb1fa000700 1 --2- 172.21.15.45:0/3208266259 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb1e00a29a0 0x7fb1e00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:37.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.362+0000 7fb1fa000700 1 -- 172.21.15.45:0/3208266259 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb1e40cee00 con 0x7fb1fc103b50 2024-03-28T17:46:37.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.362+0000 7fb200a00700 1 --2- 172.21.15.45:0/3208266259 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb1e00a29a0 0x7fb1e00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:37.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.363+0000 7fb200a00700 1 --2- 172.21.15.45:0/3208266259 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb1e00a29a0 0x7fb1e00a4e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc19b9e0 tx=0x7fb1ec01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:37.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.365+0000 7fb1fa000700 1 -- 172.21.15.45:0/3208266259 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb1e4053740 con 0x7fb1fc103b50 2024-03-28T17:46:37.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.593+0000 7fb201e00700 1 -- 172.21.15.45:0/3208266259 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1fc19bdd0 con 0x7fb1fc103b50 2024-03-28T17:46:37.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:37 smithi045 ceph-mon[31773]: pgmap v329: 97 pgs: 97 active+clean; 636 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-28T17:46:37.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.594+0000 7fb1fa000700 1 -- 172.21.15.45:0/3208266259 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22208 (secure 0 0 0) 0x7fb1e409bdf0 con 0x7fb1fc103b50 2024-03-28T17:46:37.596 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:46:37.596 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":85,"fsid":"9f4888ec-ed29-11ee-95d0-87774f69a715","created":"2024-03-28T17:36:51.623185+0000","modified":"2024-03-28T17:46:13.741486+0000","last_up_change":"2024-03-28T17:45:14.043432+0000","last_in_change":"2024-03-28T17:44:53.654489+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T17:40:54.208375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T17:46:10.564519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T17:46:11.734367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"74c56bc2-4169-4919-afd4-2ebd236f23e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6803","nonce":3795754023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6805","nonce":3795754023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6809","nonce":3795754023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6807","nonce":3795754023}]},"public_addr":"172.21.15.45:6803/3795754023","cluster_addr":"172.21.15.45:6805/3795754023","heartbeat_back_addr":"172.21.15.45:6809/3795754023","heartbeat_front_addr":"172.21.15.45:6807/3795754023","state":["exists","up"]},{"osd":1,"uuid":"78dcc21b-75b2-453d-b690-17b55b8ed1a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6801","nonce":2014529906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6803","nonce":2014529906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6807","nonce":2014529906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6805","nonce":2014529906}]},"public_addr":"172.21.15.71:6801/2014529906","cluster_addr":"172.21.15.71:6803/2014529906","heartbeat_back_addr":"172.21.15.71:6807/2014529906","heartbeat_front_addr":"172.21.15.71:6805/2014529906","state":["exists","up"]},{"osd":2,"uuid":"ca9f3f36-b71e-4962-a94f-cedf2f12eb7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6801","nonce":1850933775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6803","nonce":1850933775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6807","nonce":1850933775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6805","nonce":1850933775}]},"public_addr":"172.21.15.151:6801/1850933775","cluster_addr":"172.21.15.151:6803/1850933775","heartbeat_back_addr":"172.21.15.151:6807/1850933775","heartbeat_front_addr":"172.21.15.151:6805/1850933775","state":["exists","up"]},{"osd":3,"uuid":"d3f21f01-2973-413e-8daa-9974b8a44af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6811","nonce":3021569975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6813","nonce":3021569975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6817","nonce":3021569975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6815","nonce":3021569975}]},"public_addr":"172.21.15.45:6811/3021569975","cluster_addr":"172.21.15.45:6813/3021569975","heartbeat_back_addr":"172.21.15.45:6817/3021569975","heartbeat_front_addr":"172.21.15.45:6815/3021569975","state":["exists","up"]},{"osd":4,"uuid":"4a336f42-0605-492b-ad88-56cb02acdcf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6809","nonce":2069578333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6811","nonce":2069578333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6815","nonce":2069578333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6813","nonce":2069578333}]},"public_addr":"172.21.15.71:6809/2069578333","cluster_addr":"172.21.15.71:6811/2069578333","heartbeat_back_addr":"172.21.15.71:6815/2069578333","heartbeat_front_addr":"172.21.15.71:6813/2069578333","state":["exists","up"]},{"osd":5,"uuid":"c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6809","nonce":2629217227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6811","nonce":2629217227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6815","nonce":2629217227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6813","nonce":2629217227}]},"public_addr":"172.21.15.151:6809/2629217227","cluster_addr":"172.21.15.151:6811/2629217227","heartbeat_back_addr":"172.21.15.151:6815/2629217227","heartbeat_front_addr":"172.21.15.151:6813/2629217227","state":["exists","up"]},{"osd":6,"uuid":"58a6c5bd-a8b7-4892-b346-bcd7994bcb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6819","nonce":2077907039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6821","nonce":2077907039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6825","nonce":2077907039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6823","nonce":2077907039}]},"public_addr":"172.21.15.45:6819/2077907039","cluster_addr":"172.21.15.45:6821/2077907039","heartbeat_back_addr":"172.21.15.45:6825/2077907039","heartbeat_front_addr":"172.21.15.45:6823/2077907039","state":["exists","up"]},{"osd":7,"uuid":"f7a45d52-807b-4407-bb04-787b2a52a0ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6817","nonce":2000211251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6819","nonce":2000211251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6823","nonce":2000211251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6821","nonce":2000211251}]},"public_addr":"172.21.15.71:6817/2000211251","cluster_addr":"172.21.15.71:6819/2000211251","heartbeat_back_addr":"172.21.15.71:6823/2000211251","heartbeat_front_addr":"172.21.15.71:6821/2000211251","state":["exists","up"]},{"osd":8,"uuid":"88abcb0f-f189-49be-8315-c700c77369f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6817","nonce":3665460137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6819","nonce":3665460137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6823","nonce":3665460137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6821","nonce":3665460137}]},"public_addr":"172.21.15.151:6817/3665460137","cluster_addr":"172.21.15.151:6819/3665460137","heartbeat_back_addr":"172.21.15.151:6823/3665460137","heartbeat_front_addr":"172.21.15.151:6821/3665460137","state":["exists","up"]},{"osd":9,"uuid":"ed3fc096-961e-4392-abc2-c5aa4d050117","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6827","nonce":2122322684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6829","nonce":2122322684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6833","nonce":2122322684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6831","nonce":2122322684}]},"public_addr":"172.21.15.45:6827/2122322684","cluster_addr":"172.21.15.45:6829/2122322684","heartbeat_back_addr":"172.21.15.45:6833/2122322684","heartbeat_front_addr":"172.21.15.45:6831/2122322684","state":["exists","up"]},{"osd":10,"uuid":"ba94762b-bc5c-4c97-9cc1-17e5651c14b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6825","nonce":1099494037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6827","nonce":1099494037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6831","nonce":1099494037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6829","nonce":1099494037}]},"public_addr":"172.21.15.71:6825/1099494037","cluster_addr":"172.21.15.71:6827/1099494037","heartbeat_back_addr":"172.21.15.71:6831/1099494037","heartbeat_front_addr":"172.21.15.71:6829/1099494037","state":["exists","up"]},{"osd":11,"uuid":"1d178378-b9f5-40f6-99ef-3d3d445da640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6825","nonce":2941055364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6827","nonce":2941055364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6831","nonce":2941055364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6829","nonce":2941055364}]},"public_addr":"172.21.15.151:6825/2941055364","cluster_addr":"172.21.15.151:6827/2941055364","heartbeat_back_addr":"172.21.15.151:6831/2941055364","heartbeat_front_addr":"172.21.15.151:6829/2941055364","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:02.269586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:27.819386+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:51.552277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:25.052674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:51.873529+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:17.127867+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:51.588249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:18.381450+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:44.140534+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:17.634362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:45.469340+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:45:12.003433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3533611957":"2024-03-29T17:38:07.505611+0000","172.21.15.45:0/3625922430":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6801/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/482610468":"2024-03-29T17:37:32.424229+0000","172.21.15.45:6801/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3649804714":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/1198364079":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3211303958":"2024-03-29T17:38:07.505611+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T17:46:37.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.599+0000 7fb201e00700 1 -- 172.21.15.45:0/3208266259 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb1e00a29a0 msgr2=0x7fb1e00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:37.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.599+0000 7fb201e00700 1 --2- 172.21.15.45:0/3208266259 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb1e00a29a0 0x7fb1e00a4e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc19b9e0 tx=0x7fb1ec01a040 comp rx=0 tx=0).stop 2024-03-28T17:46:37.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.599+0000 7fb201e00700 1 -- 172.21.15.45:0/3208266259 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1fc103b50 msgr2=0x7fb1fc1a0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:37.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.599+0000 7fb201e00700 1 --2- 172.21.15.45:0/3208266259 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1fc103b50 0x7fb1fc1a0c20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb1e400e9c0 tx=0x7fb1e400ecd0 comp rx=0 tx=0).stop 2024-03-28T17:46:37.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.599+0000 7fb201e00700 1 -- 172.21.15.45:0/3208266259 shutdown_connections 2024-03-28T17:46:37.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.599+0000 7fb201e00700 1 --2- 172.21.15.45:0/3208266259 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb1e00a29a0 0x7fb1e00a4e60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:37.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.599+0000 7fb201e00700 1 --2- 172.21.15.45:0/3208266259 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1fc10c660 0x7fb1fc19a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:37.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.599+0000 7fb201e00700 1 --2- 172.21.15.45:0/3208266259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1fc103230 0x7fb1fc1a06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:37.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.599+0000 7fb201e00700 1 --2- 172.21.15.45:0/3208266259 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1fc103b50 0x7fb1fc1a0c20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:37.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.599+0000 7fb201e00700 1 -- 172.21.15.45:0/3208266259 >> 172.21.15.45:0/3208266259 conn(0x7fb1fc0fca60 msgr2=0x7fb1fc10e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:37.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.599+0000 7fb201e00700 1 -- 172.21.15.45:0/3208266259 shutdown_connections 2024-03-28T17:46:37.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:37.600+0000 7fb201e00700 1 -- 172.21.15.45:0/3208266259 wait complete. 2024-03-28T17:46:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:37 smithi071 ceph-mon[36597]: pgmap v329: 97 pgs: 97 active+clean; 636 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-28T17:46:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:37 smithi151 ceph-mon[36481]: pgmap v329: 97 pgs: 97 active+clean; 636 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-28T17:46:38.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:38 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3208266259' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:46:38.648 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-28T17:46:38.649 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-28T17:46:38.649 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-28T17:46:38.650 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-28T17:46:38.650 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-28T17:46:38.651 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-28T17:46:38.651 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-28T17:46:38.651 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-28T17:46:38.652 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-28T17:46:38.652 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-28T17:46:38.653 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-28T17:46:38.653 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-28T17:46:38.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:38 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3208266259' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:46:38.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:38 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3208266259' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:46:39.652 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:46:39.671 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:39 smithi045 ceph-mon[31773]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-28T17:46:39.721 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:46:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:39 smithi071 ceph-mon[36597]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-28T17:46:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:39 smithi151 ceph-mon[36481]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-28T17:46:40.847 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:46:40.850 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:46:40.864 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:46:40.866 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:46:40.880 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:46:40.884 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:46:40.884 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:46:40.884 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:46:40.889 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:46:41.418 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:46:41.565 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:41 smithi045 ceph-mon[31773]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 852 B/s wr, 6 op/s 2024-03-28T17:46:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:41 smithi071 ceph-mon[36597]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 852 B/s wr, 6 op/s 2024-03-28T17:46:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:41 smithi151 ceph-mon[36481]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 852 B/s wr, 6 op/s 2024-03-28T17:46:43.462 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:43 smithi045 ceph-mon[31773]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 852 B/s wr, 6 op/s 2024-03-28T17:46:43.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:43 smithi071 ceph-mon[36597]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 852 B/s wr, 6 op/s 2024-03-28T17:46:43.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:43 smithi151 ceph-mon[36481]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 852 B/s wr, 6 op/s 2024-03-28T17:46:45.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:45 smithi045 ceph-mon[31773]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 852 B/s wr, 6 op/s 2024-03-28T17:46:45.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:45 smithi071 ceph-mon[36597]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 852 B/s wr, 6 op/s 2024-03-28T17:46:45.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:45 smithi151 ceph-mon[36481]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 852 B/s wr, 6 op/s 2024-03-28T17:46:47.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:47 smithi045 ceph-mon[31773]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T17:46:47.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:47 smithi071 ceph-mon[36597]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T17:46:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:47 smithi151 ceph-mon[36481]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T17:46:49.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:49 smithi045 ceph-mon[31773]: pgmap v335: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-28T17:46:49.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:49 smithi071 ceph-mon[36597]: pgmap v335: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-28T17:46:49.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:49 smithi151 ceph-mon[36481]: pgmap v335: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-28T17:46:49.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.247+0000 7f3ff7e00700 1 -- 172.21.15.45:0/2095713819 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3ff81044e0 msgr2=0x7f3ff81048c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:49.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.247+0000 7f3ff7e00700 1 --2- 172.21.15.45:0/2095713819 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3ff81044e0 0x7f3ff81048c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3fe8009920 tx=0x7f3fe8009c30 comp rx=0 tx=0).stop 2024-03-28T17:46:49.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.247+0000 7f3ff7e00700 1 -- 172.21.15.45:0/2095713819 shutdown_connections 2024-03-28T17:46:49.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.247+0000 7f3ff7e00700 1 --2- 172.21.15.45:0/2095713819 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3ff81044e0 0x7f3ff81048c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:49.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.247+0000 7f3ff7e00700 1 --2- 172.21.15.45:0/2095713819 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ff81095c0 0x7f3ff8111190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:49.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.247+0000 7f3ff7e00700 1 --2- 172.21.15.45:0/2095713819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ff8104e90 0x7f3ff8108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:49.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.247+0000 7f3ff7e00700 1 -- 172.21.15.45:0/2095713819 >> 172.21.15.45:0/2095713819 conn(0x7f3ff80ffdb0 msgr2=0x7f3ff81021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:49.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.247+0000 7f3ff7e00700 1 -- 172.21.15.45:0/2095713819 shutdown_connections 2024-03-28T17:46:49.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.248+0000 7f3ff7e00700 1 -- 172.21.15.45:0/2095713819 wait complete. 2024-03-28T17:46:49.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.249+0000 7f3ff7e00700 1 Processor -- start 2024-03-28T17:46:49.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.249+0000 7f3ff7e00700 1 -- start start 2024-03-28T17:46:49.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.249+0000 7f3ff7e00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3ff81044e0 0x7f3ff81a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:49.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.249+0000 7f3ff7e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ff8104e90 0x7f3ff81a0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:49.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.249+0000 7f3ff7e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ff81095c0 0x7f3ff819a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:49.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.249+0000 7f3ff7e00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff81145f0 con 0x7f3ff81095c0 2024-03-28T17:46:49.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.249+0000 7f3ff7e00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff8114470 con 0x7f3ff8104e90 2024-03-28T17:46:49.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.249+0000 7f3ff7e00700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff8114770 con 0x7f3ff81044e0 2024-03-28T17:46:49.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.249+0000 7f3ff6000700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ff8104e90 0x7f3ff81a0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:49.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.249+0000 7f3ff6a00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3ff81044e0 0x7f3ff81a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:49.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.249+0000 7f3ff6000700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ff8104e90 0x7f3ff81a0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:32890/0 (socket says 172.21.15.45:32890) 2024-03-28T17:46:49.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.249+0000 7f3ff6000700 1 -- 172.21.15.45:0/214745606 learned_addr learned my addr 172.21.15.45:0/214745606 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:49.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.250+0000 7f3ff7400700 1 --2- 172.21.15.45:0/214745606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ff81095c0 0x7f3ff819a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:49.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.250+0000 7f3ff6a00700 1 -- 172.21.15.45:0/214745606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ff8104e90 msgr2=0x7f3ff81a0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:49.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.250+0000 7f3ff6a00700 1 --2- 172.21.15.45:0/214745606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ff8104e90 0x7f3ff81a0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:49.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.250+0000 7f3ff6a00700 1 -- 172.21.15.45:0/214745606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ff81095c0 msgr2=0x7f3ff819a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:49.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.250+0000 7f3ff6a00700 1 --2- 172.21.15.45:0/214745606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ff81095c0 0x7f3ff819a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:49.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.250+0000 7f3ff6a00700 1 -- 172.21.15.45:0/214745606 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff819b090 con 0x7f3ff81044e0 2024-03-28T17:46:49.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.250+0000 7f3ff6000700 1 --2- 172.21.15.45:0/214745606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ff8104e90 0x7f3ff81a0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:46:49.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.250+0000 7f3ff6a00700 1 --2- 172.21.15.45:0/214745606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3ff81044e0 0x7f3ff81a0770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3fe800a010 tx=0x7f3fe800fa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:49.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.250+0000 7f3fef400700 1 -- 172.21.15.45:0/214745606 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe80050c0 con 0x7f3ff81044e0 2024-03-28T17:46:49.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.251+0000 7f3fef400700 1 -- 172.21.15.45:0/214745606 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3fe800b560 con 0x7f3ff81044e0 2024-03-28T17:46:49.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.251+0000 7f3ff7e00700 1 -- 172.21.15.45:0/214745606 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ff819b320 con 0x7f3ff81044e0 2024-03-28T17:46:49.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.251+0000 7f3fef400700 1 -- 172.21.15.45:0/214745606 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe8003a00 con 0x7f3ff81044e0 2024-03-28T17:46:49.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.251+0000 7f3ff7e00700 1 -- 172.21.15.45:0/214745606 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ff81a7640 con 0x7f3ff81044e0 2024-03-28T17:46:49.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.252+0000 7f3ff7e00700 1 -- 172.21.15.45:0/214745606 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3ff80628a0 con 0x7f3ff81044e0 2024-03-28T17:46:49.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.254+0000 7f3fef400700 1 -- 172.21.15.45:0/214745606 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f3fe8029060 con 0x7f3ff81044e0 2024-03-28T17:46:49.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.255+0000 7f3fef400700 1 --2- 172.21.15.45:0/214745606 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3fdc0a29a0 0x7f3fdc0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:49.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.256+0000 7f3fef400700 1 -- 172.21.15.45:0/214745606 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f3fe80cf6b0 con 0x7f3ff81044e0 2024-03-28T17:46:49.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.256+0000 7f3ff6000700 1 --2- 172.21.15.45:0/214745606 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3fdc0a29a0 0x7f3fdc0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:49.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.256+0000 7f3fef400700 1 --2- 172.21.15.45:0/214745606 >> [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039] conn(0x7f3fdc0ac180 0x7f3fdc0ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:49.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.256+0000 7f3fef400700 1 -- 172.21.15.45:0/214745606 --> [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3fe800b0b0 con 0x7f3fdc0ac180 2024-03-28T17:46:49.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.256+0000 7f3fef400700 1 -- 172.21.15.45:0/214745606 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3fe80cfa60 con 0x7f3ff81044e0 2024-03-28T17:46:49.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.256+0000 7f3ff7400700 1 --2- 172.21.15.45:0/214745606 >> [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039] conn(0x7f3fdc0ac180 0x7f3fdc0ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:49.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.257+0000 7f3ff6000700 1 --2- 172.21.15.45:0/214745606 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3fdc0a29a0 0x7f3fdc0a4e60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3ff810aaf0 tx=0x7f3fe4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:49.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.257+0000 7f3ff7400700 1 --2- 172.21.15.45:0/214745606 >> [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039] conn(0x7f3fdc0ac180 0x7f3fdc0ae5a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:49.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.258+0000 7f3fef400700 1 -- 172.21.15.45:0/214745606 <== osd.6 v2:172.21.15.45:6818/2077907039 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3fe800b0b0 con 0x7f3fdc0ac180 2024-03-28T17:46:49.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.278+0000 7f3ff7e00700 1 -- 172.21.15.45:0/214745606 --> [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3ff8062d20 con 0x7f3fdc0ac180 2024-03-28T17:46:49.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.279+0000 7f3fef400700 1 -- 172.21.15.45:0/214745606 <== osd.6 v2:172.21.15.45:6818/2077907039 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f3ff8062d20 con 0x7f3fdc0ac180 2024-03-28T17:46:49.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.280+0000 7f3ff7e00700 1 -- 172.21.15.45:0/214745606 >> [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039] conn(0x7f3fdc0ac180 msgr2=0x7f3fdc0ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:49.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.280+0000 7f3ff7e00700 1 --2- 172.21.15.45:0/214745606 >> [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039] conn(0x7f3fdc0ac180 0x7f3fdc0ae5a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:49.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.280+0000 7f3ff7e00700 1 -- 172.21.15.45:0/214745606 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3fdc0a29a0 msgr2=0x7f3fdc0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:49.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.280+0000 7f3ff7e00700 1 --2- 172.21.15.45:0/214745606 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3fdc0a29a0 0x7f3fdc0a4e60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3ff810aaf0 tx=0x7f3fe4009260 comp rx=0 tx=0).stop 2024-03-28T17:46:49.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.280+0000 7f3ff7e00700 1 -- 172.21.15.45:0/214745606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3ff81044e0 msgr2=0x7f3ff81a0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:49.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.280+0000 7f3ff7e00700 1 --2- 172.21.15.45:0/214745606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3ff81044e0 0x7f3ff81a0770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3fe800a010 tx=0x7f3fe800fa70 comp rx=0 tx=0).stop 2024-03-28T17:46:49.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.280+0000 7f3ff7400700 1 -- 172.21.15.45:0/214745606 reap_dead start 2024-03-28T17:46:49.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.280+0000 7f3ff7e00700 1 -- 172.21.15.45:0/214745606 shutdown_connections 2024-03-28T17:46:49.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.280+0000 7f3ff7e00700 1 -- 172.21.15.45:0/214745606 >> 172.21.15.45:0/214745606 conn(0x7f3ff80ffdb0 msgr2=0x7f3ff8102100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:49.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.280+0000 7f3ff7e00700 1 -- 172.21.15.45:0/214745606 shutdown_connections 2024-03-28T17:46:49.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:49.280+0000 7f3ff7e00700 1 -- 172.21.15.45:0/214745606 wait complete. 2024-03-28T17:46:51.613 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:51 smithi045 ceph-mon[31773]: pgmap v336: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T17:46:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:51 smithi071 ceph-mon[36597]: pgmap v336: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T17:46:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:51 smithi151 ceph-mon[36481]: pgmap v336: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T17:46:52.569 INFO:teuthology.orchestra.run.smithi045.stdout:197568495666 2024-03-28T17:46:52.570 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-28T17:46:53.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.179+0000 7ffb21e00700 1 -- 172.21.15.45:0/4287979871 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffb1c10e9d0 msgr2=0x7ffb1c110dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:53.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.179+0000 7ffb21e00700 1 --2- 172.21.15.45:0/4287979871 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffb1c10e9d0 0x7ffb1c110dc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ffb14009f90 tx=0x7ffb1400b330 comp rx=0 tx=0).stop 2024-03-28T17:46:53.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.180+0000 7ffb21e00700 1 -- 172.21.15.45:0/4287979871 shutdown_connections 2024-03-28T17:46:53.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.180+0000 7ffb21e00700 1 --2- 172.21.15.45:0/4287979871 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffb1c10e9d0 0x7ffb1c110dc0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:53.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.180+0000 7ffb21e00700 1 --2- 172.21.15.45:0/4287979871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb1c100e20 0x7ffb1c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:53.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.180+0000 7ffb21e00700 1 --2- 172.21.15.45:0/4287979871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb1c1017d0 0x7ffb1c10e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:53.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.180+0000 7ffb21e00700 1 -- 172.21.15.45:0/4287979871 >> 172.21.15.45:0/4287979871 conn(0x7ffb1c0fca60 msgr2=0x7ffb1c0fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:53.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.180+0000 7ffb21e00700 1 -- 172.21.15.45:0/4287979871 shutdown_connections 2024-03-28T17:46:53.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.180+0000 7ffb21e00700 1 -- 172.21.15.45:0/4287979871 wait complete. 2024-03-28T17:46:53.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.181+0000 7ffb21e00700 1 Processor -- start 2024-03-28T17:46:53.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.181+0000 7ffb21e00700 1 -- start start 2024-03-28T17:46:53.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.181+0000 7ffb21e00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffb1c100e20 0x7ffb1c073940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:53.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.181+0000 7ffb21e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb1c1017d0 0x7ffb1c06cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:53.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.181+0000 7ffb21e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb1c10e9d0 0x7ffb1c06d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:53.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.182+0000 7ffb21e00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c1145d0 con 0x7ffb1c10e9d0 2024-03-28T17:46:53.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.182+0000 7ffb21e00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c114450 con 0x7ffb1c1017d0 2024-03-28T17:46:53.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.182+0000 7ffb21e00700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c114750 con 0x7ffb1c100e20 2024-03-28T17:46:53.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.182+0000 7ffb20a00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffb1c100e20 0x7ffb1c073940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:53.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.182+0000 7ffb20a00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffb1c100e20 0x7ffb1c073940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:35642/0 (socket says 172.21.15.45:35642) 2024-03-28T17:46:53.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.182+0000 7ffb20a00700 1 -- 172.21.15.45:0/3610163701 learned_addr learned my addr 172.21.15.45:0/3610163701 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:53.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.182+0000 7ffb1be00700 1 --2- 172.21.15.45:0/3610163701 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb1c1017d0 0x7ffb1c06cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:53.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.182+0000 7ffb21400700 1 --2- 172.21.15.45:0/3610163701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb1c10e9d0 0x7ffb1c06d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:53.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.182+0000 7ffb20a00700 1 -- 172.21.15.45:0/3610163701 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb1c1017d0 msgr2=0x7ffb1c06cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:53.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.182+0000 7ffb20a00700 1 --2- 172.21.15.45:0/3610163701 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb1c1017d0 0x7ffb1c06cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:53.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.182+0000 7ffb20a00700 1 -- 172.21.15.45:0/3610163701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb1c10e9d0 msgr2=0x7ffb1c06d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:53.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.182+0000 7ffb20a00700 1 --2- 172.21.15.45:0/3610163701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb1c10e9d0 0x7ffb1c06d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:53.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.182+0000 7ffb20a00700 1 -- 172.21.15.45:0/3610163701 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1c06d660 con 0x7ffb1c100e20 2024-03-28T17:46:53.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.182+0000 7ffb1be00700 1 --2- 172.21.15.45:0/3610163701 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb1c1017d0 0x7ffb1c06cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:46:53.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.182+0000 7ffb20a00700 1 --2- 172.21.15.45:0/3610163701 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffb1c100e20 0x7ffb1c073940 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ffb1000e9f0 tx=0x7ffb1000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:53.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.183+0000 7ffb19600700 1 -- 172.21.15.45:0/3610163701 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb1000cbd0 con 0x7ffb1c100e20 2024-03-28T17:46:53.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.183+0000 7ffb19600700 1 -- 172.21.15.45:0/3610163701 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb1000cd70 con 0x7ffb1c100e20 2024-03-28T17:46:53.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.183+0000 7ffb19600700 1 -- 172.21.15.45:0/3610163701 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb10010430 con 0x7ffb1c100e20 2024-03-28T17:46:53.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.183+0000 7ffb21e00700 1 -- 172.21.15.45:0/3610163701 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb1c0714e0 con 0x7ffb1c100e20 2024-03-28T17:46:53.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.183+0000 7ffb21e00700 1 -- 172.21.15.45:0/3610163701 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1c0719d0 con 0x7ffb1c100e20 2024-03-28T17:46:53.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.184+0000 7ffb21e00700 1 -- 172.21.15.45:0/3610163701 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ffb1c071f60 con 0x7ffb1c100e20 2024-03-28T17:46:53.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.186+0000 7ffb19600700 1 -- 172.21.15.45:0/3610163701 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7ffb10004750 con 0x7ffb1c100e20 2024-03-28T17:46:53.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.187+0000 7ffb19600700 1 --2- 172.21.15.45:0/3610163701 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ffb040a29a0 0x7ffb040a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:53.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.188+0000 7ffb19600700 1 -- 172.21.15.45:0/3610163701 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ffb100cec50 con 0x7ffb1c100e20 2024-03-28T17:46:53.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.188+0000 7ffb1be00700 1 --2- 172.21.15.45:0/3610163701 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ffb040a29a0 0x7ffb040a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:53.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.188+0000 7ffb19600700 1 --2- 172.21.15.45:0/3610163701 >> [v2:172.21.15.71:6800/2014529906,v1:172.21.15.71:6801/2014529906] conn(0x7ffb040ac180 0x7ffb040ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:53.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.188+0000 7ffb19600700 1 -- 172.21.15.45:0/3610163701 --> [v2:172.21.15.71:6800/2014529906,v1:172.21.15.71:6801/2014529906] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffb1001ee50 con 0x7ffb040ac180 2024-03-28T17:46:53.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.188+0000 7ffb19600700 1 -- 172.21.15.45:0/3610163701 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb100d1270 con 0x7ffb1c100e20 2024-03-28T17:46:53.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.188+0000 7ffb21400700 1 --2- 172.21.15.45:0/3610163701 >> [v2:172.21.15.71:6800/2014529906,v1:172.21.15.71:6801/2014529906] conn(0x7ffb040ac180 0x7ffb040ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:53.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.189+0000 7ffb1be00700 1 --2- 172.21.15.45:0/3610163701 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ffb040a29a0 0x7ffb040a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c110cc0 tx=0x7ffb08009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:53.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.189+0000 7ffb21400700 1 --2- 172.21.15.45:0/3610163701 >> [v2:172.21.15.71:6800/2014529906,v1:172.21.15.71:6801/2014529906] conn(0x7ffb040ac180 0x7ffb040ae5a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:53.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.190+0000 7ffb19600700 1 -- 172.21.15.45:0/3610163701 <== osd.1 v2:172.21.15.71:6800/2014529906 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ffb1001ee50 con 0x7ffb040ac180 2024-03-28T17:46:53.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.209+0000 7ffb21e00700 1 -- 172.21.15.45:0/3610163701 --> [v2:172.21.15.71:6800/2014529906,v1:172.21.15.71:6801/2014529906] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ffb1c06a6c0 con 0x7ffb040ac180 2024-03-28T17:46:53.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.209+0000 7ffb19600700 1 -- 172.21.15.45:0/3610163701 <== osd.1 v2:172.21.15.71:6800/2014529906 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ffb1c06a6c0 con 0x7ffb040ac180 2024-03-28T17:46:53.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.209+0000 7ffb21e00700 1 -- 172.21.15.45:0/3610163701 >> [v2:172.21.15.71:6800/2014529906,v1:172.21.15.71:6801/2014529906] conn(0x7ffb040ac180 msgr2=0x7ffb040ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:53.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.209+0000 7ffb21e00700 1 --2- 172.21.15.45:0/3610163701 >> [v2:172.21.15.71:6800/2014529906,v1:172.21.15.71:6801/2014529906] conn(0x7ffb040ac180 0x7ffb040ae5a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:53.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.210+0000 7ffb21e00700 1 -- 172.21.15.45:0/3610163701 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ffb040a29a0 msgr2=0x7ffb040a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:53.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.210+0000 7ffb21e00700 1 --2- 172.21.15.45:0/3610163701 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ffb040a29a0 0x7ffb040a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c110cc0 tx=0x7ffb08009260 comp rx=0 tx=0).stop 2024-03-28T17:46:53.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.210+0000 7ffb21e00700 1 -- 172.21.15.45:0/3610163701 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffb1c100e20 msgr2=0x7ffb1c073940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:53.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.210+0000 7ffb21e00700 1 --2- 172.21.15.45:0/3610163701 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffb1c100e20 0x7ffb1c073940 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ffb1000e9f0 tx=0x7ffb1000ed00 comp rx=0 tx=0).stop 2024-03-28T17:46:53.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.210+0000 7ffb21400700 1 -- 172.21.15.45:0/3610163701 reap_dead start 2024-03-28T17:46:53.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.210+0000 7ffb21e00700 1 -- 172.21.15.45:0/3610163701 shutdown_connections 2024-03-28T17:46:53.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.210+0000 7ffb21e00700 1 -- 172.21.15.45:0/3610163701 >> 172.21.15.45:0/3610163701 conn(0x7ffb1c0fca60 msgr2=0x7ffb1c10fb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:53.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.210+0000 7ffb21e00700 1 -- 172.21.15.45:0/3610163701 shutdown_connections 2024-03-28T17:46:53.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.210+0000 7ffb21e00700 1 -- 172.21.15.45:0/3610163701 wait complete. 2024-03-28T17:46:53.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:53 smithi045 ceph-mon[31773]: pgmap v337: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T17:46:53.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:53 smithi071 ceph-mon[36597]: pgmap v337: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T17:46:53.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:53 smithi151 ceph-mon[36481]: pgmap v337: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T17:46:53.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.898+0000 7f4ac6c00700 1 -- 172.21.15.45:0/588931520 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ac0102df0 msgr2=0x7f4ac0106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:53.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.898+0000 7f4ac6c00700 1 --2- 172.21.15.45:0/588931520 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ac0102df0 0x7f4ac0106e30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4aac0099c0 tx=0x7f4aac009cd0 comp rx=0 tx=0).stop 2024-03-28T17:46:53.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.898+0000 7f4ac6c00700 1 -- 172.21.15.45:0/588931520 shutdown_connections 2024-03-28T17:46:53.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.898+0000 7f4ac6c00700 1 --2- 172.21.15.45:0/588931520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ac0107510 0x7f4ac010f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:53.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.898+0000 7f4ac6c00700 1 --2- 172.21.15.45:0/588931520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ac0102440 0x7f4ac0102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:53.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.898+0000 7f4ac6c00700 1 --2- 172.21.15.45:0/588931520 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ac0102df0 0x7f4ac0106e30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:53.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.898+0000 7f4ac6c00700 1 -- 172.21.15.45:0/588931520 >> 172.21.15.45:0/588931520 conn(0x7f4ac00792f0 msgr2=0x7f4ac0079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:53.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.899+0000 7f4ac6c00700 1 -- 172.21.15.45:0/588931520 shutdown_connections 2024-03-28T17:46:53.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.899+0000 7f4ac6c00700 1 -- 172.21.15.45:0/588931520 wait complete. 2024-03-28T17:46:53.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.900+0000 7f4ac6c00700 1 Processor -- start 2024-03-28T17:46:53.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.900+0000 7f4ac6c00700 1 -- start start 2024-03-28T17:46:53.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.901+0000 7f4ac6c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ac0102440 0x7f4ac01a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:53.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.901+0000 7f4ac6c00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ac0102df0 0x7f4ac01a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:53.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.901+0000 7f4ac6c00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ac0107510 0x7f4ac019aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:53.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.901+0000 7f4ac6c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac0112550 con 0x7f4ac0102440 2024-03-28T17:46:53.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.901+0000 7f4ac6c00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac01123d0 con 0x7f4ac0102df0 2024-03-28T17:46:53.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.901+0000 7f4ac6c00700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac01126d0 con 0x7f4ac0107510 2024-03-28T17:46:53.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.901+0000 7f4abf400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ac0102df0 0x7f4ac01a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:53.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.901+0000 7f4abfe00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ac0102440 0x7f4ac01a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:53.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.901+0000 7f4abf400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ac0102df0 0x7f4ac01a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:32926/0 (socket says 172.21.15.45:32926) 2024-03-28T17:46:53.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.901+0000 7f4abf400700 1 -- 172.21.15.45:0/2356952362 learned_addr learned my addr 172.21.15.45:0/2356952362 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.901+0000 7f4ac4c00700 1 --2- 172.21.15.45:0/2356952362 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ac0107510 0x7f4ac019aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.901+0000 7f4abf400700 1 -- 172.21.15.45:0/2356952362 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ac0107510 msgr2=0x7f4ac019aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.901+0000 7f4abf400700 1 --2- 172.21.15.45:0/2356952362 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ac0107510 0x7f4ac019aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.901+0000 7f4abf400700 1 -- 172.21.15.45:0/2356952362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ac0102440 msgr2=0x7f4ac01a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.901+0000 7f4abf400700 1 --2- 172.21.15.45:0/2356952362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ac0102440 0x7f4ac01a0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.901+0000 7f4abf400700 1 -- 172.21.15.45:0/2356952362 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac019b1b0 con 0x7f4ac0102df0 2024-03-28T17:46:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.902+0000 7f4abf400700 1 --2- 172.21.15.45:0/2356952362 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ac0102df0 0x7f4ac01a0dd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4aac005f50 tx=0x7f4aac003a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.902+0000 7f4abcc00700 1 -- 172.21.15.45:0/2356952362 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac003680 con 0x7f4ac0102df0 2024-03-28T17:46:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.902+0000 7f4abcc00700 1 -- 172.21.15.45:0/2356952362 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4aac0055a0 con 0x7f4ac0102df0 2024-03-28T17:46:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.902+0000 7f4abcc00700 1 -- 172.21.15.45:0/2356952362 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac00fce0 con 0x7f4ac0102df0 2024-03-28T17:46:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.902+0000 7f4ac6c00700 1 -- 172.21.15.45:0/2356952362 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac019b440 con 0x7f4ac0102df0 2024-03-28T17:46:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.902+0000 7f4ac6c00700 1 -- 172.21.15.45:0/2356952362 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac01a7760 con 0x7f4ac0102df0 2024-03-28T17:46:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.903+0000 7f4ac6c00700 1 -- 172.21.15.45:0/2356952362 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4ac01012f0 con 0x7f4ac0102df0 2024-03-28T17:46:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.905+0000 7f4abcc00700 1 -- 172.21.15.45:0/2356952362 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f4aac00f460 con 0x7f4ac0102df0 2024-03-28T17:46:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.907+0000 7f4abcc00700 1 --2- 172.21.15.45:0/2356952362 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4aa40a2b40 0x7f4aa40a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:53.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.907+0000 7f4abcc00700 1 -- 172.21.15.45:0/2356952362 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f4aac0cfc70 con 0x7f4ac0102df0 2024-03-28T17:46:53.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.907+0000 7f4abfe00700 1 --2- 172.21.15.45:0/2356952362 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4aa40a2b40 0x7f4aa40a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:53.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.907+0000 7f4abcc00700 1 --2- 172.21.15.45:0/2356952362 >> [v2:172.21.15.71:6816/2000211251,v1:172.21.15.71:6817/2000211251] conn(0x7f4aa40ac320 0x7f4aa40ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:53.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.907+0000 7f4abcc00700 1 -- 172.21.15.45:0/2356952362 --> [v2:172.21.15.71:6816/2000211251,v1:172.21.15.71:6817/2000211251] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4aac003680 con 0x7f4aa40ac320 2024-03-28T17:46:53.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.907+0000 7f4abcc00700 1 -- 172.21.15.45:0/2356952362 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4aac0d0020 con 0x7f4ac0102df0 2024-03-28T17:46:53.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.907+0000 7f4ac4c00700 1 --2- 172.21.15.45:0/2356952362 >> [v2:172.21.15.71:6816/2000211251,v1:172.21.15.71:6817/2000211251] conn(0x7f4aa40ac320 0x7f4aa40ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:53.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.908+0000 7f4abfe00700 1 --2- 172.21.15.45:0/2356952362 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4aa40a2b40 0x7f4aa40a5000 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4ac019bb90 tx=0x7f4ab0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:53.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.908+0000 7f4ac4c00700 1 --2- 172.21.15.45:0/2356952362 >> [v2:172.21.15.71:6816/2000211251,v1:172.21.15.71:6817/2000211251] conn(0x7f4aa40ac320 0x7f4aa40ae740 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:53.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.909+0000 7f4abcc00700 1 -- 172.21.15.45:0/2356952362 <== osd.7 v2:172.21.15.71:6816/2000211251 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4aac003680 con 0x7f4aa40ac320 2024-03-28T17:46:53.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.927+0000 7f4ac6c00700 1 -- 172.21.15.45:0/2356952362 --> [v2:172.21.15.71:6816/2000211251,v1:172.21.15.71:6817/2000211251] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4ac0103910 con 0x7f4aa40ac320 2024-03-28T17:46:53.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.928+0000 7f4abcc00700 1 -- 172.21.15.45:0/2356952362 <== osd.7 v2:172.21.15.71:6816/2000211251 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4ac0103910 con 0x7f4aa40ac320 2024-03-28T17:46:53.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.928+0000 7f4ac6c00700 1 -- 172.21.15.45:0/2356952362 >> [v2:172.21.15.71:6816/2000211251,v1:172.21.15.71:6817/2000211251] conn(0x7f4aa40ac320 msgr2=0x7f4aa40ae740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:53.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.928+0000 7f4ac6c00700 1 --2- 172.21.15.45:0/2356952362 >> [v2:172.21.15.71:6816/2000211251,v1:172.21.15.71:6817/2000211251] conn(0x7f4aa40ac320 0x7f4aa40ae740 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:53.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.928+0000 7f4ac6c00700 1 -- 172.21.15.45:0/2356952362 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4aa40a2b40 msgr2=0x7f4aa40a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:53.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.928+0000 7f4ac6c00700 1 --2- 172.21.15.45:0/2356952362 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4aa40a2b40 0x7f4aa40a5000 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4ac019bb90 tx=0x7f4ab0009260 comp rx=0 tx=0).stop 2024-03-28T17:46:53.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.928+0000 7f4ac6c00700 1 -- 172.21.15.45:0/2356952362 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ac0102df0 msgr2=0x7f4ac01a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:53.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.928+0000 7f4ac6c00700 1 --2- 172.21.15.45:0/2356952362 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ac0102df0 0x7f4ac01a0dd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4aac005f50 tx=0x7f4aac003a90 comp rx=0 tx=0).stop 2024-03-28T17:46:53.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.928+0000 7f4ac4c00700 1 -- 172.21.15.45:0/2356952362 reap_dead start 2024-03-28T17:46:53.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.929+0000 7f4ac6c00700 1 -- 172.21.15.45:0/2356952362 shutdown_connections 2024-03-28T17:46:53.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.929+0000 7f4ac6c00700 1 -- 172.21.15.45:0/2356952362 >> 172.21.15.45:0/2356952362 conn(0x7f4ac00792f0 msgr2=0x7f4ac010d840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:53.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.929+0000 7f4ac6c00700 1 -- 172.21.15.45:0/2356952362 shutdown_connections 2024-03-28T17:46:53.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:53.929+0000 7f4ac6c00700 1 -- 172.21.15.45:0/2356952362 wait complete. 2024-03-28T17:46:54.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.935+0000 7f4a24400700 1 -- 172.21.15.45:0/3294890854 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a1c107150 msgr2=0x7f4a1c06a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:54.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.935+0000 7f4a24400700 1 --2- 172.21.15.45:0/3294890854 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a1c107150 0x7f4a1c06a6d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4a1000b0a0 tx=0x7f4a1000b3b0 comp rx=0 tx=0).stop 2024-03-28T17:46:54.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.935+0000 7f4a24400700 1 -- 172.21.15.45:0/3294890854 shutdown_connections 2024-03-28T17:46:54.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.935+0000 7f4a24400700 1 --2- 172.21.15.45:0/3294890854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a1c104bc0 0x7f4a1c105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:54.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.935+0000 7f4a24400700 1 --2- 172.21.15.45:0/3294890854 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a1c107150 0x7f4a1c06a6d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:54.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.935+0000 7f4a24400700 1 --2- 172.21.15.45:0/3294890854 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a1c06ac10 0x7f4a1c06b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:54.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.935+0000 7f4a24400700 1 -- 172.21.15.45:0/3294890854 >> 172.21.15.45:0/3294890854 conn(0x7f4a1c0770e0 msgr2=0x7f4a1c0774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:54.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.935+0000 7f4a24400700 1 -- 172.21.15.45:0/3294890854 shutdown_connections 2024-03-28T17:46:54.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.935+0000 7f4a24400700 1 -- 172.21.15.45:0/3294890854 wait complete. 2024-03-28T17:46:54.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.936+0000 7f4a24400700 1 Processor -- start 2024-03-28T17:46:54.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.937+0000 7f4a24400700 1 -- start start 2024-03-28T17:46:54.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.937+0000 7f4a24400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a1c06ac10 0x7f4a1c19ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:54.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.937+0000 7f4a24400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a1c104bc0 0x7f4a1c19f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:54.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.937+0000 7f4a24400700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a1c107150 0x7f4a1c1a3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:54.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.937+0000 7f4a24400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c118a40 con 0x7f4a1c06ac10 2024-03-28T17:46:54.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.937+0000 7f4a24400700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c1188c0 con 0x7f4a1c104bc0 2024-03-28T17:46:54.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.937+0000 7f4a24400700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c118bc0 con 0x7f4a1c107150 2024-03-28T17:46:54.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.938+0000 7f4a21a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a1c06ac10 0x7f4a1c19ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:54.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.938+0000 7f4a21000700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a1c104bc0 0x7f4a1c19f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:54.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.938+0000 7f4a21a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a1c06ac10 0x7f4a1c19ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35442/0 (socket says 172.21.15.45:35442) 2024-03-28T17:46:54.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.938+0000 7f4a21a00700 1 -- 172.21.15.45:0/144573375 learned_addr learned my addr 172.21.15.45:0/144573375 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:54.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.938+0000 7f4a22400700 1 --2- 172.21.15.45:0/144573375 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a1c107150 0x7f4a1c1a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:54.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.938+0000 7f4a21000700 1 -- 172.21.15.45:0/144573375 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a1c107150 msgr2=0x7f4a1c1a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:54.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.938+0000 7f4a21000700 1 --2- 172.21.15.45:0/144573375 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a1c107150 0x7f4a1c1a3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:54.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.938+0000 7f4a21000700 1 -- 172.21.15.45:0/144573375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a1c06ac10 msgr2=0x7f4a1c19ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:54.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.938+0000 7f4a21000700 1 --2- 172.21.15.45:0/144573375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a1c06ac10 0x7f4a1c19ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:54.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.938+0000 7f4a21000700 1 -- 172.21.15.45:0/144573375 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1c1a3e80 con 0x7f4a1c104bc0 2024-03-28T17:46:54.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.938+0000 7f4a22400700 1 --2- 172.21.15.45:0/144573375 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a1c107150 0x7f4a1c1a3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:46:54.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.938+0000 7f4a21a00700 1 --2- 172.21.15.45:0/144573375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a1c06ac10 0x7f4a1c19ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:46:54.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.938+0000 7f4a21000700 1 --2- 172.21.15.45:0/144573375 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a1c104bc0 0x7f4a1c19f3b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4a0800e9f0 tx=0x7f4a0800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:54.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.939+0000 7f4a1a000700 1 -- 172.21.15.45:0/144573375 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0800c9c0 con 0x7f4a1c104bc0 2024-03-28T17:46:54.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.939+0000 7f4a1a000700 1 -- 172.21.15.45:0/144573375 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a0800cb60 con 0x7f4a1c104bc0 2024-03-28T17:46:54.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.939+0000 7f4a1a000700 1 -- 172.21.15.45:0/144573375 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a08018680 con 0x7f4a1c104bc0 2024-03-28T17:46:54.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.939+0000 7f4a24400700 1 -- 172.21.15.45:0/144573375 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a1c1a4170 con 0x7f4a1c104bc0 2024-03-28T17:46:54.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.939+0000 7f4a24400700 1 -- 172.21.15.45:0/144573375 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a1c1ab9f0 con 0x7f4a1c104bc0 2024-03-28T17:46:54.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.940+0000 7f4a24400700 1 -- 172.21.15.45:0/144573375 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4a1c100b10 con 0x7f4a1c104bc0 2024-03-28T17:46:54.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.942+0000 7f4a1a000700 1 -- 172.21.15.45:0/144573375 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f4a08047a90 con 0x7f4a1c104bc0 2024-03-28T17:46:54.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.943+0000 7f4a1a000700 1 --2- 172.21.15.45:0/144573375 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4a040a29a0 0x7f4a040a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:54.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.944+0000 7f4a1a000700 1 -- 172.21.15.45:0/144573375 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f4a080ced80 con 0x7f4a1c104bc0 2024-03-28T17:46:54.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.944+0000 7f4a21a00700 1 --2- 172.21.15.45:0/144573375 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4a040a29a0 0x7f4a040a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:54.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.944+0000 7f4a1a000700 1 --2- 172.21.15.45:0/144573375 >> [v2:172.21.15.71:6824/1099494037,v1:172.21.15.71:6825/1099494037] conn(0x7f4a040ac180 0x7f4a040ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:54.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.944+0000 7f4a1a000700 1 -- 172.21.15.45:0/144573375 --> [v2:172.21.15.71:6824/1099494037,v1:172.21.15.71:6825/1099494037] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a0800be50 con 0x7f4a040ac180 2024-03-28T17:46:54.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.944+0000 7f4a1a000700 1 -- 172.21.15.45:0/144573375 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a080cf130 con 0x7f4a1c104bc0 2024-03-28T17:46:54.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.944+0000 7f4a22400700 1 --2- 172.21.15.45:0/144573375 >> [v2:172.21.15.71:6824/1099494037,v1:172.21.15.71:6825/1099494037] conn(0x7f4a040ac180 0x7f4a040ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:54.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.945+0000 7f4a21a00700 1 --2- 172.21.15.45:0/144573375 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4a040a29a0 0x7f4a040a4e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4a1000b780 tx=0x7f4a10006230 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:54.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.945+0000 7f4a22400700 1 --2- 172.21.15.45:0/144573375 >> [v2:172.21.15.71:6824/1099494037,v1:172.21.15.71:6825/1099494037] conn(0x7f4a040ac180 0x7f4a040ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:54.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.946+0000 7f4a1a000700 1 -- 172.21.15.45:0/144573375 <== osd.10 v2:172.21.15.71:6824/1099494037 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4a0800be50 con 0x7f4a040ac180 2024-03-28T17:46:54.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.965+0000 7f4a24400700 1 -- 172.21.15.45:0/144573375 --> [v2:172.21.15.71:6824/1099494037,v1:172.21.15.71:6825/1099494037] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4a1c1016a0 con 0x7f4a040ac180 2024-03-28T17:46:54.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.966+0000 7f4a1a000700 1 -- 172.21.15.45:0/144573375 <== osd.10 v2:172.21.15.71:6824/1099494037 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4a1c1016a0 con 0x7f4a040ac180 2024-03-28T17:46:54.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.966+0000 7f4a24400700 1 -- 172.21.15.45:0/144573375 >> [v2:172.21.15.71:6824/1099494037,v1:172.21.15.71:6825/1099494037] conn(0x7f4a040ac180 msgr2=0x7f4a040ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:54.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.966+0000 7f4a24400700 1 --2- 172.21.15.45:0/144573375 >> [v2:172.21.15.71:6824/1099494037,v1:172.21.15.71:6825/1099494037] conn(0x7f4a040ac180 0x7f4a040ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:54.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.966+0000 7f4a24400700 1 -- 172.21.15.45:0/144573375 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4a040a29a0 msgr2=0x7f4a040a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:54.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.966+0000 7f4a24400700 1 --2- 172.21.15.45:0/144573375 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4a040a29a0 0x7f4a040a4e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4a1000b780 tx=0x7f4a10006230 comp rx=0 tx=0).stop 2024-03-28T17:46:54.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.967+0000 7f4a24400700 1 -- 172.21.15.45:0/144573375 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a1c104bc0 msgr2=0x7f4a1c19f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:54.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.967+0000 7f4a24400700 1 --2- 172.21.15.45:0/144573375 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a1c104bc0 0x7f4a1c19f3b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4a0800e9f0 tx=0x7f4a0800ed00 comp rx=0 tx=0).stop 2024-03-28T17:46:54.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.967+0000 7f4a22400700 1 -- 172.21.15.45:0/144573375 reap_dead start 2024-03-28T17:46:54.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.967+0000 7f4a24400700 1 -- 172.21.15.45:0/144573375 shutdown_connections 2024-03-28T17:46:54.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.967+0000 7f4a24400700 1 -- 172.21.15.45:0/144573375 >> 172.21.15.45:0/144573375 conn(0x7f4a1c0770e0 msgr2=0x7f4a1c1000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:54.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.967+0000 7f4a24400700 1 -- 172.21.15.45:0/144573375 shutdown_connections 2024-03-28T17:46:54.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:54.967+0000 7f4a24400700 1 -- 172.21.15.45:0/144573375 wait complete. 2024-03-28T17:46:55.522 INFO:teuthology.orchestra.run.smithi045.stdout:55834574927 2024-03-28T17:46:55.523 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-28T17:46:55.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:55 smithi071 ceph-mon[36597]: pgmap v338: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T17:46:55.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:55 smithi151 ceph-mon[36481]: pgmap v338: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T17:46:55.879 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:55 smithi045 ceph-mon[31773]: pgmap v338: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T17:46:57.655 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:57 smithi045 ceph-mon[31773]: pgmap v339: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T17:46:57.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.728+0000 7f7ef9200700 1 -- 172.21.15.45:0/1566521751 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ef41081e0 msgr2=0x7f7ef41085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:57.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.728+0000 7f7ef9200700 1 --2- 172.21.15.45:0/1566521751 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ef41081e0 0x7f7ef41085c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4009990 tx=0x7f7ee4009ca0 comp rx=0 tx=0).stop 2024-03-28T17:46:57.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.729+0000 7f7ef9200700 1 -- 172.21.15.45:0/1566521751 shutdown_connections 2024-03-28T17:46:57.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.729+0000 7f7ef9200700 1 --2- 172.21.15.45:0/1566521751 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ef406a7e0 0x7f7ef410f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:57.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.729+0000 7f7ef9200700 1 --2- 172.21.15.45:0/1566521751 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ef41081e0 0x7f7ef41085c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:57.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.729+0000 7f7ef9200700 1 --2- 172.21.15.45:0/1566521751 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ef4069e40 0x7f7ef406a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:57.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.729+0000 7f7ef9200700 1 -- 172.21.15.45:0/1566521751 >> 172.21.15.45:0/1566521751 conn(0x7f7ef4077fc0 msgr2=0x7f7ef40783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.729+0000 7f7ef9200700 1 -- 172.21.15.45:0/1566521751 shutdown_connections 2024-03-28T17:46:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.729+0000 7f7ef9200700 1 -- 172.21.15.45:0/1566521751 wait complete. 2024-03-28T17:46:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.730+0000 7f7ef9200700 1 Processor -- start 2024-03-28T17:46:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.731+0000 7f7ef9200700 1 -- start start 2024-03-28T17:46:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.731+0000 7f7ef9200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ef4069e40 0x7f7ef4101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.731+0000 7f7ef9200700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ef406a7e0 0x7f7ef41016a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.731+0000 7f7ef9200700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ef41081e0 0x7f7ef4101be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.731+0000 7f7ef9200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef4112470 con 0x7f7ef4069e40 2024-03-28T17:46:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.731+0000 7f7ef9200700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef41122f0 con 0x7f7ef41081e0 2024-03-28T17:46:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.731+0000 7f7ef9200700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef41125f0 con 0x7f7ef406a7e0 2024-03-28T17:46:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.731+0000 7f7ef1e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ef4069e40 0x7f7ef4101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.731+0000 7f7ee9400700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ef406a7e0 0x7f7ef41016a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.731+0000 7f7ef2800700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ef41081e0 0x7f7ef4101be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.731+0000 7f7ef2800700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ef41081e0 0x7f7ef4101be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:54722/0 (socket says 172.21.15.45:54722) 2024-03-28T17:46:57.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.731+0000 7f7ef2800700 1 -- 172.21.15.45:0/1016512601 learned_addr learned my addr 172.21.15.45:0/1016512601 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:46:57.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.732+0000 7f7ef2800700 1 -- 172.21.15.45:0/1016512601 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ef406a7e0 msgr2=0x7f7ef41016a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:57.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.732+0000 7f7ef2800700 1 --2- 172.21.15.45:0/1016512601 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ef406a7e0 0x7f7ef41016a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:57.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.732+0000 7f7ef2800700 1 -- 172.21.15.45:0/1016512601 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ef4069e40 msgr2=0x7f7ef4101160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:57.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.732+0000 7f7ef2800700 1 --2- 172.21.15.45:0/1016512601 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ef4069e40 0x7f7ef4101160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:57.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.732+0000 7f7ef2800700 1 -- 172.21.15.45:0/1016512601 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef41a6e80 con 0x7f7ef41081e0 2024-03-28T17:46:57.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.732+0000 7f7ef2800700 1 --2- 172.21.15.45:0/1016512601 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ef41081e0 0x7f7ef4101be0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7eec00ef30 tx=0x7f7eec00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:57.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.732+0000 7f7eeb400700 1 -- 172.21.15.45:0/1016512601 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eec0092e0 con 0x7f7ef41081e0 2024-03-28T17:46:57.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.732+0000 7f7eeb400700 1 -- 172.21.15.45:0/1016512601 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7eec004bb0 con 0x7f7ef41081e0 2024-03-28T17:46:57.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.732+0000 7f7eeb400700 1 -- 172.21.15.45:0/1016512601 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eec004280 con 0x7f7ef41081e0 2024-03-28T17:46:57.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.732+0000 7f7ef9200700 1 -- 172.21.15.45:0/1016512601 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef41a7110 con 0x7f7ef41081e0 2024-03-28T17:46:57.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.732+0000 7f7ef9200700 1 -- 172.21.15.45:0/1016512601 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef41a7650 con 0x7f7ef41081e0 2024-03-28T17:46:57.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.733+0000 7f7ef9200700 1 -- 172.21.15.45:0/1016512601 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7ef40628a0 con 0x7f7ef41081e0 2024-03-28T17:46:57.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.735+0000 7f7eeb400700 1 -- 172.21.15.45:0/1016512601 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f7eec01b030 con 0x7f7ef41081e0 2024-03-28T17:46:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.736+0000 7f7eeb400700 1 --2- 172.21.15.45:0/1016512601 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f7ed80a29a0 0x7f7ed80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.737+0000 7f7eeb400700 1 -- 172.21.15.45:0/1016512601 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f7eec0ce510 con 0x7f7ef41081e0 2024-03-28T17:46:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.737+0000 7f7ef1e00700 1 --2- 172.21.15.45:0/1016512601 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f7ed80a29a0 0x7f7ed80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.737+0000 7f7eeb400700 1 --2- 172.21.15.45:0/1016512601 >> [v2:172.21.15.151:6800/1850933775,v1:172.21.15.151:6801/1850933775] conn(0x7f7ed80ac180 0x7f7ed80ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:46:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.737+0000 7f7eeb400700 1 -- 172.21.15.45:0/1016512601 --> [v2:172.21.15.151:6800/1850933775,v1:172.21.15.151:6801/1850933775] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ed80aec50 con 0x7f7ed80ac180 2024-03-28T17:46:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.737+0000 7f7eeb400700 1 -- 172.21.15.45:0/1016512601 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7eec0d5050 con 0x7f7ef41081e0 2024-03-28T17:46:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.737+0000 7f7ee9400700 1 --2- 172.21.15.45:0/1016512601 >> [v2:172.21.15.151:6800/1850933775,v1:172.21.15.151:6801/1850933775] conn(0x7f7ed80ac180 0x7f7ed80ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:46:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.738+0000 7f7ef1e00700 1 --2- 172.21.15.45:0/1016512601 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f7ed80a29a0 0x7f7ed80a4e60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4006010 tx=0x7f7ee400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:57.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.738+0000 7f7ee9400700 1 --2- 172.21.15.45:0/1016512601 >> [v2:172.21.15.151:6800/1850933775,v1:172.21.15.151:6801/1850933775] conn(0x7f7ed80ac180 0x7f7ed80ae5a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:46:57.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.739+0000 7f7eeb400700 1 -- 172.21.15.45:0/1016512601 <== osd.2 v2:172.21.15.151:6800/1850933775 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7ed80aec50 con 0x7f7ed80ac180 2024-03-28T17:46:57.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.758+0000 7f7ef9200700 1 -- 172.21.15.45:0/1016512601 --> [v2:172.21.15.151:6800/1850933775,v1:172.21.15.151:6801/1850933775] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7ef4062d20 con 0x7f7ed80ac180 2024-03-28T17:46:57.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.758+0000 7f7eeb400700 1 -- 172.21.15.45:0/1016512601 <== osd.2 v2:172.21.15.151:6800/1850933775 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7ef4062d20 con 0x7f7ed80ac180 2024-03-28T17:46:57.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.759+0000 7f7ef9200700 1 -- 172.21.15.45:0/1016512601 >> [v2:172.21.15.151:6800/1850933775,v1:172.21.15.151:6801/1850933775] conn(0x7f7ed80ac180 msgr2=0x7f7ed80ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:57.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.759+0000 7f7ef9200700 1 --2- 172.21.15.45:0/1016512601 >> [v2:172.21.15.151:6800/1850933775,v1:172.21.15.151:6801/1850933775] conn(0x7f7ed80ac180 0x7f7ed80ae5a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:46:57.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.759+0000 7f7ef9200700 1 -- 172.21.15.45:0/1016512601 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f7ed80a29a0 msgr2=0x7f7ed80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:57.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.759+0000 7f7ef9200700 1 --2- 172.21.15.45:0/1016512601 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f7ed80a29a0 0x7f7ed80a4e60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4006010 tx=0x7f7ee400b260 comp rx=0 tx=0).stop 2024-03-28T17:46:57.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.759+0000 7f7ef9200700 1 -- 172.21.15.45:0/1016512601 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ef41081e0 msgr2=0x7f7ef4101be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:46:57.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.759+0000 7f7ef9200700 1 --2- 172.21.15.45:0/1016512601 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ef41081e0 0x7f7ef4101be0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7eec00ef30 tx=0x7f7eec00c3a0 comp rx=0 tx=0).stop 2024-03-28T17:46:57.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.759+0000 7f7ef2800700 1 -- 172.21.15.45:0/1016512601 reap_dead start 2024-03-28T17:46:57.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.759+0000 7f7ef9200700 1 -- 172.21.15.45:0/1016512601 shutdown_connections 2024-03-28T17:46:57.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.759+0000 7f7ef9200700 1 -- 172.21.15.45:0/1016512601 >> 172.21.15.45:0/1016512601 conn(0x7f7ef4077fc0 msgr2=0x7f7ef40ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:46:57.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.759+0000 7f7ef9200700 1 -- 172.21.15.45:0/1016512601 shutdown_connections 2024-03-28T17:46:57.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:57.759+0000 7f7ef9200700 1 -- 172.21.15.45:0/1016512601 wait complete. 2024-03-28T17:46:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:57 smithi071 ceph-mon[36597]: pgmap v339: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T17:46:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:57 smithi151 ceph-mon[36481]: pgmap v339: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T17:46:59.082 INFO:teuthology.orchestra.run.smithi045.stdout:77309411403 2024-03-28T17:46:59.083 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-28T17:46:59.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:46:59 smithi045 ceph-mon[31773]: pgmap v340: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T17:46:59.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:46:59 smithi071 ceph-mon[36597]: pgmap v340: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T17:46:59.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:46:59 smithi151 ceph-mon[36481]: pgmap v340: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T17:46:59.898 INFO:teuthology.orchestra.run.smithi045.stdout:219043332140 2024-03-28T17:46:59.899 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-28T17:47:00.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.172+0000 7fe0c3000700 1 -- 172.21.15.45:0/4209663565 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0bc1081e0 msgr2=0x7fe0bc1085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:00.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.172+0000 7fe0c3000700 1 --2- 172.21.15.45:0/4209663565 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0bc1081e0 0x7fe0bc1085c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac009990 tx=0x7fe0ac009ca0 comp rx=0 tx=0).stop 2024-03-28T17:47:00.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.172+0000 7fe0c3000700 1 -- 172.21.15.45:0/4209663565 shutdown_connections 2024-03-28T17:47:00.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.172+0000 7fe0c3000700 1 --2- 172.21.15.45:0/4209663565 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0bc06a7e0 0x7fe0bc10f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:00.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.172+0000 7fe0c3000700 1 --2- 172.21.15.45:0/4209663565 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0bc1081e0 0x7fe0bc1085c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:00.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.172+0000 7fe0c3000700 1 --2- 172.21.15.45:0/4209663565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0bc069e40 0x7fe0bc06a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:00.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.172+0000 7fe0c3000700 1 -- 172.21.15.45:0/4209663565 >> 172.21.15.45:0/4209663565 conn(0x7fe0bc077fc0 msgr2=0x7fe0bc0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:00.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.172+0000 7fe0c3000700 1 -- 172.21.15.45:0/4209663565 shutdown_connections 2024-03-28T17:47:00.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.172+0000 7fe0c3000700 1 -- 172.21.15.45:0/4209663565 wait complete. 2024-03-28T17:47:00.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.174+0000 7fe0c3000700 1 Processor -- start 2024-03-28T17:47:00.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.174+0000 7fe0c3000700 1 -- start start 2024-03-28T17:47:00.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.174+0000 7fe0c3000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0bc069e40 0x7fe0bc1a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:00.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.174+0000 7fe0c3000700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0bc06a7e0 0x7fe0bc1a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:00.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.174+0000 7fe0c3000700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0bc1081e0 0x7fe0bc19a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:00.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.174+0000 7fe0c3000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0bc112490 con 0x7fe0bc069e40 2024-03-28T17:47:00.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.174+0000 7fe0c3000700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0bc112310 con 0x7fe0bc1081e0 2024-03-28T17:47:00.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.174+0000 7fe0c3000700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0bc112610 con 0x7fe0bc06a7e0 2024-03-28T17:47:00.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.174+0000 7fe0c1000700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0bc1081e0 0x7fe0bc19a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:00.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.174+0000 7fe0bb400700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0bc06a7e0 0x7fe0bc1a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:00.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.175+0000 7fe0c1000700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0bc1081e0 0x7fe0bc19a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:54756/0 (socket says 172.21.15.45:54756) 2024-03-28T17:47:00.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.175+0000 7fe0c1000700 1 -- 172.21.15.45:0/2833979687 learned_addr learned my addr 172.21.15.45:0/2833979687 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:00.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.175+0000 7fe0bbe00700 1 --2- 172.21.15.45:0/2833979687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0bc069e40 0x7fe0bc1a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:00.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.175+0000 7fe0c1000700 1 -- 172.21.15.45:0/2833979687 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0bc06a7e0 msgr2=0x7fe0bc1a0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:00.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.175+0000 7fe0c1000700 1 --2- 172.21.15.45:0/2833979687 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0bc06a7e0 0x7fe0bc1a0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:00.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.175+0000 7fe0c1000700 1 -- 172.21.15.45:0/2833979687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0bc069e40 msgr2=0x7fe0bc1a07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:00.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.175+0000 7fe0c1000700 1 --2- 172.21.15.45:0/2833979687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0bc069e40 0x7fe0bc1a07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:00.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.175+0000 7fe0c1000700 1 -- 172.21.15.45:0/2833979687 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0bc19b110 con 0x7fe0bc1081e0 2024-03-28T17:47:00.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.175+0000 7fe0bbe00700 1 --2- 172.21.15.45:0/2833979687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0bc069e40 0x7fe0bc1a07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T17:47:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.175+0000 7fe0bb400700 1 --2- 172.21.15.45:0/2833979687 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0bc06a7e0 0x7fe0bc1a0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:47:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.175+0000 7fe0c1000700 1 --2- 172.21.15.45:0/2833979687 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0bc1081e0 0x7fe0bc19a9b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4007ee0 tx=0x7fe0b400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.176+0000 7fe0b8c00700 1 -- 172.21.15.45:0/2833979687 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b400dbd0 con 0x7fe0bc1081e0 2024-03-28T17:47:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.176+0000 7fe0b8c00700 1 -- 172.21.15.45:0/2833979687 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe0b400dd70 con 0x7fe0bc1081e0 2024-03-28T17:47:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.176+0000 7fe0c3000700 1 -- 172.21.15.45:0/2833979687 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0bc19b400 con 0x7fe0bc1081e0 2024-03-28T17:47:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.176+0000 7fe0b8c00700 1 -- 172.21.15.45:0/2833979687 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b4005a60 con 0x7fe0bc1081e0 2024-03-28T17:47:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.176+0000 7fe0c3000700 1 -- 172.21.15.45:0/2833979687 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0bc1a76c0 con 0x7fe0bc1081e0 2024-03-28T17:47:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.177+0000 7fe0c3000700 1 -- 172.21.15.45:0/2833979687 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe0bc10dd50 con 0x7fe0bc1081e0 2024-03-28T17:47:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.179+0000 7fe0b8c00700 1 -- 172.21.15.45:0/2833979687 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fe0b401a020 con 0x7fe0bc1081e0 2024-03-28T17:47:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.180+0000 7fe0b8c00700 1 --2- 172.21.15.45:0/2833979687 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe0a00a29a0 0x7fe0a00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.180+0000 7fe0b8c00700 1 -- 172.21.15.45:0/2833979687 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe0b40ce1b0 con 0x7fe0bc1081e0 2024-03-28T17:47:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.181+0000 7fe0bbe00700 1 --2- 172.21.15.45:0/2833979687 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe0a00a29a0 0x7fe0a00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.181+0000 7fe0b8c00700 1 --2- 172.21.15.45:0/2833979687 >> [v2:172.21.15.45:6802/3795754023,v1:172.21.15.45:6803/3795754023] conn(0x7fe0a00ac180 0x7fe0a00ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.181+0000 7fe0b8c00700 1 -- 172.21.15.45:0/2833979687 --> [v2:172.21.15.45:6802/3795754023,v1:172.21.15.45:6803/3795754023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe0b4016ec0 con 0x7fe0a00ac180 2024-03-28T17:47:00.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.181+0000 7fe0b8c00700 1 -- 172.21.15.45:0/2833979687 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe0b40d5050 con 0x7fe0bc1081e0 2024-03-28T17:47:00.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.181+0000 7fe0bb400700 1 --2- 172.21.15.45:0/2833979687 >> [v2:172.21.15.45:6802/3795754023,v1:172.21.15.45:6803/3795754023] conn(0x7fe0a00ac180 0x7fe0a00ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:00.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.181+0000 7fe0bbe00700 1 --2- 172.21.15.45:0/2833979687 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe0a00a29a0 0x7fe0a00a4e60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac004820 tx=0x7fe0ac004690 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:00.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.182+0000 7fe0bb400700 1 --2- 172.21.15.45:0/2833979687 >> [v2:172.21.15.45:6802/3795754023,v1:172.21.15.45:6803/3795754023] conn(0x7fe0a00ac180 0x7fe0a00ae5a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:00.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.183+0000 7fe0b8c00700 1 -- 172.21.15.45:0/2833979687 <== osd.0 v2:172.21.15.45:6802/3795754023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe0b4016ec0 con 0x7fe0a00ac180 2024-03-28T17:47:00.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.215+0000 7fe0c3000700 1 -- 172.21.15.45:0/2833979687 --> [v2:172.21.15.45:6802/3795754023,v1:172.21.15.45:6803/3795754023] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe0bc103890 con 0x7fe0a00ac180 2024-03-28T17:47:00.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.216+0000 7fe0b8c00700 1 -- 172.21.15.45:0/2833979687 <== osd.0 v2:172.21.15.45:6802/3795754023 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe0bc103890 con 0x7fe0a00ac180 2024-03-28T17:47:00.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.216+0000 7fe0c3000700 1 -- 172.21.15.45:0/2833979687 >> [v2:172.21.15.45:6802/3795754023,v1:172.21.15.45:6803/3795754023] conn(0x7fe0a00ac180 msgr2=0x7fe0a00ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:00.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.216+0000 7fe0c3000700 1 --2- 172.21.15.45:0/2833979687 >> [v2:172.21.15.45:6802/3795754023,v1:172.21.15.45:6803/3795754023] conn(0x7fe0a00ac180 0x7fe0a00ae5a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:00.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.216+0000 7fe0c3000700 1 -- 172.21.15.45:0/2833979687 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe0a00a29a0 msgr2=0x7fe0a00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:00.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.216+0000 7fe0c3000700 1 --2- 172.21.15.45:0/2833979687 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe0a00a29a0 0x7fe0a00a4e60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac004820 tx=0x7fe0ac004690 comp rx=0 tx=0).stop 2024-03-28T17:47:00.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.216+0000 7fe0c3000700 1 -- 172.21.15.45:0/2833979687 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0bc1081e0 msgr2=0x7fe0bc19a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:00.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.216+0000 7fe0c3000700 1 --2- 172.21.15.45:0/2833979687 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0bc1081e0 0x7fe0bc19a9b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4007ee0 tx=0x7fe0b400d3b0 comp rx=0 tx=0).stop 2024-03-28T17:47:00.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.217+0000 7fe0c1000700 1 -- 172.21.15.45:0/2833979687 reap_dead start 2024-03-28T17:47:00.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.217+0000 7fe0c3000700 1 -- 172.21.15.45:0/2833979687 shutdown_connections 2024-03-28T17:47:00.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.217+0000 7fe0c3000700 1 -- 172.21.15.45:0/2833979687 >> 172.21.15.45:0/2833979687 conn(0x7fe0bc077fc0 msgr2=0x7fe0bc10d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:00.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.217+0000 7fe0c3000700 1 -- 172.21.15.45:0/2833979687 shutdown_connections 2024-03-28T17:47:00.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:00.217+0000 7fe0c3000700 1 -- 172.21.15.45:0/2833979687 wait complete. 2024-03-28T17:47:00.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.866+0000 7f5b63000700 1 -- 172.21.15.45:0/220972227 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b5c104e90 msgr2=0x7f5b5c108ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:00.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.866+0000 7f5b63000700 1 --2- 172.21.15.45:0/220972227 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b5c104e90 0x7f5b5c108ee0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5b48009990 tx=0x7f5b48009ca0 comp rx=0 tx=0).stop 2024-03-28T17:47:00.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.866+0000 7f5b63000700 1 -- 172.21.15.45:0/220972227 shutdown_connections 2024-03-28T17:47:00.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.866+0000 7f5b63000700 1 --2- 172.21.15.45:0/220972227 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b5c1095c0 0x7f5b5c111190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:00.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.866+0000 7f5b63000700 1 --2- 172.21.15.45:0/220972227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b5c1044e0 0x7f5b5c1048c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:00.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.866+0000 7f5b63000700 1 --2- 172.21.15.45:0/220972227 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b5c104e90 0x7f5b5c108ee0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:00.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.866+0000 7f5b63000700 1 -- 172.21.15.45:0/220972227 >> 172.21.15.45:0/220972227 conn(0x7f5b5c0ffdb0 msgr2=0x7f5b5c1021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:00.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.866+0000 7f5b63000700 1 -- 172.21.15.45:0/220972227 shutdown_connections 2024-03-28T17:47:00.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.866+0000 7f5b63000700 1 -- 172.21.15.45:0/220972227 wait complete. 2024-03-28T17:47:00.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.867+0000 7f5b63000700 1 Processor -- start 2024-03-28T17:47:00.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.868+0000 7f5b63000700 1 -- start start 2024-03-28T17:47:00.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.868+0000 7f5b63000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b5c1044e0 0x7f5b5c1a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:00.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.868+0000 7f5b63000700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b5c104e90 0x7f5b5c1a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:00.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.868+0000 7f5b63000700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b5c1095c0 0x7f5b5c19a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:00.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.868+0000 7f5b63000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c1145f0 con 0x7f5b5c1044e0 2024-03-28T17:47:00.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.868+0000 7f5b63000700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c114470 con 0x7f5b5c104e90 2024-03-28T17:47:00.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.868+0000 7f5b63000700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c114770 con 0x7f5b5c1095c0 2024-03-28T17:47:00.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.869+0000 7f5b61c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b5c1044e0 0x7f5b5c1a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:00.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.869+0000 7f5b61200700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b5c104e90 0x7f5b5c1a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:00.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.869+0000 7f5b62600700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b5c1095c0 0x7f5b5c19a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:00.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.869+0000 7f5b61200700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b5c104e90 0x7f5b5c1a0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:54740/0 (socket says 172.21.15.45:54740) 2024-03-28T17:47:00.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.869+0000 7f5b61c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b5c1044e0 0x7f5b5c1a06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35238/0 (socket says 172.21.15.45:35238) 2024-03-28T17:47:00.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.869+0000 7f5b61200700 1 -- 172.21.15.45:0/993031 learned_addr learned my addr 172.21.15.45:0/993031 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:00.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.869+0000 7f5b62600700 1 -- 172.21.15.45:0/993031 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b5c104e90 msgr2=0x7f5b5c1a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:00.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.869+0000 7f5b62600700 1 --2- 172.21.15.45:0/993031 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b5c104e90 0x7f5b5c1a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.869+0000 7f5b62600700 1 -- 172.21.15.45:0/993031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b5c1044e0 msgr2=0x7f5b5c1a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.869+0000 7f5b62600700 1 --2- 172.21.15.45:0/993031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b5c1044e0 0x7f5b5c1a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.869+0000 7f5b62600700 1 -- 172.21.15.45:0/993031 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5c19aff0 con 0x7f5b5c1095c0 2024-03-28T17:47:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.869+0000 7f5b62600700 1 --2- 172.21.15.45:0/993031 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b5c1095c0 0x7f5b5c19a890 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400efc0 tx=0x7f5b5400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.870+0000 7f5b5a000700 1 -- 172.21.15.45:0/993031 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b540092e0 con 0x7f5b5c1095c0 2024-03-28T17:47:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.870+0000 7f5b5a000700 1 -- 172.21.15.45:0/993031 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b54004bb0 con 0x7f5b5c1095c0 2024-03-28T17:47:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.870+0000 7f5b5a000700 1 -- 172.21.15.45:0/993031 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b54004280 con 0x7f5b5c1095c0 2024-03-28T17:47:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.870+0000 7f5b63000700 1 -- 172.21.15.45:0/993031 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b5c19b280 con 0x7f5b5c1095c0 2024-03-28T17:47:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.870+0000 7f5b63000700 1 -- 172.21.15.45:0/993031 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b5c1a75a0 con 0x7f5b5c1095c0 2024-03-28T17:47:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.871+0000 7f5b63000700 1 -- 172.21.15.45:0/993031 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5b5c0628a0 con 0x7f5b5c1095c0 2024-03-28T17:47:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.873+0000 7f5b5a000700 1 -- 172.21.15.45:0/993031 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f5b5401b030 con 0x7f5b5c1095c0 2024-03-28T17:47:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.875+0000 7f5b5a000700 1 --2- 172.21.15.45:0/993031 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f5b440a29a0 0x7f5b440a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.875+0000 7f5b5a000700 1 -- 172.21.15.45:0/993031 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f5b540ce5b0 con 0x7f5b5c1095c0 2024-03-28T17:47:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.875+0000 7f5b61c00700 1 --2- 172.21.15.45:0/993031 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f5b440a29a0 0x7f5b440a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:00.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.875+0000 7f5b5a000700 1 --2- 172.21.15.45:0/993031 >> [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684] conn(0x7f5b440ac180 0x7f5b440ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:00.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.875+0000 7f5b5a000700 1 -- 172.21.15.45:0/993031 --> [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b540048a0 con 0x7f5b440ac180 2024-03-28T17:47:00.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.875+0000 7f5b5a000700 1 -- 172.21.15.45:0/993031 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b540d5050 con 0x7f5b5c1095c0 2024-03-28T17:47:00.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.876+0000 7f5b61200700 1 --2- 172.21.15.45:0/993031 >> [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684] conn(0x7f5b440ac180 0x7f5b440ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:00.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.876+0000 7f5b61c00700 1 --2- 172.21.15.45:0/993031 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f5b440a29a0 0x7f5b440a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c19b9d0 tx=0x7f5b50009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:00.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.876+0000 7f5b61200700 1 --2- 172.21.15.45:0/993031 >> [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684] conn(0x7f5b440ac180 0x7f5b440ae5a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:00.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.877+0000 7f5b5a000700 1 -- 172.21.15.45:0/993031 <== osd.9 v2:172.21.15.45:6826/2122322684 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5b540048a0 con 0x7f5b440ac180 2024-03-28T17:47:00.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.896+0000 7f5b63000700 1 -- 172.21.15.45:0/993031 --> [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5b5c062d70 con 0x7f5b440ac180 2024-03-28T17:47:00.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.896+0000 7f5b5a000700 1 -- 172.21.15.45:0/993031 <== osd.9 v2:172.21.15.45:6826/2122322684 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5b5c062d70 con 0x7f5b440ac180 2024-03-28T17:47:00.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.897+0000 7f5b63000700 1 -- 172.21.15.45:0/993031 >> [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684] conn(0x7f5b440ac180 msgr2=0x7f5b440ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:00.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.897+0000 7f5b63000700 1 --2- 172.21.15.45:0/993031 >> [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684] conn(0x7f5b440ac180 0x7f5b440ae5a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:00.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.897+0000 7f5b63000700 1 -- 172.21.15.45:0/993031 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f5b440a29a0 msgr2=0x7f5b440a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:00.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.897+0000 7f5b63000700 1 --2- 172.21.15.45:0/993031 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f5b440a29a0 0x7f5b440a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c19b9d0 tx=0x7f5b50009260 comp rx=0 tx=0).stop 2024-03-28T17:47:00.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.897+0000 7f5b63000700 1 -- 172.21.15.45:0/993031 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b5c1095c0 msgr2=0x7f5b5c19a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:00.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.897+0000 7f5b63000700 1 --2- 172.21.15.45:0/993031 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b5c1095c0 0x7f5b5c19a890 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400efc0 tx=0x7f5b5400c3a0 comp rx=0 tx=0).stop 2024-03-28T17:47:00.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.897+0000 7f5b62600700 1 -- 172.21.15.45:0/993031 reap_dead start 2024-03-28T17:47:00.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.897+0000 7f5b63000700 1 -- 172.21.15.45:0/993031 shutdown_connections 2024-03-28T17:47:00.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.897+0000 7f5b63000700 1 -- 172.21.15.45:0/993031 >> 172.21.15.45:0/993031 conn(0x7f5b5c0ffdb0 msgr2=0x7f5b5c1008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:00.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.897+0000 7f5b63000700 1 -- 172.21.15.45:0/993031 shutdown_connections 2024-03-28T17:47:00.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:46:58.897+0000 7f5b63000700 1 -- 172.21.15.45:0/993031 wait complete. 2024-03-28T17:47:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:01 smithi045 ceph-mon[31773]: pgmap v341: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T17:47:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:01 smithi071 ceph-mon[36597]: pgmap v341: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T17:47:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:01 smithi151 ceph-mon[36481]: pgmap v341: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T17:47:01.980 INFO:teuthology.orchestra.run.smithi045.stdout:287762808867 2024-03-28T17:47:01.980 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-28T17:47:02.454 INFO:teuthology.orchestra.run.smithi045.stdout:309237645340 2024-03-28T17:47:02.454 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-28T17:47:03.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.093+0000 7fa58fe00700 1 -- 172.21.15.45:0/3172515526 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa58810a470 msgr2=0x7fa58810a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:03.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.093+0000 7fa58fe00700 1 --2- 172.21.15.45:0/3172515526 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa58810a470 0x7fa58810a850 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa57c009990 tx=0x7fa57c009ca0 comp rx=0 tx=0).stop 2024-03-28T17:47:03.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.093+0000 7fa58fe00700 1 -- 172.21.15.45:0/3172515526 shutdown_connections 2024-03-28T17:47:03.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.093+0000 7fa58fe00700 1 --2- 172.21.15.45:0/3172515526 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa588104da0 0x7fa5881112b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:03.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.093+0000 7fa58fe00700 1 --2- 172.21.15.45:0/3172515526 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa58810a470 0x7fa58810a850 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:03.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.093+0000 7fa58fe00700 1 --2- 172.21.15.45:0/3172515526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5881043e0 0x7fa588104860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:03.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.093+0000 7fa58fe00700 1 -- 172.21.15.45:0/3172515526 >> 172.21.15.45:0/3172515526 conn(0x7fa5880fff30 msgr2=0x7fa588102350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:03.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.094+0000 7fa58fe00700 1 -- 172.21.15.45:0/3172515526 shutdown_connections 2024-03-28T17:47:03.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.094+0000 7fa58fe00700 1 -- 172.21.15.45:0/3172515526 wait complete. 2024-03-28T17:47:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.095+0000 7fa58fe00700 1 Processor -- start 2024-03-28T17:47:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.095+0000 7fa58fe00700 1 -- start start 2024-03-28T17:47:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.095+0000 7fa58fe00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa5881043e0 0x7fa5881a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.095+0000 7fa58fe00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa588104da0 0x7fa5881a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.095+0000 7fa58fe00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa58810a470 0x7fa58819a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.095+0000 7fa58fe00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5881146f0 con 0x7fa588104da0 2024-03-28T17:47:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.095+0000 7fa58fe00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa588114570 con 0x7fa5881043e0 2024-03-28T17:47:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.095+0000 7fa58fe00700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa588114870 con 0x7fa58810a470 2024-03-28T17:47:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.096+0000 7fa58d400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa5881043e0 0x7fa5881a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.096+0000 7fa58ca00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa588104da0 0x7fa5881a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.096+0000 7fa58d400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa5881043e0 0x7fa5881a07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:54790/0 (socket says 172.21.15.45:54790) 2024-03-28T17:47:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.096+0000 7fa58d400700 1 -- 172.21.15.45:0/2367150096 learned_addr learned my addr 172.21.15.45:0/2367150096 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:03.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.096+0000 7fa58de00700 1 --2- 172.21.15.45:0/2367150096 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa58810a470 0x7fa58819a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:03.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.096+0000 7fa58d400700 1 -- 172.21.15.45:0/2367150096 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa58810a470 msgr2=0x7fa58819a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:03.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.096+0000 7fa58d400700 1 --2- 172.21.15.45:0/2367150096 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa58810a470 0x7fa58819a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:03.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.096+0000 7fa58d400700 1 -- 172.21.15.45:0/2367150096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa588104da0 msgr2=0x7fa5881a0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:03.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.096+0000 7fa58d400700 1 --2- 172.21.15.45:0/2367150096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa588104da0 0x7fa5881a0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:03.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.096+0000 7fa58d400700 1 -- 172.21.15.45:0/2367150096 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58819b110 con 0x7fa5881043e0 2024-03-28T17:47:03.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.096+0000 7fa58de00700 1 --2- 172.21.15.45:0/2367150096 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa58810a470 0x7fa58819a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:47:03.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.096+0000 7fa58d400700 1 --2- 172.21.15.45:0/2367150096 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa5881043e0 0x7fa5881a07f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa57c00bae0 tx=0x7fa57c00bb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:03.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.097+0000 7fa586000700 1 -- 172.21.15.45:0/2367150096 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c026070 con 0x7fa5881043e0 2024-03-28T17:47:03.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.097+0000 7fa586000700 1 -- 172.21.15.45:0/2367150096 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa57c003d70 con 0x7fa5881043e0 2024-03-28T17:47:03.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.097+0000 7fa586000700 1 -- 172.21.15.45:0/2367150096 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c021410 con 0x7fa5881043e0 2024-03-28T17:47:03.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.097+0000 7fa58fe00700 1 -- 172.21.15.45:0/2367150096 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa58819b3a0 con 0x7fa5881043e0 2024-03-28T17:47:03.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.097+0000 7fa58fe00700 1 -- 172.21.15.45:0/2367150096 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5881a76c0 con 0x7fa5881043e0 2024-03-28T17:47:03.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.098+0000 7fa58fe00700 1 -- 172.21.15.45:0/2367150096 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa5880628a0 con 0x7fa5881043e0 2024-03-28T17:47:03.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.100+0000 7fa586000700 1 -- 172.21.15.45:0/2367150096 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fa57c029070 con 0x7fa5881043e0 2024-03-28T17:47:03.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.101+0000 7fa586000700 1 --2- 172.21.15.45:0/2367150096 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fa5700a29a0 0x7fa5700a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:03.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.101+0000 7fa586000700 1 -- 172.21.15.45:0/2367150096 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa57c0cf850 con 0x7fa5881043e0 2024-03-28T17:47:03.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.101+0000 7fa58ca00700 1 --2- 172.21.15.45:0/2367150096 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fa5700a29a0 0x7fa5700a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:03.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.102+0000 7fa586000700 1 --2- 172.21.15.45:0/2367150096 >> [v2:172.21.15.151:6816/3665460137,v1:172.21.15.151:6817/3665460137] conn(0x7fa5700ac180 0x7fa5700ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:03.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.102+0000 7fa586000700 1 -- 172.21.15.45:0/2367150096 --> [v2:172.21.15.151:6816/3665460137,v1:172.21.15.151:6817/3665460137] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa5700aec50 con 0x7fa5700ac180 2024-03-28T17:47:03.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.102+0000 7fa586000700 1 -- 172.21.15.45:0/2367150096 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa57c0d42a0 con 0x7fa5881043e0 2024-03-28T17:47:03.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.102+0000 7fa58de00700 1 --2- 172.21.15.45:0/2367150096 >> [v2:172.21.15.151:6816/3665460137,v1:172.21.15.151:6817/3665460137] conn(0x7fa5700ac180 0x7fa5700ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:03.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.102+0000 7fa58ca00700 1 --2- 172.21.15.45:0/2367150096 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fa5700a29a0 0x7fa5700a4e60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa588102560 tx=0x7fa5740073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:03.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.103+0000 7fa58de00700 1 --2- 172.21.15.45:0/2367150096 >> [v2:172.21.15.151:6816/3665460137,v1:172.21.15.151:6817/3665460137] conn(0x7fa5700ac180 0x7fa5700ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:03.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.103+0000 7fa586000700 1 -- 172.21.15.45:0/2367150096 <== osd.8 v2:172.21.15.151:6816/3665460137 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa5700aec50 con 0x7fa5700ac180 2024-03-28T17:47:03.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.122+0000 7fa58fe00700 1 -- 172.21.15.45:0/2367150096 --> [v2:172.21.15.151:6816/3665460137,v1:172.21.15.151:6817/3665460137] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa588062d70 con 0x7fa5700ac180 2024-03-28T17:47:03.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.122+0000 7fa586000700 1 -- 172.21.15.45:0/2367150096 <== osd.8 v2:172.21.15.151:6816/3665460137 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa588062d70 con 0x7fa5700ac180 2024-03-28T17:47:03.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.122+0000 7fa58fe00700 1 -- 172.21.15.45:0/2367150096 >> [v2:172.21.15.151:6816/3665460137,v1:172.21.15.151:6817/3665460137] conn(0x7fa5700ac180 msgr2=0x7fa5700ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:03.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.122+0000 7fa58fe00700 1 --2- 172.21.15.45:0/2367150096 >> [v2:172.21.15.151:6816/3665460137,v1:172.21.15.151:6817/3665460137] conn(0x7fa5700ac180 0x7fa5700ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:03.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.123+0000 7fa58fe00700 1 -- 172.21.15.45:0/2367150096 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fa5700a29a0 msgr2=0x7fa5700a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:03.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.123+0000 7fa58fe00700 1 --2- 172.21.15.45:0/2367150096 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fa5700a29a0 0x7fa5700a4e60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa588102560 tx=0x7fa5740073d0 comp rx=0 tx=0).stop 2024-03-28T17:47:03.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.123+0000 7fa58fe00700 1 -- 172.21.15.45:0/2367150096 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa5881043e0 msgr2=0x7fa5881a07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:03.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.123+0000 7fa58fe00700 1 --2- 172.21.15.45:0/2367150096 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa5881043e0 0x7fa5881a07f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa57c00bae0 tx=0x7fa57c00bb10 comp rx=0 tx=0).stop 2024-03-28T17:47:03.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.123+0000 7fa58de00700 1 -- 172.21.15.45:0/2367150096 reap_dead start 2024-03-28T17:47:03.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.123+0000 7fa58fe00700 1 -- 172.21.15.45:0/2367150096 shutdown_connections 2024-03-28T17:47:03.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.123+0000 7fa58fe00700 1 -- 172.21.15.45:0/2367150096 >> 172.21.15.45:0/2367150096 conn(0x7fa5880fff30 msgr2=0x7fa588100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:03.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.123+0000 7fa58fe00700 1 -- 172.21.15.45:0/2367150096 shutdown_connections 2024-03-28T17:47:03.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:03.123+0000 7fa58fe00700 1 -- 172.21.15.45:0/2367150096 wait complete. 2024-03-28T17:47:03.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:03 smithi045 ceph-mon[31773]: pgmap v342: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:47:03.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:03 smithi071 ceph-mon[36597]: pgmap v342: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:47:03.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:03 smithi151 ceph-mon[36481]: pgmap v342: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:47:04.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.292+0000 7f27bf600700 1 -- 172.21.15.45:0/366679090 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f27b8102440 msgr2=0x7f27b8102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:04.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.292+0000 7f27bf600700 1 --2- 172.21.15.45:0/366679090 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f27b8102440 0x7f27b8102820 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f27ac0098f0 tx=0x7f27ac009c00 comp rx=0 tx=0).stop 2024-03-28T17:47:04.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.292+0000 7f27bf600700 1 -- 172.21.15.45:0/366679090 shutdown_connections 2024-03-28T17:47:04.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.292+0000 7f27bf600700 1 --2- 172.21.15.45:0/366679090 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f27b8107510 0x7f27b810f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:04.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.292+0000 7f27bf600700 1 --2- 172.21.15.45:0/366679090 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f27b8102440 0x7f27b8102820 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:04.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.292+0000 7f27bf600700 1 --2- 172.21.15.45:0/366679090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b8102df0 0x7f27b8106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:04.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.292+0000 7f27bf600700 1 -- 172.21.15.45:0/366679090 >> 172.21.15.45:0/366679090 conn(0x7f27b80792f0 msgr2=0x7f27b8079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:04.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.293+0000 7f27bf600700 1 -- 172.21.15.45:0/366679090 shutdown_connections 2024-03-28T17:47:04.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.293+0000 7f27bf600700 1 -- 172.21.15.45:0/366679090 wait complete. 2024-03-28T17:47:04.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.294+0000 7f27bf600700 1 Processor -- start 2024-03-28T17:47:04.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.294+0000 7f27bf600700 1 -- start start 2024-03-28T17:47:04.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.295+0000 7f27bf600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b8102440 0x7f27b81a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:04.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.295+0000 7f27bf600700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f27b8102df0 0x7f27b81a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:04.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.295+0000 7f27bf600700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f27b8107510 0x7f27b819aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:04.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.295+0000 7f27bf600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b8112570 con 0x7f27b8102440 2024-03-28T17:47:04.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.295+0000 7f27bf600700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b81123f0 con 0x7f27b8107510 2024-03-28T17:47:04.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.295+0000 7f27bf600700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b81126f0 con 0x7f27b8102df0 2024-03-28T17:47:04.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.295+0000 7f27bd600700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f27b8107510 0x7f27b819aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:04.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.295+0000 7f27b7e00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f27b8102df0 0x7f27b81a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:04.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.295+0000 7f27bd600700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f27b8107510 0x7f27b819aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:54778/0 (socket says 172.21.15.45:54778) 2024-03-28T17:47:04.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.295+0000 7f27bd600700 1 -- 172.21.15.45:0/255040313 learned_addr learned my addr 172.21.15.45:0/255040313 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:04.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.295+0000 7f27bcc00700 1 --2- 172.21.15.45:0/255040313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b8102440 0x7f27b81a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:04.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.296+0000 7f27bd600700 1 -- 172.21.15.45:0/255040313 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f27b8102df0 msgr2=0x7f27b81a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:04.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.296+0000 7f27bd600700 1 --2- 172.21.15.45:0/255040313 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f27b8102df0 0x7f27b81a0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:04.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.296+0000 7f27bd600700 1 -- 172.21.15.45:0/255040313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b8102440 msgr2=0x7f27b81a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:04.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.296+0000 7f27bd600700 1 --2- 172.21.15.45:0/255040313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b8102440 0x7f27b81a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:04.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.296+0000 7f27bd600700 1 -- 172.21.15.45:0/255040313 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b819b1a0 con 0x7f27b8107510 2024-03-28T17:47:04.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.296+0000 7f27bcc00700 1 --2- 172.21.15.45:0/255040313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b8102440 0x7f27b81a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:47:04.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.296+0000 7f27bd600700 1 --2- 172.21.15.45:0/255040313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f27b8107510 0x7f27b819aa40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f27b000ef90 tx=0x7f27b000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:04.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.296+0000 7f27b5600700 1 -- 172.21.15.45:0/255040313 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b00092e0 con 0x7f27b8107510 2024-03-28T17:47:04.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.296+0000 7f27b5600700 1 -- 172.21.15.45:0/255040313 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f27b0004bb0 con 0x7f27b8107510 2024-03-28T17:47:04.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.296+0000 7f27b5600700 1 -- 172.21.15.45:0/255040313 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b0004280 con 0x7f27b8107510 2024-03-28T17:47:04.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.296+0000 7f27bf600700 1 -- 172.21.15.45:0/255040313 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b819b430 con 0x7f27b8107510 2024-03-28T17:47:04.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.296+0000 7f27bf600700 1 -- 172.21.15.45:0/255040313 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b81a7750 con 0x7f27b8107510 2024-03-28T17:47:04.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.298+0000 7f27bf600700 1 -- 172.21.15.45:0/255040313 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f27b810ed50 con 0x7f27b8107510 2024-03-28T17:47:04.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.299+0000 7f27b5600700 1 -- 172.21.15.45:0/255040313 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f27b001b030 con 0x7f27b8107510 2024-03-28T17:47:04.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.301+0000 7f27b5600700 1 --2- 172.21.15.45:0/255040313 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f27a00a29a0 0x7f27a00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:04.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.301+0000 7f27b5600700 1 -- 172.21.15.45:0/255040313 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f27b00ce650 con 0x7f27b8107510 2024-03-28T17:47:04.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.301+0000 7f27bcc00700 1 --2- 172.21.15.45:0/255040313 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f27a00a29a0 0x7f27a00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:04.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.301+0000 7f27b5600700 1 --2- 172.21.15.45:0/255040313 >> [v2:172.21.15.71:6808/2069578333,v1:172.21.15.71:6809/2069578333] conn(0x7f27a00ac180 0x7f27a00ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:04.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.301+0000 7f27b5600700 1 -- 172.21.15.45:0/255040313 --> [v2:172.21.15.71:6808/2069578333,v1:172.21.15.71:6809/2069578333] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27b000be50 con 0x7f27a00ac180 2024-03-28T17:47:04.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.301+0000 7f27b5600700 1 -- 172.21.15.45:0/255040313 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27b00d5050 con 0x7f27b8107510 2024-03-28T17:47:04.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.302+0000 7f27b7e00700 1 --2- 172.21.15.45:0/255040313 >> [v2:172.21.15.71:6808/2069578333,v1:172.21.15.71:6809/2069578333] conn(0x7f27a00ac180 0x7f27a00ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:04.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.302+0000 7f27bcc00700 1 --2- 172.21.15.45:0/255040313 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f27a00a29a0 0x7f27a00a4e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f27ac0098c0 tx=0x7f27ac019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:04.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.302+0000 7f27b7e00700 1 --2- 172.21.15.45:0/255040313 >> [v2:172.21.15.71:6808/2069578333,v1:172.21.15.71:6809/2069578333] conn(0x7f27a00ac180 0x7f27a00ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:04.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.303+0000 7f27b5600700 1 -- 172.21.15.45:0/255040313 <== osd.4 v2:172.21.15.71:6808/2069578333 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f27b000be50 con 0x7f27a00ac180 2024-03-28T17:47:04.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.322+0000 7f27bf600700 1 -- 172.21.15.45:0/255040313 --> [v2:172.21.15.71:6808/2069578333,v1:172.21.15.71:6809/2069578333] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f27b8103910 con 0x7f27a00ac180 2024-03-28T17:47:04.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.322+0000 7f27b5600700 1 -- 172.21.15.45:0/255040313 <== osd.4 v2:172.21.15.71:6808/2069578333 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f27b8103910 con 0x7f27a00ac180 2024-03-28T17:47:04.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.323+0000 7f27bf600700 1 -- 172.21.15.45:0/255040313 >> [v2:172.21.15.71:6808/2069578333,v1:172.21.15.71:6809/2069578333] conn(0x7f27a00ac180 msgr2=0x7f27a00ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:04.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.323+0000 7f27bf600700 1 --2- 172.21.15.45:0/255040313 >> [v2:172.21.15.71:6808/2069578333,v1:172.21.15.71:6809/2069578333] conn(0x7f27a00ac180 0x7f27a00ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:04.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.323+0000 7f27bf600700 1 -- 172.21.15.45:0/255040313 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f27a00a29a0 msgr2=0x7f27a00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:04.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.323+0000 7f27bf600700 1 --2- 172.21.15.45:0/255040313 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f27a00a29a0 0x7f27a00a4e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f27ac0098c0 tx=0x7f27ac019040 comp rx=0 tx=0).stop 2024-03-28T17:47:04.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.323+0000 7f27bf600700 1 -- 172.21.15.45:0/255040313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f27b8107510 msgr2=0x7f27b819aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:04.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.323+0000 7f27bf600700 1 --2- 172.21.15.45:0/255040313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f27b8107510 0x7f27b819aa40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f27b000ef90 tx=0x7f27b000c3a0 comp rx=0 tx=0).stop 2024-03-28T17:47:04.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.323+0000 7f27bd600700 1 -- 172.21.15.45:0/255040313 reap_dead start 2024-03-28T17:47:04.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.323+0000 7f27bf600700 1 -- 172.21.15.45:0/255040313 shutdown_connections 2024-03-28T17:47:04.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.323+0000 7f27bf600700 1 -- 172.21.15.45:0/255040313 >> 172.21.15.45:0/255040313 conn(0x7f27b80792f0 msgr2=0x7f27b810e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:04.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.323+0000 7f27bf600700 1 -- 172.21.15.45:0/255040313 shutdown_connections 2024-03-28T17:47:04.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:02.323+0000 7f27bf600700 1 -- 172.21.15.45:0/255040313 wait complete. 2024-03-28T17:47:04.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.612+0000 7f83f4200700 1 -- 172.21.15.45:0/3833062250 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83ec102410 msgr2=0x7f83ec1027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:04.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.612+0000 7f83f4200700 1 --2- 172.21.15.45:0/3833062250 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83ec102410 0x7f83ec1027f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f83e00099c0 tx=0x7f83e0009cd0 comp rx=0 tx=0).stop 2024-03-28T17:47:04.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.612+0000 7f83f4200700 1 -- 172.21.15.45:0/3833062250 shutdown_connections 2024-03-28T17:47:04.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.612+0000 7f83f4200700 1 --2- 172.21.15.45:0/3833062250 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83ec102410 0x7f83ec1027f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:04.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.612+0000 7f83f4200700 1 --2- 172.21.15.45:0/3833062250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83ec1074f0 0x7f83ec10f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:04.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.612+0000 7f83f4200700 1 --2- 172.21.15.45:0/3833062250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83ec102dc0 0x7f83ec106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:04.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.612+0000 7f83f4200700 1 -- 172.21.15.45:0/3833062250 >> 172.21.15.45:0/3833062250 conn(0x7f83ec0fdc60 msgr2=0x7f83ec100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:04.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.612+0000 7f83f4200700 1 -- 172.21.15.45:0/3833062250 shutdown_connections 2024-03-28T17:47:04.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.612+0000 7f83f4200700 1 -- 172.21.15.45:0/3833062250 wait complete. 2024-03-28T17:47:04.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.613+0000 7f83f4200700 1 Processor -- start 2024-03-28T17:47:04.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.613+0000 7f83f4200700 1 -- start start 2024-03-28T17:47:04.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.614+0000 7f83f4200700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83ec102410 0x7f83ec19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:04.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.614+0000 7f83f4200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83ec102dc0 0x7f83ec19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:04.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.614+0000 7f83f4200700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83ec1074f0 0x7f83ec1987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:04.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.614+0000 7f83f4200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ec112500 con 0x7f83ec102dc0 2024-03-28T17:47:04.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.614+0000 7f83f4200700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ec112380 con 0x7f83ec102410 2024-03-28T17:47:04.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.614+0000 7f83f4200700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ec112680 con 0x7f83ec1074f0 2024-03-28T17:47:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.614+0000 7f83f1800700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83ec102410 0x7f83ec19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.614+0000 7f83f1800700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83ec102410 0x7f83ec19e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:54808/0 (socket says 172.21.15.45:54808) 2024-03-28T17:47:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.614+0000 7f83f1800700 1 -- 172.21.15.45:0/2872485159 learned_addr learned my addr 172.21.15.45:0/2872485159 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.614+0000 7f83f0e00700 1 --2- 172.21.15.45:0/2872485159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83ec102dc0 0x7f83ec19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.614+0000 7f83f2200700 1 --2- 172.21.15.45:0/2872485159 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83ec1074f0 0x7f83ec1987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.614+0000 7f83f1800700 1 -- 172.21.15.45:0/2872485159 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83ec1074f0 msgr2=0x7f83ec1987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.614+0000 7f83f1800700 1 --2- 172.21.15.45:0/2872485159 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83ec1074f0 0x7f83ec1987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.614+0000 7f83f1800700 1 -- 172.21.15.45:0/2872485159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83ec102dc0 msgr2=0x7f83ec19ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.614+0000 7f83f1800700 1 --2- 172.21.15.45:0/2872485159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83ec102dc0 0x7f83ec19ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.614+0000 7f83f1800700 1 -- 172.21.15.45:0/2872485159 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ec1990c0 con 0x7f83ec102410 2024-03-28T17:47:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.614+0000 7f83f2200700 1 --2- 172.21.15.45:0/2872485159 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83ec1074f0 0x7f83ec1987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:47:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.615+0000 7f83f1800700 1 --2- 172.21.15.45:0/2872485159 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83ec102410 0x7f83ec19e670 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f83e0006010 tx=0x7f83e000f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.615+0000 7f83ea000700 1 -- 172.21.15.45:0/2872485159 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e0003680 con 0x7f83ec102410 2024-03-28T17:47:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.615+0000 7f83ea000700 1 -- 172.21.15.45:0/2872485159 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f83e00055b0 con 0x7f83ec102410 2024-03-28T17:47:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.615+0000 7f83ea000700 1 -- 172.21.15.45:0/2872485159 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e00173f0 con 0x7f83ec102410 2024-03-28T17:47:04.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.615+0000 7f83f4200700 1 -- 172.21.15.45:0/2872485159 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83ec199350 con 0x7f83ec102410 2024-03-28T17:47:04.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.615+0000 7f83f4200700 1 -- 172.21.15.45:0/2872485159 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83ec1a5420 con 0x7f83ec102410 2024-03-28T17:47:04.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.616+0000 7f83f4200700 1 -- 172.21.15.45:0/2872485159 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f83ec0628e0 con 0x7f83ec102410 2024-03-28T17:47:04.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.618+0000 7f83ea000700 1 -- 172.21.15.45:0/2872485159 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f83e000fca0 con 0x7f83ec102410 2024-03-28T17:47:04.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.619+0000 7f83ea000700 1 --2- 172.21.15.45:0/2872485159 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f83d40a2b40 0x7f83d40a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:04.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.619+0000 7f83ea000700 1 -- 172.21.15.45:0/2872485159 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f83e00cf9a0 con 0x7f83ec102410 2024-03-28T17:47:04.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.619+0000 7f83f0e00700 1 --2- 172.21.15.45:0/2872485159 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f83d40a2b40 0x7f83d40a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:04.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.620+0000 7f83ea000700 1 --2- 172.21.15.45:0/2872485159 >> [v2:172.21.15.151:6808/2629217227,v1:172.21.15.151:6809/2629217227] conn(0x7f83d40ac320 0x7f83d40ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:04.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.620+0000 7f83ea000700 1 -- 172.21.15.45:0/2872485159 --> [v2:172.21.15.151:6808/2629217227,v1:172.21.15.151:6809/2629217227] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f83e0003680 con 0x7f83d40ac320 2024-03-28T17:47:04.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.620+0000 7f83ea000700 1 -- 172.21.15.45:0/2872485159 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f83e00d42a0 con 0x7f83ec102410 2024-03-28T17:47:04.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.620+0000 7f83f2200700 1 --2- 172.21.15.45:0/2872485159 >> [v2:172.21.15.151:6808/2629217227,v1:172.21.15.151:6809/2629217227] conn(0x7f83d40ac320 0x7f83d40ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:04.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.620+0000 7f83f0e00700 1 --2- 172.21.15.45:0/2872485159 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f83d40a2b40 0x7f83d40a5000 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f83d8009680 tx=0x7f83d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:04.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.621+0000 7f83f2200700 1 --2- 172.21.15.45:0/2872485159 >> [v2:172.21.15.151:6808/2629217227,v1:172.21.15.151:6809/2629217227] conn(0x7f83d40ac320 0x7f83d40ae740 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:04.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.621+0000 7f83ea000700 1 -- 172.21.15.45:0/2872485159 <== osd.5 v2:172.21.15.151:6808/2629217227 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f83e0003680 con 0x7f83d40ac320 2024-03-28T17:47:04.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.640+0000 7f83f4200700 1 -- 172.21.15.45:0/2872485159 --> [v2:172.21.15.151:6808/2629217227,v1:172.21.15.151:6809/2629217227] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f83ec062db0 con 0x7f83d40ac320 2024-03-28T17:47:04.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.640+0000 7f83ea000700 1 -- 172.21.15.45:0/2872485159 <== osd.5 v2:172.21.15.151:6808/2629217227 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f83ec062db0 con 0x7f83d40ac320 2024-03-28T17:47:04.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.641+0000 7f83f4200700 1 -- 172.21.15.45:0/2872485159 >> [v2:172.21.15.151:6808/2629217227,v1:172.21.15.151:6809/2629217227] conn(0x7f83d40ac320 msgr2=0x7f83d40ae740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:04.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.641+0000 7f83f4200700 1 --2- 172.21.15.45:0/2872485159 >> [v2:172.21.15.151:6808/2629217227,v1:172.21.15.151:6809/2629217227] conn(0x7f83d40ac320 0x7f83d40ae740 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:04.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.641+0000 7f83f4200700 1 -- 172.21.15.45:0/2872485159 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f83d40a2b40 msgr2=0x7f83d40a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:04.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.641+0000 7f83f4200700 1 --2- 172.21.15.45:0/2872485159 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f83d40a2b40 0x7f83d40a5000 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f83d8009680 tx=0x7f83d8009260 comp rx=0 tx=0).stop 2024-03-28T17:47:04.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.641+0000 7f83f4200700 1 -- 172.21.15.45:0/2872485159 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83ec102410 msgr2=0x7f83ec19e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:04.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.641+0000 7f83f4200700 1 --2- 172.21.15.45:0/2872485159 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83ec102410 0x7f83ec19e670 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f83e0006010 tx=0x7f83e000f690 comp rx=0 tx=0).stop 2024-03-28T17:47:04.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.641+0000 7f83f2200700 1 -- 172.21.15.45:0/2872485159 reap_dead start 2024-03-28T17:47:04.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.641+0000 7f83f4200700 1 -- 172.21.15.45:0/2872485159 shutdown_connections 2024-03-28T17:47:04.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.641+0000 7f83f4200700 1 -- 172.21.15.45:0/2872485159 >> 172.21.15.45:0/2872485159 conn(0x7f83ec0fdc60 msgr2=0x7f83ec0fffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:04.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.641+0000 7f83f4200700 1 -- 172.21.15.45:0/2872485159 shutdown_connections 2024-03-28T17:47:04.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:04.641+0000 7f83f4200700 1 -- 172.21.15.45:0/2872485159 wait complete. 2024-03-28T17:47:05.492 INFO:teuthology.orchestra.run.smithi045.stdout:141733920832 2024-03-28T17:47:05.492 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-28T17:47:05.497 INFO:teuthology.orchestra.run.smithi045.stdout:34359738455 2024-03-28T17:47:05.497 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-28T17:47:05.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:05 smithi045 ceph-mon[31773]: pgmap v343: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:47:05.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.716+0000 7f5049600700 1 -- 172.21.15.45:0/3594284995 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50441074b0 msgr2=0x7f504410f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:05.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.716+0000 7f5049600700 1 --2- 172.21.15.45:0/3594284995 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50441074b0 0x7f504410f080 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f503c00b0a0 tx=0x7f503c00b3b0 comp rx=0 tx=0).stop 2024-03-28T17:47:05.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.717+0000 7f5049600700 1 -- 172.21.15.45:0/3594284995 shutdown_connections 2024-03-28T17:47:05.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.717+0000 7f5049600700 1 --2- 172.21.15.45:0/3594284995 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f50441023d0 0x7f50441027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:05.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.717+0000 7f5049600700 1 --2- 172.21.15.45:0/3594284995 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50441074b0 0x7f504410f080 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:05.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.717+0000 7f5049600700 1 --2- 172.21.15.45:0/3594284995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5044102d80 0x7f5044106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:05.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.717+0000 7f5049600700 1 -- 172.21.15.45:0/3594284995 >> 172.21.15.45:0/3594284995 conn(0x7f50440fdc40 msgr2=0x7f5044100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:05.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.717+0000 7f5049600700 1 -- 172.21.15.45:0/3594284995 shutdown_connections 2024-03-28T17:47:05.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.717+0000 7f5049600700 1 -- 172.21.15.45:0/3594284995 wait complete. 2024-03-28T17:47:05.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.718+0000 7f5049600700 1 Processor -- start 2024-03-28T17:47:05.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.718+0000 7f5049600700 1 -- start start 2024-03-28T17:47:05.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.718+0000 7f5049600700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f50441023d0 0x7f504419f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:05.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.718+0000 7f5049600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5044102d80 0x7f504419fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:05.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.718+0000 7f5049600700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50441074b0 0x7f504410bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:05.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.718+0000 7f5049600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5044112510 con 0x7f5044102d80 2024-03-28T17:47:05.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.718+0000 7f5049600700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5044112390 con 0x7f50441074b0 2024-03-28T17:47:05.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.718+0000 7f5049600700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5044112690 con 0x7f50441023d0 2024-03-28T17:47:05.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.719+0000 7f5042000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5044102d80 0x7f504419fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:05.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.719+0000 7f5042000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5044102d80 0x7f504419fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35320/0 (socket says 172.21.15.45:35320) 2024-03-28T17:47:05.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.719+0000 7f5042000700 1 -- 172.21.15.45:0/802447129 learned_addr learned my addr 172.21.15.45:0/802447129 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:05.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.719+0000 7f5042000700 1 -- 172.21.15.45:0/802447129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f50441023d0 msgr2=0x7f504419f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:05.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.719+0000 7f5042000700 1 --2- 172.21.15.45:0/802447129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f50441023d0 0x7f504419f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:05.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.719+0000 7f5042000700 1 -- 172.21.15.45:0/802447129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50441074b0 msgr2=0x7f504410bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:05.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.719+0000 7f5042000700 1 --2- 172.21.15.45:0/802447129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50441074b0 0x7f504410bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:05.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.719+0000 7f5042000700 1 -- 172.21.15.45:0/802447129 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504410c860 con 0x7f5044102d80 2024-03-28T17:47:05.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.719+0000 7f5042000700 1 --2- 172.21.15.45:0/802447129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5044102d80 0x7f504419fb50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f503000e970 tx=0x7f503000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:05.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.720+0000 7f503b400700 1 -- 172.21.15.45:0/802447129 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503000cbd0 con 0x7f5044102d80 2024-03-28T17:47:05.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.720+0000 7f5049600700 1 -- 172.21.15.45:0/802447129 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f504410cb50 con 0x7f5044102d80 2024-03-28T17:47:05.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.720+0000 7f5049600700 1 -- 172.21.15.45:0/802447129 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f504419b170 con 0x7f5044102d80 2024-03-28T17:47:05.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.721+0000 7f503b400700 1 -- 172.21.15.45:0/802447129 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f503000cd70 con 0x7f5044102d80 2024-03-28T17:47:05.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.721+0000 7f5049600700 1 -- 172.21.15.45:0/802447129 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5044075870 con 0x7f5044102d80 2024-03-28T17:47:05.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.721+0000 7f503b400700 1 -- 172.21.15.45:0/802447129 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5030010430 con 0x7f5044102d80 2024-03-28T17:47:05.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.724+0000 7f503b400700 1 -- 172.21.15.45:0/802447129 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f50300105d0 con 0x7f5044102d80 2024-03-28T17:47:05.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.726+0000 7f503b400700 1 --2- 172.21.15.45:0/802447129 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f50280a29a0 0x7f50280a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:05.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.726+0000 7f503b400700 1 -- 172.21.15.45:0/802447129 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f50300cf430 con 0x7f5044102d80 2024-03-28T17:47:05.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.726+0000 7f5042a00700 1 --2- 172.21.15.45:0/802447129 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f50280a29a0 0x7f50280a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:05.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.727+0000 7f503b400700 1 --2- 172.21.15.45:0/802447129 >> [v2:172.21.15.45:6810/3021569975,v1:172.21.15.45:6811/3021569975] conn(0x7f50280ac180 0x7f50280ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:05.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.727+0000 7f503b400700 1 -- 172.21.15.45:0/802447129 --> [v2:172.21.15.45:6810/3021569975,v1:172.21.15.45:6811/3021569975] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5030012ec0 con 0x7f50280ac180 2024-03-28T17:47:05.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.727+0000 7f503b400700 1 -- 172.21.15.45:0/802447129 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50300cf7e0 con 0x7f5044102d80 2024-03-28T17:47:05.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.727+0000 7f5043400700 1 --2- 172.21.15.45:0/802447129 >> [v2:172.21.15.45:6810/3021569975,v1:172.21.15.45:6811/3021569975] conn(0x7f50280ac180 0x7f50280ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:05.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.728+0000 7f5042a00700 1 --2- 172.21.15.45:0/802447129 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f50280a29a0 0x7f50280a4e60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5034005950 tx=0x7f50340058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:05.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.728+0000 7f5043400700 1 --2- 172.21.15.45:0/802447129 >> [v2:172.21.15.45:6810/3021569975,v1:172.21.15.45:6811/3021569975] conn(0x7f50280ac180 0x7f50280ae5a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:05.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.730+0000 7f503b400700 1 -- 172.21.15.45:0/802447129 <== osd.3 v2:172.21.15.45:6810/3021569975 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5030012ec0 con 0x7f50280ac180 2024-03-28T17:47:05.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.748+0000 7f5049600700 1 -- 172.21.15.45:0/802447129 --> [v2:172.21.15.45:6810/3021569975,v1:172.21.15.45:6811/3021569975] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f50441038b0 con 0x7f50280ac180 2024-03-28T17:47:05.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.749+0000 7f503b400700 1 -- 172.21.15.45:0/802447129 <== osd.3 v2:172.21.15.45:6810/3021569975 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f50441038b0 con 0x7f50280ac180 2024-03-28T17:47:05.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.749+0000 7f5049600700 1 -- 172.21.15.45:0/802447129 >> [v2:172.21.15.45:6810/3021569975,v1:172.21.15.45:6811/3021569975] conn(0x7f50280ac180 msgr2=0x7f50280ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:05.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.750+0000 7f5049600700 1 --2- 172.21.15.45:0/802447129 >> [v2:172.21.15.45:6810/3021569975,v1:172.21.15.45:6811/3021569975] conn(0x7f50280ac180 0x7f50280ae5a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:05.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.750+0000 7f5049600700 1 -- 172.21.15.45:0/802447129 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f50280a29a0 msgr2=0x7f50280a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:05.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.750+0000 7f5049600700 1 --2- 172.21.15.45:0/802447129 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f50280a29a0 0x7f50280a4e60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5034005950 tx=0x7f50340058e0 comp rx=0 tx=0).stop 2024-03-28T17:47:05.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.750+0000 7f5049600700 1 -- 172.21.15.45:0/802447129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5044102d80 msgr2=0x7f504419fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:05.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.750+0000 7f5049600700 1 --2- 172.21.15.45:0/802447129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5044102d80 0x7f504419fb50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f503000e970 tx=0x7f503000ed30 comp rx=0 tx=0).stop 2024-03-28T17:47:05.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.750+0000 7f5043400700 1 -- 172.21.15.45:0/802447129 reap_dead start 2024-03-28T17:47:05.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.750+0000 7f5049600700 1 -- 172.21.15.45:0/802447129 shutdown_connections 2024-03-28T17:47:05.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.750+0000 7f5049600700 1 -- 172.21.15.45:0/802447129 >> 172.21.15.45:0/802447129 conn(0x7f50440fdc40 msgr2=0x7f504410d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:05.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.751+0000 7f5049600700 1 -- 172.21.15.45:0/802447129 shutdown_connections 2024-03-28T17:47:05.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:05.751+0000 7f5049600700 1 -- 172.21.15.45:0/802447129 wait complete. 2024-03-28T17:47:05.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:05 smithi071 ceph-mon[36597]: pgmap v343: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:47:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:05 smithi151 ceph-mon[36481]: pgmap v343: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:47:07.304 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:47:07.348 INFO:teuthology.orchestra.run.smithi045.stdout:107374182470 2024-03-28T17:47:07.348 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-28T17:47:07.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:07 smithi045 ceph-mon[31773]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:47:07.669 INFO:teuthology.orchestra.run.smithi045.stdout:163208757307 2024-03-28T17:47:07.670 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-28T17:47:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:07 smithi071 ceph-mon[36597]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:47:07.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:07 smithi151 ceph-mon[36481]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:47:07.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.184+0000 7f87ccc00700 1 -- 172.21.15.45:0/684384921 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f87c8076fe0 msgr2=0x7f87c8112b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:07.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.184+0000 7f87ccc00700 1 --2- 172.21.15.45:0/684384921 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f87c8076fe0 0x7f87c8112b00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009990 tx=0x7f87b4009ca0 comp rx=0 tx=0).stop 2024-03-28T17:47:07.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.184+0000 7f87ccc00700 1 -- 172.21.15.45:0/684384921 shutdown_connections 2024-03-28T17:47:07.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.184+0000 7f87ccc00700 1 --2- 172.21.15.45:0/684384921 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87c8113040 0x7f87c8115430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:07.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.184+0000 7f87ccc00700 1 --2- 172.21.15.45:0/684384921 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87c80766c0 0x7f87c8076aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:07.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.184+0000 7f87ccc00700 1 --2- 172.21.15.45:0/684384921 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f87c8076fe0 0x7f87c8112b00 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:07.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.184+0000 7f87ccc00700 1 -- 172.21.15.45:0/684384921 >> 172.21.15.45:0/684384921 conn(0x7f87c80ffe90 msgr2=0x7f87c81022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:07.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.184+0000 7f87ccc00700 1 -- 172.21.15.45:0/684384921 shutdown_connections 2024-03-28T17:47:07.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.184+0000 7f87ccc00700 1 -- 172.21.15.45:0/684384921 wait complete. 2024-03-28T17:47:07.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.185+0000 7f87ccc00700 1 Processor -- start 2024-03-28T17:47:07.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.185+0000 7f87ccc00700 1 -- start start 2024-03-28T17:47:07.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.185+0000 7f87ccc00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f87c80766c0 0x7f87c819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:07.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.185+0000 7f87ccc00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87c8076fe0 0x7f87c819f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:07.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.186+0000 7f87ccc00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87c8113040 0x7f87c81a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:07.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.186+0000 7f87ccc00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c8118b50 con 0x7f87c8076fe0 2024-03-28T17:47:07.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.186+0000 7f87ccc00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c81189d0 con 0x7f87c8113040 2024-03-28T17:47:07.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.186+0000 7f87ccc00700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c8118cd0 con 0x7f87c80766c0 2024-03-28T17:47:07.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.186+0000 7f87c5e00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f87c80766c0 0x7f87c819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:07.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.186+0000 7f87c5e00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f87c80766c0 0x7f87c819efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:35604/0 (socket says 172.21.15.45:35604) 2024-03-28T17:47:07.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.186+0000 7f87c5e00700 1 -- 172.21.15.45:0/4208210582 learned_addr learned my addr 172.21.15.45:0/4208210582 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:07.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.186+0000 7f87c6800700 1 --2- 172.21.15.45:0/4208210582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87c8113040 0x7f87c81a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:07.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.186+0000 7f87c5400700 1 --2- 172.21.15.45:0/4208210582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87c8076fe0 0x7f87c819f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:07.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.186+0000 7f87c5e00700 1 -- 172.21.15.45:0/4208210582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87c8113040 msgr2=0x7f87c81a3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:07.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.186+0000 7f87c5e00700 1 --2- 172.21.15.45:0/4208210582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87c8113040 0x7f87c81a3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:07.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.186+0000 7f87c5e00700 1 -- 172.21.15.45:0/4208210582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87c8076fe0 msgr2=0x7f87c819f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:07.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.186+0000 7f87c5e00700 1 --2- 172.21.15.45:0/4208210582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87c8076fe0 0x7f87c819f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:07.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.186+0000 7f87c5e00700 1 -- 172.21.15.45:0/4208210582 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c81a3fb0 con 0x7f87c80766c0 2024-03-28T17:47:07.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.186+0000 7f87c6800700 1 --2- 172.21.15.45:0/4208210582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87c8113040 0x7f87c81a3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:47:07.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.187+0000 7f87c5e00700 1 --2- 172.21.15.45:0/4208210582 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f87c80766c0 0x7f87c819efa0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f87b800e9a0 tx=0x7f87b800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:07.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.187+0000 7f87bea00700 1 -- 172.21.15.45:0/4208210582 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b800cbd0 con 0x7f87c80766c0 2024-03-28T17:47:07.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.187+0000 7f87bea00700 1 -- 172.21.15.45:0/4208210582 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87b800cd70 con 0x7f87c80766c0 2024-03-28T17:47:07.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.187+0000 7f87ccc00700 1 -- 172.21.15.45:0/4208210582 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c81a42a0 con 0x7f87c80766c0 2024-03-28T17:47:07.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.187+0000 7f87bea00700 1 -- 172.21.15.45:0/4208210582 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b8018750 con 0x7f87c80766c0 2024-03-28T17:47:07.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.187+0000 7f87ccc00700 1 -- 172.21.15.45:0/4208210582 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c81abb20 con 0x7f87c80766c0 2024-03-28T17:47:07.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.188+0000 7f87ccc00700 1 -- 172.21.15.45:0/4208210582 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f87c8062920 con 0x7f87c80766c0 2024-03-28T17:47:07.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.190+0000 7f87bea00700 1 -- 172.21.15.45:0/4208210582 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f87b8010430 con 0x7f87c80766c0 2024-03-28T17:47:07.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.191+0000 7f87bea00700 1 --2- 172.21.15.45:0/4208210582 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f87ac0a29a0 0x7f87ac0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:07.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.191+0000 7f87bea00700 1 -- 172.21.15.45:0/4208210582 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f87b80ced60 con 0x7f87c80766c0 2024-03-28T17:47:07.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.192+0000 7f87c5400700 1 --2- 172.21.15.45:0/4208210582 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f87ac0a29a0 0x7f87ac0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:07.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.192+0000 7f87bea00700 1 --2- 172.21.15.45:0/4208210582 >> [v2:172.21.15.151:6824/2941055364,v1:172.21.15.151:6825/2941055364] conn(0x7f87ac0ac180 0x7f87ac0ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:07.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.192+0000 7f87bea00700 1 -- 172.21.15.45:0/4208210582 --> [v2:172.21.15.151:6824/2941055364,v1:172.21.15.151:6825/2941055364] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87b8012ec0 con 0x7f87ac0ac180 2024-03-28T17:47:07.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.192+0000 7f87bea00700 1 -- 172.21.15.45:0/4208210582 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87b80d1380 con 0x7f87c80766c0 2024-03-28T17:47:07.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.192+0000 7f87c6800700 1 --2- 172.21.15.45:0/4208210582 >> [v2:172.21.15.151:6824/2941055364,v1:172.21.15.151:6825/2941055364] conn(0x7f87ac0ac180 0x7f87ac0ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:07.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.192+0000 7f87c5400700 1 --2- 172.21.15.45:0/4208210582 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f87ac0a29a0 0x7f87ac0a4e60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f87b400b2e0 tx=0x7f87b401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:07.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.193+0000 7f87c6800700 1 --2- 172.21.15.45:0/4208210582 >> [v2:172.21.15.151:6824/2941055364,v1:172.21.15.151:6825/2941055364] conn(0x7f87ac0ac180 0x7f87ac0ae5a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:07.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.193+0000 7f87bea00700 1 -- 172.21.15.45:0/4208210582 <== osd.11 v2:172.21.15.151:6824/2941055364 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f87b8012ec0 con 0x7f87ac0ac180 2024-03-28T17:47:07.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.212+0000 7f87ccc00700 1 -- 172.21.15.45:0/4208210582 --> [v2:172.21.15.151:6824/2941055364,v1:172.21.15.151:6825/2941055364] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f87c8062d50 con 0x7f87ac0ac180 2024-03-28T17:47:07.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.213+0000 7f87bea00700 1 -- 172.21.15.45:0/4208210582 <== osd.11 v2:172.21.15.151:6824/2941055364 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f87c8062d50 con 0x7f87ac0ac180 2024-03-28T17:47:07.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.213+0000 7f87ccc00700 1 -- 172.21.15.45:0/4208210582 >> [v2:172.21.15.151:6824/2941055364,v1:172.21.15.151:6825/2941055364] conn(0x7f87ac0ac180 msgr2=0x7f87ac0ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:07.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.213+0000 7f87ccc00700 1 --2- 172.21.15.45:0/4208210582 >> [v2:172.21.15.151:6824/2941055364,v1:172.21.15.151:6825/2941055364] conn(0x7f87ac0ac180 0x7f87ac0ae5a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:07.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.214+0000 7f87ccc00700 1 -- 172.21.15.45:0/4208210582 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f87ac0a29a0 msgr2=0x7f87ac0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:07.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.214+0000 7f87ccc00700 1 --2- 172.21.15.45:0/4208210582 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f87ac0a29a0 0x7f87ac0a4e60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f87b400b2e0 tx=0x7f87b401a040 comp rx=0 tx=0).stop 2024-03-28T17:47:07.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.214+0000 7f87ccc00700 1 -- 172.21.15.45:0/4208210582 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f87c80766c0 msgr2=0x7f87c819efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:07.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.214+0000 7f87ccc00700 1 --2- 172.21.15.45:0/4208210582 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f87c80766c0 0x7f87c819efa0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f87b800e9a0 tx=0x7f87b800ecb0 comp rx=0 tx=0).stop 2024-03-28T17:47:07.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.214+0000 7f87c6800700 1 -- 172.21.15.45:0/4208210582 reap_dead start 2024-03-28T17:47:07.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.214+0000 7f87ccc00700 1 -- 172.21.15.45:0/4208210582 shutdown_connections 2024-03-28T17:47:07.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.214+0000 7f87ccc00700 1 -- 172.21.15.45:0/4208210582 >> 172.21.15.45:0/4208210582 conn(0x7f87c80ffe90 msgr2=0x7f87c8102280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:07.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.214+0000 7f87ccc00700 1 -- 172.21.15.45:0/4208210582 shutdown_connections 2024-03-28T17:47:07.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:07.214+0000 7f87ccc00700 1 -- 172.21.15.45:0/4208210582 wait complete. 2024-03-28T17:47:08.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-28T17:47:08.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-03-28T17:47:08.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-03-28T17:47:08.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-03-28T17:47:08.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-03-28T17:47:08.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-28T17:47:08.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-03-28T17:47:08.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:08 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-28T17:47:08.797 INFO:teuthology.orchestra.run.smithi045.stdout:343597383706 2024-03-28T17:47:08.798 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-28T17:47:08.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-28T17:47:08.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-03-28T17:47:08.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-03-28T17:47:08.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-03-28T17:47:08.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-03-28T17:47:08.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-28T17:47:08.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-03-28T17:47:08.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:08 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-28T17:47:08.812 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:47:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-28T17:47:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-03-28T17:47:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-03-28T17:47:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-03-28T17:47:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-03-28T17:47:08.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-28T17:47:08.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-03-28T17:47:08.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:08 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-28T17:47:09.481 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:47:09.508 INFO:teuthology.orchestra.run.smithi045.stdout:253403070505 2024-03-28T17:47:09.509 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-28T17:47:09.696 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:09 smithi045 ceph-mon[31773]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:47:09.696 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-28T17:47:09.696 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-03-28T17:47:09.696 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-03-28T17:47:09.696 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-03-28T17:47:09.696 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-03-28T17:47:09.697 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-28T17:47:09.697 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-03-28T17:47:09.697 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:09 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-28T17:47:09.697 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:09 smithi045 ceph-mon[31773]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T17:47:09.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:09 smithi071 ceph-mon[36597]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:47:09.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:09 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-28T17:47:09.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:09 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-03-28T17:47:09.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:09 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-03-28T17:47:09.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:09 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-03-28T17:47:09.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:09 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-03-28T17:47:09.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:09 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-28T17:47:09.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:09 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-03-28T17:47:09.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:09 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-28T17:47:09.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:09 smithi071 ceph-mon[36597]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T17:47:09.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:09 smithi151 ceph-mon[36481]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:47:09.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-28T17:47:09.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-03-28T17:47:09.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-03-28T17:47:09.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-03-28T17:47:09.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-03-28T17:47:09.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-28T17:47:09.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-03-28T17:47:09.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:09 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-28T17:47:09.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:09 smithi151 ceph-mon[36481]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T17:47:09.958 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:47:09.962 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:47:10.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:10 smithi045 ceph-mon[31773]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T17:47:10.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:10 smithi071 ceph-mon[36597]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T17:47:10.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:10 smithi151 ceph-mon[36481]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T17:47:10.958 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:47:10.959 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:47:11.805 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:11 smithi045 ceph-mon[31773]: pgmap v348: 97 pgs: 1 peering, 96 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-28T17:47:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:11 smithi071 ceph-mon[36597]: pgmap v348: 97 pgs: 1 peering, 96 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-28T17:47:11.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:11 smithi151 ceph-mon[36481]: pgmap v348: 97 pgs: 1 peering, 96 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-28T17:47:12.641 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:47:13.676 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:13 smithi071 ceph-mon[36597]: pgmap v349: 97 pgs: 1 peering, 96 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 14 B/s, 2 keys/s, 0 objects/s recovering 2024-03-28T17:47:13.676 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:13 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:47:13.729 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:13 smithi045 ceph-mon[31773]: pgmap v349: 97 pgs: 1 peering, 96 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 14 B/s, 2 keys/s, 0 objects/s recovering 2024-03-28T17:47:13.729 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:13 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:47:13.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:13 smithi151 ceph-mon[36481]: pgmap v349: 97 pgs: 1 peering, 96 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 14 B/s, 2 keys/s, 0 objects/s recovering 2024-03-28T17:47:13.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:13 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:47:15.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:15 smithi045 ceph-mon[31773]: pgmap v350: 97 pgs: 1 peering, 96 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 14 B/s, 2 keys/s, 0 objects/s recovering 2024-03-28T17:47:15.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:15 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:47:15.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:15 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:47:15.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:15 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:47:15.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:15 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:47:15.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:15 smithi071 ceph-mon[36597]: pgmap v350: 97 pgs: 1 peering, 96 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 14 B/s, 2 keys/s, 0 objects/s recovering 2024-03-28T17:47:15.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:15 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:47:15.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:15 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:47:15.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:15 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:47:15.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:15 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:47:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:15 smithi151 ceph-mon[36481]: pgmap v350: 97 pgs: 1 peering, 96 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 14 B/s, 2 keys/s, 0 objects/s recovering 2024-03-28T17:47:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:15 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:47:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:15 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:47:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:15 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:47:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:15 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:47:17.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.113+0000 7f8b32800700 1 -- 172.21.15.45:0/3142941290 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b2c1074b0 msgr2=0x7f8b2c10f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:17.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.113+0000 7f8b32800700 1 --2- 172.21.15.45:0/3142941290 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b2c1074b0 0x7f8b2c10f080 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8b24009fc0 tx=0x7f8b2400b330 comp rx=0 tx=0).stop 2024-03-28T17:47:17.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.113+0000 7f8b32800700 1 -- 172.21.15.45:0/3142941290 shutdown_connections 2024-03-28T17:47:17.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.113+0000 7f8b32800700 1 --2- 172.21.15.45:0/3142941290 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b2c1074b0 0x7f8b2c10f080 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:17.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.113+0000 7f8b32800700 1 --2- 172.21.15.45:0/3142941290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b2c1023d0 0x7f8b2c1027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:17.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.113+0000 7f8b32800700 1 --2- 172.21.15.45:0/3142941290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b2c102d80 0x7f8b2c106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:17.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.113+0000 7f8b32800700 1 -- 172.21.15.45:0/3142941290 >> 172.21.15.45:0/3142941290 conn(0x7f8b2c0fdc40 msgr2=0x7f8b2c100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:17.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.113+0000 7f8b32800700 1 -- 172.21.15.45:0/3142941290 shutdown_connections 2024-03-28T17:47:17.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.113+0000 7f8b32800700 1 -- 172.21.15.45:0/3142941290 wait complete. 2024-03-28T17:47:17.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.114+0000 7f8b32800700 1 Processor -- start 2024-03-28T17:47:17.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.114+0000 7f8b32800700 1 -- start start 2024-03-28T17:47:17.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.115+0000 7f8b32800700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b2c1023d0 0x7f8b2c1a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:17.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.115+0000 7f8b32800700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b2c102d80 0x7f8b2c1a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:17.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.115+0000 7f8b32800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b2c1074b0 0x7f8b2c19a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:17.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.115+0000 7f8b32800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2c1124f0 con 0x7f8b2c1074b0 2024-03-28T17:47:17.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.115+0000 7f8b32800700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2c112370 con 0x7f8b2c102d80 2024-03-28T17:47:17.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.115+0000 7f8b32800700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2c112670 con 0x7f8b2c1023d0 2024-03-28T17:47:17.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.115+0000 7f8b2b400700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b2c1023d0 0x7f8b2c1a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:17.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.115+0000 7f8b2b400700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b2c1023d0 0x7f8b2c1a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:49446/0 (socket says 172.21.15.45:49446) 2024-03-28T17:47:17.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.115+0000 7f8b2aa00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b2c102d80 0x7f8b2c1a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:17.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.115+0000 7f8b2b400700 1 -- 172.21.15.45:0/3728984091 learned_addr learned my addr 172.21.15.45:0/3728984091 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:17.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.115+0000 7f8b2be00700 1 --2- 172.21.15.45:0/3728984091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b2c1074b0 0x7f8b2c19a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:17.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.115+0000 7f8b2b400700 1 -- 172.21.15.45:0/3728984091 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b2c102d80 msgr2=0x7f8b2c1a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:17.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.115+0000 7f8b2b400700 1 --2- 172.21.15.45:0/3728984091 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b2c102d80 0x7f8b2c1a0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:17.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.115+0000 7f8b2b400700 1 -- 172.21.15.45:0/3728984091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b2c1074b0 msgr2=0x7f8b2c19a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:17.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.115+0000 7f8b2b400700 1 --2- 172.21.15.45:0/3728984091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b2c1074b0 0x7f8b2c19a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:17.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.115+0000 7f8b2b400700 1 -- 172.21.15.45:0/3728984091 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b2c19b0a0 con 0x7f8b2c1023d0 2024-03-28T17:47:17.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.115+0000 7f8b2be00700 1 --2- 172.21.15.45:0/3728984091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b2c1074b0 0x7f8b2c19a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:47:17.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.116+0000 7f8b2b400700 1 --2- 172.21.15.45:0/3728984091 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b2c1023d0 0x7f8b2c1a0780 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c00e9c0 tx=0x7f8b1c00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:17.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.116+0000 7f8b23e00700 1 -- 172.21.15.45:0/3728984091 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c00cbd0 con 0x7f8b2c1023d0 2024-03-28T17:47:17.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.116+0000 7f8b23e00700 1 -- 172.21.15.45:0/3728984091 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b1c00cd70 con 0x7f8b2c1023d0 2024-03-28T17:47:17.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.116+0000 7f8b32800700 1 -- 172.21.15.45:0/3728984091 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b2c19b390 con 0x7f8b2c1023d0 2024-03-28T17:47:17.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.116+0000 7f8b23e00700 1 -- 172.21.15.45:0/3728984091 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c005660 con 0x7f8b2c1023d0 2024-03-28T17:47:17.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.116+0000 7f8b32800700 1 -- 172.21.15.45:0/3728984091 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b2c1a7650 con 0x7f8b2c1023d0 2024-03-28T17:47:17.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.117+0000 7f8b32800700 1 -- 172.21.15.45:0/3728984091 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b2c068e50 con 0x7f8b2c1023d0 2024-03-28T17:47:17.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.119+0000 7f8b23e00700 1 -- 172.21.15.45:0/3728984091 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f8b1c004750 con 0x7f8b2c1023d0 2024-03-28T17:47:17.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.120+0000 7f8b23e00700 1 --2- 172.21.15.45:0/3728984091 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8b100a2920 0x7f8b100a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:17.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.121+0000 7f8b23e00700 1 -- 172.21.15.45:0/3728984091 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f8b1c0cedd0 con 0x7f8b2c1023d0 2024-03-28T17:47:17.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.121+0000 7f8b2aa00700 1 --2- 172.21.15.45:0/3728984091 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8b100a2920 0x7f8b100a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:17.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.121+0000 7f8b2aa00700 1 --2- 172.21.15.45:0/3728984091 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8b100a2920 0x7f8b100a4de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c101730 tx=0x7f8b18009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:17.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.124+0000 7f8b23e00700 1 -- 172.21.15.45:0/3728984091 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8b1c09bcc0 con 0x7f8b2c1023d0 2024-03-28T17:47:17.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.360+0000 7f8b32800700 1 -- 172.21.15.45:0/3728984091 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f8b2c1a78e0 con 0x7f8b2c1023d0 2024-03-28T17:47:17.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.361+0000 7f8b23e00700 1 -- 172.21.15.45:0/3728984091 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f8b1c09b5c0 con 0x7f8b2c1023d0 2024-03-28T17:47:17.422 INFO:teuthology.orchestra.run.smithi045.stdout:287762808871 2024-03-28T17:47:17.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.364+0000 7f8b32800700 1 -- 172.21.15.45:0/3728984091 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8b100a2920 msgr2=0x7f8b100a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:17.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.364+0000 7f8b32800700 1 --2- 172.21.15.45:0/3728984091 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8b100a2920 0x7f8b100a4de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c101730 tx=0x7f8b18009260 comp rx=0 tx=0).stop 2024-03-28T17:47:17.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.364+0000 7f8b32800700 1 -- 172.21.15.45:0/3728984091 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b2c1023d0 msgr2=0x7f8b2c1a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:17.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.364+0000 7f8b32800700 1 --2- 172.21.15.45:0/3728984091 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b2c1023d0 0x7f8b2c1a0780 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c00e9c0 tx=0x7f8b1c00ecd0 comp rx=0 tx=0).stop 2024-03-28T17:47:17.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.364+0000 7f8b32800700 1 -- 172.21.15.45:0/3728984091 shutdown_connections 2024-03-28T17:47:17.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.364+0000 7f8b32800700 1 --2- 172.21.15.45:0/3728984091 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8b100a2920 0x7f8b100a4de0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:17.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.364+0000 7f8b32800700 1 --2- 172.21.15.45:0/3728984091 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b2c1023d0 0x7f8b2c1a0780 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:17.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.364+0000 7f8b32800700 1 --2- 172.21.15.45:0/3728984091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b2c1074b0 0x7f8b2c19a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:17.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.364+0000 7f8b32800700 1 --2- 172.21.15.45:0/3728984091 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b2c102d80 0x7f8b2c1a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:17.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.364+0000 7f8b32800700 1 -- 172.21.15.45:0/3728984091 >> 172.21.15.45:0/3728984091 conn(0x7f8b2c0fdc40 msgr2=0x7f8b2c10d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:17.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.364+0000 7f8b32800700 1 -- 172.21.15.45:0/3728984091 shutdown_connections 2024-03-28T17:47:17.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.364+0000 7f8b32800700 1 -- 172.21.15.45:0/3728984091 wait complete. 2024-03-28T17:47:17.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:17 smithi071 ceph-mon[36597]: pgmap v351: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 14 B/s, 2 keys/s, 0 objects/s recovering 2024-03-28T17:47:17.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:17 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/2690594924' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T17:47:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:17 smithi151 ceph-mon[36481]: pgmap v351: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 14 B/s, 2 keys/s, 0 objects/s recovering 2024-03-28T17:47:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:17 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/2690594924' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T17:47:17.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:17 smithi045 ceph-mon[31773]: pgmap v351: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 14 B/s, 2 keys/s, 0 objects/s recovering 2024-03-28T17:47:17.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:17 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/2690594924' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T17:47:18.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.011+0000 7f29ba400700 1 -- 172.21.15.45:0/3744199182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29b40743c0 msgr2=0x7f29b40747a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:18.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.011+0000 7f29ba400700 1 --2- 172.21.15.45:0/3744199182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29b40743c0 0x7f29b40747a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f29a4009990 tx=0x7f29a4009ca0 comp rx=0 tx=0).stop 2024-03-28T17:47:18.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.011+0000 7f29ba400700 1 -- 172.21.15.45:0/3744199182 shutdown_connections 2024-03-28T17:47:18.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.011+0000 7f29ba400700 1 --2- 172.21.15.45:0/3744199182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b4113360 0x7f29b4115750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:18.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.011+0000 7f29ba400700 1 --2- 172.21.15.45:0/3744199182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29b40743c0 0x7f29b40747a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:18.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.011+0000 7f29ba400700 1 --2- 172.21.15.45:0/3744199182 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29b4074ce0 0x7f29b4112e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:18.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.011+0000 7f29ba400700 1 -- 172.21.15.45:0/3744199182 >> 172.21.15.45:0/3744199182 conn(0x7f29b40792f0 msgr2=0x7f29b4079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:18.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.011+0000 7f29ba400700 1 -- 172.21.15.45:0/3744199182 shutdown_connections 2024-03-28T17:47:18.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.011+0000 7f29ba400700 1 -- 172.21.15.45:0/3744199182 wait complete. 2024-03-28T17:47:18.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.012+0000 7f29ba400700 1 Processor -- start 2024-03-28T17:47:18.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.012+0000 7f29ba400700 1 -- start start 2024-03-28T17:47:18.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.012+0000 7f29ba400700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29b40743c0 0x7f29b419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:18.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.012+0000 7f29ba400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29b4074ce0 0x7f29b419f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:18.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.012+0000 7f29ba400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b4113360 0x7f29b41a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:18.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.012+0000 7f29ba400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b4118bb0 con 0x7f29b4113360 2024-03-28T17:47:18.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.012+0000 7f29ba400700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b4118a30 con 0x7f29b4074ce0 2024-03-28T17:47:18.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.012+0000 7f29ba400700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b4118d30 con 0x7f29b40743c0 2024-03-28T17:47:18.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.013+0000 7f29b3400700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29b40743c0 0x7f29b419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:18.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.013+0000 7f29b2a00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29b4074ce0 0x7f29b419f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:18.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.013+0000 7f29b3400700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29b40743c0 0x7f29b419f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:49420/0 (socket says 172.21.15.45:49420) 2024-03-28T17:47:18.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.013+0000 7f29b3400700 1 -- 172.21.15.45:0/2690594924 learned_addr learned my addr 172.21.15.45:0/2690594924 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:18.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.013+0000 7f29b2a00700 1 --2- 172.21.15.45:0/2690594924 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29b4074ce0 0x7f29b419f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:38274/0 (socket says 172.21.15.45:38274) 2024-03-28T17:47:18.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.013+0000 7f29b3e00700 1 --2- 172.21.15.45:0/2690594924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b4113360 0x7f29b41a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:18.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.013+0000 7f29b2a00700 1 -- 172.21.15.45:0/2690594924 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29b40743c0 msgr2=0x7f29b419f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:18.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.013+0000 7f29b2a00700 1 --2- 172.21.15.45:0/2690594924 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29b40743c0 0x7f29b419f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.013+0000 7f29b2a00700 1 -- 172.21.15.45:0/2690594924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b4113360 msgr2=0x7f29b41a3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.013+0000 7f29b2a00700 1 --2- 172.21.15.45:0/2690594924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b4113360 0x7f29b41a3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.013+0000 7f29b2a00700 1 -- 172.21.15.45:0/2690594924 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b41a4050 con 0x7f29b4074ce0 2024-03-28T17:47:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.013+0000 7f29b3400700 1 --2- 172.21.15.45:0/2690594924 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29b40743c0 0x7f29b419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:47:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.014+0000 7f29b2a00700 1 --2- 172.21.15.45:0/2690594924 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29b4074ce0 0x7f29b419f580 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f29a000e9c0 tx=0x7f29a000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.014+0000 7f29abe00700 1 -- 172.21.15.45:0/2690594924 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a0004230 con 0x7f29b4074ce0 2024-03-28T17:47:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.014+0000 7f29ba400700 1 -- 172.21.15.45:0/2690594924 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b41a4340 con 0x7f29b4074ce0 2024-03-28T17:47:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.014+0000 7f29abe00700 1 -- 172.21.15.45:0/2690594924 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29a00043d0 con 0x7f29b4074ce0 2024-03-28T17:47:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.014+0000 7f29abe00700 1 -- 172.21.15.45:0/2690594924 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a0003e60 con 0x7f29b4074ce0 2024-03-28T17:47:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.014+0000 7f29ba400700 1 -- 172.21.15.45:0/2690594924 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b41abbc0 con 0x7f29b4074ce0 2024-03-28T17:47:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.015+0000 7f29ba400700 1 -- 172.21.15.45:0/2690594924 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b4068e50 con 0x7f29b4074ce0 2024-03-28T17:47:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.017+0000 7f29abe00700 1 -- 172.21.15.45:0/2690594924 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f29a000c7c0 con 0x7f29b4074ce0 2024-03-28T17:47:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.018+0000 7f29abe00700 1 --2- 172.21.15.45:0/2690594924 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f29980a2920 0x7f29980a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:18.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.018+0000 7f29abe00700 1 -- 172.21.15.45:0/2690594924 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f29a00cef20 con 0x7f29b4074ce0 2024-03-28T17:47:18.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.018+0000 7f29b3400700 1 --2- 172.21.15.45:0/2690594924 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f29980a2920 0x7f29980a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:18.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.019+0000 7f29b3400700 1 --2- 172.21.15.45:0/2690594924 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f29980a2920 0x7f29980a4de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f29a4006010 tx=0x7f29a401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:18.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.022+0000 7f29abe00700 1 -- 172.21.15.45:0/2690594924 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f29a009be10 con 0x7f29b4074ce0 2024-03-28T17:47:18.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.259+0000 7f29ba400700 1 -- 172.21.15.45:0/2690594924 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f29b41abd90 con 0x7f29b4074ce0 2024-03-28T17:47:18.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.260+0000 7f29abe00700 1 -- 172.21.15.45:0/2690594924 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f29a009b710 con 0x7f29b4074ce0 2024-03-28T17:47:18.084 INFO:teuthology.orchestra.run.smithi045.stdout:55834574931 2024-03-28T17:47:18.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.263+0000 7f29ba400700 1 -- 172.21.15.45:0/2690594924 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f29980a2920 msgr2=0x7f29980a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:18.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.263+0000 7f29ba400700 1 --2- 172.21.15.45:0/2690594924 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f29980a2920 0x7f29980a4de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f29a4006010 tx=0x7f29a401a040 comp rx=0 tx=0).stop 2024-03-28T17:47:18.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.263+0000 7f29ba400700 1 -- 172.21.15.45:0/2690594924 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29b4074ce0 msgr2=0x7f29b419f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:18.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.263+0000 7f29ba400700 1 --2- 172.21.15.45:0/2690594924 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29b4074ce0 0x7f29b419f580 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f29a000e9c0 tx=0x7f29a000ecd0 comp rx=0 tx=0).stop 2024-03-28T17:47:18.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.263+0000 7f29ba400700 1 -- 172.21.15.45:0/2690594924 shutdown_connections 2024-03-28T17:47:18.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.263+0000 7f29ba400700 1 --2- 172.21.15.45:0/2690594924 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f29980a2920 0x7f29980a4de0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:18.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.263+0000 7f29ba400700 1 --2- 172.21.15.45:0/2690594924 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29b40743c0 0x7f29b419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:18.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.263+0000 7f29ba400700 1 --2- 172.21.15.45:0/2690594924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b4113360 0x7f29b41a3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:18.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.263+0000 7f29ba400700 1 --2- 172.21.15.45:0/2690594924 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29b4074ce0 0x7f29b419f580 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:18.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.263+0000 7f29ba400700 1 -- 172.21.15.45:0/2690594924 >> 172.21.15.45:0/2690594924 conn(0x7f29b40792f0 msgr2=0x7f29b4112190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:18.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.264+0000 7f29ba400700 1 -- 172.21.15.45:0/2690594924 shutdown_connections 2024-03-28T17:47:18.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:17.264+0000 7f29ba400700 1 -- 172.21.15.45:0/2690594924 wait complete. 2024-03-28T17:47:18.397 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:18 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3728984091' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T17:47:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:18 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3728984091' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T17:47:18.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:18 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3728984091' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T17:47:19.471 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:47:19.472 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:47:19.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:19 smithi151 ceph-mon[36481]: pgmap v352: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 12 B/s, 1 keys/s, 0 objects/s recovering 2024-03-28T17:47:19.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:19 smithi045 ceph-mon[31773]: pgmap v352: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 12 B/s, 1 keys/s, 0 objects/s recovering 2024-03-28T17:47:19.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:19 smithi071 ceph-mon[36597]: pgmap v352: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 12 B/s, 1 keys/s, 0 objects/s recovering 2024-03-28T17:47:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.281+0000 7fb676a00700 1 -- 172.21.15.45:0/564286419 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb6701081c0 msgr2=0x7fb6701085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.281+0000 7fb676a00700 1 --2- 172.21.15.45:0/564286419 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb6701081c0 0x7fb6701085a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb660009990 tx=0x7fb660009ca0 comp rx=0 tx=0).stop 2024-03-28T17:47:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.281+0000 7fb676a00700 1 -- 172.21.15.45:0/564286419 shutdown_connections 2024-03-28T17:47:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.281+0000 7fb676a00700 1 --2- 172.21.15.45:0/564286419 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb6701081c0 0x7fb6701085a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:20.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.281+0000 7fb676a00700 1 --2- 172.21.15.45:0/564286419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb670102ba0 0x7fb67010efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:20.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.281+0000 7fb676a00700 1 --2- 172.21.15.45:0/564286419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6701021e0 0x7fb670102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:20.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.281+0000 7fb676a00700 1 -- 172.21.15.45:0/564286419 >> 172.21.15.45:0/564286419 conn(0x7fb6700792c0 msgr2=0x7fb6700796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:20.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.281+0000 7fb676a00700 1 -- 172.21.15.45:0/564286419 shutdown_connections 2024-03-28T17:47:20.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.281+0000 7fb676a00700 1 -- 172.21.15.45:0/564286419 wait complete. 2024-03-28T17:47:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.282+0000 7fb676a00700 1 Processor -- start 2024-03-28T17:47:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.282+0000 7fb676a00700 1 -- start start 2024-03-28T17:47:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.283+0000 7fb676a00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb6701021e0 0x7fb67019e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.283+0000 7fb676a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb670102ba0 0x7fb67019eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.283+0000 7fb676a00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6701081c0 0x7fb670198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.283+0000 7fb676a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6701124c0 con 0x7fb670102ba0 2024-03-28T17:47:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.283+0000 7fb676a00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb670112340 con 0x7fb6701081c0 2024-03-28T17:47:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.283+0000 7fb676a00700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb670112640 con 0x7fb6701021e0 2024-03-28T17:47:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.283+0000 7fb66fe00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb6701021e0 0x7fb67019e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:20.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.283+0000 7fb66fe00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb6701021e0 0x7fb67019e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:49470/0 (socket says 172.21.15.45:49470) 2024-03-28T17:47:20.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.283+0000 7fb66fe00700 1 -- 172.21.15.45:0/1274531908 learned_addr learned my addr 172.21.15.45:0/1274531908 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:20.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.283+0000 7fb674a00700 1 --2- 172.21.15.45:0/1274531908 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6701081c0 0x7fb670198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:20.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.283+0000 7fb66f400700 1 --2- 172.21.15.45:0/1274531908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb670102ba0 0x7fb67019eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:20.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.283+0000 7fb66fe00700 1 -- 172.21.15.45:0/1274531908 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6701081c0 msgr2=0x7fb670198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:20.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.283+0000 7fb66fe00700 1 --2- 172.21.15.45:0/1274531908 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6701081c0 0x7fb670198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:20.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.283+0000 7fb66fe00700 1 -- 172.21.15.45:0/1274531908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb670102ba0 msgr2=0x7fb67019eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:20.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.283+0000 7fb66fe00700 1 --2- 172.21.15.45:0/1274531908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb670102ba0 0x7fb67019eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:20.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.283+0000 7fb66fe00700 1 -- 172.21.15.45:0/1274531908 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb670199020 con 0x7fb6701021e0 2024-03-28T17:47:20.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.284+0000 7fb66fe00700 1 --2- 172.21.15.45:0/1274531908 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb6701021e0 0x7fb67019e620 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb660000c00 tx=0x7fb660005590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:20.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.284+0000 7fb66cc00700 1 -- 172.21.15.45:0/1274531908 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb660027070 con 0x7fb6701021e0 2024-03-28T17:47:20.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.284+0000 7fb66cc00700 1 -- 172.21.15.45:0/1274531908 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb6600056b0 con 0x7fb6701021e0 2024-03-28T17:47:20.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.284+0000 7fb676a00700 1 -- 172.21.15.45:0/1274531908 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6701992b0 con 0x7fb6701021e0 2024-03-28T17:47:20.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.284+0000 7fb66cc00700 1 -- 172.21.15.45:0/1274531908 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb660017650 con 0x7fb6701021e0 2024-03-28T17:47:20.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.284+0000 7fb676a00700 1 -- 172.21.15.45:0/1274531908 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6701a55a0 con 0x7fb6701021e0 2024-03-28T17:47:20.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.285+0000 7fb676a00700 1 -- 172.21.15.45:0/1274531908 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb670199610 con 0x7fb6701021e0 2024-03-28T17:47:20.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.287+0000 7fb66cc00700 1 -- 172.21.15.45:0/1274531908 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fb66002a070 con 0x7fb6701021e0 2024-03-28T17:47:20.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.288+0000 7fb66cc00700 1 --2- 172.21.15.45:0/1274531908 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb6540a29a0 0x7fb6540a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.288+0000 7fb66cc00700 1 -- 172.21.15.45:0/1274531908 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb6600cfba0 con 0x7fb6701021e0 2024-03-28T17:47:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.289+0000 7fb66f400700 1 --2- 172.21.15.45:0/1274531908 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb6540a29a0 0x7fb6540a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.289+0000 7fb66f400700 1 --2- 172.21.15.45:0/1274531908 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb6540a29a0 0x7fb6540a4e60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb670199eb0 tx=0x7fb65c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:20.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.296+0000 7fb66cc00700 1 -- 172.21.15.45:0/1274531908 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb66005e740 con 0x7fb6701021e0 2024-03-28T17:47:20.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.541+0000 7fb676a00700 1 -- 172.21.15.45:0/1274531908 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fb670199c20 con 0x7fb6701021e0 2024-03-28T17:47:20.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.542+0000 7fb66cc00700 1 -- 172.21.15.45:0/1274531908 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb660013070 con 0x7fb6701021e0 2024-03-28T17:47:20.542 INFO:teuthology.orchestra.run.smithi045.stdout:197568495672 2024-03-28T17:47:20.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.544+0000 7fb676a00700 1 -- 172.21.15.45:0/1274531908 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb6540a29a0 msgr2=0x7fb6540a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:20.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.544+0000 7fb676a00700 1 --2- 172.21.15.45:0/1274531908 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb6540a29a0 0x7fb6540a4e60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb670199eb0 tx=0x7fb65c009260 comp rx=0 tx=0).stop 2024-03-28T17:47:20.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.544+0000 7fb676a00700 1 -- 172.21.15.45:0/1274531908 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb6701021e0 msgr2=0x7fb67019e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:20.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.544+0000 7fb676a00700 1 --2- 172.21.15.45:0/1274531908 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb6701021e0 0x7fb67019e620 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb660000c00 tx=0x7fb660005590 comp rx=0 tx=0).stop 2024-03-28T17:47:20.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.545+0000 7fb676a00700 1 -- 172.21.15.45:0/1274531908 shutdown_connections 2024-03-28T17:47:20.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.545+0000 7fb676a00700 1 --2- 172.21.15.45:0/1274531908 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb6540a29a0 0x7fb6540a4e60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:20.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.545+0000 7fb676a00700 1 --2- 172.21.15.45:0/1274531908 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb6701021e0 0x7fb67019e620 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:20.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.545+0000 7fb676a00700 1 --2- 172.21.15.45:0/1274531908 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6701081c0 0x7fb670198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:20.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.545+0000 7fb676a00700 1 --2- 172.21.15.45:0/1274531908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb670102ba0 0x7fb67019eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:20.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.545+0000 7fb676a00700 1 -- 172.21.15.45:0/1274531908 >> 172.21.15.45:0/1274531908 conn(0x7fb6700792c0 msgr2=0x7fb670100dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:20.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.545+0000 7fb676a00700 1 -- 172.21.15.45:0/1274531908 shutdown_connections 2024-03-28T17:47:20.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:20.545+0000 7fb676a00700 1 -- 172.21.15.45:0/1274531908 wait complete. 2024-03-28T17:47:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:21 smithi151 ceph-mon[36481]: pgmap v353: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 11 B/s, 1 keys/s, 0 objects/s recovering 2024-03-28T17:47:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:21 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1274531908' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T17:47:21.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:21 smithi045 ceph-mon[31773]: pgmap v353: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 11 B/s, 1 keys/s, 0 objects/s recovering 2024-03-28T17:47:21.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:21 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1274531908' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T17:47:21.796 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:47:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:21 smithi071 ceph-mon[36597]: pgmap v353: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 11 B/s, 1 keys/s, 0 objects/s recovering 2024-03-28T17:47:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:21 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1274531908' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T17:47:21.849 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808867 got 287762808871 for osd.9 2024-03-28T17:47:21.849 DEBUG:teuthology.parallel:result is None 2024-03-28T17:47:23.343 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495666 got 197568495672 for osd.6 2024-03-28T17:47:23.343 DEBUG:teuthology.parallel:result is None 2024-03-28T17:47:23.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:23 smithi151 ceph-mon[36481]: pgmap v354: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 9 B/s, 1 keys/s, 0 objects/s recovering 2024-03-28T17:47:23.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:23 smithi045 ceph-mon[31773]: pgmap v354: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 9 B/s, 1 keys/s, 0 objects/s recovering 2024-03-28T17:47:23.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:23 smithi071 ceph-mon[36597]: pgmap v354: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 9 B/s, 1 keys/s, 0 objects/s recovering 2024-03-28T17:47:24.874 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574927 got 55834574931 for osd.1 2024-03-28T17:47:24.874 DEBUG:teuthology.parallel:result is None 2024-03-28T17:47:25.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.434+0000 7ff216400700 1 -- 172.21.15.45:0/3034122605 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff21006a7e0 msgr2=0x7ff21010f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:25.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.434+0000 7ff216400700 1 --2- 172.21.15.45:0/3034122605 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff21006a7e0 0x7ff21010f020 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff20800b0d0 tx=0x7ff20800b3e0 comp rx=0 tx=0).stop 2024-03-28T17:47:25.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.434+0000 7ff216400700 1 -- 172.21.15.45:0/3034122605 shutdown_connections 2024-03-28T17:47:25.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.434+0000 7ff216400700 1 --2- 172.21.15.45:0/3034122605 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff21006a7e0 0x7ff21010f020 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:25.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.434+0000 7ff216400700 1 --2- 172.21.15.45:0/3034122605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2101081e0 0x7ff2101085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:25.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.434+0000 7ff216400700 1 --2- 172.21.15.45:0/3034122605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff210069e40 0x7ff21006a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:25.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.434+0000 7ff216400700 1 -- 172.21.15.45:0/3034122605 >> 172.21.15.45:0/3034122605 conn(0x7ff210077fc0 msgr2=0x7ff2100783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:25.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.434+0000 7ff216400700 1 -- 172.21.15.45:0/3034122605 shutdown_connections 2024-03-28T17:47:25.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.434+0000 7ff216400700 1 -- 172.21.15.45:0/3034122605 wait complete. 2024-03-28T17:47:25.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.436+0000 7ff216400700 1 Processor -- start 2024-03-28T17:47:25.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.436+0000 7ff216400700 1 -- start start 2024-03-28T17:47:25.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.436+0000 7ff216400700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff210069e40 0x7ff21019e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:25.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.437+0000 7ff216400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff21006a7e0 0x7ff21019eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:25.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.437+0000 7ff216400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2101081e0 0x7ff2101987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:25.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.437+0000 7ff216400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff210112480 con 0x7ff2101081e0 2024-03-28T17:47:25.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.437+0000 7ff216400700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff210112300 con 0x7ff21006a7e0 2024-03-28T17:47:25.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.437+0000 7ff216400700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff210112600 con 0x7ff210069e40 2024-03-28T17:47:25.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.437+0000 7ff20f400700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff210069e40 0x7ff21019e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:25.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.437+0000 7ff20f400700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff210069e40 0x7ff21019e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:49538/0 (socket says 172.21.15.45:49538) 2024-03-28T17:47:25.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.437+0000 7ff20f400700 1 -- 172.21.15.45:0/2713222266 learned_addr learned my addr 172.21.15.45:0/2713222266 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:25.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.437+0000 7ff207e00700 1 --2- 172.21.15.45:0/2713222266 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff21006a7e0 0x7ff21019eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:25.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.437+0000 7ff20fe00700 1 --2- 172.21.15.45:0/2713222266 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2101081e0 0x7ff2101987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:25.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.437+0000 7ff20f400700 1 -- 172.21.15.45:0/2713222266 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff21006a7e0 msgr2=0x7ff21019eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:25.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.437+0000 7ff20f400700 1 --2- 172.21.15.45:0/2713222266 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff21006a7e0 0x7ff21019eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:25.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.437+0000 7ff20f400700 1 -- 172.21.15.45:0/2713222266 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2101081e0 msgr2=0x7ff2101987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:25.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.437+0000 7ff20f400700 1 --2- 172.21.15.45:0/2713222266 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2101081e0 0x7ff2101987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:25.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.437+0000 7ff20f400700 1 -- 172.21.15.45:0/2713222266 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff210198fc0 con 0x7ff210069e40 2024-03-28T17:47:25.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.437+0000 7ff20fe00700 1 --2- 172.21.15.45:0/2713222266 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2101081e0 0x7ff2101987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T17:47:25.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.438+0000 7ff20f400700 1 --2- 172.21.15.45:0/2713222266 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff210069e40 0x7ff21019e610 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff20000e9a0 tx=0x7ff20000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:25.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.438+0000 7ff20cc00700 1 -- 172.21.15.45:0/2713222266 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20000cbd0 con 0x7ff210069e40 2024-03-28T17:47:25.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.438+0000 7ff20cc00700 1 -- 172.21.15.45:0/2713222266 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff20000cd70 con 0x7ff210069e40 2024-03-28T17:47:25.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.438+0000 7ff20cc00700 1 -- 172.21.15.45:0/2713222266 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff200018750 con 0x7ff210069e40 2024-03-28T17:47:25.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.438+0000 7ff216400700 1 -- 172.21.15.45:0/2713222266 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2101992b0 con 0x7ff210069e40 2024-03-28T17:47:25.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.438+0000 7ff216400700 1 -- 172.21.15.45:0/2713222266 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2101a5470 con 0x7ff210069e40 2024-03-28T17:47:25.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.440+0000 7ff216400700 1 -- 172.21.15.45:0/2713222266 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff210103890 con 0x7ff210069e40 2024-03-28T17:47:25.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.441+0000 7ff20cc00700 1 -- 172.21.15.45:0/2713222266 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7ff200010430 con 0x7ff210069e40 2024-03-28T17:47:25.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.442+0000 7ff20cc00700 1 --2- 172.21.15.45:0/2713222266 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff1f40a29a0 0x7ff1f40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:25.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.443+0000 7ff20cc00700 1 -- 172.21.15.45:0/2713222266 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff2000cee10 con 0x7ff210069e40 2024-03-28T17:47:25.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.443+0000 7ff207e00700 1 --2- 172.21.15.45:0/2713222266 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff1f40a29a0 0x7ff1f40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:25.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.443+0000 7ff207e00700 1 --2- 172.21.15.45:0/2713222266 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff1f40a29a0 0x7ff1f40a4e60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff2100785a0 tx=0x7ff1f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:25.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.446+0000 7ff20cc00700 1 -- 172.21.15.45:0/2713222266 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff2000d2030 con 0x7ff210069e40 2024-03-28T17:47:25.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:25 smithi045 ceph-mon[31773]: pgmap v355: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:47:25.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:25 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1013073594' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T17:47:25.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.693+0000 7ff216400700 1 -- 172.21.15.45:0/2713222266 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7ff210199cc0 con 0x7ff210069e40 2024-03-28T17:47:25.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.694+0000 7ff20cc00700 1 -- 172.21.15.45:0/2713222266 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff20009bd00 con 0x7ff210069e40 2024-03-28T17:47:25.694 INFO:teuthology.orchestra.run.smithi045.stdout:163208757310 2024-03-28T17:47:25.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.696+0000 7ff216400700 1 -- 172.21.15.45:0/2713222266 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff1f40a29a0 msgr2=0x7ff1f40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:25.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.696+0000 7ff216400700 1 --2- 172.21.15.45:0/2713222266 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff1f40a29a0 0x7ff1f40a4e60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff2100785a0 tx=0x7ff1f8009260 comp rx=0 tx=0).stop 2024-03-28T17:47:25.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.696+0000 7ff216400700 1 -- 172.21.15.45:0/2713222266 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff210069e40 msgr2=0x7ff21019e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:25.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.696+0000 7ff216400700 1 --2- 172.21.15.45:0/2713222266 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff210069e40 0x7ff21019e610 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff20000e9a0 tx=0x7ff20000ecb0 comp rx=0 tx=0).stop 2024-03-28T17:47:25.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.697+0000 7ff216400700 1 -- 172.21.15.45:0/2713222266 shutdown_connections 2024-03-28T17:47:25.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.697+0000 7ff216400700 1 --2- 172.21.15.45:0/2713222266 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff1f40a29a0 0x7ff1f40a4e60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:25.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.697+0000 7ff216400700 1 --2- 172.21.15.45:0/2713222266 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff210069e40 0x7ff21019e610 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:25.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.697+0000 7ff216400700 1 --2- 172.21.15.45:0/2713222266 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2101081e0 0x7ff2101987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:25.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.697+0000 7ff216400700 1 --2- 172.21.15.45:0/2713222266 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff21006a7e0 0x7ff21019eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:25.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.697+0000 7ff216400700 1 -- 172.21.15.45:0/2713222266 >> 172.21.15.45:0/2713222266 conn(0x7ff210077fc0 msgr2=0x7ff2100ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:25.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.697+0000 7ff216400700 1 -- 172.21.15.45:0/2713222266 shutdown_connections 2024-03-28T17:47:25.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.697+0000 7ff216400700 1 -- 172.21.15.45:0/2713222266 wait complete. 2024-03-28T17:47:25.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.319+0000 7f6938c00700 1 -- 172.21.15.45:0/2416200238 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6934069e40 msgr2=0x7f693406a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:25.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.319+0000 7f6938c00700 1 --2- 172.21.15.45:0/2416200238 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6934069e40 0x7f693406a2a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f691c0098f0 tx=0x7f691c009c00 comp rx=0 tx=0).stop 2024-03-28T17:47:25.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.319+0000 7f6938c00700 1 -- 172.21.15.45:0/2416200238 shutdown_connections 2024-03-28T17:47:25.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.319+0000 7f6938c00700 1 --2- 172.21.15.45:0/2416200238 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f693406a7e0 0x7f693410f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:25.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.319+0000 7f6938c00700 1 --2- 172.21.15.45:0/2416200238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69341081e0 0x7f69341085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:25.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.319+0000 7f6938c00700 1 --2- 172.21.15.45:0/2416200238 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6934069e40 0x7f693406a2a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:25.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.319+0000 7f6938c00700 1 -- 172.21.15.45:0/2416200238 >> 172.21.15.45:0/2416200238 conn(0x7f6934077fc0 msgr2=0x7f69340783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:25.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.319+0000 7f6938c00700 1 -- 172.21.15.45:0/2416200238 shutdown_connections 2024-03-28T17:47:25.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.319+0000 7f6938c00700 1 -- 172.21.15.45:0/2416200238 wait complete. 2024-03-28T17:47:25.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.320+0000 7f6938c00700 1 Processor -- start 2024-03-28T17:47:25.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.320+0000 7f6938c00700 1 -- start start 2024-03-28T17:47:25.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.321+0000 7f6938c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6934069e40 0x7f693419e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:25.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.321+0000 7f6938c00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f693406a7e0 0x7f693419eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:25.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.321+0000 7f6938c00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69341081e0 0x7f69341987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:25.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.321+0000 7f6938c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6934112490 con 0x7f6934069e40 2024-03-28T17:47:25.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.321+0000 7f6938c00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6934112310 con 0x7f693406a7e0 2024-03-28T17:47:25.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.321+0000 7f6938c00700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6934112610 con 0x7f69341081e0 2024-03-28T17:47:25.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.321+0000 7f6931e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6934069e40 0x7f693419e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:25.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.321+0000 7f6929400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f693406a7e0 0x7f693419eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:25.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.321+0000 7f6931e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6934069e40 0x7f693419e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47908/0 (socket says 172.21.15.45:47908) 2024-03-28T17:47:25.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.321+0000 7f6929400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f693406a7e0 0x7f693419eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:38344/0 (socket says 172.21.15.45:38344) 2024-03-28T17:47:25.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.321+0000 7f6931e00700 1 -- 172.21.15.45:0/1740525460 learned_addr learned my addr 172.21.15.45:0/1740525460 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:25.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.321+0000 7f6932800700 1 --2- 172.21.15.45:0/1740525460 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69341081e0 0x7f69341987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:25.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.321+0000 7f6929400700 1 -- 172.21.15.45:0/1740525460 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69341081e0 msgr2=0x7f69341987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:25.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.321+0000 7f6929400700 1 --2- 172.21.15.45:0/1740525460 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69341081e0 0x7f69341987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:25.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.321+0000 7f6929400700 1 -- 172.21.15.45:0/1740525460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6934069e40 msgr2=0x7f693419e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:25.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.321+0000 7f6929400700 1 --2- 172.21.15.45:0/1740525460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6934069e40 0x7f693419e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:25.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.321+0000 7f6929400700 1 -- 172.21.15.45:0/1740525460 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6934198f90 con 0x7f693406a7e0 2024-03-28T17:47:25.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.322+0000 7f6929400700 1 --2- 172.21.15.45:0/1740525460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f693406a7e0 0x7f693419eb70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f691c000c00 tx=0x7f691c005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:25.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.322+0000 7f692b400700 1 -- 172.21.15.45:0/1740525460 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c025070 con 0x7f693406a7e0 2024-03-28T17:47:25.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.322+0000 7f692b400700 1 -- 172.21.15.45:0/1740525460 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f691c0054f0 con 0x7f693406a7e0 2024-03-28T17:47:25.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.322+0000 7f692b400700 1 -- 172.21.15.45:0/1740525460 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c004d50 con 0x7f693406a7e0 2024-03-28T17:47:25.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.322+0000 7f6938c00700 1 -- 172.21.15.45:0/1740525460 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6934199220 con 0x7f693406a7e0 2024-03-28T17:47:25.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.322+0000 7f6938c00700 1 -- 172.21.15.45:0/1740525460 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69341a5490 con 0x7f693406a7e0 2024-03-28T17:47:25.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.324+0000 7f6938c00700 1 -- 172.21.15.45:0/1740525460 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6934103890 con 0x7f693406a7e0 2024-03-28T17:47:25.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.327+0000 7f692b400700 1 -- 172.21.15.45:0/1740525460 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f691c028070 con 0x7f693406a7e0 2024-03-28T17:47:25.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.328+0000 7f692b400700 1 --2- 172.21.15.45:0/1740525460 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f69180a2a70 0x7f69180a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:25.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.329+0000 7f692b400700 1 -- 172.21.15.45:0/1740525460 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f691c0cf1d0 con 0x7f693406a7e0 2024-03-28T17:47:25.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.329+0000 7f6931e00700 1 --2- 172.21.15.45:0/1740525460 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f69180a2a70 0x7f69180a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:25.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.329+0000 7f6931e00700 1 --2- 172.21.15.45:0/1740525460 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f69180a2a70 0x7f69180a4f30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6924007c10 tx=0x7f6924007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:25.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.337+0000 7f692b400700 1 -- 172.21.15.45:0/1740525460 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f691c05c740 con 0x7f693406a7e0 2024-03-28T17:47:25.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.577+0000 7f6938c00700 1 -- 172.21.15.45:0/1740525460 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f6934199ce0 con 0x7f693406a7e0 2024-03-28T17:47:25.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.577+0000 7f692b400700 1 -- 172.21.15.45:0/1740525460 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f691c013030 con 0x7f693406a7e0 2024-03-28T17:47:25.712 INFO:teuthology.orchestra.run.smithi045.stdout:34359738459 2024-03-28T17:47:25.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.580+0000 7f6938c00700 1 -- 172.21.15.45:0/1740525460 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f69180a2a70 msgr2=0x7f69180a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:25.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.580+0000 7f6938c00700 1 --2- 172.21.15.45:0/1740525460 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f69180a2a70 0x7f69180a4f30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6924007c10 tx=0x7f6924007480 comp rx=0 tx=0).stop 2024-03-28T17:47:25.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.580+0000 7f6938c00700 1 -- 172.21.15.45:0/1740525460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f693406a7e0 msgr2=0x7f693419eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:25.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.580+0000 7f6938c00700 1 --2- 172.21.15.45:0/1740525460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f693406a7e0 0x7f693419eb70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f691c000c00 tx=0x7f691c005040 comp rx=0 tx=0).stop 2024-03-28T17:47:25.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.581+0000 7f6938c00700 1 -- 172.21.15.45:0/1740525460 shutdown_connections 2024-03-28T17:47:25.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.581+0000 7f6938c00700 1 --2- 172.21.15.45:0/1740525460 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f69180a2a70 0x7f69180a4f30 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:25.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.581+0000 7f6938c00700 1 --2- 172.21.15.45:0/1740525460 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69341081e0 0x7f69341987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:25.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.581+0000 7f6938c00700 1 --2- 172.21.15.45:0/1740525460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6934069e40 0x7f693419e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:25.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.581+0000 7f6938c00700 1 --2- 172.21.15.45:0/1740525460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f693406a7e0 0x7f693419eb70 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:25.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.581+0000 7f6938c00700 1 -- 172.21.15.45:0/1740525460 >> 172.21.15.45:0/1740525460 conn(0x7f6934077fc0 msgr2=0x7f69340ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:25.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.581+0000 7f6938c00700 1 -- 172.21.15.45:0/1740525460 shutdown_connections 2024-03-28T17:47:25.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.581+0000 7f6938c00700 1 -- 172.21.15.45:0/1740525460 wait complete. 2024-03-28T17:47:25.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:25 smithi071 ceph-mon[36597]: pgmap v355: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:47:25.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:25 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1013073594' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T17:47:25.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:25 smithi151 ceph-mon[36481]: pgmap v355: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:47:25.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:25 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1013073594' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T17:47:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:26 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1740525460' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T17:47:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:26 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/2713222266' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T17:47:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:26 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1740525460' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T17:47:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:26 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/2713222266' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T17:47:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:26 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1740525460' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T17:47:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:26 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/2713222266' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T17:47:26.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.893+0000 7f8dbbc00700 1 -- 172.21.15.45:0/3052742400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8db406b2e0 msgr2=0x7f8db4106de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:26.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.893+0000 7f8dbbc00700 1 --2- 172.21.15.45:0/3052742400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8db406b2e0 0x7f8db4106de0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8da00099c0 tx=0x7f8da0009cd0 comp rx=0 tx=0).stop 2024-03-28T17:47:26.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.893+0000 7f8dbbc00700 1 -- 172.21.15.45:0/3052742400 shutdown_connections 2024-03-28T17:47:26.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.893+0000 7f8dbbc00700 1 --2- 172.21.15.45:0/3052742400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8db4107510 0x7f8db410f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:26.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.893+0000 7f8dbbc00700 1 --2- 172.21.15.45:0/3052742400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8db406a930 0x7f8db406ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:26.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.893+0000 7f8dbbc00700 1 --2- 172.21.15.45:0/3052742400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8db406b2e0 0x7f8db4106de0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:26.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.893+0000 7f8dbbc00700 1 -- 172.21.15.45:0/3052742400 >> 172.21.15.45:0/3052742400 conn(0x7f8db40770e0 msgr2=0x7f8db40774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:26.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.893+0000 7f8dbbc00700 1 -- 172.21.15.45:0/3052742400 shutdown_connections 2024-03-28T17:47:26.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.893+0000 7f8dbbc00700 1 -- 172.21.15.45:0/3052742400 wait complete. 2024-03-28T17:47:26.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.894+0000 7f8dbbc00700 1 Processor -- start 2024-03-28T17:47:26.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.895+0000 7f8dbbc00700 1 -- start start 2024-03-28T17:47:26.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.895+0000 7f8dbbc00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8db406a930 0x7f8db41a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:26.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.895+0000 7f8dbbc00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8db406b2e0 0x7f8db41a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:26.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.895+0000 7f8dbbc00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8db4107510 0x7f8db419a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:26.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.895+0000 7f8dbbc00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db41124b0 con 0x7f8db406a930 2024-03-28T17:47:26.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.895+0000 7f8dbbc00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db4112330 con 0x7f8db4107510 2024-03-28T17:47:26.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.895+0000 7f8dbbc00700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db4112630 con 0x7f8db406b2e0 2024-03-28T17:47:26.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.895+0000 7f8db9c00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8db4107510 0x7f8db419a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:26.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.896+0000 7f8db9c00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8db4107510 0x7f8db419a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:33646/0 (socket says 172.21.15.45:33646) 2024-03-28T17:47:26.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.896+0000 7f8db9c00700 1 -- 172.21.15.45:0/851174840 learned_addr learned my addr 172.21.15.45:0/851174840 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:26.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.896+0000 7f8db9c00700 1 -- 172.21.15.45:0/851174840 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8db406b2e0 msgr2=0x7f8db41a0d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T17:47:26.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.896+0000 7f8db9c00700 1 --2- 172.21.15.45:0/851174840 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8db406b2e0 0x7f8db41a0d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:26.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.896+0000 7f8db9c00700 1 -- 172.21.15.45:0/851174840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8db406a930 msgr2=0x7f8db41a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:26.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.896+0000 7f8db9c00700 1 --2- 172.21.15.45:0/851174840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8db406a930 0x7f8db41a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:26.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.896+0000 7f8db9c00700 1 -- 172.21.15.45:0/851174840 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db419b130 con 0x7f8db4107510 2024-03-28T17:47:26.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.896+0000 7f8db9c00700 1 --2- 172.21.15.45:0/851174840 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8db4107510 0x7f8db419a9d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8dac007d70 tx=0x7f8dac00a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:26.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.897+0000 7f8db1600700 1 -- 172.21.15.45:0/851174840 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dac016020 con 0x7f8db4107510 2024-03-28T17:47:26.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.897+0000 7f8db1600700 1 -- 172.21.15.45:0/851174840 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8dac00ad40 con 0x7f8db4107510 2024-03-28T17:47:26.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.897+0000 7f8dbbc00700 1 -- 172.21.15.45:0/851174840 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db419b420 con 0x7f8db4107510 2024-03-28T17:47:26.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.897+0000 7f8dbbc00700 1 -- 172.21.15.45:0/851174840 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db41a76e0 con 0x7f8db4107510 2024-03-28T17:47:26.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.897+0000 7f8db1600700 1 -- 172.21.15.45:0/851174840 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dac0116f0 con 0x7f8db4107510 2024-03-28T17:47:26.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.898+0000 7f8dbbc00700 1 -- 172.21.15.45:0/851174840 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8db4068e50 con 0x7f8db4107510 2024-03-28T17:47:26.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.900+0000 7f8db1600700 1 -- 172.21.15.45:0/851174840 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f8dac019020 con 0x7f8db4107510 2024-03-28T17:47:26.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.902+0000 7f8db1600700 1 --2- 172.21.15.45:0/851174840 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8d9c0a29a0 0x7f8d9c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.902+0000 7f8db9200700 1 --2- 172.21.15.45:0/851174840 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8d9c0a29a0 0x7f8d9c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.903+0000 7f8db1600700 1 -- 172.21.15.45:0/851174840 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f8dac0ceaf0 con 0x7f8db4107510 2024-03-28T17:47:26.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.903+0000 7f8db9200700 1 --2- 172.21.15.45:0/851174840 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8d9c0a29a0 0x7f8d9c0a4e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8da8005950 tx=0x7f8da80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:47:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:26.905+0000 7f8db1600700 1 -- 172.21.15.45:0/851174840 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8dac020020 con 0x7f8db4107510 2024-03-28T17:47:27.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.145+0000 7f8dbbc00700 1 -- 172.21.15.45:0/851174840 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f8db41a7970 con 0x7f8db4107510 2024-03-28T17:47:27.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.146+0000 7f8db1600700 1 -- 172.21.15.45:0/851174840 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f8dac019310 con 0x7f8db4107510 2024-03-28T17:47:27.146 INFO:teuthology.orchestra.run.smithi045.stdout:309237645346 2024-03-28T17:47:27.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.148+0000 7f8dbbc00700 1 -- 172.21.15.45:0/851174840 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8d9c0a29a0 msgr2=0x7f8d9c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:27.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.148+0000 7f8dbbc00700 1 --2- 172.21.15.45:0/851174840 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8d9c0a29a0 0x7f8d9c0a4e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8da8005950 tx=0x7f8da80058e0 comp rx=0 tx=0).stop 2024-03-28T17:47:27.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.148+0000 7f8dbbc00700 1 -- 172.21.15.45:0/851174840 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8db4107510 msgr2=0x7f8db419a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:27.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.148+0000 7f8dbbc00700 1 --2- 172.21.15.45:0/851174840 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8db4107510 0x7f8db419a9d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8dac007d70 tx=0x7f8dac00a340 comp rx=0 tx=0).stop 2024-03-28T17:47:27.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.149+0000 7f8dbbc00700 1 -- 172.21.15.45:0/851174840 shutdown_connections 2024-03-28T17:47:27.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.149+0000 7f8dbbc00700 1 --2- 172.21.15.45:0/851174840 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f8d9c0a29a0 0x7f8d9c0a4e60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:27.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.149+0000 7f8dbbc00700 1 --2- 172.21.15.45:0/851174840 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8db4107510 0x7f8db419a9d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:27.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.149+0000 7f8dbbc00700 1 --2- 172.21.15.45:0/851174840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8db406a930 0x7f8db41a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:27.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.149+0000 7f8dbbc00700 1 --2- 172.21.15.45:0/851174840 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8db406b2e0 0x7f8db41a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:27.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.149+0000 7f8dbbc00700 1 -- 172.21.15.45:0/851174840 >> 172.21.15.45:0/851174840 conn(0x7f8db40770e0 msgr2=0x7f8db410d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:27.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.149+0000 7f8dbbc00700 1 -- 172.21.15.45:0/851174840 shutdown_connections 2024-03-28T17:47:27.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.149+0000 7f8dbbc00700 1 -- 172.21.15.45:0/851174840 wait complete. 2024-03-28T17:47:27.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.156+0000 7f294d200700 1 -- 172.21.15.45:0/1935330153 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29481073b0 msgr2=0x7f294810ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:27.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.156+0000 7f294d200700 1 --2- 172.21.15.45:0/1935330153 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29481073b0 0x7f294810ef80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2940009fc0 tx=0x7f294000b330 comp rx=0 tx=0).stop 2024-03-28T17:47:27.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.157+0000 7f294d200700 1 -- 172.21.15.45:0/1935330153 shutdown_connections 2024-03-28T17:47:27.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.157+0000 7f294d200700 1 --2- 172.21.15.45:0/1935330153 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29481073b0 0x7f294810ef80 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:27.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.157+0000 7f294d200700 1 --2- 172.21.15.45:0/1935330153 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29481022d0 0x7f29481026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:27.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.157+0000 7f294d200700 1 --2- 172.21.15.45:0/1935330153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2948102c80 0x7f2948106cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:27.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.157+0000 7f294d200700 1 -- 172.21.15.45:0/1935330153 >> 172.21.15.45:0/1935330153 conn(0x7f29480fdb60 msgr2=0x7f29480fff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:27.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.157+0000 7f294d200700 1 -- 172.21.15.45:0/1935330153 shutdown_connections 2024-03-28T17:47:27.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.157+0000 7f294d200700 1 -- 172.21.15.45:0/1935330153 wait complete. 2024-03-28T17:47:27.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.158+0000 7f294d200700 1 Processor -- start 2024-03-28T17:47:27.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.159+0000 7f294d200700 1 -- start start 2024-03-28T17:47:27.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.159+0000 7f294d200700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29481022d0 0x7f29481a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:27.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.159+0000 7f294d200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2948102c80 0x7f29481a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:27.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.159+0000 7f294d200700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29481073b0 0x7f294819a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:27.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.159+0000 7f294d200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29481123f0 con 0x7f2948102c80 2024-03-28T17:47:27.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.159+0000 7f294d200700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2948112270 con 0x7f29481022d0 2024-03-28T17:47:27.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.159+0000 7f294d200700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2948112570 con 0x7f29481073b0 2024-03-28T17:47:27.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.159+0000 7f2947400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29481022d0 0x7f29481a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:27.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.160+0000 7f2947400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29481022d0 0x7f29481a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:33670/0 (socket says 172.21.15.45:33670) 2024-03-28T17:47:27.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.160+0000 7f2947400700 1 -- 172.21.15.45:0/3268664886 learned_addr learned my addr 172.21.15.45:0/3268664886 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:27.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.160+0000 7f2947e00700 1 --2- 172.21.15.45:0/3268664886 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29481073b0 0x7f294819a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:27.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.160+0000 7f2947400700 1 -- 172.21.15.45:0/3268664886 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29481073b0 msgr2=0x7f294819a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:27.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.160+0000 7f2947400700 1 --2- 172.21.15.45:0/3268664886 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29481073b0 0x7f294819a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:27.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.160+0000 7f2947400700 1 -- 172.21.15.45:0/3268664886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2948102c80 msgr2=0x7f29481a0bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T17:47:27.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.160+0000 7f2947400700 1 --2- 172.21.15.45:0/3268664886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2948102c80 0x7f29481a0bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:27.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.160+0000 7f2947400700 1 -- 172.21.15.45:0/3268664886 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294819b060 con 0x7f29481022d0 2024-03-28T17:47:27.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.160+0000 7f2947e00700 1 --2- 172.21.15.45:0/3268664886 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29481073b0 0x7f294819a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:47:27.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.160+0000 7f2947400700 1 --2- 172.21.15.45:0/3268664886 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29481022d0 0x7f29481a06b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f293800ec10 tx=0x7f293800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:27.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.160+0000 7f293fe00700 1 -- 172.21.15.45:0/3268664886 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293800cd40 con 0x7f29481022d0 2024-03-28T17:47:27.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.161+0000 7f293fe00700 1 -- 172.21.15.45:0/3268664886 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2938004500 con 0x7f29481022d0 2024-03-28T17:47:27.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.161+0000 7f293fe00700 1 -- 172.21.15.45:0/3268664886 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2938010690 con 0x7f29481022d0 2024-03-28T17:47:27.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.161+0000 7f294d200700 1 -- 172.21.15.45:0/3268664886 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f294819b350 con 0x7f29481022d0 2024-03-28T17:47:27.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.161+0000 7f294d200700 1 -- 172.21.15.45:0/3268664886 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29481a7510 con 0x7f29481022d0 2024-03-28T17:47:27.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.162+0000 7f294d200700 1 -- 172.21.15.45:0/3268664886 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29481037b0 con 0x7f29481022d0 2024-03-28T17:47:27.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.164+0000 7f293fe00700 1 -- 172.21.15.45:0/3268664886 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f2938004020 con 0x7f29481022d0 2024-03-28T17:47:27.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.166+0000 7f293fe00700 1 --2- 172.21.15.45:0/3268664886 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f292c0a29a0 0x7f292c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:27.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.166+0000 7f293fe00700 1 -- 172.21.15.45:0/3268664886 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f29380cef70 con 0x7f29481022d0 2024-03-28T17:47:27.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.166+0000 7f2946a00700 1 --2- 172.21.15.45:0/3268664886 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f292c0a29a0 0x7f292c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:27.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.167+0000 7f2946a00700 1 --2- 172.21.15.45:0/3268664886 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f292c0a29a0 0x7f292c0a4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f294810f430 tx=0x7f2934009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:27.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.169+0000 7f293fe00700 1 -- 172.21.15.45:0/3268664886 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f293809be60 con 0x7f29481022d0 2024-03-28T17:47:27.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.406+0000 7f294d200700 1 -- 172.21.15.45:0/3268664886 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f29481a77d0 con 0x7f29481022d0 2024-03-28T17:47:27.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.407+0000 7f293fe00700 1 -- 172.21.15.45:0/3268664886 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f293809b760 con 0x7f29481022d0 2024-03-28T17:47:27.408 INFO:teuthology.orchestra.run.smithi045.stdout:107374182474 2024-03-28T17:47:27.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.410+0000 7f294d200700 1 -- 172.21.15.45:0/3268664886 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f292c0a29a0 msgr2=0x7f292c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:27.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.410+0000 7f294d200700 1 --2- 172.21.15.45:0/3268664886 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f292c0a29a0 0x7f292c0a4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f294810f430 tx=0x7f2934009260 comp rx=0 tx=0).stop 2024-03-28T17:47:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.410+0000 7f294d200700 1 -- 172.21.15.45:0/3268664886 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29481022d0 msgr2=0x7f29481a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.410+0000 7f294d200700 1 --2- 172.21.15.45:0/3268664886 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29481022d0 0x7f29481a06b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f293800ec10 tx=0x7f293800ef20 comp rx=0 tx=0).stop 2024-03-28T17:47:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.410+0000 7f294d200700 1 -- 172.21.15.45:0/3268664886 shutdown_connections 2024-03-28T17:47:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.410+0000 7f294d200700 1 --2- 172.21.15.45:0/3268664886 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f292c0a29a0 0x7f292c0a4e60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.410+0000 7f294d200700 1 --2- 172.21.15.45:0/3268664886 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29481073b0 0x7f294819a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.410+0000 7f294d200700 1 --2- 172.21.15.45:0/3268664886 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29481022d0 0x7f29481a06b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.410+0000 7f294d200700 1 --2- 172.21.15.45:0/3268664886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2948102c80 0x7f29481a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:27.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.410+0000 7f294d200700 1 -- 172.21.15.45:0/3268664886 >> 172.21.15.45:0/3268664886 conn(0x7f29480fdb60 msgr2=0x7f29480fe500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:27.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.411+0000 7f294d200700 1 -- 172.21.15.45:0/3268664886 shutdown_connections 2024-03-28T17:47:27.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:27.411+0000 7f294d200700 1 -- 172.21.15.45:0/3268664886 wait complete. 2024-03-28T17:47:27.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:27 smithi045 ceph-mon[31773]: pgmap v356: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:47:27.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:27 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/851174840' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T17:47:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:27 smithi071 ceph-mon[36597]: pgmap v356: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:47:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:27 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/851174840' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T17:47:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:27 smithi151 ceph-mon[36481]: pgmap v356: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:47:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:27 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/851174840' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T17:47:28.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.758+0000 7f66d6400700 1 -- 172.21.15.45:0/2669463508 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66d00743c0 msgr2=0x7f66d00747a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:28.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.758+0000 7f66d6400700 1 --2- 172.21.15.45:0/2669463508 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66d00743c0 0x7f66d00747a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f66c00098f0 tx=0x7f66c0009c00 comp rx=0 tx=0).stop 2024-03-28T17:47:28.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.758+0000 7f66d6400700 1 -- 172.21.15.45:0/2669463508 shutdown_connections 2024-03-28T17:47:28.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.758+0000 7f66d6400700 1 --2- 172.21.15.45:0/2669463508 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66d00743c0 0x7f66d00747a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:28.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.758+0000 7f66d6400700 1 --2- 172.21.15.45:0/2669463508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66d010eb90 0x7f66d0110f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:28.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.758+0000 7f66d6400700 1 --2- 172.21.15.45:0/2669463508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66d0074ce0 0x7f66d010e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:28.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.758+0000 7f66d6400700 1 -- 172.21.15.45:0/2669463508 >> 172.21.15.45:0/2669463508 conn(0x7f66d00792c0 msgr2=0x7f66d00796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:28.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.758+0000 7f66d6400700 1 -- 172.21.15.45:0/2669463508 shutdown_connections 2024-03-28T17:47:28.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.758+0000 7f66d6400700 1 -- 172.21.15.45:0/2669463508 wait complete. 2024-03-28T17:47:28.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.759+0000 7f66d6400700 1 Processor -- start 2024-03-28T17:47:28.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.759+0000 7f66d6400700 1 -- start start 2024-03-28T17:47:28.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.759+0000 7f66d6400700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66d00743c0 0x7f66d01a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:28.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.759+0000 7f66d6400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66d0074ce0 0x7f66d01a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:28.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.759+0000 7f66d6400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66d010eb90 0x7f66d019a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:28.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.759+0000 7f66d6400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d0114680 con 0x7f66d010eb90 2024-03-28T17:47:28.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.759+0000 7f66d6400700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d0114500 con 0x7f66d0074ce0 2024-03-28T17:47:28.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.759+0000 7f66d6400700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d0114800 con 0x7f66d00743c0 2024-03-28T17:47:28.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.760+0000 7f66cf400700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66d00743c0 0x7f66d01a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:28.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.760+0000 7f66cf400700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66d00743c0 0x7f66d01a07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:49500/0 (socket says 172.21.15.45:49500) 2024-03-28T17:47:28.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.760+0000 7f66cea00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66d0074ce0 0x7f66d01a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:28.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.760+0000 7f66cf400700 1 -- 172.21.15.45:0/1013073594 learned_addr learned my addr 172.21.15.45:0/1013073594 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:28.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.760+0000 7f66cfe00700 1 --2- 172.21.15.45:0/1013073594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66d010eb90 0x7f66d019a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:28.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.760+0000 7f66cf400700 1 -- 172.21.15.45:0/1013073594 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66d0074ce0 msgr2=0x7f66d01a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:28.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.760+0000 7f66cf400700 1 --2- 172.21.15.45:0/1013073594 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66d0074ce0 0x7f66d01a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:28.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.760+0000 7f66cf400700 1 -- 172.21.15.45:0/1013073594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66d010eb90 msgr2=0x7f66d019a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:28.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.760+0000 7f66cf400700 1 --2- 172.21.15.45:0/1013073594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66d010eb90 0x7f66d019a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:28.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.760+0000 7f66cf400700 1 -- 172.21.15.45:0/1013073594 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d019b120 con 0x7f66d00743c0 2024-03-28T17:47:28.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.760+0000 7f66cf400700 1 --2- 172.21.15.45:0/1013073594 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66d00743c0 0x7f66d01a07b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f66c0009230 tx=0x7f66c000fa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:28.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.761+0000 7f66c7e00700 1 -- 172.21.15.45:0/1013073594 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c0005180 con 0x7f66d00743c0 2024-03-28T17:47:28.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.761+0000 7f66c7e00700 1 -- 172.21.15.45:0/1013073594 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66c000b560 con 0x7f66d00743c0 2024-03-28T17:47:28.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.761+0000 7f66d6400700 1 -- 172.21.15.45:0/1013073594 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d019b3b0 con 0x7f66d00743c0 2024-03-28T17:47:28.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.761+0000 7f66c7e00700 1 -- 172.21.15.45:0/1013073594 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c0003a70 con 0x7f66d00743c0 2024-03-28T17:47:28.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.761+0000 7f66d6400700 1 -- 172.21.15.45:0/1013073594 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d01a7680 con 0x7f66d00743c0 2024-03-28T17:47:28.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.762+0000 7f66d6400700 1 -- 172.21.15.45:0/1013073594 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d0068e50 con 0x7f66d00743c0 2024-03-28T17:47:28.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.764+0000 7f66c7e00700 1 -- 172.21.15.45:0/1013073594 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f66c0029060 con 0x7f66d00743c0 2024-03-28T17:47:28.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.765+0000 7f66c7e00700 1 --2- 172.21.15.45:0/1013073594 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f66b40a2920 0x7f66b40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:28.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.765+0000 7f66c7e00700 1 -- 172.21.15.45:0/1013073594 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f66c00cf500 con 0x7f66d00743c0 2024-03-28T17:47:28.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.765+0000 7f66cea00700 1 --2- 172.21.15.45:0/1013073594 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f66b40a2920 0x7f66b40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:28.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.766+0000 7f66cea00700 1 --2- 172.21.15.45:0/1013073594 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f66b40a2920 0x7f66b40a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f66bc009750 tx=0x7f66bc009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:28.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:24.769+0000 7f66c7e00700 1 -- 172.21.15.45:0/1013073594 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f66c0013030 con 0x7f66d00743c0 2024-03-28T17:47:28.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.014+0000 7f66d6400700 1 -- 172.21.15.45:0/1013073594 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f66d002da30 con 0x7f66d00743c0 2024-03-28T17:47:28.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.014+0000 7f66c7e00700 1 -- 172.21.15.45:0/1013073594 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f66c009c480 con 0x7f66d00743c0 2024-03-28T17:47:28.424 INFO:teuthology.orchestra.run.smithi045.stdout:77309411408 2024-03-28T17:47:28.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.017+0000 7f66d6400700 1 -- 172.21.15.45:0/1013073594 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f66b40a2920 msgr2=0x7f66b40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:28.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.017+0000 7f66d6400700 1 --2- 172.21.15.45:0/1013073594 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f66b40a2920 0x7f66b40a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f66bc009750 tx=0x7f66bc009310 comp rx=0 tx=0).stop 2024-03-28T17:47:28.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.017+0000 7f66d6400700 1 -- 172.21.15.45:0/1013073594 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66d00743c0 msgr2=0x7f66d01a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:28.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.017+0000 7f66d6400700 1 --2- 172.21.15.45:0/1013073594 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66d00743c0 0x7f66d01a07b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f66c0009230 tx=0x7f66c000fa70 comp rx=0 tx=0).stop 2024-03-28T17:47:28.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.017+0000 7f66d6400700 1 -- 172.21.15.45:0/1013073594 shutdown_connections 2024-03-28T17:47:28.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.017+0000 7f66d6400700 1 --2- 172.21.15.45:0/1013073594 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f66b40a2920 0x7f66b40a4de0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:28.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.017+0000 7f66d6400700 1 --2- 172.21.15.45:0/1013073594 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66d00743c0 0x7f66d01a07b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:28.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.017+0000 7f66d6400700 1 --2- 172.21.15.45:0/1013073594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66d010eb90 0x7f66d019a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:28.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.017+0000 7f66d6400700 1 --2- 172.21.15.45:0/1013073594 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66d0074ce0 0x7f66d01a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:28.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.017+0000 7f66d6400700 1 -- 172.21.15.45:0/1013073594 >> 172.21.15.45:0/1013073594 conn(0x7f66d00792c0 msgr2=0x7f66d00ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:28.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.018+0000 7f66d6400700 1 -- 172.21.15.45:0/1013073594 shutdown_connections 2024-03-28T17:47:28.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:25.018+0000 7f66d6400700 1 -- 172.21.15.45:0/1013073594 wait complete. 2024-03-28T17:47:28.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:28 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3268664886' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T17:47:28.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:28 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3268664886' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T17:47:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:28 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3268664886' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T17:47:29.606 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:29 smithi045 ceph-mon[31773]: pgmap v357: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:47:29.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:29 smithi071 ceph-mon[36597]: pgmap v357: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:47:29.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:29 smithi151 ceph-mon[36481]: pgmap v357: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:47:30.509 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645340 got 309237645346 for osd.10 2024-03-28T17:47:30.509 DEBUG:teuthology.parallel:result is None 2024-03-28T17:47:30.640 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182470 got 107374182474 for osd.3 2024-03-28T17:47:30.640 DEBUG:teuthology.parallel:result is None 2024-03-28T17:47:31.141 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738455 got 34359738459 for osd.0 2024-03-28T17:47:31.141 DEBUG:teuthology.parallel:result is None 2024-03-28T17:47:31.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:31 smithi045 ceph-mon[31773]: pgmap v358: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T17:47:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:31 smithi071 ceph-mon[36597]: pgmap v358: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T17:47:31.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:31 smithi151 ceph-mon[36481]: pgmap v358: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T17:47:32.625 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411403 got 77309411408 for osd.2 2024-03-28T17:47:32.625 DEBUG:teuthology.parallel:result is None 2024-03-28T17:47:33.441 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757307 got 163208757310 for osd.5 2024-03-28T17:47:33.441 DEBUG:teuthology.parallel:result is None 2024-03-28T17:47:33.612 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:33 smithi071 ceph-mon[36597]: pgmap v359: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:47:33.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:33 smithi045 ceph-mon[31773]: pgmap v359: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:47:33.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:33 smithi151 ceph-mon[36481]: pgmap v359: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:47:35.552 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:35 smithi045 ceph-mon[31773]: pgmap v360: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:47:35.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.714+0000 7f853c000700 1 -- 172.21.15.45:0/3476603125 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f85340743c0 msgr2=0x7f85340747a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:35.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.714+0000 7f853c000700 1 --2- 172.21.15.45:0/3476603125 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f85340743c0 0x7f85340747a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8528009990 tx=0x7f8528009ca0 comp rx=0 tx=0).stop 2024-03-28T17:47:35.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.714+0000 7f853c000700 1 -- 172.21.15.45:0/3476603125 shutdown_connections 2024-03-28T17:47:35.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.714+0000 7f853c000700 1 --2- 172.21.15.45:0/3476603125 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8534113360 0x7f8534115750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:35.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.714+0000 7f853c000700 1 --2- 172.21.15.45:0/3476603125 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f85340743c0 0x7f85340747a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:35.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.714+0000 7f853c000700 1 --2- 172.21.15.45:0/3476603125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8534074ce0 0x7f8534112e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:35.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.714+0000 7f853c000700 1 -- 172.21.15.45:0/3476603125 >> 172.21.15.45:0/3476603125 conn(0x7f85340792f0 msgr2=0x7f8534079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:35.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.714+0000 7f853c000700 1 -- 172.21.15.45:0/3476603125 shutdown_connections 2024-03-28T17:47:35.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.714+0000 7f853c000700 1 -- 172.21.15.45:0/3476603125 wait complete. 2024-03-28T17:47:35.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.715+0000 7f853c000700 1 Processor -- start 2024-03-28T17:47:35.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.716+0000 7f853c000700 1 -- start start 2024-03-28T17:47:35.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.716+0000 7f853c000700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85340743c0 0x7f853419efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:35.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.716+0000 7f853c000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8534074ce0 0x7f853419f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:35.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.716+0000 7f853c000700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8534113360 0x7f85341a3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:35.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.716+0000 7f853c000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8534118b90 con 0x7f8534074ce0 2024-03-28T17:47:35.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.716+0000 7f853c000700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8534118a10 con 0x7f8534113360 2024-03-28T17:47:35.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.716+0000 7f853c000700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8534118d10 con 0x7f85340743c0 2024-03-28T17:47:35.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.716+0000 7f8539600700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85340743c0 0x7f853419efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:35.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.716+0000 7f8539600700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85340743c0 0x7f853419efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:57328/0 (socket says 172.21.15.45:57328) 2024-03-28T17:47:35.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.716+0000 7f8539600700 1 -- 172.21.15.45:0/307237086 learned_addr learned my addr 172.21.15.45:0/307237086 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:35.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.716+0000 7f853a000700 1 --2- 172.21.15.45:0/307237086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8534113360 0x7f85341a3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:35.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.717+0000 7f8538c00700 1 --2- 172.21.15.45:0/307237086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8534074ce0 0x7f853419f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:35.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.717+0000 7f8539600700 1 -- 172.21.15.45:0/307237086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8534113360 msgr2=0x7f85341a3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:35.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.717+0000 7f8539600700 1 --2- 172.21.15.45:0/307237086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8534113360 0x7f85341a3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:35.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.717+0000 7f8539600700 1 -- 172.21.15.45:0/307237086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8534074ce0 msgr2=0x7f853419f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:35.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.717+0000 7f8539600700 1 --2- 172.21.15.45:0/307237086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8534074ce0 0x7f853419f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:35.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.717+0000 7f8539600700 1 -- 172.21.15.45:0/307237086 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85341a3fa0 con 0x7f85340743c0 2024-03-28T17:47:35.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.717+0000 7f853a000700 1 --2- 172.21.15.45:0/307237086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8534113360 0x7f85341a3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:47:35.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.717+0000 7f8539600700 1 --2- 172.21.15.45:0/307237086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85340743c0 0x7f853419efe0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8528000c00 tx=0x7f8528005590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:35.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.717+0000 7f8532000700 1 -- 172.21.15.45:0/307237086 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8528027070 con 0x7f85340743c0 2024-03-28T17:47:35.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.718+0000 7f8532000700 1 -- 172.21.15.45:0/307237086 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f85280056b0 con 0x7f85340743c0 2024-03-28T17:47:35.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.718+0000 7f853c000700 1 -- 172.21.15.45:0/307237086 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85341a4230 con 0x7f85340743c0 2024-03-28T17:47:35.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.718+0000 7f8532000700 1 -- 172.21.15.45:0/307237086 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8528017650 con 0x7f85340743c0 2024-03-28T17:47:35.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.718+0000 7f853c000700 1 -- 172.21.15.45:0/307237086 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85341abb10 con 0x7f85340743c0 2024-03-28T17:47:35.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.720+0000 7f853c000700 1 -- 172.21.15.45:0/307237086 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8534075af0 con 0x7f85340743c0 2024-03-28T17:47:35.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.721+0000 7f8532000700 1 -- 172.21.15.45:0/307237086 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f852802a070 con 0x7f85340743c0 2024-03-28T17:47:35.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.722+0000 7f8532000700 1 --2- 172.21.15.45:0/307237086 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f851c0a2920 0x7f851c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:35.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.722+0000 7f8532000700 1 -- 172.21.15.45:0/307237086 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f85280cfb20 con 0x7f85340743c0 2024-03-28T17:47:35.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.723+0000 7f8538c00700 1 --2- 172.21.15.45:0/307237086 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f851c0a2920 0x7f851c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:35.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.723+0000 7f8538c00700 1 --2- 172.21.15.45:0/307237086 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f851c0a2920 0x7f851c0a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f85341151d0 tx=0x7f8520009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:35.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.726+0000 7f8532000700 1 -- 172.21.15.45:0/307237086 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8528013070 con 0x7f85340743c0 2024-03-28T17:47:35.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:35 smithi071 ceph-mon[36597]: pgmap v360: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:47:35.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:35 smithi151 ceph-mon[36481]: pgmap v360: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:47:35.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.622+0000 7f732e400700 1 -- 172.21.15.45:0/4221722372 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f73281074b0 msgr2=0x7f732810f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:35.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.622+0000 7f732e400700 1 --2- 172.21.15.45:0/4221722372 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f73281074b0 0x7f732810f080 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f732000b0d0 tx=0x7f732000b3e0 comp rx=0 tx=0).stop 2024-03-28T17:47:35.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.622+0000 7f732e400700 1 -- 172.21.15.45:0/4221722372 shutdown_connections 2024-03-28T17:47:35.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.622+0000 7f732e400700 1 --2- 172.21.15.45:0/4221722372 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f73281074b0 0x7f732810f080 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:35.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.622+0000 7f732e400700 1 --2- 172.21.15.45:0/4221722372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73281023d0 0x7f73281027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:35.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.622+0000 7f732e400700 1 --2- 172.21.15.45:0/4221722372 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7328102d80 0x7f7328106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:35.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.622+0000 7f732e400700 1 -- 172.21.15.45:0/4221722372 >> 172.21.15.45:0/4221722372 conn(0x7f73280fdc40 msgr2=0x7f7328100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:35.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.622+0000 7f732e400700 1 -- 172.21.15.45:0/4221722372 shutdown_connections 2024-03-28T17:47:35.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.622+0000 7f732e400700 1 -- 172.21.15.45:0/4221722372 wait complete. 2024-03-28T17:47:35.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.624+0000 7f732e400700 1 Processor -- start 2024-03-28T17:47:35.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.624+0000 7f732e400700 1 -- start start 2024-03-28T17:47:35.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.624+0000 7f732e400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73281023d0 0x7f73281a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:35.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.624+0000 7f732e400700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7328102d80 0x7f73281a0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:35.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.624+0000 7f732e400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73281074b0 0x7f732819a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:35.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.624+0000 7f732e400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7328112510 con 0x7f73281023d0 2024-03-28T17:47:35.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.624+0000 7f732e400700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7328112390 con 0x7f73281074b0 2024-03-28T17:47:35.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.624+0000 7f732e400700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7328112690 con 0x7f7328102d80 2024-03-28T17:47:35.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.624+0000 7f7326a00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7328102d80 0x7f73281a0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:35.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.624+0000 7f7326a00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7328102d80 0x7f73281a0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:57308/0 (socket says 172.21.15.45:57308) 2024-03-28T17:47:35.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.624+0000 7f7326a00700 1 -- 172.21.15.45:0/3820287716 learned_addr learned my addr 172.21.15.45:0/3820287716 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:35.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.625+0000 7f7327e00700 1 --2- 172.21.15.45:0/3820287716 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73281074b0 0x7f732819a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:35.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.625+0000 7f7327400700 1 --2- 172.21.15.45:0/3820287716 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73281023d0 0x7f73281a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:35.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.625+0000 7f7326a00700 1 -- 172.21.15.45:0/3820287716 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73281074b0 msgr2=0x7f732819a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:35.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.625+0000 7f7326a00700 1 --2- 172.21.15.45:0/3820287716 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73281074b0 0x7f732819a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:35.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.625+0000 7f7326a00700 1 -- 172.21.15.45:0/3820287716 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73281023d0 msgr2=0x7f73281a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:35.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.625+0000 7f7326a00700 1 --2- 172.21.15.45:0/3820287716 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73281023d0 0x7f73281a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:35.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.625+0000 7f7326a00700 1 -- 172.21.15.45:0/3820287716 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f732819b0e0 con 0x7f7328102d80 2024-03-28T17:47:35.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.625+0000 7f7327e00700 1 --2- 172.21.15.45:0/3820287716 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73281074b0 0x7f732819a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:47:35.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.625+0000 7f7327400700 1 --2- 172.21.15.45:0/3820287716 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73281023d0 0x7f73281a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:47:35.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.625+0000 7f7326a00700 1 --2- 172.21.15.45:0/3820287716 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7328102d80 0x7f73281a0cb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f731400e9f0 tx=0x7f731400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:35.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.626+0000 7f731fe00700 1 -- 172.21.15.45:0/3820287716 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731400cbd0 con 0x7f7328102d80 2024-03-28T17:47:35.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.626+0000 7f731fe00700 1 -- 172.21.15.45:0/3820287716 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f731400cd70 con 0x7f7328102d80 2024-03-28T17:47:35.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.626+0000 7f732e400700 1 -- 172.21.15.45:0/3820287716 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f732819b3d0 con 0x7f7328102d80 2024-03-28T17:47:35.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.626+0000 7f731fe00700 1 -- 172.21.15.45:0/3820287716 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7314010430 con 0x7f7328102d80 2024-03-28T17:47:35.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.626+0000 7f732e400700 1 -- 172.21.15.45:0/3820287716 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73281a7640 con 0x7f7328102d80 2024-03-28T17:47:35.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.627+0000 7f732e400700 1 -- 172.21.15.45:0/3820287716 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7328068e50 con 0x7f7328102d80 2024-03-28T17:47:35.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.629+0000 7f731fe00700 1 -- 172.21.15.45:0/3820287716 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f7314004830 con 0x7f7328102d80 2024-03-28T17:47:35.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.630+0000 7f731fe00700 1 --2- 172.21.15.45:0/3820287716 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f730c0a29a0 0x7f730c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:35.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.630+0000 7f731fe00700 1 -- 172.21.15.45:0/3820287716 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f73140cee00 con 0x7f7328102d80 2024-03-28T17:47:35.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.630+0000 7f7327400700 1 --2- 172.21.15.45:0/3820287716 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f730c0a29a0 0x7f730c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:35.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.631+0000 7f7327400700 1 --2- 172.21.15.45:0/3820287716 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f730c0a29a0 0x7f730c0a4e60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7328108ae0 tx=0x7f7318009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:35.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.634+0000 7f731fe00700 1 -- 172.21.15.45:0/3820287716 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f731409bcf0 con 0x7f7328102d80 2024-03-28T17:47:35.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.871+0000 7f732e400700 1 -- 172.21.15.45:0/3820287716 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f73281a78d0 con 0x7f7328102d80 2024-03-28T17:47:35.964 INFO:teuthology.orchestra.run.smithi045.stdout:219043332148 2024-03-28T17:47:35.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.872+0000 7f731fe00700 1 -- 172.21.15.45:0/3820287716 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f731409b5f0 con 0x7f7328102d80 2024-03-28T17:47:35.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.876+0000 7f732e400700 1 -- 172.21.15.45:0/3820287716 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f730c0a29a0 msgr2=0x7f730c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:35.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.876+0000 7f732e400700 1 --2- 172.21.15.45:0/3820287716 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f730c0a29a0 0x7f730c0a4e60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7328108ae0 tx=0x7f7318009260 comp rx=0 tx=0).stop 2024-03-28T17:47:35.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.876+0000 7f732e400700 1 -- 172.21.15.45:0/3820287716 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7328102d80 msgr2=0x7f73281a0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:35.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.876+0000 7f732e400700 1 --2- 172.21.15.45:0/3820287716 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7328102d80 0x7f73281a0cb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f731400e9f0 tx=0x7f731400ed00 comp rx=0 tx=0).stop 2024-03-28T17:47:35.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.876+0000 7f732e400700 1 -- 172.21.15.45:0/3820287716 shutdown_connections 2024-03-28T17:47:35.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.876+0000 7f732e400700 1 --2- 172.21.15.45:0/3820287716 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f730c0a29a0 0x7f730c0a4e60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:35.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.876+0000 7f732e400700 1 --2- 172.21.15.45:0/3820287716 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73281074b0 0x7f732819a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:35.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.876+0000 7f732e400700 1 --2- 172.21.15.45:0/3820287716 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73281023d0 0x7f73281a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:35.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.877+0000 7f732e400700 1 --2- 172.21.15.45:0/3820287716 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7328102d80 0x7f73281a0cb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:35.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.877+0000 7f732e400700 1 -- 172.21.15.45:0/3820287716 >> 172.21.15.45:0/3820287716 conn(0x7f73280fdc40 msgr2=0x7f732810ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:35.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.877+0000 7f732e400700 1 -- 172.21.15.45:0/3820287716 shutdown_connections 2024-03-28T17:47:35.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.877+0000 7f732e400700 1 -- 172.21.15.45:0/3820287716 wait complete. 2024-03-28T17:47:35.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.971+0000 7f853c000700 1 -- 172.21.15.45:0/307237086 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f85341a02f0 con 0x7f85340743c0 2024-03-28T17:47:35.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.971+0000 7f8532000700 1 -- 172.21.15.45:0/307237086 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f852809caa0 con 0x7f85340743c0 2024-03-28T17:47:35.972 INFO:teuthology.orchestra.run.smithi045.stdout:141733920838 2024-03-28T17:47:35.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.974+0000 7f853c000700 1 -- 172.21.15.45:0/307237086 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f851c0a2920 msgr2=0x7f851c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:35.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.974+0000 7f853c000700 1 --2- 172.21.15.45:0/307237086 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f851c0a2920 0x7f851c0a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f85341151d0 tx=0x7f8520009260 comp rx=0 tx=0).stop 2024-03-28T17:47:35.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.974+0000 7f853c000700 1 -- 172.21.15.45:0/307237086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85340743c0 msgr2=0x7f853419efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:35.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.974+0000 7f853c000700 1 --2- 172.21.15.45:0/307237086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85340743c0 0x7f853419efe0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8528000c00 tx=0x7f8528005590 comp rx=0 tx=0).stop 2024-03-28T17:47:35.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.974+0000 7f853c000700 1 -- 172.21.15.45:0/307237086 shutdown_connections 2024-03-28T17:47:35.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.974+0000 7f853c000700 1 --2- 172.21.15.45:0/307237086 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f851c0a2920 0x7f851c0a4de0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:35.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.974+0000 7f853c000700 1 --2- 172.21.15.45:0/307237086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85340743c0 0x7f853419efe0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:35.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.974+0000 7f853c000700 1 --2- 172.21.15.45:0/307237086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8534113360 0x7f85341a3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:35.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.974+0000 7f853c000700 1 --2- 172.21.15.45:0/307237086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8534074ce0 0x7f853419f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:35.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.974+0000 7f853c000700 1 -- 172.21.15.45:0/307237086 >> 172.21.15.45:0/307237086 conn(0x7f85340792f0 msgr2=0x7f85341120b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:35.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.975+0000 7f853c000700 1 -- 172.21.15.45:0/307237086 shutdown_connections 2024-03-28T17:47:35.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:35.975+0000 7f853c000700 1 -- 172.21.15.45:0/307237086 wait complete. 2024-03-28T17:47:36.502 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:36 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3820287716' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T17:47:36.502 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:36 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/307237086' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T17:47:36.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.676+0000 7f41bfe00700 1 -- 172.21.15.45:0/1939408276 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41c0103b60 msgr2=0x7f41c010c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:36.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.676+0000 7f41bfe00700 1 --2- 172.21.15.45:0/1939408276 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41c0103b60 0x7f41c010c050 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f41a8009990 tx=0x7f41a8009ca0 comp rx=0 tx=0).stop 2024-03-28T17:47:36.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.676+0000 7f41bfe00700 1 -- 172.21.15.45:0/1939408276 shutdown_connections 2024-03-28T17:47:36.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.676+0000 7f41bfe00700 1 --2- 172.21.15.45:0/1939408276 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41c0103240 0x7f41c0103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:36.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.676+0000 7f41bfe00700 1 --2- 172.21.15.45:0/1939408276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41c010c670 0x7f41c010eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:36.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.676+0000 7f41bfe00700 1 --2- 172.21.15.45:0/1939408276 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41c0103b60 0x7f41c010c050 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:36.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.676+0000 7f41bfe00700 1 -- 172.21.15.45:0/1939408276 >> 172.21.15.45:0/1939408276 conn(0x7f41c00fca70 msgr2=0x7f41c00fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:36.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.676+0000 7f41bfe00700 1 -- 172.21.15.45:0/1939408276 shutdown_connections 2024-03-28T17:47:36.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.676+0000 7f41bfe00700 1 -- 172.21.15.45:0/1939408276 wait complete. 2024-03-28T17:47:36.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.677+0000 7f41bfe00700 1 Processor -- start 2024-03-28T17:47:36.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.678+0000 7f41bfe00700 1 -- start start 2024-03-28T17:47:36.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.678+0000 7f41bfe00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41c0103240 0x7f41c019e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:36.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.678+0000 7f41bfe00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41c0103b60 0x7f41c019eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:36.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.678+0000 7f41bfe00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41c010c670 0x7f41c01986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:36.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.678+0000 7f41bfe00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c0112390 con 0x7f41c010c670 2024-03-28T17:47:36.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.678+0000 7f41bfe00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c0112210 con 0x7f41c0103b60 2024-03-28T17:47:36.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.678+0000 7f41bfe00700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c0112510 con 0x7f41c0103240 2024-03-28T17:47:36.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.678+0000 7f41bea00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41c0103240 0x7f41c019e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:36.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.678+0000 7f41b7e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41c0103b60 0x7f41c019eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:36.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.678+0000 7f41bea00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41c0103240 0x7f41c019e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:35924/0 (socket says 172.21.15.45:35924) 2024-03-28T17:47:36.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.678+0000 7f41bea00700 1 -- 172.21.15.45:0/3647803142 learned_addr learned my addr 172.21.15.45:0/3647803142 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:36.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.678+0000 7f41bf400700 1 --2- 172.21.15.45:0/3647803142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41c010c670 0x7f41c01986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:36.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.679+0000 7f41b7e00700 1 -- 172.21.15.45:0/3647803142 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41c0103240 msgr2=0x7f41c019e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:36.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.679+0000 7f41b7e00700 1 --2- 172.21.15.45:0/3647803142 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41c0103240 0x7f41c019e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:36.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.679+0000 7f41b7e00700 1 -- 172.21.15.45:0/3647803142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41c010c670 msgr2=0x7f41c01986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:36.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.679+0000 7f41b7e00700 1 --2- 172.21.15.45:0/3647803142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41c010c670 0x7f41c01986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:36.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.679+0000 7f41b7e00700 1 -- 172.21.15.45:0/3647803142 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c0198f90 con 0x7f41c0103b60 2024-03-28T17:47:36.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.679+0000 7f41b7e00700 1 --2- 172.21.15.45:0/3647803142 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41c0103b60 0x7f41c019eb20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f41a80098e0 tx=0x7f41a800b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:36.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.679+0000 7f41b7400700 1 -- 172.21.15.45:0/3647803142 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a8027070 con 0x7f41c0103b60 2024-03-28T17:47:36.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.679+0000 7f41b7400700 1 -- 172.21.15.45:0/3647803142 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f41a8003b60 con 0x7f41c0103b60 2024-03-28T17:47:36.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.679+0000 7f41bfe00700 1 -- 172.21.15.45:0/3647803142 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c0199220 con 0x7f41c0103b60 2024-03-28T17:47:36.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.679+0000 7f41b7400700 1 -- 172.21.15.45:0/3647803142 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a8022d50 con 0x7f41c0103b60 2024-03-28T17:47:36.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.679+0000 7f41bfe00700 1 -- 172.21.15.45:0/3647803142 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c01a5560 con 0x7f41c0103b60 2024-03-28T17:47:36.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.681+0000 7f41bfe00700 1 -- 172.21.15.45:0/3647803142 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c006afb0 con 0x7f41c0103b60 2024-03-28T17:47:36.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.682+0000 7f41b7400700 1 -- 172.21.15.45:0/3647803142 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f41a802a070 con 0x7f41c0103b60 2024-03-28T17:47:36.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.684+0000 7f41b7400700 1 --2- 172.21.15.45:0/3647803142 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f41a40a29a0 0x7f41a40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:36.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.684+0000 7f41b7400700 1 -- 172.21.15.45:0/3647803142 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f41a80cf8d0 con 0x7f41c0103b60 2024-03-28T17:47:36.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.684+0000 7f41bea00700 1 --2- 172.21.15.45:0/3647803142 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f41a40a29a0 0x7f41a40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:36.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.685+0000 7f41bea00700 1 --2- 172.21.15.45:0/3647803142 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f41a40a29a0 0x7f41a40a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f41b0009d20 tx=0x7f41b0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:36.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.688+0000 7f41b7400700 1 -- 172.21.15.45:0/3647803142 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f41a805e740 con 0x7f41c0103b60 2024-03-28T17:47:36.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:36 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3820287716' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T17:47:36.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:36 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/307237086' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T17:47:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:36 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3820287716' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T17:47:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:36 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/307237086' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T17:47:36.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.921+0000 7f41bfe00700 1 -- 172.21.15.45:0/3647803142 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f41c0199be0 con 0x7f41c0103b60 2024-03-28T17:47:36.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.921+0000 7f41b7400700 1 -- 172.21.15.45:0/3647803142 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f41a8013070 con 0x7f41c0103b60 2024-03-28T17:47:36.922 INFO:teuthology.orchestra.run.smithi045.stdout:253403070512 2024-03-28T17:47:36.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.924+0000 7f41bfe00700 1 -- 172.21.15.45:0/3647803142 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f41a40a29a0 msgr2=0x7f41a40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:36.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.924+0000 7f41bfe00700 1 --2- 172.21.15.45:0/3647803142 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f41a40a29a0 0x7f41a40a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f41b0009d20 tx=0x7f41b0009260 comp rx=0 tx=0).stop 2024-03-28T17:47:36.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.924+0000 7f41bfe00700 1 -- 172.21.15.45:0/3647803142 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41c0103b60 msgr2=0x7f41c019eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:36.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.924+0000 7f41bfe00700 1 --2- 172.21.15.45:0/3647803142 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41c0103b60 0x7f41c019eb20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f41a80098e0 tx=0x7f41a800b880 comp rx=0 tx=0).stop 2024-03-28T17:47:36.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.924+0000 7f41bfe00700 1 -- 172.21.15.45:0/3647803142 shutdown_connections 2024-03-28T17:47:36.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.924+0000 7f41bfe00700 1 --2- 172.21.15.45:0/3647803142 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f41a40a29a0 0x7f41a40a4e60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:36.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.924+0000 7f41bfe00700 1 --2- 172.21.15.45:0/3647803142 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41c0103240 0x7f41c019e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:36.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.924+0000 7f41bfe00700 1 --2- 172.21.15.45:0/3647803142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41c010c670 0x7f41c01986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:36.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.924+0000 7f41bfe00700 1 --2- 172.21.15.45:0/3647803142 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41c0103b60 0x7f41c019eb20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:36.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.924+0000 7f41bfe00700 1 -- 172.21.15.45:0/3647803142 >> 172.21.15.45:0/3647803142 conn(0x7f41c00fca70 msgr2=0x7f41c00fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:36.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.925+0000 7f41bfe00700 1 -- 172.21.15.45:0/3647803142 shutdown_connections 2024-03-28T17:47:36.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:36.925+0000 7f41bfe00700 1 -- 172.21.15.45:0/3647803142 wait complete. 2024-03-28T17:47:37.628 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:37 smithi045 ceph-mon[31773]: pgmap v361: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:47:37.628 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:37 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3647803142' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T17:47:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:37 smithi071 ceph-mon[36597]: pgmap v361: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:47:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:37 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3647803142' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T17:47:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:37 smithi151 ceph-mon[36481]: pgmap v361: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:47:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:37 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3647803142' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T17:47:38.249 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070505 got 253403070512 for osd.8 2024-03-28T17:47:38.249 DEBUG:teuthology.parallel:result is None 2024-03-28T17:47:38.817 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920832 got 141733920838 for osd.4 2024-03-28T17:47:38.817 DEBUG:teuthology.parallel:result is None 2024-03-28T17:47:38.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.606+0000 7fb7df400700 1 -- 172.21.15.45:0/3936402685 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7d810e7a0 msgr2=0x7fb7d810eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:38.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.606+0000 7fb7df400700 1 --2- 172.21.15.45:0/3936402685 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7d810e7a0 0x7fb7d810eb80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc009990 tx=0x7fb7cc009ca0 comp rx=0 tx=0).stop 2024-03-28T17:47:38.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.606+0000 7fb7df400700 1 -- 172.21.15.45:0/3936402685 shutdown_connections 2024-03-28T17:47:38.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.606+0000 7fb7df400700 1 --2- 172.21.15.45:0/3936402685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7d8109160 0x7fb7d8115670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:38.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.606+0000 7fb7df400700 1 --2- 172.21.15.45:0/3936402685 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7d810e7a0 0x7fb7d810eb80 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:38.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.606+0000 7fb7df400700 1 --2- 172.21.15.45:0/3936402685 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7d81087a0 0x7fb7d8108c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:38.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.606+0000 7fb7df400700 1 -- 172.21.15.45:0/3936402685 >> 172.21.15.45:0/3936402685 conn(0x7fb7d8075960 msgr2=0x7fb7d8077d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:38.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.606+0000 7fb7df400700 1 -- 172.21.15.45:0/3936402685 shutdown_connections 2024-03-28T17:47:38.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.606+0000 7fb7df400700 1 -- 172.21.15.45:0/3936402685 wait complete. 2024-03-28T17:47:38.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.607+0000 7fb7df400700 1 Processor -- start 2024-03-28T17:47:38.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.607+0000 7fb7df400700 1 -- start start 2024-03-28T17:47:38.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.607+0000 7fb7df400700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7d81087a0 0x7fb7d819ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:38.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.607+0000 7fb7df400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7d8109160 0x7fb7d819f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:38.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.607+0000 7fb7df400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7d810e7a0 0x7fb7d81a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:38.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.607+0000 7fb7df400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d8118b20 con 0x7fb7d810e7a0 2024-03-28T17:47:38.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.607+0000 7fb7df400700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d81189a0 con 0x7fb7d8109160 2024-03-28T17:47:38.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.607+0000 7fb7df400700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d8118ca0 con 0x7fb7d81087a0 2024-03-28T17:47:38.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.608+0000 7fb7d7e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7d8109160 0x7fb7d819f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:38.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.608+0000 7fb7dca00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7d81087a0 0x7fb7d819ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:38.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.608+0000 7fb7d7e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7d8109160 0x7fb7d819f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:45264/0 (socket says 172.21.15.45:45264) 2024-03-28T17:47:38.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.608+0000 7fb7d7e00700 1 -- 172.21.15.45:0/3365445051 learned_addr learned my addr 172.21.15.45:0/3365445051 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:38.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.608+0000 7fb7dd400700 1 --2- 172.21.15.45:0/3365445051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7d810e7a0 0x7fb7d81a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:38.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.608+0000 7fb7d7e00700 1 -- 172.21.15.45:0/3365445051 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7d81087a0 msgr2=0x7fb7d819ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:38.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.608+0000 7fb7d7e00700 1 --2- 172.21.15.45:0/3365445051 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7d81087a0 0x7fb7d819ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:38.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.608+0000 7fb7d7e00700 1 -- 172.21.15.45:0/3365445051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7d810e7a0 msgr2=0x7fb7d81a3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:38.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.608+0000 7fb7d7e00700 1 --2- 172.21.15.45:0/3365445051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7d810e7a0 0x7fb7d81a3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:38.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.608+0000 7fb7d7e00700 1 -- 172.21.15.45:0/3365445051 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d81a3f40 con 0x7fb7d8109160 2024-03-28T17:47:38.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.608+0000 7fb7dca00700 1 --2- 172.21.15.45:0/3365445051 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7d81087a0 0x7fb7d819ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:47:38.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.608+0000 7fb7d7e00700 1 --2- 172.21.15.45:0/3365445051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7d8109160 0x7fb7d819f470 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb7c400d980 tx=0x7fb7c400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:38.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.609+0000 7fb7d5600700 1 -- 172.21.15.45:0/3365445051 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c4016070 con 0x7fb7d8109160 2024-03-28T17:47:38.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.609+0000 7fb7d5600700 1 -- 172.21.15.45:0/3365445051 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7c4004d10 con 0x7fb7d8109160 2024-03-28T17:47:38.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.609+0000 7fb7df400700 1 -- 172.21.15.45:0/3365445051 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d81a4230 con 0x7fb7d8109160 2024-03-28T17:47:38.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.609+0000 7fb7d5600700 1 -- 172.21.15.45:0/3365445051 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c4005670 con 0x7fb7d8109160 2024-03-28T17:47:38.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.609+0000 7fb7df400700 1 -- 172.21.15.45:0/3365445051 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d81abab0 con 0x7fb7d8109160 2024-03-28T17:47:38.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.610+0000 7fb7df400700 1 -- 172.21.15.45:0/3365445051 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d8068e50 con 0x7fb7d8109160 2024-03-28T17:47:38.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.612+0000 7fb7d5600700 1 -- 172.21.15.45:0/3365445051 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb7c4004750 con 0x7fb7d8109160 2024-03-28T17:47:38.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.613+0000 7fb7d5600700 1 --2- 172.21.15.45:0/3365445051 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb7c00a2910 0x7fb7c00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:38.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.613+0000 7fb7d5600700 1 -- 172.21.15.45:0/3365445051 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb7c40ce7e0 con 0x7fb7d8109160 2024-03-28T17:47:38.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.613+0000 7fb7dca00700 1 --2- 172.21.15.45:0/3365445051 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb7c00a2910 0x7fb7c00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:38.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.614+0000 7fb7dca00700 1 --2- 172.21.15.45:0/3365445051 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb7c00a2910 0x7fb7c00a4dd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc006010 tx=0x7fb7cc00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:38.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.617+0000 7fb7d5600700 1 -- 172.21.15.45:0/3365445051 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb7c401f020 con 0x7fb7d8109160 2024-03-28T17:47:38.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.849+0000 7fb7df400700 1 -- 172.21.15.45:0/3365445051 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7fb7d802da30 con 0x7fb7d8109160 2024-03-28T17:47:38.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.849+0000 7fb7d5600700 1 -- 172.21.15.45:0/3365445051 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fb7c4022350 con 0x7fb7d8109160 2024-03-28T17:47:38.864 INFO:teuthology.orchestra.run.smithi045.stdout:343597383712 2024-03-28T17:47:38.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.852+0000 7fb7df400700 1 -- 172.21.15.45:0/3365445051 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb7c00a2910 msgr2=0x7fb7c00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:38.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.852+0000 7fb7df400700 1 --2- 172.21.15.45:0/3365445051 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb7c00a2910 0x7fb7c00a4dd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc006010 tx=0x7fb7cc00b540 comp rx=0 tx=0).stop 2024-03-28T17:47:38.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.852+0000 7fb7df400700 1 -- 172.21.15.45:0/3365445051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7d8109160 msgr2=0x7fb7d819f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:38.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.852+0000 7fb7df400700 1 --2- 172.21.15.45:0/3365445051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7d8109160 0x7fb7d819f470 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb7c400d980 tx=0x7fb7c400dc90 comp rx=0 tx=0).stop 2024-03-28T17:47:38.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.852+0000 7fb7df400700 1 -- 172.21.15.45:0/3365445051 shutdown_connections 2024-03-28T17:47:38.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.852+0000 7fb7df400700 1 --2- 172.21.15.45:0/3365445051 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb7c00a2910 0x7fb7c00a4dd0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:38.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.852+0000 7fb7df400700 1 --2- 172.21.15.45:0/3365445051 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7d81087a0 0x7fb7d819ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:38.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.852+0000 7fb7df400700 1 --2- 172.21.15.45:0/3365445051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7d810e7a0 0x7fb7d81a3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:38.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.852+0000 7fb7df400700 1 --2- 172.21.15.45:0/3365445051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7d8109160 0x7fb7d819f470 secure :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb7c400d980 tx=0x7fb7c400dc90 comp rx=0 tx=0).stop 2024-03-28T17:47:38.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.852+0000 7fb7df400700 1 -- 172.21.15.45:0/3365445051 >> 172.21.15.45:0/3365445051 conn(0x7fb7d8075960 msgr2=0x7fb7d8075ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:38.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.852+0000 7fb7df400700 1 -- 172.21.15.45:0/3365445051 shutdown_connections 2024-03-28T17:47:38.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:38.852+0000 7fb7df400700 1 -- 172.21.15.45:0/3365445051 wait complete. 2024-03-28T17:47:39.509 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332140 got 219043332148 for osd.7 2024-03-28T17:47:39.509 DEBUG:teuthology.parallel:result is None 2024-03-28T17:47:39.643 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:39 smithi045 ceph-mon[31773]: pgmap v362: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:47:39.643 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:39 smithi045 ceph-mon[31773]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-28T17:47:39.643 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:39 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3365445051' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T17:47:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:39 smithi071 ceph-mon[36597]: pgmap v362: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:47:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:39 smithi071 ceph-mon[36597]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-28T17:47:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:39 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3365445051' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T17:47:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:39 smithi151 ceph-mon[36481]: pgmap v362: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:47:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:39 smithi151 ceph-mon[36481]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-28T17:47:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:39 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3365445051' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T17:47:40.065 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383706 got 343597383712 for osd.11 2024-03-28T17:47:40.065 DEBUG:teuthology.parallel:result is None 2024-03-28T17:47:40.065 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-28T17:47:40.065 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T17:47:40.526 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:47:41.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:41 smithi045 ceph-mon[31773]: pgmap v363: 97 pgs: 97 active+clean; 641 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T17:47:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:41 smithi071 ceph-mon[36597]: pgmap v363: 97 pgs: 97 active+clean; 641 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T17:47:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:41 smithi151 ceph-mon[36481]: pgmap v363: 97 pgs: 97 active+clean; 641 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T17:47:42.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.749+0000 7f4e65800700 1 -- 172.21.15.45:0/2778643009 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e601074f0 msgr2=0x7f4e6010f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:42.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.750+0000 7f4e65800700 1 --2- 172.21.15.45:0/2778643009 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e601074f0 0x7f4e6010f0c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4e5800b0d0 tx=0x7f4e5800b3e0 comp rx=0 tx=0).stop 2024-03-28T17:47:42.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.750+0000 7f4e65800700 1 -- 172.21.15.45:0/2778643009 shutdown_connections 2024-03-28T17:47:42.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.750+0000 7f4e65800700 1 --2- 172.21.15.45:0/2778643009 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e601074f0 0x7f4e6010f0c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:42.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.750+0000 7f4e65800700 1 --2- 172.21.15.45:0/2778643009 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e60102410 0x7f4e601027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:42.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.750+0000 7f4e65800700 1 --2- 172.21.15.45:0/2778643009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e60102dc0 0x7f4e60106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:42.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.750+0000 7f4e65800700 1 -- 172.21.15.45:0/2778643009 >> 172.21.15.45:0/2778643009 conn(0x7f4e600fdc60 msgr2=0x7f4e60100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:42.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.750+0000 7f4e65800700 1 -- 172.21.15.45:0/2778643009 shutdown_connections 2024-03-28T17:47:42.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.750+0000 7f4e65800700 1 -- 172.21.15.45:0/2778643009 wait complete. 2024-03-28T17:47:42.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.751+0000 7f4e65800700 1 Processor -- start 2024-03-28T17:47:42.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.751+0000 7f4e65800700 1 -- start start 2024-03-28T17:47:42.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.751+0000 7f4e65800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e60102410 0x7f4e601a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:42.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.751+0000 7f4e65800700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e60102dc0 0x7f4e601a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:42.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.752+0000 7f4e65800700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e601074f0 0x7f4e6019a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:42.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.752+0000 7f4e65800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e60112530 con 0x7f4e60102410 2024-03-28T17:47:42.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.752+0000 7f4e65800700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e601123b0 con 0x7f4e601074f0 2024-03-28T17:47:42.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.752+0000 7f4e65800700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e601126b0 con 0x7f4e60102dc0 2024-03-28T17:47:42.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.752+0000 7f4e5ea00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e60102410 0x7f4e601a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:42.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.752+0000 7f4e5e000700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e60102dc0 0x7f4e601a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:42.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.752+0000 7f4e5e000700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e60102dc0 0x7f4e601a0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:35960/0 (socket says 172.21.15.45:35960) 2024-03-28T17:47:42.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.752+0000 7f4e5e000700 1 -- 172.21.15.45:0/2175739703 learned_addr learned my addr 172.21.15.45:0/2175739703 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:42.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.752+0000 7f4e5f400700 1 --2- 172.21.15.45:0/2175739703 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e601074f0 0x7f4e6019a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:42.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.752+0000 7f4e5e000700 1 -- 172.21.15.45:0/2175739703 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e601074f0 msgr2=0x7f4e6019a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:42.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.752+0000 7f4e5e000700 1 --2- 172.21.15.45:0/2175739703 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e601074f0 0x7f4e6019a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:42.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.752+0000 7f4e5e000700 1 -- 172.21.15.45:0/2175739703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e60102410 msgr2=0x7f4e601a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:42.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.752+0000 7f4e5e000700 1 --2- 172.21.15.45:0/2175739703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e60102410 0x7f4e601a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:42.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.752+0000 7f4e5e000700 1 -- 172.21.15.45:0/2175739703 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6019b070 con 0x7f4e60102dc0 2024-03-28T17:47:42.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.753+0000 7f4e5e000700 1 --2- 172.21.15.45:0/2175739703 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e60102dc0 0x7f4e601a0c90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c00e970 tx=0x7f4e4c00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:42.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.753+0000 7f4e57400700 1 -- 172.21.15.45:0/2175739703 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c004a30 con 0x7f4e60102dc0 2024-03-28T17:47:42.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.753+0000 7f4e57400700 1 -- 172.21.15.45:0/2175739703 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e4c00ccc0 con 0x7f4e60102dc0 2024-03-28T17:47:42.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.753+0000 7f4e57400700 1 -- 172.21.15.45:0/2175739703 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c005650 con 0x7f4e60102dc0 2024-03-28T17:47:42.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.753+0000 7f4e65800700 1 -- 172.21.15.45:0/2175739703 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e6019b360 con 0x7f4e60102dc0 2024-03-28T17:47:42.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.753+0000 7f4e65800700 1 -- 172.21.15.45:0/2175739703 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e601a7620 con 0x7f4e60102dc0 2024-03-28T17:47:42.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.754+0000 7f4e65800700 1 -- 172.21.15.45:0/2175739703 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e60068e50 con 0x7f4e60102dc0 2024-03-28T17:47:42.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.756+0000 7f4e57400700 1 -- 172.21.15.45:0/2175739703 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4e4c00c7e0 con 0x7f4e60102dc0 2024-03-28T17:47:42.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.757+0000 7f4e57400700 1 --2- 172.21.15.45:0/2175739703 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4e440a2910 0x7f4e440a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:42.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.757+0000 7f4e57400700 1 -- 172.21.15.45:0/2175739703 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f4e4c0ced60 con 0x7f4e60102dc0 2024-03-28T17:47:42.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.757+0000 7f4e5ea00700 1 --2- 172.21.15.45:0/2175739703 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4e440a2910 0x7f4e440a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:42.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.758+0000 7f4e5ea00700 1 --2- 172.21.15.45:0/2175739703 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4e440a2910 0x7f4e440a4dd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4e6019ba50 tx=0x7f4e50009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:42.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.761+0000 7f4e57400700 1 -- 172.21.15.45:0/2175739703 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4e4c09bdb0 con 0x7f4e60102dc0 2024-03-28T17:47:42.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.987+0000 7f4e65800700 1 -- 172.21.15.45:0/2175739703 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4e600ff500 con 0x7f4e440a2910 2024-03-28T17:47:42.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:42.995+0000 7f4e57400700 1 -- 172.21.15.45:0/2175739703 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259158 (secure 0 0 0) 0x7f4e600ff500 con 0x7f4e440a2910 2024-03-28T17:47:42.996 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:47:43.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:43.003+0000 7f4e65800700 1 -- 172.21.15.45:0/2175739703 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4e440a2910 msgr2=0x7f4e440a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:43.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:43.003+0000 7f4e65800700 1 --2- 172.21.15.45:0/2175739703 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4e440a2910 0x7f4e440a4dd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4e6019ba50 tx=0x7f4e50009260 comp rx=0 tx=0).stop 2024-03-28T17:47:43.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:43.003+0000 7f4e65800700 1 -- 172.21.15.45:0/2175739703 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e60102dc0 msgr2=0x7f4e601a0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:43.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:43.003+0000 7f4e65800700 1 --2- 172.21.15.45:0/2175739703 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e60102dc0 0x7f4e601a0c90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c00e970 tx=0x7f4e4c00ec80 comp rx=0 tx=0).stop 2024-03-28T17:47:43.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:43.003+0000 7f4e65800700 1 -- 172.21.15.45:0/2175739703 shutdown_connections 2024-03-28T17:47:43.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:43.003+0000 7f4e65800700 1 --2- 172.21.15.45:0/2175739703 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4e440a2910 0x7f4e440a4dd0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:43.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:43.003+0000 7f4e65800700 1 --2- 172.21.15.45:0/2175739703 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e601074f0 0x7f4e6019a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:43.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:43.003+0000 7f4e65800700 1 --2- 172.21.15.45:0/2175739703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4e60102410 0x7f4e601a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:43.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:43.003+0000 7f4e65800700 1 --2- 172.21.15.45:0/2175739703 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e60102dc0 0x7f4e601a0c90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:43.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:43.003+0000 7f4e65800700 1 -- 172.21.15.45:0/2175739703 >> 172.21.15.45:0/2175739703 conn(0x7f4e600fdc60 msgr2=0x7f4e6010d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:43.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:43.004+0000 7f4e65800700 1 -- 172.21.15.45:0/2175739703 shutdown_connections 2024-03-28T17:47:43.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:43.004+0000 7f4e65800700 1 -- 172.21.15.45:0/2175739703 wait complete. 2024-03-28T17:47:43.006 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-28T17:47:43.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:43 smithi045 ceph-mon[31773]: pgmap v364: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T17:47:43.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:43 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:47:43.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:43 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:47:43.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:43 smithi071 ceph-mon[36597]: pgmap v364: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T17:47:43.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:43 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:47:43.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:43 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:47:43.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:43 smithi151 ceph-mon[36481]: pgmap v364: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T17:47:43.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:43 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:47:43.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:43 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:47:43.836 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-03-28T17:47:41.682510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656149,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":653,"num_read_kb":447,"num_write":349,"num_write_kb":3055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180853,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":749,"ondisk_log_size":749,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":364880,"kb_used_data":11556,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576944,"statfs":{"total":1151940427776,"available":1151566790656,"internally_reserved":0,"allocated":11833344,"data_stored":7436184,"data_compressed":114219,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":55,"apply_latency_ms":55,"commit_latency_ns":55000000,"apply_latency_ns":55000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":282,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":67,"num_read_kb":32,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005046"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683759+0000","last_change":"2024-03-28T17:46:12.783038+0000","last_active":"2024-03-28T17:47:37.683759+0000","last_peered":"2024-03-28T17:47:37.683759+0000","last_clean":"2024-03-28T17:47:37.683759+0000","last_became_active":"2024-03-28T17:46:12.782668+0000","last_became_peered":"2024-03-28T17:46:12.782668+0000","last_unstale":"2024-03-28T17:47:37.683759+0000","last_undegraded":"2024-03-28T17:47:37.683759+0000","last_fullsized":"2024-03-28T17:47:37.683759+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:03.424959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683400+0000","last_change":"2024-03-28T17:46:12.781828+0000","last_active":"2024-03-28T17:47:37.683400+0000","last_peered":"2024-03-28T17:47:37.683400+0000","last_clean":"2024-03-28T17:47:37.683400+0000","last_became_active":"2024-03-28T17:46:12.781424+0000","last_became_peered":"2024-03-28T17:46:12.781424+0000","last_unstale":"2024-03-28T17:47:37.683400+0000","last_undegraded":"2024-03-28T17:47:37.683400+0000","last_fullsized":"2024-03-28T17:47:37.683400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:15:16.639473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683172+0000","last_change":"2024-03-28T17:46:12.785431+0000","last_active":"2024-03-28T17:47:37.683172+0000","last_peered":"2024-03-28T17:47:37.683172+0000","last_clean":"2024-03-28T17:47:37.683172+0000","last_became_active":"2024-03-28T17:46:12.784992+0000","last_became_peered":"2024-03-28T17:46:12.784992+0000","last_unstale":"2024-03-28T17:47:37.683172+0000","last_undegraded":"2024-03-28T17:47:37.683172+0000","last_fullsized":"2024-03-28T17:47:37.683172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:03:36.926613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.662861+0000","last_change":"2024-03-28T17:46:12.765274+0000","last_active":"2024-03-28T17:47:37.662861+0000","last_peered":"2024-03-28T17:47:37.662861+0000","last_clean":"2024-03-28T17:47:37.662861+0000","last_became_active":"2024-03-28T17:46:12.764890+0000","last_became_peered":"2024-03-28T17:46:12.764890+0000","last_unstale":"2024-03-28T17:47:37.662861+0000","last_undegraded":"2024-03-28T17:47:37.662861+0000","last_fullsized":"2024-03-28T17:47:37.662861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:58:37.149756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.670408+0000","last_change":"2024-03-28T17:46:12.782791+0000","last_active":"2024-03-28T17:47:37.670408+0000","last_peered":"2024-03-28T17:47:37.670408+0000","last_clean":"2024-03-28T17:47:37.670408+0000","last_became_active":"2024-03-28T17:46:12.782420+0000","last_became_peered":"2024-03-28T17:46:12.782420+0000","last_unstale":"2024-03-28T17:47:37.670408+0000","last_undegraded":"2024-03-28T17:47:37.670408+0000","last_fullsized":"2024-03-28T17:47:37.670408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:30.512863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.950584+0000","last_change":"2024-03-28T17:46:12.780713+0000","last_active":"2024-03-28T17:47:09.950584+0000","last_peered":"2024-03-28T17:47:09.950584+0000","last_clean":"2024-03-28T17:47:09.950584+0000","last_became_active":"2024-03-28T17:46:12.780305+0000","last_became_peered":"2024-03-28T17:46:12.780305+0000","last_unstale":"2024-03-28T17:47:09.950584+0000","last_undegraded":"2024-03-28T17:47:09.950584+0000","last_fullsized":"2024-03-28T17:47:09.950584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:08:09.840491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.653070+0000","last_change":"2024-03-28T17:46:12.778120+0000","last_active":"2024-03-28T17:47:37.653070+0000","last_peered":"2024-03-28T17:47:37.653070+0000","last_clean":"2024-03-28T17:47:37.653070+0000","last_became_active":"2024-03-28T17:46:12.777724+0000","last_became_peered":"2024-03-28T17:46:12.777724+0000","last_unstale":"2024-03-28T17:47:37.653070+0000","last_undegraded":"2024-03-28T17:47:37.653070+0000","last_fullsized":"2024-03-28T17:47:37.653070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:29:50.980366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687801+0000","last_change":"2024-03-28T17:46:12.793447+0000","last_active":"2024-03-28T17:47:37.687801+0000","last_peered":"2024-03-28T17:47:37.687801+0000","last_clean":"2024-03-28T17:47:37.687801+0000","last_became_active":"2024-03-28T17:46:12.793070+0000","last_became_peered":"2024-03-28T17:46:12.793070+0000","last_unstale":"2024-03-28T17:47:37.687801+0000","last_undegraded":"2024-03-28T17:47:37.687801+0000","last_fullsized":"2024-03-28T17:47:37.687801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:01.364577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.668799+0000","last_change":"2024-03-28T17:46:12.763908+0000","last_active":"2024-03-28T17:47:37.668799+0000","last_peered":"2024-03-28T17:47:37.668799+0000","last_clean":"2024-03-28T17:47:37.668799+0000","last_became_active":"2024-03-28T17:46:12.763267+0000","last_became_peered":"2024-03-28T17:46:12.763267+0000","last_unstale":"2024-03-28T17:47:37.668799+0000","last_undegraded":"2024-03-28T17:47:37.668799+0000","last_fullsized":"2024-03-28T17:47:37.668799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:10.692974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.950940+0000","last_change":"2024-03-28T17:46:12.762648+0000","last_active":"2024-03-28T17:47:09.950940+0000","last_peered":"2024-03-28T17:47:09.950940+0000","last_clean":"2024-03-28T17:47:09.950940+0000","last_became_active":"2024-03-28T17:46:12.762300+0000","last_became_peered":"2024-03-28T17:46:12.762300+0000","last_unstale":"2024-03-28T17:47:09.950940+0000","last_undegraded":"2024-03-28T17:47:09.950940+0000","last_fullsized":"2024-03-28T17:47:09.950940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:02.210828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.665668+0000","last_change":"2024-03-28T17:46:12.766482+0000","last_active":"2024-03-28T17:47:37.665668+0000","last_peered":"2024-03-28T17:47:37.665668+0000","last_clean":"2024-03-28T17:47:37.665668+0000","last_became_active":"2024-03-28T17:46:12.766155+0000","last_became_peered":"2024-03-28T17:46:12.766155+0000","last_unstale":"2024-03-28T17:47:37.665668+0000","last_undegraded":"2024-03-28T17:47:37.665668+0000","last_fullsized":"2024-03-28T17:47:37.665668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:46:24.261744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.642911+0000","last_change":"2024-03-28T17:46:12.766100+0000","last_active":"2024-03-28T17:47:37.642911+0000","last_peered":"2024-03-28T17:47:37.642911+0000","last_clean":"2024-03-28T17:47:37.642911+0000","last_became_active":"2024-03-28T17:46:12.765762+0000","last_became_peered":"2024-03-28T17:46:12.765762+0000","last_unstale":"2024-03-28T17:47:37.642911+0000","last_undegraded":"2024-03-28T17:47:37.642911+0000","last_fullsized":"2024-03-28T17:47:37.642911+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:11.666653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.669912+0000","last_change":"2024-03-28T17:46:12.753465+0000","last_active":"2024-03-28T17:47:37.669912+0000","last_peered":"2024-03-28T17:47:37.669912+0000","last_clean":"2024-03-28T17:47:37.669912+0000","last_became_active":"2024-03-28T17:46:12.751931+0000","last_became_peered":"2024-03-28T17:46:12.751931+0000","last_unstale":"2024-03-28T17:47:37.669912+0000","last_undegraded":"2024-03-28T17:47:37.669912+0000","last_fullsized":"2024-03-28T17:47:37.669912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:39:38.757878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.684424+0000","last_change":"2024-03-28T17:46:12.784177+0000","last_active":"2024-03-28T17:47:37.684424+0000","last_peered":"2024-03-28T17:47:37.684424+0000","last_clean":"2024-03-28T17:47:37.684424+0000","last_became_active":"2024-03-28T17:46:12.783823+0000","last_became_peered":"2024-03-28T17:46:12.783823+0000","last_unstale":"2024-03-28T17:47:37.684424+0000","last_undegraded":"2024-03-28T17:47:37.684424+0000","last_fullsized":"2024-03-28T17:47:37.684424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:11:33.221095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685100+0000","last_change":"2024-03-28T17:46:12.777152+0000","last_active":"2024-03-28T17:47:37.685100+0000","last_peered":"2024-03-28T17:47:37.685100+0000","last_clean":"2024-03-28T17:47:37.685100+0000","last_became_active":"2024-03-28T17:46:12.776758+0000","last_became_peered":"2024-03-28T17:46:12.776758+0000","last_unstale":"2024-03-28T17:47:37.685100+0000","last_undegraded":"2024-03-28T17:47:37.685100+0000","last_fullsized":"2024-03-28T17:47:37.685100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:20:56.587470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.686198+0000","last_change":"2024-03-28T17:46:12.782384+0000","last_active":"2024-03-28T17:47:37.686198+0000","last_peered":"2024-03-28T17:47:37.686198+0000","last_clean":"2024-03-28T17:47:37.686198+0000","last_became_active":"2024-03-28T17:46:12.781971+0000","last_became_peered":"2024-03-28T17:46:12.781971+0000","last_unstale":"2024-03-28T17:47:37.686198+0000","last_undegraded":"2024-03-28T17:47:37.686198+0000","last_fullsized":"2024-03-28T17:47:37.686198+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:19:32.652629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685817+0000","last_change":"2024-03-28T17:46:12.789364+0000","last_active":"2024-03-28T17:47:37.685817+0000","last_peered":"2024-03-28T17:47:37.685817+0000","last_clean":"2024-03-28T17:47:37.685817+0000","last_became_active":"2024-03-28T17:46:12.788976+0000","last_became_peered":"2024-03-28T17:46:12.788976+0000","last_unstale":"2024-03-28T17:47:37.685817+0000","last_undegraded":"2024-03-28T17:47:37.685817+0000","last_fullsized":"2024-03-28T17:47:37.685817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:43:37.183705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.663470+0000","last_change":"2024-03-28T17:46:12.782664+0000","last_active":"2024-03-28T17:47:37.663470+0000","last_peered":"2024-03-28T17:47:37.663470+0000","last_clean":"2024-03-28T17:47:37.663470+0000","last_became_active":"2024-03-28T17:46:12.782305+0000","last_became_peered":"2024-03-28T17:46:12.782305+0000","last_unstale":"2024-03-28T17:47:37.663470+0000","last_undegraded":"2024-03-28T17:47:37.663470+0000","last_fullsized":"2024-03-28T17:47:37.663470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:47:18.373517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.684298+0000","last_change":"2024-03-28T17:46:12.794494+0000","last_active":"2024-03-28T17:47:37.684298+0000","last_peered":"2024-03-28T17:47:37.684298+0000","last_clean":"2024-03-28T17:47:37.684298+0000","last_became_active":"2024-03-28T17:46:12.794151+0000","last_became_peered":"2024-03-28T17:46:12.794151+0000","last_unstale":"2024-03-28T17:47:37.684298+0000","last_undegraded":"2024-03-28T17:47:37.684298+0000","last_fullsized":"2024-03-28T17:47:37.684298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:17:24.596333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.475712+0000","last_change":"2024-03-28T17:46:11.759643+0000","last_active":"2024-03-28T17:47:09.475712+0000","last_peered":"2024-03-28T17:47:09.475712+0000","last_clean":"2024-03-28T17:47:09.475712+0000","last_became_active":"2024-03-28T17:46:11.759273+0000","last_became_peered":"2024-03-28T17:46:11.759273+0000","last_unstale":"2024-03-28T17:47:09.475712+0000","last_undegraded":"2024-03-28T17:47:09.475712+0000","last_fullsized":"2024-03-28T17:47:09.475712+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:09:51.764471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.681633+0000","last_change":"2024-03-28T17:46:12.786503+0000","last_active":"2024-03-28T17:47:37.681633+0000","last_peered":"2024-03-28T17:47:37.681633+0000","last_clean":"2024-03-28T17:47:37.681633+0000","last_became_active":"2024-03-28T17:46:12.784651+0000","last_became_peered":"2024-03-28T17:46:12.784651+0000","last_unstale":"2024-03-28T17:47:37.681633+0000","last_undegraded":"2024-03-28T17:47:37.681633+0000","last_fullsized":"2024-03-28T17:47:37.681633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:30:22.401767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.480673+0000","last_change":"2024-03-28T17:46:11.780340+0000","last_active":"2024-03-28T17:47:09.480673+0000","last_peered":"2024-03-28T17:47:09.480673+0000","last_clean":"2024-03-28T17:47:09.480673+0000","last_became_active":"2024-03-28T17:46:11.779917+0000","last_became_peered":"2024-03-28T17:46:11.779917+0000","last_unstale":"2024-03-28T17:47:09.480673+0000","last_undegraded":"2024-03-28T17:47:09.480673+0000","last_fullsized":"2024-03-28T17:47:09.480673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:22:45.617444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685946+0000","last_change":"2024-03-28T17:46:12.795086+0000","last_active":"2024-03-28T17:47:37.685946+0000","last_peered":"2024-03-28T17:47:37.685946+0000","last_clean":"2024-03-28T17:47:37.685946+0000","last_became_active":"2024-03-28T17:46:12.794124+0000","last_became_peered":"2024-03-28T17:46:12.794124+0000","last_unstale":"2024-03-28T17:47:37.685946+0000","last_undegraded":"2024-03-28T17:47:37.685946+0000","last_fullsized":"2024-03-28T17:47:37.685946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:11:05.347047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.648328+0000","last_change":"2024-03-28T17:47:09.648328+0000","last_active":"2024-03-28T17:47:09.648328+0000","last_peered":"2024-03-28T17:47:09.648328+0000","last_clean":"2024-03-28T17:47:09.648328+0000","last_became_active":"2024-03-28T17:47:09.644797+0000","last_became_peered":"2024-03-28T17:47:09.644797+0000","last_unstale":"2024-03-28T17:47:09.648328+0000","last_undegraded":"2024-03-28T17:47:09.648328+0000","last_fullsized":"2024-03-28T17:47:09.648328+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:14:30.233156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.665594+0000","last_change":"2024-03-28T17:46:12.768118+0000","last_active":"2024-03-28T17:47:37.665594+0000","last_peered":"2024-03-28T17:47:37.665594+0000","last_clean":"2024-03-28T17:47:37.665594+0000","last_became_active":"2024-03-28T17:46:12.767765+0000","last_became_peered":"2024-03-28T17:46:12.767765+0000","last_unstale":"2024-03-28T17:47:37.665594+0000","last_undegraded":"2024-03-28T17:47:37.665594+0000","last_fullsized":"2024-03-28T17:47:37.665594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:16:12.268968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461621+0000","last_change":"2024-03-28T17:46:11.760204+0000","last_active":"2024-03-28T17:47:09.461621+0000","last_peered":"2024-03-28T17:47:09.461621+0000","last_clean":"2024-03-28T17:47:09.461621+0000","last_became_active":"2024-03-28T17:46:11.759829+0000","last_became_peered":"2024-03-28T17:46:11.759829+0000","last_unstale":"2024-03-28T17:47:09.461621+0000","last_undegraded":"2024-03-28T17:47:09.461621+0000","last_fullsized":"2024-03-28T17:47:09.461621+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:29:02.075424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.663263+0000","last_change":"2024-03-28T17:46:12.762229+0000","last_active":"2024-03-28T17:47:37.663263+0000","last_peered":"2024-03-28T17:47:37.663263+0000","last_clean":"2024-03-28T17:47:37.663263+0000","last_became_active":"2024-03-28T17:46:12.761889+0000","last_became_peered":"2024-03-28T17:46:12.761889+0000","last_unstale":"2024-03-28T17:47:37.663263+0000","last_undegraded":"2024-03-28T17:47:37.663263+0000","last_fullsized":"2024-03-28T17:47:37.663263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:15.023966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'7","reported_seq":112,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:38.059415+0000","last_change":"2024-03-28T17:46:11.763152+0000","last_active":"2024-03-28T17:47:38.059415+0000","last_peered":"2024-03-28T17:47:38.059415+0000","last_clean":"2024-03-28T17:47:38.059415+0000","last_became_active":"2024-03-28T17:46:11.762754+0000","last_became_peered":"2024-03-28T17:46:11.762754+0000","last_unstale":"2024-03-28T17:47:38.059415+0000","last_undegraded":"2024-03-28T17:47:38.059415+0000","last_fullsized":"2024-03-28T17:47:38.059415+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:17:06.659908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.649470+0000","last_change":"2024-03-28T17:46:12.780701+0000","last_active":"2024-03-28T17:47:37.649470+0000","last_peered":"2024-03-28T17:47:37.649470+0000","last_clean":"2024-03-28T17:47:37.649470+0000","last_became_active":"2024-03-28T17:46:12.780339+0000","last_became_peered":"2024-03-28T17:46:12.780339+0000","last_unstale":"2024-03-28T17:47:37.649470+0000","last_undegraded":"2024-03-28T17:47:37.649470+0000","last_fullsized":"2024-03-28T17:47:37.649470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:59.774888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.950680+0000","last_change":"2024-03-28T17:46:11.763387+0000","last_active":"2024-03-28T17:47:09.950680+0000","last_peered":"2024-03-28T17:47:09.950680+0000","last_clean":"2024-03-28T17:47:09.950680+0000","last_became_active":"2024-03-28T17:46:11.763028+0000","last_became_peered":"2024-03-28T17:46:11.763028+0000","last_unstale":"2024-03-28T17:47:09.950680+0000","last_undegraded":"2024-03-28T17:47:09.950680+0000","last_fullsized":"2024-03-28T17:47:09.950680+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:21:31.585047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685166+0000","last_change":"2024-03-28T17:46:12.794167+0000","last_active":"2024-03-28T17:47:37.685166+0000","last_peered":"2024-03-28T17:47:37.685166+0000","last_clean":"2024-03-28T17:47:37.685166+0000","last_became_active":"2024-03-28T17:46:12.793830+0000","last_became_peered":"2024-03-28T17:46:12.793830+0000","last_unstale":"2024-03-28T17:47:37.685166+0000","last_undegraded":"2024-03-28T17:47:37.685166+0000","last_fullsized":"2024-03-28T17:47:37.685166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:20.907752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.470172+0000","last_change":"2024-03-28T17:46:11.769322+0000","last_active":"2024-03-28T17:47:09.470172+0000","last_peered":"2024-03-28T17:47:09.470172+0000","last_clean":"2024-03-28T17:47:09.470172+0000","last_became_active":"2024-03-28T17:46:11.768931+0000","last_became_peered":"2024-03-28T17:46:11.768931+0000","last_unstale":"2024-03-28T17:47:09.470172+0000","last_undegraded":"2024-03-28T17:47:09.470172+0000","last_fullsized":"2024-03-28T17:47:09.470172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:29:34.372947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683329+0000","last_change":"2024-03-28T17:46:12.781306+0000","last_active":"2024-03-28T17:47:37.683329+0000","last_peered":"2024-03-28T17:47:37.683329+0000","last_clean":"2024-03-28T17:47:37.683329+0000","last_became_active":"2024-03-28T17:46:12.779161+0000","last_became_peered":"2024-03-28T17:46:12.779161+0000","last_unstale":"2024-03-28T17:47:37.683329+0000","last_undegraded":"2024-03-28T17:47:37.683329+0000","last_fullsized":"2024-03-28T17:47:37.683329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:06:10.302067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.464702+0000","last_change":"2024-03-28T17:46:11.762852+0000","last_active":"2024-03-28T17:47:09.464702+0000","last_peered":"2024-03-28T17:47:09.464702+0000","last_clean":"2024-03-28T17:47:09.464702+0000","last_became_active":"2024-03-28T17:46:11.762428+0000","last_became_peered":"2024-03-28T17:46:11.762428+0000","last_unstale":"2024-03-28T17:47:09.464702+0000","last_undegraded":"2024-03-28T17:47:09.464702+0000","last_fullsized":"2024-03-28T17:47:09.464702+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:38:20.433408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685573+0000","last_change":"2024-03-28T17:46:12.786295+0000","last_active":"2024-03-28T17:47:37.685573+0000","last_peered":"2024-03-28T17:47:37.685573+0000","last_clean":"2024-03-28T17:47:37.685573+0000","last_became_active":"2024-03-28T17:46:12.785894+0000","last_became_peered":"2024-03-28T17:46:12.785894+0000","last_unstale":"2024-03-28T17:47:37.685573+0000","last_undegraded":"2024-03-28T17:47:37.685573+0000","last_fullsized":"2024-03-28T17:47:37.685573+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:06.271660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":102,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:36.056344+0000","last_change":"2024-03-28T17:46:11.766787+0000","last_active":"2024-03-28T17:47:36.056344+0000","last_peered":"2024-03-28T17:47:36.056344+0000","last_clean":"2024-03-28T17:47:36.056344+0000","last_became_active":"2024-03-28T17:46:11.766352+0000","last_became_peered":"2024-03-28T17:46:11.766352+0000","last_unstale":"2024-03-28T17:47:36.056344+0000","last_undegraded":"2024-03-28T17:47:36.056344+0000","last_fullsized":"2024-03-28T17:47:36.056344+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:54:48.157597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":80,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.666145+0000","last_change":"2024-03-28T17:46:12.785724+0000","last_active":"2024-03-28T17:47:37.666145+0000","last_peered":"2024-03-28T17:47:37.666145+0000","last_clean":"2024-03-28T17:47:37.666145+0000","last_became_active":"2024-03-28T17:46:12.785353+0000","last_became_peered":"2024-03-28T17:46:12.785353+0000","last_unstale":"2024-03-28T17:47:37.666145+0000","last_undegraded":"2024-03-28T17:47:37.666145+0000","last_fullsized":"2024-03-28T17:47:37.666145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:21:22.692899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.463079+0000","last_change":"2024-03-28T17:46:11.763507+0000","last_active":"2024-03-28T17:47:09.463079+0000","last_peered":"2024-03-28T17:47:09.463079+0000","last_clean":"2024-03-28T17:47:09.463079+0000","last_became_active":"2024-03-28T17:46:11.763177+0000","last_became_peered":"2024-03-28T17:46:11.763177+0000","last_unstale":"2024-03-28T17:47:09.463079+0000","last_undegraded":"2024-03-28T17:47:09.463079+0000","last_fullsized":"2024-03-28T17:47:09.463079+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:56:39.696699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685855+0000","last_change":"2024-03-28T17:47:09.479162+0000","last_active":"2024-03-28T17:47:37.685855+0000","last_peered":"2024-03-28T17:47:37.685855+0000","last_clean":"2024-03-28T17:47:37.685855+0000","last_became_active":"2024-03-28T17:47:09.478349+0000","last_became_peered":"2024-03-28T17:47:09.478349+0000","last_unstale":"2024-03-28T17:47:37.685855+0000","last_undegraded":"2024-03-28T17:47:37.685855+0000","last_fullsized":"2024-03-28T17:47:37.685855+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:41.056361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.465137+0000","last_change":"2024-03-28T17:46:11.774961+0000","last_active":"2024-03-28T17:47:09.465137+0000","last_peered":"2024-03-28T17:47:09.465137+0000","last_clean":"2024-03-28T17:47:09.465137+0000","last_became_active":"2024-03-28T17:46:11.774645+0000","last_became_peered":"2024-03-28T17:46:11.774645+0000","last_unstale":"2024-03-28T17:47:09.465137+0000","last_undegraded":"2024-03-28T17:47:09.465137+0000","last_fullsized":"2024-03-28T17:47:09.465137+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:38:23.983739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.667399+0000","last_change":"2024-03-28T17:46:12.763698+0000","last_active":"2024-03-28T17:47:37.667399+0000","last_peered":"2024-03-28T17:47:37.667399+0000","last_clean":"2024-03-28T17:47:37.667399+0000","last_became_active":"2024-03-28T17:46:12.763238+0000","last_became_peered":"2024-03-28T17:46:12.763238+0000","last_unstale":"2024-03-28T17:47:37.667399+0000","last_undegraded":"2024-03-28T17:47:37.667399+0000","last_fullsized":"2024-03-28T17:47:37.667399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:12:58.252855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'30","reported_seq":218,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.062773+0000","last_change":"2024-03-28T17:46:11.763096+0000","last_active":"2024-03-28T17:47:37.062773+0000","last_peered":"2024-03-28T17:47:37.062773+0000","last_clean":"2024-03-28T17:47:37.062773+0000","last_became_active":"2024-03-28T17:46:11.762730+0000","last_became_peered":"2024-03-28T17:46:11.762730+0000","last_unstale":"2024-03-28T17:47:37.062773+0000","last_undegraded":"2024-03-28T17:47:37.062773+0000","last_fullsized":"2024-03-28T17:47:37.062773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:45:59.469096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60873,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":169,"num_read_kb":144,"num_write":31,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.651310+0000","last_change":"2024-03-28T17:46:12.760366+0000","last_active":"2024-03-28T17:47:37.651310+0000","last_peered":"2024-03-28T17:47:37.651310+0000","last_clean":"2024-03-28T17:47:37.651310+0000","last_became_active":"2024-03-28T17:46:12.759957+0000","last_became_peered":"2024-03-28T17:46:12.759957+0000","last_unstale":"2024-03-28T17:47:37.651310+0000","last_undegraded":"2024-03-28T17:47:37.651310+0000","last_fullsized":"2024-03-28T17:47:37.651310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:08:02.579659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.463494+0000","last_change":"2024-03-28T17:46:11.787108+0000","last_active":"2024-03-28T17:47:09.463494+0000","last_peered":"2024-03-28T17:47:09.463494+0000","last_clean":"2024-03-28T17:47:09.463494+0000","last_became_active":"2024-03-28T17:46:11.786734+0000","last_became_peered":"2024-03-28T17:46:11.786734+0000","last_unstale":"2024-03-28T17:47:09.463494+0000","last_undegraded":"2024-03-28T17:47:09.463494+0000","last_fullsized":"2024-03-28T17:47:09.463494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:04:00.480545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.678596+0000","last_change":"2024-03-28T17:46:12.786362+0000","last_active":"2024-03-28T17:47:37.678596+0000","last_peered":"2024-03-28T17:47:37.678596+0000","last_clean":"2024-03-28T17:47:37.678596+0000","last_became_active":"2024-03-28T17:46:12.785946+0000","last_became_peered":"2024-03-28T17:46:12.785946+0000","last_unstale":"2024-03-28T17:47:37.678596+0000","last_undegraded":"2024-03-28T17:47:37.678596+0000","last_fullsized":"2024-03-28T17:47:37.678596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:13:57.857818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.464833+0000","last_change":"2024-03-28T17:46:11.786292+0000","last_active":"2024-03-28T17:47:09.464833+0000","last_peered":"2024-03-28T17:47:09.464833+0000","last_clean":"2024-03-28T17:47:09.464833+0000","last_became_active":"2024-03-28T17:46:11.785953+0000","last_became_peered":"2024-03-28T17:46:11.785953+0000","last_unstale":"2024-03-28T17:47:09.464833+0000","last_undegraded":"2024-03-28T17:47:09.464833+0000","last_fullsized":"2024-03-28T17:47:09.464833+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:02:18.093585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.669772+0000","last_change":"2024-03-28T17:46:12.770669+0000","last_active":"2024-03-28T17:47:37.669772+0000","last_peered":"2024-03-28T17:47:37.669772+0000","last_clean":"2024-03-28T17:47:37.669772+0000","last_became_active":"2024-03-28T17:46:12.770301+0000","last_became_peered":"2024-03-28T17:46:12.770301+0000","last_unstale":"2024-03-28T17:47:37.669772+0000","last_undegraded":"2024-03-28T17:47:37.669772+0000","last_fullsized":"2024-03-28T17:47:37.669772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:59:00.601357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'5","reported_seq":110,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:38.112253+0000","last_change":"2024-03-28T17:46:11.759686+0000","last_active":"2024-03-28T17:47:38.112253+0000","last_peered":"2024-03-28T17:47:38.112253+0000","last_clean":"2024-03-28T17:47:38.112253+0000","last_became_active":"2024-03-28T17:46:11.759309+0000","last_became_peered":"2024-03-28T17:46:11.759309+0000","last_unstale":"2024-03-28T17:47:38.112253+0000","last_undegraded":"2024-03-28T17:47:38.112253+0000","last_fullsized":"2024-03-28T17:47:38.112253+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:31:09.999279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.652849+0000","last_change":"2024-03-28T17:46:12.760484+0000","last_active":"2024-03-28T17:47:37.652849+0000","last_peered":"2024-03-28T17:47:37.652849+0000","last_clean":"2024-03-28T17:47:37.652849+0000","last_became_active":"2024-03-28T17:46:12.760121+0000","last_became_peered":"2024-03-28T17:46:12.760121+0000","last_unstale":"2024-03-28T17:47:37.652849+0000","last_undegraded":"2024-03-28T17:47:37.652849+0000","last_fullsized":"2024-03-28T17:47:37.652849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:33:51.457426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461251+0000","last_change":"2024-03-28T17:46:11.769133+0000","last_active":"2024-03-28T17:47:09.461251+0000","last_peered":"2024-03-28T17:47:09.461251+0000","last_clean":"2024-03-28T17:47:09.461251+0000","last_became_active":"2024-03-28T17:46:11.768857+0000","last_became_peered":"2024-03-28T17:46:11.768857+0000","last_unstale":"2024-03-28T17:47:09.461251+0000","last_undegraded":"2024-03-28T17:47:09.461251+0000","last_fullsized":"2024-03-28T17:47:09.461251+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:19:26.065368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.662697+0000","last_change":"2024-03-28T17:46:12.765246+0000","last_active":"2024-03-28T17:47:37.662697+0000","last_peered":"2024-03-28T17:47:37.662697+0000","last_clean":"2024-03-28T17:47:37.662697+0000","last_became_active":"2024-03-28T17:46:12.764788+0000","last_became_peered":"2024-03-28T17:46:12.764788+0000","last_unstale":"2024-03-28T17:47:37.662697+0000","last_undegraded":"2024-03-28T17:47:37.662697+0000","last_fullsized":"2024-03-28T17:47:37.662697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:36:22.864337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461733+0000","last_change":"2024-03-28T17:46:11.761005+0000","last_active":"2024-03-28T17:47:09.461733+0000","last_peered":"2024-03-28T17:47:09.461733+0000","last_clean":"2024-03-28T17:47:09.461733+0000","last_became_active":"2024-03-28T17:46:11.760660+0000","last_became_peered":"2024-03-28T17:46:11.760660+0000","last_unstale":"2024-03-28T17:47:09.461733+0000","last_undegraded":"2024-03-28T17:47:09.461733+0000","last_fullsized":"2024-03-28T17:47:09.461733+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:01:10.408853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.644450+0000","last_change":"2024-03-28T17:46:12.762851+0000","last_active":"2024-03-28T17:47:37.644450+0000","last_peered":"2024-03-28T17:47:37.644450+0000","last_clean":"2024-03-28T17:47:37.644450+0000","last_became_active":"2024-03-28T17:46:12.762508+0000","last_became_peered":"2024-03-28T17:46:12.762508+0000","last_unstale":"2024-03-28T17:47:37.644450+0000","last_undegraded":"2024-03-28T17:47:37.644450+0000","last_fullsized":"2024-03-28T17:47:37.644450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:55:29.389985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.953352+0000","last_change":"2024-03-28T17:47:09.953352+0000","last_active":"2024-03-28T17:47:09.953352+0000","last_peered":"2024-03-28T17:47:09.953352+0000","last_clean":"2024-03-28T17:47:09.953352+0000","last_became_active":"2024-03-28T17:47:09.952822+0000","last_became_peered":"2024-03-28T17:47:09.952822+0000","last_unstale":"2024-03-28T17:47:09.953352+0000","last_undegraded":"2024-03-28T17:47:09.953352+0000","last_fullsized":"2024-03-28T17:47:09.953352+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:22:48.635040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.950822+0000","last_change":"2024-03-28T17:46:12.794290+0000","last_active":"2024-03-28T17:47:09.950822+0000","last_peered":"2024-03-28T17:47:09.950822+0000","last_clean":"2024-03-28T17:47:09.950822+0000","last_became_active":"2024-03-28T17:46:12.793906+0000","last_became_peered":"2024-03-28T17:46:12.793906+0000","last_unstale":"2024-03-28T17:47:09.950822+0000","last_undegraded":"2024-03-28T17:47:09.950822+0000","last_fullsized":"2024-03-28T17:47:09.950822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:26:04.996019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.463013+0000","last_change":"2024-03-28T17:46:11.762676+0000","last_active":"2024-03-28T17:47:09.463013+0000","last_peered":"2024-03-28T17:47:09.463013+0000","last_clean":"2024-03-28T17:47:09.463013+0000","last_became_active":"2024-03-28T17:46:11.761458+0000","last_became_peered":"2024-03-28T17:46:11.761458+0000","last_unstale":"2024-03-28T17:47:09.463013+0000","last_undegraded":"2024-03-28T17:47:09.463013+0000","last_fullsized":"2024-03-28T17:47:09.463013+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:22:26.642798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.950765+0000","last_change":"2024-03-28T17:46:12.787107+0000","last_active":"2024-03-28T17:47:09.950765+0000","last_peered":"2024-03-28T17:47:09.950765+0000","last_clean":"2024-03-28T17:47:09.950765+0000","last_became_active":"2024-03-28T17:46:12.786610+0000","last_became_peered":"2024-03-28T17:46:12.786610+0000","last_unstale":"2024-03-28T17:47:09.950765+0000","last_undegraded":"2024-03-28T17:47:09.950765+0000","last_fullsized":"2024-03-28T17:47:09.950765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:30:14.880483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.665610+0000","last_change":"2024-03-28T17:47:09.645023+0000","last_active":"2024-03-28T17:47:37.665610+0000","last_peered":"2024-03-28T17:47:37.665610+0000","last_clean":"2024-03-28T17:47:37.665610+0000","last_became_active":"2024-03-28T17:47:09.644507+0000","last_became_peered":"2024-03-28T17:47:09.644507+0000","last_unstale":"2024-03-28T17:47:37.665610+0000","last_undegraded":"2024-03-28T17:47:37.665610+0000","last_fullsized":"2024-03-28T17:47:37.665610+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:00:48.144635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"87'5","reported_seq":106,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:38.109319+0000","last_change":"2024-03-28T17:46:11.784261+0000","last_active":"2024-03-28T17:47:38.109319+0000","last_peered":"2024-03-28T17:47:38.109319+0000","last_clean":"2024-03-28T17:47:38.109319+0000","last_became_active":"2024-03-28T17:46:11.782957+0000","last_became_peered":"2024-03-28T17:46:11.782957+0000","last_unstale":"2024-03-28T17:47:38.109319+0000","last_undegraded":"2024-03-28T17:47:38.109319+0000","last_fullsized":"2024-03-28T17:47:38.109319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:08:48.009774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":82,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687427+0000","last_change":"2024-03-28T17:46:12.784024+0000","last_active":"2024-03-28T17:47:37.687427+0000","last_peered":"2024-03-28T17:47:37.687427+0000","last_clean":"2024-03-28T17:47:37.687427+0000","last_became_active":"2024-03-28T17:46:12.782157+0000","last_became_peered":"2024-03-28T17:46:12.782157+0000","last_unstale":"2024-03-28T17:47:37.687427+0000","last_undegraded":"2024-03-28T17:47:37.687427+0000","last_fullsized":"2024-03-28T17:47:37.687427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:33.679894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.644050+0000","last_change":"2024-03-28T17:46:11.759602+0000","last_active":"2024-03-28T17:47:09.644050+0000","last_peered":"2024-03-28T17:47:09.644050+0000","last_clean":"2024-03-28T17:47:09.644050+0000","last_became_active":"2024-03-28T17:46:11.759248+0000","last_became_peered":"2024-03-28T17:46:11.759248+0000","last_unstale":"2024-03-28T17:47:09.644050+0000","last_undegraded":"2024-03-28T17:47:09.644050+0000","last_fullsized":"2024-03-28T17:47:09.644050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:15:59.997948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":352,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.462874+0000","last_change":"2024-03-28T17:44:51.172493+0000","last_active":"2024-03-28T17:47:09.462874+0000","last_peered":"2024-03-28T17:47:09.462874+0000","last_clean":"2024-03-28T17:47:09.462874+0000","last_became_active":"2024-03-28T17:44:51.172175+0000","last_became_peered":"2024-03-28T17:44:51.172175+0000","last_unstale":"2024-03-28T17:47:09.462874+0000","last_undegraded":"2024-03-28T17:47:09.462874+0000","last_fullsized":"2024-03-28T17:47:09.462874+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:40:54.580958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:40:54.580958+0000","last_clean_scrub_stamp":"2024-03-28T17:40:54.580958+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:34.942490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.667471+0000","last_change":"2024-03-28T17:46:12.766728+0000","last_active":"2024-03-28T17:47:37.667471+0000","last_peered":"2024-03-28T17:47:37.667471+0000","last_clean":"2024-03-28T17:47:37.667471+0000","last_became_active":"2024-03-28T17:46:12.763671+0000","last_became_peered":"2024-03-28T17:46:12.763671+0000","last_unstale":"2024-03-28T17:47:37.667471+0000","last_undegraded":"2024-03-28T17:47:37.667471+0000","last_fullsized":"2024-03-28T17:47:37.667471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:25:39.802551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.462937+0000","last_change":"2024-03-28T17:46:11.763361+0000","last_active":"2024-03-28T17:47:09.462937+0000","last_peered":"2024-03-28T17:47:09.462937+0000","last_clean":"2024-03-28T17:47:09.462937+0000","last_became_active":"2024-03-28T17:46:11.763049+0000","last_became_peered":"2024-03-28T17:46:11.763049+0000","last_unstale":"2024-03-28T17:47:09.462937+0000","last_undegraded":"2024-03-28T17:47:09.462937+0000","last_fullsized":"2024-03-28T17:47:09.462937+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:01.767964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461612+0000","last_change":"2024-03-28T17:46:11.774957+0000","last_active":"2024-03-28T17:47:09.461612+0000","last_peered":"2024-03-28T17:47:09.461612+0000","last_clean":"2024-03-28T17:47:09.461612+0000","last_became_active":"2024-03-28T17:46:11.774658+0000","last_became_peered":"2024-03-28T17:46:11.774658+0000","last_unstale":"2024-03-28T17:47:09.461612+0000","last_undegraded":"2024-03-28T17:47:09.461612+0000","last_fullsized":"2024-03-28T17:47:09.461612+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:40.254318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.673786+0000","last_change":"2024-03-28T17:46:12.778206+0000","last_active":"2024-03-28T17:47:37.673786+0000","last_peered":"2024-03-28T17:47:37.673786+0000","last_clean":"2024-03-28T17:47:37.673786+0000","last_became_active":"2024-03-28T17:46:12.777780+0000","last_became_peered":"2024-03-28T17:46:12.777780+0000","last_unstale":"2024-03-28T17:47:37.673786+0000","last_undegraded":"2024-03-28T17:47:37.673786+0000","last_fullsized":"2024-03-28T17:47:37.673786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:13.241794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'10","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.967124+0000","last_change":"2024-03-28T17:47:09.961000+0000","last_active":"2024-03-28T17:47:09.967124+0000","last_peered":"2024-03-28T17:47:09.967124+0000","last_clean":"2024-03-28T17:47:09.967124+0000","last_became_active":"2024-03-28T17:47:09.953155+0000","last_became_peered":"2024-03-28T17:47:09.953155+0000","last_unstale":"2024-03-28T17:47:09.967124+0000","last_undegraded":"2024-03-28T17:47:09.967124+0000","last_fullsized":"2024-03-28T17:47:09.967124+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:31:57.258152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683011+0000","last_change":"2024-03-28T17:46:12.790489+0000","last_active":"2024-03-28T17:47:37.683011+0000","last_peered":"2024-03-28T17:47:37.683011+0000","last_clean":"2024-03-28T17:47:37.683011+0000","last_became_active":"2024-03-28T17:46:12.790108+0000","last_became_peered":"2024-03-28T17:46:12.790108+0000","last_unstale":"2024-03-28T17:47:37.683011+0000","last_undegraded":"2024-03-28T17:47:37.683011+0000","last_fullsized":"2024-03-28T17:47:37.683011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:16:21.210182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.644162+0000","last_change":"2024-03-28T17:46:11.762658+0000","last_active":"2024-03-28T17:47:09.644162+0000","last_peered":"2024-03-28T17:47:09.644162+0000","last_clean":"2024-03-28T17:47:09.644162+0000","last_became_active":"2024-03-28T17:46:11.762296+0000","last_became_peered":"2024-03-28T17:46:11.762296+0000","last_unstale":"2024-03-28T17:47:09.644162+0000","last_undegraded":"2024-03-28T17:47:09.644162+0000","last_fullsized":"2024-03-28T17:47:09.644162+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:42:52.768285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683517+0000","last_change":"2024-03-28T17:47:09.488306+0000","last_active":"2024-03-28T17:47:37.683517+0000","last_peered":"2024-03-28T17:47:37.683517+0000","last_clean":"2024-03-28T17:47:37.683517+0000","last_became_active":"2024-03-28T17:47:09.487292+0000","last_became_peered":"2024-03-28T17:47:09.487292+0000","last_unstale":"2024-03-28T17:47:37.683517+0000","last_undegraded":"2024-03-28T17:47:37.683517+0000","last_fullsized":"2024-03-28T17:47:37.683517+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:55.612951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.951467+0000","last_change":"2024-03-28T17:46:11.769261+0000","last_active":"2024-03-28T17:47:09.951467+0000","last_peered":"2024-03-28T17:47:09.951467+0000","last_clean":"2024-03-28T17:47:09.951467+0000","last_became_active":"2024-03-28T17:46:11.768918+0000","last_became_peered":"2024-03-28T17:46:11.768918+0000","last_unstale":"2024-03-28T17:47:09.951467+0000","last_undegraded":"2024-03-28T17:47:09.951467+0000","last_fullsized":"2024-03-28T17:47:09.951467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:26:38.276425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.625250+0000","last_change":"2024-03-28T17:46:12.762617+0000","last_active":"2024-03-28T17:47:37.625250+0000","last_peered":"2024-03-28T17:47:37.625250+0000","last_clean":"2024-03-28T17:47:37.625250+0000","last_became_active":"2024-03-28T17:46:12.762273+0000","last_became_peered":"2024-03-28T17:46:12.762273+0000","last_unstale":"2024-03-28T17:47:37.625250+0000","last_undegraded":"2024-03-28T17:47:37.625250+0000","last_fullsized":"2024-03-28T17:47:37.625250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:45.848454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.476133+0000","last_change":"2024-03-28T17:46:11.770831+0000","last_active":"2024-03-28T17:47:09.476133+0000","last_peered":"2024-03-28T17:47:09.476133+0000","last_clean":"2024-03-28T17:47:09.476133+0000","last_became_active":"2024-03-28T17:46:11.770303+0000","last_became_peered":"2024-03-28T17:46:11.770303+0000","last_unstale":"2024-03-28T17:47:09.476133+0000","last_undegraded":"2024-03-28T17:47:09.476133+0000","last_fullsized":"2024-03-28T17:47:09.476133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:48:25.566922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685772+0000","last_change":"2024-03-28T17:46:12.768885+0000","last_active":"2024-03-28T17:47:37.685772+0000","last_peered":"2024-03-28T17:47:37.685772+0000","last_clean":"2024-03-28T17:47:37.685772+0000","last_became_active":"2024-03-28T17:46:12.768525+0000","last_became_peered":"2024-03-28T17:46:12.768525+0000","last_unstale":"2024-03-28T17:47:37.685772+0000","last_undegraded":"2024-03-28T17:47:37.685772+0000","last_fullsized":"2024-03-28T17:47:37.685772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:35:30.779279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.463366+0000","last_change":"2024-03-28T17:46:11.786661+0000","last_active":"2024-03-28T17:47:09.463366+0000","last_peered":"2024-03-28T17:47:09.463366+0000","last_clean":"2024-03-28T17:47:09.463366+0000","last_became_active":"2024-03-28T17:46:11.786226+0000","last_became_peered":"2024-03-28T17:46:11.786226+0000","last_unstale":"2024-03-28T17:47:09.463366+0000","last_undegraded":"2024-03-28T17:47:09.463366+0000","last_fullsized":"2024-03-28T17:47:09.463366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:51:24.781368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687564+0000","last_change":"2024-03-28T17:46:12.797531+0000","last_active":"2024-03-28T17:47:37.687564+0000","last_peered":"2024-03-28T17:47:37.687564+0000","last_clean":"2024-03-28T17:47:37.687564+0000","last_became_active":"2024-03-28T17:46:12.797161+0000","last_became_peered":"2024-03-28T17:46:12.797161+0000","last_unstale":"2024-03-28T17:47:37.687564+0000","last_undegraded":"2024-03-28T17:47:37.687564+0000","last_fullsized":"2024-03-28T17:47:37.687564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:01.534666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461439+0000","last_change":"2024-03-28T17:46:11.766589+0000","last_active":"2024-03-28T17:47:09.461439+0000","last_peered":"2024-03-28T17:47:09.461439+0000","last_clean":"2024-03-28T17:47:09.461439+0000","last_became_active":"2024-03-28T17:46:11.766269+0000","last_became_peered":"2024-03-28T17:46:11.766269+0000","last_unstale":"2024-03-28T17:47:09.461439+0000","last_undegraded":"2024-03-28T17:47:09.461439+0000","last_fullsized":"2024-03-28T17:47:09.461439+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:12:26.055262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.670126+0000","last_change":"2024-03-28T17:46:12.765723+0000","last_active":"2024-03-28T17:47:37.670126+0000","last_peered":"2024-03-28T17:47:37.670126+0000","last_clean":"2024-03-28T17:47:37.670126+0000","last_became_active":"2024-03-28T17:46:12.765366+0000","last_became_peered":"2024-03-28T17:46:12.765366+0000","last_unstale":"2024-03-28T17:47:37.670126+0000","last_undegraded":"2024-03-28T17:47:37.670126+0000","last_fullsized":"2024-03-28T17:47:37.670126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:39:55.103246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.463644+0000","last_change":"2024-03-28T17:46:11.767686+0000","last_active":"2024-03-28T17:47:09.463644+0000","last_peered":"2024-03-28T17:47:09.463644+0000","last_clean":"2024-03-28T17:47:09.463644+0000","last_became_active":"2024-03-28T17:46:11.767365+0000","last_became_peered":"2024-03-28T17:46:11.767365+0000","last_unstale":"2024-03-28T17:47:09.463644+0000","last_undegraded":"2024-03-28T17:47:09.463644+0000","last_fullsized":"2024-03-28T17:47:09.463644+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:49.909554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.669228+0000","last_change":"2024-03-28T17:46:12.764129+0000","last_active":"2024-03-28T17:47:37.669228+0000","last_peered":"2024-03-28T17:47:37.669228+0000","last_clean":"2024-03-28T17:47:37.669228+0000","last_became_active":"2024-03-28T17:46:12.763716+0000","last_became_peered":"2024-03-28T17:46:12.763716+0000","last_unstale":"2024-03-28T17:47:37.669228+0000","last_undegraded":"2024-03-28T17:47:37.669228+0000","last_fullsized":"2024-03-28T17:47:37.669228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:08.365578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.464912+0000","last_change":"2024-03-28T17:46:11.787374+0000","last_active":"2024-03-28T17:47:09.464912+0000","last_peered":"2024-03-28T17:47:09.464912+0000","last_clean":"2024-03-28T17:47:09.464912+0000","last_became_active":"2024-03-28T17:46:11.787047+0000","last_became_peered":"2024-03-28T17:46:11.787047+0000","last_unstale":"2024-03-28T17:47:09.464912+0000","last_undegraded":"2024-03-28T17:47:09.464912+0000","last_fullsized":"2024-03-28T17:47:09.464912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:33.657488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.652865+0000","last_change":"2024-03-28T17:47:09.480465+0000","last_active":"2024-03-28T17:47:37.652865+0000","last_peered":"2024-03-28T17:47:37.652865+0000","last_clean":"2024-03-28T17:47:37.652865+0000","last_became_active":"2024-03-28T17:47:09.475758+0000","last_became_peered":"2024-03-28T17:47:09.475758+0000","last_unstale":"2024-03-28T17:47:37.652865+0000","last_undegraded":"2024-03-28T17:47:37.652865+0000","last_fullsized":"2024-03-28T17:47:37.652865+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:08.610192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.473185+0000","last_change":"2024-03-28T17:46:11.780388+0000","last_active":"2024-03-28T17:47:09.473185+0000","last_peered":"2024-03-28T17:47:09.473185+0000","last_clean":"2024-03-28T17:47:09.473185+0000","last_became_active":"2024-03-28T17:46:11.780016+0000","last_became_peered":"2024-03-28T17:46:11.780016+0000","last_unstale":"2024-03-28T17:47:09.473185+0000","last_undegraded":"2024-03-28T17:47:09.473185+0000","last_fullsized":"2024-03-28T17:47:09.473185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:51:06.181772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.651895+0000","last_change":"2024-03-28T17:46:12.764748+0000","last_active":"2024-03-28T17:47:37.651895+0000","last_peered":"2024-03-28T17:47:37.651895+0000","last_clean":"2024-03-28T17:47:37.651895+0000","last_became_active":"2024-03-28T17:46:12.764407+0000","last_became_peered":"2024-03-28T17:46:12.764407+0000","last_unstale":"2024-03-28T17:47:37.651895+0000","last_undegraded":"2024-03-28T17:47:37.651895+0000","last_fullsized":"2024-03-28T17:47:37.651895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:41:54.185079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.652654+0000","last_change":"2024-03-28T17:46:12.762486+0000","last_active":"2024-03-28T17:47:37.652654+0000","last_peered":"2024-03-28T17:47:37.652654+0000","last_clean":"2024-03-28T17:47:37.652654+0000","last_became_active":"2024-03-28T17:46:12.762107+0000","last_became_peered":"2024-03-28T17:46:12.762107+0000","last_unstale":"2024-03-28T17:47:37.652654+0000","last_undegraded":"2024-03-28T17:47:37.652654+0000","last_fullsized":"2024-03-28T17:47:37.652654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:31.062456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.664866+0000","last_change":"2024-03-28T17:46:12.764569+0000","last_active":"2024-03-28T17:47:37.664866+0000","last_peered":"2024-03-28T17:47:37.664866+0000","last_clean":"2024-03-28T17:47:37.664866+0000","last_became_active":"2024-03-28T17:46:12.764112+0000","last_became_peered":"2024-03-28T17:46:12.764112+0000","last_unstale":"2024-03-28T17:47:37.664866+0000","last_undegraded":"2024-03-28T17:47:37.664866+0000","last_fullsized":"2024-03-28T17:47:37.664866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:39:55.526432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.684845+0000","last_change":"2024-03-28T17:46:12.786031+0000","last_active":"2024-03-28T17:47:37.684845+0000","last_peered":"2024-03-28T17:47:37.684845+0000","last_clean":"2024-03-28T17:47:37.684845+0000","last_became_active":"2024-03-28T17:46:12.785679+0000","last_became_peered":"2024-03-28T17:46:12.785679+0000","last_unstale":"2024-03-28T17:47:37.684845+0000","last_undegraded":"2024-03-28T17:47:37.684845+0000","last_fullsized":"2024-03-28T17:47:37.684845+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:41:22.537979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685174+0000","last_change":"2024-03-28T17:46:12.793285+0000","last_active":"2024-03-28T17:47:37.685174+0000","last_peered":"2024-03-28T17:47:37.685174+0000","last_clean":"2024-03-28T17:47:37.685174+0000","last_became_active":"2024-03-28T17:46:12.788935+0000","last_became_peered":"2024-03-28T17:46:12.788935+0000","last_unstale":"2024-03-28T17:47:37.685174+0000","last_undegraded":"2024-03-28T17:47:37.685174+0000","last_fullsized":"2024-03-28T17:47:37.685174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:53:58.886607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687008+0000","last_change":"2024-03-28T17:47:09.486124+0000","last_active":"2024-03-28T17:47:37.687008+0000","last_peered":"2024-03-28T17:47:37.687008+0000","last_clean":"2024-03-28T17:47:37.687008+0000","last_became_active":"2024-03-28T17:47:09.481292+0000","last_became_peered":"2024-03-28T17:47:09.481292+0000","last_unstale":"2024-03-28T17:47:37.687008+0000","last_undegraded":"2024-03-28T17:47:37.687008+0000","last_fullsized":"2024-03-28T17:47:37.687008+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:18.656764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.645140+0000","last_change":"2024-03-28T17:46:12.765140+0000","last_active":"2024-03-28T17:47:37.645140+0000","last_peered":"2024-03-28T17:47:37.645140+0000","last_clean":"2024-03-28T17:47:37.645140+0000","last_became_active":"2024-03-28T17:46:12.764760+0000","last_became_peered":"2024-03-28T17:46:12.764760+0000","last_unstale":"2024-03-28T17:47:37.645140+0000","last_undegraded":"2024-03-28T17:47:37.645140+0000","last_fullsized":"2024-03-28T17:47:37.645140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:41.576437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.658894+0000","last_change":"2024-03-28T17:46:12.760271+0000","last_active":"2024-03-28T17:47:37.658894+0000","last_peered":"2024-03-28T17:47:37.658894+0000","last_clean":"2024-03-28T17:47:37.658894+0000","last_became_active":"2024-03-28T17:46:12.759939+0000","last_became_peered":"2024-03-28T17:46:12.759939+0000","last_unstale":"2024-03-28T17:47:37.658894+0000","last_undegraded":"2024-03-28T17:47:37.658894+0000","last_fullsized":"2024-03-28T17:47:37.658894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:06:42.529826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.681026+0000","last_change":"2024-03-28T17:46:12.792360+0000","last_active":"2024-03-28T17:47:37.681026+0000","last_peered":"2024-03-28T17:47:37.681026+0000","last_clean":"2024-03-28T17:47:37.681026+0000","last_became_active":"2024-03-28T17:46:12.791979+0000","last_became_peered":"2024-03-28T17:46:12.791979+0000","last_unstale":"2024-03-28T17:47:37.681026+0000","last_undegraded":"2024-03-28T17:47:37.681026+0000","last_fullsized":"2024-03-28T17:47:37.681026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:39.866969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685045+0000","last_change":"2024-03-28T17:46:12.784228+0000","last_active":"2024-03-28T17:47:37.685045+0000","last_peered":"2024-03-28T17:47:37.685045+0000","last_clean":"2024-03-28T17:47:37.685045+0000","last_became_active":"2024-03-28T17:46:12.783890+0000","last_became_peered":"2024-03-28T17:46:12.783890+0000","last_unstale":"2024-03-28T17:47:37.685045+0000","last_undegraded":"2024-03-28T17:47:37.685045+0000","last_fullsized":"2024-03-28T17:47:37.685045+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:58:13.388705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685578+0000","last_change":"2024-03-28T17:46:12.782675+0000","last_active":"2024-03-28T17:47:37.685578+0000","last_peered":"2024-03-28T17:47:37.685578+0000","last_clean":"2024-03-28T17:47:37.685578+0000","last_became_active":"2024-03-28T17:46:12.779255+0000","last_became_peered":"2024-03-28T17:46:12.779255+0000","last_unstale":"2024-03-28T17:47:37.685578+0000","last_undegraded":"2024-03-28T17:47:37.685578+0000","last_fullsized":"2024-03-28T17:47:37.685578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:04:31.447512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.671485+0000","last_change":"2024-03-28T17:46:12.786308+0000","last_active":"2024-03-28T17:47:37.671485+0000","last_peered":"2024-03-28T17:47:37.671485+0000","last_clean":"2024-03-28T17:47:37.671485+0000","last_became_active":"2024-03-28T17:46:12.785948+0000","last_became_peered":"2024-03-28T17:46:12.785948+0000","last_unstale":"2024-03-28T17:47:37.671485+0000","last_undegraded":"2024-03-28T17:47:37.671485+0000","last_fullsized":"2024-03-28T17:47:37.671485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:43.292925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687349+0000","last_change":"2024-03-28T17:46:12.797278+0000","last_active":"2024-03-28T17:47:37.687349+0000","last_peered":"2024-03-28T17:47:37.687349+0000","last_clean":"2024-03-28T17:47:37.687349+0000","last_became_active":"2024-03-28T17:46:12.796593+0000","last_became_peered":"2024-03-28T17:46:12.796593+0000","last_unstale":"2024-03-28T17:47:37.687349+0000","last_undegraded":"2024-03-28T17:47:37.687349+0000","last_fullsized":"2024-03-28T17:47:37.687349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:07:34.433680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.638776+0000","last_change":"2024-03-28T17:46:12.764249+0000","last_active":"2024-03-28T17:47:37.638776+0000","last_peered":"2024-03-28T17:47:37.638776+0000","last_clean":"2024-03-28T17:47:37.638776+0000","last_became_active":"2024-03-28T17:46:12.763909+0000","last_became_peered":"2024-03-28T17:46:12.763909+0000","last_unstale":"2024-03-28T17:47:37.638776+0000","last_undegraded":"2024-03-28T17:47:37.638776+0000","last_fullsized":"2024-03-28T17:47:37.638776+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:40:51.780187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":469,"ondisk_log_size":469,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250098,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65547,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":517,"num_read_kb":328,"num_write":166,"num_write_kb":115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":196998,"data_compressed":5625,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383713,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":456197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645349,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1089536,"data_stored":1108942,"data_compressed":38073,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808876,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":456550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070513,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":456353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":1048956,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495676,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":455822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":1003520,"data_stored":516737,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":457093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":518619,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182477,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":456016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920840,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":456791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757314,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":1048108,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34554,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34554,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":61122,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":63004,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":62842,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T17:47:43.841 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T17:47:44.354 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:47:44.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:44 smithi045 ceph-mon[31773]: from='client.24868 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:47:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:47:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:47:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:47:44.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:44 smithi071 ceph-mon[36597]: from='client.24868 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:47:44.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:47:44.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:47:44.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:47:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:44 smithi151 ceph-mon[36481]: from='client.24868 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:47:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:47:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:47:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:47:45.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:45 smithi151 ceph-mon[36481]: pgmap v365: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-28T17:47:45.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:45 smithi045 ceph-mon[31773]: pgmap v365: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-28T17:47:45.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:45 smithi071 ceph-mon[36597]: pgmap v365: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-28T17:47:46.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.679+0000 7f4a98400700 1 -- 172.21.15.45:0/4020519088 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a901081e0 msgr2=0x7f4a901085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:46.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.679+0000 7f4a98400700 1 --2- 172.21.15.45:0/4020519088 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a901081e0 0x7f4a901085c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4a84009990 tx=0x7f4a84009ca0 comp rx=0 tx=0).stop 2024-03-28T17:47:46.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.679+0000 7f4a98400700 1 -- 172.21.15.45:0/4020519088 shutdown_connections 2024-03-28T17:47:46.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.679+0000 7f4a98400700 1 --2- 172.21.15.45:0/4020519088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a9006a7e0 0x7f4a9010f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:46.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.679+0000 7f4a98400700 1 --2- 172.21.15.45:0/4020519088 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a901081e0 0x7f4a901085c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:46.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.679+0000 7f4a98400700 1 --2- 172.21.15.45:0/4020519088 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a90069e40 0x7f4a9006a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:46.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.679+0000 7f4a98400700 1 -- 172.21.15.45:0/4020519088 >> 172.21.15.45:0/4020519088 conn(0x7f4a90077fc0 msgr2=0x7f4a900783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:46.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.679+0000 7f4a98400700 1 -- 172.21.15.45:0/4020519088 shutdown_connections 2024-03-28T17:47:46.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.679+0000 7f4a98400700 1 -- 172.21.15.45:0/4020519088 wait complete. 2024-03-28T17:47:46.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.680+0000 7f4a98400700 1 Processor -- start 2024-03-28T17:47:46.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.680+0000 7f4a98400700 1 -- start start 2024-03-28T17:47:46.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.681+0000 7f4a98400700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a90069e40 0x7f4a9019e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:46.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.681+0000 7f4a98400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a9006a7e0 0x7f4a9019eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:46.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.681+0000 7f4a98400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a901081e0 0x7f4a90198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:46.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.681+0000 7f4a98400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a90112490 con 0x7f4a901081e0 2024-03-28T17:47:46.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.681+0000 7f4a98400700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a90112310 con 0x7f4a9006a7e0 2024-03-28T17:47:46.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.681+0000 7f4a98400700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a90112610 con 0x7f4a90069e40 2024-03-28T17:47:46.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.681+0000 7f4a95a00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a90069e40 0x7f4a9019e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:46.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.681+0000 7f4a95000700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a9006a7e0 0x7f4a9019eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:46.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.681+0000 7f4a95a00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a90069e40 0x7f4a9019e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:56782/0 (socket says 172.21.15.45:56782) 2024-03-28T17:47:46.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.681+0000 7f4a95a00700 1 -- 172.21.15.45:0/1315774374 learned_addr learned my addr 172.21.15.45:0/1315774374 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:46.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.681+0000 7f4a96400700 1 --2- 172.21.15.45:0/1315774374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a901081e0 0x7f4a90198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:46.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.681+0000 7f4a95a00700 1 -- 172.21.15.45:0/1315774374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a9006a7e0 msgr2=0x7f4a9019eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:46.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.681+0000 7f4a95a00700 1 --2- 172.21.15.45:0/1315774374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a9006a7e0 0x7f4a9019eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:46.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.681+0000 7f4a95a00700 1 -- 172.21.15.45:0/1315774374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a901081e0 msgr2=0x7f4a90198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:46.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.681+0000 7f4a95a00700 1 --2- 172.21.15.45:0/1315774374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a901081e0 0x7f4a90198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:46.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.681+0000 7f4a95a00700 1 -- 172.21.15.45:0/1315774374 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a90198ff0 con 0x7f4a90069e40 2024-03-28T17:47:46.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.682+0000 7f4a95a00700 1 --2- 172.21.15.45:0/1315774374 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a90069e40 0x7f4a9019e640 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4a84000c00 tx=0x7f4a840057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:46.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.682+0000 7f4a8e000700 1 -- 172.21.15.45:0/1315774374 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a84027070 con 0x7f4a90069e40 2024-03-28T17:47:46.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.682+0000 7f4a8e000700 1 -- 172.21.15.45:0/1315774374 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a84022470 con 0x7f4a90069e40 2024-03-28T17:47:46.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.682+0000 7f4a98400700 1 -- 172.21.15.45:0/1315774374 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a90199280 con 0x7f4a90069e40 2024-03-28T17:47:46.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.682+0000 7f4a8e000700 1 -- 172.21.15.45:0/1315774374 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a84017650 con 0x7f4a90069e40 2024-03-28T17:47:46.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.682+0000 7f4a98400700 1 -- 172.21.15.45:0/1315774374 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a901a53f0 con 0x7f4a90069e40 2024-03-28T17:47:46.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.683+0000 7f4a98400700 1 -- 172.21.15.45:0/1315774374 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a90103890 con 0x7f4a90069e40 2024-03-28T17:47:46.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.685+0000 7f4a8e000700 1 -- 172.21.15.45:0/1315774374 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4a8402a070 con 0x7f4a90069e40 2024-03-28T17:47:46.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.686+0000 7f4a8e000700 1 --2- 172.21.15.45:0/1315774374 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4a780a2890 0x7f4a780a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:46.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.687+0000 7f4a8e000700 1 -- 172.21.15.45:0/1315774374 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f4a840cf850 con 0x7f4a90069e40 2024-03-28T17:47:46.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.687+0000 7f4a95000700 1 --2- 172.21.15.45:0/1315774374 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4a780a2890 0x7f4a780a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:46.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.687+0000 7f4a95000700 1 --2- 172.21.15.45:0/1315774374 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4a780a2890 0x7f4a780a4d50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4a90199ed0 tx=0x7f4a7c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:46.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.690+0000 7f4a8e000700 1 -- 172.21.15.45:0/1315774374 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4a84013070 con 0x7f4a90069e40 2024-03-28T17:47:46.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.916+0000 7f4a98400700 1 -- 172.21.15.45:0/1315774374 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4a900625d0 con 0x7f4a780a2890 2024-03-28T17:47:46.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.926+0000 7f4a8e000700 1 -- 172.21.15.45:0/1315774374 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259068 (secure 0 0 0) 0x7f4a900625d0 con 0x7f4a780a2890 2024-03-28T17:47:46.931 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:47:46.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.932+0000 7f4a98400700 1 -- 172.21.15.45:0/1315774374 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4a780a2890 msgr2=0x7f4a780a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:46.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.932+0000 7f4a98400700 1 --2- 172.21.15.45:0/1315774374 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4a780a2890 0x7f4a780a4d50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4a90199ed0 tx=0x7f4a7c009260 comp rx=0 tx=0).stop 2024-03-28T17:47:46.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.932+0000 7f4a98400700 1 -- 172.21.15.45:0/1315774374 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a90069e40 msgr2=0x7f4a9019e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:46.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.932+0000 7f4a98400700 1 --2- 172.21.15.45:0/1315774374 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a90069e40 0x7f4a9019e640 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4a84000c00 tx=0x7f4a840057c0 comp rx=0 tx=0).stop 2024-03-28T17:47:46.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.932+0000 7f4a98400700 1 -- 172.21.15.45:0/1315774374 shutdown_connections 2024-03-28T17:47:46.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.932+0000 7f4a98400700 1 --2- 172.21.15.45:0/1315774374 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4a780a2890 0x7f4a780a4d50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:46.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.932+0000 7f4a98400700 1 --2- 172.21.15.45:0/1315774374 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a90069e40 0x7f4a9019e640 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:46.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.932+0000 7f4a98400700 1 --2- 172.21.15.45:0/1315774374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a901081e0 0x7f4a90198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:46.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.932+0000 7f4a98400700 1 --2- 172.21.15.45:0/1315774374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a9006a7e0 0x7f4a9019eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:46.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.932+0000 7f4a98400700 1 -- 172.21.15.45:0/1315774374 >> 172.21.15.45:0/1315774374 conn(0x7f4a90077fc0 msgr2=0x7f4a900ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:46.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.932+0000 7f4a98400700 1 -- 172.21.15.45:0/1315774374 shutdown_connections 2024-03-28T17:47:46.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:46.932+0000 7f4a98400700 1 -- 172.21.15.45:0/1315774374 wait complete. 2024-03-28T17:47:46.934 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-28T17:47:47.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:47 smithi045 ceph-mon[31773]: pgmap v366: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-28T17:47:47.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:47 smithi071 ceph-mon[36597]: pgmap v366: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-28T17:47:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:47 smithi151 ceph-mon[36481]: pgmap v366: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-28T17:47:47.843 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":366,"stamp":"2024-03-28T17:47:45.684258+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656398,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":684,"num_read_kb":463,"num_write":353,"num_write_kb":3059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180853,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":786,"ondisk_log_size":786,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":364868,"kb_used_data":11556,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576956,"statfs":{"total":1151940427776,"available":1151566802944,"internally_reserved":0,"allocated":11833344,"data_stored":7436574,"data_compressed":114219,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":531,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":78,"num_read_kb":38,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005005"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683759+0000","last_change":"2024-03-28T17:46:12.783038+0000","last_active":"2024-03-28T17:47:37.683759+0000","last_peered":"2024-03-28T17:47:37.683759+0000","last_clean":"2024-03-28T17:47:37.683759+0000","last_became_active":"2024-03-28T17:46:12.782668+0000","last_became_peered":"2024-03-28T17:46:12.782668+0000","last_unstale":"2024-03-28T17:47:37.683759+0000","last_undegraded":"2024-03-28T17:47:37.683759+0000","last_fullsized":"2024-03-28T17:47:37.683759+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:03.424959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683400+0000","last_change":"2024-03-28T17:46:12.781828+0000","last_active":"2024-03-28T17:47:37.683400+0000","last_peered":"2024-03-28T17:47:37.683400+0000","last_clean":"2024-03-28T17:47:37.683400+0000","last_became_active":"2024-03-28T17:46:12.781424+0000","last_became_peered":"2024-03-28T17:46:12.781424+0000","last_unstale":"2024-03-28T17:47:37.683400+0000","last_undegraded":"2024-03-28T17:47:37.683400+0000","last_fullsized":"2024-03-28T17:47:37.683400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:15:16.639473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683172+0000","last_change":"2024-03-28T17:46:12.785431+0000","last_active":"2024-03-28T17:47:37.683172+0000","last_peered":"2024-03-28T17:47:37.683172+0000","last_clean":"2024-03-28T17:47:37.683172+0000","last_became_active":"2024-03-28T17:46:12.784992+0000","last_became_peered":"2024-03-28T17:46:12.784992+0000","last_unstale":"2024-03-28T17:47:37.683172+0000","last_undegraded":"2024-03-28T17:47:37.683172+0000","last_fullsized":"2024-03-28T17:47:37.683172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:03:36.926613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.662861+0000","last_change":"2024-03-28T17:46:12.765274+0000","last_active":"2024-03-28T17:47:37.662861+0000","last_peered":"2024-03-28T17:47:37.662861+0000","last_clean":"2024-03-28T17:47:37.662861+0000","last_became_active":"2024-03-28T17:46:12.764890+0000","last_became_peered":"2024-03-28T17:46:12.764890+0000","last_unstale":"2024-03-28T17:47:37.662861+0000","last_undegraded":"2024-03-28T17:47:37.662861+0000","last_fullsized":"2024-03-28T17:47:37.662861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:58:37.149756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.670408+0000","last_change":"2024-03-28T17:46:12.782791+0000","last_active":"2024-03-28T17:47:37.670408+0000","last_peered":"2024-03-28T17:47:37.670408+0000","last_clean":"2024-03-28T17:47:37.670408+0000","last_became_active":"2024-03-28T17:46:12.782420+0000","last_became_peered":"2024-03-28T17:46:12.782420+0000","last_unstale":"2024-03-28T17:47:37.670408+0000","last_undegraded":"2024-03-28T17:47:37.670408+0000","last_fullsized":"2024-03-28T17:47:37.670408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:30.512863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.657246+0000","last_change":"2024-03-28T17:46:12.780713+0000","last_active":"2024-03-28T17:47:37.657246+0000","last_peered":"2024-03-28T17:47:37.657246+0000","last_clean":"2024-03-28T17:47:37.657246+0000","last_became_active":"2024-03-28T17:46:12.780305+0000","last_became_peered":"2024-03-28T17:46:12.780305+0000","last_unstale":"2024-03-28T17:47:37.657246+0000","last_undegraded":"2024-03-28T17:47:37.657246+0000","last_fullsized":"2024-03-28T17:47:37.657246+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:08:09.840491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.653070+0000","last_change":"2024-03-28T17:46:12.778120+0000","last_active":"2024-03-28T17:47:37.653070+0000","last_peered":"2024-03-28T17:47:37.653070+0000","last_clean":"2024-03-28T17:47:37.653070+0000","last_became_active":"2024-03-28T17:46:12.777724+0000","last_became_peered":"2024-03-28T17:46:12.777724+0000","last_unstale":"2024-03-28T17:47:37.653070+0000","last_undegraded":"2024-03-28T17:47:37.653070+0000","last_fullsized":"2024-03-28T17:47:37.653070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:29:50.980366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687801+0000","last_change":"2024-03-28T17:46:12.793447+0000","last_active":"2024-03-28T17:47:37.687801+0000","last_peered":"2024-03-28T17:47:37.687801+0000","last_clean":"2024-03-28T17:47:37.687801+0000","last_became_active":"2024-03-28T17:46:12.793070+0000","last_became_peered":"2024-03-28T17:46:12.793070+0000","last_unstale":"2024-03-28T17:47:37.687801+0000","last_undegraded":"2024-03-28T17:47:37.687801+0000","last_fullsized":"2024-03-28T17:47:37.687801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:01.364577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.668799+0000","last_change":"2024-03-28T17:46:12.763908+0000","last_active":"2024-03-28T17:47:37.668799+0000","last_peered":"2024-03-28T17:47:37.668799+0000","last_clean":"2024-03-28T17:47:37.668799+0000","last_became_active":"2024-03-28T17:46:12.763267+0000","last_became_peered":"2024-03-28T17:46:12.763267+0000","last_unstale":"2024-03-28T17:47:37.668799+0000","last_undegraded":"2024-03-28T17:47:37.668799+0000","last_fullsized":"2024-03-28T17:47:37.668799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:10.692974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.660471+0000","last_change":"2024-03-28T17:46:12.762648+0000","last_active":"2024-03-28T17:47:37.660471+0000","last_peered":"2024-03-28T17:47:37.660471+0000","last_clean":"2024-03-28T17:47:37.660471+0000","last_became_active":"2024-03-28T17:46:12.762300+0000","last_became_peered":"2024-03-28T17:46:12.762300+0000","last_unstale":"2024-03-28T17:47:37.660471+0000","last_undegraded":"2024-03-28T17:47:37.660471+0000","last_fullsized":"2024-03-28T17:47:37.660471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:02.210828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.665668+0000","last_change":"2024-03-28T17:46:12.766482+0000","last_active":"2024-03-28T17:47:37.665668+0000","last_peered":"2024-03-28T17:47:37.665668+0000","last_clean":"2024-03-28T17:47:37.665668+0000","last_became_active":"2024-03-28T17:46:12.766155+0000","last_became_peered":"2024-03-28T17:46:12.766155+0000","last_unstale":"2024-03-28T17:47:37.665668+0000","last_undegraded":"2024-03-28T17:47:37.665668+0000","last_fullsized":"2024-03-28T17:47:37.665668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:46:24.261744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.642911+0000","last_change":"2024-03-28T17:46:12.766100+0000","last_active":"2024-03-28T17:47:37.642911+0000","last_peered":"2024-03-28T17:47:37.642911+0000","last_clean":"2024-03-28T17:47:37.642911+0000","last_became_active":"2024-03-28T17:46:12.765762+0000","last_became_peered":"2024-03-28T17:46:12.765762+0000","last_unstale":"2024-03-28T17:47:37.642911+0000","last_undegraded":"2024-03-28T17:47:37.642911+0000","last_fullsized":"2024-03-28T17:47:37.642911+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:11.666653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.669912+0000","last_change":"2024-03-28T17:46:12.753465+0000","last_active":"2024-03-28T17:47:37.669912+0000","last_peered":"2024-03-28T17:47:37.669912+0000","last_clean":"2024-03-28T17:47:37.669912+0000","last_became_active":"2024-03-28T17:46:12.751931+0000","last_became_peered":"2024-03-28T17:46:12.751931+0000","last_unstale":"2024-03-28T17:47:37.669912+0000","last_undegraded":"2024-03-28T17:47:37.669912+0000","last_fullsized":"2024-03-28T17:47:37.669912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:39:38.757878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.684424+0000","last_change":"2024-03-28T17:46:12.784177+0000","last_active":"2024-03-28T17:47:37.684424+0000","last_peered":"2024-03-28T17:47:37.684424+0000","last_clean":"2024-03-28T17:47:37.684424+0000","last_became_active":"2024-03-28T17:46:12.783823+0000","last_became_peered":"2024-03-28T17:46:12.783823+0000","last_unstale":"2024-03-28T17:47:37.684424+0000","last_undegraded":"2024-03-28T17:47:37.684424+0000","last_fullsized":"2024-03-28T17:47:37.684424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:11:33.221095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685100+0000","last_change":"2024-03-28T17:46:12.777152+0000","last_active":"2024-03-28T17:47:37.685100+0000","last_peered":"2024-03-28T17:47:37.685100+0000","last_clean":"2024-03-28T17:47:37.685100+0000","last_became_active":"2024-03-28T17:46:12.776758+0000","last_became_peered":"2024-03-28T17:46:12.776758+0000","last_unstale":"2024-03-28T17:47:37.685100+0000","last_undegraded":"2024-03-28T17:47:37.685100+0000","last_fullsized":"2024-03-28T17:47:37.685100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:20:56.587470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.686198+0000","last_change":"2024-03-28T17:46:12.782384+0000","last_active":"2024-03-28T17:47:37.686198+0000","last_peered":"2024-03-28T17:47:37.686198+0000","last_clean":"2024-03-28T17:47:37.686198+0000","last_became_active":"2024-03-28T17:46:12.781971+0000","last_became_peered":"2024-03-28T17:46:12.781971+0000","last_unstale":"2024-03-28T17:47:37.686198+0000","last_undegraded":"2024-03-28T17:47:37.686198+0000","last_fullsized":"2024-03-28T17:47:37.686198+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:19:32.652629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685817+0000","last_change":"2024-03-28T17:46:12.789364+0000","last_active":"2024-03-28T17:47:37.685817+0000","last_peered":"2024-03-28T17:47:37.685817+0000","last_clean":"2024-03-28T17:47:37.685817+0000","last_became_active":"2024-03-28T17:46:12.788976+0000","last_became_peered":"2024-03-28T17:46:12.788976+0000","last_unstale":"2024-03-28T17:47:37.685817+0000","last_undegraded":"2024-03-28T17:47:37.685817+0000","last_fullsized":"2024-03-28T17:47:37.685817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:43:37.183705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.663470+0000","last_change":"2024-03-28T17:46:12.782664+0000","last_active":"2024-03-28T17:47:37.663470+0000","last_peered":"2024-03-28T17:47:37.663470+0000","last_clean":"2024-03-28T17:47:37.663470+0000","last_became_active":"2024-03-28T17:46:12.782305+0000","last_became_peered":"2024-03-28T17:46:12.782305+0000","last_unstale":"2024-03-28T17:47:37.663470+0000","last_undegraded":"2024-03-28T17:47:37.663470+0000","last_fullsized":"2024-03-28T17:47:37.663470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:47:18.373517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.684298+0000","last_change":"2024-03-28T17:46:12.794494+0000","last_active":"2024-03-28T17:47:37.684298+0000","last_peered":"2024-03-28T17:47:37.684298+0000","last_clean":"2024-03-28T17:47:37.684298+0000","last_became_active":"2024-03-28T17:46:12.794151+0000","last_became_peered":"2024-03-28T17:46:12.794151+0000","last_unstale":"2024-03-28T17:47:37.684298+0000","last_undegraded":"2024-03-28T17:47:37.684298+0000","last_fullsized":"2024-03-28T17:47:37.684298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:17:24.596333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.475712+0000","last_change":"2024-03-28T17:46:11.759643+0000","last_active":"2024-03-28T17:47:09.475712+0000","last_peered":"2024-03-28T17:47:09.475712+0000","last_clean":"2024-03-28T17:47:09.475712+0000","last_became_active":"2024-03-28T17:46:11.759273+0000","last_became_peered":"2024-03-28T17:46:11.759273+0000","last_unstale":"2024-03-28T17:47:09.475712+0000","last_undegraded":"2024-03-28T17:47:09.475712+0000","last_fullsized":"2024-03-28T17:47:09.475712+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:09:51.764471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.681633+0000","last_change":"2024-03-28T17:46:12.786503+0000","last_active":"2024-03-28T17:47:37.681633+0000","last_peered":"2024-03-28T17:47:37.681633+0000","last_clean":"2024-03-28T17:47:37.681633+0000","last_became_active":"2024-03-28T17:46:12.784651+0000","last_became_peered":"2024-03-28T17:46:12.784651+0000","last_unstale":"2024-03-28T17:47:37.681633+0000","last_undegraded":"2024-03-28T17:47:37.681633+0000","last_fullsized":"2024-03-28T17:47:37.681633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:30:22.401767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.480673+0000","last_change":"2024-03-28T17:46:11.780340+0000","last_active":"2024-03-28T17:47:09.480673+0000","last_peered":"2024-03-28T17:47:09.480673+0000","last_clean":"2024-03-28T17:47:09.480673+0000","last_became_active":"2024-03-28T17:46:11.779917+0000","last_became_peered":"2024-03-28T17:46:11.779917+0000","last_unstale":"2024-03-28T17:47:09.480673+0000","last_undegraded":"2024-03-28T17:47:09.480673+0000","last_fullsized":"2024-03-28T17:47:09.480673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:22:45.617444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685946+0000","last_change":"2024-03-28T17:46:12.795086+0000","last_active":"2024-03-28T17:47:37.685946+0000","last_peered":"2024-03-28T17:47:37.685946+0000","last_clean":"2024-03-28T17:47:37.685946+0000","last_became_active":"2024-03-28T17:46:12.794124+0000","last_became_peered":"2024-03-28T17:46:12.794124+0000","last_unstale":"2024-03-28T17:47:37.685946+0000","last_undegraded":"2024-03-28T17:47:37.685946+0000","last_fullsized":"2024-03-28T17:47:37.685946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:11:05.347047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.648328+0000","last_change":"2024-03-28T17:47:09.648328+0000","last_active":"2024-03-28T17:47:09.648328+0000","last_peered":"2024-03-28T17:47:09.648328+0000","last_clean":"2024-03-28T17:47:09.648328+0000","last_became_active":"2024-03-28T17:47:09.644797+0000","last_became_peered":"2024-03-28T17:47:09.644797+0000","last_unstale":"2024-03-28T17:47:09.648328+0000","last_undegraded":"2024-03-28T17:47:09.648328+0000","last_fullsized":"2024-03-28T17:47:09.648328+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:14:30.233156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.665594+0000","last_change":"2024-03-28T17:46:12.768118+0000","last_active":"2024-03-28T17:47:37.665594+0000","last_peered":"2024-03-28T17:47:37.665594+0000","last_clean":"2024-03-28T17:47:37.665594+0000","last_became_active":"2024-03-28T17:46:12.767765+0000","last_became_peered":"2024-03-28T17:46:12.767765+0000","last_unstale":"2024-03-28T17:47:37.665594+0000","last_undegraded":"2024-03-28T17:47:37.665594+0000","last_fullsized":"2024-03-28T17:47:37.665594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:16:12.268968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461621+0000","last_change":"2024-03-28T17:46:11.760204+0000","last_active":"2024-03-28T17:47:09.461621+0000","last_peered":"2024-03-28T17:47:09.461621+0000","last_clean":"2024-03-28T17:47:09.461621+0000","last_became_active":"2024-03-28T17:46:11.759829+0000","last_became_peered":"2024-03-28T17:46:11.759829+0000","last_unstale":"2024-03-28T17:47:09.461621+0000","last_undegraded":"2024-03-28T17:47:09.461621+0000","last_fullsized":"2024-03-28T17:47:09.461621+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:29:02.075424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.663263+0000","last_change":"2024-03-28T17:46:12.762229+0000","last_active":"2024-03-28T17:47:37.663263+0000","last_peered":"2024-03-28T17:47:37.663263+0000","last_clean":"2024-03-28T17:47:37.663263+0000","last_became_active":"2024-03-28T17:46:12.761889+0000","last_became_peered":"2024-03-28T17:46:12.761889+0000","last_unstale":"2024-03-28T17:47:37.663263+0000","last_undegraded":"2024-03-28T17:47:37.663263+0000","last_fullsized":"2024-03-28T17:47:37.663263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:15.023966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'7","reported_seq":117,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:43.064910+0000","last_change":"2024-03-28T17:46:11.763152+0000","last_active":"2024-03-28T17:47:43.064910+0000","last_peered":"2024-03-28T17:47:43.064910+0000","last_clean":"2024-03-28T17:47:43.064910+0000","last_became_active":"2024-03-28T17:46:11.762754+0000","last_became_peered":"2024-03-28T17:46:11.762754+0000","last_unstale":"2024-03-28T17:47:43.064910+0000","last_undegraded":"2024-03-28T17:47:43.064910+0000","last_fullsized":"2024-03-28T17:47:43.064910+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:17:06.659908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.649470+0000","last_change":"2024-03-28T17:46:12.780701+0000","last_active":"2024-03-28T17:47:37.649470+0000","last_peered":"2024-03-28T17:47:37.649470+0000","last_clean":"2024-03-28T17:47:37.649470+0000","last_became_active":"2024-03-28T17:46:12.780339+0000","last_became_peered":"2024-03-28T17:46:12.780339+0000","last_unstale":"2024-03-28T17:47:37.649470+0000","last_undegraded":"2024-03-28T17:47:37.649470+0000","last_fullsized":"2024-03-28T17:47:37.649470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:59.774888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.950680+0000","last_change":"2024-03-28T17:46:11.763387+0000","last_active":"2024-03-28T17:47:09.950680+0000","last_peered":"2024-03-28T17:47:09.950680+0000","last_clean":"2024-03-28T17:47:09.950680+0000","last_became_active":"2024-03-28T17:46:11.763028+0000","last_became_peered":"2024-03-28T17:46:11.763028+0000","last_unstale":"2024-03-28T17:47:09.950680+0000","last_undegraded":"2024-03-28T17:47:09.950680+0000","last_fullsized":"2024-03-28T17:47:09.950680+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:21:31.585047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685166+0000","last_change":"2024-03-28T17:46:12.794167+0000","last_active":"2024-03-28T17:47:37.685166+0000","last_peered":"2024-03-28T17:47:37.685166+0000","last_clean":"2024-03-28T17:47:37.685166+0000","last_became_active":"2024-03-28T17:46:12.793830+0000","last_became_peered":"2024-03-28T17:46:12.793830+0000","last_unstale":"2024-03-28T17:47:37.685166+0000","last_undegraded":"2024-03-28T17:47:37.685166+0000","last_fullsized":"2024-03-28T17:47:37.685166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:20.907752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.470172+0000","last_change":"2024-03-28T17:46:11.769322+0000","last_active":"2024-03-28T17:47:09.470172+0000","last_peered":"2024-03-28T17:47:09.470172+0000","last_clean":"2024-03-28T17:47:09.470172+0000","last_became_active":"2024-03-28T17:46:11.768931+0000","last_became_peered":"2024-03-28T17:46:11.768931+0000","last_unstale":"2024-03-28T17:47:09.470172+0000","last_undegraded":"2024-03-28T17:47:09.470172+0000","last_fullsized":"2024-03-28T17:47:09.470172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:29:34.372947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683329+0000","last_change":"2024-03-28T17:46:12.781306+0000","last_active":"2024-03-28T17:47:37.683329+0000","last_peered":"2024-03-28T17:47:37.683329+0000","last_clean":"2024-03-28T17:47:37.683329+0000","last_became_active":"2024-03-28T17:46:12.779161+0000","last_became_peered":"2024-03-28T17:46:12.779161+0000","last_unstale":"2024-03-28T17:47:37.683329+0000","last_undegraded":"2024-03-28T17:47:37.683329+0000","last_fullsized":"2024-03-28T17:47:37.683329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:06:10.302067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.464702+0000","last_change":"2024-03-28T17:46:11.762852+0000","last_active":"2024-03-28T17:47:09.464702+0000","last_peered":"2024-03-28T17:47:09.464702+0000","last_clean":"2024-03-28T17:47:09.464702+0000","last_became_active":"2024-03-28T17:46:11.762428+0000","last_became_peered":"2024-03-28T17:46:11.762428+0000","last_unstale":"2024-03-28T17:47:09.464702+0000","last_undegraded":"2024-03-28T17:47:09.464702+0000","last_fullsized":"2024-03-28T17:47:09.464702+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:38:20.433408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685573+0000","last_change":"2024-03-28T17:46:12.786295+0000","last_active":"2024-03-28T17:47:37.685573+0000","last_peered":"2024-03-28T17:47:37.685573+0000","last_clean":"2024-03-28T17:47:37.685573+0000","last_became_active":"2024-03-28T17:46:12.785894+0000","last_became_peered":"2024-03-28T17:46:12.785894+0000","last_unstale":"2024-03-28T17:47:37.685573+0000","last_undegraded":"2024-03-28T17:47:37.685573+0000","last_fullsized":"2024-03-28T17:47:37.685573+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:06.271660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'4","reported_seq":108,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:41.062178+0000","last_change":"2024-03-28T17:46:11.766787+0000","last_active":"2024-03-28T17:47:41.062178+0000","last_peered":"2024-03-28T17:47:41.062178+0000","last_clean":"2024-03-28T17:47:41.062178+0000","last_became_active":"2024-03-28T17:46:11.766352+0000","last_became_peered":"2024-03-28T17:46:11.766352+0000","last_unstale":"2024-03-28T17:47:41.062178+0000","last_undegraded":"2024-03-28T17:47:41.062178+0000","last_fullsized":"2024-03-28T17:47:41.062178+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:54:48.157597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":85,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.666145+0000","last_change":"2024-03-28T17:46:12.785724+0000","last_active":"2024-03-28T17:47:37.666145+0000","last_peered":"2024-03-28T17:47:37.666145+0000","last_clean":"2024-03-28T17:47:37.666145+0000","last_became_active":"2024-03-28T17:46:12.785353+0000","last_became_peered":"2024-03-28T17:46:12.785353+0000","last_unstale":"2024-03-28T17:47:37.666145+0000","last_undegraded":"2024-03-28T17:47:37.666145+0000","last_fullsized":"2024-03-28T17:47:37.666145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:21:22.692899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.463079+0000","last_change":"2024-03-28T17:46:11.763507+0000","last_active":"2024-03-28T17:47:09.463079+0000","last_peered":"2024-03-28T17:47:09.463079+0000","last_clean":"2024-03-28T17:47:09.463079+0000","last_became_active":"2024-03-28T17:46:11.763177+0000","last_became_peered":"2024-03-28T17:46:11.763177+0000","last_unstale":"2024-03-28T17:47:09.463079+0000","last_undegraded":"2024-03-28T17:47:09.463079+0000","last_fullsized":"2024-03-28T17:47:09.463079+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:56:39.696699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685855+0000","last_change":"2024-03-28T17:47:09.479162+0000","last_active":"2024-03-28T17:47:37.685855+0000","last_peered":"2024-03-28T17:47:37.685855+0000","last_clean":"2024-03-28T17:47:37.685855+0000","last_became_active":"2024-03-28T17:47:09.478349+0000","last_became_peered":"2024-03-28T17:47:09.478349+0000","last_unstale":"2024-03-28T17:47:37.685855+0000","last_undegraded":"2024-03-28T17:47:37.685855+0000","last_fullsized":"2024-03-28T17:47:37.685855+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:41.056361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.465137+0000","last_change":"2024-03-28T17:46:11.774961+0000","last_active":"2024-03-28T17:47:09.465137+0000","last_peered":"2024-03-28T17:47:09.465137+0000","last_clean":"2024-03-28T17:47:09.465137+0000","last_became_active":"2024-03-28T17:46:11.774645+0000","last_became_peered":"2024-03-28T17:46:11.774645+0000","last_unstale":"2024-03-28T17:47:09.465137+0000","last_undegraded":"2024-03-28T17:47:09.465137+0000","last_fullsized":"2024-03-28T17:47:09.465137+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:38:23.983739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.667399+0000","last_change":"2024-03-28T17:46:12.763698+0000","last_active":"2024-03-28T17:47:37.667399+0000","last_peered":"2024-03-28T17:47:37.667399+0000","last_clean":"2024-03-28T17:47:37.667399+0000","last_became_active":"2024-03-28T17:46:12.763238+0000","last_became_peered":"2024-03-28T17:46:12.763238+0000","last_unstale":"2024-03-28T17:47:37.667399+0000","last_undegraded":"2024-03-28T17:47:37.667399+0000","last_fullsized":"2024-03-28T17:47:37.667399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:12:58.252855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'33","reported_seq":232,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:42.068032+0000","last_change":"2024-03-28T17:46:11.763096+0000","last_active":"2024-03-28T17:47:42.068032+0000","last_peered":"2024-03-28T17:47:42.068032+0000","last_clean":"2024-03-28T17:47:42.068032+0000","last_became_active":"2024-03-28T17:46:11.762730+0000","last_became_peered":"2024-03-28T17:46:11.762730+0000","last_unstale":"2024-03-28T17:47:42.068032+0000","last_undegraded":"2024-03-28T17:47:42.068032+0000","last_fullsized":"2024-03-28T17:47:42.068032+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:45:59.469096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61122,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":180,"num_read_kb":150,"num_write":34,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.651310+0000","last_change":"2024-03-28T17:46:12.760366+0000","last_active":"2024-03-28T17:47:37.651310+0000","last_peered":"2024-03-28T17:47:37.651310+0000","last_clean":"2024-03-28T17:47:37.651310+0000","last_became_active":"2024-03-28T17:46:12.759957+0000","last_became_peered":"2024-03-28T17:46:12.759957+0000","last_unstale":"2024-03-28T17:47:37.651310+0000","last_undegraded":"2024-03-28T17:47:37.651310+0000","last_fullsized":"2024-03-28T17:47:37.651310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:08:02.579659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.463494+0000","last_change":"2024-03-28T17:46:11.787108+0000","last_active":"2024-03-28T17:47:09.463494+0000","last_peered":"2024-03-28T17:47:09.463494+0000","last_clean":"2024-03-28T17:47:09.463494+0000","last_became_active":"2024-03-28T17:46:11.786734+0000","last_became_peered":"2024-03-28T17:46:11.786734+0000","last_unstale":"2024-03-28T17:47:09.463494+0000","last_undegraded":"2024-03-28T17:47:09.463494+0000","last_fullsized":"2024-03-28T17:47:09.463494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:04:00.480545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.678596+0000","last_change":"2024-03-28T17:46:12.786362+0000","last_active":"2024-03-28T17:47:37.678596+0000","last_peered":"2024-03-28T17:47:37.678596+0000","last_clean":"2024-03-28T17:47:37.678596+0000","last_became_active":"2024-03-28T17:46:12.785946+0000","last_became_peered":"2024-03-28T17:46:12.785946+0000","last_unstale":"2024-03-28T17:47:37.678596+0000","last_undegraded":"2024-03-28T17:47:37.678596+0000","last_fullsized":"2024-03-28T17:47:37.678596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:13:57.857818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.464833+0000","last_change":"2024-03-28T17:46:11.786292+0000","last_active":"2024-03-28T17:47:09.464833+0000","last_peered":"2024-03-28T17:47:09.464833+0000","last_clean":"2024-03-28T17:47:09.464833+0000","last_became_active":"2024-03-28T17:46:11.785953+0000","last_became_peered":"2024-03-28T17:46:11.785953+0000","last_unstale":"2024-03-28T17:47:09.464833+0000","last_undegraded":"2024-03-28T17:47:09.464833+0000","last_fullsized":"2024-03-28T17:47:09.464833+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:02:18.093585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.669772+0000","last_change":"2024-03-28T17:46:12.770669+0000","last_active":"2024-03-28T17:47:37.669772+0000","last_peered":"2024-03-28T17:47:37.669772+0000","last_clean":"2024-03-28T17:47:37.669772+0000","last_became_active":"2024-03-28T17:46:12.770301+0000","last_became_peered":"2024-03-28T17:46:12.770301+0000","last_unstale":"2024-03-28T17:47:37.669772+0000","last_undegraded":"2024-03-28T17:47:37.669772+0000","last_fullsized":"2024-03-28T17:47:37.669772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:59:00.601357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'5","reported_seq":115,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:43.120718+0000","last_change":"2024-03-28T17:46:11.759686+0000","last_active":"2024-03-28T17:47:43.120718+0000","last_peered":"2024-03-28T17:47:43.120718+0000","last_clean":"2024-03-28T17:47:43.120718+0000","last_became_active":"2024-03-28T17:46:11.759309+0000","last_became_peered":"2024-03-28T17:46:11.759309+0000","last_unstale":"2024-03-28T17:47:43.120718+0000","last_undegraded":"2024-03-28T17:47:43.120718+0000","last_fullsized":"2024-03-28T17:47:43.120718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:31:09.999279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.652849+0000","last_change":"2024-03-28T17:46:12.760484+0000","last_active":"2024-03-28T17:47:37.652849+0000","last_peered":"2024-03-28T17:47:37.652849+0000","last_clean":"2024-03-28T17:47:37.652849+0000","last_became_active":"2024-03-28T17:46:12.760121+0000","last_became_peered":"2024-03-28T17:46:12.760121+0000","last_unstale":"2024-03-28T17:47:37.652849+0000","last_undegraded":"2024-03-28T17:47:37.652849+0000","last_fullsized":"2024-03-28T17:47:37.652849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:33:51.457426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461251+0000","last_change":"2024-03-28T17:46:11.769133+0000","last_active":"2024-03-28T17:47:09.461251+0000","last_peered":"2024-03-28T17:47:09.461251+0000","last_clean":"2024-03-28T17:47:09.461251+0000","last_became_active":"2024-03-28T17:46:11.768857+0000","last_became_peered":"2024-03-28T17:46:11.768857+0000","last_unstale":"2024-03-28T17:47:09.461251+0000","last_undegraded":"2024-03-28T17:47:09.461251+0000","last_fullsized":"2024-03-28T17:47:09.461251+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:19:26.065368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.662697+0000","last_change":"2024-03-28T17:46:12.765246+0000","last_active":"2024-03-28T17:47:37.662697+0000","last_peered":"2024-03-28T17:47:37.662697+0000","last_clean":"2024-03-28T17:47:37.662697+0000","last_became_active":"2024-03-28T17:46:12.764788+0000","last_became_peered":"2024-03-28T17:46:12.764788+0000","last_unstale":"2024-03-28T17:47:37.662697+0000","last_undegraded":"2024-03-28T17:47:37.662697+0000","last_fullsized":"2024-03-28T17:47:37.662697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:36:22.864337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461733+0000","last_change":"2024-03-28T17:46:11.761005+0000","last_active":"2024-03-28T17:47:09.461733+0000","last_peered":"2024-03-28T17:47:09.461733+0000","last_clean":"2024-03-28T17:47:09.461733+0000","last_became_active":"2024-03-28T17:46:11.760660+0000","last_became_peered":"2024-03-28T17:46:11.760660+0000","last_unstale":"2024-03-28T17:47:09.461733+0000","last_undegraded":"2024-03-28T17:47:09.461733+0000","last_fullsized":"2024-03-28T17:47:09.461733+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:01:10.408853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.644450+0000","last_change":"2024-03-28T17:46:12.762851+0000","last_active":"2024-03-28T17:47:37.644450+0000","last_peered":"2024-03-28T17:47:37.644450+0000","last_clean":"2024-03-28T17:47:37.644450+0000","last_became_active":"2024-03-28T17:46:12.762508+0000","last_became_peered":"2024-03-28T17:46:12.762508+0000","last_unstale":"2024-03-28T17:47:37.644450+0000","last_undegraded":"2024-03-28T17:47:37.644450+0000","last_fullsized":"2024-03-28T17:47:37.644450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:55:29.389985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.953352+0000","last_change":"2024-03-28T17:47:09.953352+0000","last_active":"2024-03-28T17:47:09.953352+0000","last_peered":"2024-03-28T17:47:09.953352+0000","last_clean":"2024-03-28T17:47:09.953352+0000","last_became_active":"2024-03-28T17:47:09.952822+0000","last_became_peered":"2024-03-28T17:47:09.952822+0000","last_unstale":"2024-03-28T17:47:09.953352+0000","last_undegraded":"2024-03-28T17:47:09.953352+0000","last_fullsized":"2024-03-28T17:47:09.953352+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:22:48.635040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.684289+0000","last_change":"2024-03-28T17:46:12.794290+0000","last_active":"2024-03-28T17:47:37.684289+0000","last_peered":"2024-03-28T17:47:37.684289+0000","last_clean":"2024-03-28T17:47:37.684289+0000","last_became_active":"2024-03-28T17:46:12.793906+0000","last_became_peered":"2024-03-28T17:46:12.793906+0000","last_unstale":"2024-03-28T17:47:37.684289+0000","last_undegraded":"2024-03-28T17:47:37.684289+0000","last_fullsized":"2024-03-28T17:47:37.684289+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:26:04.996019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.463013+0000","last_change":"2024-03-28T17:46:11.762676+0000","last_active":"2024-03-28T17:47:09.463013+0000","last_peered":"2024-03-28T17:47:09.463013+0000","last_clean":"2024-03-28T17:47:09.463013+0000","last_became_active":"2024-03-28T17:46:11.761458+0000","last_became_peered":"2024-03-28T17:46:11.761458+0000","last_unstale":"2024-03-28T17:47:09.463013+0000","last_undegraded":"2024-03-28T17:47:09.463013+0000","last_fullsized":"2024-03-28T17:47:09.463013+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:22:26.642798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.686763+0000","last_change":"2024-03-28T17:46:12.787107+0000","last_active":"2024-03-28T17:47:37.686763+0000","last_peered":"2024-03-28T17:47:37.686763+0000","last_clean":"2024-03-28T17:47:37.686763+0000","last_became_active":"2024-03-28T17:46:12.786610+0000","last_became_peered":"2024-03-28T17:46:12.786610+0000","last_unstale":"2024-03-28T17:47:37.686763+0000","last_undegraded":"2024-03-28T17:47:37.686763+0000","last_fullsized":"2024-03-28T17:47:37.686763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:30:14.880483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.665610+0000","last_change":"2024-03-28T17:47:09.645023+0000","last_active":"2024-03-28T17:47:37.665610+0000","last_peered":"2024-03-28T17:47:37.665610+0000","last_clean":"2024-03-28T17:47:37.665610+0000","last_became_active":"2024-03-28T17:47:09.644507+0000","last_became_peered":"2024-03-28T17:47:09.644507+0000","last_unstale":"2024-03-28T17:47:37.665610+0000","last_undegraded":"2024-03-28T17:47:37.665610+0000","last_fullsized":"2024-03-28T17:47:37.665610+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:00:48.144635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"87'5","reported_seq":111,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:43.120107+0000","last_change":"2024-03-28T17:46:11.784261+0000","last_active":"2024-03-28T17:47:43.120107+0000","last_peered":"2024-03-28T17:47:43.120107+0000","last_clean":"2024-03-28T17:47:43.120107+0000","last_became_active":"2024-03-28T17:46:11.782957+0000","last_became_peered":"2024-03-28T17:46:11.782957+0000","last_unstale":"2024-03-28T17:47:43.120107+0000","last_undegraded":"2024-03-28T17:47:43.120107+0000","last_fullsized":"2024-03-28T17:47:43.120107+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:08:48.009774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687427+0000","last_change":"2024-03-28T17:46:12.784024+0000","last_active":"2024-03-28T17:47:37.687427+0000","last_peered":"2024-03-28T17:47:37.687427+0000","last_clean":"2024-03-28T17:47:37.687427+0000","last_became_active":"2024-03-28T17:46:12.782157+0000","last_became_peered":"2024-03-28T17:46:12.782157+0000","last_unstale":"2024-03-28T17:47:37.687427+0000","last_undegraded":"2024-03-28T17:47:37.687427+0000","last_fullsized":"2024-03-28T17:47:37.687427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:33.679894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.644050+0000","last_change":"2024-03-28T17:46:11.759602+0000","last_active":"2024-03-28T17:47:09.644050+0000","last_peered":"2024-03-28T17:47:09.644050+0000","last_clean":"2024-03-28T17:47:09.644050+0000","last_became_active":"2024-03-28T17:46:11.759248+0000","last_became_peered":"2024-03-28T17:46:11.759248+0000","last_unstale":"2024-03-28T17:47:09.644050+0000","last_undegraded":"2024-03-28T17:47:09.644050+0000","last_fullsized":"2024-03-28T17:47:09.644050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:15:59.997948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":352,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.462874+0000","last_change":"2024-03-28T17:44:51.172493+0000","last_active":"2024-03-28T17:47:09.462874+0000","last_peered":"2024-03-28T17:47:09.462874+0000","last_clean":"2024-03-28T17:47:09.462874+0000","last_became_active":"2024-03-28T17:44:51.172175+0000","last_became_peered":"2024-03-28T17:44:51.172175+0000","last_unstale":"2024-03-28T17:47:09.462874+0000","last_undegraded":"2024-03-28T17:47:09.462874+0000","last_fullsized":"2024-03-28T17:47:09.462874+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:40:54.580958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:40:54.580958+0000","last_clean_scrub_stamp":"2024-03-28T17:40:54.580958+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:34.942490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.667471+0000","last_change":"2024-03-28T17:46:12.766728+0000","last_active":"2024-03-28T17:47:37.667471+0000","last_peered":"2024-03-28T17:47:37.667471+0000","last_clean":"2024-03-28T17:47:37.667471+0000","last_became_active":"2024-03-28T17:46:12.763671+0000","last_became_peered":"2024-03-28T17:46:12.763671+0000","last_unstale":"2024-03-28T17:47:37.667471+0000","last_undegraded":"2024-03-28T17:47:37.667471+0000","last_fullsized":"2024-03-28T17:47:37.667471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:25:39.802551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.462937+0000","last_change":"2024-03-28T17:46:11.763361+0000","last_active":"2024-03-28T17:47:09.462937+0000","last_peered":"2024-03-28T17:47:09.462937+0000","last_clean":"2024-03-28T17:47:09.462937+0000","last_became_active":"2024-03-28T17:46:11.763049+0000","last_became_peered":"2024-03-28T17:46:11.763049+0000","last_unstale":"2024-03-28T17:47:09.462937+0000","last_undegraded":"2024-03-28T17:47:09.462937+0000","last_fullsized":"2024-03-28T17:47:09.462937+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:01.767964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461612+0000","last_change":"2024-03-28T17:46:11.774957+0000","last_active":"2024-03-28T17:47:09.461612+0000","last_peered":"2024-03-28T17:47:09.461612+0000","last_clean":"2024-03-28T17:47:09.461612+0000","last_became_active":"2024-03-28T17:46:11.774658+0000","last_became_peered":"2024-03-28T17:46:11.774658+0000","last_unstale":"2024-03-28T17:47:09.461612+0000","last_undegraded":"2024-03-28T17:47:09.461612+0000","last_fullsized":"2024-03-28T17:47:09.461612+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:40.254318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.673786+0000","last_change":"2024-03-28T17:46:12.778206+0000","last_active":"2024-03-28T17:47:37.673786+0000","last_peered":"2024-03-28T17:47:37.673786+0000","last_clean":"2024-03-28T17:47:37.673786+0000","last_became_active":"2024-03-28T17:46:12.777780+0000","last_became_peered":"2024-03-28T17:46:12.777780+0000","last_unstale":"2024-03-28T17:47:37.673786+0000","last_undegraded":"2024-03-28T17:47:37.673786+0000","last_fullsized":"2024-03-28T17:47:37.673786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:13.241794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'10","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.967124+0000","last_change":"2024-03-28T17:47:09.961000+0000","last_active":"2024-03-28T17:47:09.967124+0000","last_peered":"2024-03-28T17:47:09.967124+0000","last_clean":"2024-03-28T17:47:09.967124+0000","last_became_active":"2024-03-28T17:47:09.953155+0000","last_became_peered":"2024-03-28T17:47:09.953155+0000","last_unstale":"2024-03-28T17:47:09.967124+0000","last_undegraded":"2024-03-28T17:47:09.967124+0000","last_fullsized":"2024-03-28T17:47:09.967124+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:31:57.258152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683011+0000","last_change":"2024-03-28T17:46:12.790489+0000","last_active":"2024-03-28T17:47:37.683011+0000","last_peered":"2024-03-28T17:47:37.683011+0000","last_clean":"2024-03-28T17:47:37.683011+0000","last_became_active":"2024-03-28T17:46:12.790108+0000","last_became_peered":"2024-03-28T17:46:12.790108+0000","last_unstale":"2024-03-28T17:47:37.683011+0000","last_undegraded":"2024-03-28T17:47:37.683011+0000","last_fullsized":"2024-03-28T17:47:37.683011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:16:21.210182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.644162+0000","last_change":"2024-03-28T17:46:11.762658+0000","last_active":"2024-03-28T17:47:09.644162+0000","last_peered":"2024-03-28T17:47:09.644162+0000","last_clean":"2024-03-28T17:47:09.644162+0000","last_became_active":"2024-03-28T17:46:11.762296+0000","last_became_peered":"2024-03-28T17:46:11.762296+0000","last_unstale":"2024-03-28T17:47:09.644162+0000","last_undegraded":"2024-03-28T17:47:09.644162+0000","last_fullsized":"2024-03-28T17:47:09.644162+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:42:52.768285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683517+0000","last_change":"2024-03-28T17:47:09.488306+0000","last_active":"2024-03-28T17:47:37.683517+0000","last_peered":"2024-03-28T17:47:37.683517+0000","last_clean":"2024-03-28T17:47:37.683517+0000","last_became_active":"2024-03-28T17:47:09.487292+0000","last_became_peered":"2024-03-28T17:47:09.487292+0000","last_unstale":"2024-03-28T17:47:37.683517+0000","last_undegraded":"2024-03-28T17:47:37.683517+0000","last_fullsized":"2024-03-28T17:47:37.683517+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:55.612951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.951467+0000","last_change":"2024-03-28T17:46:11.769261+0000","last_active":"2024-03-28T17:47:09.951467+0000","last_peered":"2024-03-28T17:47:09.951467+0000","last_clean":"2024-03-28T17:47:09.951467+0000","last_became_active":"2024-03-28T17:46:11.768918+0000","last_became_peered":"2024-03-28T17:46:11.768918+0000","last_unstale":"2024-03-28T17:47:09.951467+0000","last_undegraded":"2024-03-28T17:47:09.951467+0000","last_fullsized":"2024-03-28T17:47:09.951467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:26:38.276425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.625250+0000","last_change":"2024-03-28T17:46:12.762617+0000","last_active":"2024-03-28T17:47:37.625250+0000","last_peered":"2024-03-28T17:47:37.625250+0000","last_clean":"2024-03-28T17:47:37.625250+0000","last_became_active":"2024-03-28T17:46:12.762273+0000","last_became_peered":"2024-03-28T17:46:12.762273+0000","last_unstale":"2024-03-28T17:47:37.625250+0000","last_undegraded":"2024-03-28T17:47:37.625250+0000","last_fullsized":"2024-03-28T17:47:37.625250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:45.848454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.476133+0000","last_change":"2024-03-28T17:46:11.770831+0000","last_active":"2024-03-28T17:47:09.476133+0000","last_peered":"2024-03-28T17:47:09.476133+0000","last_clean":"2024-03-28T17:47:09.476133+0000","last_became_active":"2024-03-28T17:46:11.770303+0000","last_became_peered":"2024-03-28T17:46:11.770303+0000","last_unstale":"2024-03-28T17:47:09.476133+0000","last_undegraded":"2024-03-28T17:47:09.476133+0000","last_fullsized":"2024-03-28T17:47:09.476133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:48:25.566922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685772+0000","last_change":"2024-03-28T17:46:12.768885+0000","last_active":"2024-03-28T17:47:37.685772+0000","last_peered":"2024-03-28T17:47:37.685772+0000","last_clean":"2024-03-28T17:47:37.685772+0000","last_became_active":"2024-03-28T17:46:12.768525+0000","last_became_peered":"2024-03-28T17:46:12.768525+0000","last_unstale":"2024-03-28T17:47:37.685772+0000","last_undegraded":"2024-03-28T17:47:37.685772+0000","last_fullsized":"2024-03-28T17:47:37.685772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:35:30.779279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.463366+0000","last_change":"2024-03-28T17:46:11.786661+0000","last_active":"2024-03-28T17:47:09.463366+0000","last_peered":"2024-03-28T17:47:09.463366+0000","last_clean":"2024-03-28T17:47:09.463366+0000","last_became_active":"2024-03-28T17:46:11.786226+0000","last_became_peered":"2024-03-28T17:46:11.786226+0000","last_unstale":"2024-03-28T17:47:09.463366+0000","last_undegraded":"2024-03-28T17:47:09.463366+0000","last_fullsized":"2024-03-28T17:47:09.463366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:51:24.781368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687564+0000","last_change":"2024-03-28T17:46:12.797531+0000","last_active":"2024-03-28T17:47:37.687564+0000","last_peered":"2024-03-28T17:47:37.687564+0000","last_clean":"2024-03-28T17:47:37.687564+0000","last_became_active":"2024-03-28T17:46:12.797161+0000","last_became_peered":"2024-03-28T17:46:12.797161+0000","last_unstale":"2024-03-28T17:47:37.687564+0000","last_undegraded":"2024-03-28T17:47:37.687564+0000","last_fullsized":"2024-03-28T17:47:37.687564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:01.534666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461439+0000","last_change":"2024-03-28T17:46:11.766589+0000","last_active":"2024-03-28T17:47:09.461439+0000","last_peered":"2024-03-28T17:47:09.461439+0000","last_clean":"2024-03-28T17:47:09.461439+0000","last_became_active":"2024-03-28T17:46:11.766269+0000","last_became_peered":"2024-03-28T17:46:11.766269+0000","last_unstale":"2024-03-28T17:47:09.461439+0000","last_undegraded":"2024-03-28T17:47:09.461439+0000","last_fullsized":"2024-03-28T17:47:09.461439+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:12:26.055262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.670126+0000","last_change":"2024-03-28T17:46:12.765723+0000","last_active":"2024-03-28T17:47:37.670126+0000","last_peered":"2024-03-28T17:47:37.670126+0000","last_clean":"2024-03-28T17:47:37.670126+0000","last_became_active":"2024-03-28T17:46:12.765366+0000","last_became_peered":"2024-03-28T17:46:12.765366+0000","last_unstale":"2024-03-28T17:47:37.670126+0000","last_undegraded":"2024-03-28T17:47:37.670126+0000","last_fullsized":"2024-03-28T17:47:37.670126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:39:55.103246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.463644+0000","last_change":"2024-03-28T17:46:11.767686+0000","last_active":"2024-03-28T17:47:09.463644+0000","last_peered":"2024-03-28T17:47:09.463644+0000","last_clean":"2024-03-28T17:47:09.463644+0000","last_became_active":"2024-03-28T17:46:11.767365+0000","last_became_peered":"2024-03-28T17:46:11.767365+0000","last_unstale":"2024-03-28T17:47:09.463644+0000","last_undegraded":"2024-03-28T17:47:09.463644+0000","last_fullsized":"2024-03-28T17:47:09.463644+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:49.909554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.669228+0000","last_change":"2024-03-28T17:46:12.764129+0000","last_active":"2024-03-28T17:47:37.669228+0000","last_peered":"2024-03-28T17:47:37.669228+0000","last_clean":"2024-03-28T17:47:37.669228+0000","last_became_active":"2024-03-28T17:46:12.763716+0000","last_became_peered":"2024-03-28T17:46:12.763716+0000","last_unstale":"2024-03-28T17:47:37.669228+0000","last_undegraded":"2024-03-28T17:47:37.669228+0000","last_fullsized":"2024-03-28T17:47:37.669228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:08.365578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.464912+0000","last_change":"2024-03-28T17:46:11.787374+0000","last_active":"2024-03-28T17:47:09.464912+0000","last_peered":"2024-03-28T17:47:09.464912+0000","last_clean":"2024-03-28T17:47:09.464912+0000","last_became_active":"2024-03-28T17:46:11.787047+0000","last_became_peered":"2024-03-28T17:46:11.787047+0000","last_unstale":"2024-03-28T17:47:09.464912+0000","last_undegraded":"2024-03-28T17:47:09.464912+0000","last_fullsized":"2024-03-28T17:47:09.464912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:33.657488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.652865+0000","last_change":"2024-03-28T17:47:09.480465+0000","last_active":"2024-03-28T17:47:37.652865+0000","last_peered":"2024-03-28T17:47:37.652865+0000","last_clean":"2024-03-28T17:47:37.652865+0000","last_became_active":"2024-03-28T17:47:09.475758+0000","last_became_peered":"2024-03-28T17:47:09.475758+0000","last_unstale":"2024-03-28T17:47:37.652865+0000","last_undegraded":"2024-03-28T17:47:37.652865+0000","last_fullsized":"2024-03-28T17:47:37.652865+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:08.610192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.473185+0000","last_change":"2024-03-28T17:46:11.780388+0000","last_active":"2024-03-28T17:47:09.473185+0000","last_peered":"2024-03-28T17:47:09.473185+0000","last_clean":"2024-03-28T17:47:09.473185+0000","last_became_active":"2024-03-28T17:46:11.780016+0000","last_became_peered":"2024-03-28T17:46:11.780016+0000","last_unstale":"2024-03-28T17:47:09.473185+0000","last_undegraded":"2024-03-28T17:47:09.473185+0000","last_fullsized":"2024-03-28T17:47:09.473185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:51:06.181772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.651895+0000","last_change":"2024-03-28T17:46:12.764748+0000","last_active":"2024-03-28T17:47:37.651895+0000","last_peered":"2024-03-28T17:47:37.651895+0000","last_clean":"2024-03-28T17:47:37.651895+0000","last_became_active":"2024-03-28T17:46:12.764407+0000","last_became_peered":"2024-03-28T17:46:12.764407+0000","last_unstale":"2024-03-28T17:47:37.651895+0000","last_undegraded":"2024-03-28T17:47:37.651895+0000","last_fullsized":"2024-03-28T17:47:37.651895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:41:54.185079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.652654+0000","last_change":"2024-03-28T17:46:12.762486+0000","last_active":"2024-03-28T17:47:37.652654+0000","last_peered":"2024-03-28T17:47:37.652654+0000","last_clean":"2024-03-28T17:47:37.652654+0000","last_became_active":"2024-03-28T17:46:12.762107+0000","last_became_peered":"2024-03-28T17:46:12.762107+0000","last_unstale":"2024-03-28T17:47:37.652654+0000","last_undegraded":"2024-03-28T17:47:37.652654+0000","last_fullsized":"2024-03-28T17:47:37.652654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:31.062456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.664866+0000","last_change":"2024-03-28T17:46:12.764569+0000","last_active":"2024-03-28T17:47:37.664866+0000","last_peered":"2024-03-28T17:47:37.664866+0000","last_clean":"2024-03-28T17:47:37.664866+0000","last_became_active":"2024-03-28T17:46:12.764112+0000","last_became_peered":"2024-03-28T17:46:12.764112+0000","last_unstale":"2024-03-28T17:47:37.664866+0000","last_undegraded":"2024-03-28T17:47:37.664866+0000","last_fullsized":"2024-03-28T17:47:37.664866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:39:55.526432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.684845+0000","last_change":"2024-03-28T17:46:12.786031+0000","last_active":"2024-03-28T17:47:37.684845+0000","last_peered":"2024-03-28T17:47:37.684845+0000","last_clean":"2024-03-28T17:47:37.684845+0000","last_became_active":"2024-03-28T17:46:12.785679+0000","last_became_peered":"2024-03-28T17:46:12.785679+0000","last_unstale":"2024-03-28T17:47:37.684845+0000","last_undegraded":"2024-03-28T17:47:37.684845+0000","last_fullsized":"2024-03-28T17:47:37.684845+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:41:22.537979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685174+0000","last_change":"2024-03-28T17:46:12.793285+0000","last_active":"2024-03-28T17:47:37.685174+0000","last_peered":"2024-03-28T17:47:37.685174+0000","last_clean":"2024-03-28T17:47:37.685174+0000","last_became_active":"2024-03-28T17:46:12.788935+0000","last_became_peered":"2024-03-28T17:46:12.788935+0000","last_unstale":"2024-03-28T17:47:37.685174+0000","last_undegraded":"2024-03-28T17:47:37.685174+0000","last_fullsized":"2024-03-28T17:47:37.685174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:53:58.886607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687008+0000","last_change":"2024-03-28T17:47:09.486124+0000","last_active":"2024-03-28T17:47:37.687008+0000","last_peered":"2024-03-28T17:47:37.687008+0000","last_clean":"2024-03-28T17:47:37.687008+0000","last_became_active":"2024-03-28T17:47:09.481292+0000","last_became_peered":"2024-03-28T17:47:09.481292+0000","last_unstale":"2024-03-28T17:47:37.687008+0000","last_undegraded":"2024-03-28T17:47:37.687008+0000","last_fullsized":"2024-03-28T17:47:37.687008+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:18.656764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.645140+0000","last_change":"2024-03-28T17:46:12.765140+0000","last_active":"2024-03-28T17:47:37.645140+0000","last_peered":"2024-03-28T17:47:37.645140+0000","last_clean":"2024-03-28T17:47:37.645140+0000","last_became_active":"2024-03-28T17:46:12.764760+0000","last_became_peered":"2024-03-28T17:46:12.764760+0000","last_unstale":"2024-03-28T17:47:37.645140+0000","last_undegraded":"2024-03-28T17:47:37.645140+0000","last_fullsized":"2024-03-28T17:47:37.645140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:41.576437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.658894+0000","last_change":"2024-03-28T17:46:12.760271+0000","last_active":"2024-03-28T17:47:37.658894+0000","last_peered":"2024-03-28T17:47:37.658894+0000","last_clean":"2024-03-28T17:47:37.658894+0000","last_became_active":"2024-03-28T17:46:12.759939+0000","last_became_peered":"2024-03-28T17:46:12.759939+0000","last_unstale":"2024-03-28T17:47:37.658894+0000","last_undegraded":"2024-03-28T17:47:37.658894+0000","last_fullsized":"2024-03-28T17:47:37.658894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:06:42.529826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.681026+0000","last_change":"2024-03-28T17:46:12.792360+0000","last_active":"2024-03-28T17:47:37.681026+0000","last_peered":"2024-03-28T17:47:37.681026+0000","last_clean":"2024-03-28T17:47:37.681026+0000","last_became_active":"2024-03-28T17:46:12.791979+0000","last_became_peered":"2024-03-28T17:46:12.791979+0000","last_unstale":"2024-03-28T17:47:37.681026+0000","last_undegraded":"2024-03-28T17:47:37.681026+0000","last_fullsized":"2024-03-28T17:47:37.681026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:39.866969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685045+0000","last_change":"2024-03-28T17:46:12.784228+0000","last_active":"2024-03-28T17:47:37.685045+0000","last_peered":"2024-03-28T17:47:37.685045+0000","last_clean":"2024-03-28T17:47:37.685045+0000","last_became_active":"2024-03-28T17:46:12.783890+0000","last_became_peered":"2024-03-28T17:46:12.783890+0000","last_unstale":"2024-03-28T17:47:37.685045+0000","last_undegraded":"2024-03-28T17:47:37.685045+0000","last_fullsized":"2024-03-28T17:47:37.685045+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:58:13.388705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685578+0000","last_change":"2024-03-28T17:46:12.782675+0000","last_active":"2024-03-28T17:47:37.685578+0000","last_peered":"2024-03-28T17:47:37.685578+0000","last_clean":"2024-03-28T17:47:37.685578+0000","last_became_active":"2024-03-28T17:46:12.779255+0000","last_became_peered":"2024-03-28T17:46:12.779255+0000","last_unstale":"2024-03-28T17:47:37.685578+0000","last_undegraded":"2024-03-28T17:47:37.685578+0000","last_fullsized":"2024-03-28T17:47:37.685578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:04:31.447512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.671485+0000","last_change":"2024-03-28T17:46:12.786308+0000","last_active":"2024-03-28T17:47:37.671485+0000","last_peered":"2024-03-28T17:47:37.671485+0000","last_clean":"2024-03-28T17:47:37.671485+0000","last_became_active":"2024-03-28T17:46:12.785948+0000","last_became_peered":"2024-03-28T17:46:12.785948+0000","last_unstale":"2024-03-28T17:47:37.671485+0000","last_undegraded":"2024-03-28T17:47:37.671485+0000","last_fullsized":"2024-03-28T17:47:37.671485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:43.292925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687349+0000","last_change":"2024-03-28T17:46:12.797278+0000","last_active":"2024-03-28T17:47:37.687349+0000","last_peered":"2024-03-28T17:47:37.687349+0000","last_clean":"2024-03-28T17:47:37.687349+0000","last_became_active":"2024-03-28T17:46:12.796593+0000","last_became_peered":"2024-03-28T17:46:12.796593+0000","last_unstale":"2024-03-28T17:47:37.687349+0000","last_undegraded":"2024-03-28T17:47:37.687349+0000","last_fullsized":"2024-03-28T17:47:37.687349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:07:34.433680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.638776+0000","last_change":"2024-03-28T17:46:12.764249+0000","last_active":"2024-03-28T17:47:37.638776+0000","last_peered":"2024-03-28T17:47:37.638776+0000","last_clean":"2024-03-28T17:47:37.638776+0000","last_became_active":"2024-03-28T17:46:12.763909+0000","last_became_peered":"2024-03-28T17:46:12.763909+0000","last_unstale":"2024-03-28T17:47:37.638776+0000","last_undegraded":"2024-03-28T17:47:37.638776+0000","last_fullsized":"2024-03-28T17:47:37.638776+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:40:51.780187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250098,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65796,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":548,"num_read_kb":344,"num_write":170,"num_write_kb":119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":197388,"data_compressed":5625,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":147,"ondisk_log_size":147,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383714,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":456197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645350,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":1109332,"data_compressed":38073,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808877,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":456550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070514,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":456353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332151,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":1048956,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495677,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":455822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":1003520,"data_stored":516737,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":457093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":518619,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182478,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":456016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920841,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":456791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757315,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":1048108,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34554,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34554,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":61122,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":63004,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":63232,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T17:47:47.847 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-28T17:47:47.847 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-28T17:47:47.847 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-28T17:47:47.847 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-28T17:47:48.365 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:47:48.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:48 smithi045 ceph-mon[31773]: from='client.24874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:47:48.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:48 smithi071 ceph-mon[36597]: from='client.24874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:47:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:48 smithi151 ceph-mon[36481]: from='client.24874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:47:49.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:49 smithi045 ceph-mon[31773]: pgmap v367: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-28T17:47:49.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:49 smithi071 ceph-mon[36597]: pgmap v367: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-28T17:47:49.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:49 smithi151 ceph-mon[36481]: pgmap v367: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-28T17:47:50.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.570+0000 7f6ca3400700 1 -- 172.21.15.45:0/1578711599 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c9c102440 msgr2=0x7f6c9c102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:50.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.570+0000 7f6ca3400700 1 --2- 172.21.15.45:0/1578711599 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c9c102440 0x7f6c9c102820 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6c90009990 tx=0x7f6c90009ca0 comp rx=0 tx=0).stop 2024-03-28T17:47:50.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.571+0000 7f6ca3400700 1 -- 172.21.15.45:0/1578711599 shutdown_connections 2024-03-28T17:47:50.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.571+0000 7f6ca3400700 1 --2- 172.21.15.45:0/1578711599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c9c107510 0x7f6c9c10f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:50.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.571+0000 7f6ca3400700 1 --2- 172.21.15.45:0/1578711599 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c9c102440 0x7f6c9c102820 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:50.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.571+0000 7f6ca3400700 1 --2- 172.21.15.45:0/1578711599 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6c9c102df0 0x7f6c9c106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:50.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.571+0000 7f6ca3400700 1 -- 172.21.15.45:0/1578711599 >> 172.21.15.45:0/1578711599 conn(0x7f6c9c0792f0 msgr2=0x7f6c9c079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:50.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.571+0000 7f6ca3400700 1 -- 172.21.15.45:0/1578711599 shutdown_connections 2024-03-28T17:47:50.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.571+0000 7f6ca3400700 1 -- 172.21.15.45:0/1578711599 wait complete. 2024-03-28T17:47:50.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.572+0000 7f6ca3400700 1 Processor -- start 2024-03-28T17:47:50.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.572+0000 7f6ca3400700 1 -- start start 2024-03-28T17:47:50.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.572+0000 7f6ca3400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c9c102440 0x7f6c9c1a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:50.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.573+0000 7f6ca3400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c9c102df0 0x7f6c9c1a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:50.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.573+0000 7f6ca3400700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6c9c107510 0x7f6c9c19aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:50.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.573+0000 7f6ca3400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c112560 con 0x7f6c9c102df0 2024-03-28T17:47:50.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.573+0000 7f6ca3400700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c1123e0 con 0x7f6c9c102440 2024-03-28T17:47:50.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.573+0000 7f6ca3400700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c1126e0 con 0x7f6c9c107510 2024-03-28T17:47:50.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.573+0000 7f6ca0a00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c9c102440 0x7f6c9c1a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:50.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.573+0000 7f6ca0a00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c9c102440 0x7f6c9c1a0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:51072/0 (socket says 172.21.15.45:51072) 2024-03-28T17:47:50.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.573+0000 7f6ca0a00700 1 -- 172.21.15.45:0/1893649297 learned_addr learned my addr 172.21.15.45:0/1893649297 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:50.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.573+0000 7f6ca1400700 1 --2- 172.21.15.45:0/1893649297 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6c9c107510 0x7f6c9c19aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:50.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.573+0000 7f6c9be00700 1 --2- 172.21.15.45:0/1893649297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c9c102df0 0x7f6c9c1a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:50.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.573+0000 7f6ca0a00700 1 -- 172.21.15.45:0/1893649297 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6c9c107510 msgr2=0x7f6c9c19aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:50.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.573+0000 7f6ca0a00700 1 --2- 172.21.15.45:0/1893649297 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6c9c107510 0x7f6c9c19aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:50.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.573+0000 7f6ca0a00700 1 -- 172.21.15.45:0/1893649297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c9c102df0 msgr2=0x7f6c9c1a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:50.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.573+0000 7f6ca0a00700 1 --2- 172.21.15.45:0/1893649297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c9c102df0 0x7f6c9c1a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:50.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.573+0000 7f6ca0a00700 1 -- 172.21.15.45:0/1893649297 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9c19b1b0 con 0x7f6c9c102440 2024-03-28T17:47:50.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.573+0000 7f6ca0a00700 1 --2- 172.21.15.45:0/1893649297 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c9c102440 0x7f6c9c1a0890 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6c90005910 tx=0x7f6c900047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:50.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.574+0000 7f6c99600700 1 -- 172.21.15.45:0/1893649297 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90027070 con 0x7f6c9c102440 2024-03-28T17:47:50.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.574+0000 7f6c99600700 1 -- 172.21.15.45:0/1893649297 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6c9000fde0 con 0x7f6c9c102440 2024-03-28T17:47:50.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.574+0000 7f6ca3400700 1 -- 172.21.15.45:0/1893649297 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c9c19b440 con 0x7f6c9c102440 2024-03-28T17:47:50.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.574+0000 7f6c99600700 1 -- 172.21.15.45:0/1893649297 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c900173f0 con 0x7f6c9c102440 2024-03-28T17:47:50.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.574+0000 7f6ca3400700 1 -- 172.21.15.45:0/1893649297 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9c1a7760 con 0x7f6c9c102440 2024-03-28T17:47:50.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.575+0000 7f6ca3400700 1 -- 172.21.15.45:0/1893649297 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c9c068e50 con 0x7f6c9c102440 2024-03-28T17:47:50.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.577+0000 7f6c99600700 1 -- 172.21.15.45:0/1893649297 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6c9002a070 con 0x7f6c9c102440 2024-03-28T17:47:50.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.578+0000 7f6c99600700 1 --2- 172.21.15.45:0/1893649297 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6c840a2910 0x7f6c840a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:50.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.578+0000 7f6c99600700 1 -- 172.21.15.45:0/1893649297 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6c900cf690 con 0x7f6c9c102440 2024-03-28T17:47:50.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.579+0000 7f6c9be00700 1 --2- 172.21.15.45:0/1893649297 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6c840a2910 0x7f6c840a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:50.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.579+0000 7f6c9be00700 1 --2- 172.21.15.45:0/1893649297 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6c840a2910 0x7f6c840a4dd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c10f530 tx=0x7f6c88009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:50.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.587+0000 7f6c99600700 1 -- 172.21.15.45:0/1893649297 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6c90013070 con 0x7f6c9c102440 2024-03-28T17:47:50.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.868+0000 7f6ca3400700 1 -- 172.21.15.45:0/1893649297 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6c9c19bf80 con 0x7f6c9c102440 2024-03-28T17:47:50.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.869+0000 7f6c99600700 1 -- 172.21.15.45:0/1893649297 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f6c9009c770 con 0x7f6c9c102440 2024-03-28T17:47:50.869 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:47:50.870 INFO:teuthology.orchestra.run.smithi045.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-28T17:47:50.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.871+0000 7f6ca3400700 1 -- 172.21.15.45:0/1893649297 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6c840a2910 msgr2=0x7f6c840a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:50.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.871+0000 7f6ca3400700 1 --2- 172.21.15.45:0/1893649297 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6c840a2910 0x7f6c840a4dd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c10f530 tx=0x7f6c88009260 comp rx=0 tx=0).stop 2024-03-28T17:47:50.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.872+0000 7f6ca3400700 1 -- 172.21.15.45:0/1893649297 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c9c102440 msgr2=0x7f6c9c1a0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:50.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.872+0000 7f6ca3400700 1 --2- 172.21.15.45:0/1893649297 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c9c102440 0x7f6c9c1a0890 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6c90005910 tx=0x7f6c900047b0 comp rx=0 tx=0).stop 2024-03-28T17:47:50.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.872+0000 7f6ca3400700 1 -- 172.21.15.45:0/1893649297 shutdown_connections 2024-03-28T17:47:50.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.872+0000 7f6ca3400700 1 --2- 172.21.15.45:0/1893649297 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6c840a2910 0x7f6c840a4dd0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:50.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.872+0000 7f6ca3400700 1 --2- 172.21.15.45:0/1893649297 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6c9c107510 0x7f6c9c19aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:50.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.872+0000 7f6ca3400700 1 --2- 172.21.15.45:0/1893649297 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c9c102440 0x7f6c9c1a0890 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:50.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.872+0000 7f6ca3400700 1 --2- 172.21.15.45:0/1893649297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c9c102df0 0x7f6c9c1a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:50.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.872+0000 7f6ca3400700 1 -- 172.21.15.45:0/1893649297 >> 172.21.15.45:0/1893649297 conn(0x7f6c9c0792f0 msgr2=0x7f6c9c1086e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:50.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.872+0000 7f6ca3400700 1 -- 172.21.15.45:0/1893649297 shutdown_connections 2024-03-28T17:47:50.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:50.872+0000 7f6ca3400700 1 -- 172.21.15.45:0/1893649297 wait complete. 2024-03-28T17:47:51.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:51 smithi045 ceph-mon[31773]: pgmap v368: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-28T17:47:51.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:51 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1893649297' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T17:47:51.775 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-28T17:47:51.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T17:47:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:51 smithi071 ceph-mon[36597]: pgmap v368: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-28T17:47:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:51 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1893649297' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T17:47:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:51 smithi151 ceph-mon[36481]: pgmap v368: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-28T17:47:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:51 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1893649297' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T17:47:52.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.011+0000 7f80d6200640 1 -- 172.21.15.45:0/4008676169 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80d0108ed0 msgr2=0x7f80d0109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:52.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.011+0000 7f80d6200640 1 --2- 172.21.15.45:0/4008676169 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80d0108ed0 0x7f80d0109350 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f80b8009900 tx=0x7f80b802bc50 comp rx=0 tx=0).stop 2024-03-28T17:47:52.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.011+0000 7f80d6200640 1 -- 172.21.15.45:0/4008676169 shutdown_connections 2024-03-28T17:47:52.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.011+0000 7f80d6200640 1 --2- 172.21.15.45:0/4008676169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80d0109890 0x7f80d0110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.011+0000 7f80d6200640 1 --2- 172.21.15.45:0/4008676169 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80d0108ed0 0x7f80d0109350 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.011+0000 7f80d6200640 1 --2- 172.21.15.45:0/4008676169 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80d0107ce0 0x7f80d0108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.011+0000 7f80d6200640 1 -- 172.21.15.45:0/4008676169 >> 172.21.15.45:0/4008676169 conn(0x7f80d0076cb0 msgr2=0x7f80d00790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:52.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.011+0000 7f80d6200640 1 -- 172.21.15.45:0/4008676169 shutdown_connections 2024-03-28T17:47:52.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.011+0000 7f80d6200640 1 -- 172.21.15.45:0/4008676169 wait complete. 2024-03-28T17:47:52.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.012+0000 7f80d6200640 1 Processor -- start 2024-03-28T17:47:52.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.013+0000 7f80d6200640 1 -- start start 2024-03-28T17:47:52.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.013+0000 7f80d6200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80d0107ce0 0x7f80d019df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:52.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.013+0000 7f80d6200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80d0108ed0 0x7f80d019e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:52.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.013+0000 7f80d6200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80d0109890 0x7f80d01a5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:52.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.013+0000 7f80d6200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d0113990 con 0x7f80d0107ce0 2024-03-28T17:47:52.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.013+0000 7f80d6200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d0113810 con 0x7f80d0108ed0 2024-03-28T17:47:52.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.013+0000 7f80d6200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d0113b10 con 0x7f80d0109890 2024-03-28T17:47:52.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.013+0000 7f80cf400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80d0107ce0 0x7f80d019df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:52.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.013+0000 7f80cf400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80d0107ce0 0x7f80d019df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44500/0 (socket says 172.21.15.45:44500) 2024-03-28T17:47:52.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.013+0000 7f80c7e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80d0108ed0 0x7f80d019e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:52.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.013+0000 7f80cfe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80d0109890 0x7f80d01a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:52.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.013+0000 7f80cf400640 1 -- 172.21.15.45:0/2021078308 learned_addr learned my addr 172.21.15.45:0/2021078308 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:52.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.014+0000 7f80cfe00640 1 -- 172.21.15.45:0/2021078308 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80d0108ed0 msgr2=0x7f80d019e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:52.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.014+0000 7f80cfe00640 1 --2- 172.21.15.45:0/2021078308 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80d0108ed0 0x7f80d019e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.014+0000 7f80cfe00640 1 -- 172.21.15.45:0/2021078308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80d0107ce0 msgr2=0x7f80d019df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:52.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.014+0000 7f80cfe00640 1 --2- 172.21.15.45:0/2021078308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80d0107ce0 0x7f80d019df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.014+0000 7f80cfe00640 1 -- 172.21.15.45:0/2021078308 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d01a5b90 con 0x7f80d0109890 2024-03-28T17:47:52.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.014+0000 7f80c7e00640 1 --2- 172.21.15.45:0/2021078308 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80d0108ed0 0x7f80d019e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:47:52.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.014+0000 7f80cfe00640 1 --2- 172.21.15.45:0/2021078308 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80d0109890 0x7f80d01a5520 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f80c8009fd0 tx=0x7f80c800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:52.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.014+0000 7f80ccc00640 1 -- 172.21.15.45:0/2021078308 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c800ce20 con 0x7f80d0109890 2024-03-28T17:47:52.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.014+0000 7f80ccc00640 1 -- 172.21.15.45:0/2021078308 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80c8005020 con 0x7f80d0109890 2024-03-28T17:47:52.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.014+0000 7f80ccc00640 1 -- 172.21.15.45:0/2021078308 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c8010720 con 0x7f80d0109890 2024-03-28T17:47:52.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.014+0000 7f80d6200640 1 -- 172.21.15.45:0/2021078308 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80d01a5e80 con 0x7f80d0109890 2024-03-28T17:47:52.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.015+0000 7f80d6200640 1 -- 172.21.15.45:0/2021078308 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80d01a62f0 con 0x7f80d0109890 2024-03-28T17:47:52.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.016+0000 7f80d6200640 1 -- 172.21.15.45:0/2021078308 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80d010efa0 con 0x7f80d0109890 2024-03-28T17:47:52.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.017+0000 7f80ccc00640 1 -- 172.21.15.45:0/2021078308 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f80c8005540 con 0x7f80d0109890 2024-03-28T17:47:52.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.019+0000 7f80ccc00640 1 --2- 172.21.15.45:0/2021078308 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80b40a2880 0x7f80b40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:52.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.019+0000 7f80ccc00640 1 -- 172.21.15.45:0/2021078308 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f80c80cf090 con 0x7f80d0109890 2024-03-28T17:47:52.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.019+0000 7f80cf400640 1 --2- 172.21.15.45:0/2021078308 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80b40a2880 0x7f80b40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:52.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.020+0000 7f80cf400640 1 --2- 172.21.15.45:0/2021078308 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80b40a2880 0x7f80b40a4d40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f80c00058c0 tx=0x7f80c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:52.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.023+0000 7f80ccc00640 1 -- 172.21.15.45:0/2021078308 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f80c809c140 con 0x7f80d0109890 2024-03-28T17:47:52.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.229+0000 7f80d6200640 1 -- 172.21.15.45:0/2021078308 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80d002d2b0 con 0x7f80d0109890 2024-03-28T17:47:52.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.231+0000 7f80ccc00640 1 -- 172.21.15.45:0/2021078308 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22583 (secure 0 0 0) 0x7f80c809ba40 con 0x7f80d0109890 2024-03-28T17:47:52.231 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:47:52.232 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"9f4888ec-ed29-11ee-95d0-87774f69a715","created":"2024-03-28T17:36:51.623185+0000","modified":"2024-03-28T17:47:09.455843+0000","last_up_change":"2024-03-28T17:45:14.043432+0000","last_in_change":"2024-03-28T17:44:53.654489+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T17:40:54.208375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T17:46:10.564519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T17:46:11.734367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"74c56bc2-4169-4919-afd4-2ebd236f23e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6803","nonce":3795754023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6805","nonce":3795754023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6809","nonce":3795754023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6807","nonce":3795754023}]},"public_addr":"172.21.15.45:6803/3795754023","cluster_addr":"172.21.15.45:6805/3795754023","heartbeat_back_addr":"172.21.15.45:6809/3795754023","heartbeat_front_addr":"172.21.15.45:6807/3795754023","state":["exists","up"]},{"osd":1,"uuid":"78dcc21b-75b2-453d-b690-17b55b8ed1a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6801","nonce":2014529906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6803","nonce":2014529906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6807","nonce":2014529906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6805","nonce":2014529906}]},"public_addr":"172.21.15.71:6801/2014529906","cluster_addr":"172.21.15.71:6803/2014529906","heartbeat_back_addr":"172.21.15.71:6807/2014529906","heartbeat_front_addr":"172.21.15.71:6805/2014529906","state":["exists","up"]},{"osd":2,"uuid":"ca9f3f36-b71e-4962-a94f-cedf2f12eb7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6801","nonce":1850933775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6803","nonce":1850933775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6807","nonce":1850933775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6805","nonce":1850933775}]},"public_addr":"172.21.15.151:6801/1850933775","cluster_addr":"172.21.15.151:6803/1850933775","heartbeat_back_addr":"172.21.15.151:6807/1850933775","heartbeat_front_addr":"172.21.15.151:6805/1850933775","state":["exists","up"]},{"osd":3,"uuid":"d3f21f01-2973-413e-8daa-9974b8a44af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6811","nonce":3021569975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6813","nonce":3021569975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6817","nonce":3021569975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6815","nonce":3021569975}]},"public_addr":"172.21.15.45:6811/3021569975","cluster_addr":"172.21.15.45:6813/3021569975","heartbeat_back_addr":"172.21.15.45:6817/3021569975","heartbeat_front_addr":"172.21.15.45:6815/3021569975","state":["exists","up"]},{"osd":4,"uuid":"4a336f42-0605-492b-ad88-56cb02acdcf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6809","nonce":2069578333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6811","nonce":2069578333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6815","nonce":2069578333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6813","nonce":2069578333}]},"public_addr":"172.21.15.71:6809/2069578333","cluster_addr":"172.21.15.71:6811/2069578333","heartbeat_back_addr":"172.21.15.71:6815/2069578333","heartbeat_front_addr":"172.21.15.71:6813/2069578333","state":["exists","up"]},{"osd":5,"uuid":"c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6809","nonce":2629217227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6811","nonce":2629217227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6815","nonce":2629217227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6813","nonce":2629217227}]},"public_addr":"172.21.15.151:6809/2629217227","cluster_addr":"172.21.15.151:6811/2629217227","heartbeat_back_addr":"172.21.15.151:6815/2629217227","heartbeat_front_addr":"172.21.15.151:6813/2629217227","state":["exists","up"]},{"osd":6,"uuid":"58a6c5bd-a8b7-4892-b346-bcd7994bcb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6819","nonce":2077907039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6821","nonce":2077907039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6825","nonce":2077907039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6823","nonce":2077907039}]},"public_addr":"172.21.15.45:6819/2077907039","cluster_addr":"172.21.15.45:6821/2077907039","heartbeat_back_addr":"172.21.15.45:6825/2077907039","heartbeat_front_addr":"172.21.15.45:6823/2077907039","state":["exists","up"]},{"osd":7,"uuid":"f7a45d52-807b-4407-bb04-787b2a52a0ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6817","nonce":2000211251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6819","nonce":2000211251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6823","nonce":2000211251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6821","nonce":2000211251}]},"public_addr":"172.21.15.71:6817/2000211251","cluster_addr":"172.21.15.71:6819/2000211251","heartbeat_back_addr":"172.21.15.71:6823/2000211251","heartbeat_front_addr":"172.21.15.71:6821/2000211251","state":["exists","up"]},{"osd":8,"uuid":"88abcb0f-f189-49be-8315-c700c77369f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6817","nonce":3665460137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6819","nonce":3665460137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6823","nonce":3665460137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6821","nonce":3665460137}]},"public_addr":"172.21.15.151:6817/3665460137","cluster_addr":"172.21.15.151:6819/3665460137","heartbeat_back_addr":"172.21.15.151:6823/3665460137","heartbeat_front_addr":"172.21.15.151:6821/3665460137","state":["exists","up"]},{"osd":9,"uuid":"ed3fc096-961e-4392-abc2-c5aa4d050117","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6827","nonce":2122322684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6829","nonce":2122322684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6833","nonce":2122322684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6831","nonce":2122322684}]},"public_addr":"172.21.15.45:6827/2122322684","cluster_addr":"172.21.15.45:6829/2122322684","heartbeat_back_addr":"172.21.15.45:6833/2122322684","heartbeat_front_addr":"172.21.15.45:6831/2122322684","state":["exists","up"]},{"osd":10,"uuid":"ba94762b-bc5c-4c97-9cc1-17e5651c14b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6825","nonce":1099494037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6827","nonce":1099494037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6831","nonce":1099494037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6829","nonce":1099494037}]},"public_addr":"172.21.15.71:6825/1099494037","cluster_addr":"172.21.15.71:6827/1099494037","heartbeat_back_addr":"172.21.15.71:6831/1099494037","heartbeat_front_addr":"172.21.15.71:6829/1099494037","state":["exists","up"]},{"osd":11,"uuid":"1d178378-b9f5-40f6-99ef-3d3d445da640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6825","nonce":2941055364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6827","nonce":2941055364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6831","nonce":2941055364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6829","nonce":2941055364}]},"public_addr":"172.21.15.151:6825/2941055364","cluster_addr":"172.21.15.151:6827/2941055364","heartbeat_back_addr":"172.21.15.151:6831/2941055364","heartbeat_front_addr":"172.21.15.151:6829/2941055364","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:02.269586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:27.819386+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:51.552277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:25.052674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:51.873529+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:17.127867+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:51.588249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:18.381450+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:44.140534+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:17.634362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:45.469340+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:45:12.003433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3533611957":"2024-03-29T17:38:07.505611+0000","172.21.15.45:0/3625922430":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6801/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/482610468":"2024-03-29T17:37:32.424229+0000","172.21.15.45:6801/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3649804714":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/1198364079":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3211303958":"2024-03-29T17:38:07.505611+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T17:47:52.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.234+0000 7f80d6200640 1 -- 172.21.15.45:0/2021078308 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80b40a2880 msgr2=0x7f80b40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:52.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.234+0000 7f80d6200640 1 --2- 172.21.15.45:0/2021078308 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80b40a2880 0x7f80b40a4d40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f80c00058c0 tx=0x7f80c0005f90 comp rx=0 tx=0).stop 2024-03-28T17:47:52.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.234+0000 7f80d6200640 1 -- 172.21.15.45:0/2021078308 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80d0109890 msgr2=0x7f80d01a5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:52.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.234+0000 7f80d6200640 1 --2- 172.21.15.45:0/2021078308 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80d0109890 0x7f80d01a5520 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f80c8009fd0 tx=0x7f80c800ef10 comp rx=0 tx=0).stop 2024-03-28T17:47:52.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.234+0000 7f80d6200640 1 -- 172.21.15.45:0/2021078308 shutdown_connections 2024-03-28T17:47:52.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.234+0000 7f80d6200640 1 --2- 172.21.15.45:0/2021078308 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80b40a2880 0x7f80b40a4d40 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.234+0000 7f80d6200640 1 --2- 172.21.15.45:0/2021078308 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80d0109890 0x7f80d01a5520 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.234+0000 7f80d6200640 1 --2- 172.21.15.45:0/2021078308 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80d0108ed0 0x7f80d019e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.234+0000 7f80d6200640 1 --2- 172.21.15.45:0/2021078308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80d0107ce0 0x7f80d019df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.234+0000 7f80d6200640 1 -- 172.21.15.45:0/2021078308 >> 172.21.15.45:0/2021078308 conn(0x7f80d0076cb0 msgr2=0x7f80d010c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:52.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.234+0000 7f80d6200640 1 -- 172.21.15.45:0/2021078308 shutdown_connections 2024-03-28T17:47:52.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.234+0000 7f80d6200640 1 -- 172.21.15.45:0/2021078308 wait complete. 2024-03-28T17:47:52.256 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T17:40:54.208375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T17:46:10.564519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T17:46:11.734367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T17:47:52.256 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T17:47:52.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.486+0000 7fa3d9200640 1 -- 172.21.15.45:0/401923715 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3d4108eb0 msgr2=0x7fa3d4109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:52.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.486+0000 7fa3d9200640 1 --2- 172.21.15.45:0/401923715 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3d4108eb0 0x7fa3d4109330 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc009900 tx=0x7fa3bc02bc30 comp rx=0 tx=0).stop 2024-03-28T17:47:52.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.486+0000 7fa3d9200640 1 -- 172.21.15.45:0/401923715 shutdown_connections 2024-03-28T17:47:52.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.486+0000 7fa3d9200640 1 --2- 172.21.15.45:0/401923715 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d4109870 0x7fa3d4110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.486+0000 7fa3d9200640 1 --2- 172.21.15.45:0/401923715 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3d4108eb0 0x7fa3d4109330 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.486+0000 7fa3d9200640 1 --2- 172.21.15.45:0/401923715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3d4107cc0 0x7fa3d41080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.486+0000 7fa3d9200640 1 -- 172.21.15.45:0/401923715 >> 172.21.15.45:0/401923715 conn(0x7fa3d4076c60 msgr2=0x7fa3d40790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:52.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.486+0000 7fa3d9200640 1 -- 172.21.15.45:0/401923715 shutdown_connections 2024-03-28T17:47:52.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.486+0000 7fa3d9200640 1 -- 172.21.15.45:0/401923715 wait complete. 2024-03-28T17:47:52.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.488+0000 7fa3d9200640 1 Processor -- start 2024-03-28T17:47:52.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.488+0000 7fa3d9200640 1 -- start start 2024-03-28T17:47:52.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.488+0000 7fa3d9200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3d4107cc0 0x7fa3d419df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:52.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.488+0000 7fa3d9200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d4108eb0 0x7fa3d419e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:52.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.488+0000 7fa3d9200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3d4109870 0x7fa3d41a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:52.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.488+0000 7fa3d9200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d4113970 con 0x7fa3d4109870 2024-03-28T17:47:52.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.489+0000 7fa3d9200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d41137f0 con 0x7fa3d4108eb0 2024-03-28T17:47:52.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.489+0000 7fa3d9200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d4113af0 con 0x7fa3d4107cc0 2024-03-28T17:47:52.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.489+0000 7fa3d1e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3d4107cc0 0x7fa3d419df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:52.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.489+0000 7fa3d1e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3d4107cc0 0x7fa3d419df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:56832/0 (socket says 172.21.15.45:56832) 2024-03-28T17:47:52.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.489+0000 7fa3d1e00640 1 -- 172.21.15.45:0/1271149687 learned_addr learned my addr 172.21.15.45:0/1271149687 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:52.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.489+0000 7fa3cbe00640 1 --2- 172.21.15.45:0/1271149687 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d4108eb0 0x7fa3d419e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:52.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.489+0000 7fa3d2800640 1 --2- 172.21.15.45:0/1271149687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3d4109870 0x7fa3d41a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:52.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.489+0000 7fa3d1e00640 1 -- 172.21.15.45:0/1271149687 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d4108eb0 msgr2=0x7fa3d419e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:52.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.489+0000 7fa3d1e00640 1 --2- 172.21.15.45:0/1271149687 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d4108eb0 0x7fa3d419e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.489+0000 7fa3d1e00640 1 -- 172.21.15.45:0/1271149687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3d4109870 msgr2=0x7fa3d41a5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:52.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.489+0000 7fa3d1e00640 1 --2- 172.21.15.45:0/1271149687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3d4109870 0x7fa3d41a5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.489+0000 7fa3d1e00640 1 -- 172.21.15.45:0/1271149687 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d41a5bb0 con 0x7fa3d4107cc0 2024-03-28T17:47:52.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.489+0000 7fa3d1e00640 1 --2- 172.21.15.45:0/1271149687 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3d4107cc0 0x7fa3d419df90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400e9c0 tx=0x7fa3c400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:52.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.490+0000 7fa3caa00640 1 -- 172.21.15.45:0/1271149687 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c400cd80 con 0x7fa3d4107cc0 2024-03-28T17:47:52.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.490+0000 7fa3caa00640 1 -- 172.21.15.45:0/1271149687 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3c40035f0 con 0x7fa3d4107cc0 2024-03-28T17:47:52.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.490+0000 7fa3caa00640 1 -- 172.21.15.45:0/1271149687 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4010690 con 0x7fa3d4107cc0 2024-03-28T17:47:52.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.490+0000 7fa3d9200640 1 -- 172.21.15.45:0/1271149687 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d41a5ea0 con 0x7fa3d4107cc0 2024-03-28T17:47:52.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.490+0000 7fa3d9200640 1 -- 172.21.15.45:0/1271149687 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d41a6250 con 0x7fa3d4107cc0 2024-03-28T17:47:52.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.491+0000 7fa3d9200640 1 -- 172.21.15.45:0/1271149687 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d410efe0 con 0x7fa3d4107cc0 2024-03-28T17:47:52.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.493+0000 7fa3caa00640 1 -- 172.21.15.45:0/1271149687 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa3c40037a0 con 0x7fa3d4107cc0 2024-03-28T17:47:52.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.494+0000 7fa3caa00640 1 --2- 172.21.15.45:0/1271149687 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fa3b80a2800 0x7fa3b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:52.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.494+0000 7fa3caa00640 1 -- 172.21.15.45:0/1271149687 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa3c40ceef0 con 0x7fa3d4107cc0 2024-03-28T17:47:52.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.494+0000 7fa3cbe00640 1 --2- 172.21.15.45:0/1271149687 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fa3b80a2800 0x7fa3b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:52.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.495+0000 7fa3cbe00640 1 --2- 172.21.15.45:0/1271149687 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fa3b80a2800 0x7fa3b80a4cc0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc02c240 tx=0x7fa3bc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:52.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.498+0000 7fa3caa00640 1 -- 172.21.15.45:0/1271149687 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa3c409bf40 con 0x7fa3d4107cc0 2024-03-28T17:47:52.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:52 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/2021078308' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:47:52.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.702+0000 7fa3d9200640 1 -- 172.21.15.45:0/1271149687 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa3d402d320 con 0x7fa3d4107cc0 2024-03-28T17:47:52.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.703+0000 7fa3caa00640 1 -- 172.21.15.45:0/1271149687 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa3c409b840 con 0x7fa3d4107cc0 2024-03-28T17:47:52.704 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-28T17:47:52.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.705+0000 7fa3d9200640 1 -- 172.21.15.45:0/1271149687 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fa3b80a2800 msgr2=0x7fa3b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:52.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.705+0000 7fa3d9200640 1 --2- 172.21.15.45:0/1271149687 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fa3b80a2800 0x7fa3b80a4cc0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc02c240 tx=0x7fa3bc02df90 comp rx=0 tx=0).stop 2024-03-28T17:47:52.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.705+0000 7fa3d9200640 1 -- 172.21.15.45:0/1271149687 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3d4107cc0 msgr2=0x7fa3d419df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:52.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.705+0000 7fa3d9200640 1 --2- 172.21.15.45:0/1271149687 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3d4107cc0 0x7fa3d419df90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400e9c0 tx=0x7fa3c400ee80 comp rx=0 tx=0).stop 2024-03-28T17:47:52.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.705+0000 7fa3d9200640 1 -- 172.21.15.45:0/1271149687 shutdown_connections 2024-03-28T17:47:52.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.705+0000 7fa3d9200640 1 --2- 172.21.15.45:0/1271149687 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fa3b80a2800 0x7fa3b80a4cc0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.705+0000 7fa3d9200640 1 --2- 172.21.15.45:0/1271149687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3d4109870 0x7fa3d41a5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.705+0000 7fa3d9200640 1 --2- 172.21.15.45:0/1271149687 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d4108eb0 0x7fa3d419e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.705+0000 7fa3d9200640 1 --2- 172.21.15.45:0/1271149687 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3d4107cc0 0x7fa3d419df90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.705+0000 7fa3d9200640 1 -- 172.21.15.45:0/1271149687 >> 172.21.15.45:0/1271149687 conn(0x7fa3d4076c60 msgr2=0x7fa3d410c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:52.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.706+0000 7fa3d9200640 1 -- 172.21.15.45:0/1271149687 shutdown_connections 2024-03-28T17:47:52.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.706+0000 7fa3d9200640 1 -- 172.21.15.45:0/1271149687 wait complete. 2024-03-28T17:47:52.727 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T17:47:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:52 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/2021078308' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:47:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:52 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/2021078308' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:47:52.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.951+0000 7f80cb200640 1 -- 172.21.15.45:0/2372771726 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80c4103150 msgr2=0x7f80c41099e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:52.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.951+0000 7f80cb200640 1 --2- 172.21.15.45:0/2372771726 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80c4103150 0x7f80c41099e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f80c0009ca0 tx=0x7f80c002c0f0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.952+0000 7f80cb200640 1 -- 172.21.15.45:0/2372771726 shutdown_connections 2024-03-28T17:47:52.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.952+0000 7f80cb200640 1 --2- 172.21.15.45:0/2372771726 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80c4103150 0x7f80c41099e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.952+0000 7f80cb200640 1 --2- 172.21.15.45:0/2372771726 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80c4102790 0x7f80c4102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.952+0000 7f80cb200640 1 --2- 172.21.15.45:0/2372771726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80c41015a0 0x7f80c41019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.952+0000 7f80cb200640 1 -- 172.21.15.45:0/2372771726 >> 172.21.15.45:0/2372771726 conn(0x7f80c40fcb10 msgr2=0x7f80c40fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:52.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.952+0000 7f80cb200640 1 -- 172.21.15.45:0/2372771726 shutdown_connections 2024-03-28T17:47:52.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.952+0000 7f80cb200640 1 -- 172.21.15.45:0/2372771726 wait complete. 2024-03-28T17:47:52.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.953+0000 7f80cb200640 1 Processor -- start 2024-03-28T17:47:52.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.953+0000 7f80cb200640 1 -- start start 2024-03-28T17:47:52.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.953+0000 7f80cb200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80c41015a0 0x7f80c4199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:52.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.953+0000 7f80cb200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80c4102790 0x7f80c4199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:52.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.953+0000 7f80cb200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80c4103150 0x7f80c41a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:52.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.953+0000 7f80cb200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c410d200 con 0x7f80c41015a0 2024-03-28T17:47:52.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.953+0000 7f80cb200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c410d080 con 0x7f80c4103150 2024-03-28T17:47:52.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.953+0000 7f80cb200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c410d380 con 0x7f80c4102790 2024-03-28T17:47:52.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.953+0000 7f80c9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80c41015a0 0x7f80c4199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:52.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.953+0000 7f80ca800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80c4103150 0x7f80c41a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:52.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.953+0000 7f80c9400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80c4102790 0x7f80c4199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:52.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.953+0000 7f80ca800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80c4103150 0x7f80c41a0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:51128/0 (socket says 172.21.15.45:51128) 2024-03-28T17:47:52.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.954+0000 7f80ca800640 1 -- 172.21.15.45:0/3118642827 learned_addr learned my addr 172.21.15.45:0/3118642827 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:52.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.954+0000 7f80ca800640 1 -- 172.21.15.45:0/3118642827 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80c4102790 msgr2=0x7f80c4199f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:52.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.954+0000 7f80ca800640 1 --2- 172.21.15.45:0/3118642827 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80c4102790 0x7f80c4199f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.954+0000 7f80ca800640 1 -- 172.21.15.45:0/3118642827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80c41015a0 msgr2=0x7f80c4199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:52.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.954+0000 7f80ca800640 1 --2- 172.21.15.45:0/3118642827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80c41015a0 0x7f80c4199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:52.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.954+0000 7f80ca800640 1 -- 172.21.15.45:0/3118642827 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c41a1650 con 0x7f80c4103150 2024-03-28T17:47:52.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.954+0000 7f80ca800640 1 --2- 172.21.15.45:0/3118642827 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80c4103150 0x7f80c41a0fe0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f80c002c630 tx=0x7f80c0032ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:52.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.954+0000 7f80bea00640 1 -- 172.21.15.45:0/3118642827 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c0041020 con 0x7f80c4103150 2024-03-28T17:47:52.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.954+0000 7f80bea00640 1 -- 172.21.15.45:0/3118642827 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80c0034cd0 con 0x7f80c4103150 2024-03-28T17:47:52.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.954+0000 7f80cb200640 1 -- 172.21.15.45:0/3118642827 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c41a18e0 con 0x7f80c4103150 2024-03-28T17:47:52.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.954+0000 7f80bea00640 1 -- 172.21.15.45:0/3118642827 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c0039040 con 0x7f80c4103150 2024-03-28T17:47:52.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.955+0000 7f80cb200640 1 -- 172.21.15.45:0/3118642827 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c41a1e20 con 0x7f80c4103150 2024-03-28T17:47:52.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.956+0000 7f80cb200640 1 -- 172.21.15.45:0/3118642827 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c41a2050 con 0x7f80c4103150 2024-03-28T17:47:52.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.957+0000 7f80bea00640 1 -- 172.21.15.45:0/3118642827 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f80c0044020 con 0x7f80c4103150 2024-03-28T17:47:52.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.959+0000 7f80bea00640 1 --2- 172.21.15.45:0/3118642827 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80ac0a2880 0x7f80ac0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:52.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.959+0000 7f80bea00640 1 -- 172.21.15.45:0/3118642827 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f80c00f0930 con 0x7f80c4103150 2024-03-28T17:47:52.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.959+0000 7f80c9e00640 1 --2- 172.21.15.45:0/3118642827 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80ac0a2880 0x7f80ac0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:52.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.960+0000 7f80c9e00640 1 --2- 172.21.15.45:0/3118642827 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80ac0a2880 0x7f80ac0a4d40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f80c41025f0 tx=0x7f80b800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:52.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:52.967+0000 7f80bea00640 1 -- 172.21.15.45:0/3118642827 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f80c004b020 con 0x7f80c4103150 2024-03-28T17:47:53.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.169+0000 7f80cb200640 1 -- 172.21.15.45:0/3118642827 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f80c41a23e0 con 0x7f80c4103150 2024-03-28T17:47:53.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.170+0000 7f80bea00640 1 -- 172.21.15.45:0/3118642827 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f80c00bd980 con 0x7f80c4103150 2024-03-28T17:47:53.171 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-03-28T17:47:53.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.173+0000 7f80cb200640 1 -- 172.21.15.45:0/3118642827 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80ac0a2880 msgr2=0x7f80ac0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:53.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.173+0000 7f80cb200640 1 --2- 172.21.15.45:0/3118642827 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80ac0a2880 0x7f80ac0a4d40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f80c41025f0 tx=0x7f80b800b390 comp rx=0 tx=0).stop 2024-03-28T17:47:53.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.173+0000 7f80cb200640 1 -- 172.21.15.45:0/3118642827 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80c4103150 msgr2=0x7f80c41a0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:53.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.173+0000 7f80cb200640 1 --2- 172.21.15.45:0/3118642827 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80c4103150 0x7f80c41a0fe0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f80c002c630 tx=0x7f80c0032ea0 comp rx=0 tx=0).stop 2024-03-28T17:47:53.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.173+0000 7f80cb200640 1 -- 172.21.15.45:0/3118642827 shutdown_connections 2024-03-28T17:47:53.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.173+0000 7f80cb200640 1 --2- 172.21.15.45:0/3118642827 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f80ac0a2880 0x7f80ac0a4d40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:53.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.173+0000 7f80cb200640 1 --2- 172.21.15.45:0/3118642827 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80c4103150 0x7f80c41a0fe0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:53.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.173+0000 7f80cb200640 1 --2- 172.21.15.45:0/3118642827 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80c4102790 0x7f80c4199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:53.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.173+0000 7f80cb200640 1 --2- 172.21.15.45:0/3118642827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80c41015a0 0x7f80c4199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:53.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.173+0000 7f80cb200640 1 -- 172.21.15.45:0/3118642827 >> 172.21.15.45:0/3118642827 conn(0x7f80c40fcb10 msgr2=0x7f80c40738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:53.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.173+0000 7f80cb200640 1 -- 172.21.15.45:0/3118642827 shutdown_connections 2024-03-28T17:47:53.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.173+0000 7f80cb200640 1 -- 172.21.15.45:0/3118642827 wait complete. 2024-03-28T17:47:53.194 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T17:47:53.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.425+0000 7f63e0c00640 1 -- 172.21.15.45:0/3206568481 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63dc1049b0 msgr2=0x7f63dc104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:53.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.425+0000 7f63e0c00640 1 --2- 172.21.15.45:0/3206568481 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63dc1049b0 0x7f63dc104e30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f63c8009900 tx=0x7f63c802bc40 comp rx=0 tx=0).stop 2024-03-28T17:47:53.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.425+0000 7f63e0c00640 1 -- 172.21.15.45:0/3206568481 shutdown_connections 2024-03-28T17:47:53.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.425+0000 7f63e0c00640 1 --2- 172.21.15.45:0/3206568481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63dc105370 0x7f63dc10bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:53.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.425+0000 7f63e0c00640 1 --2- 172.21.15.45:0/3206568481 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63dc1049b0 0x7f63dc104e30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:53.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.425+0000 7f63e0c00640 1 --2- 172.21.15.45:0/3206568481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63dc1037c0 0x7f63dc103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:53.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.425+0000 7f63e0c00640 1 -- 172.21.15.45:0/3206568481 >> 172.21.15.45:0/3206568481 conn(0x7f63dc0fed50 msgr2=0x7f63dc101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:53.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.425+0000 7f63e0c00640 1 -- 172.21.15.45:0/3206568481 shutdown_connections 2024-03-28T17:47:53.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.426+0000 7f63e0c00640 1 -- 172.21.15.45:0/3206568481 wait complete. 2024-03-28T17:47:53.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.426+0000 7f63e0c00640 1 Processor -- start 2024-03-28T17:47:53.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.427+0000 7f63e0c00640 1 -- start start 2024-03-28T17:47:53.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.427+0000 7f63e0c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63dc1037c0 0x7f63dc199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:53.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.427+0000 7f63e0c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63dc1049b0 0x7f63dc19a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:53.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.427+0000 7f63e0c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63dc105370 0x7f63dc1a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:53.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.427+0000 7f63e0c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc10f470 con 0x7f63dc1037c0 2024-03-28T17:47:53.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.427+0000 7f63e0c00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc10f2f0 con 0x7f63dc1049b0 2024-03-28T17:47:53.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.427+0000 7f63e0c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc10f5f0 con 0x7f63dc105370 2024-03-28T17:47:53.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.427+0000 7f63dbe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63dc105370 0x7f63dc1a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:53.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.427+0000 7f63daa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63dc1049b0 0x7f63dc19a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:53.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.427+0000 7f63dbe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63dc105370 0x7f63dc1a1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:56870/0 (socket says 172.21.15.45:56870) 2024-03-28T17:47:53.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.427+0000 7f63dbe00640 1 -- 172.21.15.45:0/3635284699 learned_addr learned my addr 172.21.15.45:0/3635284699 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:53.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.427+0000 7f63daa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63dc1049b0 0x7f63dc19a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:51146/0 (socket says 172.21.15.45:51146) 2024-03-28T17:47:53.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.428+0000 7f63db400640 1 --2- 172.21.15.45:0/3635284699 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63dc1037c0 0x7f63dc199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:53.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.428+0000 7f63dbe00640 1 -- 172.21.15.45:0/3635284699 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63dc1049b0 msgr2=0x7f63dc19a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:53.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.428+0000 7f63dbe00640 1 --2- 172.21.15.45:0/3635284699 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63dc1049b0 0x7f63dc19a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:53.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.428+0000 7f63dbe00640 1 -- 172.21.15.45:0/3635284699 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63dc1037c0 msgr2=0x7f63dc199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:53.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.428+0000 7f63dbe00640 1 --2- 172.21.15.45:0/3635284699 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63dc1037c0 0x7f63dc199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:53.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.428+0000 7f63dbe00640 1 -- 172.21.15.45:0/3635284699 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63dc1a16e0 con 0x7f63dc105370 2024-03-28T17:47:53.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.428+0000 7f63dbe00640 1 --2- 172.21.15.45:0/3635284699 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63dc105370 0x7f63dc1a1070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f63d4009fd0 tx=0x7f63d400b260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:53.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.428+0000 7f63d3e00640 1 -- 172.21.15.45:0/3635284699 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d4017070 con 0x7f63dc105370 2024-03-28T17:47:53.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.428+0000 7f63d3e00640 1 -- 172.21.15.45:0/3635284699 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f63d4003e50 con 0x7f63dc105370 2024-03-28T17:47:53.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.428+0000 7f63d3e00640 1 -- 172.21.15.45:0/3635284699 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d40106c0 con 0x7f63dc105370 2024-03-28T17:47:53.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.429+0000 7f63e0c00640 1 -- 172.21.15.45:0/3635284699 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63dc1a19d0 con 0x7f63dc105370 2024-03-28T17:47:53.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.429+0000 7f63e0c00640 1 -- 172.21.15.45:0/3635284699 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63dc1a1ea0 con 0x7f63dc105370 2024-03-28T17:47:53.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.430+0000 7f63e0c00640 1 -- 172.21.15.45:0/3635284699 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63dc068800 con 0x7f63dc105370 2024-03-28T17:47:53.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.431+0000 7f63d3e00640 1 -- 172.21.15.45:0/3635284699 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f63d4003970 con 0x7f63dc105370 2024-03-28T17:47:53.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.433+0000 7f63d3e00640 1 --2- 172.21.15.45:0/3635284699 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f63c00a2880 0x7f63c00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:53.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.433+0000 7f63d3e00640 1 -- 172.21.15.45:0/3635284699 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f63d40ce7c0 con 0x7f63dc105370 2024-03-28T17:47:53.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.433+0000 7f63db400640 1 --2- 172.21.15.45:0/3635284699 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f63c00a2880 0x7f63c00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:53.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.434+0000 7f63db400640 1 --2- 172.21.15.45:0/3635284699 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f63c00a2880 0x7f63c00a4d40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f63dc104810 tx=0x7f63cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:53.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.437+0000 7f63d3e00640 1 -- 172.21.15.45:0/3635284699 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f63d409b870 con 0x7f63dc105370 2024-03-28T17:47:53.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.635+0000 7f63e0c00640 1 -- 172.21.15.45:0/3635284699 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f63dc0622a0 con 0x7f63dc105370 2024-03-28T17:47:53.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.636+0000 7f63d3e00640 1 -- 172.21.15.45:0/3635284699 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f63d409b170 con 0x7f63dc105370 2024-03-28T17:47:53.636 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-03-28T17:47:53.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.638+0000 7f63e0c00640 1 -- 172.21.15.45:0/3635284699 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f63c00a2880 msgr2=0x7f63c00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:53.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.638+0000 7f63e0c00640 1 --2- 172.21.15.45:0/3635284699 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f63c00a2880 0x7f63c00a4d40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f63dc104810 tx=0x7f63cc005f90 comp rx=0 tx=0).stop 2024-03-28T17:47:53.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.638+0000 7f63e0c00640 1 -- 172.21.15.45:0/3635284699 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63dc105370 msgr2=0x7f63dc1a1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:53.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.638+0000 7f63e0c00640 1 --2- 172.21.15.45:0/3635284699 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63dc105370 0x7f63dc1a1070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f63d4009fd0 tx=0x7f63d400b260 comp rx=0 tx=0).stop 2024-03-28T17:47:53.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.639+0000 7f63e0c00640 1 -- 172.21.15.45:0/3635284699 shutdown_connections 2024-03-28T17:47:53.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.639+0000 7f63e0c00640 1 --2- 172.21.15.45:0/3635284699 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f63c00a2880 0x7f63c00a4d40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:53.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.639+0000 7f63e0c00640 1 --2- 172.21.15.45:0/3635284699 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63dc105370 0x7f63dc1a1070 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:53.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.639+0000 7f63e0c00640 1 --2- 172.21.15.45:0/3635284699 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63dc1049b0 0x7f63dc19a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:53.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.639+0000 7f63e0c00640 1 --2- 172.21.15.45:0/3635284699 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63dc1037c0 0x7f63dc199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:53.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.639+0000 7f63e0c00640 1 -- 172.21.15.45:0/3635284699 >> 172.21.15.45:0/3635284699 conn(0x7f63dc0fed50 msgr2=0x7f63dc107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:53.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.639+0000 7f63e0c00640 1 -- 172.21.15.45:0/3635284699 shutdown_connections 2024-03-28T17:47:53.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.639+0000 7f63e0c00640 1 -- 172.21.15.45:0/3635284699 wait complete. 2024-03-28T17:47:53.654 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:53 smithi045 ceph-mon[31773]: pgmap v369: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T17:47:53.655 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:53 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1271149687' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:47:53.655 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:53 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3118642827' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T17:47:53.659 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:47:53.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:53 smithi151 ceph-mon[36481]: pgmap v369: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T17:47:53.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:53 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1271149687' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:47:53.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:53 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3118642827' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T17:47:53.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:53 smithi071 ceph-mon[36597]: pgmap v369: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T17:47:53.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:53 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1271149687' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:47:53.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:53 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3118642827' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T17:47:53.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.878+0000 7f668ce00640 1 -- 172.21.15.45:0/1717885636 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6688076060 msgr2=0x7f6688076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:53.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.878+0000 7f668ce00640 1 --2- 172.21.15.45:0/1717885636 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6688076060 0x7f6688076480 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6678009900 tx=0x7f667802bc30 comp rx=0 tx=0).stop 2024-03-28T17:47:53.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.878+0000 7f668ce00640 1 -- 172.21.15.45:0/1717885636 shutdown_connections 2024-03-28T17:47:53.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.878+0000 7f668ce00640 1 --2- 172.21.15.45:0/1717885636 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6688077c40 0x7f66880784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:53.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.878+0000 7f668ce00640 1 --2- 172.21.15.45:0/1717885636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66880772a0 0x7f6688077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:53.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.878+0000 7f668ce00640 1 --2- 172.21.15.45:0/1717885636 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6688076060 0x7f6688076480 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:53.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.878+0000 7f668ce00640 1 -- 172.21.15.45:0/1717885636 >> 172.21.15.45:0/1717885636 conn(0x7f66880fee50 msgr2=0x7f6688101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:53.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.878+0000 7f668ce00640 1 -- 172.21.15.45:0/1717885636 shutdown_connections 2024-03-28T17:47:53.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.878+0000 7f668ce00640 1 -- 172.21.15.45:0/1717885636 wait complete. 2024-03-28T17:47:53.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.879+0000 7f668ce00640 1 Processor -- start 2024-03-28T17:47:53.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.879+0000 7f668ce00640 1 -- start start 2024-03-28T17:47:53.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.880+0000 7f668ce00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6688076060 0x7f668819dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:53.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.880+0000 7f668ce00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66880772a0 0x7f668819e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:53.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.880+0000 7f668ce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6688077c40 0x7f66881a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:53.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.880+0000 7f668ce00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66881139b0 con 0x7f6688077c40 2024-03-28T17:47:53.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.880+0000 7f668ce00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6688113830 con 0x7f66880772a0 2024-03-28T17:47:53.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.880+0000 7f668ce00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6688113b30 con 0x7f6688076060 2024-03-28T17:47:53.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.880+0000 7f6685e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6688076060 0x7f668819dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:53.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.880+0000 7f6685e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6688076060 0x7f668819dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:56884/0 (socket says 172.21.15.45:56884) 2024-03-28T17:47:53.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.880+0000 7f6685e00640 1 -- 172.21.15.45:0/4188382621 learned_addr learned my addr 172.21.15.45:0/4188382621 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:53.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.880+0000 7f667d400640 1 --2- 172.21.15.45:0/4188382621 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66880772a0 0x7f668819e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:53.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.880+0000 7f6686800640 1 --2- 172.21.15.45:0/4188382621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6688077c40 0x7f66881a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:53.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.880+0000 7f6685e00640 1 -- 172.21.15.45:0/4188382621 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66880772a0 msgr2=0x7f668819e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.880+0000 7f6685e00640 1 --2- 172.21.15.45:0/4188382621 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66880772a0 0x7f668819e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.880+0000 7f6685e00640 1 -- 172.21.15.45:0/4188382621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6688077c40 msgr2=0x7f66881a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.880+0000 7f6685e00640 1 --2- 172.21.15.45:0/4188382621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6688077c40 0x7f66881a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.880+0000 7f6685e00640 1 -- 172.21.15.45:0/4188382621 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66881a5bd0 con 0x7f6688076060 2024-03-28T17:47:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.880+0000 7f667d400640 1 --2- 172.21.15.45:0/4188382621 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66880772a0 0x7f668819e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:47:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.880+0000 7f6686800640 1 --2- 172.21.15.45:0/4188382621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6688077c40 0x7f66881a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:47:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.881+0000 7f6685e00640 1 --2- 172.21.15.45:0/4188382621 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6688076060 0x7f668819dfb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6678005ab0 tx=0x7f667802c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.881+0000 7f667f400640 1 -- 172.21.15.45:0/4188382621 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6678043020 con 0x7f6688076060 2024-03-28T17:47:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.881+0000 7f668ce00640 1 -- 172.21.15.45:0/4188382621 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66881a5e60 con 0x7f6688076060 2024-03-28T17:47:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.881+0000 7f667f400640 1 -- 172.21.15.45:0/4188382621 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66780054e0 con 0x7f6688076060 2024-03-28T17:47:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.881+0000 7f667f400640 1 -- 172.21.15.45:0/4188382621 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6678003ce0 con 0x7f6688076060 2024-03-28T17:47:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.881+0000 7f668ce00640 1 -- 172.21.15.45:0/4188382621 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66881a62c0 con 0x7f6688076060 2024-03-28T17:47:53.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.882+0000 7f668ce00640 1 -- 172.21.15.45:0/4188382621 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66881103a0 con 0x7f6688076060 2024-03-28T17:47:53.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.884+0000 7f667f400640 1 -- 172.21.15.45:0/4188382621 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6678046020 con 0x7f6688076060 2024-03-28T17:47:53.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.885+0000 7f667f400640 1 --2- 172.21.15.45:0/4188382621 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f666c0a2800 0x7f666c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:53.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.886+0000 7f667f400640 1 -- 172.21.15.45:0/4188382621 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f66780f0fe0 con 0x7f6688076060 2024-03-28T17:47:53.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.886+0000 7f667d400640 1 --2- 172.21.15.45:0/4188382621 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f666c0a2800 0x7f666c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:53.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.886+0000 7f667d400640 1 --2- 172.21.15.45:0/4188382621 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f666c0a2800 0x7f666c0a4cc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f668819f4c0 tx=0x7f667000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:53.892+0000 7f667f400640 1 -- 172.21.15.45:0/4188382621 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f667804d020 con 0x7f6688076060 2024-03-28T17:47:54.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.137+0000 7f668ce00640 1 -- 172.21.15.45:0/4188382621 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f668802d070 con 0x7f6688076060 2024-03-28T17:47:54.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.139+0000 7f667f400640 1 -- 172.21.15.45:0/4188382621 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f66780be070 con 0x7f6688076060 2024-03-28T17:47:54.139 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:47:54.139 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6839/2313500150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":2313500150},{"type":"v1","addr":"172.21.15.45:6839","nonce":2313500150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6841/2647972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":2647972571},{"type":"v1","addr":"172.21.15.45:6841","nonce":2647972571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T17:46:12.261135+0000","modified":"2024-03-28T17:46:17.967524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24584,"mds_1":24530,"mds_2":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6835/2876354410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":2876354410},{"type":"v1","addr":"172.21.15.151:6835","nonce":2876354410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24500":{"gid":24500,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/4234998385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":4234998385},{"type":"v1","addr":"172.21.15.71:6835","nonce":4234998385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6837/1238964845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":1238964845},{"type":"v1","addr":"172.21.15.151:6837","nonce":1238964845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.151:6839/3003564901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6838","nonce":3003564901},{"type":"v1","addr":"172.21.15.151:6839","nonce":3003564901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6837/2008752655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2008752655},{"type":"v1","addr":"172.21.15.71:6837","nonce":2008752655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.71:6839/2709938800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":2709938800},{"type":"v1","addr":"172.21.15.71:6839","nonce":2709938800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24584,"qdb_cluster":[24529,24530,24584]},"id":1}]} 2024-03-28T17:47:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 22 2024-03-28T17:47:54.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.142+0000 7f668ce00640 1 -- 172.21.15.45:0/4188382621 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f666c0a2800 msgr2=0x7f666c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:54.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.142+0000 7f668ce00640 1 --2- 172.21.15.45:0/4188382621 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f666c0a2800 0x7f666c0a4cc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f668819f4c0 tx=0x7f667000b390 comp rx=0 tx=0).stop 2024-03-28T17:47:54.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.142+0000 7f668ce00640 1 -- 172.21.15.45:0/4188382621 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6688076060 msgr2=0x7f668819dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:54.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.142+0000 7f668ce00640 1 --2- 172.21.15.45:0/4188382621 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6688076060 0x7f668819dfb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6678005ab0 tx=0x7f667802c6c0 comp rx=0 tx=0).stop 2024-03-28T17:47:54.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.142+0000 7f668ce00640 1 -- 172.21.15.45:0/4188382621 shutdown_connections 2024-03-28T17:47:54.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.142+0000 7f668ce00640 1 --2- 172.21.15.45:0/4188382621 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f666c0a2800 0x7f666c0a4cc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:54.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.142+0000 7f668ce00640 1 --2- 172.21.15.45:0/4188382621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6688077c40 0x7f66881a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:54.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.142+0000 7f668ce00640 1 --2- 172.21.15.45:0/4188382621 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66880772a0 0x7f668819e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:54.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.142+0000 7f668ce00640 1 --2- 172.21.15.45:0/4188382621 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6688076060 0x7f668819dfb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:54.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.142+0000 7f668ce00640 1 -- 172.21.15.45:0/4188382621 >> 172.21.15.45:0/4188382621 conn(0x7f66880fee50 msgr2=0x7f6688100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:54.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.142+0000 7f668ce00640 1 -- 172.21.15.45:0/4188382621 shutdown_connections 2024-03-28T17:47:54.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.142+0000 7f668ce00640 1 -- 172.21.15.45:0/4188382621 wait complete. 2024-03-28T17:47:54.163 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T17:47:54.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.373+0000 7fc839600640 1 -- 172.21.15.45:0/2599070135 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc834104ad0 msgr2=0x7fc834104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:54.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.373+0000 7fc839600640 1 --2- 172.21.15.45:0/2599070135 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc834104ad0 0x7fc834104f50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc81c009860 tx=0x7fc81c02bb70 comp rx=0 tx=0).stop 2024-03-28T17:47:54.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.373+0000 7fc839600640 1 -- 172.21.15.45:0/2599070135 shutdown_connections 2024-03-28T17:47:54.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.373+0000 7fc839600640 1 --2- 172.21.15.45:0/2599070135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc834105490 0x7fc83410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:54.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.373+0000 7fc839600640 1 --2- 172.21.15.45:0/2599070135 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc834104ad0 0x7fc834104f50 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:54.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.373+0000 7fc839600640 1 --2- 172.21.15.45:0/2599070135 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc8341038e0 0x7fc834103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:54.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.373+0000 7fc839600640 1 -- 172.21.15.45:0/2599070135 >> 172.21.15.45:0/2599070135 conn(0x7fc8340fee90 msgr2=0x7fc8341012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:54.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.373+0000 7fc839600640 1 -- 172.21.15.45:0/2599070135 shutdown_connections 2024-03-28T17:47:54.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.374+0000 7fc839600640 1 -- 172.21.15.45:0/2599070135 wait complete. 2024-03-28T17:47:54.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.374+0000 7fc839600640 1 Processor -- start 2024-03-28T17:47:54.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.375+0000 7fc839600640 1 -- start start 2024-03-28T17:47:54.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.375+0000 7fc839600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc8341038e0 0x7fc834199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:54.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.375+0000 7fc839600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc834104ad0 0x7fc83419a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:54.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.375+0000 7fc839600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc834105490 0x7fc8341a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:54.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.375+0000 7fc839600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83410f540 con 0x7fc834105490 2024-03-28T17:47:54.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.375+0000 7fc839600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83410f3c0 con 0x7fc8341038e0 2024-03-28T17:47:54.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.375+0000 7fc839600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83410f6c0 con 0x7fc834104ad0 2024-03-28T17:47:54.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.375+0000 7fc832a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc8341038e0 0x7fc834199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:54.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.375+0000 7fc832000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc834104ad0 0x7fc83419a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:54.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.375+0000 7fc833400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc834105490 0x7fc8341a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:54.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.375+0000 7fc832a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc8341038e0 0x7fc834199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:51180/0 (socket says 172.21.15.45:51180) 2024-03-28T17:47:54.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.375+0000 7fc832000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc834104ad0 0x7fc83419a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:56902/0 (socket says 172.21.15.45:56902) 2024-03-28T17:47:54.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.375+0000 7fc832a00640 1 -- 172.21.15.45:0/2546247743 learned_addr learned my addr 172.21.15.45:0/2546247743 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:54.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.376+0000 7fc832a00640 1 -- 172.21.15.45:0/2546247743 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc834104ad0 msgr2=0x7fc83419a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:54.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.376+0000 7fc832a00640 1 --2- 172.21.15.45:0/2546247743 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc834104ad0 0x7fc83419a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:54.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.376+0000 7fc832a00640 1 -- 172.21.15.45:0/2546247743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc834105490 msgr2=0x7fc8341a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:54.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.376+0000 7fc832a00640 1 --2- 172.21.15.45:0/2546247743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc834105490 0x7fc8341a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:54.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.376+0000 7fc832a00640 1 -- 172.21.15.45:0/2546247743 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8341a1800 con 0x7fc8341038e0 2024-03-28T17:47:54.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.376+0000 7fc832000640 1 --2- 172.21.15.45:0/2546247743 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc834104ad0 0x7fc83419a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:47:54.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.376+0000 7fc832a00640 1 --2- 172.21.15.45:0/2546247743 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc8341038e0 0x7fc834199be0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc82400e9a0 tx=0x7fc82400ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:54.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.376+0000 7fc82b400640 1 -- 172.21.15.45:0/2546247743 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82400cd80 con 0x7fc8341038e0 2024-03-28T17:47:54.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.376+0000 7fc82b400640 1 -- 172.21.15.45:0/2546247743 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc824010910 con 0x7fc8341038e0 2024-03-28T17:47:54.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.376+0000 7fc82b400640 1 -- 172.21.15.45:0/2546247743 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc824018a30 con 0x7fc8341038e0 2024-03-28T17:47:54.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.376+0000 7fc839600640 1 -- 172.21.15.45:0/2546247743 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8341a1af0 con 0x7fc8341038e0 2024-03-28T17:47:54.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.376+0000 7fc839600640 1 -- 172.21.15.45:0/2546247743 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8341a1fc0 con 0x7fc8341038e0 2024-03-28T17:47:54.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.377+0000 7fc839600640 1 -- 172.21.15.45:0/2546247743 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8340688e0 con 0x7fc8341038e0 2024-03-28T17:47:54.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.379+0000 7fc82b400640 1 -- 172.21.15.45:0/2546247743 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc824010ac0 con 0x7fc8341038e0 2024-03-28T17:47:54.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.380+0000 7fc82b400640 1 --2- 172.21.15.45:0/2546247743 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc8180a2880 0x7fc8180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:54.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.380+0000 7fc82b400640 1 -- 172.21.15.45:0/2546247743 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc8240ced90 con 0x7fc8341038e0 2024-03-28T17:47:54.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.380+0000 7fc832000640 1 --2- 172.21.15.45:0/2546247743 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc8180a2880 0x7fc8180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:54.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.381+0000 7fc832000640 1 --2- 172.21.15.45:0/2546247743 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc8180a2880 0x7fc8180a4d40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc81c009830 tx=0x7fc81c02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:54.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.384+0000 7fc82b400640 1 -- 172.21.15.45:0/2546247743 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc824053740 con 0x7fc8341038e0 2024-03-28T17:47:54.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:54 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3635284699' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T17:47:54.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:54 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/4188382621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:47:54.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.589+0000 7fc839600640 1 -- 172.21.15.45:0/2546247743 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8340622a0 con 0x7fc8341038e0 2024-03-28T17:47:54.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.591+0000 7fc82b400640 1 -- 172.21.15.45:0/2546247743 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22583 (secure 0 0 0) 0x7fc82409bde0 con 0x7fc8341038e0 2024-03-28T17:47:54.592 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:47:54.593 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"9f4888ec-ed29-11ee-95d0-87774f69a715","created":"2024-03-28T17:36:51.623185+0000","modified":"2024-03-28T17:47:09.455843+0000","last_up_change":"2024-03-28T17:45:14.043432+0000","last_in_change":"2024-03-28T17:44:53.654489+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T17:40:54.208375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T17:46:10.564519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T17:46:11.734367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"74c56bc2-4169-4919-afd4-2ebd236f23e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6803","nonce":3795754023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6805","nonce":3795754023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6809","nonce":3795754023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6807","nonce":3795754023}]},"public_addr":"172.21.15.45:6803/3795754023","cluster_addr":"172.21.15.45:6805/3795754023","heartbeat_back_addr":"172.21.15.45:6809/3795754023","heartbeat_front_addr":"172.21.15.45:6807/3795754023","state":["exists","up"]},{"osd":1,"uuid":"78dcc21b-75b2-453d-b690-17b55b8ed1a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6801","nonce":2014529906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6803","nonce":2014529906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6807","nonce":2014529906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6805","nonce":2014529906}]},"public_addr":"172.21.15.71:6801/2014529906","cluster_addr":"172.21.15.71:6803/2014529906","heartbeat_back_addr":"172.21.15.71:6807/2014529906","heartbeat_front_addr":"172.21.15.71:6805/2014529906","state":["exists","up"]},{"osd":2,"uuid":"ca9f3f36-b71e-4962-a94f-cedf2f12eb7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6801","nonce":1850933775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6803","nonce":1850933775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6807","nonce":1850933775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6805","nonce":1850933775}]},"public_addr":"172.21.15.151:6801/1850933775","cluster_addr":"172.21.15.151:6803/1850933775","heartbeat_back_addr":"172.21.15.151:6807/1850933775","heartbeat_front_addr":"172.21.15.151:6805/1850933775","state":["exists","up"]},{"osd":3,"uuid":"d3f21f01-2973-413e-8daa-9974b8a44af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6811","nonce":3021569975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6813","nonce":3021569975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6817","nonce":3021569975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6815","nonce":3021569975}]},"public_addr":"172.21.15.45:6811/3021569975","cluster_addr":"172.21.15.45:6813/3021569975","heartbeat_back_addr":"172.21.15.45:6817/3021569975","heartbeat_front_addr":"172.21.15.45:6815/3021569975","state":["exists","up"]},{"osd":4,"uuid":"4a336f42-0605-492b-ad88-56cb02acdcf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6809","nonce":2069578333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6811","nonce":2069578333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6815","nonce":2069578333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6813","nonce":2069578333}]},"public_addr":"172.21.15.71:6809/2069578333","cluster_addr":"172.21.15.71:6811/2069578333","heartbeat_back_addr":"172.21.15.71:6815/2069578333","heartbeat_front_addr":"172.21.15.71:6813/2069578333","state":["exists","up"]},{"osd":5,"uuid":"c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6809","nonce":2629217227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6811","nonce":2629217227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6815","nonce":2629217227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6813","nonce":2629217227}]},"public_addr":"172.21.15.151:6809/2629217227","cluster_addr":"172.21.15.151:6811/2629217227","heartbeat_back_addr":"172.21.15.151:6815/2629217227","heartbeat_front_addr":"172.21.15.151:6813/2629217227","state":["exists","up"]},{"osd":6,"uuid":"58a6c5bd-a8b7-4892-b346-bcd7994bcb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6819","nonce":2077907039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6821","nonce":2077907039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6825","nonce":2077907039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6823","nonce":2077907039}]},"public_addr":"172.21.15.45:6819/2077907039","cluster_addr":"172.21.15.45:6821/2077907039","heartbeat_back_addr":"172.21.15.45:6825/2077907039","heartbeat_front_addr":"172.21.15.45:6823/2077907039","state":["exists","up"]},{"osd":7,"uuid":"f7a45d52-807b-4407-bb04-787b2a52a0ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6817","nonce":2000211251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6819","nonce":2000211251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6823","nonce":2000211251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6821","nonce":2000211251}]},"public_addr":"172.21.15.71:6817/2000211251","cluster_addr":"172.21.15.71:6819/2000211251","heartbeat_back_addr":"172.21.15.71:6823/2000211251","heartbeat_front_addr":"172.21.15.71:6821/2000211251","state":["exists","up"]},{"osd":8,"uuid":"88abcb0f-f189-49be-8315-c700c77369f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6817","nonce":3665460137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6819","nonce":3665460137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6823","nonce":3665460137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6821","nonce":3665460137}]},"public_addr":"172.21.15.151:6817/3665460137","cluster_addr":"172.21.15.151:6819/3665460137","heartbeat_back_addr":"172.21.15.151:6823/3665460137","heartbeat_front_addr":"172.21.15.151:6821/3665460137","state":["exists","up"]},{"osd":9,"uuid":"ed3fc096-961e-4392-abc2-c5aa4d050117","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6827","nonce":2122322684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6829","nonce":2122322684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6833","nonce":2122322684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6831","nonce":2122322684}]},"public_addr":"172.21.15.45:6827/2122322684","cluster_addr":"172.21.15.45:6829/2122322684","heartbeat_back_addr":"172.21.15.45:6833/2122322684","heartbeat_front_addr":"172.21.15.45:6831/2122322684","state":["exists","up"]},{"osd":10,"uuid":"ba94762b-bc5c-4c97-9cc1-17e5651c14b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6825","nonce":1099494037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6827","nonce":1099494037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6831","nonce":1099494037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6829","nonce":1099494037}]},"public_addr":"172.21.15.71:6825/1099494037","cluster_addr":"172.21.15.71:6827/1099494037","heartbeat_back_addr":"172.21.15.71:6831/1099494037","heartbeat_front_addr":"172.21.15.71:6829/1099494037","state":["exists","up"]},{"osd":11,"uuid":"1d178378-b9f5-40f6-99ef-3d3d445da640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6825","nonce":2941055364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6827","nonce":2941055364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6831","nonce":2941055364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6829","nonce":2941055364}]},"public_addr":"172.21.15.151:6825/2941055364","cluster_addr":"172.21.15.151:6827/2941055364","heartbeat_back_addr":"172.21.15.151:6831/2941055364","heartbeat_front_addr":"172.21.15.151:6829/2941055364","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:02.269586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:27.819386+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:51.552277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:25.052674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:51.873529+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:17.127867+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:51.588249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:18.381450+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:44.140534+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:17.634362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:45.469340+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:45:12.003433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3533611957":"2024-03-29T17:38:07.505611+0000","172.21.15.45:0/3625922430":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6801/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/482610468":"2024-03-29T17:37:32.424229+0000","172.21.15.45:6801/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3649804714":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/1198364079":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3211303958":"2024-03-29T17:38:07.505611+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T17:47:54.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.593+0000 7fc839600640 1 -- 172.21.15.45:0/2546247743 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc8180a2880 msgr2=0x7fc8180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:54.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.593+0000 7fc839600640 1 --2- 172.21.15.45:0/2546247743 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc8180a2880 0x7fc8180a4d40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc81c009830 tx=0x7fc81c02e040 comp rx=0 tx=0).stop 2024-03-28T17:47:54.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.593+0000 7fc839600640 1 -- 172.21.15.45:0/2546247743 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc8341038e0 msgr2=0x7fc834199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:54.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.593+0000 7fc839600640 1 --2- 172.21.15.45:0/2546247743 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc8341038e0 0x7fc834199be0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc82400e9a0 tx=0x7fc82400ee60 comp rx=0 tx=0).stop 2024-03-28T17:47:54.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.593+0000 7fc839600640 1 -- 172.21.15.45:0/2546247743 shutdown_connections 2024-03-28T17:47:54.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.593+0000 7fc839600640 1 --2- 172.21.15.45:0/2546247743 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc8180a2880 0x7fc8180a4d40 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:54.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.593+0000 7fc839600640 1 --2- 172.21.15.45:0/2546247743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc834105490 0x7fc8341a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:54.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.593+0000 7fc839600640 1 --2- 172.21.15.45:0/2546247743 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc834104ad0 0x7fc83419a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:54.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.593+0000 7fc839600640 1 --2- 172.21.15.45:0/2546247743 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc8341038e0 0x7fc834199be0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:54.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.593+0000 7fc839600640 1 -- 172.21.15.45:0/2546247743 >> 172.21.15.45:0/2546247743 conn(0x7fc8340fee90 msgr2=0x7fc834107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:54.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.594+0000 7fc839600640 1 -- 172.21.15.45:0/2546247743 shutdown_connections 2024-03-28T17:47:54.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.594+0000 7fc839600640 1 -- 172.21.15.45:0/2546247743 wait complete. 2024-03-28T17:47:54.614 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T17:40:54.208375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T17:46:10.564519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T17:46:11.734367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T17:47:54.615 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T17:47:54.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:54 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3635284699' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T17:47:54.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:54 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/4188382621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:47:54.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:54 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3635284699' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T17:47:54.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:54 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/4188382621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:47:54.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.842+0000 7f3b81a00640 1 -- 172.21.15.45:0/3169477399 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b7c106040 msgr2=0x7f3b7c108500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:54.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.842+0000 7f3b81a00640 1 --2- 172.21.15.45:0/3169477399 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b7c106040 0x7f3b7c108500 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009ca0 tx=0x7f3b7402c100 comp rx=0 tx=0).stop 2024-03-28T17:47:54.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.842+0000 7f3b81a00640 1 -- 172.21.15.45:0/3169477399 shutdown_connections 2024-03-28T17:47:54.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.842+0000 7f3b81a00640 1 --2- 172.21.15.45:0/3169477399 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b7c106040 0x7f3b7c108500 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:54.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.842+0000 7f3b81a00640 1 --2- 172.21.15.45:0/3169477399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b7c103630 0x7f3b7c105a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:54.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.842+0000 7f3b81a00640 1 --2- 172.21.15.45:0/3169477399 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b7c100d00 0x7f3b7c1030f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:54.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.842+0000 7f3b81a00640 1 -- 172.21.15.45:0/3169477399 >> 172.21.15.45:0/3169477399 conn(0x7f3b7c0fad20 msgr2=0x7f3b7c0fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:54.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.842+0000 7f3b81a00640 1 -- 172.21.15.45:0/3169477399 shutdown_connections 2024-03-28T17:47:54.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.842+0000 7f3b81a00640 1 -- 172.21.15.45:0/3169477399 wait complete. 2024-03-28T17:47:54.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.843+0000 7f3b81a00640 1 Processor -- start 2024-03-28T17:47:54.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.843+0000 7f3b81a00640 1 -- start start 2024-03-28T17:47:54.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.844+0000 7f3b81a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b7c100d00 0x7f3b7c102c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:54.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.844+0000 7f3b81a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b7c103630 0x7f3b7c1012a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:54.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.844+0000 7f3b81a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b7c106040 0x7f3b7c1017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:54.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.844+0000 7f3b81a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7c10c100 con 0x7f3b7c103630 2024-03-28T17:47:54.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.844+0000 7f3b81a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7c10bf80 con 0x7f3b7c100d00 2024-03-28T17:47:54.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.844+0000 7f3b81a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7c10c280 con 0x7f3b7c106040 2024-03-28T17:47:54.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.844+0000 7f3b7be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b7c100d00 0x7f3b7c102c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:54.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.844+0000 7f3b7be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b7c100d00 0x7f3b7c102c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:51186/0 (socket says 172.21.15.45:51186) 2024-03-28T17:47:54.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.844+0000 7f3b7be00640 1 -- 172.21.15.45:0/4138611273 learned_addr learned my addr 172.21.15.45:0/4138611273 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:54.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.844+0000 7f3b73e00640 1 --2- 172.21.15.45:0/4138611273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b7c103630 0x7f3b7c1012a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:54.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.844+0000 7f3b81000640 1 --2- 172.21.15.45:0/4138611273 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b7c106040 0x7f3b7c1017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:54.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.844+0000 7f3b7be00640 1 -- 172.21.15.45:0/4138611273 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b7c106040 msgr2=0x7f3b7c1017e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:54.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.844+0000 7f3b7be00640 1 --2- 172.21.15.45:0/4138611273 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b7c106040 0x7f3b7c1017e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:54.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.844+0000 7f3b7be00640 1 -- 172.21.15.45:0/4138611273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b7c103630 msgr2=0x7f3b7c1012a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:54.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.845+0000 7f3b7be00640 1 --2- 172.21.15.45:0/4138611273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b7c103630 0x7f3b7c1012a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:54.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.845+0000 7f3b7be00640 1 -- 172.21.15.45:0/4138611273 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b7c1020a0 con 0x7f3b7c100d00 2024-03-28T17:47:54.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.845+0000 7f3b73e00640 1 --2- 172.21.15.45:0/4138611273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b7c103630 0x7f3b7c1012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:47:54.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.845+0000 7f3b7be00640 1 --2- 172.21.15.45:0/4138611273 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b7c100d00 0x7f3b7c102c10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c00e9c0 tx=0x7f3b6c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:54.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.845+0000 7f3b79600640 1 -- 172.21.15.45:0/4138611273 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c00cd80 con 0x7f3b7c100d00 2024-03-28T17:47:54.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.845+0000 7f3b79600640 1 -- 172.21.15.45:0/4138611273 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b6c005500 con 0x7f3b7c100d00 2024-03-28T17:47:54.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.845+0000 7f3b79600640 1 -- 172.21.15.45:0/4138611273 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c010690 con 0x7f3b7c100d00 2024-03-28T17:47:54.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.845+0000 7f3b81a00640 1 -- 172.21.15.45:0/4138611273 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b7c1a1990 con 0x7f3b7c100d00 2024-03-28T17:47:54.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.845+0000 7f3b81a00640 1 -- 172.21.15.45:0/4138611273 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b7c1a1eb0 con 0x7f3b7c100d00 2024-03-28T17:47:54.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.846+0000 7f3b81a00640 1 -- 172.21.15.45:0/4138611273 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b7c068800 con 0x7f3b7c100d00 2024-03-28T17:47:54.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.848+0000 7f3b79600640 1 -- 172.21.15.45:0/4138611273 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3b6c0046c0 con 0x7f3b7c100d00 2024-03-28T17:47:54.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.849+0000 7f3b79600640 1 --2- 172.21.15.45:0/4138611273 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3b600a2800 0x7f3b600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:54.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.849+0000 7f3b79600640 1 -- 172.21.15.45:0/4138611273 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3b6c0cee90 con 0x7f3b7c100d00 2024-03-28T17:47:54.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.850+0000 7f3b73e00640 1 --2- 172.21.15.45:0/4138611273 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3b600a2800 0x7f3b600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:54.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.850+0000 7f3b73e00640 1 --2- 172.21.15.45:0/4138611273 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3b600a2800 0x7f3b600a4cc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3b640058c0 tx=0x7f3b64005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:54.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:54.854+0000 7f3b79600640 1 -- 172.21.15.45:0/4138611273 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3b6c09bee0 con 0x7f3b7c100d00 2024-03-28T17:47:55.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.056+0000 7f3b81a00640 1 -- 172.21.15.45:0/4138611273 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3b7c1a2810 con 0x7f3b7c100d00 2024-03-28T17:47:55.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.056+0000 7f3b79600640 1 -- 172.21.15.45:0/4138611273 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3b6c09b7e0 con 0x7f3b7c100d00 2024-03-28T17:47:55.058 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-28T17:47:55.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.059+0000 7f3b81a00640 1 -- 172.21.15.45:0/4138611273 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3b600a2800 msgr2=0x7f3b600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:55.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.059+0000 7f3b81a00640 1 --2- 172.21.15.45:0/4138611273 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3b600a2800 0x7f3b600a4cc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3b640058c0 tx=0x7f3b64005850 comp rx=0 tx=0).stop 2024-03-28T17:47:55.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.059+0000 7f3b81a00640 1 -- 172.21.15.45:0/4138611273 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b7c100d00 msgr2=0x7f3b7c102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:55.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.059+0000 7f3b81a00640 1 --2- 172.21.15.45:0/4138611273 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b7c100d00 0x7f3b7c102c10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c00e9c0 tx=0x7f3b6c00ee80 comp rx=0 tx=0).stop 2024-03-28T17:47:55.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.059+0000 7f3b81a00640 1 -- 172.21.15.45:0/4138611273 shutdown_connections 2024-03-28T17:47:55.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.059+0000 7f3b81a00640 1 --2- 172.21.15.45:0/4138611273 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3b600a2800 0x7f3b600a4cc0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:55.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.059+0000 7f3b81a00640 1 --2- 172.21.15.45:0/4138611273 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3b7c106040 0x7f3b7c1017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:55.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.059+0000 7f3b81a00640 1 --2- 172.21.15.45:0/4138611273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b7c103630 0x7f3b7c1012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:55.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.059+0000 7f3b81a00640 1 --2- 172.21.15.45:0/4138611273 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b7c100d00 0x7f3b7c102c10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:55.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.059+0000 7f3b81a00640 1 -- 172.21.15.45:0/4138611273 >> 172.21.15.45:0/4138611273 conn(0x7f3b7c0fad20 msgr2=0x7f3b7c0fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:55.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.059+0000 7f3b81a00640 1 -- 172.21.15.45:0/4138611273 shutdown_connections 2024-03-28T17:47:55.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.060+0000 7f3b81a00640 1 -- 172.21.15.45:0/4138611273 wait complete. 2024-03-28T17:47:55.077 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T17:47:55.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.303+0000 7f48de400640 1 -- 172.21.15.45:0/646287182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48d8103150 msgr2=0x7f48d81099e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:55.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.303+0000 7f48de400640 1 --2- 172.21.15.45:0/646287182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48d8103150 0x7f48d81099e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f48d0009900 tx=0x7f48d002bc30 comp rx=0 tx=0).stop 2024-03-28T17:47:55.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.303+0000 7f48de400640 1 -- 172.21.15.45:0/646287182 shutdown_connections 2024-03-28T17:47:55.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.303+0000 7f48de400640 1 --2- 172.21.15.45:0/646287182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48d8103150 0x7f48d81099e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:55.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.303+0000 7f48de400640 1 --2- 172.21.15.45:0/646287182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48d8102790 0x7f48d8102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:55.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.303+0000 7f48de400640 1 --2- 172.21.15.45:0/646287182 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48d81015a0 0x7f48d81019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:55.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.303+0000 7f48de400640 1 -- 172.21.15.45:0/646287182 >> 172.21.15.45:0/646287182 conn(0x7f48d80fcb10 msgr2=0x7f48d80fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:55.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.303+0000 7f48de400640 1 -- 172.21.15.45:0/646287182 shutdown_connections 2024-03-28T17:47:55.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.303+0000 7f48de400640 1 -- 172.21.15.45:0/646287182 wait complete. 2024-03-28T17:47:55.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.304+0000 7f48de400640 1 Processor -- start 2024-03-28T17:47:55.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.304+0000 7f48de400640 1 -- start start 2024-03-28T17:47:55.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.304+0000 7f48de400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48d81015a0 0x7f48d8197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:55.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.304+0000 7f48de400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48d8102790 0x7f48d8197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:55.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.304+0000 7f48de400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48d8103150 0x7f48d819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:55.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.304+0000 7f48de400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d810d250 con 0x7f48d8102790 2024-03-28T17:47:55.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.304+0000 7f48de400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d810d0d0 con 0x7f48d81015a0 2024-03-28T17:47:55.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.304+0000 7f48de400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d810d3d0 con 0x7f48d8103150 2024-03-28T17:47:55.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.305+0000 7f48dd000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48d81015a0 0x7f48d8197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:55.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.305+0000 7f48d7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48d8102790 0x7f48d8197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:55.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.305+0000 7f48dd000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48d81015a0 0x7f48d8197800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:51206/0 (socket says 172.21.15.45:51206) 2024-03-28T17:47:55.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.305+0000 7f48dd000640 1 -- 172.21.15.45:0/3928986388 learned_addr learned my addr 172.21.15.45:0/3928986388 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:55.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.305+0000 7f48dda00640 1 --2- 172.21.15.45:0/3928986388 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48d8103150 0x7f48d819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:55.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.305+0000 7f48dd000640 1 -- 172.21.15.45:0/3928986388 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48d8103150 msgr2=0x7f48d819edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:55.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.305+0000 7f48dd000640 1 --2- 172.21.15.45:0/3928986388 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48d8103150 0x7f48d819edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:55.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.305+0000 7f48dd000640 1 -- 172.21.15.45:0/3928986388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48d8102790 msgr2=0x7f48d8197d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:55.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.305+0000 7f48dd000640 1 --2- 172.21.15.45:0/3928986388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48d8102790 0x7f48d8197d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:55.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.305+0000 7f48dd000640 1 -- 172.21.15.45:0/3928986388 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d819f420 con 0x7f48d81015a0 2024-03-28T17:47:55.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.305+0000 7f48dd000640 1 --2- 172.21.15.45:0/3928986388 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48d81015a0 0x7f48d8197800 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f48c800ebc0 tx=0x7f48c800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:55.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.305+0000 7f48d5600640 1 -- 172.21.15.45:0/3928986388 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c8003b70 con 0x7f48d81015a0 2024-03-28T17:47:55.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.306+0000 7f48d5600640 1 -- 172.21.15.45:0/3928986388 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48c8003d10 con 0x7f48d81015a0 2024-03-28T17:47:55.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.306+0000 7f48de400640 1 -- 172.21.15.45:0/3928986388 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d819f710 con 0x7f48d81015a0 2024-03-28T17:47:55.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.306+0000 7f48d5600640 1 -- 172.21.15.45:0/3928986388 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c8019980 con 0x7f48d81015a0 2024-03-28T17:47:55.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.306+0000 7f48de400640 1 -- 172.21.15.45:0/3928986388 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d819fcb0 con 0x7f48d81015a0 2024-03-28T17:47:55.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.307+0000 7f48de400640 1 -- 172.21.15.45:0/3928986388 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d8108860 con 0x7f48d81015a0 2024-03-28T17:47:55.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.309+0000 7f48d5600640 1 -- 172.21.15.45:0/3928986388 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f48c8019b20 con 0x7f48d81015a0 2024-03-28T17:47:55.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.310+0000 7f48d5600640 1 --2- 172.21.15.45:0/3928986388 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f48bc0a2880 0x7f48bc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:55.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.310+0000 7f48d5600640 1 -- 172.21.15.45:0/3928986388 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f48c80cf210 con 0x7f48d81015a0 2024-03-28T17:47:55.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.310+0000 7f48d7e00640 1 --2- 172.21.15.45:0/3928986388 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f48bc0a2880 0x7f48bc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:55.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.311+0000 7f48d7e00640 1 --2- 172.21.15.45:0/3928986388 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f48bc0a2880 0x7f48bc0a4d40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f48d8198d10 tx=0x7f48c400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:55.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.314+0000 7f48d5600640 1 -- 172.21.15.45:0/3928986388 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f48c809c260 con 0x7f48d81015a0 2024-03-28T17:47:55.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.516+0000 7f48de400640 1 -- 172.21.15.45:0/3928986388 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f48d80622a0 con 0x7f48d81015a0 2024-03-28T17:47:55.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.517+0000 7f48d5600640 1 -- 172.21.15.45:0/3928986388 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f48c809bb60 con 0x7f48d81015a0 2024-03-28T17:47:55.518 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-03-28T17:47:55.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.519+0000 7f48de400640 1 -- 172.21.15.45:0/3928986388 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f48bc0a2880 msgr2=0x7f48bc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:55.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.519+0000 7f48de400640 1 --2- 172.21.15.45:0/3928986388 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f48bc0a2880 0x7f48bc0a4d40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f48d8198d10 tx=0x7f48c400b410 comp rx=0 tx=0).stop 2024-03-28T17:47:55.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.519+0000 7f48de400640 1 -- 172.21.15.45:0/3928986388 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48d81015a0 msgr2=0x7f48d8197800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:55.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.519+0000 7f48de400640 1 --2- 172.21.15.45:0/3928986388 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48d81015a0 0x7f48d8197800 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f48c800ebc0 tx=0x7f48c800c6a0 comp rx=0 tx=0).stop 2024-03-28T17:47:55.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.520+0000 7f48de400640 1 -- 172.21.15.45:0/3928986388 shutdown_connections 2024-03-28T17:47:55.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.520+0000 7f48de400640 1 --2- 172.21.15.45:0/3928986388 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f48bc0a2880 0x7f48bc0a4d40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:55.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.520+0000 7f48de400640 1 --2- 172.21.15.45:0/3928986388 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48d8103150 0x7f48d819edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:55.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.520+0000 7f48de400640 1 --2- 172.21.15.45:0/3928986388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48d8102790 0x7f48d8197d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:55.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.520+0000 7f48de400640 1 --2- 172.21.15.45:0/3928986388 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48d81015a0 0x7f48d8197800 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:55.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.520+0000 7f48de400640 1 -- 172.21.15.45:0/3928986388 >> 172.21.15.45:0/3928986388 conn(0x7f48d80fcb10 msgr2=0x7f48d8105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:55.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.520+0000 7f48de400640 1 -- 172.21.15.45:0/3928986388 shutdown_connections 2024-03-28T17:47:55.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.520+0000 7f48de400640 1 -- 172.21.15.45:0/3928986388 wait complete. 2024-03-28T17:47:55.541 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T17:47:55.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:55 smithi045 ceph-mon[31773]: pgmap v370: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T17:47:55.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:55 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/2546247743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:47:55.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:55 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/4138611273' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:47:55.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.766+0000 7fe2cb200640 1 -- 172.21.15.45:0/2812671617 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe2cc102000 msgr2=0x7fe2cc1043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:55.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.766+0000 7fe2cb200640 1 --2- 172.21.15.45:0/2812671617 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe2cc102000 0x7fe2cc1043f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc009900 tx=0x7fe2bc02bc40 comp rx=0 tx=0).stop 2024-03-28T17:47:55.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.766+0000 7fe2cb200640 1 -- 172.21.15.45:0/2812671617 shutdown_connections 2024-03-28T17:47:55.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.766+0000 7fe2cb200640 1 --2- 172.21.15.45:0/2812671617 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2cc107260 0x7fe2cc1096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:55.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.766+0000 7fe2cb200640 1 --2- 172.21.15.45:0/2812671617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2cc104930 0x7fe2cc106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:55.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.766+0000 7fe2cb200640 1 --2- 172.21.15.45:0/2812671617 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe2cc102000 0x7fe2cc1043f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:55.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.766+0000 7fe2cb200640 1 -- 172.21.15.45:0/2812671617 >> 172.21.15.45:0/2812671617 conn(0x7fe2cc0fba10 msgr2=0x7fe2cc0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:55.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.766+0000 7fe2cb200640 1 -- 172.21.15.45:0/2812671617 shutdown_connections 2024-03-28T17:47:55.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.766+0000 7fe2cb200640 1 -- 172.21.15.45:0/2812671617 wait complete. 2024-03-28T17:47:55.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.767+0000 7fe2cb200640 1 Processor -- start 2024-03-28T17:47:55.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.767+0000 7fe2cb200640 1 -- start start 2024-03-28T17:47:55.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.767+0000 7fe2cb200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2cc102000 0x7fe2cc199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:55.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.767+0000 7fe2cb200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2cc104930 0x7fe2cc199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:55.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.767+0000 7fe2cb200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe2cc107260 0x7fe2cc1a1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:55.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.767+0000 7fe2cb200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc10d260 con 0x7fe2cc104930 2024-03-28T17:47:55.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.767+0000 7fe2cb200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc10d0e0 con 0x7fe2cc102000 2024-03-28T17:47:55.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.767+0000 7fe2cb200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc10d3e0 con 0x7fe2cc107260 2024-03-28T17:47:55.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.768+0000 7fe2c9e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2cc102000 0x7fe2cc199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:55.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.768+0000 7fe2c9e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2cc102000 0x7fe2cc199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:51216/0 (socket says 172.21.15.45:51216) 2024-03-28T17:47:55.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.768+0000 7fe2c9e00640 1 -- 172.21.15.45:0/220917749 learned_addr learned my addr 172.21.15.45:0/220917749 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:55.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.768+0000 7fe2c9400640 1 --2- 172.21.15.45:0/220917749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2cc104930 0x7fe2cc199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:55.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.768+0000 7fe2ca800640 1 --2- 172.21.15.45:0/220917749 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe2cc107260 0x7fe2cc1a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:55.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.768+0000 7fe2c9e00640 1 -- 172.21.15.45:0/220917749 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe2cc107260 msgr2=0x7fe2cc1a1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:55.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.768+0000 7fe2c9e00640 1 --2- 172.21.15.45:0/220917749 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe2cc107260 0x7fe2cc1a1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:55.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.768+0000 7fe2c9e00640 1 -- 172.21.15.45:0/220917749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2cc104930 msgr2=0x7fe2cc199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:55.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.768+0000 7fe2c9e00640 1 --2- 172.21.15.45:0/220917749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2cc104930 0x7fe2cc199fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:55.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.768+0000 7fe2c9e00640 1 -- 172.21.15.45:0/220917749 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc1a16b0 con 0x7fe2cc102000 2024-03-28T17:47:55.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.768+0000 7fe2c9400640 1 --2- 172.21.15.45:0/220917749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2cc104930 0x7fe2cc199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:47:55.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.768+0000 7fe2c9e00640 1 --2- 172.21.15.45:0/220917749 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2cc102000 0x7fe2cc199a90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc02c8f0 tx=0x7fe2bc02c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:55.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.769+0000 7fe2c2a00640 1 -- 172.21.15.45:0/220917749 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc041070 con 0x7fe2cc102000 2024-03-28T17:47:55.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.769+0000 7fe2c2a00640 1 -- 172.21.15.45:0/220917749 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2bc004450 con 0x7fe2cc102000 2024-03-28T17:47:55.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.769+0000 7fe2c2a00640 1 -- 172.21.15.45:0/220917749 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc034440 con 0x7fe2cc102000 2024-03-28T17:47:55.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.769+0000 7fe2cb200640 1 -- 172.21.15.45:0/220917749 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2cc1a1940 con 0x7fe2cc102000 2024-03-28T17:47:55.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.769+0000 7fe2cb200640 1 -- 172.21.15.45:0/220917749 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2cc1a1de0 con 0x7fe2cc102000 2024-03-28T17:47:55.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.770+0000 7fe2cb200640 1 -- 172.21.15.45:0/220917749 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2cc108db0 con 0x7fe2cc102000 2024-03-28T17:47:55.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.772+0000 7fe2c2a00640 1 -- 172.21.15.45:0/220917749 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe2bc044070 con 0x7fe2cc102000 2024-03-28T17:47:55.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.773+0000 7fe2c2a00640 1 --2- 172.21.15.45:0/220917749 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe2b00a2880 0x7fe2b00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:55.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.773+0000 7fe2c2a00640 1 -- 172.21.15.45:0/220917749 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe2bc0f0590 con 0x7fe2cc102000 2024-03-28T17:47:55.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.774+0000 7fe2c9400640 1 --2- 172.21.15.45:0/220917749 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe2b00a2880 0x7fe2b00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:55.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.774+0000 7fe2c9400640 1 --2- 172.21.15.45:0/220917749 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe2b00a2880 0x7fe2b00a4d40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc19afa0 tx=0x7fe2b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:55.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.782+0000 7fe2c2a00640 1 -- 172.21.15.45:0/220917749 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe2bc0bcf90 con 0x7fe2cc102000 2024-03-28T17:47:55.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:55 smithi071 ceph-mon[36597]: pgmap v370: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T17:47:55.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:55 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/2546247743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:47:55.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:55 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/4138611273' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:47:55.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:55 smithi151 ceph-mon[36481]: pgmap v370: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T17:47:55.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:55 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/2546247743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:47:55.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:55 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/4138611273' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:47:55.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.981+0000 7fe2cb200640 1 -- 172.21.15.45:0/220917749 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe2cc0622a0 con 0x7fe2cc102000 2024-03-28T17:47:55.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.982+0000 7fe2c2a00640 1 -- 172.21.15.45:0/220917749 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe2bc0c1030 con 0x7fe2cc102000 2024-03-28T17:47:55.984 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-03-28T17:47:55.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.985+0000 7fe2cb200640 1 -- 172.21.15.45:0/220917749 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe2b00a2880 msgr2=0x7fe2b00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:55.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.985+0000 7fe2cb200640 1 --2- 172.21.15.45:0/220917749 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe2b00a2880 0x7fe2b00a4d40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc19afa0 tx=0x7fe2b8005f90 comp rx=0 tx=0).stop 2024-03-28T17:47:55.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.985+0000 7fe2cb200640 1 -- 172.21.15.45:0/220917749 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2cc102000 msgr2=0x7fe2cc199a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:55.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.985+0000 7fe2cb200640 1 --2- 172.21.15.45:0/220917749 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2cc102000 0x7fe2cc199a90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc02c8f0 tx=0x7fe2bc02c660 comp rx=0 tx=0).stop 2024-03-28T17:47:55.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.985+0000 7fe2cb200640 1 -- 172.21.15.45:0/220917749 shutdown_connections 2024-03-28T17:47:55.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.985+0000 7fe2cb200640 1 --2- 172.21.15.45:0/220917749 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe2b00a2880 0x7fe2b00a4d40 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:55.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.985+0000 7fe2cb200640 1 --2- 172.21.15.45:0/220917749 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe2cc107260 0x7fe2cc1a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:55.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.985+0000 7fe2cb200640 1 --2- 172.21.15.45:0/220917749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2cc104930 0x7fe2cc199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:55.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.985+0000 7fe2cb200640 1 --2- 172.21.15.45:0/220917749 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2cc102000 0x7fe2cc199a90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:55.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.985+0000 7fe2cb200640 1 -- 172.21.15.45:0/220917749 >> 172.21.15.45:0/220917749 conn(0x7fe2cc0fba10 msgr2=0x7fe2cc101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:55.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.985+0000 7fe2cb200640 1 -- 172.21.15.45:0/220917749 shutdown_connections 2024-03-28T17:47:55.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:55.985+0000 7fe2cb200640 1 -- 172.21.15.45:0/220917749 wait complete. 2024-03-28T17:47:56.004 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:47:56.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.217+0000 7fb7fe800640 1 -- 172.21.15.45:0/19653746 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7f81036f0 msgr2=0x7fb7f8103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:56.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.217+0000 7fb7fe800640 1 --2- 172.21.15.45:0/19653746 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7f81036f0 0x7fb7f8103b10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009ca0 tx=0x7fb7e802c100 comp rx=0 tx=0).stop 2024-03-28T17:47:56.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.217+0000 7fb7fe800640 1 -- 172.21.15.45:0/19653746 shutdown_connections 2024-03-28T17:47:56.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.217+0000 7fb7fe800640 1 --2- 172.21.15.45:0/19653746 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7f81052a0 0x7fb7f810bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:56.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.217+0000 7fb7fe800640 1 --2- 172.21.15.45:0/19653746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7f81048e0 0x7fb7f8104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:56.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.217+0000 7fb7fe800640 1 --2- 172.21.15.45:0/19653746 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7f81036f0 0x7fb7f8103b10 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:56.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.217+0000 7fb7fe800640 1 -- 172.21.15.45:0/19653746 >> 172.21.15.45:0/19653746 conn(0x7fb7f80fec80 msgr2=0x7fb7f81010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:56.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.217+0000 7fb7fe800640 1 -- 172.21.15.45:0/19653746 shutdown_connections 2024-03-28T17:47:56.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.217+0000 7fb7fe800640 1 -- 172.21.15.45:0/19653746 wait complete. 2024-03-28T17:47:56.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.218+0000 7fb7fe800640 1 Processor -- start 2024-03-28T17:47:56.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.218+0000 7fb7fe800640 1 -- start start 2024-03-28T17:47:56.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.219+0000 7fb7fe800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7f81036f0 0x7fb7f81999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:56.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.219+0000 7fb7fe800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7f81048e0 0x7fb7f8199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:56.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.219+0000 7fb7fe800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7f81052a0 0x7fb7f81a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:56.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.219+0000 7fb7fe800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f810f370 con 0x7fb7f81048e0 2024-03-28T17:47:56.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.219+0000 7fb7fe800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f810f1f0 con 0x7fb7f81036f0 2024-03-28T17:47:56.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.219+0000 7fb7fe800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f810f4f0 con 0x7fb7f81052a0 2024-03-28T17:47:56.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.219+0000 7fb7fd400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7f81036f0 0x7fb7f81999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:56.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.219+0000 7fb7fca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7f81048e0 0x7fb7f8199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:56.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.219+0000 7fb7fd400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7f81036f0 0x7fb7f81999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:35430/0 (socket says 172.21.15.45:35430) 2024-03-28T17:47:56.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.219+0000 7fb7fd400640 1 -- 172.21.15.45:0/1170458933 learned_addr learned my addr 172.21.15.45:0/1170458933 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:56.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.219+0000 7fb7fde00640 1 --2- 172.21.15.45:0/1170458933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7f81052a0 0x7fb7f81a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:56.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.220+0000 7fb7fd400640 1 -- 172.21.15.45:0/1170458933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7f81052a0 msgr2=0x7fb7f81a0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:56.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.220+0000 7fb7fd400640 1 --2- 172.21.15.45:0/1170458933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7f81052a0 0x7fb7f81a0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:56.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.220+0000 7fb7fd400640 1 -- 172.21.15.45:0/1170458933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7f81048e0 msgr2=0x7fb7f8199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:56.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.220+0000 7fb7fd400640 1 --2- 172.21.15.45:0/1170458933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7f81048e0 0x7fb7f8199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:56.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.220+0000 7fb7fd400640 1 -- 172.21.15.45:0/1170458933 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f81a1610 con 0x7fb7f81036f0 2024-03-28T17:47:56.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.220+0000 7fb7fd400640 1 --2- 172.21.15.45:0/1170458933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7f81036f0 0x7fb7f81999f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8000f80 tx=0x7fb7e8035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:56.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.220+0000 7fb7f6000640 1 -- 172.21.15.45:0/1170458933 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8039040 con 0x7fb7f81036f0 2024-03-28T17:47:56.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.220+0000 7fb7f6000640 1 -- 172.21.15.45:0/1170458933 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7e8044070 con 0x7fb7f81036f0 2024-03-28T17:47:56.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.220+0000 7fb7f6000640 1 -- 172.21.15.45:0/1170458933 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8004430 con 0x7fb7f81036f0 2024-03-28T17:47:56.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.220+0000 7fb7fe800640 1 -- 172.21.15.45:0/1170458933 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f81a18a0 con 0x7fb7f81036f0 2024-03-28T17:47:56.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.220+0000 7fb7fe800640 1 -- 172.21.15.45:0/1170458933 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f81a1d10 con 0x7fb7f81036f0 2024-03-28T17:47:56.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.221+0000 7fb7fe800640 1 -- 172.21.15.45:0/1170458933 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f810aa10 con 0x7fb7f81036f0 2024-03-28T17:47:56.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.224+0000 7fb7f6000640 1 -- 172.21.15.45:0/1170458933 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb7e80045d0 con 0x7fb7f81036f0 2024-03-28T17:47:56.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.225+0000 7fb7f6000640 1 --2- 172.21.15.45:0/1170458933 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb7dc0a2880 0x7fb7dc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:56.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.225+0000 7fb7f6000640 1 -- 172.21.15.45:0/1170458933 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb7e80f1050 con 0x7fb7f81036f0 2024-03-28T17:47:56.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.225+0000 7fb7fca00640 1 --2- 172.21.15.45:0/1170458933 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb7dc0a2880 0x7fb7dc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:56.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.226+0000 7fb7fca00640 1 --2- 172.21.15.45:0/1170458933 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb7dc0a2880 0x7fb7dc0a4d40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb7f819af00 tx=0x7fb7e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:56.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.228+0000 7fb7f6000640 1 -- 172.21.15.45:0/1170458933 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb7e80be100 con 0x7fb7f81036f0 2024-03-28T17:47:56.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.472+0000 7fb7fe800640 1 -- 172.21.15.45:0/1170458933 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7f802d070 con 0x7fb7f81036f0 2024-03-28T17:47:56.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.474+0000 7fb7f6000640 1 -- 172.21.15.45:0/1170458933 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11393 (secure 0 0 0) 0x7fb7e80bda00 con 0x7fb7f81036f0 2024-03-28T17:47:56.475 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:47:56.475 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6839/2313500150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":2313500150},{"type":"v1","addr":"172.21.15.45:6839","nonce":2313500150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6841/2647972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":2647972571},{"type":"v1","addr":"172.21.15.45:6841","nonce":2647972571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T17:46:12.261135+0000","modified":"2024-03-28T17:46:17.967524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24584,"mds_1":24530,"mds_2":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6835/2876354410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":2876354410},{"type":"v1","addr":"172.21.15.151:6835","nonce":2876354410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24500":{"gid":24500,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/4234998385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":4234998385},{"type":"v1","addr":"172.21.15.71:6835","nonce":4234998385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6837/1238964845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":1238964845},{"type":"v1","addr":"172.21.15.151:6837","nonce":1238964845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.151:6839/3003564901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6838","nonce":3003564901},{"type":"v1","addr":"172.21.15.151:6839","nonce":3003564901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6837/2008752655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2008752655},{"type":"v1","addr":"172.21.15.71:6837","nonce":2008752655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.71:6839/2709938800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":2709938800},{"type":"v1","addr":"172.21.15.71:6839","nonce":2709938800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24584,"qdb_cluster":[24529,24530,24584]},"id":1}]} 2024-03-28T17:47:56.475 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 22 2024-03-28T17:47:56.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.476+0000 7fb7fe800640 1 -- 172.21.15.45:0/1170458933 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb7dc0a2880 msgr2=0x7fb7dc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:56.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.476+0000 7fb7fe800640 1 --2- 172.21.15.45:0/1170458933 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb7dc0a2880 0x7fb7dc0a4d40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb7f819af00 tx=0x7fb7e4005f90 comp rx=0 tx=0).stop 2024-03-28T17:47:56.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.476+0000 7fb7fe800640 1 -- 172.21.15.45:0/1170458933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7f81036f0 msgr2=0x7fb7f81999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.476+0000 7fb7fe800640 1 --2- 172.21.15.45:0/1170458933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7f81036f0 0x7fb7f81999f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8000f80 tx=0x7fb7e8035d10 comp rx=0 tx=0).stop 2024-03-28T17:47:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.476+0000 7fb7fe800640 1 -- 172.21.15.45:0/1170458933 shutdown_connections 2024-03-28T17:47:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.476+0000 7fb7fe800640 1 --2- 172.21.15.45:0/1170458933 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb7dc0a2880 0x7fb7dc0a4d40 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.476+0000 7fb7fe800640 1 --2- 172.21.15.45:0/1170458933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7f81052a0 0x7fb7f81a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.476+0000 7fb7fe800640 1 --2- 172.21.15.45:0/1170458933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7f81048e0 0x7fb7f8199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.476+0000 7fb7fe800640 1 --2- 172.21.15.45:0/1170458933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7f81036f0 0x7fb7f81999f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.476+0000 7fb7fe800640 1 -- 172.21.15.45:0/1170458933 >> 172.21.15.45:0/1170458933 conn(0x7fb7f80fec80 msgr2=0x7fb7f8107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.476+0000 7fb7fe800640 1 -- 172.21.15.45:0/1170458933 shutdown_connections 2024-03-28T17:47:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.476+0000 7fb7fe800640 1 -- 172.21.15.45:0/1170458933 wait complete. 2024-03-28T17:47:56.496 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T17:47:56.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:56 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3928986388' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T17:47:56.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:56 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/220917749' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T17:47:56.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.717+0000 7f5a88e00640 1 -- 172.21.15.45:0/2130051112 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a841052a0 msgr2=0x7f5a8410bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:56.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.717+0000 7f5a88e00640 1 --2- 172.21.15.45:0/2130051112 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a841052a0 0x7f5a8410bb30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c009ca0 tx=0x7f5a7c02c0f0 comp rx=0 tx=0).stop 2024-03-28T17:47:56.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.717+0000 7f5a88e00640 1 -- 172.21.15.45:0/2130051112 shutdown_connections 2024-03-28T17:47:56.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.717+0000 7f5a88e00640 1 --2- 172.21.15.45:0/2130051112 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a841052a0 0x7f5a8410bb30 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:56.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.717+0000 7f5a88e00640 1 --2- 172.21.15.45:0/2130051112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a841048e0 0x7f5a84104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:56.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.717+0000 7f5a88e00640 1 --2- 172.21.15.45:0/2130051112 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a841036f0 0x7f5a84103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:56.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.717+0000 7f5a88e00640 1 -- 172.21.15.45:0/2130051112 >> 172.21.15.45:0/2130051112 conn(0x7f5a840fec80 msgr2=0x7f5a841010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:56.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.717+0000 7f5a88e00640 1 -- 172.21.15.45:0/2130051112 shutdown_connections 2024-03-28T17:47:56.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.717+0000 7f5a88e00640 1 -- 172.21.15.45:0/2130051112 wait complete. 2024-03-28T17:47:56.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.718+0000 7f5a88e00640 1 Processor -- start 2024-03-28T17:47:56.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.718+0000 7f5a88e00640 1 -- start start 2024-03-28T17:47:56.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.718+0000 7f5a88e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a841036f0 0x7f5a84199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:56.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.718+0000 7f5a88e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a841048e0 0x7f5a84199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:56.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.718+0000 7f5a88e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a841052a0 0x7f5a841a0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:56.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.718+0000 7f5a88e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8410f350 con 0x7f5a841048e0 2024-03-28T17:47:56.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.719+0000 7f5a88e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8410f1d0 con 0x7f5a841036f0 2024-03-28T17:47:56.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.719+0000 7f5a88e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8410f4d0 con 0x7f5a841052a0 2024-03-28T17:47:56.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.719+0000 7f5a83400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a841036f0 0x7f5a84199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:56.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.719+0000 7f5a83400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a841036f0 0x7f5a84199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:35452/0 (socket says 172.21.15.45:35452) 2024-03-28T17:47:56.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.719+0000 7f5a83400640 1 -- 172.21.15.45:0/3052486036 learned_addr learned my addr 172.21.15.45:0/3052486036 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:56.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.719+0000 7f5a7aa00640 1 --2- 172.21.15.45:0/3052486036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a841048e0 0x7f5a84199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:56.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.719+0000 7f5a83e00640 1 --2- 172.21.15.45:0/3052486036 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a841052a0 0x7f5a841a0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:56.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.719+0000 7f5a83400640 1 -- 172.21.15.45:0/3052486036 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a841052a0 msgr2=0x7f5a841a0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:56.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.719+0000 7f5a83400640 1 --2- 172.21.15.45:0/3052486036 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a841052a0 0x7f5a841a0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:56.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.719+0000 7f5a83400640 1 -- 172.21.15.45:0/3052486036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a841048e0 msgr2=0x7f5a84199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:56.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.719+0000 7f5a83400640 1 --2- 172.21.15.45:0/3052486036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a841048e0 0x7f5a84199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:56.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.719+0000 7f5a83400640 1 -- 172.21.15.45:0/3052486036 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a841a1580 con 0x7f5a841036f0 2024-03-28T17:47:56.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.719+0000 7f5a83e00640 1 --2- 172.21.15.45:0/3052486036 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a841052a0 0x7f5a841a0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:47:56.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.719+0000 7f5a83400640 1 --2- 172.21.15.45:0/3052486036 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a841036f0 0x7f5a84199960 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5a7400e9c0 tx=0x7f5a7400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:56.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.720+0000 7f5a80c00640 1 -- 172.21.15.45:0/3052486036 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7400cd80 con 0x7f5a841036f0 2024-03-28T17:47:56.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.720+0000 7f5a80c00640 1 -- 172.21.15.45:0/3052486036 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a740035f0 con 0x7f5a841036f0 2024-03-28T17:47:56.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.720+0000 7f5a88e00640 1 -- 172.21.15.45:0/3052486036 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a841a1870 con 0x7f5a841036f0 2024-03-28T17:47:56.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.720+0000 7f5a80c00640 1 -- 172.21.15.45:0/3052486036 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a74010690 con 0x7f5a841036f0 2024-03-28T17:47:56.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.720+0000 7f5a88e00640 1 -- 172.21.15.45:0/3052486036 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a841a1e10 con 0x7f5a841036f0 2024-03-28T17:47:56.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.721+0000 7f5a88e00640 1 -- 172.21.15.45:0/3052486036 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a84068800 con 0x7f5a841036f0 2024-03-28T17:47:56.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.722+0000 7f5a80c00640 1 -- 172.21.15.45:0/3052486036 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5a740037a0 con 0x7f5a841036f0 2024-03-28T17:47:56.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.724+0000 7f5a80c00640 1 --2- 172.21.15.45:0/3052486036 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f5a680a2880 0x7f5a680a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:56.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.724+0000 7f5a80c00640 1 -- 172.21.15.45:0/3052486036 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5a740cede0 con 0x7f5a841036f0 2024-03-28T17:47:56.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.724+0000 7f5a7aa00640 1 --2- 172.21.15.45:0/3052486036 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f5a680a2880 0x7f5a680a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:56.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.725+0000 7f5a7aa00640 1 --2- 172.21.15.45:0/3052486036 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f5a680a2880 0x7f5a680a4d40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5a8419ae70 tx=0x7f5a6c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:56.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.729+0000 7f5a80c00640 1 -- 172.21.15.45:0/3052486036 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5a7409be30 con 0x7f5a841036f0 2024-03-28T17:47:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:56 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3928986388' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T17:47:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:56 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/220917749' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T17:47:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:56 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3928986388' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T17:47:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:56 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/220917749' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T17:47:56.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.939+0000 7f5a88e00640 1 -- 172.21.15.45:0/3052486036 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a840622a0 con 0x7f5a841036f0 2024-03-28T17:47:56.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.940+0000 7f5a80c00640 1 -- 172.21.15.45:0/3052486036 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22583 (secure 0 0 0) 0x7f5a7409b730 con 0x7f5a841036f0 2024-03-28T17:47:56.942 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:47:56.942 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"9f4888ec-ed29-11ee-95d0-87774f69a715","created":"2024-03-28T17:36:51.623185+0000","modified":"2024-03-28T17:47:09.455843+0000","last_up_change":"2024-03-28T17:45:14.043432+0000","last_in_change":"2024-03-28T17:44:53.654489+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T17:40:54.208375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T17:46:10.564519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T17:46:11.734367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"74c56bc2-4169-4919-afd4-2ebd236f23e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6803","nonce":3795754023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6805","nonce":3795754023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6809","nonce":3795754023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6807","nonce":3795754023}]},"public_addr":"172.21.15.45:6803/3795754023","cluster_addr":"172.21.15.45:6805/3795754023","heartbeat_back_addr":"172.21.15.45:6809/3795754023","heartbeat_front_addr":"172.21.15.45:6807/3795754023","state":["exists","up"]},{"osd":1,"uuid":"78dcc21b-75b2-453d-b690-17b55b8ed1a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6801","nonce":2014529906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6803","nonce":2014529906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6807","nonce":2014529906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6805","nonce":2014529906}]},"public_addr":"172.21.15.71:6801/2014529906","cluster_addr":"172.21.15.71:6803/2014529906","heartbeat_back_addr":"172.21.15.71:6807/2014529906","heartbeat_front_addr":"172.21.15.71:6805/2014529906","state":["exists","up"]},{"osd":2,"uuid":"ca9f3f36-b71e-4962-a94f-cedf2f12eb7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6801","nonce":1850933775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6803","nonce":1850933775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6807","nonce":1850933775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6805","nonce":1850933775}]},"public_addr":"172.21.15.151:6801/1850933775","cluster_addr":"172.21.15.151:6803/1850933775","heartbeat_back_addr":"172.21.15.151:6807/1850933775","heartbeat_front_addr":"172.21.15.151:6805/1850933775","state":["exists","up"]},{"osd":3,"uuid":"d3f21f01-2973-413e-8daa-9974b8a44af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6811","nonce":3021569975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6813","nonce":3021569975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6817","nonce":3021569975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6815","nonce":3021569975}]},"public_addr":"172.21.15.45:6811/3021569975","cluster_addr":"172.21.15.45:6813/3021569975","heartbeat_back_addr":"172.21.15.45:6817/3021569975","heartbeat_front_addr":"172.21.15.45:6815/3021569975","state":["exists","up"]},{"osd":4,"uuid":"4a336f42-0605-492b-ad88-56cb02acdcf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6809","nonce":2069578333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6811","nonce":2069578333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6815","nonce":2069578333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6813","nonce":2069578333}]},"public_addr":"172.21.15.71:6809/2069578333","cluster_addr":"172.21.15.71:6811/2069578333","heartbeat_back_addr":"172.21.15.71:6815/2069578333","heartbeat_front_addr":"172.21.15.71:6813/2069578333","state":["exists","up"]},{"osd":5,"uuid":"c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6809","nonce":2629217227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6811","nonce":2629217227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6815","nonce":2629217227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6813","nonce":2629217227}]},"public_addr":"172.21.15.151:6809/2629217227","cluster_addr":"172.21.15.151:6811/2629217227","heartbeat_back_addr":"172.21.15.151:6815/2629217227","heartbeat_front_addr":"172.21.15.151:6813/2629217227","state":["exists","up"]},{"osd":6,"uuid":"58a6c5bd-a8b7-4892-b346-bcd7994bcb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6819","nonce":2077907039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6821","nonce":2077907039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6825","nonce":2077907039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6823","nonce":2077907039}]},"public_addr":"172.21.15.45:6819/2077907039","cluster_addr":"172.21.15.45:6821/2077907039","heartbeat_back_addr":"172.21.15.45:6825/2077907039","heartbeat_front_addr":"172.21.15.45:6823/2077907039","state":["exists","up"]},{"osd":7,"uuid":"f7a45d52-807b-4407-bb04-787b2a52a0ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6817","nonce":2000211251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6819","nonce":2000211251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6823","nonce":2000211251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6821","nonce":2000211251}]},"public_addr":"172.21.15.71:6817/2000211251","cluster_addr":"172.21.15.71:6819/2000211251","heartbeat_back_addr":"172.21.15.71:6823/2000211251","heartbeat_front_addr":"172.21.15.71:6821/2000211251","state":["exists","up"]},{"osd":8,"uuid":"88abcb0f-f189-49be-8315-c700c77369f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6817","nonce":3665460137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6819","nonce":3665460137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6823","nonce":3665460137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6821","nonce":3665460137}]},"public_addr":"172.21.15.151:6817/3665460137","cluster_addr":"172.21.15.151:6819/3665460137","heartbeat_back_addr":"172.21.15.151:6823/3665460137","heartbeat_front_addr":"172.21.15.151:6821/3665460137","state":["exists","up"]},{"osd":9,"uuid":"ed3fc096-961e-4392-abc2-c5aa4d050117","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6827","nonce":2122322684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6829","nonce":2122322684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6833","nonce":2122322684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6831","nonce":2122322684}]},"public_addr":"172.21.15.45:6827/2122322684","cluster_addr":"172.21.15.45:6829/2122322684","heartbeat_back_addr":"172.21.15.45:6833/2122322684","heartbeat_front_addr":"172.21.15.45:6831/2122322684","state":["exists","up"]},{"osd":10,"uuid":"ba94762b-bc5c-4c97-9cc1-17e5651c14b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6825","nonce":1099494037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6827","nonce":1099494037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6831","nonce":1099494037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6829","nonce":1099494037}]},"public_addr":"172.21.15.71:6825/1099494037","cluster_addr":"172.21.15.71:6827/1099494037","heartbeat_back_addr":"172.21.15.71:6831/1099494037","heartbeat_front_addr":"172.21.15.71:6829/1099494037","state":["exists","up"]},{"osd":11,"uuid":"1d178378-b9f5-40f6-99ef-3d3d445da640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6825","nonce":2941055364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6827","nonce":2941055364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6831","nonce":2941055364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6829","nonce":2941055364}]},"public_addr":"172.21.15.151:6825/2941055364","cluster_addr":"172.21.15.151:6827/2941055364","heartbeat_back_addr":"172.21.15.151:6831/2941055364","heartbeat_front_addr":"172.21.15.151:6829/2941055364","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:02.269586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:27.819386+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:51.552277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:25.052674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:51.873529+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:17.127867+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:51.588249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:18.381450+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:44.140534+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:17.634362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:45.469340+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:45:12.003433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3533611957":"2024-03-29T17:38:07.505611+0000","172.21.15.45:0/3625922430":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6801/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/482610468":"2024-03-29T17:37:32.424229+0000","172.21.15.45:6801/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3649804714":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/1198364079":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3211303958":"2024-03-29T17:38:07.505611+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T17:47:56.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.943+0000 7f5a88e00640 1 -- 172.21.15.45:0/3052486036 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f5a680a2880 msgr2=0x7f5a680a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:56.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.943+0000 7f5a88e00640 1 --2- 172.21.15.45:0/3052486036 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f5a680a2880 0x7f5a680a4d40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5a8419ae70 tx=0x7f5a6c005f90 comp rx=0 tx=0).stop 2024-03-28T17:47:56.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.943+0000 7f5a88e00640 1 -- 172.21.15.45:0/3052486036 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a841036f0 msgr2=0x7f5a84199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:56.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.943+0000 7f5a88e00640 1 --2- 172.21.15.45:0/3052486036 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a841036f0 0x7f5a84199960 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5a7400e9c0 tx=0x7f5a7400ee80 comp rx=0 tx=0).stop 2024-03-28T17:47:56.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.943+0000 7f5a88e00640 1 -- 172.21.15.45:0/3052486036 shutdown_connections 2024-03-28T17:47:56.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.943+0000 7f5a88e00640 1 --2- 172.21.15.45:0/3052486036 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f5a680a2880 0x7f5a680a4d40 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:56.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.943+0000 7f5a88e00640 1 --2- 172.21.15.45:0/3052486036 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a841052a0 0x7f5a841a0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:56.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.943+0000 7f5a88e00640 1 --2- 172.21.15.45:0/3052486036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a841048e0 0x7f5a84199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:56.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.943+0000 7f5a88e00640 1 --2- 172.21.15.45:0/3052486036 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a841036f0 0x7f5a84199960 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:56.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.943+0000 7f5a88e00640 1 -- 172.21.15.45:0/3052486036 >> 172.21.15.45:0/3052486036 conn(0x7f5a840fec80 msgr2=0x7f5a84107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:56.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.943+0000 7f5a88e00640 1 -- 172.21.15.45:0/3052486036 shutdown_connections 2024-03-28T17:47:56.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:56.944+0000 7f5a88e00640 1 -- 172.21.15.45:0/3052486036 wait complete. 2024-03-28T17:47:56.963 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:47:57.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.185+0000 7f7cf9800640 1 -- 172.21.15.45:0/3058717603 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7cf41016b0 msgr2=0x7f7cf4101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:57.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.185+0000 7f7cf9800640 1 --2- 172.21.15.45:0/3058717603 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7cf41016b0 0x7f7cf4101ad0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4009900 tx=0x7f7ce402bc30 comp rx=0 tx=0).stop 2024-03-28T17:47:57.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.186+0000 7f7cf9800640 1 -- 172.21.15.45:0/3058717603 shutdown_connections 2024-03-28T17:47:57.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.186+0000 7f7cf9800640 1 --2- 172.21.15.45:0/3058717603 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cf4103260 0x7f7cf4109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.186+0000 7f7cf9800640 1 --2- 172.21.15.45:0/3058717603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cf41028a0 0x7f7cf4102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.186+0000 7f7cf9800640 1 --2- 172.21.15.45:0/3058717603 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7cf41016b0 0x7f7cf4101ad0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.186+0000 7f7cf9800640 1 -- 172.21.15.45:0/3058717603 >> 172.21.15.45:0/3058717603 conn(0x7f7cf40fcc80 msgr2=0x7f7cf40ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:57.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.186+0000 7f7cf9800640 1 -- 172.21.15.45:0/3058717603 shutdown_connections 2024-03-28T17:47:57.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.186+0000 7f7cf9800640 1 -- 172.21.15.45:0/3058717603 wait complete. 2024-03-28T17:47:57.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.187+0000 7f7cf9800640 1 Processor -- start 2024-03-28T17:47:57.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.187+0000 7f7cf9800640 1 -- start start 2024-03-28T17:47:57.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.187+0000 7f7cf9800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7cf41016b0 0x7f7cf4199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.187+0000 7f7cf9800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cf41028a0 0x7f7cf419a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.187+0000 7f7cf9800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cf4103260 0x7f7cf41a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.188+0000 7f7cf9800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf410d350 con 0x7f7cf4103260 2024-03-28T17:47:57.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.188+0000 7f7cf9800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf410d1d0 con 0x7f7cf41016b0 2024-03-28T17:47:57.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.188+0000 7f7cf9800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf410d4d0 con 0x7f7cf41028a0 2024-03-28T17:47:57.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.188+0000 7f7cf2a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7cf41016b0 0x7f7cf4199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.188+0000 7f7cf2000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cf41028a0 0x7f7cf419a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.188+0000 7f7cf2a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7cf41016b0 0x7f7cf4199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:35462/0 (socket says 172.21.15.45:35462) 2024-03-28T17:47:57.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.188+0000 7f7cf2000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cf41028a0 0x7f7cf419a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:47694/0 (socket says 172.21.15.45:47694) 2024-03-28T17:47:57.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.188+0000 7f7cf2a00640 1 -- 172.21.15.45:0/778208371 learned_addr learned my addr 172.21.15.45:0/778208371 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:57.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.188+0000 7f7cf3400640 1 --2- 172.21.15.45:0/778208371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cf4103260 0x7f7cf41a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.188+0000 7f7cf2000640 1 -- 172.21.15.45:0/778208371 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7cf41016b0 msgr2=0x7f7cf4199c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:57.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.188+0000 7f7cf2000640 1 --2- 172.21.15.45:0/778208371 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7cf41016b0 0x7f7cf4199c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.188+0000 7f7cf2000640 1 -- 172.21.15.45:0/778208371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cf4103260 msgr2=0x7f7cf41a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:57.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.188+0000 7f7cf2000640 1 --2- 172.21.15.45:0/778208371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cf4103260 0x7f7cf41a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.188+0000 7f7cf2000640 1 -- 172.21.15.45:0/778208371 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf41a1860 con 0x7f7cf41028a0 2024-03-28T17:47:57.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.188+0000 7f7cf2a00640 1 --2- 172.21.15.45:0/778208371 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7cf41016b0 0x7f7cf4199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:47:57.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.189+0000 7f7cf2000640 1 --2- 172.21.15.45:0/778208371 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cf41028a0 0x7f7cf419a190 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000e9f0 tx=0x7f7ce000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:57.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.189+0000 7f7ceb400640 1 -- 172.21.15.45:0/778208371 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce000ccc0 con 0x7f7cf41028a0 2024-03-28T17:47:57.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.189+0000 7f7ceb400640 1 -- 172.21.15.45:0/778208371 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ce0010910 con 0x7f7cf41028a0 2024-03-28T17:47:57.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.189+0000 7f7ceb400640 1 -- 172.21.15.45:0/778208371 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0018800 con 0x7f7cf41028a0 2024-03-28T17:47:57.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.189+0000 7f7cf9800640 1 -- 172.21.15.45:0/778208371 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf41a1b50 con 0x7f7cf41028a0 2024-03-28T17:47:57.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.189+0000 7f7cf9800640 1 -- 172.21.15.45:0/778208371 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf41a20f0 con 0x7f7cf41028a0 2024-03-28T17:47:57.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.190+0000 7f7cf9800640 1 -- 172.21.15.45:0/778208371 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf41a22c0 con 0x7f7cf41028a0 2024-03-28T17:47:57.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.193+0000 7f7ceb400640 1 -- 172.21.15.45:0/778208371 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7ce0010ac0 con 0x7f7cf41028a0 2024-03-28T17:47:57.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.194+0000 7f7ceb400640 1 --2- 172.21.15.45:0/778208371 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f7cd80a2880 0x7f7cd80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.194+0000 7f7ceb400640 1 -- 172.21.15.45:0/778208371 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f7ce00cede0 con 0x7f7cf41028a0 2024-03-28T17:47:57.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.194+0000 7f7cf2a00640 1 --2- 172.21.15.45:0/778208371 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f7cd80a2880 0x7f7cd80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.195+0000 7f7cf2a00640 1 --2- 172.21.15.45:0/778208371 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f7cd80a2880 0x7f7cd80a4d40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7cf419b120 tx=0x7f7ce402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:57.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.198+0000 7f7ceb400640 1 -- 172.21.15.45:0/778208371 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7ce009be30 con 0x7f7cf41028a0 2024-03-28T17:47:57.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.444+0000 7f7cf9800640 1 -- 172.21.15.45:0/778208371 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cf41a2490 con 0x7f7cf41028a0 2024-03-28T17:47:57.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.445+0000 7f7ceb400640 1 -- 172.21.15.45:0/778208371 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f7ce009b730 con 0x7f7cf41028a0 2024-03-28T17:47:57.445 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:47:57.445 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6839/2313500150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":2313500150},{"type":"v1","addr":"172.21.15.45:6839","nonce":2313500150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6841/2647972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":2647972571},{"type":"v1","addr":"172.21.15.45:6841","nonce":2647972571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T17:46:12.261135+0000","modified":"2024-03-28T17:46:17.967524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24584,"mds_1":24530,"mds_2":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6835/2876354410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":2876354410},{"type":"v1","addr":"172.21.15.151:6835","nonce":2876354410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24500":{"gid":24500,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/4234998385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":4234998385},{"type":"v1","addr":"172.21.15.71:6835","nonce":4234998385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6837/1238964845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":1238964845},{"type":"v1","addr":"172.21.15.151:6837","nonce":1238964845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.151:6839/3003564901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6838","nonce":3003564901},{"type":"v1","addr":"172.21.15.151:6839","nonce":3003564901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6837/2008752655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2008752655},{"type":"v1","addr":"172.21.15.71:6837","nonce":2008752655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.71:6839/2709938800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":2709938800},{"type":"v1","addr":"172.21.15.71:6839","nonce":2709938800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24584,"qdb_cluster":[24529,24530,24584]},"id":1}]} 2024-03-28T17:47:57.446 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 22 2024-03-28T17:47:57.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.448+0000 7f7cf9800640 1 -- 172.21.15.45:0/778208371 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f7cd80a2880 msgr2=0x7f7cd80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:57.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.448+0000 7f7cf9800640 1 --2- 172.21.15.45:0/778208371 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f7cd80a2880 0x7f7cd80a4d40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7cf419b120 tx=0x7f7ce402df90 comp rx=0 tx=0).stop 2024-03-28T17:47:57.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.448+0000 7f7cf9800640 1 -- 172.21.15.45:0/778208371 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cf41028a0 msgr2=0x7f7cf419a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:57.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.448+0000 7f7cf9800640 1 --2- 172.21.15.45:0/778208371 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cf41028a0 0x7f7cf419a190 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000e9f0 tx=0x7f7ce000edf0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.448+0000 7f7cf9800640 1 -- 172.21.15.45:0/778208371 shutdown_connections 2024-03-28T17:47:57.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.448+0000 7f7cf9800640 1 --2- 172.21.15.45:0/778208371 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f7cd80a2880 0x7f7cd80a4d40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.448+0000 7f7cf9800640 1 --2- 172.21.15.45:0/778208371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7cf4103260 0x7f7cf41a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.448+0000 7f7cf9800640 1 --2- 172.21.15.45:0/778208371 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cf41028a0 0x7f7cf419a190 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.448+0000 7f7cf9800640 1 --2- 172.21.15.45:0/778208371 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7cf41016b0 0x7f7cf4199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.448+0000 7f7cf9800640 1 -- 172.21.15.45:0/778208371 >> 172.21.15.45:0/778208371 conn(0x7f7cf40fcc80 msgr2=0x7f7cf40738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:57.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.448+0000 7f7cf9800640 1 -- 172.21.15.45:0/778208371 shutdown_connections 2024-03-28T17:47:57.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.448+0000 7f7cf9800640 1 -- 172.21.15.45:0/778208371 wait complete. 2024-03-28T17:47:57.468 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T17:46:12.261135+0000', 'modified': '2024-03-28T17:46:17.967524+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24584, 'mds_1': 24530, 'mds_2': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24499': {'gid': 24499, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.151:6835/2876354410', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6834', 'nonce': 2876354410}, {'type': 'v1', 'addr': '172.21.15.151:6835', 'nonce': 2876354410}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24500': {'gid': 24500, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.71:6835/4234998385', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6834', 'nonce': 4234998385}, {'type': 'v1', 'addr': '172.21.15.71:6835', 'nonce': 4234998385}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24511': {'gid': 24511, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.151:6837/1238964845', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6836', 'nonce': 1238964845}, {'type': 'v1', 'addr': '172.21.15.151:6837', 'nonce': 1238964845}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24529': {'gid': 24529, 'name': 'i', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.151:6839/3003564901', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6838', 'nonce': 3003564901}, {'type': 'v1', 'addr': '172.21.15.151:6839', 'nonce': 3003564901}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24530': {'gid': 24530, 'name': 'h', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6837/2008752655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6836', 'nonce': 2008752655}, {'type': 'v1', 'addr': '172.21.15.71:6837', 'nonce': 2008752655}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24584': {'gid': 24584, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.71:6839/2709938800', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6838', 'nonce': 2709938800}, {'type': 'v1', 'addr': '172.21.15.71:6839', 'nonce': 2709938800}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24584, 'qdb_cluster': [24529, 24530, 24584]} 2024-03-28T17:47:57.468 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-28T17:47:57.469 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-28T17:47:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:57 smithi045 ceph-mon[31773]: pgmap v371: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:47:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:57 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1170458933' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:47:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:57 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3052486036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:47:57.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.694+0000 7f5610000640 1 -- 172.21.15.45:0/2733138792 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56081017b0 msgr2=0x7f5608101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:57.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.694+0000 7f5610000640 1 --2- 172.21.15.45:0/2733138792 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56081017b0 0x7f5608101c30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f55f4009900 tx=0x7f55f402bc50 comp rx=0 tx=0).stop 2024-03-28T17:47:57.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.694+0000 7f5610000640 1 -- 172.21.15.45:0/2733138792 shutdown_connections 2024-03-28T17:47:57.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.694+0000 7f5610000640 1 --2- 172.21.15.45:0/2733138792 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5608102170 0x7f5608108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.694+0000 7f5610000640 1 --2- 172.21.15.45:0/2733138792 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56081017b0 0x7f5608101c30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.694+0000 7f5610000640 1 --2- 172.21.15.45:0/2733138792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56081005c0 0x7f56081009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.694+0000 7f5610000640 1 -- 172.21.15.45:0/2733138792 >> 172.21.15.45:0/2733138792 conn(0x7f56080fbb30 msgr2=0x7f56080fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:57.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.695+0000 7f5610000640 1 -- 172.21.15.45:0/2733138792 shutdown_connections 2024-03-28T17:47:57.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.695+0000 7f5610000640 1 -- 172.21.15.45:0/2733138792 wait complete. 2024-03-28T17:47:57.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.696+0000 7f5610000640 1 Processor -- start 2024-03-28T17:47:57.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.696+0000 7f5610000640 1 -- start start 2024-03-28T17:47:57.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.696+0000 7f5610000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56081005c0 0x7f5608199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.696+0000 7f5610000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56081017b0 0x7f560819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.696+0000 7f5610000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5608102170 0x7f56081a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.696+0000 7f5610000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560810c280 con 0x7f5608102170 2024-03-28T17:47:57.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.696+0000 7f5610000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560810c100 con 0x7f56081017b0 2024-03-28T17:47:57.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.696+0000 7f5610000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560810c400 con 0x7f56081005c0 2024-03-28T17:47:57.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.696+0000 7f560cc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56081017b0 0x7f560819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.696+0000 7f560d600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56081005c0 0x7f5608199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.696+0000 7f560cc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56081017b0 0x7f560819a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:35480/0 (socket says 172.21.15.45:35480) 2024-03-28T17:47:57.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.696+0000 7f560d600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56081005c0 0x7f5608199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:47716/0 (socket says 172.21.15.45:47716) 2024-03-28T17:47:57.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.696+0000 7f560cc00640 1 -- 172.21.15.45:0/2574455101 learned_addr learned my addr 172.21.15.45:0/2574455101 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:57.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.697+0000 7f560e000640 1 --2- 172.21.15.45:0/2574455101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5608102170 0x7f56081a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.697+0000 7f560cc00640 1 -- 172.21.15.45:0/2574455101 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56081005c0 msgr2=0x7f5608199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:57.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.697+0000 7f560cc00640 1 --2- 172.21.15.45:0/2574455101 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56081005c0 0x7f5608199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.697+0000 7f560cc00640 1 -- 172.21.15.45:0/2574455101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5608102170 msgr2=0x7f56081a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:57.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.697+0000 7f560cc00640 1 --2- 172.21.15.45:0/2574455101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5608102170 0x7f56081a1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.697+0000 7f560cc00640 1 -- 172.21.15.45:0/2574455101 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56081a18c0 con 0x7f56081017b0 2024-03-28T17:47:57.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.697+0000 7f560e000640 1 --2- 172.21.15.45:0/2574455101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5608102170 0x7f56081a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:47:57.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.697+0000 7f560d600640 1 --2- 172.21.15.45:0/2574455101 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56081005c0 0x7f5608199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:47:57.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.697+0000 7f560cc00640 1 --2- 172.21.15.45:0/2574455101 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56081017b0 0x7f560819a1c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f55f4000c00 tx=0x7f55f40049c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:57.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.698+0000 7f5606000640 1 -- 172.21.15.45:0/2574455101 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f4041070 con 0x7f56081017b0 2024-03-28T17:47:57.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.698+0000 7f5606000640 1 -- 172.21.15.45:0/2574455101 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f55f4004450 con 0x7f56081017b0 2024-03-28T17:47:57.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.698+0000 7f5610000640 1 -- 172.21.15.45:0/2574455101 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56081a1b50 con 0x7f56081017b0 2024-03-28T17:47:57.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.698+0000 7f5606000640 1 -- 172.21.15.45:0/2574455101 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f4034440 con 0x7f56081017b0 2024-03-28T17:47:57.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.698+0000 7f5610000640 1 -- 172.21.15.45:0/2574455101 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56081a20f0 con 0x7f56081017b0 2024-03-28T17:47:57.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.702+0000 7f5606000640 1 -- 172.21.15.45:0/2574455101 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f55f4044070 con 0x7f56081017b0 2024-03-28T17:47:57.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.702+0000 7f5612d71740 1 Processor -- start 2024-03-28T17:47:57.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.702+0000 7f5612d71740 1 -- start start 2024-03-28T17:47:57.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.703+0000 7f5612d71740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfbba0 0x561ef9dfbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.703+0000 7f5612d71740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9dfc590 0x561ef9e01090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.703+0000 7f5612d71740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef9e01700 0x561ef9e03b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.703+0000 7f5612d71740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef9d60c90 con 0x561ef9e01700 2024-03-28T17:47:57.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.703+0000 7f5612d71740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef9d09980 con 0x561ef9dfc590 2024-03-28T17:47:57.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.703+0000 7f5612d71740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef9d5cd30 con 0x561ef9dfbba0 2024-03-28T17:47:57.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.703+0000 7f560e000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfbba0 0x561ef9dfbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.703+0000 7f560d600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9dfc590 0x561ef9e01090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.703+0000 7f560e000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfbba0 0x561ef9dfbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:47720/0 (socket says 172.21.15.45:47720) 2024-03-28T17:47:57.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.703+0000 7f560d600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9dfc590 0x561ef9e01090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:35482/0 (socket says 172.21.15.45:35482) 2024-03-28T17:47:57.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.703+0000 7f560e000640 1 -- 172.21.15.45:0/714049912 learned_addr learned my addr 172.21.15.45:0/714049912 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:57.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.703+0000 7f560cc00640 1 --2- 172.21.15.45:0/714049912 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef9e01700 0x561ef9e03b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.703+0000 7f560d600640 1 -- 172.21.15.45:0/714049912 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfbba0 msgr2=0x561ef9dfbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:57.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.704+0000 7f560d600640 1 --2- 172.21.15.45:0/714049912 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfbba0 0x561ef9dfbfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.704+0000 7f560d600640 1 -- 172.21.15.45:0/714049912 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef9e01700 msgr2=0x561ef9e03b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:57.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.704+0000 7f560d600640 1 --2- 172.21.15.45:0/714049912 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef9e01700 0x561ef9e03b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.704+0000 7f560d600640 1 -- 172.21.15.45:0/714049912 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ef9d21510 con 0x561ef9dfc590 2024-03-28T17:47:57.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.704+0000 7f560cc00640 1 --2- 172.21.15.45:0/714049912 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef9e01700 0x561ef9e03b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:47:57.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.704+0000 7f560d600640 1 --2- 172.21.15.45:0/714049912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9dfc590 0x561ef9e01090 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f55fc005fd0 tx=0x7f55fc0108e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f90f6c96e322d6e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:57.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.704+0000 7f5606000640 1 --2- 172.21.15.45:0/2574455101 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f55f00a2880 0x7f55f00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.704+0000 7f55fa600640 1 -- 172.21.15.45:0/714049912 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55fc004230 con 0x561ef9dfc590 2024-03-28T17:47:57.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.704+0000 7f55fa600640 1 -- 172.21.15.45:0/714049912 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f55fc0043d0 con 0x561ef9dfc590 2024-03-28T17:47:57.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.704+0000 7f55fa600640 1 -- 172.21.15.45:0/714049912 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55fc011620 con 0x561ef9dfc590 2024-03-28T17:47:57.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.704+0000 7f5606000640 1 -- 172.21.15.45:0/2574455101 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f55f40f06a0 con 0x7f56081017b0 2024-03-28T17:47:57.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.704+0000 7f560e000640 1 --2- 172.21.15.45:0/2574455101 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f55f00a2880 0x7f55f00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.705+0000 7f5612d71740 1 -- 172.21.15.45:0/714049912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9dfc590 msgr2=0x561ef9e01090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:57.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.705+0000 7f5612d71740 1 --2- 172.21.15.45:0/714049912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9dfc590 0x561ef9e01090 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f55fc005fd0 tx=0x7f55fc0108e0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.705+0000 7f560e000640 1 --2- 172.21.15.45:0/2574455101 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f55f00a2880 0x7f55f00a4d40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x561ef9dfcdf0 tx=0x7f5600009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:57.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.705+0000 7f5612d71740 1 -- 172.21.15.45:0/714049912 shutdown_connections 2024-03-28T17:47:57.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.705+0000 7f5612d71740 1 --2- 172.21.15.45:0/714049912 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef9e01700 0x561ef9e03b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.705+0000 7f5612d71740 1 --2- 172.21.15.45:0/714049912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9dfc590 0x561ef9e01090 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.705+0000 7f5612d71740 1 --2- 172.21.15.45:0/714049912 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfbba0 0x561ef9dfbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.705+0000 7f5612d71740 1 -- 172.21.15.45:0/714049912 >> 172.21.15.45:0/714049912 conn(0x561ef9df7110 msgr2=0x561ef9df7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:57.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.705+0000 7f5612d71740 1 -- 172.21.15.45:0/714049912 shutdown_connections 2024-03-28T17:47:57.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.705+0000 7f5612d71740 1 -- 172.21.15.45:0/714049912 wait complete. 2024-03-28T17:47:57.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.706+0000 7f5612d71740 1 -- start start 2024-03-28T17:47:57.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.706+0000 7f5612d71740 1 Processor -- start 2024-03-28T17:47:57.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.706+0000 7f5612d71740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfbba0 0x561ef9e060e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.706+0000 7f5612d71740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9dfc590 0x561ef9e06620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.706+0000 7f5612d71740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef9e01700 0x561ef9e0d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.706+0000 7f5612d71740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef9e046b0 con 0x561ef9e01700 2024-03-28T17:47:57.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.706+0000 7f5612d71740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef9d1cdd0 con 0x561ef9dfc590 2024-03-28T17:47:57.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.706+0000 7f5612d71740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef9e04830 con 0x561ef9dfbba0 2024-03-28T17:47:57.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.707+0000 7f5612d71740 10 client.0 _ll_register_callbacks cb 0x561ef9a74ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:47:57.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.707+0000 7f560cc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9dfc590 0x561ef9e06620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.707+0000 7f560d600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfbba0 0x561ef9e060e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.707+0000 7f560cc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9dfc590 0x561ef9e06620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:35494/0 (socket says 172.21.15.45:35494) 2024-03-28T17:47:57.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.707+0000 7f560cc00640 1 -- 172.21.15.45:0/2849294947 learned_addr learned my addr 172.21.15.45:0/2849294947 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:57.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.707+0000 7f560e000640 1 --2- 172.21.15.45:0/2849294947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef9e01700 0x561ef9e0d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.707+0000 7f560cc00640 1 -- 172.21.15.45:0/2849294947 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfbba0 msgr2=0x561ef9e060e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:57.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.707+0000 7f560cc00640 1 --2- 172.21.15.45:0/2849294947 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfbba0 0x561ef9e060e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.707+0000 7f560cc00640 1 -- 172.21.15.45:0/2849294947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef9e01700 msgr2=0x561ef9e0d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:57.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.707+0000 7f560cc00640 1 --2- 172.21.15.45:0/2849294947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef9e01700 0x561ef9e0d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.707+0000 7f560cc00640 1 -- 172.21.15.45:0/2849294947 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561ef9a752d0 con 0x561ef9dfc590 2024-03-28T17:47:57.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.707+0000 7f560e000640 1 --2- 172.21.15.45:0/2849294947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef9e01700 0x561ef9e0d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:47:57.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.707+0000 7f560cc00640 1 --2- 172.21.15.45:0/2849294947 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9dfc590 0x561ef9e06620 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f55f40f62d0 tx=0x7f55f40056c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80c315bc0c85cbf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:57.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.707+0000 7f55eea00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:47:57.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.707+0000 7f55eea00640 1 -- 172.21.15.45:0/2849294947 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f404b050 con 0x561ef9dfc590 2024-03-28T17:47:57.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.708+0000 7f55eea00640 1 -- 172.21.15.45:0/2849294947 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f55f40b9660 con 0x561ef9dfc590 2024-03-28T17:47:57.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.708+0000 7f5612d71740 1 -- 172.21.15.45:0/2849294947 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f55fc0043d0 con 0x561ef9dfc590 2024-03-28T17:47:57.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.708+0000 7f55eea00640 1 -- 172.21.15.45:0/2849294947 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f55f40bc950 con 0x561ef9dfc590 2024-03-28T17:47:57.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.708+0000 7f5612d71740 10 client.24989 fetch_fsmap learned FSMap version 22 2024-03-28T17:47:57.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.708+0000 7f5612d71740 1 -- 172.21.15.45:0/2849294947 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x561ef9b3f6b0 con 0x561ef9dfc590 2024-03-28T17:47:57.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.708+0000 7f55eea00640 1 -- 172.21.15.45:0/2849294947 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f55fc0043d0 con 0x561ef9dfc590 2024-03-28T17:47:57.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.708+0000 7f55eea00640 1 -- 172.21.15.45:0/2849294947 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f55f40c5300 con 0x561ef9dfc590 2024-03-28T17:47:57.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.708+0000 7f5612d71740 10 client.24989 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T17:47:57.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.708+0000 7f5612d71740 10 client.24989 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T17:47:57.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.708+0000 7f5612d71740 20 client.24989 populate_metadata read hostname 'smithi045' 2024-03-28T17:47:57.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.708+0000 7f5612d71740 1 --2- 172.21.15.45:0/2849294947 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x561ef9e12800 0x561ef9e117c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.708+0000 7f5612d71740 4 client.24989 mds_command: new command op to 24529 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:47:57.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.708+0000 7f5612d71740 1 -- 172.21.15.45:0/2849294947 --> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561ef9d02520 con 0x561ef9e12800 2024-03-28T17:47:57.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.709+0000 7f560d600640 1 --2- 172.21.15.45:0/2849294947 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x561ef9e12800 0x561ef9e117c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.709+0000 7f560d600640 1 --2- 172.21.15.45:0/2849294947 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x561ef9e12800 0x561ef9e117c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e2bb1241c46e8623 server_cookie=4c4d4f441dea0a85 in_seq=0 out_seq=0 2024-03-28T17:47:57.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.709+0000 7f55eea00640 10 client.24989 ms_handle_connect on v2:172.21.15.151:6838/3003564901 2024-03-28T17:47:57.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.710+0000 7f55eea00640 1 -- 172.21.15.45:0/2849294947 <== mds.2 v2:172.21.15.151:6838/3003564901 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561ef9d02520 con 0x561ef9e12800 2024-03-28T17:47:57.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.710+0000 7f55eea00640 10 client.24989 handle_command_reply: tid=0 2024-03-28T17:47:57.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.710+0000 7f5612d71740 1 client.24989 shutdown 2024-03-28T17:47:57.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.710+0000 7f5612d71740 1 -- 172.21.15.45:0/2849294947 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9dfc590 msgr2=0x561ef9e06620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:57.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.710+0000 7f5612d71740 1 --2- 172.21.15.45:0/2849294947 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9dfc590 0x561ef9e06620 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f55f40f62d0 tx=0x7f55f40056c0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.710+0000 7f5612d71740 1 -- 172.21.15.45:0/2849294947 shutdown_connections 2024-03-28T17:47:57.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.710+0000 7f5612d71740 1 --2- 172.21.15.45:0/2849294947 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x561ef9e12800 0x561ef9e117c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.710+0000 7f5612d71740 1 --2- 172.21.15.45:0/2849294947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef9e01700 0x561ef9e0d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.710+0000 7f5612d71740 1 --2- 172.21.15.45:0/2849294947 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9dfc590 0x561ef9e06620 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.710+0000 7f55eea00640 0 client.24989 ms_handle_reset on v2:172.21.15.151:6838/3003564901 2024-03-28T17:47:57.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.710+0000 7f5612d71740 1 --2- 172.21.15.45:0/2849294947 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfbba0 0x561ef9e060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.710+0000 7f5612d71740 1 -- 172.21.15.45:0/2849294947 >> 172.21.15.45:0/2849294947 conn(0x561ef9df7110 msgr2=0x561ef9df6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:57.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.711+0000 7f5612d71740 1 -- 172.21.15.45:0/2849294947 shutdown_connections 2024-03-28T17:47:57.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.711+0000 7f5612d71740 1 -- 172.21.15.45:0/2849294947 wait complete. 2024-03-28T17:47:57.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.711+0000 7f5612d71740 20 client.24989 trim_cache size 0 max 16384 2024-03-28T17:47:57.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.730+0000 7f5612d71740 1 Processor -- start 2024-03-28T17:47:57.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.730+0000 7f5612d71740 1 -- start start 2024-03-28T17:47:57.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.730+0000 7f5612d71740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef9dfbba0 0x561ef9e117c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.730+0000 7f5612d71740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfc590 0x561ef9df5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.730+0000 7f5612d71740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9e01700 0x561ef9df5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.730+0000 7f5612d71740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef9e12020 con 0x561ef9dfbba0 2024-03-28T17:47:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.730+0000 7f5612d71740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef9e12680 con 0x561ef9e01700 2024-03-28T17:47:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.730+0000 7f5612d71740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef9e12ce0 con 0x561ef9dfc590 2024-03-28T17:47:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.730+0000 7f560e000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9e01700 0x561ef9df5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.730+0000 7f560cc00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfc590 0x561ef9df5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.731+0000 7f560cc00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfc590 0x561ef9df5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:47738/0 (socket says 172.21.15.45:47738) 2024-03-28T17:47:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.731+0000 7f560cc00640 1 -- 172.21.15.45:0/363784493 learned_addr learned my addr 172.21.15.45:0/363784493 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.731+0000 7f560d600640 1 --2- 172.21.15.45:0/363784493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef9dfbba0 0x561ef9e117c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.731+0000 7f560e000640 1 -- 172.21.15.45:0/363784493 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfc590 msgr2=0x561ef9df5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.731+0000 7f560e000640 1 --2- 172.21.15.45:0/363784493 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfc590 0x561ef9df5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.731+0000 7f560e000640 1 -- 172.21.15.45:0/363784493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef9dfbba0 msgr2=0x561ef9e117c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.731+0000 7f560e000640 1 --2- 172.21.15.45:0/363784493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef9dfbba0 0x561ef9e117c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.731+0000 7f560e000640 1 -- 172.21.15.45:0/363784493 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ef9df6f30 con 0x561ef9e01700 2024-03-28T17:47:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.731+0000 7f560cc00640 1 --2- 172.21.15.45:0/363784493 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfc590 0x561ef9df5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:47:57.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.731+0000 7f560d600640 1 --2- 172.21.15.45:0/363784493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef9dfbba0 0x561ef9e117c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:47:57.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.731+0000 7f560e000640 1 --2- 172.21.15.45:0/363784493 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9e01700 0x561ef9df5fc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5600005f10 tx=0x7f5600016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=576e868b54cda71b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:57.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.731+0000 7f55f9c00640 1 -- 172.21.15.45:0/363784493 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5600004b20 con 0x561ef9e01700 2024-03-28T17:47:57.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.732+0000 7f55f9c00640 1 -- 172.21.15.45:0/363784493 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f560001fde0 con 0x561ef9e01700 2024-03-28T17:47:57.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.732+0000 7f55f9c00640 1 -- 172.21.15.45:0/363784493 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5600005450 con 0x561ef9e01700 2024-03-28T17:47:57.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.732+0000 7f5612d71740 1 -- 172.21.15.45:0/363784493 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9e01700 msgr2=0x561ef9df5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:57.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.732+0000 7f5612d71740 1 --2- 172.21.15.45:0/363784493 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9e01700 0x561ef9df5fc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5600005f10 tx=0x7f5600016000 comp rx=0 tx=0).stop 2024-03-28T17:47:57.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.732+0000 7f5612d71740 1 -- 172.21.15.45:0/363784493 shutdown_connections 2024-03-28T17:47:57.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.732+0000 7f5612d71740 1 --2- 172.21.15.45:0/363784493 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9e01700 0x561ef9df5fc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.732+0000 7f5612d71740 1 --2- 172.21.15.45:0/363784493 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfc590 0x561ef9df5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.732+0000 7f5612d71740 1 --2- 172.21.15.45:0/363784493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef9dfbba0 0x561ef9e117c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.732+0000 7f5612d71740 1 -- 172.21.15.45:0/363784493 >> 172.21.15.45:0/363784493 conn(0x561ef9df7110 msgr2=0x561ef9df1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:57.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.732+0000 7f5612d71740 1 -- 172.21.15.45:0/363784493 shutdown_connections 2024-03-28T17:47:57.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.733+0000 7f5612d71740 1 -- 172.21.15.45:0/363784493 wait complete. 2024-03-28T17:47:57.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.733+0000 7f5612d71740 1 -- start start 2024-03-28T17:47:57.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.733+0000 7f5612d71740 1 Processor -- start 2024-03-28T17:47:57.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.734+0000 7f5612d71740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfbba0 0x561ef9e0c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.734+0000 7f5612d71740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef9dfc590 0x561ef9e0cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.734+0000 7f5612d71740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9e01700 0x561ef9e09320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.734+0000 7f5612d71740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef9e139a0 con 0x561ef9dfc590 2024-03-28T17:47:57.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.734+0000 7f5612d71740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef9e13340 con 0x561ef9e01700 2024-03-28T17:47:57.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.734+0000 7f5612d71740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef9e14000 con 0x561ef9dfbba0 2024-03-28T17:47:57.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.734+0000 7f5612d71740 10 client.0 _ll_register_callbacks cb 0x7f55f4003f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:47:57.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.734+0000 7f560d600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfbba0 0x561ef9e0c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.734+0000 7f560d600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfbba0 0x561ef9e0c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:47744/0 (socket says 172.21.15.45:47744) 2024-03-28T17:47:57.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.734+0000 7f560d600640 1 -- 172.21.15.45:0/3289109942 learned_addr learned my addr 172.21.15.45:0/3289109942 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:57.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.734+0000 7f560cc00640 1 --2- 172.21.15.45:0/3289109942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef9dfc590 0x561ef9e0cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.734+0000 7f560e000640 1 --2- 172.21.15.45:0/3289109942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9e01700 0x561ef9e09320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.734+0000 7f560d600640 1 -- 172.21.15.45:0/3289109942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9e01700 msgr2=0x561ef9e09320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:57.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.734+0000 7f560d600640 1 --2- 172.21.15.45:0/3289109942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9e01700 0x561ef9e09320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.734+0000 7f560d600640 1 -- 172.21.15.45:0/3289109942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef9dfc590 msgr2=0x561ef9e0cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:57.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.734+0000 7f560d600640 1 --2- 172.21.15.45:0/3289109942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef9dfc590 0x561ef9e0cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.734+0000 7f560d600640 1 -- 172.21.15.45:0/3289109942 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561ef9e09b50 con 0x561ef9dfbba0 2024-03-28T17:47:57.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.735+0000 7f560d600640 1 --2- 172.21.15.45:0/3289109942 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfbba0 0x561ef9e0c810 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f55fc005fd0 tx=0x7f55fc0103d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d305cdbef7f99f28 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:57.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.735+0000 7f55eea00640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:47:57.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.735+0000 7f55eea00640 1 -- 172.21.15.45:0/3289109942 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55fc02a250 con 0x561ef9dfbba0 2024-03-28T17:47:57.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.735+0000 7f55eea00640 1 -- 172.21.15.45:0/3289109942 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f55fc004070 con 0x561ef9dfbba0 2024-03-28T17:47:57.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.735+0000 7f5612d71740 1 -- 172.21.15.45:0/3289109942 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f560001fde0 con 0x561ef9dfbba0 2024-03-28T17:47:57.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.735+0000 7f55eea00640 1 -- 172.21.15.45:0/3289109942 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f55fc00bb00 con 0x561ef9dfbba0 2024-03-28T17:47:57.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.735+0000 7f5612d71740 10 client.24970 fetch_fsmap learned FSMap version 22 2024-03-28T17:47:57.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.735+0000 7f5612d71740 1 -- 172.21.15.45:0/3289109942 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x561ef9e76de0 con 0x561ef9dfbba0 2024-03-28T17:47:57.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.735+0000 7f55eea00640 1 -- 172.21.15.45:0/3289109942 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f560001fde0 con 0x561ef9dfbba0 2024-03-28T17:47:57.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.736+0000 7f55eea00640 1 -- 172.21.15.45:0/3289109942 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f55fc0035f0 con 0x561ef9dfbba0 2024-03-28T17:47:57.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.736+0000 7f5612d71740 10 client.24970 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T17:47:57.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.736+0000 7f5612d71740 10 client.24970 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T17:47:57.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.736+0000 7f5612d71740 20 client.24970 populate_metadata read hostname 'smithi045' 2024-03-28T17:47:57.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.736+0000 7f5612d71740 1 --2- 172.21.15.45:0/3289109942 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x561ef9e7b190 0x561ef9e770d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.736+0000 7f5612d71740 4 client.24970 mds_command: new command op to 24529 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T17:47:57.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.736+0000 7f5612d71740 1 -- 172.21.15.45:0/3289109942 --> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561ef9d6e1d0 con 0x561ef9e7b190 2024-03-28T17:47:57.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.736+0000 7f560cc00640 1 --2- 172.21.15.45:0/3289109942 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x561ef9e7b190 0x561ef9e770d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.736+0000 7f560cc00640 1 --2- 172.21.15.45:0/3289109942 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x561ef9e7b190 0x561ef9e770d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c0150709c8b13ac3 server_cookie=37cb35b51d3ae88 in_seq=0 out_seq=0 2024-03-28T17:47:57.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.736+0000 7f55eea00640 10 client.24970 ms_handle_connect on v2:172.21.15.151:6838/3003564901 2024-03-28T17:47:57.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.737+0000 7f55eea00640 1 -- 172.21.15.45:0/3289109942 <== mds.2 v2:172.21.15.151:6838/3003564901 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+321 (crc 0 0 0) 0x561ef9d6e1d0 con 0x561ef9e7b190 2024-03-28T17:47:57.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.737+0000 7f55eea00640 10 client.24970 handle_command_reply: tid=0 2024-03-28T17:47:57.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.737+0000 7f5612d71740 1 client.24970 shutdown 2024-03-28T17:47:57.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.737+0000 7f5612d71740 1 -- 172.21.15.45:0/3289109942 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfbba0 msgr2=0x561ef9e0c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:57.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.737+0000 7f5612d71740 1 --2- 172.21.15.45:0/3289109942 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfbba0 0x561ef9e0c810 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f55fc005fd0 tx=0x7f55fc0103d0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.737+0000 7f5612d71740 1 -- 172.21.15.45:0/3289109942 shutdown_connections 2024-03-28T17:47:57.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.737+0000 7f5612d71740 1 --2- 172.21.15.45:0/3289109942 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x561ef9e7b190 0x561ef9e770d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.737+0000 7f5612d71740 1 --2- 172.21.15.45:0/3289109942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561ef9e01700 0x561ef9e09320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.737+0000 7f5612d71740 1 --2- 172.21.15.45:0/3289109942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef9dfc590 0x561ef9e0cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.737+0000 7f5612d71740 1 --2- 172.21.15.45:0/3289109942 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561ef9dfbba0 0x561ef9e0c810 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.737+0000 7f55eea00640 0 client.24970 ms_handle_reset on v2:172.21.15.151:6838/3003564901 2024-03-28T17:47:57.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.737+0000 7f5612d71740 1 -- 172.21.15.45:0/3289109942 >> 172.21.15.45:0/3289109942 conn(0x561ef9df7110 msgr2=0x561ef9e117c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:57.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.737+0000 7f5612d71740 1 -- 172.21.15.45:0/3289109942 shutdown_connections 2024-03-28T17:47:57.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.737+0000 7f5612d71740 1 -- 172.21.15.45:0/3289109942 wait complete. 2024-03-28T17:47:57.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.737+0000 7f5612d71740 20 client.24970 trim_cache size 0 max 16384 2024-03-28T17:47:57.741 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:47:57.741 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "9f4888ec-ed29-11ee-95d0-87774f69a715", 2024-03-28T17:47:57.741 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 2, 2024-03-28T17:47:57.741 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 24529, 2024-03-28T17:47:57.741 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-03-28T17:47:57.741 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-03-28T17:47:57.742 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-03-28T17:47:57.742 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 100.775274255, 2024-03-28T17:47:57.742 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 22, 2024-03-28T17:47:57.742 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 87, 2024-03-28T17:47:57.742 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 85, 2024-03-28T17:47:57.742 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 123.07843328 2024-03-28T17:47:57.742 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:47:57.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.738+0000 7f5610000640 1 -- 172.21.15.45:0/2574455101 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f55f00a2880 msgr2=0x7f55f00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:57.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.738+0000 7f5610000640 1 --2- 172.21.15.45:0/2574455101 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f55f00a2880 0x7f55f00a4d40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x561ef9dfcdf0 tx=0x7f5600009500 comp rx=0 tx=0).stop 2024-03-28T17:47:57.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.738+0000 7f5610000640 1 -- 172.21.15.45:0/2574455101 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56081017b0 msgr2=0x7f560819a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:57.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.738+0000 7f5610000640 1 --2- 172.21.15.45:0/2574455101 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56081017b0 0x7f560819a1c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f55f4000c00 tx=0x7f55f40049c0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.738+0000 7f5610000640 1 -- 172.21.15.45:0/2574455101 shutdown_connections 2024-03-28T17:47:57.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.738+0000 7f5610000640 1 --2- 172.21.15.45:0/2574455101 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f55f00a2880 0x7f55f00a4d40 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.738+0000 7f5610000640 1 --2- 172.21.15.45:0/2574455101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5608102170 0x7f56081a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.738+0000 7f5610000640 1 --2- 172.21.15.45:0/2574455101 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56081017b0 0x7f560819a1c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.738+0000 7f5610000640 1 --2- 172.21.15.45:0/2574455101 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56081005c0 0x7f5608199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.738+0000 7f5610000640 1 -- 172.21.15.45:0/2574455101 >> 172.21.15.45:0/2574455101 conn(0x7f56080fbb30 msgr2=0x7f5608069940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:57.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.738+0000 7f5610000640 1 -- 172.21.15.45:0/2574455101 shutdown_connections 2024-03-28T17:47:57.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.738+0000 7f5610000640 1 -- 172.21.15.45:0/2574455101 wait complete. 2024-03-28T17:47:57.756 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-28T17:47:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:57 smithi071 ceph-mon[36597]: pgmap v371: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:47:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:57 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1170458933' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:47:57.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:57 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3052486036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:47:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:57 smithi151 ceph-mon[36481]: pgmap v371: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:47:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:57 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1170458933' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:47:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:57 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3052486036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:47:57.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.975+0000 7f19f7200640 1 -- 172.21.15.45:0/643548710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19f8108db0 msgr2=0x7f19f8109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:57.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.975+0000 7f19f7200640 1 --2- 172.21.15.45:0/643548710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19f8108db0 0x7f19f8109230 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f19e0009900 tx=0x7f19e002bc30 comp rx=0 tx=0).stop 2024-03-28T17:47:57.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.975+0000 7f19f7200640 1 -- 172.21.15.45:0/643548710 shutdown_connections 2024-03-28T17:47:57.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.975+0000 7f19f7200640 1 --2- 172.21.15.45:0/643548710 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f19f8109770 0x7f19f8110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.975+0000 7f19f7200640 1 --2- 172.21.15.45:0/643548710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19f8108db0 0x7f19f8109230 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.975+0000 7f19f7200640 1 --2- 172.21.15.45:0/643548710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19f8107bc0 0x7f19f8107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.975+0000 7f19f7200640 1 -- 172.21.15.45:0/643548710 >> 172.21.15.45:0/643548710 conn(0x7f19f8076b70 msgr2=0x7f19f8078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:57.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.975+0000 7f19f7200640 1 -- 172.21.15.45:0/643548710 shutdown_connections 2024-03-28T17:47:57.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.975+0000 7f19f7200640 1 -- 172.21.15.45:0/643548710 wait complete. 2024-03-28T17:47:57.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.976+0000 7f19f7200640 1 Processor -- start 2024-03-28T17:47:57.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.976+0000 7f19f7200640 1 -- start start 2024-03-28T17:47:57.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.976+0000 7f19f7200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f19f8107bc0 0x7f19f819deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.976+0000 7f19f7200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19f8108db0 0x7f19f819e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.976+0000 7f19f7200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19f8109770 0x7f19f81a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.976+0000 7f19f7200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f8113890 con 0x7f19f8108db0 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.976+0000 7f19f7200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f8113710 con 0x7f19f8109770 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.976+0000 7f19f7200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f8113a10 con 0x7f19f8107bc0 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.976+0000 7f19f5400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19f8108db0 0x7f19f819e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.977+0000 7f19f6800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19f8109770 0x7f19f81a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.977+0000 7f19f5e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f19f8107bc0 0x7f19f819deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.977+0000 7f19f5400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19f8108db0 0x7f19f819e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44578/0 (socket says 172.21.15.45:44578) 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.977+0000 7f19f5400640 1 -- 172.21.15.45:0/3854754483 learned_addr learned my addr 172.21.15.45:0/3854754483 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.977+0000 7f19f5400640 1 -- 172.21.15.45:0/3854754483 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f19f8107bc0 msgr2=0x7f19f819deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.977+0000 7f19f5400640 1 --2- 172.21.15.45:0/3854754483 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f19f8107bc0 0x7f19f819deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.977+0000 7f19f5400640 1 -- 172.21.15.45:0/3854754483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19f8109770 msgr2=0x7f19f81a5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.977+0000 7f19f5400640 1 --2- 172.21.15.45:0/3854754483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19f8109770 0x7f19f81a5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.977+0000 7f19f5400640 1 -- 172.21.15.45:0/3854754483 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f81a5ad0 con 0x7f19f8108db0 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.977+0000 7f19f6800640 1 --2- 172.21.15.45:0/3854754483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19f8109770 0x7f19f81a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.977+0000 7f19f5400640 1 --2- 172.21.15.45:0/3854754483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19f8108db0 0x7f19f819e3f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f19e0000c00 tx=0x7f19e002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.978+0000 7f19eea00640 1 -- 172.21.15.45:0/3854754483 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0043020 con 0x7f19f8108db0 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.978+0000 7f19f7200640 1 -- 172.21.15.45:0/3854754483 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f81a5d60 con 0x7f19f8108db0 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.978+0000 7f19eea00640 1 -- 172.21.15.45:0/3854754483 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19e00054e0 con 0x7f19f8108db0 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.978+0000 7f19eea00640 1 -- 172.21.15.45:0/3854754483 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0003c40 con 0x7f19f8108db0 2024-03-28T17:47:57.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.978+0000 7f19f7200640 1 -- 172.21.15.45:0/3854754483 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f81a6220 con 0x7f19f8108db0 2024-03-28T17:47:57.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.981+0000 7f19eea00640 1 -- 172.21.15.45:0/3854754483 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f19e0046020 con 0x7f19f8108db0 2024-03-28T17:47:57.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.981+0000 7f19ff97e740 1 Processor -- start 2024-03-28T17:47:57.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.981+0000 7f19ff97e740 1 -- start start 2024-03-28T17:47:57.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.981+0000 7f19ff97e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee17d80 0x563eeee181a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.982+0000 7f19ff97e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563eeee18770 0x563eeee1d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.982+0000 7f19ff97e740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563eeee1d8e0 0x563eeee1fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.982+0000 7f19ff97e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563eeed7cc90 con 0x563eeee18770 2024-03-28T17:47:57.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.982+0000 7f19ff97e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x563eeed25980 con 0x563eeee17d80 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.982+0000 7f19ff97e740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x563eeed78d30 con 0x563eeee1d8e0 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.982+0000 7f19f6800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee17d80 0x563eeee181a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.982+0000 7f19f6800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee17d80 0x563eeee181a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:35546/0 (socket says 172.21.15.45:35546) 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.982+0000 7f19f5400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563eeee1d8e0 0x563eeee1fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.982+0000 7f19f6800640 1 -- 172.21.15.45:0/287927843 learned_addr learned my addr 172.21.15.45:0/287927843 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.982+0000 7f19f5e00640 1 --2- 172.21.15.45:0/287927843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563eeee18770 0x563eeee1d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.982+0000 7f19f6800640 1 -- 172.21.15.45:0/287927843 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563eeee1d8e0 msgr2=0x563eeee1fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.982+0000 7f19f6800640 1 --2- 172.21.15.45:0/287927843 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563eeee1d8e0 0x563eeee1fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.982+0000 7f19f6800640 1 -- 172.21.15.45:0/287927843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563eeee18770 msgr2=0x563eeee1d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.982+0000 7f19f6800640 1 --2- 172.21.15.45:0/287927843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563eeee18770 0x563eeee1d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.982+0000 7f19f6800640 1 -- 172.21.15.45:0/287927843 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563eeed3d510 con 0x563eeee17d80 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.982+0000 7f19f5e00640 1 --2- 172.21.15.45:0/287927843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563eeee18770 0x563eeee1d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:47:57.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.982+0000 7f19eea00640 1 --2- 172.21.15.45:0/3854754483 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f19dc0a2800 0x7f19dc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.983+0000 7f19f6800640 1 --2- 172.21.15.45:0/287927843 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee17d80 0x563eeee181a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f19f0011540 tx=0x7f19f0011a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5b43d15615ec25a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:57.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.983+0000 7f19eea00640 1 -- 172.21.15.45:0/3854754483 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f19e00f0af0 con 0x7f19f8108db0 2024-03-28T17:47:57.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.983+0000 7f19f5e00640 1 --2- 172.21.15.45:0/3854754483 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f19dc0a2800 0x7f19dc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.983+0000 7f19e7000640 1 -- 172.21.15.45:0/287927843 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f000f5c0 con 0x563eeee17d80 2024-03-28T17:47:57.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.983+0000 7f19e7000640 1 -- 172.21.15.45:0/287927843 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19f000fc40 con 0x563eeee17d80 2024-03-28T17:47:57.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.983+0000 7f19e7000640 1 -- 172.21.15.45:0/287927843 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f0019d20 con 0x563eeee17d80 2024-03-28T17:47:57.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.984+0000 7f19ff97e740 1 -- 172.21.15.45:0/287927843 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee17d80 msgr2=0x563eeee181a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:57.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.984+0000 7f19ff97e740 1 --2- 172.21.15.45:0/287927843 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee17d80 0x563eeee181a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f19f0011540 tx=0x7f19f0011a00 comp rx=0 tx=0).stop 2024-03-28T17:47:57.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.984+0000 7f19f5e00640 1 --2- 172.21.15.45:0/3854754483 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f19dc0a2800 0x7f19dc0a4cc0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x563eeee18b70 tx=0x7f19e8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:57.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.984+0000 7f19ff97e740 1 -- 172.21.15.45:0/287927843 shutdown_connections 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.984+0000 7f19ff97e740 1 --2- 172.21.15.45:0/287927843 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563eeee1d8e0 0x563eeee1fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.984+0000 7f19ff97e740 1 --2- 172.21.15.45:0/287927843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563eeee18770 0x563eeee1d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.984+0000 7f19ff97e740 1 --2- 172.21.15.45:0/287927843 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee17d80 0x563eeee181a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.984+0000 7f19ff97e740 1 -- 172.21.15.45:0/287927843 >> 172.21.15.45:0/287927843 conn(0x563eeee132f0 msgr2=0x563eeee136f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.984+0000 7f19ff97e740 1 -- 172.21.15.45:0/287927843 shutdown_connections 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.984+0000 7f19ff97e740 1 -- 172.21.15.45:0/287927843 wait complete. 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.984+0000 7f19ff97e740 1 -- start start 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.984+0000 7f19ff97e740 1 Processor -- start 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.985+0000 7f19ff97e740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563eeee17d80 0x563eeee222f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.985+0000 7f19ff97e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee18770 0x563eeee22830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.985+0000 7f19ff97e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563eeee1d8e0 0x563eeee29870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.985+0000 7f19ff97e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563eeee20890 con 0x563eeee1d8e0 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.985+0000 7f19ff97e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x563eeed38dd0 con 0x563eeee18770 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.985+0000 7f19ff97e740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x563eeee20a10 con 0x563eeee17d80 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.985+0000 7f19ff97e740 10 client.0 _ll_register_callbacks cb 0x563eeea90ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.985+0000 7f19f5400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee18770 0x563eeee22830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.985+0000 7f19f5400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee18770 0x563eeee22830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:35556/0 (socket says 172.21.15.45:35556) 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.985+0000 7f19f5e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563eeee17d80 0x563eeee222f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.985+0000 7f19f5400640 1 -- 172.21.15.45:0/2775732922 learned_addr learned my addr 172.21.15.45:0/2775732922 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.985+0000 7f19f6800640 1 --2- 172.21.15.45:0/2775732922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563eeee1d8e0 0x563eeee29870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.985+0000 7f19f5400640 1 -- 172.21.15.45:0/2775732922 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563eeee17d80 msgr2=0x563eeee222f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.985+0000 7f19f5400640 1 --2- 172.21.15.45:0/2775732922 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563eeee17d80 0x563eeee222f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.985+0000 7f19f5400640 1 -- 172.21.15.45:0/2775732922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563eeee1d8e0 msgr2=0x563eeee29870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.985+0000 7f19f5400640 1 --2- 172.21.15.45:0/2775732922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563eeee1d8e0 0x563eeee29870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.985+0000 7f19f5400640 1 -- 172.21.15.45:0/2775732922 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563eeea912d0 con 0x563eeee18770 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.986+0000 7f19f5400640 1 --2- 172.21.15.45:0/2775732922 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee18770 0x563eeee22830 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f19e0006010 tx=0x7f19e00b90b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba83b9bad2212d90 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.986+0000 7f19db400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.986+0000 7f19db400640 1 -- 172.21.15.45:0/2775732922 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e00fa040 con 0x563eeee18770 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.986+0000 7f19db400640 1 -- 172.21.15.45:0/2775732922 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f19e00ee6b0 con 0x563eeee18770 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.986+0000 7f19ff97e740 1 -- 172.21.15.45:0/2775732922 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f19f000fc40 con 0x563eeee18770 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.986+0000 7f19db400640 1 -- 172.21.15.45:0/2775732922 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f19e00bd210 con 0x563eeee18770 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.986+0000 7f19ff97e740 10 client.25007 fetch_fsmap learned FSMap version 22 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.986+0000 7f19ff97e740 1 -- 172.21.15.45:0/2775732922 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x563eeeb5b6b0 con 0x563eeee18770 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.986+0000 7f19db400640 1 -- 172.21.15.45:0/2775732922 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19f000fc40 con 0x563eeee18770 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.987+0000 7f19db400640 1 -- 172.21.15.45:0/2775732922 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f19e00c1100 con 0x563eeee18770 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.987+0000 7f19ff97e740 10 client.25007 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.987+0000 7f19ff97e740 10 client.25007 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T17:47:57.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.987+0000 7f19ff97e740 20 client.25007 populate_metadata read hostname 'smithi045' 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.987+0000 7f19ff97e740 1 --2- 172.21.15.45:0/2775732922 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x563eeee2e9f0 0x563eeee2d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.987+0000 7f19ff97e740 4 client.25007 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.987+0000 7f19ff97e740 1 -- 172.21.15.45:0/2775732922 --> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563eeed1e520 con 0x563eeee2e9f0 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.987+0000 7f19f5e00640 1 --2- 172.21.15.45:0/2775732922 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x563eeee2e9f0 0x563eeee2d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.987+0000 7f19f5e00640 1 --2- 172.21.15.45:0/2775732922 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x563eeee2e9f0 0x563eeee2d9d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=754baa43a0018292 server_cookie=c1aa21bc8a748d21 in_seq=0 out_seq=0 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.987+0000 7f19db400640 10 client.25007 ms_handle_connect on v2:172.21.15.71:6836/2008752655 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.988+0000 7f19db400640 1 -- 172.21.15.45:0/2775732922 <== mds.1 v2:172.21.15.71:6836/2008752655 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563eeed1e520 con 0x563eeee2e9f0 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.988+0000 7f19db400640 10 client.25007 handle_command_reply: tid=0 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.988+0000 7f19ff97e740 1 client.25007 shutdown 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.989+0000 7f19ff97e740 1 -- 172.21.15.45:0/2775732922 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee18770 msgr2=0x563eeee22830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.989+0000 7f19ff97e740 1 --2- 172.21.15.45:0/2775732922 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee18770 0x563eeee22830 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f19e0006010 tx=0x7f19e00b90b0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.989+0000 7f19ff97e740 1 -- 172.21.15.45:0/2775732922 shutdown_connections 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.989+0000 7f19ff97e740 1 --2- 172.21.15.45:0/2775732922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563eeee1d8e0 0x563eeee29870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.989+0000 7f19ff97e740 1 --2- 172.21.15.45:0/2775732922 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee18770 0x563eeee22830 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.989+0000 7f19ff97e740 1 --2- 172.21.15.45:0/2775732922 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x563eeee2e9f0 0x563eeee2d9d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.989+0000 7f19ff97e740 1 --2- 172.21.15.45:0/2775732922 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563eeee17d80 0x563eeee222f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:57.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.989+0000 7f19ff97e740 1 -- 172.21.15.45:0/2775732922 >> 172.21.15.45:0/2775732922 conn(0x563eeee132f0 msgr2=0x563eeee124f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:57.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.989+0000 7f19db400640 0 client.25007 ms_handle_reset on v2:172.21.15.71:6836/2008752655 2024-03-28T17:47:57.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.989+0000 7f19ff97e740 1 -- 172.21.15.45:0/2775732922 shutdown_connections 2024-03-28T17:47:57.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.989+0000 7f19ff97e740 1 -- 172.21.15.45:0/2775732922 wait complete. 2024-03-28T17:47:57.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:57.989+0000 7f19ff97e740 20 client.25007 trim_cache size 0 max 16384 2024-03-28T17:47:58.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.007+0000 7f19ff97e740 1 Processor -- start 2024-03-28T17:47:58.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.007+0000 7f19ff97e740 1 -- start start 2024-03-28T17:47:58.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.008+0000 7f19ff97e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee17d80 0x563eeee16c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.008+0000 7f19ff97e740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563eeee18770 0x563eeee36e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.008+0000 7f19ff97e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563eeee1d8e0 0x563eeee373d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.008+0000 7f19ff97e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563eeee2e210 con 0x563eeee1d8e0 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.008+0000 7f19ff97e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x563eeee2e870 con 0x563eeee17d80 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.008+0000 7f19ff97e740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x563eeee2eed0 con 0x563eeee18770 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.008+0000 7f19f5e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee17d80 0x563eeee16c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.008+0000 7f19f5400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563eeee18770 0x563eeee36e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.008+0000 7f19f5e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee17d80 0x563eeee16c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:35568/0 (socket says 172.21.15.45:35568) 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.008+0000 7f19f5e00640 1 -- 172.21.15.45:0/1167184185 learned_addr learned my addr 172.21.15.45:0/1167184185 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.008+0000 7f19f6800640 1 --2- 172.21.15.45:0/1167184185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563eeee1d8e0 0x563eeee373d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.008+0000 7f19f5e00640 1 -- 172.21.15.45:0/1167184185 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563eeee18770 msgr2=0x563eeee36e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.008+0000 7f19f5e00640 1 --2- 172.21.15.45:0/1167184185 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563eeee18770 0x563eeee36e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.008+0000 7f19f5e00640 1 -- 172.21.15.45:0/1167184185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563eeee1d8e0 msgr2=0x563eeee373d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.008+0000 7f19f5e00640 1 --2- 172.21.15.45:0/1167184185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563eeee1d8e0 0x563eeee373d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.008+0000 7f19f5e00640 1 -- 172.21.15.45:0/1167184185 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563eeee05170 con 0x563eeee17d80 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.008+0000 7f19f5400640 1 --2- 172.21.15.45:0/1167184185 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563eeee18770 0x563eeee36e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.008+0000 7f19f6800640 1 --2- 172.21.15.45:0/1167184185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563eeee1d8e0 0x563eeee373d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.009+0000 7f19f5e00640 1 --2- 172.21.15.45:0/1167184185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee17d80 0x563eeee16c60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f19e8012010 tx=0x7f19e8009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=516f77f5149c65f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.009+0000 7f19e6600640 1 -- 172.21.15.45:0/1167184185 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e800fd90 con 0x563eeee17d80 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.009+0000 7f19e6600640 1 -- 172.21.15.45:0/1167184185 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19e8003e50 con 0x563eeee17d80 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.009+0000 7f19e6600640 1 -- 172.21.15.45:0/1167184185 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e8004280 con 0x563eeee17d80 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.009+0000 7f19ff97e740 1 -- 172.21.15.45:0/1167184185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee17d80 msgr2=0x563eeee16c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.009+0000 7f19ff97e740 1 --2- 172.21.15.45:0/1167184185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee17d80 0x563eeee16c60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f19e8012010 tx=0x7f19e8009f90 comp rx=0 tx=0).stop 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.010+0000 7f19ff97e740 1 -- 172.21.15.45:0/1167184185 shutdown_connections 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.010+0000 7f19ff97e740 1 --2- 172.21.15.45:0/1167184185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563eeee1d8e0 0x563eeee373d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.010+0000 7f19ff97e740 1 --2- 172.21.15.45:0/1167184185 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563eeee18770 0x563eeee36e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.010+0000 7f19ff97e740 1 --2- 172.21.15.45:0/1167184185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee17d80 0x563eeee16c60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.010+0000 7f19ff97e740 1 -- 172.21.15.45:0/1167184185 >> 172.21.15.45:0/1167184185 conn(0x563eeee132f0 msgr2=0x563eeee16550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.010+0000 7f19ff97e740 1 -- 172.21.15.45:0/1167184185 shutdown_connections 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.010+0000 7f19ff97e740 1 -- 172.21.15.45:0/1167184185 wait complete. 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.010+0000 7f19ff97e740 1 -- start start 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.010+0000 7f19ff97e740 1 Processor -- start 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.011+0000 7f19ff97e740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563eeee17d80 0x563eeee28ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.011+0000 7f19ff97e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee18770 0x563eeee29000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.011+0000 7f19ff97e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563eeee1d8e0 0x563eeee271a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.011+0000 7f19ff97e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563eeee2fb90 con 0x563eeee1d8e0 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.011+0000 7f19ff97e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x563eeee2f530 con 0x563eeee18770 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.011+0000 7f19ff97e740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x563eeee301f0 con 0x563eeee17d80 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.011+0000 7f19ff97e740 10 client.0 _ll_register_callbacks cb 0x7f19e0005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.011+0000 7f19f5e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563eeee17d80 0x563eeee28ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.011+0000 7f19f5e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563eeee17d80 0x563eeee28ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:47800/0 (socket says 172.21.15.45:47800) 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.011+0000 7f19f5e00640 1 -- 172.21.15.45:0/1410170110 learned_addr learned my addr 172.21.15.45:0/1410170110 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.011+0000 7f19f5400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee18770 0x563eeee29000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.011+0000 7f19f6800640 1 --2- 172.21.15.45:0/1410170110 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563eeee1d8e0 0x563eeee271a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.011+0000 7f19f5400640 1 -- 172.21.15.45:0/1410170110 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563eeee17d80 msgr2=0x563eeee28ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.011+0000 7f19f5400640 1 --2- 172.21.15.45:0/1410170110 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563eeee17d80 0x563eeee28ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.011+0000 7f19f5400640 1 -- 172.21.15.45:0/1410170110 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563eeee1d8e0 msgr2=0x563eeee271a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.011+0000 7f19f5400640 1 --2- 172.21.15.45:0/1410170110 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563eeee1d8e0 0x563eeee271a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.011+0000 7f19f5400640 1 -- 172.21.15.45:0/1410170110 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563eeee279a0 con 0x563eeee18770 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.011+0000 7f19f5e00640 1 --2- 172.21.15.45:0/1410170110 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563eeee17d80 0x563eeee28ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.012+0000 7f19f5400640 1 --2- 172.21.15.45:0/1410170110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee18770 0x563eeee29000 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f19e00b9e10 tx=0x7f19e0005690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9377126f0edc83e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.012+0000 7f19db400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.012+0000 7f19db400640 1 -- 172.21.15.45:0/1410170110 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e00bcda0 con 0x563eeee18770 2024-03-28T17:47:58.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.012+0000 7f19db400640 1 -- 172.21.15.45:0/1410170110 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f19e00ee620 con 0x563eeee18770 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.012+0000 7f19ff97e740 1 -- 172.21.15.45:0/1410170110 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f19e8003e50 con 0x563eeee18770 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.012+0000 7f19db400640 1 -- 172.21.15.45:0/1410170110 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f19e00bcf80 con 0x563eeee18770 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.012+0000 7f19ff97e740 10 client.25013 fetch_fsmap learned FSMap version 22 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.012+0000 7f19ff97e740 1 -- 172.21.15.45:0/1410170110 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x563eeee8a680 con 0x563eeee18770 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.012+0000 7f19db400640 1 -- 172.21.15.45:0/1410170110 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19e8003e50 con 0x563eeee18770 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.012+0000 7f19db400640 1 -- 172.21.15.45:0/1410170110 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f19e00c5750 con 0x563eeee18770 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.013+0000 7f19ff97e740 10 client.25013 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.013+0000 7f19ff97e740 10 client.25013 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.013+0000 7f19ff97e740 20 client.25013 populate_metadata read hostname 'smithi045' 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.013+0000 7f19ff97e740 1 --2- 172.21.15.45:0/1410170110 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x563eeee8eb20 0x563eeee8a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.013+0000 7f19ff97e740 4 client.25013 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.013+0000 7f19ff97e740 1 -- 172.21.15.45:0/1410170110 --> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563eeed8a1d0 con 0x563eeee8eb20 2024-03-28T17:47:58.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.013+0000 7f19f5e00640 1 --2- 172.21.15.45:0/1410170110 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x563eeee8eb20 0x563eeee8a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.013+0000 7f19f5e00640 1 --2- 172.21.15.45:0/1410170110 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x563eeee8eb20 0x563eeee8a910 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=30dccd9f9d0cc406 server_cookie=3b8c9f172c0fc3ad in_seq=0 out_seq=0 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.013+0000 7f19db400640 10 client.25013 ms_handle_connect on v2:172.21.15.71:6836/2008752655 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.014+0000 7f19db400640 1 -- 172.21.15.45:0/1410170110 <== mds.1 v2:172.21.15.71:6836/2008752655 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x563eeed8a1d0 con 0x563eeee8eb20 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.014+0000 7f19db400640 10 client.25013 handle_command_reply: tid=0 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.014+0000 7f19ff97e740 1 client.25013 shutdown 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.014+0000 7f19ff97e740 1 -- 172.21.15.45:0/1410170110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee18770 msgr2=0x563eeee29000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.014+0000 7f19ff97e740 1 --2- 172.21.15.45:0/1410170110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee18770 0x563eeee29000 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f19e00b9e10 tx=0x7f19e0005690 comp rx=0 tx=0).stop 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.014+0000 7f19ff97e740 1 -- 172.21.15.45:0/1410170110 shutdown_connections 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.014+0000 7f19ff97e740 1 --2- 172.21.15.45:0/1410170110 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563eeee1d8e0 0x563eeee271a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.014+0000 7f19ff97e740 1 --2- 172.21.15.45:0/1410170110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563eeee18770 0x563eeee29000 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.014+0000 7f19ff97e740 1 --2- 172.21.15.45:0/1410170110 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x563eeee8eb20 0x563eeee8a910 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.014+0000 7f19ff97e740 1 --2- 172.21.15.45:0/1410170110 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563eeee17d80 0x563eeee28ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.014+0000 7f19ff97e740 1 -- 172.21.15.45:0/1410170110 >> 172.21.15.45:0/1410170110 conn(0x563eeee132f0 msgr2=0x563eeee16de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.014+0000 7f19db400640 0 client.25013 ms_handle_reset on v2:172.21.15.71:6836/2008752655 2024-03-28T17:47:58.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.014+0000 7f19ff97e740 1 -- 172.21.15.45:0/1410170110 shutdown_connections 2024-03-28T17:47:58.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.014+0000 7f19ff97e740 1 -- 172.21.15.45:0/1410170110 wait complete. 2024-03-28T17:47:58.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.014+0000 7f19ff97e740 20 client.25013 trim_cache size 0 max 16384 2024-03-28T17:47:58.021 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:47:58.021 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "9f4888ec-ed29-11ee-95d0-87774f69a715", 2024-03-28T17:47:58.021 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 1, 2024-03-28T17:47:58.021 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 24530, 2024-03-28T17:47:58.021 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-03-28T17:47:58.022 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-03-28T17:47:58.022 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-03-28T17:47:58.022 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 102.085565262, 2024-03-28T17:47:58.022 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 22, 2024-03-28T17:47:58.022 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 87, 2024-03-28T17:47:58.022 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 85, 2024-03-28T17:47:58.022 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 126.408434478 2024-03-28T17:47:58.022 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.015+0000 7f19f7200640 1 -- 172.21.15.45:0/3854754483 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f19dc0a2800 msgr2=0x7f19dc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.015+0000 7f19f7200640 1 --2- 172.21.15.45:0/3854754483 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f19dc0a2800 0x7f19dc0a4cc0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x563eeee18b70 tx=0x7f19e8009500 comp rx=0 tx=0).stop 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.015+0000 7f19f7200640 1 -- 172.21.15.45:0/3854754483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19f8108db0 msgr2=0x7f19f819e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.015+0000 7f19f7200640 1 --2- 172.21.15.45:0/3854754483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19f8108db0 0x7f19f819e3f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f19e0000c00 tx=0x7f19e002c650 comp rx=0 tx=0).stop 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.015+0000 7f19f7200640 1 -- 172.21.15.45:0/3854754483 shutdown_connections 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.015+0000 7f19f7200640 1 --2- 172.21.15.45:0/3854754483 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f19dc0a2800 0x7f19dc0a4cc0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.015+0000 7f19f7200640 1 --2- 172.21.15.45:0/3854754483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19f8109770 0x7f19f81a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.015+0000 7f19f7200640 1 --2- 172.21.15.45:0/3854754483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19f8108db0 0x7f19f819e3f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.015+0000 7f19f7200640 1 --2- 172.21.15.45:0/3854754483 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f19f8107bc0 0x7f19f819deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.015+0000 7f19f7200640 1 -- 172.21.15.45:0/3854754483 >> 172.21.15.45:0/3854754483 conn(0x7f19f8076b70 msgr2=0x7f19f810c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:58.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.015+0000 7f19f7200640 1 -- 172.21.15.45:0/3854754483 shutdown_connections 2024-03-28T17:47:58.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.015+0000 7f19f7200640 1 -- 172.21.15.45:0/3854754483 wait complete. 2024-03-28T17:47:58.032 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T17:47:58.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.241+0000 7f72e7200640 1 -- 172.21.15.45:0/2310591144 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72e8107bc0 msgr2=0x7f72e8107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:58.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.241+0000 7f72e7200640 1 --2- 172.21.15.45:0/2310591144 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72e8107bc0 0x7f72e8107fe0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009900 tx=0x7f72d802bc30 comp rx=0 tx=0).stop 2024-03-28T17:47:58.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.241+0000 7f72e7200640 1 -- 172.21.15.45:0/2310591144 shutdown_connections 2024-03-28T17:47:58.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.241+0000 7f72e7200640 1 --2- 172.21.15.45:0/2310591144 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72e8109770 0x7f72e8110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.241+0000 7f72e7200640 1 --2- 172.21.15.45:0/2310591144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72e8108db0 0x7f72e8109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.241+0000 7f72e7200640 1 --2- 172.21.15.45:0/2310591144 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72e8107bc0 0x7f72e8107fe0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.241+0000 7f72e7200640 1 -- 172.21.15.45:0/2310591144 >> 172.21.15.45:0/2310591144 conn(0x7f72e8076b70 msgr2=0x7f72e8078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:58.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.241+0000 7f72e7200640 1 -- 172.21.15.45:0/2310591144 shutdown_connections 2024-03-28T17:47:58.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.241+0000 7f72e7200640 1 -- 172.21.15.45:0/2310591144 wait complete. 2024-03-28T17:47:58.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.242+0000 7f72e7200640 1 Processor -- start 2024-03-28T17:47:58.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.242+0000 7f72e7200640 1 -- start start 2024-03-28T17:47:58.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.243+0000 7f72e7200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72e8107bc0 0x7f72e819df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.243+0000 7f72e7200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72e8108db0 0x7f72e819e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.243+0000 7f72e7200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72e8109770 0x7f72e81a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.243+0000 7f72e7200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e8113860 con 0x7f72e8108db0 2024-03-28T17:47:58.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.243+0000 7f72e7200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e81136e0 con 0x7f72e8107bc0 2024-03-28T17:47:58.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.243+0000 7f72e7200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e81139e0 con 0x7f72e8109770 2024-03-28T17:47:58.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.243+0000 7f72e5e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72e8107bc0 0x7f72e819df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.243+0000 7f72e5e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72e8107bc0 0x7f72e819df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:35592/0 (socket says 172.21.15.45:35592) 2024-03-28T17:47:58.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.243+0000 7f72dfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72e8108db0 0x7f72e819e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.243+0000 7f72e5e00640 1 -- 172.21.15.45:0/3969694437 learned_addr learned my addr 172.21.15.45:0/3969694437 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:58.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.243+0000 7f72e6800640 1 --2- 172.21.15.45:0/3969694437 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72e8109770 0x7f72e81a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.243+0000 7f72e5e00640 1 -- 172.21.15.45:0/3969694437 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72e8109770 msgr2=0x7f72e81a5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:58.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.243+0000 7f72e5e00640 1 --2- 172.21.15.45:0/3969694437 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72e8109770 0x7f72e81a5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.243+0000 7f72e5e00640 1 -- 172.21.15.45:0/3969694437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72e8108db0 msgr2=0x7f72e819e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:58.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.243+0000 7f72e5e00640 1 --2- 172.21.15.45:0/3969694437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72e8108db0 0x7f72e819e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.243+0000 7f72e5e00640 1 -- 172.21.15.45:0/3969694437 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e81a5ab0 con 0x7f72e8107bc0 2024-03-28T17:47:58.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.243+0000 7f72e6800640 1 --2- 172.21.15.45:0/3969694437 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72e8109770 0x7f72e81a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:47:58.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.244+0000 7f72e5e00640 1 --2- 172.21.15.45:0/3969694437 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72e8107bc0 0x7f72e819df30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f72d8000c00 tx=0x7f72d802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:58.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.244+0000 7f72dea00640 1 -- 172.21.15.45:0/3969694437 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d8043020 con 0x7f72e8107bc0 2024-03-28T17:47:58.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.244+0000 7f72dea00640 1 -- 172.21.15.45:0/3969694437 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72d80054e0 con 0x7f72e8107bc0 2024-03-28T17:47:58.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.244+0000 7f72dea00640 1 -- 172.21.15.45:0/3969694437 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d8003c40 con 0x7f72e8107bc0 2024-03-28T17:47:58.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.244+0000 7f72e7200640 1 -- 172.21.15.45:0/3969694437 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e81a5d40 con 0x7f72e8107bc0 2024-03-28T17:47:58.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.244+0000 7f72e7200640 1 -- 172.21.15.45:0/3969694437 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e81a6180 con 0x7f72e8107bc0 2024-03-28T17:47:58.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.247+0000 7f72efb76740 1 Processor -- start 2024-03-28T17:47:58.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.247+0000 7f72efb76740 1 -- start start 2024-03-28T17:47:58.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.248+0000 7f72dea00640 1 -- 172.21.15.45:0/3969694437 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f72d8046020 con 0x7f72e8107bc0 2024-03-28T17:47:58.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.248+0000 7f72efb76740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5625421d7d80 0x5625421d81a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.248+0000 7f72efb76740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5625421d8770 0x5625421dd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.248+0000 7f72efb76740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421dd8e0 0x5625421dfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.248+0000 7f72efb76740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56254213cc90 con 0x5625421d8770 2024-03-28T17:47:58.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.248+0000 7f72efb76740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625420e5980 con 0x5625421d7d80 2024-03-28T17:47:58.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.248+0000 7f72efb76740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x562542138d30 con 0x5625421dd8e0 2024-03-28T17:47:58.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.248+0000 7f72e5e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5625421d8770 0x5625421dd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.248+0000 7f72dfe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421dd8e0 0x5625421dfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.248+0000 7f72e6800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5625421d7d80 0x5625421d81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.248+0000 7f72e6800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5625421d7d80 0x5625421d81a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:35600/0 (socket says 172.21.15.45:35600) 2024-03-28T17:47:58.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.248+0000 7f72e6800640 1 -- 172.21.15.45:0/2402226129 learned_addr learned my addr 172.21.15.45:0/2402226129 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:58.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.248+0000 7f72dfe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421dd8e0 0x5625421dfd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:47828/0 (socket says 172.21.15.45:47828) 2024-03-28T17:47:58.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.249+0000 7f72dfe00640 1 -- 172.21.15.45:0/2402226129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5625421d7d80 msgr2=0x5625421d81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:58.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.249+0000 7f72dfe00640 1 --2- 172.21.15.45:0/2402226129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5625421d7d80 0x5625421d81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.249+0000 7f72dfe00640 1 -- 172.21.15.45:0/2402226129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5625421d8770 msgr2=0x5625421dd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:58.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.249+0000 7f72dfe00640 1 --2- 172.21.15.45:0/2402226129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5625421d8770 0x5625421dd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.249+0000 7f72dfe00640 1 -- 172.21.15.45:0/2402226129 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5625420fd510 con 0x5625421dd8e0 2024-03-28T17:47:58.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.249+0000 7f72e6800640 1 --2- 172.21.15.45:0/2402226129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5625421d7d80 0x5625421d81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:47:58.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.249+0000 7f72dfe00640 1 --2- 172.21.15.45:0/2402226129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421dd8e0 0x5625421dfd70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009bf0 tx=0x7f72d0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ce4cd426f2cbc03 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:58.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.249+0000 7f72d7400640 1 -- 172.21.15.45:0/2402226129 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d001e070 con 0x5625421dd8e0 2024-03-28T17:47:58.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.249+0000 7f72d7400640 1 -- 172.21.15.45:0/2402226129 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72d0009da0 con 0x5625421dd8e0 2024-03-28T17:47:58.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.249+0000 7f72d7400640 1 -- 172.21.15.45:0/2402226129 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d0018830 con 0x5625421dd8e0 2024-03-28T17:47:58.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.249+0000 7f72dea00640 1 --2- 172.21.15.45:0/3969694437 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f72cc0a2880 0x7f72cc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.250+0000 7f72dea00640 1 -- 172.21.15.45:0/3969694437 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f72d80f09b0 con 0x7f72e8107bc0 2024-03-28T17:47:58.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.250+0000 7f72e6800640 1 --2- 172.21.15.45:0/3969694437 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f72cc0a2880 0x7f72cc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.250+0000 7f72efb76740 1 -- 172.21.15.45:0/2402226129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421dd8e0 msgr2=0x5625421dfd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:58.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.250+0000 7f72efb76740 1 --2- 172.21.15.45:0/2402226129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421dd8e0 0x5625421dfd70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009bf0 tx=0x7f72d0011810 comp rx=0 tx=0).stop 2024-03-28T17:47:58.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.250+0000 7f72efb76740 1 -- 172.21.15.45:0/2402226129 shutdown_connections 2024-03-28T17:47:58.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.250+0000 7f72efb76740 1 --2- 172.21.15.45:0/2402226129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421dd8e0 0x5625421dfd70 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.250+0000 7f72efb76740 1 --2- 172.21.15.45:0/2402226129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5625421d8770 0x5625421dd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.250+0000 7f72efb76740 1 --2- 172.21.15.45:0/2402226129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5625421d7d80 0x5625421d81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.250+0000 7f72efb76740 1 -- 172.21.15.45:0/2402226129 >> 172.21.15.45:0/2402226129 conn(0x5625421d32f0 msgr2=0x5625421d36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:58.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.250+0000 7f72efb76740 1 -- 172.21.15.45:0/2402226129 shutdown_connections 2024-03-28T17:47:58.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.250+0000 7f72efb76740 1 -- 172.21.15.45:0/2402226129 wait complete. 2024-03-28T17:47:58.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.250+0000 7f72e6800640 1 --2- 172.21.15.45:0/3969694437 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f72cc0a2880 0x7f72cc0a4d40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x5625421d8fd0 tx=0x7f72e0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:58.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.251+0000 7f72efb76740 1 -- start start 2024-03-28T17:47:58.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.251+0000 7f72efb76740 1 Processor -- start 2024-03-28T17:47:58.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.251+0000 7f72efb76740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421d7d80 0x5625421e2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.251+0000 7f72efb76740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5625421d8770 0x5625421e2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.251+0000 7f72efb76740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5625421dd8e0 0x5625421e97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.251+0000 7f72efb76740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625421e0890 con 0x5625421d8770 2024-03-28T17:47:58.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.251+0000 7f72efb76740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625420f8dd0 con 0x5625421dd8e0 2024-03-28T17:47:58.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.251+0000 7f72efb76740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625421e0a10 con 0x5625421d7d80 2024-03-28T17:47:58.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.252+0000 7f72efb76740 10 client.0 _ll_register_callbacks cb 0x562541e50ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:47:58.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.252+0000 7f72e5e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421d7d80 0x5625421e2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.252+0000 7f72e5e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421d7d80 0x5625421e2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:47840/0 (socket says 172.21.15.45:47840) 2024-03-28T17:47:58.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.252+0000 7f72e5e00640 1 -- 172.21.15.45:0/1930878146 learned_addr learned my addr 172.21.15.45:0/1930878146 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:58.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.252+0000 7f72e6800640 1 --2- 172.21.15.45:0/1930878146 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5625421dd8e0 0x5625421e97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.252+0000 7f72dfe00640 1 --2- 172.21.15.45:0/1930878146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5625421d8770 0x5625421e2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.252+0000 7f72e5e00640 1 -- 172.21.15.45:0/1930878146 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5625421dd8e0 msgr2=0x5625421e97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:58.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.252+0000 7f72e5e00640 1 --2- 172.21.15.45:0/1930878146 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5625421dd8e0 0x5625421e97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.252+0000 7f72e5e00640 1 -- 172.21.15.45:0/1930878146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5625421d8770 msgr2=0x5625421e2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:58.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.252+0000 7f72e5e00640 1 --2- 172.21.15.45:0/1930878146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5625421d8770 0x5625421e2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.252+0000 7f72e5e00640 1 -- 172.21.15.45:0/1930878146 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562541e512d0 con 0x5625421d7d80 2024-03-28T17:47:58.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.252+0000 7f72e5e00640 1 --2- 172.21.15.45:0/1930878146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421d7d80 0x5625421e2240 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f72d8006010 tx=0x7f72d80b9600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12620853fa99b8ca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:58.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.252+0000 7f72cbe00640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:47:58.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.252+0000 7f72cbe00640 1 -- 172.21.15.45:0/1930878146 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d80f0e60 con 0x5625421d7d80 2024-03-28T17:47:58.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.253+0000 7f72cbe00640 1 -- 172.21.15.45:0/1930878146 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f72d80bb400 con 0x5625421d7d80 2024-03-28T17:47:58.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.253+0000 7f72efb76740 1 -- 172.21.15.45:0/1930878146 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f72d0009da0 con 0x5625421d7d80 2024-03-28T17:47:58.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.253+0000 7f72cbe00640 1 -- 172.21.15.45:0/1930878146 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72d80dbe60 con 0x5625421d7d80 2024-03-28T17:47:58.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.253+0000 7f72efb76740 10 client.25000 fetch_fsmap learned FSMap version 22 2024-03-28T17:47:58.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.253+0000 7f72efb76740 1 -- 172.21.15.45:0/1930878146 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x562541f1b6b0 con 0x5625421d7d80 2024-03-28T17:47:58.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.253+0000 7f72cbe00640 1 -- 172.21.15.45:0/1930878146 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72d0009da0 con 0x5625421d7d80 2024-03-28T17:47:58.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.253+0000 7f72cbe00640 1 -- 172.21.15.45:0/1930878146 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f72d80bd5c0 con 0x5625421d7d80 2024-03-28T17:47:58.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.253+0000 7f72efb76740 10 client.25000 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T17:47:58.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.253+0000 7f72efb76740 10 client.25000 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T17:47:58.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.253+0000 7f72efb76740 20 client.25000 populate_metadata read hostname 'smithi045' 2024-03-28T17:47:58.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.253+0000 7f72efb76740 1 --2- 172.21.15.45:0/1930878146 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5625421eea90 0x5625421ed920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.253+0000 7f72efb76740 4 client.25000 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:47:58.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.253+0000 7f72efb76740 1 -- 172.21.15.45:0/1930878146 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5625420de520 con 0x5625421eea90 2024-03-28T17:47:58.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.254+0000 7f72dfe00640 1 --2- 172.21.15.45:0/1930878146 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5625421eea90 0x5625421ed920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.254+0000 7f72dfe00640 1 --2- 172.21.15.45:0/1930878146 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5625421eea90 0x5625421ed920 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9371035673629d49 server_cookie=c25e3e34e9163089 in_seq=0 out_seq=0 2024-03-28T17:47:58.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.254+0000 7f72cbe00640 10 client.25000 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:47:58.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.255+0000 7f72cbe00640 1 -- 172.21.15.45:0/1930878146 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5625420de520 con 0x5625421eea90 2024-03-28T17:47:58.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.255+0000 7f72cbe00640 10 client.25000 handle_command_reply: tid=0 2024-03-28T17:47:58.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.255+0000 7f72efb76740 1 client.25000 shutdown 2024-03-28T17:47:58.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.255+0000 7f72efb76740 1 -- 172.21.15.45:0/1930878146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421d7d80 msgr2=0x5625421e2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:58.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.255+0000 7f72efb76740 1 --2- 172.21.15.45:0/1930878146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421d7d80 0x5625421e2240 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f72d8006010 tx=0x7f72d80b9600 comp rx=0 tx=0).stop 2024-03-28T17:47:58.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.255+0000 7f72efb76740 1 -- 172.21.15.45:0/1930878146 shutdown_connections 2024-03-28T17:47:58.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.255+0000 7f72efb76740 1 --2- 172.21.15.45:0/1930878146 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5625421eea90 0x5625421ed920 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.255+0000 7f72efb76740 1 --2- 172.21.15.45:0/1930878146 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5625421dd8e0 0x5625421e97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.255+0000 7f72efb76740 1 --2- 172.21.15.45:0/1930878146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5625421d8770 0x5625421e2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.255+0000 7f72efb76740 1 --2- 172.21.15.45:0/1930878146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421d7d80 0x5625421e2240 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.255+0000 7f72cbe00640 0 client.25000 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:47:58.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.255+0000 7f72efb76740 1 -- 172.21.15.45:0/1930878146 >> 172.21.15.45:0/1930878146 conn(0x5625421d32f0 msgr2=0x5625421d24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:58.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.256+0000 7f72efb76740 1 -- 172.21.15.45:0/1930878146 shutdown_connections 2024-03-28T17:47:58.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.256+0000 7f72efb76740 1 -- 172.21.15.45:0/1930878146 wait complete. 2024-03-28T17:47:58.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.256+0000 7f72efb76740 20 client.25000 trim_cache size 0 max 16384 2024-03-28T17:47:58.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.274+0000 7f72efb76740 1 Processor -- start 2024-03-28T17:47:58.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.275+0000 7f72efb76740 1 -- start start 2024-03-28T17:47:58.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.275+0000 7f72efb76740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5625421d7d80 0x5625421d9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.275+0000 7f72efb76740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421d8770 0x5625421e9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.275+0000 7f72efb76740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5625421dd8e0 0x5625421da110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.275+0000 7f72efb76740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625421ee2b0 con 0x5625421d7d80 2024-03-28T17:47:58.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.275+0000 7f72efb76740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625421ee910 con 0x5625421dd8e0 2024-03-28T17:47:58.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.275+0000 7f72efb76740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625421eef70 con 0x5625421d8770 2024-03-28T17:47:58.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.275+0000 7f72e5e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5625421d7d80 0x5625421d9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.275+0000 7f72dfe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421d8770 0x5625421e9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.275+0000 7f72e6800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5625421dd8e0 0x5625421da110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.275+0000 7f72e5e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5625421d7d80 0x5625421d9bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44652/0 (socket says 172.21.15.45:44652) 2024-03-28T17:47:58.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.275+0000 7f72e5e00640 1 -- 172.21.15.45:0/105664567 learned_addr learned my addr 172.21.15.45:0/105664567 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:58.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.275+0000 7f72dfe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421d8770 0x5625421e9750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:47852/0 (socket says 172.21.15.45:47852) 2024-03-28T17:47:58.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.276+0000 7f72e5e00640 1 -- 172.21.15.45:0/105664567 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421d8770 msgr2=0x5625421e9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:58.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.276+0000 7f72e5e00640 1 --2- 172.21.15.45:0/105664567 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421d8770 0x5625421e9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.276+0000 7f72e5e00640 1 -- 172.21.15.45:0/105664567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5625421dd8e0 msgr2=0x5625421da110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:58.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.276+0000 7f72e5e00640 1 --2- 172.21.15.45:0/105664567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5625421dd8e0 0x5625421da110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.276+0000 7f72e5e00640 1 -- 172.21.15.45:0/105664567 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5625421c5170 con 0x5625421d7d80 2024-03-28T17:47:58.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.276+0000 7f72dfe00640 1 --2- 172.21.15.45:0/105664567 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421d8770 0x5625421e9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:47:58.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.276+0000 7f72e6800640 1 --2- 172.21.15.45:0/105664567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5625421dd8e0 0x5625421da110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:47:58.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.276+0000 7f72e5e00640 1 --2- 172.21.15.45:0/105664567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5625421d7d80 0x5625421d9bd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f72d80b9d30 tx=0x7f72d8005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12a07ec18341fd31 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:58.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.276+0000 7f72d6a00640 1 -- 172.21.15.45:0/105664567 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d80db070 con 0x5625421d7d80 2024-03-28T17:47:58.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.276+0000 7f72d6a00640 1 -- 172.21.15.45:0/105664567 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72d8005690 con 0x5625421d7d80 2024-03-28T17:47:58.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.276+0000 7f72d6a00640 1 -- 172.21.15.45:0/105664567 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d80c14b0 con 0x5625421d7d80 2024-03-28T17:47:58.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.277+0000 7f72efb76740 1 -- 172.21.15.45:0/105664567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5625421d7d80 msgr2=0x5625421d9bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:58.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.277+0000 7f72efb76740 1 --2- 172.21.15.45:0/105664567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5625421d7d80 0x5625421d9bd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f72d80b9d30 tx=0x7f72d8005e50 comp rx=0 tx=0).stop 2024-03-28T17:47:58.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.277+0000 7f72efb76740 1 -- 172.21.15.45:0/105664567 shutdown_connections 2024-03-28T17:47:58.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.277+0000 7f72efb76740 1 --2- 172.21.15.45:0/105664567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5625421dd8e0 0x5625421da110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.277+0000 7f72efb76740 1 --2- 172.21.15.45:0/105664567 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421d8770 0x5625421e9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.277+0000 7f72efb76740 1 --2- 172.21.15.45:0/105664567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5625421d7d80 0x5625421d9bd0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.277+0000 7f72efb76740 1 -- 172.21.15.45:0/105664567 >> 172.21.15.45:0/105664567 conn(0x5625421d32f0 msgr2=0x5625421d36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:58.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.277+0000 7f72efb76740 1 -- 172.21.15.45:0/105664567 shutdown_connections 2024-03-28T17:47:58.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.277+0000 7f72efb76740 1 -- 172.21.15.45:0/105664567 wait complete. 2024-03-28T17:47:58.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.277+0000 7f72efb76740 1 -- start start 2024-03-28T17:47:58.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.278+0000 7f72efb76740 1 Processor -- start 2024-03-28T17:47:58.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.278+0000 7f72efb76740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5625421d8770 0x5625421e6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.278+0000 7f72efb76740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421dd8e0 0x5625421e69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.278+0000 7f72efb76740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562542247ae0 0x562542249ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.278+0000 7f72efb76740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625421efc30 con 0x562542247ae0 2024-03-28T17:47:58.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.278+0000 7f72efb76740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625421ef5d0 con 0x5625421d8770 2024-03-28T17:47:58.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.278+0000 7f72efb76740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625421f0290 con 0x5625421dd8e0 2024-03-28T17:47:58.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.278+0000 7f72efb76740 10 client.0 _ll_register_callbacks cb 0x7f72d8009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:47:58.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.278+0000 7f72e6800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562542247ae0 0x562542249ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.278+0000 7f72dfe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421dd8e0 0x5625421e69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.278+0000 7f72dfe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421dd8e0 0x5625421e69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:47864/0 (socket says 172.21.15.45:47864) 2024-03-28T17:47:58.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.278+0000 7f72e6800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562542247ae0 0x562542249ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44666/0 (socket says 172.21.15.45:44666) 2024-03-28T17:47:58.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.278+0000 7f72dfe00640 1 -- 172.21.15.45:0/3610828478 learned_addr learned my addr 172.21.15.45:0/3610828478 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:47:58.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.278+0000 7f72e5e00640 1 --2- 172.21.15.45:0/3610828478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5625421d8770 0x5625421e6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.279+0000 7f72dfe00640 1 -- 172.21.15.45:0/3610828478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5625421d8770 msgr2=0x5625421e6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:58.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.279+0000 7f72dfe00640 1 --2- 172.21.15.45:0/3610828478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5625421d8770 0x5625421e6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.279+0000 7f72dfe00640 1 -- 172.21.15.45:0/3610828478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562542247ae0 msgr2=0x562542249ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:47:58.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.279+0000 7f72dfe00640 1 --2- 172.21.15.45:0/3610828478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562542247ae0 0x562542249ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.279+0000 7f72dfe00640 1 -- 172.21.15.45:0/3610828478 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5625421e6fe0 con 0x5625421dd8e0 2024-03-28T17:47:58.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.279+0000 7f72dfe00640 1 --2- 172.21.15.45:0/3610828478 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421dd8e0 0x5625421e69a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f72d00097e0 tx=0x7f72d0009490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4c773c731e902bc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:47:58.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.279+0000 7f72cbe00640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:47:58.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.279+0000 7f72cbe00640 1 -- 172.21.15.45:0/3610828478 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d001e040 con 0x5625421dd8e0 2024-03-28T17:47:58.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.279+0000 7f72efb76740 1 -- 172.21.15.45:0/3610828478 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56254214a1d0 con 0x5625421dd8e0 2024-03-28T17:47:58.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.279+0000 7f72cbe00640 1 -- 172.21.15.45:0/3610828478 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f72d002b240 con 0x5625421dd8e0 2024-03-28T17:47:58.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.279+0000 7f72cbe00640 1 -- 172.21.15.45:0/3610828478 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72d0009ce0 con 0x5625421dd8e0 2024-03-28T17:47:58.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.280+0000 7f72efb76740 10 client.25006 fetch_fsmap learned FSMap version 22 2024-03-28T17:47:58.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.280+0000 7f72efb76740 1 -- 172.21.15.45:0/3610828478 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x56254224b560 con 0x5625421dd8e0 2024-03-28T17:47:58.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.280+0000 7f72cbe00640 1 -- 172.21.15.45:0/3610828478 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56254214a1d0 con 0x5625421dd8e0 2024-03-28T17:47:58.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.280+0000 7f72cbe00640 1 -- 172.21.15.45:0/3610828478 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f72d00035f0 con 0x5625421dd8e0 2024-03-28T17:47:58.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.280+0000 7f72efb76740 10 client.25006 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T17:47:58.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.280+0000 7f72efb76740 10 client.25006 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T17:47:58.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.280+0000 7f72efb76740 20 client.25006 populate_metadata read hostname 'smithi045' 2024-03-28T17:47:58.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.280+0000 7f72efb76740 1 --2- 172.21.15.45:0/3610828478 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56254224f9f0 0x56254224b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:47:58.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.280+0000 7f72efb76740 4 client.25006 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T17:47:58.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.280+0000 7f72efb76740 1 -- 172.21.15.45:0/3610828478 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562542144520 con 0x56254224f9f0 2024-03-28T17:47:58.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.280+0000 7f72e5e00640 1 --2- 172.21.15.45:0/3610828478 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56254224f9f0 0x56254224b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:47:58.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.281+0000 7f72e5e00640 1 --2- 172.21.15.45:0/3610828478 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56254224f9f0 0x56254224b7f0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3be1c03bd4a10611 server_cookie=31a9789d93b62c44 in_seq=0 out_seq=0 2024-03-28T17:47:58.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.281+0000 7f72cbe00640 10 client.25006 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:47:58.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.281+0000 7f72cbe00640 1 -- 172.21.15.45:0/3610828478 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x562542144520 con 0x56254224f9f0 2024-03-28T17:47:58.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.281+0000 7f72cbe00640 10 client.25006 handle_command_reply: tid=0 2024-03-28T17:47:58.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.281+0000 7f72efb76740 1 client.25006 shutdown 2024-03-28T17:47:58.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.281+0000 7f72efb76740 1 -- 172.21.15.45:0/3610828478 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421dd8e0 msgr2=0x5625421e69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:58.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.281+0000 7f72efb76740 1 --2- 172.21.15.45:0/3610828478 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421dd8e0 0x5625421e69a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f72d00097e0 tx=0x7f72d0009490 comp rx=0 tx=0).stop 2024-03-28T17:47:58.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.281+0000 7f72efb76740 1 -- 172.21.15.45:0/3610828478 shutdown_connections 2024-03-28T17:47:58.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.281+0000 7f72efb76740 1 --2- 172.21.15.45:0/3610828478 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56254224f9f0 0x56254224b7f0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.281+0000 7f72efb76740 1 --2- 172.21.15.45:0/3610828478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562542247ae0 0x562542249ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.281+0000 7f72efb76740 1 --2- 172.21.15.45:0/3610828478 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5625421dd8e0 0x5625421e69a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.281+0000 7f72efb76740 1 --2- 172.21.15.45:0/3610828478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5625421d8770 0x5625421e6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.281+0000 7f72efb76740 1 -- 172.21.15.45:0/3610828478 >> 172.21.15.45:0/3610828478 conn(0x5625421d32f0 msgr2=0x5625421d9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:58.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.281+0000 7f72cbe00640 0 client.25006 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:47:58.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.281+0000 7f72efb76740 1 -- 172.21.15.45:0/3610828478 shutdown_connections 2024-03-28T17:47:58.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.281+0000 7f72efb76740 1 -- 172.21.15.45:0/3610828478 wait complete. 2024-03-28T17:47:58.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.281+0000 7f72efb76740 20 client.25006 trim_cache size 0 max 16384 2024-03-28T17:47:58.286 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:47:58.286 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "9f4888ec-ed29-11ee-95d0-87774f69a715", 2024-03-28T17:47:58.286 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2024-03-28T17:47:58.286 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 24584, 2024-03-28T17:47:58.286 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-03-28T17:47:58.286 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-03-28T17:47:58.286 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-03-28T17:47:58.286 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 105.037398874, 2024-03-28T17:47:58.287 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 22, 2024-03-28T17:47:58.287 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 87, 2024-03-28T17:47:58.287 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 85, 2024-03-28T17:47:58.287 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 112.48445546400001 2024-03-28T17:47:58.287 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:47:58.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.282+0000 7f72e7200640 1 -- 172.21.15.45:0/3969694437 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f72cc0a2880 msgr2=0x7f72cc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:58.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.282+0000 7f72e7200640 1 --2- 172.21.15.45:0/3969694437 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f72cc0a2880 0x7f72cc0a4d40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x5625421d8fd0 tx=0x7f72e0009500 comp rx=0 tx=0).stop 2024-03-28T17:47:58.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.282+0000 7f72e7200640 1 -- 172.21.15.45:0/3969694437 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72e8107bc0 msgr2=0x7f72e819df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:47:58.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.282+0000 7f72e7200640 1 --2- 172.21.15.45:0/3969694437 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72e8107bc0 0x7f72e819df30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f72d8000c00 tx=0x7f72d802c650 comp rx=0 tx=0).stop 2024-03-28T17:47:58.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.282+0000 7f72e7200640 1 -- 172.21.15.45:0/3969694437 shutdown_connections 2024-03-28T17:47:58.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.282+0000 7f72e7200640 1 --2- 172.21.15.45:0/3969694437 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f72cc0a2880 0x7f72cc0a4d40 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.282+0000 7f72e7200640 1 --2- 172.21.15.45:0/3969694437 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72e8109770 0x7f72e81a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.282+0000 7f72e7200640 1 --2- 172.21.15.45:0/3969694437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72e8108db0 0x7f72e819e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.282+0000 7f72e7200640 1 --2- 172.21.15.45:0/3969694437 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72e8107bc0 0x7f72e819df30 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:47:58.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.282+0000 7f72e7200640 1 -- 172.21.15.45:0/3969694437 >> 172.21.15.45:0/3969694437 conn(0x7f72e8076b70 msgr2=0x7f72e810c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:47:58.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.282+0000 7f72e7200640 1 -- 172.21.15.45:0/3969694437 shutdown_connections 2024-03-28T17:47:58.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:47:58.282+0000 7f72e7200640 1 -- 172.21.15.45:0/3969694437 wait complete. 2024-03-28T17:47:58.301 INFO:tasks.cephadm:Setup complete, yielding 2024-03-28T17:47:58.302 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-28T17:47:58.311 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi045.front.sepia.ceph.com 2024-03-28T17:47:58.311 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- bash -c 'ceph orch status' 2024-03-28T17:47:58.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:58 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/778208371' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:47:58.790 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:47:58.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:58 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/778208371' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:47:58.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:58 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/778208371' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:47:59.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:47:59 smithi045 ceph-mon[31773]: pgmap v372: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:47:59.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:47:59 smithi071 ceph-mon[36597]: pgmap v372: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:47:59.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:47:59 smithi151 ceph-mon[36481]: pgmap v372: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:01.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.234+0000 7f633b400700 1 -- 172.21.15.45:0/1392718659 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6334105fb0 msgr2=0x7f633410c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:01.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.234+0000 7f633b400700 1 --2- 172.21.15.45:0/1392718659 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6334105fb0 0x7f633410c860 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f632c00b0a0 tx=0x7f632c00b3b0 comp rx=0 tx=0).stop 2024-03-28T17:48:01.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.234+0000 7f633b400700 1 -- 172.21.15.45:0/1392718659 shutdown_connections 2024-03-28T17:48:01.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.234+0000 7f633b400700 1 --2- 172.21.15.45:0/1392718659 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6334105fb0 0x7f633410c860 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:01.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.234+0000 7f633b400700 1 --2- 172.21.15.45:0/1392718659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6334104410 0x7f6334104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:01.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.234+0000 7f633b400700 1 --2- 172.21.15.45:0/1392718659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6334105610 0x7f6334105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:01.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.234+0000 7f633b400700 1 -- 172.21.15.45:0/1392718659 >> 172.21.15.45:0/1392718659 conn(0x7f63340ff9b0 msgr2=0x7f6334101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:01.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.234+0000 7f633b400700 1 -- 172.21.15.45:0/1392718659 shutdown_connections 2024-03-28T17:48:01.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.234+0000 7f633b400700 1 -- 172.21.15.45:0/1392718659 wait complete. 2024-03-28T17:48:01.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.235+0000 7f633b400700 1 Processor -- start 2024-03-28T17:48:01.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.236+0000 7f633b400700 1 -- start start 2024-03-28T17:48:01.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.236+0000 7f633b400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6334104410 0x7f633419a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:01.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.236+0000 7f633b400700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6334105610 0x7f633419aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:01.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.236+0000 7f633b400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6334105fb0 0x7f63341a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:01.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.236+0000 7f633b400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6334110030 con 0x7f6334105fb0 2024-03-28T17:48:01.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.236+0000 7f633b400700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633410feb0 con 0x7f6334104410 2024-03-28T17:48:01.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.236+0000 7f633b400700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63341101b0 con 0x7f6334105610 2024-03-28T17:48:01.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.236+0000 7f6338a00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6334104410 0x7f633419a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:01.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.236+0000 7f6338a00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6334104410 0x7f633419a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:35640/0 (socket says 172.21.15.45:35640) 2024-03-28T17:48:01.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.236+0000 7f6338a00700 1 -- 172.21.15.45:0/2706434383 learned_addr learned my addr 172.21.15.45:0/2706434383 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:01.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.236+0000 7f6333e00700 1 --2- 172.21.15.45:0/2706434383 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6334105610 0x7f633419aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:01.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.236+0000 7f6339400700 1 --2- 172.21.15.45:0/2706434383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6334105fb0 0x7f63341a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:01.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.237+0000 7f6338a00700 1 -- 172.21.15.45:0/2706434383 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6334105610 msgr2=0x7f633419aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:01.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.237+0000 7f6338a00700 1 --2- 172.21.15.45:0/2706434383 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6334105610 0x7f633419aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:01.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.237+0000 7f6338a00700 1 -- 172.21.15.45:0/2706434383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6334105fb0 msgr2=0x7f63341a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:01.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.237+0000 7f6338a00700 1 --2- 172.21.15.45:0/2706434383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6334105fb0 0x7f63341a1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:01.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.237+0000 7f6338a00700 1 -- 172.21.15.45:0/2706434383 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63341a21e0 con 0x7f6334104410 2024-03-28T17:48:01.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.237+0000 7f6338a00700 1 --2- 172.21.15.45:0/2706434383 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6334104410 0x7f633419a580 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f632800e970 tx=0x7f632800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:01.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.237+0000 7f6331600700 1 -- 172.21.15.45:0/2706434383 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6328004220 con 0x7f6334104410 2024-03-28T17:48:01.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.237+0000 7f6331600700 1 -- 172.21.15.45:0/2706434383 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f63280043c0 con 0x7f6334104410 2024-03-28T17:48:01.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.237+0000 7f6331600700 1 -- 172.21.15.45:0/2706434383 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6328003df0 con 0x7f6334104410 2024-03-28T17:48:01.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.237+0000 7f633b400700 1 -- 172.21.15.45:0/2706434383 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63341a24d0 con 0x7f6334104410 2024-03-28T17:48:01.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.237+0000 7f633b400700 1 -- 172.21.15.45:0/2706434383 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63341a2a60 con 0x7f6334104410 2024-03-28T17:48:01.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.238+0000 7f633b400700 1 -- 172.21.15.45:0/2706434383 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6334068e50 con 0x7f6334104410 2024-03-28T17:48:01.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.240+0000 7f6331600700 1 -- 172.21.15.45:0/2706434383 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f632800c7e0 con 0x7f6334104410 2024-03-28T17:48:01.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.242+0000 7f6331600700 1 --2- 172.21.15.45:0/2706434383 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f631c0a2910 0x7f631c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:01.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.242+0000 7f6331600700 1 -- 172.21.15.45:0/2706434383 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f63280cebe0 con 0x7f6334104410 2024-03-28T17:48:01.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.242+0000 7f6333e00700 1 --2- 172.21.15.45:0/2706434383 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f631c0a2910 0x7f631c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:01.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.243+0000 7f6333e00700 1 --2- 172.21.15.45:0/2706434383 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f631c0a2910 0x7f631c0a4dd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6320009c60 tx=0x7f6320009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:01.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.244+0000 7f6331600700 1 -- 172.21.15.45:0/2706434383 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f632809bc30 con 0x7f6334104410 2024-03-28T17:48:01.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.491+0000 7f633b400700 1 -- 172.21.15.45:0/2706434383 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f63340625d0 con 0x7f631c0a2910 2024-03-28T17:48:01.492 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:01 smithi045 ceph-mon[31773]: pgmap v373: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T17:48:01.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.493+0000 7f6331600700 1 -- 172.21.15.45:0/2706434383 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f63340625d0 con 0x7f631c0a2910 2024-03-28T17:48:01.494 INFO:teuthology.orchestra.run.smithi045.stdout:Backend: cephadm 2024-03-28T17:48:01.494 INFO:teuthology.orchestra.run.smithi045.stdout:Available: Yes 2024-03-28T17:48:01.494 INFO:teuthology.orchestra.run.smithi045.stdout:Paused: No 2024-03-28T17:48:01.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.497+0000 7f633b400700 1 -- 172.21.15.45:0/2706434383 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f631c0a2910 msgr2=0x7f631c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:01.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.497+0000 7f633b400700 1 --2- 172.21.15.45:0/2706434383 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f631c0a2910 0x7f631c0a4dd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6320009c60 tx=0x7f6320009260 comp rx=0 tx=0).stop 2024-03-28T17:48:01.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.497+0000 7f633b400700 1 -- 172.21.15.45:0/2706434383 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6334104410 msgr2=0x7f633419a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:01.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.497+0000 7f633b400700 1 --2- 172.21.15.45:0/2706434383 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6334104410 0x7f633419a580 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f632800e970 tx=0x7f632800ec80 comp rx=0 tx=0).stop 2024-03-28T17:48:01.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.497+0000 7f633b400700 1 -- 172.21.15.45:0/2706434383 shutdown_connections 2024-03-28T17:48:01.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.497+0000 7f633b400700 1 --2- 172.21.15.45:0/2706434383 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f631c0a2910 0x7f631c0a4dd0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:01.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.497+0000 7f633b400700 1 --2- 172.21.15.45:0/2706434383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6334105fb0 0x7f63341a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:01.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.497+0000 7f633b400700 1 --2- 172.21.15.45:0/2706434383 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6334104410 0x7f633419a580 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:01.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.497+0000 7f633b400700 1 --2- 172.21.15.45:0/2706434383 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6334105610 0x7f633419aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:01.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.498+0000 7f633b400700 1 -- 172.21.15.45:0/2706434383 >> 172.21.15.45:0/2706434383 conn(0x7f63340ff9b0 msgr2=0x7f6334108840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:01.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.498+0000 7f633b400700 1 -- 172.21.15.45:0/2706434383 shutdown_connections 2024-03-28T17:48:01.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:01.498+0000 7f633b400700 1 -- 172.21.15.45:0/2706434383 wait complete. 2024-03-28T17:48:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:01 smithi071 ceph-mon[36597]: pgmap v373: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T17:48:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:01 smithi151 ceph-mon[36481]: pgmap v373: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T17:48:02.370 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-28T17:48:02.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:02 smithi045 ceph-mon[31773]: from='client.25037 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:48:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:02 smithi071 ceph-mon[36597]: from='client.25037 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:48:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:02 smithi151 ceph-mon[36481]: from='client.25037 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:48:02.828 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:48:03.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:03 smithi045 ceph-mon[31773]: pgmap v374: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:48:03.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:03 smithi071 ceph-mon[36597]: pgmap v374: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:48:03.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:03 smithi151 ceph-mon[36481]: pgmap v374: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:48:04.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.989+0000 7fb48be00700 1 -- 172.21.15.45:0/1435132546 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb48c1048d0 msgr2=0x7fb48c10c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:04.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.989+0000 7fb48be00700 1 --2- 172.21.15.45:0/1435132546 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb48c1048d0 0x7fb48c10c760 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb48400b0d0 tx=0x7fb48400b3e0 comp rx=0 tx=0).stop 2024-03-28T17:48:04.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.989+0000 7fb48be00700 1 -- 172.21.15.45:0/1435132546 shutdown_connections 2024-03-28T17:48:04.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.989+0000 7fb48be00700 1 --2- 172.21.15.45:0/1435132546 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb48c1048d0 0x7fb48c10c760 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:04.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.989+0000 7fb48be00700 1 --2- 172.21.15.45:0/1435132546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb48c1059f0 0x7fb48c105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:04.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.989+0000 7fb48be00700 1 --2- 172.21.15.45:0/1435132546 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb48c103f30 0x7fb48c104390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:04.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.989+0000 7fb48be00700 1 -- 172.21.15.45:0/1435132546 >> 172.21.15.45:0/1435132546 conn(0x7fb48c0ff8b0 msgr2=0x7fb48c101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:04.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.989+0000 7fb48be00700 1 -- 172.21.15.45:0/1435132546 shutdown_connections 2024-03-28T17:48:04.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.989+0000 7fb48be00700 1 -- 172.21.15.45:0/1435132546 wait complete. 2024-03-28T17:48:04.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.990+0000 7fb48be00700 1 Processor -- start 2024-03-28T17:48:04.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.991+0000 7fb48be00700 1 -- start start 2024-03-28T17:48:04.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.991+0000 7fb48be00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb48c103f30 0x7fb48c19a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:04.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.991+0000 7fb48be00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb48c1048d0 0x7fb48c19aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:04.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.991+0000 7fb48be00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb48c1059f0 0x7fb48c1a1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:04.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.991+0000 7fb48be00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c110030 con 0x7fb48c1048d0 2024-03-28T17:48:04.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.991+0000 7fb48be00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c10feb0 con 0x7fb48c103f30 2024-03-28T17:48:04.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.991+0000 7fb48be00700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c1101b0 con 0x7fb48c1059f0 2024-03-28T17:48:04.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.991+0000 7fb48aa00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb48c103f30 0x7fb48c19a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:04.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.991+0000 7fb48a000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb48c1048d0 0x7fb48c19aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:04.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.991+0000 7fb48aa00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb48c103f30 0x7fb48c19a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:35660/0 (socket says 172.21.15.45:35660) 2024-03-28T17:48:04.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.991+0000 7fb48aa00700 1 -- 172.21.15.45:0/2747225174 learned_addr learned my addr 172.21.15.45:0/2747225174 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:04.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.991+0000 7fb48b400700 1 --2- 172.21.15.45:0/2747225174 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb48c1059f0 0x7fb48c1a1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:04.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.992+0000 7fb48aa00700 1 -- 172.21.15.45:0/2747225174 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb48c1059f0 msgr2=0x7fb48c1a1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:04.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.992+0000 7fb48aa00700 1 --2- 172.21.15.45:0/2747225174 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb48c1059f0 0x7fb48c1a1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:04.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.992+0000 7fb48aa00700 1 -- 172.21.15.45:0/2747225174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb48c1048d0 msgr2=0x7fb48c19aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:04.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.992+0000 7fb48aa00700 1 --2- 172.21.15.45:0/2747225174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb48c1048d0 0x7fb48c19aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:04.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.992+0000 7fb48aa00700 1 -- 172.21.15.45:0/2747225174 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48c1a2120 con 0x7fb48c103f30 2024-03-28T17:48:04.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.992+0000 7fb48aa00700 1 --2- 172.21.15.45:0/2747225174 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb48c103f30 0x7fb48c19a4c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb47c00c990 tx=0x7fb47c00cca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:04.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.992+0000 7fb483400700 1 -- 172.21.15.45:0/2747225174 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb47c007940 con 0x7fb48c103f30 2024-03-28T17:48:04.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.992+0000 7fb483400700 1 -- 172.21.15.45:0/2747225174 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb47c00f450 con 0x7fb48c103f30 2024-03-28T17:48:04.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.992+0000 7fb48be00700 1 -- 172.21.15.45:0/2747225174 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb48c1a2410 con 0x7fb48c103f30 2024-03-28T17:48:04.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.992+0000 7fb483400700 1 -- 172.21.15.45:0/2747225174 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb47c018660 con 0x7fb48c103f30 2024-03-28T17:48:04.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.993+0000 7fb48be00700 1 -- 172.21.15.45:0/2747225174 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb48c1a29a0 con 0x7fb48c103f30 2024-03-28T17:48:04.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.994+0000 7fb48be00700 1 -- 172.21.15.45:0/2747225174 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb48c068e50 con 0x7fb48c103f30 2024-03-28T17:48:04.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.995+0000 7fb483400700 1 -- 172.21.15.45:0/2747225174 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb47c00f990 con 0x7fb48c103f30 2024-03-28T17:48:04.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.997+0000 7fb483400700 1 --2- 172.21.15.45:0/2747225174 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb4700a2910 0x7fb4700a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:04.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.997+0000 7fb483400700 1 -- 172.21.15.45:0/2747225174 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb47c0ce8c0 con 0x7fb48c103f30 2024-03-28T17:48:04.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.997+0000 7fb48a000700 1 --2- 172.21.15.45:0/2747225174 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb4700a2910 0x7fb4700a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:04.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:04.998+0000 7fb48a000700 1 --2- 172.21.15.45:0/2747225174 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb4700a2910 0x7fb4700a4dd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb478009c60 tx=0x7fb478009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:05.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:05.001+0000 7fb483400700 1 -- 172.21.15.45:0/2747225174 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb47c014030 con 0x7fb48c103f30 2024-03-28T17:48:05.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:05.243+0000 7fb48be00700 1 -- 172.21.15.45:0/2747225174 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb48c0625d0 con 0x7fb4700a2910 2024-03-28T17:48:05.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:05.261+0000 7fb483400700 1 -- 172.21.15.45:0/2747225174 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7fb48c0625d0 con 0x7fb4700a2910 2024-03-28T17:48:05.261 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-28T17:48:05.262 INFO:teuthology.orchestra.run.smithi045.stdout:mds.a smithi045 running (2m) 23s ago 2m 12.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 6c14419b3a77 2024-03-28T17:48:05.262 INFO:teuthology.orchestra.run.smithi045.stdout:mds.b smithi071 running (119s) 50s ago 119s 30.1M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 1963dfdb35f4 2024-03-28T17:48:05.262 INFO:teuthology.orchestra.run.smithi045.stdout:mds.c smithi151 running (2m) 50s ago 2m 19.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 89a8ca9efd55 2024-03-28T17:48:05.262 INFO:teuthology.orchestra.run.smithi045.stdout:mds.d smithi045 running (2m) 23s ago 2m 12.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 1902e980607f 2024-03-28T17:48:05.262 INFO:teuthology.orchestra.run.smithi045.stdout:mds.e smithi071 running (2m) 50s ago 2m 17.0M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 605a78da2aac 2024-03-28T17:48:05.262 INFO:teuthology.orchestra.run.smithi045.stdout:mds.f smithi151 running (2m) 50s ago 2m 17.0M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 124475ae99a5 2024-03-28T17:48:05.262 INFO:teuthology.orchestra.run.smithi045.stdout:mds.g smithi045 running (2m) 23s ago 2m 12.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 764585b7da2e 2024-03-28T17:48:05.262 INFO:teuthology.orchestra.run.smithi045.stdout:mds.h smithi071 running (2m) 50s ago 2m 16.5M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 4961e27f07c5 2024-03-28T17:48:05.262 INFO:teuthology.orchestra.run.smithi045.stdout:mds.i smithi151 running (2m) 50s ago 2m 22.0M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 7e607e6e801e 2024-03-28T17:48:05.262 INFO:teuthology.orchestra.run.smithi045.stdout:mds.j smithi045 running (2m) 23s ago 2m 12.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 5768fb00f3ea 2024-03-28T17:48:05.263 INFO:teuthology.orchestra.run.smithi045.stdout:mds.k smithi071 running (2m) 50s ago 2m 14.0M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 9f17d11e5016 2024-03-28T17:48:05.263 INFO:teuthology.orchestra.run.smithi045.stdout:mds.l smithi151 running (2m) 50s ago 2m 16.5M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 3ac9123a6649 2024-03-28T17:48:05.263 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi045 *:9283,8765 running (10m) 23s ago 10m 513M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 9e7be486edd5 2024-03-28T17:48:05.263 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi071 *:8443,8765 running (8m) 50s ago 8m 454M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 21216561732b 2024-03-28T17:48:05.263 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.z smithi151 *:8443,8765 running (8m) 50s ago 8m 451M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 ff14039b17a0 2024-03-28T17:48:05.263 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 running (11m) 23s ago 11m 47.1M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 5eba5f57af17 2024-03-28T17:48:05.263 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi071 running (8m) 50s ago 8m 40.6M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 43886e29daec 2024-03-28T17:48:05.263 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi151 running (9m) 50s ago 9m 39.7M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 cb1942f6cc87 2024-03-28T17:48:05.263 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 running (8m) 23s ago 8m 44.4M 4273M 19.0.0-2145-g3be9a5c9 c664afe56bb0 3a92410ff4be 2024-03-28T17:48:05.263 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi071 running (7m) 50s ago 7m 58.3M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 06a184dd4a5f 2024-03-28T17:48:05.263 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi151 running (7m) 50s ago 7m 46.8M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 c892070f1089 2024-03-28T17:48:05.263 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 running (6m) 23s ago 6m 41.8M 4273M 19.0.0-2145-g3be9a5c9 c664afe56bb0 326180babd3e 2024-03-28T17:48:05.264 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi071 running (6m) 50s ago 6m 41.8M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 6d5b8605342d 2024-03-28T17:48:05.264 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi151 running (5m) 50s ago 5m 43.3M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 ca1c31dd2ad2 2024-03-28T17:48:05.264 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi045 running (5m) 23s ago 5m 49.7M 4273M 19.0.0-2145-g3be9a5c9 c664afe56bb0 3fc2b326dee1 2024-03-28T17:48:05.264 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi071 running (4m) 50s ago 4m 54.8M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 2e171835dcd7 2024-03-28T17:48:05.264 INFO:teuthology.orchestra.run.smithi045.stdout:osd.8 smithi151 running (4m) 50s ago 4m 54.0M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 b9de0fce55e0 2024-03-28T17:48:05.264 INFO:teuthology.orchestra.run.smithi045.stdout:osd.9 smithi045 running (3m) 23s ago 3m 51.7M 4273M 19.0.0-2145-g3be9a5c9 c664afe56bb0 11ff6f9a8e3a 2024-03-28T17:48:05.264 INFO:teuthology.orchestra.run.smithi045.stdout:osd.10 smithi071 running (3m) 50s ago 3m 42.4M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 c0c00039a91a 2024-03-28T17:48:05.264 INFO:teuthology.orchestra.run.smithi045.stdout:osd.11 smithi151 running (2m) 50s ago 2m 50.8M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 945f45df3d19 2024-03-28T17:48:05.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:05.264+0000 7fb48be00700 1 -- 172.21.15.45:0/2747225174 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb4700a2910 msgr2=0x7fb4700a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:05.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:05.264+0000 7fb48be00700 1 --2- 172.21.15.45:0/2747225174 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb4700a2910 0x7fb4700a4dd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb478009c60 tx=0x7fb478009260 comp rx=0 tx=0).stop 2024-03-28T17:48:05.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:05.264+0000 7fb48be00700 1 -- 172.21.15.45:0/2747225174 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb48c103f30 msgr2=0x7fb48c19a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:05.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:05.264+0000 7fb48be00700 1 --2- 172.21.15.45:0/2747225174 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb48c103f30 0x7fb48c19a4c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb47c00c990 tx=0x7fb47c00cca0 comp rx=0 tx=0).stop 2024-03-28T17:48:05.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:05.264+0000 7fb48be00700 1 -- 172.21.15.45:0/2747225174 shutdown_connections 2024-03-28T17:48:05.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:05.264+0000 7fb48be00700 1 --2- 172.21.15.45:0/2747225174 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb4700a2910 0x7fb4700a4dd0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:05.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:05.264+0000 7fb48be00700 1 --2- 172.21.15.45:0/2747225174 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb48c1059f0 0x7fb48c1a1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:05.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:05.264+0000 7fb48be00700 1 --2- 172.21.15.45:0/2747225174 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb48c103f30 0x7fb48c19a4c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:05.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:05.264+0000 7fb48be00700 1 --2- 172.21.15.45:0/2747225174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb48c1048d0 0x7fb48c19aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:05.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:05.264+0000 7fb48be00700 1 -- 172.21.15.45:0/2747225174 >> 172.21.15.45:0/2747225174 conn(0x7fb48c0ff8b0 msgr2=0x7fb48c108740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:05.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:05.264+0000 7fb48be00700 1 -- 172.21.15.45:0/2747225174 shutdown_connections 2024-03-28T17:48:05.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:05.264+0000 7fb48be00700 1 -- 172.21.15.45:0/2747225174 wait complete. 2024-03-28T17:48:05.632 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:05 smithi045 ceph-mon[31773]: pgmap v375: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:05.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:05 smithi071 ceph-mon[36597]: pgmap v375: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:05 smithi151 ceph-mon[36481]: pgmap v375: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:06.075 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-28T17:48:06.544 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:48:06.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:06 smithi045 ceph-mon[31773]: from='client.25043 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:48:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:06 smithi071 ceph-mon[36597]: from='client.25043 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:48:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:06 smithi151 ceph-mon[36481]: from='client.25043 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:48:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:07 smithi045 ceph-mon[31773]: pgmap v376: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:48:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:07 smithi071 ceph-mon[36597]: pgmap v376: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:48:07.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:07 smithi151 ceph-mon[36481]: pgmap v376: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:48:08.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.770+0000 7ff4d6c00700 1 -- 172.21.15.45:0/733555250 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4d0105580 msgr2=0x7ff4d0107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:08.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.770+0000 7ff4d6c00700 1 --2- 172.21.15.45:0/733555250 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4d0105580 0x7ff4d0107970 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8009990 tx=0x7ff4b8009ca0 comp rx=0 tx=0).stop 2024-03-28T17:48:08.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.771+0000 7ff4d6c00700 1 -- 172.21.15.45:0/733555250 shutdown_connections 2024-03-28T17:48:08.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.771+0000 7ff4d6c00700 1 --2- 172.21.15.45:0/733555250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d0107eb0 0x7ff4d010a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:08.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.771+0000 7ff4d6c00700 1 --2- 172.21.15.45:0/733555250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4d0102c50 0x7ff4d0105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:08.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.771+0000 7ff4d6c00700 1 --2- 172.21.15.45:0/733555250 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4d0105580 0x7ff4d0107970 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:08.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.771+0000 7ff4d6c00700 1 -- 172.21.15.45:0/733555250 >> 172.21.15.45:0/733555250 conn(0x7ff4d00fc660 msgr2=0x7ff4d00feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:08.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.771+0000 7ff4d6c00700 1 -- 172.21.15.45:0/733555250 shutdown_connections 2024-03-28T17:48:08.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.771+0000 7ff4d6c00700 1 -- 172.21.15.45:0/733555250 wait complete. 2024-03-28T17:48:08.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.772+0000 7ff4d6c00700 1 Processor -- start 2024-03-28T17:48:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.772+0000 7ff4d6c00700 1 -- start start 2024-03-28T17:48:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.773+0000 7ff4d6c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4d0102c50 0x7ff4d01984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.773+0000 7ff4d6c00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d0105580 0x7ff4d01989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.773+0000 7ff4d6c00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4d0107eb0 0x7ff4d019fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.773+0000 7ff4d6c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d010ddf0 con 0x7ff4d0102c50 2024-03-28T17:48:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.773+0000 7ff4d6c00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d010dc70 con 0x7ff4d0105580 2024-03-28T17:48:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.773+0000 7ff4d6c00700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d010df70 con 0x7ff4d0107eb0 2024-03-28T17:48:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.773+0000 7ff4c7400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d0105580 0x7ff4d01989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.773+0000 7ff4c7400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d0105580 0x7ff4d01989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:50414/0 (socket says 172.21.15.45:50414) 2024-03-28T17:48:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.773+0000 7ff4c7400700 1 -- 172.21.15.45:0/3377888327 learned_addr learned my addr 172.21.15.45:0/3377888327 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.773+0000 7ff4cfe00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4d0102c50 0x7ff4d01984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.773+0000 7ff4d4c00700 1 --2- 172.21.15.45:0/3377888327 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4d0107eb0 0x7ff4d019fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:08.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.773+0000 7ff4c7400700 1 -- 172.21.15.45:0/3377888327 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4d0107eb0 msgr2=0x7ff4d019fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:08.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.773+0000 7ff4c7400700 1 --2- 172.21.15.45:0/3377888327 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4d0107eb0 0x7ff4d019fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:08.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.773+0000 7ff4c7400700 1 -- 172.21.15.45:0/3377888327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4d0102c50 msgr2=0x7ff4d01984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:08.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.773+0000 7ff4c7400700 1 --2- 172.21.15.45:0/3377888327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4d0102c50 0x7ff4d01984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:08.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.774+0000 7ff4c7400700 1 -- 172.21.15.45:0/3377888327 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d01a0100 con 0x7ff4d0105580 2024-03-28T17:48:08.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.774+0000 7ff4d4c00700 1 --2- 172.21.15.45:0/3377888327 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4d0107eb0 0x7ff4d019fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:48:08.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.774+0000 7ff4c7400700 1 --2- 172.21.15.45:0/3377888327 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d0105580 0x7ff4d01989e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8005910 tx=0x7ff4b800bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:08.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.774+0000 7ff4cd600700 1 -- 172.21.15.45:0/3377888327 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8026070 con 0x7ff4d0105580 2024-03-28T17:48:08.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.774+0000 7ff4cd600700 1 -- 172.21.15.45:0/3377888327 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4b8003d70 con 0x7ff4d0105580 2024-03-28T17:48:08.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.774+0000 7ff4cd600700 1 -- 172.21.15.45:0/3377888327 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8021670 con 0x7ff4d0105580 2024-03-28T17:48:08.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.774+0000 7ff4d6c00700 1 -- 172.21.15.45:0/3377888327 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d01a0390 con 0x7ff4d0105580 2024-03-28T17:48:08.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.774+0000 7ff4d6c00700 1 -- 172.21.15.45:0/3377888327 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d01a0840 con 0x7ff4d0105580 2024-03-28T17:48:08.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.775+0000 7ff4d6c00700 1 -- 172.21.15.45:0/3377888327 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d0068e50 con 0x7ff4d0105580 2024-03-28T17:48:08.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.778+0000 7ff4cd600700 1 -- 172.21.15.45:0/3377888327 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff4b8029070 con 0x7ff4d0105580 2024-03-28T17:48:08.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.780+0000 7ff4cd600700 1 --2- 172.21.15.45:0/3377888327 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff4b40a2910 0x7ff4b40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:08.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.780+0000 7ff4cd600700 1 -- 172.21.15.45:0/3377888327 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff4b80cf7a0 con 0x7ff4d0105580 2024-03-28T17:48:08.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.780+0000 7ff4cfe00700 1 --2- 172.21.15.45:0/3377888327 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff4b40a2910 0x7ff4b40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:08.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.781+0000 7ff4cfe00700 1 --2- 172.21.15.45:0/3377888327 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff4b40a2910 0x7ff4b40a4dd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0009c60 tx=0x7ff4c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:08.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:08.781+0000 7ff4cd600700 1 -- 172.21.15.45:0/3377888327 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff4b8013070 con 0x7ff4d0105580 2024-03-28T17:48:09.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:09.022+0000 7ff4d6c00700 1 -- 172.21.15.45:0/3377888327 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4d0062690 con 0x7ff4b40a2910 2024-03-28T17:48:09.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:09.033+0000 7ff4cd600700 1 -- 172.21.15.45:0/3377888327 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7ff4d0062690 con 0x7ff4b40a2910 2024-03-28T17:48:09.033 INFO:teuthology.orchestra.run.smithi045.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-28T17:48:09.033 INFO:teuthology.orchestra.run.smithi045.stdout:mds.all 12/12 54s ago 2m smithi045=a;smithi045=d;smithi045=g;smithi045=j;smithi071=b;smithi071=e;smithi071=h;smithi071=k;smithi151=c;smithi151=f;smithi151=i;smithi151=l;count:12 2024-03-28T17:48:09.034 INFO:teuthology.orchestra.run.smithi045.stdout:mgr 3/3 54s ago 8m smithi045=x;smithi071=y;smithi151=z;count:3 2024-03-28T17:48:09.034 INFO:teuthology.orchestra.run.smithi045.stdout:mon 3/3 54s ago 9m smithi045:172.21.15.45=a;smithi071:172.21.15.71=b;smithi151:172.21.15.151=c;count:3 2024-03-28T17:48:09.034 INFO:teuthology.orchestra.run.smithi045.stdout:osd 12 54s ago - 2024-03-28T17:48:09.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:09.036+0000 7ff4d6c00700 1 -- 172.21.15.45:0/3377888327 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff4b40a2910 msgr2=0x7ff4b40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:09.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:09.036+0000 7ff4d6c00700 1 --2- 172.21.15.45:0/3377888327 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff4b40a2910 0x7ff4b40a4dd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0009c60 tx=0x7ff4c0009260 comp rx=0 tx=0).stop 2024-03-28T17:48:09.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:09.036+0000 7ff4d6c00700 1 -- 172.21.15.45:0/3377888327 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d0105580 msgr2=0x7ff4d01989e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:09.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:09.036+0000 7ff4d6c00700 1 --2- 172.21.15.45:0/3377888327 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d0105580 0x7ff4d01989e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8005910 tx=0x7ff4b800bbb0 comp rx=0 tx=0).stop 2024-03-28T17:48:09.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:09.036+0000 7ff4d6c00700 1 -- 172.21.15.45:0/3377888327 shutdown_connections 2024-03-28T17:48:09.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:09.036+0000 7ff4d6c00700 1 --2- 172.21.15.45:0/3377888327 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff4b40a2910 0x7ff4b40a4dd0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:09.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:09.036+0000 7ff4d6c00700 1 --2- 172.21.15.45:0/3377888327 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4d0107eb0 0x7ff4d019fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:09.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:09.036+0000 7ff4d6c00700 1 --2- 172.21.15.45:0/3377888327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4d0102c50 0x7ff4d01984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:09.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:09.036+0000 7ff4d6c00700 1 --2- 172.21.15.45:0/3377888327 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d0105580 0x7ff4d01989e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:09.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:09.036+0000 7ff4d6c00700 1 -- 172.21.15.45:0/3377888327 >> 172.21.15.45:0/3377888327 conn(0x7ff4d00fc660 msgr2=0x7ff4d0103730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:09.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:09.036+0000 7ff4d6c00700 1 -- 172.21.15.45:0/3377888327 shutdown_connections 2024-03-28T17:48:09.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:09.036+0000 7ff4d6c00700 1 -- 172.21.15.45:0/3377888327 wait complete. 2024-03-28T17:48:09.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:09 smithi045 ceph-mon[31773]: pgmap v377: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:09.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:09 smithi071 ceph-mon[36597]: pgmap v377: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:09.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:09 smithi151 ceph-mon[36481]: pgmap v377: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:09.845 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-28T17:48:10.318 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:48:10.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:10 smithi045 ceph-mon[31773]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:48:10.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:10 smithi071 ceph-mon[36597]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:48:10.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:10 smithi151 ceph-mon[36481]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:48:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:11 smithi045 ceph-mon[31773]: pgmap v378: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T17:48:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:11 smithi071 ceph-mon[36597]: pgmap v378: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T17:48:11.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:11 smithi151 ceph-mon[36481]: pgmap v378: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T17:48:12.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.578+0000 7f53c0a00700 1 -- 172.21.15.45:0/1026653534 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53bc102c60 msgr2=0x7f53bc109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:12.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.578+0000 7f53c0a00700 1 --2- 172.21.15.45:0/1026653534 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53bc102c60 0x7f53bc109510 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f53b400b0d0 tx=0x7f53b400b3e0 comp rx=0 tx=0).stop 2024-03-28T17:48:12.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.578+0000 7f53c0a00700 1 -- 172.21.15.45:0/1026653534 shutdown_connections 2024-03-28T17:48:12.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.578+0000 7f53c0a00700 1 --2- 172.21.15.45:0/1026653534 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53bc1010c0 0x7f53bc1014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:12.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.578+0000 7f53c0a00700 1 --2- 172.21.15.45:0/1026653534 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53bc102c60 0x7f53bc109510 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:12.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.578+0000 7f53c0a00700 1 --2- 172.21.15.45:0/1026653534 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53bc1022c0 0x7f53bc102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:12.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.578+0000 7f53c0a00700 1 -- 172.21.15.45:0/1026653534 >> 172.21.15.45:0/1026653534 conn(0x7f53bc0fc660 msgr2=0x7f53bc0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:12.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.578+0000 7f53c0a00700 1 -- 172.21.15.45:0/1026653534 shutdown_connections 2024-03-28T17:48:12.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.578+0000 7f53c0a00700 1 -- 172.21.15.45:0/1026653534 wait complete. 2024-03-28T17:48:12.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.579+0000 7f53c0a00700 1 Processor -- start 2024-03-28T17:48:12.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.579+0000 7f53c0a00700 1 -- start start 2024-03-28T17:48:12.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.579+0000 7f53c0a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53bc1010c0 0x7f53bc198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:12.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.579+0000 7f53c0a00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53bc1022c0 0x7f53bc198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:12.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.579+0000 7f53c0a00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53bc102c60 0x7f53bc19f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:12.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.579+0000 7f53c0a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc10cce0 con 0x7f53bc1010c0 2024-03-28T17:48:12.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.579+0000 7f53c0a00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc10cb60 con 0x7f53bc1022c0 2024-03-28T17:48:12.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.579+0000 7f53c0a00700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc10ce60 con 0x7f53bc102c60 2024-03-28T17:48:12.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.580+0000 7f53b9400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53bc1022c0 0x7f53bc198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:12.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.580+0000 7f53ba800700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53bc102c60 0x7f53bc19f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:12.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.580+0000 7f53b9400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53bc1022c0 0x7f53bc198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:50418/0 (socket says 172.21.15.45:50418) 2024-03-28T17:48:12.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.580+0000 7f53b9400700 1 -- 172.21.15.45:0/2258083885 learned_addr learned my addr 172.21.15.45:0/2258083885 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:12.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.580+0000 7f53b9e00700 1 --2- 172.21.15.45:0/2258083885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53bc1010c0 0x7f53bc198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:12.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.580+0000 7f53ba800700 1 -- 172.21.15.45:0/2258083885 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53bc1022c0 msgr2=0x7f53bc198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:12.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.580+0000 7f53ba800700 1 --2- 172.21.15.45:0/2258083885 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53bc1022c0 0x7f53bc198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:12.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.580+0000 7f53ba800700 1 -- 172.21.15.45:0/2258083885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53bc1010c0 msgr2=0x7f53bc198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:12.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.580+0000 7f53ba800700 1 --2- 172.21.15.45:0/2258083885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53bc1010c0 0x7f53bc198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:12.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.580+0000 7f53ba800700 1 -- 172.21.15.45:0/2258083885 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53bc1a0090 con 0x7f53bc102c60 2024-03-28T17:48:12.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.580+0000 7f53b9400700 1 --2- 172.21.15.45:0/2258083885 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53bc1022c0 0x7f53bc198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:12.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.581+0000 7f53ba800700 1 --2- 172.21.15.45:0/2258083885 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53bc102c60 0x7f53bc19f9f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f53b400b0d0 tx=0x7f53b40079a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:12.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.581+0000 7f53b2a00700 1 -- 172.21.15.45:0/2258083885 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b4021070 con 0x7f53bc102c60 2024-03-28T17:48:12.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.581+0000 7f53b2a00700 1 -- 172.21.15.45:0/2258083885 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f53b401db30 con 0x7f53bc102c60 2024-03-28T17:48:12.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.581+0000 7f53b2a00700 1 -- 172.21.15.45:0/2258083885 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b401baf0 con 0x7f53bc102c60 2024-03-28T17:48:12.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.581+0000 7f53c0a00700 1 -- 172.21.15.45:0/2258083885 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53bc1a0320 con 0x7f53bc102c60 2024-03-28T17:48:12.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.581+0000 7f53c0a00700 1 -- 172.21.15.45:0/2258083885 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53bc1a0780 con 0x7f53bc102c60 2024-03-28T17:48:12.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.582+0000 7f53c0a00700 1 -- 172.21.15.45:0/2258083885 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53bc068e50 con 0x7f53bc102c60 2024-03-28T17:48:12.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.584+0000 7f53b2a00700 1 -- 172.21.15.45:0/2258083885 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f53b4024070 con 0x7f53bc102c60 2024-03-28T17:48:12.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.585+0000 7f53b2a00700 1 --2- 172.21.15.45:0/2258083885 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f53a00a2910 0x7f53a00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:12.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.585+0000 7f53b2a00700 1 -- 172.21.15.45:0/2258083885 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f53b40cfb10 con 0x7f53bc102c60 2024-03-28T17:48:12.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.586+0000 7f53b9e00700 1 --2- 172.21.15.45:0/2258083885 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f53a00a2910 0x7f53a00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:12.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.586+0000 7f53b9e00700 1 --2- 172.21.15.45:0/2258083885 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f53a00a2910 0x7f53a00a4dd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f53bc102120 tx=0x7f53ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:12.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.589+0000 7f53b2a00700 1 -- 172.21.15.45:0/2258083885 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f53b409cbf0 con 0x7f53bc102c60 2024-03-28T17:48:12.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.829+0000 7f53c0a00700 1 -- 172.21.15.45:0/2258083885 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f53bc06a550 con 0x7f53a00a2910 2024-03-28T17:48:12.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.832+0000 7f53b2a00700 1 -- 172.21.15.45:0/2258083885 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f53bc06a550 con 0x7f53a00a2910 2024-03-28T17:48:12.833 INFO:teuthology.orchestra.run.smithi045.stdout:HOST ADDR LABELS STATUS 2024-03-28T17:48:12.833 INFO:teuthology.orchestra.run.smithi045.stdout:smithi045 172.21.15.45 2024-03-28T17:48:12.833 INFO:teuthology.orchestra.run.smithi045.stdout:smithi071 172.21.15.71 2024-03-28T17:48:12.833 INFO:teuthology.orchestra.run.smithi045.stdout:smithi151 172.21.15.151 2024-03-28T17:48:12.833 INFO:teuthology.orchestra.run.smithi045.stdout:3 hosts in cluster 2024-03-28T17:48:12.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.835+0000 7f53c0a00700 1 -- 172.21.15.45:0/2258083885 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f53a00a2910 msgr2=0x7f53a00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:12.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.835+0000 7f53c0a00700 1 --2- 172.21.15.45:0/2258083885 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f53a00a2910 0x7f53a00a4dd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f53bc102120 tx=0x7f53ac009260 comp rx=0 tx=0).stop 2024-03-28T17:48:12.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.835+0000 7f53c0a00700 1 -- 172.21.15.45:0/2258083885 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53bc102c60 msgr2=0x7f53bc19f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:12.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.835+0000 7f53c0a00700 1 --2- 172.21.15.45:0/2258083885 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53bc102c60 0x7f53bc19f9f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f53b400b0d0 tx=0x7f53b40079a0 comp rx=0 tx=0).stop 2024-03-28T17:48:12.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.836+0000 7f53c0a00700 1 -- 172.21.15.45:0/2258083885 shutdown_connections 2024-03-28T17:48:12.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.836+0000 7f53c0a00700 1 --2- 172.21.15.45:0/2258083885 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f53a00a2910 0x7f53a00a4dd0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:12.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.836+0000 7f53c0a00700 1 --2- 172.21.15.45:0/2258083885 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53bc102c60 0x7f53bc19f9f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:12.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.836+0000 7f53c0a00700 1 --2- 172.21.15.45:0/2258083885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53bc1010c0 0x7f53bc198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:12.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.836+0000 7f53c0a00700 1 --2- 172.21.15.45:0/2258083885 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53bc1022c0 0x7f53bc198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:12.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.836+0000 7f53c0a00700 1 -- 172.21.15.45:0/2258083885 >> 172.21.15.45:0/2258083885 conn(0x7f53bc0fc660 msgr2=0x7f53bc069e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:12.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.836+0000 7f53c0a00700 1 -- 172.21.15.45:0/2258083885 shutdown_connections 2024-03-28T17:48:12.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:12.836+0000 7f53c0a00700 1 -- 172.21.15.45:0/2258083885 wait complete. 2024-03-28T17:48:13.716 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:13 smithi045 ceph-mon[31773]: pgmap v379: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:48:13.721 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-28T17:48:13.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:13 smithi071 ceph-mon[36597]: pgmap v379: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:48:13.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:13 smithi151 ceph-mon[36481]: pgmap v379: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:48:14.203 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:48:14.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:14 smithi045 ceph-mon[31773]: from='client.25030 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:48:14.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:14 smithi071 ceph-mon[36597]: from='client.25030 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:48:14.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:14 smithi151 ceph-mon[36481]: from='client.25030 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:48:15.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:15 smithi045 ceph-mon[31773]: pgmap v380: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:15.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:15 smithi071 ceph-mon[36597]: pgmap v380: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:15.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:15 smithi151 ceph-mon[36481]: pgmap v380: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:16.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.325+0000 7f4f0be00700 1 -- 172.21.15.45:0/3633663490 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f04109a10 msgr2=0x7f4f04109e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:16.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.325+0000 7f4f0be00700 1 --2- 172.21.15.45:0/3633663490 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f04109a10 0x7f4f04109e70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4ef00099c0 tx=0x7f4ef0009cd0 comp rx=0 tx=0).stop 2024-03-28T17:48:16.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.326+0000 7f4f0be00700 1 -- 172.21.15.45:0/3633663490 shutdown_connections 2024-03-28T17:48:16.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.326+0000 7f4f0be00700 1 --2- 172.21.15.45:0/3633663490 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f0410a3b0 0x7f4f04110c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:16.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.326+0000 7f4f0be00700 1 --2- 172.21.15.45:0/3633663490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f04108810 0x7f4f04108c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:16.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.326+0000 7f4f0be00700 1 --2- 172.21.15.45:0/3633663490 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f04109a10 0x7f4f04109e70 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:16.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.326+0000 7f4f0be00700 1 -- 172.21.15.45:0/3633663490 >> 172.21.15.45:0/3633663490 conn(0x7f4f040777e0 msgr2=0x7f4f04079c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:16.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.326+0000 7f4f0be00700 1 -- 172.21.15.45:0/3633663490 shutdown_connections 2024-03-28T17:48:16.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.326+0000 7f4f0be00700 1 -- 172.21.15.45:0/3633663490 wait complete. 2024-03-28T17:48:16.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.327+0000 7f4f0be00700 1 Processor -- start 2024-03-28T17:48:16.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.327+0000 7f4f0be00700 1 -- start start 2024-03-28T17:48:16.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.327+0000 7f4f0be00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f04108810 0x7f4f0419ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:16.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.327+0000 7f4f0be00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f04109a10 0x7f4f0419ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:16.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.327+0000 7f4f0be00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f0410a3b0 0x7f4f041a5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:16.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.327+0000 7f4f0be00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f04114450 con 0x7f4f04109a10 2024-03-28T17:48:16.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.327+0000 7f4f0be00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f041142d0 con 0x7f4f0410a3b0 2024-03-28T17:48:16.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.327+0000 7f4f0be00700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f041145d0 con 0x7f4f04108810 2024-03-28T17:48:16.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.328+0000 7f4f09400700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f04108810 0x7f4f0419ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:16.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.328+0000 7f4f09e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f0410a3b0 0x7f4f041a5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:16.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.328+0000 7f4f09400700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f04108810 0x7f4f0419ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:59012/0 (socket says 172.21.15.45:59012) 2024-03-28T17:48:16.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.328+0000 7f4f09400700 1 -- 172.21.15.45:0/2238783991 learned_addr learned my addr 172.21.15.45:0/2238783991 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:16.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.328+0000 7f4f09e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f0410a3b0 0x7f4f041a5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:60050/0 (socket says 172.21.15.45:60050) 2024-03-28T17:48:16.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.328+0000 7f4f08a00700 1 --2- 172.21.15.45:0/2238783991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f04109a10 0x7f4f0419ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:16.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.328+0000 7f4f09400700 1 -- 172.21.15.45:0/2238783991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f0410a3b0 msgr2=0x7f4f041a5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:16.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.328+0000 7f4f09400700 1 --2- 172.21.15.45:0/2238783991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f0410a3b0 0x7f4f041a5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:16.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.328+0000 7f4f09400700 1 -- 172.21.15.45:0/2238783991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f04109a10 msgr2=0x7f4f0419ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:16.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.328+0000 7f4f09400700 1 --2- 172.21.15.45:0/2238783991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f04109a10 0x7f4f0419ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:16.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.328+0000 7f4f09400700 1 -- 172.21.15.45:0/2238783991 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f041a6660 con 0x7f4f04108810 2024-03-28T17:48:16.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.328+0000 7f4f09400700 1 --2- 172.21.15.45:0/2238783991 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f04108810 0x7f4f0419ea00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8009f40 tx=0x7f4ef800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:16.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.329+0000 7f4f02000700 1 -- 172.21.15.45:0/2238783991 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef80092e0 con 0x7f4f04108810 2024-03-28T17:48:16.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.329+0000 7f4f02000700 1 -- 172.21.15.45:0/2238783991 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ef800c9c0 con 0x7f4f04108810 2024-03-28T17:48:16.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.329+0000 7f4f0be00700 1 -- 172.21.15.45:0/2238783991 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f041a68f0 con 0x7f4f04108810 2024-03-28T17:48:16.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.329+0000 7f4f02000700 1 -- 172.21.15.45:0/2238783991 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8004580 con 0x7f4f04108810 2024-03-28T17:48:16.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.329+0000 7f4f0be00700 1 -- 172.21.15.45:0/2238783991 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f041a6e80 con 0x7f4f04108810 2024-03-28T17:48:16.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.330+0000 7f4f0be00700 1 -- 172.21.15.45:0/2238783991 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f04068e50 con 0x7f4f04108810 2024-03-28T17:48:16.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.332+0000 7f4f02000700 1 -- 172.21.15.45:0/2238783991 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4ef801b030 con 0x7f4f04108810 2024-03-28T17:48:16.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.333+0000 7f4f02000700 1 --2- 172.21.15.45:0/2238783991 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4eec0a2910 0x7f4eec0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:16.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.333+0000 7f4f02000700 1 -- 172.21.15.45:0/2238783991 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f4ef80ce930 con 0x7f4f04108810 2024-03-28T17:48:16.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.333+0000 7f4f08a00700 1 --2- 172.21.15.45:0/2238783991 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4eec0a2910 0x7f4eec0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:16.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.334+0000 7f4f08a00700 1 --2- 172.21.15.45:0/2238783991 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4eec0a2910 0x7f4eec0a4dd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0000c00 tx=0x7f4ef0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:16.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.337+0000 7f4f02000700 1 -- 172.21.15.45:0/2238783991 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4ef8022020 con 0x7f4f04108810 2024-03-28T17:48:16.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.572+0000 7f4f0be00700 1 -- 172.21.15.45:0/2238783991 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f040625d0 con 0x7f4eec0a2910 2024-03-28T17:48:16.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.575+0000 7f4f02000700 1 -- 172.21.15.45:0/2238783991 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7f4f040625d0 con 0x7f4eec0a2910 2024-03-28T17:48:16.576 INFO:teuthology.orchestra.run.smithi045.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-28T17:48:16.576 INFO:teuthology.orchestra.run.smithi045.stdout:smithi045 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB75120098480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-03-28T17:48:16.576 INFO:teuthology.orchestra.run.smithi045.stdout:smithi071 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62040172400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-28T17:48:16.576 INFO:teuthology.orchestra.run.smithi045.stdout:smithi151 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300JJ400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-28T17:48:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.578+0000 7f4f0be00700 1 -- 172.21.15.45:0/2238783991 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4eec0a2910 msgr2=0x7f4eec0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.578+0000 7f4f0be00700 1 --2- 172.21.15.45:0/2238783991 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4eec0a2910 0x7f4eec0a4dd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0000c00 tx=0x7f4ef0005fb0 comp rx=0 tx=0).stop 2024-03-28T17:48:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.578+0000 7f4f0be00700 1 -- 172.21.15.45:0/2238783991 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f04108810 msgr2=0x7f4f0419ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.578+0000 7f4f0be00700 1 --2- 172.21.15.45:0/2238783991 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f04108810 0x7f4f0419ea00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8009f40 tx=0x7f4ef800c3a0 comp rx=0 tx=0).stop 2024-03-28T17:48:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.578+0000 7f4f0be00700 1 -- 172.21.15.45:0/2238783991 shutdown_connections 2024-03-28T17:48:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.578+0000 7f4f0be00700 1 --2- 172.21.15.45:0/2238783991 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f4eec0a2910 0x7f4eec0a4dd0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.578+0000 7f4f0be00700 1 --2- 172.21.15.45:0/2238783991 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f04108810 0x7f4f0419ea00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.578+0000 7f4f0be00700 1 --2- 172.21.15.45:0/2238783991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f0410a3b0 0x7f4f041a5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:16.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.578+0000 7f4f0be00700 1 --2- 172.21.15.45:0/2238783991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f04109a10 0x7f4f0419ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:16.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.579+0000 7f4f0be00700 1 -- 172.21.15.45:0/2238783991 >> 172.21.15.45:0/2238783991 conn(0x7f4f040777e0 msgr2=0x7f4f0410ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:16.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.579+0000 7f4f0be00700 1 -- 172.21.15.45:0/2238783991 shutdown_connections 2024-03-28T17:48:16.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:16.579+0000 7f4f0be00700 1 -- 172.21.15.45:0/2238783991 wait complete. 2024-03-28T17:48:17.473 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-28T17:48:17.486 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi045.front.sepia.ceph.com 2024-03-28T17:48:17.486 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-28T17:48:17.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:17 smithi045 ceph-mon[31773]: pgmap v381: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:48:17.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:17 smithi045 ceph-mon[31773]: from='client.25036 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:48:17.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:17 smithi071 ceph-mon[36597]: pgmap v381: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:48:17.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:17 smithi071 ceph-mon[36597]: from='client.25036 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:48:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:17 smithi151 ceph-mon[36481]: pgmap v381: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:48:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:17 smithi151 ceph-mon[36481]: from='client.25036 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:48:17.951 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:48:19.570 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:19 smithi045 ceph-mon[31773]: pgmap v382: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:19.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:19 smithi071 ceph-mon[36597]: pgmap v382: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:19.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:19 smithi151 ceph-mon[36481]: pgmap v382: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:20.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.072+0000 7f4999800700 1 -- 172.21.15.45:0/2396430178 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4994103300 msgr2=0x7f4994103760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:20.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.072+0000 7f4999800700 1 --2- 172.21.15.45:0/2396430178 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4994103300 0x7f4994103760 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f497c0098f0 tx=0x7f497c009c00 comp rx=0 tx=0).stop 2024-03-28T17:48:20.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.072+0000 7f4999800700 1 -- 172.21.15.45:0/2396430178 shutdown_connections 2024-03-28T17:48:20.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.072+0000 7f4999800700 1 --2- 172.21.15.45:0/2396430178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4994103ca0 0x7f499410a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:20.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.072+0000 7f4999800700 1 --2- 172.21.15.45:0/2396430178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4994102100 0x7f4994102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:20.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.072+0000 7f4999800700 1 --2- 172.21.15.45:0/2396430178 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4994103300 0x7f4994103760 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:20.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.072+0000 7f4999800700 1 -- 172.21.15.45:0/2396430178 >> 172.21.15.45:0/2396430178 conn(0x7f49940fd6a0 msgr2=0x7f49940ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:20.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.072+0000 7f4999800700 1 -- 172.21.15.45:0/2396430178 shutdown_connections 2024-03-28T17:48:20.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.072+0000 7f4999800700 1 -- 172.21.15.45:0/2396430178 wait complete. 2024-03-28T17:48:20.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.073+0000 7f4999800700 1 Processor -- start 2024-03-28T17:48:20.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.074+0000 7f4999800700 1 -- start start 2024-03-28T17:48:20.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.074+0000 7f4999800700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4994102100 0x7f4994198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:20.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.074+0000 7f4999800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4994103300 0x7f49941988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:20.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.074+0000 7f4999800700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4994103ca0 0x7f499419f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:20.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.074+0000 7f4999800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499410dd20 con 0x7f4994103300 2024-03-28T17:48:20.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.074+0000 7f4999800700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499410dba0 con 0x7f4994103ca0 2024-03-28T17:48:20.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.074+0000 7f4999800700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499410dea0 con 0x7f4994102100 2024-03-28T17:48:20.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.074+0000 7f4993e00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4994102100 0x7f4994198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:20.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.074+0000 7f4993e00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4994102100 0x7f4994198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:59032/0 (socket says 172.21.15.45:59032) 2024-03-28T17:48:20.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.074+0000 7f4993e00700 1 -- 172.21.15.45:0/4264711908 learned_addr learned my addr 172.21.15.45:0/4264711908 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:20.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.075+0000 7f4998e00700 1 --2- 172.21.15.45:0/4264711908 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4994103ca0 0x7f499419f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:20.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.075+0000 7f498b400700 1 --2- 172.21.15.45:0/4264711908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4994103300 0x7f49941988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:20.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.075+0000 7f4993e00700 1 -- 172.21.15.45:0/4264711908 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4994103ca0 msgr2=0x7f499419f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:20.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.075+0000 7f4993e00700 1 --2- 172.21.15.45:0/4264711908 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4994103ca0 0x7f499419f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:20.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.075+0000 7f4993e00700 1 -- 172.21.15.45:0/4264711908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4994103300 msgr2=0x7f49941988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:20.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.075+0000 7f4993e00700 1 --2- 172.21.15.45:0/4264711908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4994103300 0x7f49941988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:20.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.075+0000 7f4993e00700 1 -- 172.21.15.45:0/4264711908 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499419ffa0 con 0x7f4994102100 2024-03-28T17:48:20.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.075+0000 7f4993e00700 1 --2- 172.21.15.45:0/4264711908 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4994102100 0x7f4994198390 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4984009f40 tx=0x7f498400c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:20.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.075+0000 7f4991600700 1 -- 172.21.15.45:0/4264711908 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49840092e0 con 0x7f4994102100 2024-03-28T17:48:20.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.075+0000 7f4991600700 1 -- 172.21.15.45:0/4264711908 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4984003a00 con 0x7f4994102100 2024-03-28T17:48:20.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.075+0000 7f4999800700 1 -- 172.21.15.45:0/4264711908 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49941a0230 con 0x7f4994102100 2024-03-28T17:48:20.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.075+0000 7f4991600700 1 -- 172.21.15.45:0/4264711908 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4984004680 con 0x7f4994102100 2024-03-28T17:48:20.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.076+0000 7f4999800700 1 -- 172.21.15.45:0/4264711908 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49941a07c0 con 0x7f4994102100 2024-03-28T17:48:20.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.077+0000 7f4999800700 1 -- 172.21.15.45:0/4264711908 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4994068e50 con 0x7f4994102100 2024-03-28T17:48:20.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.078+0000 7f4991600700 1 -- 172.21.15.45:0/4264711908 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f498401b030 con 0x7f4994102100 2024-03-28T17:48:20.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.080+0000 7f4991600700 1 --2- 172.21.15.45:0/4264711908 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f49780a2910 0x7f49780a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:20.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.080+0000 7f4991600700 1 -- 172.21.15.45:0/4264711908 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f49840ce910 con 0x7f4994102100 2024-03-28T17:48:20.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.080+0000 7f498b400700 1 --2- 172.21.15.45:0/4264711908 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f49780a2910 0x7f49780a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:20.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.081+0000 7f498b400700 1 --2- 172.21.15.45:0/4264711908 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f49780a2910 0x7f49780a4dd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f497c0098c0 tx=0x7f497c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:20.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.084+0000 7f4991600700 1 -- 172.21.15.45:0/4264711908 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4984022020 con 0x7f4994102100 2024-03-28T17:48:20.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.365+0000 7f4999800700 1 -- 172.21.15.45:0/4264711908 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4994062da0 con 0x7f4994102100 2024-03-28T17:48:20.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.365+0000 7f4991600700 1 -- 172.21.15.45:0/4264711908 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+2992 (secure 0 0 0) 0x7f4984010040 con 0x7f4994102100 2024-03-28T17:48:20.366 INFO:teuthology.orchestra.run.smithi045.stdout:e22 2024-03-28T17:48:20.366 INFO:teuthology.orchestra.run.smithi045.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T17:48:20.366 INFO:teuthology.orchestra.run.smithi045.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T17:48:20.367 INFO:teuthology.orchestra.run.smithi045.stdout:legacy client fscid: 1 2024-03-28T17:48:20.367 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:48:20.367 INFO:teuthology.orchestra.run.smithi045.stdout:Filesystem 'cephfs' (1) 2024-03-28T17:48:20.367 INFO:teuthology.orchestra.run.smithi045.stdout:fs_name cephfs 2024-03-28T17:48:20.367 INFO:teuthology.orchestra.run.smithi045.stdout:epoch 22 2024-03-28T17:48:20.367 INFO:teuthology.orchestra.run.smithi045.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-28T17:48:20.367 INFO:teuthology.orchestra.run.smithi045.stdout:created 2024-03-28T17:46:12.261135+0000 2024-03-28T17:48:20.367 INFO:teuthology.orchestra.run.smithi045.stdout:modified 2024-03-28T17:46:17.967524+0000 2024-03-28T17:48:20.367 INFO:teuthology.orchestra.run.smithi045.stdout:tableserver 0 2024-03-28T17:48:20.367 INFO:teuthology.orchestra.run.smithi045.stdout:root 0 2024-03-28T17:48:20.367 INFO:teuthology.orchestra.run.smithi045.stdout:session_timeout 300 2024-03-28T17:48:20.367 INFO:teuthology.orchestra.run.smithi045.stdout:session_autoclose 300 2024-03-28T17:48:20.367 INFO:teuthology.orchestra.run.smithi045.stdout:max_file_size 1099511627776 2024-03-28T17:48:20.368 INFO:teuthology.orchestra.run.smithi045.stdout:max_xattr_size 65536 2024-03-28T17:48:20.368 INFO:teuthology.orchestra.run.smithi045.stdout:required_client_features {} 2024-03-28T17:48:20.368 INFO:teuthology.orchestra.run.smithi045.stdout:last_failure 0 2024-03-28T17:48:20.368 INFO:teuthology.orchestra.run.smithi045.stdout:last_failure_osd_epoch 0 2024-03-28T17:48:20.368 INFO:teuthology.orchestra.run.smithi045.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T17:48:20.368 INFO:teuthology.orchestra.run.smithi045.stdout:max_mds 3 2024-03-28T17:48:20.368 INFO:teuthology.orchestra.run.smithi045.stdout:in 0,1,2 2024-03-28T17:48:20.368 INFO:teuthology.orchestra.run.smithi045.stdout:up {0=24584,1=24530,2=24529} 2024-03-28T17:48:20.368 INFO:teuthology.orchestra.run.smithi045.stdout:failed 2024-03-28T17:48:20.368 INFO:teuthology.orchestra.run.smithi045.stdout:damaged 2024-03-28T17:48:20.368 INFO:teuthology.orchestra.run.smithi045.stdout:stopped 2024-03-28T17:48:20.368 INFO:teuthology.orchestra.run.smithi045.stdout:data_pools [3] 2024-03-28T17:48:20.368 INFO:teuthology.orchestra.run.smithi045.stdout:metadata_pool 2 2024-03-28T17:48:20.368 INFO:teuthology.orchestra.run.smithi045.stdout:inline_data disabled 2024-03-28T17:48:20.369 INFO:teuthology.orchestra.run.smithi045.stdout:balancer 2024-03-28T17:48:20.369 INFO:teuthology.orchestra.run.smithi045.stdout:bal_rank_mask -1 2024-03-28T17:48:20.369 INFO:teuthology.orchestra.run.smithi045.stdout:standby_count_wanted 0 2024-03-28T17:48:20.369 INFO:teuthology.orchestra.run.smithi045.stdout:qdb_cluster leader: 24584 members: 24529,24530,24584 2024-03-28T17:48:20.369 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.b{0:24584} state up:active seq 3 addr [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T17:48:20.369 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.c{0:24511} state up:standby-replay seq 1 addr [v2:172.21.15.151:6836/1238964845,v1:172.21.15.151:6837/1238964845] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T17:48:20.369 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.h{1:24530} state up:active seq 8 addr [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T17:48:20.369 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.e{1:24500} state up:standby-replay seq 1 addr [v2:172.21.15.71:6834/4234998385,v1:172.21.15.71:6835/4234998385] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T17:48:20.369 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.i{2:24529} state up:active seq 7 addr [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T17:48:20.369 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.f{2:24499} state up:standby-replay seq 1 addr [v2:172.21.15.151:6834/2876354410,v1:172.21.15.151:6835/2876354410] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T17:48:20.369 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:48:20.369 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:48:20.370 INFO:teuthology.orchestra.run.smithi045.stdout:Standby daemons: 2024-03-28T17:48:20.370 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:48:20.370 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.j{-1:14388} state up:standby seq 1 addr [v2:172.21.15.45:6834/333952790,v1:172.21.15.45:6835/333952790] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T17:48:20.370 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.g{-1:14454} state up:standby seq 1 addr [v2:172.21.15.45:6838/2313500150,v1:172.21.15.45:6839/2313500150] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T17:48:20.370 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.a{-1:14472} state up:standby seq 1 addr [v2:172.21.15.45:6840/2647972571,v1:172.21.15.45:6841/2647972571] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T17:48:20.370 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.l{-1:24433} state up:standby seq 1 addr [v2:172.21.15.151:6832/1898596562,v1:172.21.15.151:6833/1898596562] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T17:48:20.370 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.k{-1:24452} state up:standby seq 1 addr [v2:172.21.15.71:6832/3145454285,v1:172.21.15.71:6833/3145454285] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T17:48:20.370 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.d{-1:24488} state up:standby seq 1 addr [v2:172.21.15.45:6836/1262971887,v1:172.21.15.45:6837/1262971887] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T17:48:20.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.369+0000 7f4999800700 1 -- 172.21.15.45:0/4264711908 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f49780a2910 msgr2=0x7f49780a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:20.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.369+0000 7f4999800700 1 --2- 172.21.15.45:0/4264711908 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f49780a2910 0x7f49780a4dd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f497c0098c0 tx=0x7f497c019040 comp rx=0 tx=0).stop 2024-03-28T17:48:20.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.369+0000 7f4999800700 1 -- 172.21.15.45:0/4264711908 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4994102100 msgr2=0x7f4994198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:20.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.369+0000 7f4999800700 1 --2- 172.21.15.45:0/4264711908 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4994102100 0x7f4994198390 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4984009f40 tx=0x7f498400c450 comp rx=0 tx=0).stop 2024-03-28T17:48:20.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.369+0000 7f4999800700 1 -- 172.21.15.45:0/4264711908 shutdown_connections 2024-03-28T17:48:20.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.369+0000 7f4999800700 1 --2- 172.21.15.45:0/4264711908 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f49780a2910 0x7f49780a4dd0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:20.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.369+0000 7f4999800700 1 --2- 172.21.15.45:0/4264711908 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4994102100 0x7f4994198390 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:20.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.369+0000 7f4999800700 1 --2- 172.21.15.45:0/4264711908 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4994103ca0 0x7f499419f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:20.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.369+0000 7f4999800700 1 --2- 172.21.15.45:0/4264711908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4994103300 0x7f49941988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:20.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.369+0000 7f4999800700 1 -- 172.21.15.45:0/4264711908 >> 172.21.15.45:0/4264711908 conn(0x7f49940fd6a0 msgr2=0x7f4994106680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:20.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.369+0000 7f4999800700 1 -- 172.21.15.45:0/4264711908 shutdown_connections 2024-03-28T17:48:20.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:20.369+0000 7f4999800700 1 -- 172.21.15.45:0/4264711908 wait complete. 2024-03-28T17:48:20.372 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 22 2024-03-28T17:48:20.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:20 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/4264711908' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T17:48:20.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:20 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/4264711908' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T17:48:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:20 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/4264711908' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T17:48:21.209 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-28T17:48:21.692 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:48:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:21 smithi071 ceph-mon[36597]: pgmap v383: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T17:48:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:21 smithi151 ceph-mon[36481]: pgmap v383: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T17:48:21.844 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:21 smithi045 ceph-mon[31773]: pgmap v383: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T17:48:23.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:23 smithi045 ceph-mon[31773]: pgmap v384: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:48:23.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.794+0000 7ff6aa200700 1 -- 172.21.15.45:0/2304120639 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6a4101520 msgr2=0x7ff6a4101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:23.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.794+0000 7ff6aa200700 1 --2- 172.21.15.45:0/2304120639 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6a4101520 0x7ff6a4101980 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff6940098f0 tx=0x7ff694009c00 comp rx=0 tx=0).stop 2024-03-28T17:48:23.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.794+0000 7ff6aa200700 1 -- 172.21.15.45:0/2304120639 shutdown_connections 2024-03-28T17:48:23.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.794+0000 7ff6aa200700 1 --2- 172.21.15.45:0/2304120639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6a4101ec0 0x7ff6a4108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:23.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.794+0000 7ff6aa200700 1 --2- 172.21.15.45:0/2304120639 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6a4100320 0x7ff6a4100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:23.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.794+0000 7ff6aa200700 1 --2- 172.21.15.45:0/2304120639 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6a4101520 0x7ff6a4101980 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:23.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.794+0000 7ff6aa200700 1 -- 172.21.15.45:0/2304120639 >> 172.21.15.45:0/2304120639 conn(0x7ff6a40fb900 msgr2=0x7ff6a40fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:23.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.794+0000 7ff6aa200700 1 -- 172.21.15.45:0/2304120639 shutdown_connections 2024-03-28T17:48:23.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.794+0000 7ff6aa200700 1 -- 172.21.15.45:0/2304120639 wait complete. 2024-03-28T17:48:23.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.795+0000 7ff6aa200700 1 Processor -- start 2024-03-28T17:48:23.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.795+0000 7ff6aa200700 1 -- start start 2024-03-28T17:48:23.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.796+0000 7ff6aa200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6a4100320 0x7ff6a41042b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:23.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.796+0000 7ff6aa200700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6a4101520 0x7ff6a4102900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:23.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.796+0000 7ff6aa200700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6a4101ec0 0x7ff6a4102e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:23.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.796+0000 7ff6aa200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a410cbf0 con 0x7ff6a4100320 2024-03-28T17:48:23.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.796+0000 7ff6aa200700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a410ca70 con 0x7ff6a4101ec0 2024-03-28T17:48:23.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.796+0000 7ff6aa200700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a410cd70 con 0x7ff6a4101520 2024-03-28T17:48:23.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.796+0000 7ff6a8e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6a4100320 0x7ff6a41042b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:23.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.796+0000 7ff69be00700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6a4101520 0x7ff6a4102900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:23.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.796+0000 7ff6a8e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6a4100320 0x7ff6a41042b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45742/0 (socket says 172.21.15.45:45742) 2024-03-28T17:48:23.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.796+0000 7ff6a8e00700 1 -- 172.21.15.45:0/1945814191 learned_addr learned my addr 172.21.15.45:0/1945814191 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:23.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.796+0000 7ff6a9800700 1 --2- 172.21.15.45:0/1945814191 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6a4101ec0 0x7ff6a4102e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:23.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.797+0000 7ff69be00700 1 -- 172.21.15.45:0/1945814191 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6a4101ec0 msgr2=0x7ff6a4102e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:23.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.797+0000 7ff69be00700 1 --2- 172.21.15.45:0/1945814191 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6a4101ec0 0x7ff6a4102e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:23.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.797+0000 7ff69be00700 1 -- 172.21.15.45:0/1945814191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6a4100320 msgr2=0x7ff6a41042b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:23.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.797+0000 7ff69be00700 1 --2- 172.21.15.45:0/1945814191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6a4100320 0x7ff6a41042b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:23.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.797+0000 7ff69be00700 1 -- 172.21.15.45:0/1945814191 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a41036f0 con 0x7ff6a4101520 2024-03-28T17:48:23.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.797+0000 7ff69be00700 1 --2- 172.21.15.45:0/1945814191 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6a4101520 0x7ff6a4102900 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff694009230 tx=0x7ff694005250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:23.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.797+0000 7ff6a2000700 1 -- 172.21.15.45:0/1945814191 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69400f950 con 0x7ff6a4101520 2024-03-28T17:48:23.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.797+0000 7ff6a2000700 1 -- 172.21.15.45:0/1945814191 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff69400ba40 con 0x7ff6a4101520 2024-03-28T17:48:23.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.797+0000 7ff6a2000700 1 -- 172.21.15.45:0/1945814191 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff694003b80 con 0x7ff6a4101520 2024-03-28T17:48:23.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.797+0000 7ff6aa200700 1 -- 172.21.15.45:0/1945814191 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6a41a2430 con 0x7ff6a4101520 2024-03-28T17:48:23.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.797+0000 7ff6aa200700 1 -- 172.21.15.45:0/1945814191 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6a41a2830 con 0x7ff6a4101520 2024-03-28T17:48:23.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.798+0000 7ff6aa200700 1 -- 172.21.15.45:0/1945814191 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6a4068e50 con 0x7ff6a4101520 2024-03-28T17:48:23.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.800+0000 7ff6a2000700 1 -- 172.21.15.45:0/1945814191 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff69400bbf0 con 0x7ff6a4101520 2024-03-28T17:48:23.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.801+0000 7ff6a2000700 1 --2- 172.21.15.45:0/1945814191 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff6840a2ab0 0x7ff6840a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:23.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.802+0000 7ff6a2000700 1 -- 172.21.15.45:0/1945814191 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff6940cf470 con 0x7ff6a4101520 2024-03-28T17:48:23.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.802+0000 7ff6a8e00700 1 --2- 172.21.15.45:0/1945814191 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff6840a2ab0 0x7ff6840a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:23.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.803+0000 7ff6a8e00700 1 --2- 172.21.15.45:0/1945814191 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff6840a2ab0 0x7ff6840a4f70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff68c009cf0 tx=0x7ff68c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:23.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:23.805+0000 7ff6a2000700 1 -- 172.21.15.45:0/1945814191 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff694013030 con 0x7ff6a4101520 2024-03-28T17:48:23.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:23 smithi071 ceph-mon[36597]: pgmap v384: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:48:23.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:23 smithi151 ceph-mon[36481]: pgmap v384: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:48:24.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:24.035+0000 7ff6aa200700 1 -- 172.21.15.45:0/1945814191 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7ff6a41a3140 con 0x7ff6a4101520 2024-03-28T17:48:24.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:24.036+0000 7ff6a2000700 1 -- 172.21.15.45:0/1945814191 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v87) v1 ==== 56+0+5278 (secure 0 0 0) 0x7ff69409c550 con 0x7ff6a4101520 2024-03-28T17:48:24.037 INFO:teuthology.orchestra.run.smithi045.stdout:epoch 87 2024-03-28T17:48:24.038 INFO:teuthology.orchestra.run.smithi045.stdout:fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 2024-03-28T17:48:24.038 INFO:teuthology.orchestra.run.smithi045.stdout:created 2024-03-28T17:36:51.623185+0000 2024-03-28T17:48:24.038 INFO:teuthology.orchestra.run.smithi045.stdout:modified 2024-03-28T17:47:09.455843+0000 2024-03-28T17:48:24.038 INFO:teuthology.orchestra.run.smithi045.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-28T17:48:24.038 INFO:teuthology.orchestra.run.smithi045.stdout:crush_version 26 2024-03-28T17:48:24.038 INFO:teuthology.orchestra.run.smithi045.stdout:full_ratio 0.95 2024-03-28T17:48:24.038 INFO:teuthology.orchestra.run.smithi045.stdout:backfillfull_ratio 0.9 2024-03-28T17:48:24.038 INFO:teuthology.orchestra.run.smithi045.stdout:nearfull_ratio 0.85 2024-03-28T17:48:24.038 INFO:teuthology.orchestra.run.smithi045.stdout:require_min_compat_client luminous 2024-03-28T17:48:24.038 INFO:teuthology.orchestra.run.smithi045.stdout:min_compat_client luminous 2024-03-28T17:48:24.038 INFO:teuthology.orchestra.run.smithi045.stdout:require_osd_release squid 2024-03-28T17:48:24.038 INFO:teuthology.orchestra.run.smithi045.stdout:stretch_mode_enabled false 2024-03-28T17:48:24.039 INFO:teuthology.orchestra.run.smithi045.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-28T17:48:24.039 INFO:teuthology.orchestra.run.smithi045.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-03-28T17:48:24.039 INFO:teuthology.orchestra.run.smithi045.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-03-28T17:48:24.039 INFO:teuthology.orchestra.run.smithi045.stdout:max_osd 12 2024-03-28T17:48:24.039 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 up in weight 1 up_from 8 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6802/3795754023,v1:172.21.15.45:6803/3795754023] [v2:172.21.15.45:6804/3795754023,v1:172.21.15.45:6805/3795754023] exists,up 74c56bc2-4169-4919-afd4-2ebd236f23e7 2024-03-28T17:48:24.039 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 up in weight 1 up_from 13 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6800/2014529906,v1:172.21.15.71:6801/2014529906] [v2:172.21.15.71:6802/2014529906,v1:172.21.15.71:6803/2014529906] exists,up 78dcc21b-75b2-453d-b690-17b55b8ed1a0 2024-03-28T17:48:24.039 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 up in weight 1 up_from 18 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6800/1850933775,v1:172.21.15.151:6801/1850933775] [v2:172.21.15.151:6802/1850933775,v1:172.21.15.151:6803/1850933775] exists,up ca9f3f36-b71e-4962-a94f-cedf2f12eb7f 2024-03-28T17:48:24.039 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 up in weight 1 up_from 25 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6810/3021569975,v1:172.21.15.45:6811/3021569975] [v2:172.21.15.45:6812/3021569975,v1:172.21.15.45:6813/3021569975] exists,up d3f21f01-2973-413e-8daa-9974b8a44af5 2024-03-28T17:48:24.039 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 up in weight 1 up_from 33 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6808/2069578333,v1:172.21.15.71:6809/2069578333] [v2:172.21.15.71:6810/2069578333,v1:172.21.15.71:6811/2069578333] exists,up 4a336f42-0605-492b-ad88-56cb02acdcf8 2024-03-28T17:48:24.039 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 up in weight 1 up_from 38 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6808/2629217227,v1:172.21.15.151:6809/2629217227] [v2:172.21.15.151:6810/2629217227,v1:172.21.15.151:6811/2629217227] exists,up c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4 2024-03-28T17:48:24.039 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 up in weight 1 up_from 46 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6818/2077907039,v1:172.21.15.45:6819/2077907039] [v2:172.21.15.45:6820/2077907039,v1:172.21.15.45:6821/2077907039] exists,up 58a6c5bd-a8b7-4892-b346-bcd7994bcb6b 2024-03-28T17:48:24.039 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 up in weight 1 up_from 51 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6816/2000211251,v1:172.21.15.71:6817/2000211251] [v2:172.21.15.71:6818/2000211251,v1:172.21.15.71:6819/2000211251] exists,up f7a45d52-807b-4407-bb04-787b2a52a0ac 2024-03-28T17:48:24.039 INFO:teuthology.orchestra.run.smithi045.stdout:osd.8 up in weight 1 up_from 59 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6816/3665460137,v1:172.21.15.151:6817/3665460137] [v2:172.21.15.151:6818/3665460137,v1:172.21.15.151:6819/3665460137] exists,up 88abcb0f-f189-49be-8315-c700c77369f3 2024-03-28T17:48:24.039 INFO:teuthology.orchestra.run.smithi045.stdout:osd.9 up in weight 1 up_from 67 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6826/2122322684,v1:172.21.15.45:6827/2122322684] [v2:172.21.15.45:6828/2122322684,v1:172.21.15.45:6829/2122322684] exists,up ed3fc096-961e-4392-abc2-c5aa4d050117 2024-03-28T17:48:24.040 INFO:teuthology.orchestra.run.smithi045.stdout:osd.10 up in weight 1 up_from 72 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6824/1099494037,v1:172.21.15.71:6825/1099494037] [v2:172.21.15.71:6826/1099494037,v1:172.21.15.71:6827/1099494037] exists,up ba94762b-bc5c-4c97-9cc1-17e5651c14b9 2024-03-28T17:48:24.040 INFO:teuthology.orchestra.run.smithi045.stdout:osd.11 up in weight 1 up_from 80 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6824/2941055364,v1:172.21.15.151:6825/2941055364] [v2:172.21.15.151:6826/2941055364,v1:172.21.15.151:6827/2941055364] exists,up 1d178378-b9f5-40f6-99ef-3d3d445da640 2024-03-28T17:48:24.040 INFO:teuthology.orchestra.run.smithi045.stdout:pg_upmap_items 2.4 [2,11] 2024-03-28T17:48:24.040 INFO:teuthology.orchestra.run.smithi045.stdout:pg_upmap_items 2.6 [8,11] 2024-03-28T17:48:24.040 INFO:teuthology.orchestra.run.smithi045.stdout:pg_upmap_items 2.1d [2,4] 2024-03-28T17:48:24.040 INFO:teuthology.orchestra.run.smithi045.stdout:pg_upmap_items 3.0 [5,9] 2024-03-28T17:48:24.040 INFO:teuthology.orchestra.run.smithi045.stdout:pg_upmap_items 3.6 [4,6] 2024-03-28T17:48:24.040 INFO:teuthology.orchestra.run.smithi045.stdout:pg_upmap_items 3.c [3,9] 2024-03-28T17:48:24.040 INFO:teuthology.orchestra.run.smithi045.stdout:pg_upmap_items 3.17 [4,8] 2024-03-28T17:48:24.040 INFO:teuthology.orchestra.run.smithi045.stdout:pg_upmap_items 3.24 [4,9] 2024-03-28T17:48:24.040 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:0/3533611957 expires 2024-03-29T17:38:07.505611+0000 2024-03-28T17:48:24.040 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:0/3625922430 expires 2024-03-29T17:38:07.505611+0000 2024-03-28T17:48:24.040 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:6801/3951447106 expires 2024-03-29T17:38:07.505611+0000 2024-03-28T17:48:24.040 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:6800/3951447106 expires 2024-03-29T17:38:07.505611+0000 2024-03-28T17:48:24.041 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:6800/4186185610 expires 2024-03-29T17:37:32.424229+0000 2024-03-28T17:48:24.041 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:0/482610468 expires 2024-03-29T17:37:32.424229+0000 2024-03-28T17:48:24.041 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:6801/4186185610 expires 2024-03-29T17:37:32.424229+0000 2024-03-28T17:48:24.041 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:0/3649804714 expires 2024-03-29T17:37:32.424229+0000 2024-03-28T17:48:24.041 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:0/1198364079 expires 2024-03-29T17:37:32.424229+0000 2024-03-28T17:48:24.041 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:0/3211303958 expires 2024-03-29T17:38:07.505611+0000 2024-03-28T17:48:24.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:24.039+0000 7ff6aa200700 1 -- 172.21.15.45:0/1945814191 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff6840a2ab0 msgr2=0x7ff6840a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:24.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:24.039+0000 7ff6aa200700 1 --2- 172.21.15.45:0/1945814191 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff6840a2ab0 0x7ff6840a4f70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff68c009cf0 tx=0x7ff68c009310 comp rx=0 tx=0).stop 2024-03-28T17:48:24.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:24.039+0000 7ff6aa200700 1 -- 172.21.15.45:0/1945814191 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6a4101520 msgr2=0x7ff6a4102900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:24.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:24.039+0000 7ff6aa200700 1 --2- 172.21.15.45:0/1945814191 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6a4101520 0x7ff6a4102900 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff694009230 tx=0x7ff694005250 comp rx=0 tx=0).stop 2024-03-28T17:48:24.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:24.039+0000 7ff6aa200700 1 -- 172.21.15.45:0/1945814191 shutdown_connections 2024-03-28T17:48:24.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:24.039+0000 7ff6aa200700 1 --2- 172.21.15.45:0/1945814191 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff6840a2ab0 0x7ff6840a4f70 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:24.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:24.039+0000 7ff6aa200700 1 --2- 172.21.15.45:0/1945814191 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6a4101ec0 0x7ff6a4102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:24.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:24.039+0000 7ff6aa200700 1 --2- 172.21.15.45:0/1945814191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6a4100320 0x7ff6a41042b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:24.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:24.039+0000 7ff6aa200700 1 --2- 172.21.15.45:0/1945814191 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6a4101520 0x7ff6a4102900 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:24.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:24.039+0000 7ff6aa200700 1 -- 172.21.15.45:0/1945814191 >> 172.21.15.45:0/1945814191 conn(0x7ff6a40fb900 msgr2=0x7ff6a40fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:24.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:24.040+0000 7ff6aa200700 1 -- 172.21.15.45:0/1945814191 shutdown_connections 2024-03-28T17:48:24.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:24.040+0000 7ff6aa200700 1 -- 172.21.15.45:0/1945814191 wait complete. 2024-03-28T17:48:24.789 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:24 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1945814191' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T17:48:24.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:24 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1945814191' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T17:48:24.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:24 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1945814191' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T17:48:24.849 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-28T17:48:24.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T17:48:25.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.092+0000 7f3f38400640 1 -- 172.21.15.45:0/412970879 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f30104ad0 msgr2=0x7f3f30104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:25.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.092+0000 7f3f38400640 1 --2- 172.21.15.45:0/412970879 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f30104ad0 0x7f3f30104f50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c009900 tx=0x7f3f1c02bc40 comp rx=0 tx=0).stop 2024-03-28T17:48:25.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.092+0000 7f3f38400640 1 -- 172.21.15.45:0/412970879 shutdown_connections 2024-03-28T17:48:25.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.092+0000 7f3f38400640 1 --2- 172.21.15.45:0/412970879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f30105490 0x7f3f3010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:25.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.092+0000 7f3f38400640 1 --2- 172.21.15.45:0/412970879 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f30104ad0 0x7f3f30104f50 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:25.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.092+0000 7f3f38400640 1 --2- 172.21.15.45:0/412970879 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f301038e0 0x7f3f30103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:25.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.092+0000 7f3f38400640 1 -- 172.21.15.45:0/412970879 >> 172.21.15.45:0/412970879 conn(0x7f3f300fee90 msgr2=0x7f3f301012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:25.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.092+0000 7f3f38400640 1 -- 172.21.15.45:0/412970879 shutdown_connections 2024-03-28T17:48:25.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.092+0000 7f3f38400640 1 -- 172.21.15.45:0/412970879 wait complete. 2024-03-28T17:48:25.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.093+0000 7f3f38400640 1 Processor -- start 2024-03-28T17:48:25.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.093+0000 7f3f38400640 1 -- start start 2024-03-28T17:48:25.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.094+0000 7f3f38400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f301038e0 0x7f3f30199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:25.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.094+0000 7f3f38400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f30104ad0 0x7f3f3019a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:25.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.094+0000 7f3f38400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f30105490 0x7f3f301a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:25.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.094+0000 7f3f38400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3010f590 con 0x7f3f301038e0 2024-03-28T17:48:25.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.094+0000 7f3f38400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3010f410 con 0x7f3f30104ad0 2024-03-28T17:48:25.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.094+0000 7f3f38400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3010f710 con 0x7f3f30105490 2024-03-28T17:48:25.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.094+0000 7f3f35000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f30104ad0 0x7f3f3019a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:25.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.094+0000 7f3f35a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f301038e0 0x7f3f30199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:25.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.094+0000 7f3f36400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f30105490 0x7f3f301a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:25.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.094+0000 7f3f35000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f30104ad0 0x7f3f3019a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:60114/0 (socket says 172.21.15.45:60114) 2024-03-28T17:48:25.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.094+0000 7f3f35000640 1 -- 172.21.15.45:0/1726014162 learned_addr learned my addr 172.21.15.45:0/1726014162 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:25.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.095+0000 7f3f36400640 1 -- 172.21.15.45:0/1726014162 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f30104ad0 msgr2=0x7f3f3019a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:25.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.095+0000 7f3f36400640 1 --2- 172.21.15.45:0/1726014162 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f30104ad0 0x7f3f3019a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:25.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.095+0000 7f3f36400640 1 -- 172.21.15.45:0/1726014162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f301038e0 msgr2=0x7f3f30199b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:25.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.095+0000 7f3f36400640 1 --2- 172.21.15.45:0/1726014162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f301038e0 0x7f3f30199b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:25.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.095+0000 7f3f36400640 1 -- 172.21.15.45:0/1726014162 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f301a1770 con 0x7f3f30105490 2024-03-28T17:48:25.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.095+0000 7f3f35000640 1 --2- 172.21.15.45:0/1726014162 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f30104ad0 0x7f3f3019a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:25.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.095+0000 7f3f36400640 1 --2- 172.21.15.45:0/1726014162 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f30105490 0x7f3f301a1100 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3f2800cab0 tx=0x7f3f2800cf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:25.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.095+0000 7f3f2e000640 1 -- 172.21.15.45:0/1726014162 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2800ae20 con 0x7f3f30105490 2024-03-28T17:48:25.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.095+0000 7f3f2e000640 1 -- 172.21.15.45:0/1726014162 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f2800f930 con 0x7f3f30105490 2024-03-28T17:48:25.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.095+0000 7f3f38400640 1 -- 172.21.15.45:0/1726014162 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f301a1a60 con 0x7f3f30105490 2024-03-28T17:48:25.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.095+0000 7f3f2e000640 1 -- 172.21.15.45:0/1726014162 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f28018b30 con 0x7f3f30105490 2024-03-28T17:48:25.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.095+0000 7f3f38400640 1 -- 172.21.15.45:0/1726014162 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f301a1f80 con 0x7f3f30105490 2024-03-28T17:48:25.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.096+0000 7f3f38400640 1 -- 172.21.15.45:0/1726014162 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f3010ab30 con 0x7f3f30105490 2024-03-28T17:48:25.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.098+0000 7f3f2e000640 1 -- 172.21.15.45:0/1726014162 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3f2800fae0 con 0x7f3f30105490 2024-03-28T17:48:25.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.100+0000 7f3f2e000640 1 --2- 172.21.15.45:0/1726014162 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3f180a2880 0x7f3f180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:25.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.100+0000 7f3f2e000640 1 -- 172.21.15.45:0/1726014162 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3f280cf100 con 0x7f3f30105490 2024-03-28T17:48:25.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.100+0000 7f3f35a00640 1 --2- 172.21.15.45:0/1726014162 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3f180a2880 0x7f3f180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:25.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.100+0000 7f3f35a00640 1 --2- 172.21.15.45:0/1726014162 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3f180a2880 0x7f3f180a4d40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3f30104930 tx=0x7f3f2400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:25.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.104+0000 7f3f2e000640 1 -- 172.21.15.45:0/1726014162 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3f28014030 con 0x7f3f30105490 2024-03-28T17:48:25.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.309+0000 7f3f38400640 1 -- 172.21.15.45:0/1726014162 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f30062340 con 0x7f3f30105490 2024-03-28T17:48:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.311+0000 7f3f2e000640 1 -- 172.21.15.45:0/1726014162 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22583 (secure 0 0 0) 0x7f3f2809c150 con 0x7f3f30105490 2024-03-28T17:48:25.312 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:48:25.312 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"9f4888ec-ed29-11ee-95d0-87774f69a715","created":"2024-03-28T17:36:51.623185+0000","modified":"2024-03-28T17:47:09.455843+0000","last_up_change":"2024-03-28T17:45:14.043432+0000","last_in_change":"2024-03-28T17:44:53.654489+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T17:40:54.208375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T17:46:10.564519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T17:46:11.734367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"74c56bc2-4169-4919-afd4-2ebd236f23e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6803","nonce":3795754023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6805","nonce":3795754023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6809","nonce":3795754023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6807","nonce":3795754023}]},"public_addr":"172.21.15.45:6803/3795754023","cluster_addr":"172.21.15.45:6805/3795754023","heartbeat_back_addr":"172.21.15.45:6809/3795754023","heartbeat_front_addr":"172.21.15.45:6807/3795754023","state":["exists","up"]},{"osd":1,"uuid":"78dcc21b-75b2-453d-b690-17b55b8ed1a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6801","nonce":2014529906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6803","nonce":2014529906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6807","nonce":2014529906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6805","nonce":2014529906}]},"public_addr":"172.21.15.71:6801/2014529906","cluster_addr":"172.21.15.71:6803/2014529906","heartbeat_back_addr":"172.21.15.71:6807/2014529906","heartbeat_front_addr":"172.21.15.71:6805/2014529906","state":["exists","up"]},{"osd":2,"uuid":"ca9f3f36-b71e-4962-a94f-cedf2f12eb7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6801","nonce":1850933775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6803","nonce":1850933775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6807","nonce":1850933775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6805","nonce":1850933775}]},"public_addr":"172.21.15.151:6801/1850933775","cluster_addr":"172.21.15.151:6803/1850933775","heartbeat_back_addr":"172.21.15.151:6807/1850933775","heartbeat_front_addr":"172.21.15.151:6805/1850933775","state":["exists","up"]},{"osd":3,"uuid":"d3f21f01-2973-413e-8daa-9974b8a44af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6811","nonce":3021569975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6813","nonce":3021569975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6817","nonce":3021569975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6815","nonce":3021569975}]},"public_addr":"172.21.15.45:6811/3021569975","cluster_addr":"172.21.15.45:6813/3021569975","heartbeat_back_addr":"172.21.15.45:6817/3021569975","heartbeat_front_addr":"172.21.15.45:6815/3021569975","state":["exists","up"]},{"osd":4,"uuid":"4a336f42-0605-492b-ad88-56cb02acdcf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6809","nonce":2069578333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6811","nonce":2069578333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6815","nonce":2069578333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6813","nonce":2069578333}]},"public_addr":"172.21.15.71:6809/2069578333","cluster_addr":"172.21.15.71:6811/2069578333","heartbeat_back_addr":"172.21.15.71:6815/2069578333","heartbeat_front_addr":"172.21.15.71:6813/2069578333","state":["exists","up"]},{"osd":5,"uuid":"c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6809","nonce":2629217227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6811","nonce":2629217227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6815","nonce":2629217227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6813","nonce":2629217227}]},"public_addr":"172.21.15.151:6809/2629217227","cluster_addr":"172.21.15.151:6811/2629217227","heartbeat_back_addr":"172.21.15.151:6815/2629217227","heartbeat_front_addr":"172.21.15.151:6813/2629217227","state":["exists","up"]},{"osd":6,"uuid":"58a6c5bd-a8b7-4892-b346-bcd7994bcb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6819","nonce":2077907039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6821","nonce":2077907039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6825","nonce":2077907039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6823","nonce":2077907039}]},"public_addr":"172.21.15.45:6819/2077907039","cluster_addr":"172.21.15.45:6821/2077907039","heartbeat_back_addr":"172.21.15.45:6825/2077907039","heartbeat_front_addr":"172.21.15.45:6823/2077907039","state":["exists","up"]},{"osd":7,"uuid":"f7a45d52-807b-4407-bb04-787b2a52a0ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6817","nonce":2000211251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6819","nonce":2000211251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6823","nonce":2000211251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6821","nonce":2000211251}]},"public_addr":"172.21.15.71:6817/2000211251","cluster_addr":"172.21.15.71:6819/2000211251","heartbeat_back_addr":"172.21.15.71:6823/2000211251","heartbeat_front_addr":"172.21.15.71:6821/2000211251","state":["exists","up"]},{"osd":8,"uuid":"88abcb0f-f189-49be-8315-c700c77369f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6817","nonce":3665460137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6819","nonce":3665460137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6823","nonce":3665460137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6821","nonce":3665460137}]},"public_addr":"172.21.15.151:6817/3665460137","cluster_addr":"172.21.15.151:6819/3665460137","heartbeat_back_addr":"172.21.15.151:6823/3665460137","heartbeat_front_addr":"172.21.15.151:6821/3665460137","state":["exists","up"]},{"osd":9,"uuid":"ed3fc096-961e-4392-abc2-c5aa4d050117","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6827","nonce":2122322684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6829","nonce":2122322684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6833","nonce":2122322684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6831","nonce":2122322684}]},"public_addr":"172.21.15.45:6827/2122322684","cluster_addr":"172.21.15.45:6829/2122322684","heartbeat_back_addr":"172.21.15.45:6833/2122322684","heartbeat_front_addr":"172.21.15.45:6831/2122322684","state":["exists","up"]},{"osd":10,"uuid":"ba94762b-bc5c-4c97-9cc1-17e5651c14b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6825","nonce":1099494037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6827","nonce":1099494037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6831","nonce":1099494037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6829","nonce":1099494037}]},"public_addr":"172.21.15.71:6825/1099494037","cluster_addr":"172.21.15.71:6827/1099494037","heartbeat_back_addr":"172.21.15.71:6831/1099494037","heartbeat_front_addr":"172.21.15.71:6829/1099494037","state":["exists","up"]},{"osd":11,"uuid":"1d178378-b9f5-40f6-99ef-3d3d445da640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6825","nonce":2941055364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6827","nonce":2941055364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6831","nonce":2941055364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6829","nonce":2941055364}]},"public_addr":"172.21.15.151:6825/2941055364","cluster_addr":"172.21.15.151:6827/2941055364","heartbeat_back_addr":"172.21.15.151:6831/2941055364","heartbeat_front_addr":"172.21.15.151:6829/2941055364","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:02.269586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:27.819386+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:51.552277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:25.052674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:51.873529+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:17.127867+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:51.588249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:18.381450+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:44.140534+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:17.634362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:45.469340+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:45:12.003433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3533611957":"2024-03-29T17:38:07.505611+0000","172.21.15.45:0/3625922430":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6801/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/482610468":"2024-03-29T17:37:32.424229+0000","172.21.15.45:6801/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3649804714":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/1198364079":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3211303958":"2024-03-29T17:38:07.505611+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T17:48:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.313+0000 7f3f38400640 1 -- 172.21.15.45:0/1726014162 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3f180a2880 msgr2=0x7f3f180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.313+0000 7f3f38400640 1 --2- 172.21.15.45:0/1726014162 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3f180a2880 0x7f3f180a4d40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3f30104930 tx=0x7f3f2400b390 comp rx=0 tx=0).stop 2024-03-28T17:48:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.313+0000 7f3f38400640 1 -- 172.21.15.45:0/1726014162 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f30105490 msgr2=0x7f3f301a1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.313+0000 7f3f38400640 1 --2- 172.21.15.45:0/1726014162 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f30105490 0x7f3f301a1100 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3f2800cab0 tx=0x7f3f2800cf70 comp rx=0 tx=0).stop 2024-03-28T17:48:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.314+0000 7f3f38400640 1 -- 172.21.15.45:0/1726014162 shutdown_connections 2024-03-28T17:48:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.314+0000 7f3f38400640 1 --2- 172.21.15.45:0/1726014162 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3f180a2880 0x7f3f180a4d40 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.314+0000 7f3f38400640 1 --2- 172.21.15.45:0/1726014162 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f30105490 0x7f3f301a1100 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.314+0000 7f3f38400640 1 --2- 172.21.15.45:0/1726014162 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f30104ad0 0x7f3f3019a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.314+0000 7f3f38400640 1 --2- 172.21.15.45:0/1726014162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f301038e0 0x7f3f30199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.314+0000 7f3f38400640 1 -- 172.21.15.45:0/1726014162 >> 172.21.15.45:0/1726014162 conn(0x7f3f300fee90 msgr2=0x7f3f30107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:25.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.314+0000 7f3f38400640 1 -- 172.21.15.45:0/1726014162 shutdown_connections 2024-03-28T17:48:25.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.314+0000 7f3f38400640 1 -- 172.21.15.45:0/1726014162 wait complete. 2024-03-28T17:48:25.333 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T17:40:54.208375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T17:46:10.564519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T17:46:11.734367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T17:48:25.333 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T17:48:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.564+0000 7fc2a9000640 1 -- 172.21.15.45:0/2917451620 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc2a4100e00 msgr2=0x7fc2a4101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.564+0000 7fc2a9000640 1 --2- 172.21.15.45:0/2917451620 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc2a4100e00 0x7fc2a4101280 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc28c009900 tx=0x7fc28c02bc30 comp rx=0 tx=0).stop 2024-03-28T17:48:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.565+0000 7fc2a9000640 1 -- 172.21.15.45:0/2917451620 shutdown_connections 2024-03-28T17:48:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.565+0000 7fc2a9000640 1 --2- 172.21.15.45:0/2917451620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2a41017c0 0x7fc2a4108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.565+0000 7fc2a9000640 1 --2- 172.21.15.45:0/2917451620 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc2a4100e00 0x7fc2a4101280 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.565+0000 7fc2a9000640 1 --2- 172.21.15.45:0/2917451620 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2a40ffc10 0x7fc2a4100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.565+0000 7fc2a9000640 1 -- 172.21.15.45:0/2917451620 >> 172.21.15.45:0/2917451620 conn(0x7fc2a40fb180 msgr2=0x7fc2a40fd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.565+0000 7fc2a9000640 1 -- 172.21.15.45:0/2917451620 shutdown_connections 2024-03-28T17:48:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.565+0000 7fc2a9000640 1 -- 172.21.15.45:0/2917451620 wait complete. 2024-03-28T17:48:25.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.566+0000 7fc2a9000640 1 Processor -- start 2024-03-28T17:48:25.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.566+0000 7fc2a9000640 1 -- start start 2024-03-28T17:48:25.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.566+0000 7fc2a9000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2a40ffc10 0x7fc2a4199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:25.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.566+0000 7fc2a9000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2a4100e00 0x7fc2a419a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:25.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.566+0000 7fc2a9000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc2a41017c0 0x7fc2a41a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:25.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.566+0000 7fc2a9000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a410b070 con 0x7fc2a4100e00 2024-03-28T17:48:25.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.566+0000 7fc2a9000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a410aef0 con 0x7fc2a40ffc10 2024-03-28T17:48:25.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.566+0000 7fc2a9000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a410b1f0 con 0x7fc2a41017c0 2024-03-28T17:48:25.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.567+0000 7fc2a3400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2a40ffc10 0x7fc2a4199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:25.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.567+0000 7fc2a3400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2a40ffc10 0x7fc2a4199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:60124/0 (socket says 172.21.15.45:60124) 2024-03-28T17:48:25.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.567+0000 7fc2a3400640 1 -- 172.21.15.45:0/2783354323 learned_addr learned my addr 172.21.15.45:0/2783354323 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:25.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.567+0000 7fc2a2a00640 1 --2- 172.21.15.45:0/2783354323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2a4100e00 0x7fc2a419a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:25.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.567+0000 7fc2a3e00640 1 --2- 172.21.15.45:0/2783354323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc2a41017c0 0x7fc2a41a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:25.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.567+0000 7fc2a3400640 1 -- 172.21.15.45:0/2783354323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc2a41017c0 msgr2=0x7fc2a41a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:25.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.567+0000 7fc2a3400640 1 --2- 172.21.15.45:0/2783354323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc2a41017c0 0x7fc2a41a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:25.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.567+0000 7fc2a3400640 1 -- 172.21.15.45:0/2783354323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2a4100e00 msgr2=0x7fc2a419a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:25.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.567+0000 7fc2a3400640 1 --2- 172.21.15.45:0/2783354323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2a4100e00 0x7fc2a419a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:25.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.567+0000 7fc2a3400640 1 -- 172.21.15.45:0/2783354323 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a41a1770 con 0x7fc2a40ffc10 2024-03-28T17:48:25.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.567+0000 7fc2a3400640 1 --2- 172.21.15.45:0/2783354323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2a40ffc10 0x7fc2a4199b50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc29400ebc0 tx=0x7fc29400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:25.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.568+0000 7fc29be00640 1 -- 172.21.15.45:0/2783354323 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc294003b70 con 0x7fc2a40ffc10 2024-03-28T17:48:25.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.568+0000 7fc29be00640 1 -- 172.21.15.45:0/2783354323 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc294003d10 con 0x7fc2a40ffc10 2024-03-28T17:48:25.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.568+0000 7fc29be00640 1 -- 172.21.15.45:0/2783354323 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc294019980 con 0x7fc2a40ffc10 2024-03-28T17:48:25.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.568+0000 7fc2a9000640 1 -- 172.21.15.45:0/2783354323 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a41a1a60 con 0x7fc2a40ffc10 2024-03-28T17:48:25.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.568+0000 7fc2a9000640 1 -- 172.21.15.45:0/2783354323 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a41a1f30 con 0x7fc2a40ffc10 2024-03-28T17:48:25.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.569+0000 7fc2a9000640 1 -- 172.21.15.45:0/2783354323 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a4068800 con 0x7fc2a40ffc10 2024-03-28T17:48:25.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.570+0000 7fc29be00640 1 -- 172.21.15.45:0/2783354323 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc294019b20 con 0x7fc2a40ffc10 2024-03-28T17:48:25.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.572+0000 7fc29be00640 1 --2- 172.21.15.45:0/2783354323 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc2880a2880 0x7fc2880a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:25.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.572+0000 7fc29be00640 1 -- 172.21.15.45:0/2783354323 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc2940cf210 con 0x7fc2a40ffc10 2024-03-28T17:48:25.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.572+0000 7fc2a2a00640 1 --2- 172.21.15.45:0/2783354323 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc2880a2880 0x7fc2880a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:25.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.573+0000 7fc2a2a00640 1 --2- 172.21.15.45:0/2783354323 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc2880a2880 0x7fc2880a4d40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc28c000c00 tx=0x7fc28c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:25.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.576+0000 7fc29be00640 1 -- 172.21.15.45:0/2783354323 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc29409c260 con 0x7fc2a40ffc10 2024-03-28T17:48:25.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:25 smithi045 ceph-mon[31773]: pgmap v385: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:25.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:25 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1726014162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:48:25.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.781+0000 7fc2a9000640 1 -- 172.21.15.45:0/2783354323 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc2a40622a0 con 0x7fc2a40ffc10 2024-03-28T17:48:25.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.782+0000 7fc29be00640 1 -- 172.21.15.45:0/2783354323 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc29409bb60 con 0x7fc2a40ffc10 2024-03-28T17:48:25.782 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-28T17:48:25.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.785+0000 7fc2a9000640 1 -- 172.21.15.45:0/2783354323 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc2880a2880 msgr2=0x7fc2880a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:25.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.785+0000 7fc2a9000640 1 --2- 172.21.15.45:0/2783354323 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc2880a2880 0x7fc2880a4d40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc28c000c00 tx=0x7fc28c02df90 comp rx=0 tx=0).stop 2024-03-28T17:48:25.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.785+0000 7fc2a9000640 1 -- 172.21.15.45:0/2783354323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2a40ffc10 msgr2=0x7fc2a4199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:25.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.785+0000 7fc2a9000640 1 --2- 172.21.15.45:0/2783354323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2a40ffc10 0x7fc2a4199b50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc29400ebc0 tx=0x7fc29400c6a0 comp rx=0 tx=0).stop 2024-03-28T17:48:25.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.785+0000 7fc2a9000640 1 -- 172.21.15.45:0/2783354323 shutdown_connections 2024-03-28T17:48:25.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.785+0000 7fc2a9000640 1 --2- 172.21.15.45:0/2783354323 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc2880a2880 0x7fc2880a4d40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:25.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.785+0000 7fc2a9000640 1 --2- 172.21.15.45:0/2783354323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc2a41017c0 0x7fc2a41a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:25.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.785+0000 7fc2a9000640 1 --2- 172.21.15.45:0/2783354323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2a4100e00 0x7fc2a419a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:25.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.785+0000 7fc2a9000640 1 --2- 172.21.15.45:0/2783354323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2a40ffc10 0x7fc2a4199b50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:25.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.785+0000 7fc2a9000640 1 -- 172.21.15.45:0/2783354323 >> 172.21.15.45:0/2783354323 conn(0x7fc2a40fb180 msgr2=0x7fc2a41040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:25.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.785+0000 7fc2a9000640 1 -- 172.21.15.45:0/2783354323 shutdown_connections 2024-03-28T17:48:25.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:25.785+0000 7fc2a9000640 1 -- 172.21.15.45:0/2783354323 wait complete. 2024-03-28T17:48:25.805 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T17:48:25.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:25 smithi071 ceph-mon[36597]: pgmap v385: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:25.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:25 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1726014162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:48:25.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:25 smithi151 ceph-mon[36481]: pgmap v385: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:25.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:25 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1726014162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:48:26.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.023+0000 7f96c5c00640 1 -- 172.21.15.45:0/294530887 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96c0108db0 msgr2=0x7f96c0109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:26.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.023+0000 7f96c5c00640 1 --2- 172.21.15.45:0/294530887 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96c0108db0 0x7f96c0109230 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009ca0 tx=0x7f96ac02c100 comp rx=0 tx=0).stop 2024-03-28T17:48:26.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.023+0000 7f96c5c00640 1 -- 172.21.15.45:0/294530887 shutdown_connections 2024-03-28T17:48:26.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.023+0000 7f96c5c00640 1 --2- 172.21.15.45:0/294530887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96c0109770 0x7f96c0110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:26.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.023+0000 7f96c5c00640 1 --2- 172.21.15.45:0/294530887 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96c0108db0 0x7f96c0109230 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:26.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.023+0000 7f96c5c00640 1 --2- 172.21.15.45:0/294530887 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96c0107bc0 0x7f96c0107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:26.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.023+0000 7f96c5c00640 1 -- 172.21.15.45:0/294530887 >> 172.21.15.45:0/294530887 conn(0x7f96c0076b70 msgr2=0x7f96c0078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:26.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.024+0000 7f96c5c00640 1 -- 172.21.15.45:0/294530887 shutdown_connections 2024-03-28T17:48:26.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.024+0000 7f96c5c00640 1 -- 172.21.15.45:0/294530887 wait complete. 2024-03-28T17:48:26.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.025+0000 7f96c5c00640 1 Processor -- start 2024-03-28T17:48:26.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.025+0000 7f96c5c00640 1 -- start start 2024-03-28T17:48:26.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.025+0000 7f96c5c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96c0107bc0 0x7f96c019de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:26.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.025+0000 7f96c5c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96c0108db0 0x7f96c019e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:26.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.026+0000 7f96c5c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96c0109770 0x7f96c01a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:26.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.026+0000 7f96c5c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c0113840 con 0x7f96c0109770 2024-03-28T17:48:26.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.026+0000 7f96c5c00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c01136c0 con 0x7f96c0108db0 2024-03-28T17:48:26.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.026+0000 7f96c5c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c01139c0 con 0x7f96c0107bc0 2024-03-28T17:48:26.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.026+0000 7f96bfe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96c0107bc0 0x7f96c019de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:26.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.026+0000 7f96bf400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96c0108db0 0x7f96c019e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:26.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.026+0000 7f96bfe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96c0107bc0 0x7f96c019de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:41238/0 (socket says 172.21.15.45:41238) 2024-03-28T17:48:26.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.026+0000 7f96bfe00640 1 -- 172.21.15.45:0/2602687272 learned_addr learned my addr 172.21.15.45:0/2602687272 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:26.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.026+0000 7f96bf400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96c0108db0 0x7f96c019e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53410/0 (socket says 172.21.15.45:53410) 2024-03-28T17:48:26.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.026+0000 7f96c5200640 1 --2- 172.21.15.45:0/2602687272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96c0109770 0x7f96c01a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:26.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.026+0000 7f96bf400640 1 -- 172.21.15.45:0/2602687272 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96c0107bc0 msgr2=0x7f96c019de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:26.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.026+0000 7f96bf400640 1 --2- 172.21.15.45:0/2602687272 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96c0107bc0 0x7f96c019de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:26.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.026+0000 7f96bf400640 1 -- 172.21.15.45:0/2602687272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96c0109770 msgr2=0x7f96c01a53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:26.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.026+0000 7f96bf400640 1 --2- 172.21.15.45:0/2602687272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96c0109770 0x7f96c01a53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:26.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.026+0000 7f96bf400640 1 -- 172.21.15.45:0/2602687272 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c01a5a40 con 0x7f96c0108db0 2024-03-28T17:48:26.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.026+0000 7f96c5200640 1 --2- 172.21.15.45:0/2602687272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96c0109770 0x7f96c01a53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:48:26.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.026+0000 7f96bf400640 1 --2- 172.21.15.45:0/2602687272 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96c0108db0 0x7f96c019e360 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009bf0 tx=0x7f96ac035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:26.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.027+0000 7f96bcc00640 1 -- 172.21.15.45:0/2602687272 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac039040 con 0x7f96c0108db0 2024-03-28T17:48:26.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.027+0000 7f96bcc00640 1 -- 172.21.15.45:0/2602687272 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96ac003970 con 0x7f96c0108db0 2024-03-28T17:48:26.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.027+0000 7f96bcc00640 1 -- 172.21.15.45:0/2602687272 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac004360 con 0x7f96c0108db0 2024-03-28T17:48:26.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.027+0000 7f96c5c00640 1 -- 172.21.15.45:0/2602687272 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c01a5cd0 con 0x7f96c0108db0 2024-03-28T17:48:26.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.027+0000 7f96c5c00640 1 -- 172.21.15.45:0/2602687272 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c01a6190 con 0x7f96c0108db0 2024-03-28T17:48:26.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.028+0000 7f96c5c00640 1 -- 172.21.15.45:0/2602687272 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c010ee80 con 0x7f96c0108db0 2024-03-28T17:48:26.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.030+0000 7f96bcc00640 1 -- 172.21.15.45:0/2602687272 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f96ac046030 con 0x7f96c0108db0 2024-03-28T17:48:26.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.031+0000 7f96bcc00640 1 --2- 172.21.15.45:0/2602687272 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f96a40a2880 0x7f96a40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:26.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.031+0000 7f96bcc00640 1 -- 172.21.15.45:0/2602687272 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f96ac0f0700 con 0x7f96c0108db0 2024-03-28T17:48:26.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.032+0000 7f96bfe00640 1 --2- 172.21.15.45:0/2602687272 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f96a40a2880 0x7f96a40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:26.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.032+0000 7f96bfe00640 1 --2- 172.21.15.45:0/2602687272 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f96a40a2880 0x7f96a40a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f96b00058c0 tx=0x7f96b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:26.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.040+0000 7f96bcc00640 1 -- 172.21.15.45:0/2602687272 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f96ac04d060 con 0x7f96c0108db0 2024-03-28T17:48:26.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.248+0000 7f96c5c00640 1 -- 172.21.15.45:0/2602687272 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f96c00622a0 con 0x7f96c0108db0 2024-03-28T17:48:26.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.249+0000 7f96bcc00640 1 -- 172.21.15.45:0/2602687272 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f96ac0bd7e0 con 0x7f96c0108db0 2024-03-28T17:48:26.249 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-03-28T17:48:26.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.252+0000 7f96c5c00640 1 -- 172.21.15.45:0/2602687272 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f96a40a2880 msgr2=0x7f96a40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:26.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.252+0000 7f96c5c00640 1 --2- 172.21.15.45:0/2602687272 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f96a40a2880 0x7f96a40a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f96b00058c0 tx=0x7f96b0005f90 comp rx=0 tx=0).stop 2024-03-28T17:48:26.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.252+0000 7f96c5c00640 1 -- 172.21.15.45:0/2602687272 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96c0108db0 msgr2=0x7f96c019e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:26.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.252+0000 7f96c5c00640 1 --2- 172.21.15.45:0/2602687272 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96c0108db0 0x7f96c019e360 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009bf0 tx=0x7f96ac035d10 comp rx=0 tx=0).stop 2024-03-28T17:48:26.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.252+0000 7f96c5c00640 1 -- 172.21.15.45:0/2602687272 shutdown_connections 2024-03-28T17:48:26.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.252+0000 7f96c5c00640 1 --2- 172.21.15.45:0/2602687272 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f96a40a2880 0x7f96a40a4d40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:26.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.252+0000 7f96c5c00640 1 --2- 172.21.15.45:0/2602687272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96c0109770 0x7f96c01a53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:26.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.252+0000 7f96c5c00640 1 --2- 172.21.15.45:0/2602687272 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96c0108db0 0x7f96c019e360 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:26.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.252+0000 7f96c5c00640 1 --2- 172.21.15.45:0/2602687272 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96c0107bc0 0x7f96c019de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:26.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.252+0000 7f96c5c00640 1 -- 172.21.15.45:0/2602687272 >> 172.21.15.45:0/2602687272 conn(0x7f96c0076b70 msgr2=0x7f96c010c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:26.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.252+0000 7f96c5c00640 1 -- 172.21.15.45:0/2602687272 shutdown_connections 2024-03-28T17:48:26.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.252+0000 7f96c5c00640 1 -- 172.21.15.45:0/2602687272 wait complete. 2024-03-28T17:48:26.272 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T17:48:26.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.502+0000 7fcc76000640 1 -- 172.21.15.45:0/1873362993 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc701038e0 msgr2=0x7fcc70103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:26.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.502+0000 7fcc76000640 1 --2- 172.21.15.45:0/1873362993 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc701038e0 0x7fcc70103d00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcc60009900 tx=0x7fcc6002bc30 comp rx=0 tx=0).stop 2024-03-28T17:48:26.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.502+0000 7fcc76000640 1 -- 172.21.15.45:0/1873362993 shutdown_connections 2024-03-28T17:48:26.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.502+0000 7fcc76000640 1 --2- 172.21.15.45:0/1873362993 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc70105490 0x7fcc7010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:26.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.502+0000 7fcc76000640 1 --2- 172.21.15.45:0/1873362993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc70104ad0 0x7fcc70104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:26.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.502+0000 7fcc76000640 1 --2- 172.21.15.45:0/1873362993 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc701038e0 0x7fcc70103d00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:26.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.502+0000 7fcc76000640 1 -- 172.21.15.45:0/1873362993 >> 172.21.15.45:0/1873362993 conn(0x7fcc700fee50 msgr2=0x7fcc70101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:26.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.502+0000 7fcc76000640 1 -- 172.21.15.45:0/1873362993 shutdown_connections 2024-03-28T17:48:26.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.502+0000 7fcc76000640 1 -- 172.21.15.45:0/1873362993 wait complete. 2024-03-28T17:48:26.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.503+0000 7fcc76000640 1 Processor -- start 2024-03-28T17:48:26.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.503+0000 7fcc76000640 1 -- start start 2024-03-28T17:48:26.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.503+0000 7fcc76000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc701038e0 0x7fcc70199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:26.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.503+0000 7fcc76000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc70104ad0 0x7fcc7019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:26.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.503+0000 7fcc76000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc70105490 0x7fcc701a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:26.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.503+0000 7fcc76000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7010f5b0 con 0x7fcc70104ad0 2024-03-28T17:48:26.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.503+0000 7fcc76000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7010f430 con 0x7fcc701038e0 2024-03-28T17:48:26.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.503+0000 7fcc76000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7010f730 con 0x7fcc70105490 2024-03-28T17:48:26.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.504+0000 7fcc6f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc701038e0 0x7fcc70199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:26.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.504+0000 7fcc6f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc701038e0 0x7fcc70199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53426/0 (socket says 172.21.15.45:53426) 2024-03-28T17:48:26.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.504+0000 7fcc6f400640 1 -- 172.21.15.45:0/2916204528 learned_addr learned my addr 172.21.15.45:0/2916204528 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:26.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.504+0000 7fcc6ea00640 1 --2- 172.21.15.45:0/2916204528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc70104ad0 0x7fcc7019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:26.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.504+0000 7fcc6fe00640 1 --2- 172.21.15.45:0/2916204528 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc70105490 0x7fcc701a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:26.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.504+0000 7fcc6f400640 1 -- 172.21.15.45:0/2916204528 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc70105490 msgr2=0x7fcc701a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:26.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.504+0000 7fcc6f400640 1 --2- 172.21.15.45:0/2916204528 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc70105490 0x7fcc701a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:26.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.504+0000 7fcc6f400640 1 -- 172.21.15.45:0/2916204528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc70104ad0 msgr2=0x7fcc7019a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:26.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.504+0000 7fcc6f400640 1 --2- 172.21.15.45:0/2916204528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc70104ad0 0x7fcc7019a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:26.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.504+0000 7fcc6f400640 1 -- 172.21.15.45:0/2916204528 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc701a1880 con 0x7fcc701038e0 2024-03-28T17:48:26.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.504+0000 7fcc6f400640 1 --2- 172.21.15.45:0/2916204528 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc701038e0 0x7fcc70199c70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcc60000c00 tx=0x7fcc6002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:26.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.504+0000 7fcc67e00640 1 -- 172.21.15.45:0/2916204528 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc60043020 con 0x7fcc701038e0 2024-03-28T17:48:26.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.504+0000 7fcc67e00640 1 -- 172.21.15.45:0/2916204528 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc600054e0 con 0x7fcc701038e0 2024-03-28T17:48:26.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.504+0000 7fcc76000640 1 -- 172.21.15.45:0/2916204528 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc701a1b10 con 0x7fcc701038e0 2024-03-28T17:48:26.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.505+0000 7fcc67e00640 1 -- 172.21.15.45:0/2916204528 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc60003c40 con 0x7fcc701038e0 2024-03-28T17:48:26.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.505+0000 7fcc76000640 1 -- 172.21.15.45:0/2916204528 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc701a2060 con 0x7fcc701038e0 2024-03-28T17:48:26.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.506+0000 7fcc76000640 1 -- 172.21.15.45:0/2916204528 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc70068490 con 0x7fcc701038e0 2024-03-28T17:48:26.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.507+0000 7fcc67e00640 1 -- 172.21.15.45:0/2916204528 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcc60046020 con 0x7fcc701038e0 2024-03-28T17:48:26.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.509+0000 7fcc67e00640 1 --2- 172.21.15.45:0/2916204528 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcc540a2800 0x7fcc540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:26.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.509+0000 7fcc67e00640 1 -- 172.21.15.45:0/2916204528 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fcc600f0ae0 con 0x7fcc701038e0 2024-03-28T17:48:26.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.509+0000 7fcc6ea00640 1 --2- 172.21.15.45:0/2916204528 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcc540a2800 0x7fcc540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:26.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.510+0000 7fcc6ea00640 1 --2- 172.21.15.45:0/2916204528 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcc540a2800 0x7fcc540a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c009bd0 tx=0x7fcc5c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:26.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.513+0000 7fcc67e00640 1 -- 172.21.15.45:0/2916204528 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcc6004d020 con 0x7fcc701038e0 2024-03-28T17:48:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:26 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/2783354323' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:48:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:26 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/2602687272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T17:48:26.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.718+0000 7fcc76000640 1 -- 172.21.15.45:0/2916204528 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcc700622a0 con 0x7fcc701038e0 2024-03-28T17:48:26.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.718+0000 7fcc67e00640 1 -- 172.21.15.45:0/2916204528 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcc600bdb30 con 0x7fcc701038e0 2024-03-28T17:48:26.719 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-03-28T17:48:26.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.721+0000 7fcc76000640 1 -- 172.21.15.45:0/2916204528 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcc540a2800 msgr2=0x7fcc540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:26.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.721+0000 7fcc76000640 1 --2- 172.21.15.45:0/2916204528 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcc540a2800 0x7fcc540a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c009bd0 tx=0x7fcc5c005f90 comp rx=0 tx=0).stop 2024-03-28T17:48:26.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.721+0000 7fcc76000640 1 -- 172.21.15.45:0/2916204528 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc701038e0 msgr2=0x7fcc70199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:26.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.721+0000 7fcc76000640 1 --2- 172.21.15.45:0/2916204528 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc701038e0 0x7fcc70199c70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcc60000c00 tx=0x7fcc6002c650 comp rx=0 tx=0).stop 2024-03-28T17:48:26.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.721+0000 7fcc76000640 1 -- 172.21.15.45:0/2916204528 shutdown_connections 2024-03-28T17:48:26.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.721+0000 7fcc76000640 1 --2- 172.21.15.45:0/2916204528 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcc540a2800 0x7fcc540a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:26.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.721+0000 7fcc76000640 1 --2- 172.21.15.45:0/2916204528 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc70105490 0x7fcc701a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:26.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.721+0000 7fcc76000640 1 --2- 172.21.15.45:0/2916204528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc70104ad0 0x7fcc7019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:26.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.721+0000 7fcc76000640 1 --2- 172.21.15.45:0/2916204528 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc701038e0 0x7fcc70199c70 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:26.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.721+0000 7fcc76000640 1 -- 172.21.15.45:0/2916204528 >> 172.21.15.45:0/2916204528 conn(0x7fcc700fee50 msgr2=0x7fcc70107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:26.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.721+0000 7fcc76000640 1 -- 172.21.15.45:0/2916204528 shutdown_connections 2024-03-28T17:48:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.722+0000 7fcc76000640 1 -- 172.21.15.45:0/2916204528 wait complete. 2024-03-28T17:48:26.742 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:48:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:26 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/2783354323' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:48:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:26 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/2602687272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T17:48:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:26 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/2783354323' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:48:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:26 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/2602687272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T17:48:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.963+0000 7f833be00640 1 -- 172.21.15.45:0/1748016351 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f833c1036f0 msgr2=0x7f833c103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.963+0000 7f833be00640 1 --2- 172.21.15.45:0/1748016351 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f833c1036f0 0x7f833c103b10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f832c009900 tx=0x7f832c02bc50 comp rx=0 tx=0).stop 2024-03-28T17:48:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.964+0000 7f833be00640 1 -- 172.21.15.45:0/1748016351 shutdown_connections 2024-03-28T17:48:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.964+0000 7f833be00640 1 --2- 172.21.15.45:0/1748016351 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f833c1052a0 0x7f833c10bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.964+0000 7f833be00640 1 --2- 172.21.15.45:0/1748016351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f833c1048e0 0x7f833c104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.964+0000 7f833be00640 1 --2- 172.21.15.45:0/1748016351 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f833c1036f0 0x7f833c103b10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.964+0000 7f833be00640 1 -- 172.21.15.45:0/1748016351 >> 172.21.15.45:0/1748016351 conn(0x7f833c0fec80 msgr2=0x7f833c1010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.964+0000 7f833be00640 1 -- 172.21.15.45:0/1748016351 shutdown_connections 2024-03-28T17:48:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.964+0000 7f833be00640 1 -- 172.21.15.45:0/1748016351 wait complete. 2024-03-28T17:48:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.965+0000 7f833be00640 1 Processor -- start 2024-03-28T17:48:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.965+0000 7f833be00640 1 -- start start 2024-03-28T17:48:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.966+0000 7f833be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f833c1036f0 0x7f833c1999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.966+0000 7f833be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f833c1048e0 0x7f833c199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.966+0000 7f833be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f833c1052a0 0x7f833c1a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.966+0000 7f833be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c10f3c0 con 0x7f833c1048e0 2024-03-28T17:48:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.966+0000 7f833be00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c10f240 con 0x7f833c1036f0 2024-03-28T17:48:26.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.966+0000 7f833be00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c10f540 con 0x7f833c1052a0 2024-03-28T17:48:26.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.966+0000 7f833aa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f833c1036f0 0x7f833c1999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:26.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.966+0000 7f833aa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f833c1036f0 0x7f833c1999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53458/0 (socket says 172.21.15.45:53458) 2024-03-28T17:48:26.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.966+0000 7f833aa00640 1 -- 172.21.15.45:0/3791299624 learned_addr learned my addr 172.21.15.45:0/3791299624 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:26.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.966+0000 7f833a000640 1 --2- 172.21.15.45:0/3791299624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f833c1048e0 0x7f833c199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:26.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.966+0000 7f833b400640 1 --2- 172.21.15.45:0/3791299624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f833c1052a0 0x7f833c1a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:26.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.966+0000 7f833aa00640 1 -- 172.21.15.45:0/3791299624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f833c1052a0 msgr2=0x7f833c1a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.966+0000 7f833aa00640 1 --2- 172.21.15.45:0/3791299624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f833c1052a0 0x7f833c1a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.966+0000 7f833aa00640 1 -- 172.21.15.45:0/3791299624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f833c1048e0 msgr2=0x7f833c199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.966+0000 7f833aa00640 1 --2- 172.21.15.45:0/3791299624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f833c1048e0 0x7f833c199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.966+0000 7f833aa00640 1 -- 172.21.15.45:0/3791299624 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c1a1600 con 0x7f833c1036f0 2024-03-28T17:48:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.967+0000 7f833aa00640 1 --2- 172.21.15.45:0/3791299624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f833c1036f0 0x7f833c1999e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f832c02c260 tx=0x7f832c02c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.967+0000 7f8340a00640 1 -- 172.21.15.45:0/3791299624 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832c041070 con 0x7f833c1036f0 2024-03-28T17:48:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.967+0000 7f8340a00640 1 -- 172.21.15.45:0/3791299624 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f832c004450 con 0x7f833c1036f0 2024-03-28T17:48:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.967+0000 7f833be00640 1 -- 172.21.15.45:0/3791299624 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f833c1a1890 con 0x7f833c1036f0 2024-03-28T17:48:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.967+0000 7f8340a00640 1 -- 172.21.15.45:0/3791299624 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832c034440 con 0x7f833c1036f0 2024-03-28T17:48:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.967+0000 7f833be00640 1 -- 172.21.15.45:0/3791299624 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f833c1a1de0 con 0x7f833c1036f0 2024-03-28T17:48:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.968+0000 7f833be00640 1 -- 172.21.15.45:0/3791299624 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f833c10a9b0 con 0x7f833c1036f0 2024-03-28T17:48:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.970+0000 7f8340a00640 1 -- 172.21.15.45:0/3791299624 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f832c044070 con 0x7f833c1036f0 2024-03-28T17:48:26.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.971+0000 7f8340a00640 1 --2- 172.21.15.45:0/3791299624 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f83200a2800 0x7f83200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:26.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.971+0000 7f8340a00640 1 -- 172.21.15.45:0/3791299624 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f832c0f0550 con 0x7f833c1036f0 2024-03-28T17:48:26.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.971+0000 7f833a000640 1 --2- 172.21.15.45:0/3791299624 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f83200a2800 0x7f83200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:26.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.972+0000 7f833a000640 1 --2- 172.21.15.45:0/3791299624 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f83200a2800 0x7f83200a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f833c19aef0 tx=0x7f8328009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:26.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:26.975+0000 7f8340a00640 1 -- 172.21.15.45:0/3791299624 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f832c0bcf50 con 0x7f833c1036f0 2024-03-28T17:48:27.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.221+0000 7f833be00640 1 -- 172.21.15.45:0/3791299624 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f833c0622a0 con 0x7f833c1036f0 2024-03-28T17:48:27.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.223+0000 7f8340a00640 1 -- 172.21.15.45:0/3791299624 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f832c0c0ff0 con 0x7f833c1036f0 2024-03-28T17:48:27.223 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:48:27.224 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6839/2313500150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":2313500150},{"type":"v1","addr":"172.21.15.45:6839","nonce":2313500150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6841/2647972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":2647972571},{"type":"v1","addr":"172.21.15.45:6841","nonce":2647972571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T17:46:12.261135+0000","modified":"2024-03-28T17:46:17.967524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24584,"mds_1":24530,"mds_2":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6835/2876354410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":2876354410},{"type":"v1","addr":"172.21.15.151:6835","nonce":2876354410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24500":{"gid":24500,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/4234998385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":4234998385},{"type":"v1","addr":"172.21.15.71:6835","nonce":4234998385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6837/1238964845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":1238964845},{"type":"v1","addr":"172.21.15.151:6837","nonce":1238964845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.151:6839/3003564901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6838","nonce":3003564901},{"type":"v1","addr":"172.21.15.151:6839","nonce":3003564901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6837/2008752655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2008752655},{"type":"v1","addr":"172.21.15.71:6837","nonce":2008752655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.71:6839/2709938800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":2709938800},{"type":"v1","addr":"172.21.15.71:6839","nonce":2709938800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24584,"qdb_cluster":[24529,24530,24584]},"id":1}]} 2024-03-28T17:48:27.224 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 22 2024-03-28T17:48:27.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.226+0000 7f833be00640 1 -- 172.21.15.45:0/3791299624 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f83200a2800 msgr2=0x7f83200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:27.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.226+0000 7f833be00640 1 --2- 172.21.15.45:0/3791299624 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f83200a2800 0x7f83200a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f833c19aef0 tx=0x7f8328009500 comp rx=0 tx=0).stop 2024-03-28T17:48:27.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.226+0000 7f833be00640 1 -- 172.21.15.45:0/3791299624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f833c1036f0 msgr2=0x7f833c1999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:27.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.226+0000 7f833be00640 1 --2- 172.21.15.45:0/3791299624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f833c1036f0 0x7f833c1999e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f832c02c260 tx=0x7f832c02c670 comp rx=0 tx=0).stop 2024-03-28T17:48:27.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.226+0000 7f833be00640 1 -- 172.21.15.45:0/3791299624 shutdown_connections 2024-03-28T17:48:27.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.226+0000 7f833be00640 1 --2- 172.21.15.45:0/3791299624 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f83200a2800 0x7f83200a4cc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:27.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.226+0000 7f833be00640 1 --2- 172.21.15.45:0/3791299624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f833c1052a0 0x7f833c1a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:27.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.226+0000 7f833be00640 1 --2- 172.21.15.45:0/3791299624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f833c1048e0 0x7f833c199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:27.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.226+0000 7f833be00640 1 --2- 172.21.15.45:0/3791299624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f833c1036f0 0x7f833c1999e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:27.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.226+0000 7f833be00640 1 -- 172.21.15.45:0/3791299624 >> 172.21.15.45:0/3791299624 conn(0x7f833c0fec80 msgr2=0x7f833c107b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:27.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.226+0000 7f833be00640 1 -- 172.21.15.45:0/3791299624 shutdown_connections 2024-03-28T17:48:27.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.226+0000 7f833be00640 1 -- 172.21.15.45:0/3791299624 wait complete. 2024-03-28T17:48:27.248 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T17:48:27.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.472+0000 7f2f0be00640 1 -- 172.21.15.45:0/3734016025 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f0c107b00 msgr2=0x7f2f0c107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:27.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.472+0000 7f2f0be00640 1 --2- 172.21.15.45:0/3734016025 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f0c107b00 0x7f2f0c107f20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2efc009900 tx=0x7f2efc02bc30 comp rx=0 tx=0).stop 2024-03-28T17:48:27.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.472+0000 7f2f0be00640 1 -- 172.21.15.45:0/3734016025 shutdown_connections 2024-03-28T17:48:27.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.472+0000 7f2f0be00640 1 --2- 172.21.15.45:0/3734016025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f0c1096b0 0x7f2f0c10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:27.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.472+0000 7f2f0be00640 1 --2- 172.21.15.45:0/3734016025 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f0c108cf0 0x7f2f0c109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:27.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.472+0000 7f2f0be00640 1 --2- 172.21.15.45:0/3734016025 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f0c107b00 0x7f2f0c107f20 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:27.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.472+0000 7f2f0be00640 1 -- 172.21.15.45:0/3734016025 >> 172.21.15.45:0/3734016025 conn(0x7f2f0c076ab0 msgr2=0x7f2f0c078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:27.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.472+0000 7f2f0be00640 1 -- 172.21.15.45:0/3734016025 shutdown_connections 2024-03-28T17:48:27.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.472+0000 7f2f0be00640 1 -- 172.21.15.45:0/3734016025 wait complete. 2024-03-28T17:48:27.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.474+0000 7f2f0be00640 1 Processor -- start 2024-03-28T17:48:27.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.474+0000 7f2f0be00640 1 -- start start 2024-03-28T17:48:27.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.474+0000 7f2f0be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f0c107b00 0x7f2f0c19de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:27.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.474+0000 7f2f0be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f0c108cf0 0x7f2f0c19e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:27.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.474+0000 7f2f0be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f0c1096b0 0x7f2f0c1a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:27.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.474+0000 7f2f0be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c1137a0 con 0x7f2f0c1096b0 2024-03-28T17:48:27.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.474+0000 7f2f0be00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c113620 con 0x7f2f0c108cf0 2024-03-28T17:48:27.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.474+0000 7f2f0be00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c113920 con 0x7f2f0c107b00 2024-03-28T17:48:27.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.475+0000 7f2f0aa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f0c107b00 0x7f2f0c19de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:27.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.475+0000 7f2f03e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f0c108cf0 0x7f2f0c19e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:27.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.475+0000 7f2f0aa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f0c107b00 0x7f2f0c19de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:41304/0 (socket says 172.21.15.45:41304) 2024-03-28T17:48:27.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.475+0000 7f2f0aa00640 1 -- 172.21.15.45:0/1351026974 learned_addr learned my addr 172.21.15.45:0/1351026974 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:27.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.475+0000 7f2f0b400640 1 --2- 172.21.15.45:0/1351026974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f0c1096b0 0x7f2f0c1a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:27.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.475+0000 7f2f03e00640 1 -- 172.21.15.45:0/1351026974 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f0c107b00 msgr2=0x7f2f0c19de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:27.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.475+0000 7f2f03e00640 1 --2- 172.21.15.45:0/1351026974 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f0c107b00 0x7f2f0c19de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:27.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.475+0000 7f2f03e00640 1 -- 172.21.15.45:0/1351026974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f0c1096b0 msgr2=0x7f2f0c1a53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:27.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.475+0000 7f2f03e00640 1 --2- 172.21.15.45:0/1351026974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f0c1096b0 0x7f2f0c1a53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:27.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.475+0000 7f2f03e00640 1 -- 172.21.15.45:0/1351026974 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0c1a5a30 con 0x7f2f0c108cf0 2024-03-28T17:48:27.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.475+0000 7f2f03e00640 1 --2- 172.21.15.45:0/1351026974 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f0c108cf0 0x7f2f0c19e350 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2ef400e9f0 tx=0x7f2ef400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:27.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.476+0000 7f2f03400640 1 -- 172.21.15.45:0/1351026974 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef400ccc0 con 0x7f2f0c108cf0 2024-03-28T17:48:27.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.476+0000 7f2f03400640 1 -- 172.21.15.45:0/1351026974 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ef4010910 con 0x7f2f0c108cf0 2024-03-28T17:48:27.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.476+0000 7f2f0be00640 1 -- 172.21.15.45:0/1351026974 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f0c1a5d20 con 0x7f2f0c108cf0 2024-03-28T17:48:27.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.476+0000 7f2f03400640 1 -- 172.21.15.45:0/1351026974 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4018960 con 0x7f2f0c108cf0 2024-03-28T17:48:27.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.476+0000 7f2f0be00640 1 -- 172.21.15.45:0/1351026974 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f0c1a6260 con 0x7f2f0c108cf0 2024-03-28T17:48:27.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.477+0000 7f2f0be00640 1 -- 172.21.15.45:0/1351026974 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f0c10ee20 con 0x7f2f0c108cf0 2024-03-28T17:48:27.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.479+0000 7f2f03400640 1 -- 172.21.15.45:0/1351026974 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2ef4010ac0 con 0x7f2f0c108cf0 2024-03-28T17:48:27.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.480+0000 7f2f03400640 1 --2- 172.21.15.45:0/1351026974 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2ef00a2880 0x7f2ef00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:27.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.480+0000 7f2f03400640 1 -- 172.21.15.45:0/1351026974 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f2ef40ceda0 con 0x7f2f0c108cf0 2024-03-28T17:48:27.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.480+0000 7f2f0aa00640 1 --2- 172.21.15.45:0/1351026974 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2ef00a2880 0x7f2ef00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:27.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.481+0000 7f2f0aa00640 1 --2- 172.21.15.45:0/1351026974 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2ef00a2880 0x7f2ef00a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2efc006010 tx=0x7f2efc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:27.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.484+0000 7f2f03400640 1 -- 172.21.15.45:0/1351026974 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2ef409b7a0 con 0x7f2f0c108cf0 2024-03-28T17:48:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:27 smithi045 ceph-mon[31773]: pgmap v386: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:48:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:27 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/2916204528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T17:48:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:27 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3791299624' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:48:27.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.679+0000 7f2f0be00640 1 -- 172.21.15.45:0/1351026974 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f0c062340 con 0x7f2f0c108cf0 2024-03-28T17:48:27.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.681+0000 7f2f03400640 1 -- 172.21.15.45:0/1351026974 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22583 (secure 0 0 0) 0x7f2ef409f840 con 0x7f2f0c108cf0 2024-03-28T17:48:27.682 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:48:27.682 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"9f4888ec-ed29-11ee-95d0-87774f69a715","created":"2024-03-28T17:36:51.623185+0000","modified":"2024-03-28T17:47:09.455843+0000","last_up_change":"2024-03-28T17:45:14.043432+0000","last_in_change":"2024-03-28T17:44:53.654489+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T17:40:54.208375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T17:46:10.564519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T17:46:11.734367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"74c56bc2-4169-4919-afd4-2ebd236f23e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6803","nonce":3795754023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6805","nonce":3795754023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6809","nonce":3795754023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6807","nonce":3795754023}]},"public_addr":"172.21.15.45:6803/3795754023","cluster_addr":"172.21.15.45:6805/3795754023","heartbeat_back_addr":"172.21.15.45:6809/3795754023","heartbeat_front_addr":"172.21.15.45:6807/3795754023","state":["exists","up"]},{"osd":1,"uuid":"78dcc21b-75b2-453d-b690-17b55b8ed1a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6801","nonce":2014529906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6803","nonce":2014529906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6807","nonce":2014529906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6805","nonce":2014529906}]},"public_addr":"172.21.15.71:6801/2014529906","cluster_addr":"172.21.15.71:6803/2014529906","heartbeat_back_addr":"172.21.15.71:6807/2014529906","heartbeat_front_addr":"172.21.15.71:6805/2014529906","state":["exists","up"]},{"osd":2,"uuid":"ca9f3f36-b71e-4962-a94f-cedf2f12eb7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6801","nonce":1850933775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6803","nonce":1850933775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6807","nonce":1850933775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6805","nonce":1850933775}]},"public_addr":"172.21.15.151:6801/1850933775","cluster_addr":"172.21.15.151:6803/1850933775","heartbeat_back_addr":"172.21.15.151:6807/1850933775","heartbeat_front_addr":"172.21.15.151:6805/1850933775","state":["exists","up"]},{"osd":3,"uuid":"d3f21f01-2973-413e-8daa-9974b8a44af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6811","nonce":3021569975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6813","nonce":3021569975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6817","nonce":3021569975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6815","nonce":3021569975}]},"public_addr":"172.21.15.45:6811/3021569975","cluster_addr":"172.21.15.45:6813/3021569975","heartbeat_back_addr":"172.21.15.45:6817/3021569975","heartbeat_front_addr":"172.21.15.45:6815/3021569975","state":["exists","up"]},{"osd":4,"uuid":"4a336f42-0605-492b-ad88-56cb02acdcf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6809","nonce":2069578333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6811","nonce":2069578333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6815","nonce":2069578333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6813","nonce":2069578333}]},"public_addr":"172.21.15.71:6809/2069578333","cluster_addr":"172.21.15.71:6811/2069578333","heartbeat_back_addr":"172.21.15.71:6815/2069578333","heartbeat_front_addr":"172.21.15.71:6813/2069578333","state":["exists","up"]},{"osd":5,"uuid":"c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6809","nonce":2629217227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6811","nonce":2629217227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6815","nonce":2629217227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6813","nonce":2629217227}]},"public_addr":"172.21.15.151:6809/2629217227","cluster_addr":"172.21.15.151:6811/2629217227","heartbeat_back_addr":"172.21.15.151:6815/2629217227","heartbeat_front_addr":"172.21.15.151:6813/2629217227","state":["exists","up"]},{"osd":6,"uuid":"58a6c5bd-a8b7-4892-b346-bcd7994bcb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6819","nonce":2077907039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6821","nonce":2077907039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6825","nonce":2077907039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6823","nonce":2077907039}]},"public_addr":"172.21.15.45:6819/2077907039","cluster_addr":"172.21.15.45:6821/2077907039","heartbeat_back_addr":"172.21.15.45:6825/2077907039","heartbeat_front_addr":"172.21.15.45:6823/2077907039","state":["exists","up"]},{"osd":7,"uuid":"f7a45d52-807b-4407-bb04-787b2a52a0ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6817","nonce":2000211251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6819","nonce":2000211251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6823","nonce":2000211251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6821","nonce":2000211251}]},"public_addr":"172.21.15.71:6817/2000211251","cluster_addr":"172.21.15.71:6819/2000211251","heartbeat_back_addr":"172.21.15.71:6823/2000211251","heartbeat_front_addr":"172.21.15.71:6821/2000211251","state":["exists","up"]},{"osd":8,"uuid":"88abcb0f-f189-49be-8315-c700c77369f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6817","nonce":3665460137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6819","nonce":3665460137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6823","nonce":3665460137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6821","nonce":3665460137}]},"public_addr":"172.21.15.151:6817/3665460137","cluster_addr":"172.21.15.151:6819/3665460137","heartbeat_back_addr":"172.21.15.151:6823/3665460137","heartbeat_front_addr":"172.21.15.151:6821/3665460137","state":["exists","up"]},{"osd":9,"uuid":"ed3fc096-961e-4392-abc2-c5aa4d050117","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6827","nonce":2122322684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6829","nonce":2122322684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6833","nonce":2122322684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6831","nonce":2122322684}]},"public_addr":"172.21.15.45:6827/2122322684","cluster_addr":"172.21.15.45:6829/2122322684","heartbeat_back_addr":"172.21.15.45:6833/2122322684","heartbeat_front_addr":"172.21.15.45:6831/2122322684","state":["exists","up"]},{"osd":10,"uuid":"ba94762b-bc5c-4c97-9cc1-17e5651c14b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6825","nonce":1099494037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6827","nonce":1099494037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6831","nonce":1099494037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6829","nonce":1099494037}]},"public_addr":"172.21.15.71:6825/1099494037","cluster_addr":"172.21.15.71:6827/1099494037","heartbeat_back_addr":"172.21.15.71:6831/1099494037","heartbeat_front_addr":"172.21.15.71:6829/1099494037","state":["exists","up"]},{"osd":11,"uuid":"1d178378-b9f5-40f6-99ef-3d3d445da640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6825","nonce":2941055364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6827","nonce":2941055364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6831","nonce":2941055364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6829","nonce":2941055364}]},"public_addr":"172.21.15.151:6825/2941055364","cluster_addr":"172.21.15.151:6827/2941055364","heartbeat_back_addr":"172.21.15.151:6831/2941055364","heartbeat_front_addr":"172.21.15.151:6829/2941055364","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:02.269586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:27.819386+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:51.552277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:25.052674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:51.873529+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:17.127867+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:51.588249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:18.381450+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:44.140534+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:17.634362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:45.469340+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:45:12.003433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3533611957":"2024-03-29T17:38:07.505611+0000","172.21.15.45:0/3625922430":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6801/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/482610468":"2024-03-29T17:37:32.424229+0000","172.21.15.45:6801/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3649804714":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/1198364079":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3211303958":"2024-03-29T17:38:07.505611+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T17:48:27.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.684+0000 7f2f0be00640 1 -- 172.21.15.45:0/1351026974 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2ef00a2880 msgr2=0x7f2ef00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:27.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.684+0000 7f2f0be00640 1 --2- 172.21.15.45:0/1351026974 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2ef00a2880 0x7f2ef00a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2efc006010 tx=0x7f2efc02df90 comp rx=0 tx=0).stop 2024-03-28T17:48:27.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.684+0000 7f2f0be00640 1 -- 172.21.15.45:0/1351026974 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f0c108cf0 msgr2=0x7f2f0c19e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:27.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.684+0000 7f2f0be00640 1 --2- 172.21.15.45:0/1351026974 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f0c108cf0 0x7f2f0c19e350 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2ef400e9f0 tx=0x7f2ef400eeb0 comp rx=0 tx=0).stop 2024-03-28T17:48:27.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.684+0000 7f2f0be00640 1 -- 172.21.15.45:0/1351026974 shutdown_connections 2024-03-28T17:48:27.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.684+0000 7f2f0be00640 1 --2- 172.21.15.45:0/1351026974 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2ef00a2880 0x7f2ef00a4d40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:27.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.684+0000 7f2f0be00640 1 --2- 172.21.15.45:0/1351026974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f0c1096b0 0x7f2f0c1a53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:27.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.684+0000 7f2f0be00640 1 --2- 172.21.15.45:0/1351026974 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f0c108cf0 0x7f2f0c19e350 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:27.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.684+0000 7f2f0be00640 1 --2- 172.21.15.45:0/1351026974 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f0c107b00 0x7f2f0c19de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:27.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.684+0000 7f2f0be00640 1 -- 172.21.15.45:0/1351026974 >> 172.21.15.45:0/1351026974 conn(0x7f2f0c076ab0 msgr2=0x7f2f0c10bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:27.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.684+0000 7f2f0be00640 1 -- 172.21.15.45:0/1351026974 shutdown_connections 2024-03-28T17:48:27.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.684+0000 7f2f0be00640 1 -- 172.21.15.45:0/1351026974 wait complete. 2024-03-28T17:48:27.704 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T17:40:54.208375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T17:46:10.564519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T17:46:11.734367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T17:48:27.705 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T17:48:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:27 smithi071 ceph-mon[36597]: pgmap v386: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:48:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:27 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/2916204528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T17:48:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:27 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3791299624' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:48:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:27 smithi151 ceph-mon[36481]: pgmap v386: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:48:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:27 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/2916204528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T17:48:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:27 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3791299624' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:48:27.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.929+0000 7f9c50a00640 1 -- 172.21.15.45:0/3578722530 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c4c074100 msgr2=0x7f9c4c074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:27.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.929+0000 7f9c50a00640 1 --2- 172.21.15.45:0/3578722530 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c4c074100 0x7f9c4c074580 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9c34009900 tx=0x7f9c3402bc50 comp rx=0 tx=0).stop 2024-03-28T17:48:27.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.929+0000 7f9c50a00640 1 -- 172.21.15.45:0/3578722530 shutdown_connections 2024-03-28T17:48:27.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.929+0000 7f9c50a00640 1 --2- 172.21.15.45:0/3578722530 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c4c10db70 0x7f9c4c10ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:27.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.929+0000 7f9c50a00640 1 --2- 172.21.15.45:0/3578722530 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c4c074100 0x7f9c4c074580 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:27.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.929+0000 7f9c50a00640 1 --2- 172.21.15.45:0/3578722530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c4c0756e0 0x7f9c4c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:27.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.929+0000 7f9c50a00640 1 -- 172.21.15.45:0/3578722530 >> 172.21.15.45:0/3578722530 conn(0x7f9c4c0fcb10 msgr2=0x7f9c4c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:27.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.929+0000 7f9c50a00640 1 -- 172.21.15.45:0/3578722530 shutdown_connections 2024-03-28T17:48:27.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.929+0000 7f9c50a00640 1 -- 172.21.15.45:0/3578722530 wait complete. 2024-03-28T17:48:27.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.930+0000 7f9c50a00640 1 Processor -- start 2024-03-28T17:48:27.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.931+0000 7f9c50a00640 1 -- start start 2024-03-28T17:48:27.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.931+0000 7f9c50a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c4c074100 0x7f9c4c19dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:27.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.931+0000 7f9c50a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c4c0756e0 0x7f9c4c19e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:27.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.931+0000 7f9c50a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c4c10db70 0x7f9c4c1a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:27.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.931+0000 7f9c50a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c113840 con 0x7f9c4c074100 2024-03-28T17:48:27.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.931+0000 7f9c50a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c1136c0 con 0x7f9c4c10db70 2024-03-28T17:48:27.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.931+0000 7f9c50a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c1139c0 con 0x7f9c4c0756e0 2024-03-28T17:48:27.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.931+0000 7f9c4b200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c4c10db70 0x7f9c4c1a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:27.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.931+0000 7f9c4a800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c4c074100 0x7f9c4c19dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:27.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.931+0000 7f9c49e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c4c0756e0 0x7f9c4c19e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:27.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.931+0000 7f9c4b200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c4c10db70 0x7f9c4c1a5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53488/0 (socket says 172.21.15.45:53488) 2024-03-28T17:48:27.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.931+0000 7f9c4b200640 1 -- 172.21.15.45:0/2802150859 learned_addr learned my addr 172.21.15.45:0/2802150859 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:27.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.931+0000 7f9c49e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c4c0756e0 0x7f9c4c19e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:41320/0 (socket says 172.21.15.45:41320) 2024-03-28T17:48:27.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.932+0000 7f9c4b200640 1 -- 172.21.15.45:0/2802150859 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c4c0756e0 msgr2=0x7f9c4c19e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:27.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.932+0000 7f9c4b200640 1 --2- 172.21.15.45:0/2802150859 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c4c0756e0 0x7f9c4c19e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:27.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.932+0000 7f9c4b200640 1 -- 172.21.15.45:0/2802150859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c4c074100 msgr2=0x7f9c4c19dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:27.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.932+0000 7f9c4b200640 1 --2- 172.21.15.45:0/2802150859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c4c074100 0x7f9c4c19dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:27.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.932+0000 7f9c4b200640 1 -- 172.21.15.45:0/2802150859 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c4c1a5b00 con 0x7f9c4c10db70 2024-03-28T17:48:27.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.932+0000 7f9c4b200640 1 --2- 172.21.15.45:0/2802150859 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c4c10db70 0x7f9c4c1a5490 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9c4400efd0 tx=0x7f9c4400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:27.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.932+0000 7f9c43400640 1 -- 172.21.15.45:0/2802150859 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c44003b70 con 0x7f9c4c10db70 2024-03-28T17:48:27.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.932+0000 7f9c50a00640 1 -- 172.21.15.45:0/2802150859 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4c1a5df0 con 0x7f9c4c10db70 2024-03-28T17:48:27.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.932+0000 7f9c43400640 1 -- 172.21.15.45:0/2802150859 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c44003d10 con 0x7f9c4c10db70 2024-03-28T17:48:27.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.932+0000 7f9c43400640 1 -- 172.21.15.45:0/2802150859 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c440109f0 con 0x7f9c4c10db70 2024-03-28T17:48:27.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.932+0000 7f9c50a00640 1 -- 172.21.15.45:0/2802150859 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c4c1a62b0 con 0x7f9c4c10db70 2024-03-28T17:48:27.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.933+0000 7f9c50a00640 1 -- 172.21.15.45:0/2802150859 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c4c103d40 con 0x7f9c4c10db70 2024-03-28T17:48:27.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.935+0000 7f9c43400640 1 -- 172.21.15.45:0/2802150859 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9c44004020 con 0x7f9c4c10db70 2024-03-28T17:48:27.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.936+0000 7f9c43400640 1 --2- 172.21.15.45:0/2802150859 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9c300a2880 0x7f9c300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:27.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.937+0000 7f9c43400640 1 -- 172.21.15.45:0/2802150859 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9c440cee40 con 0x7f9c4c10db70 2024-03-28T17:48:27.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.937+0000 7f9c4a800640 1 --2- 172.21.15.45:0/2802150859 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9c300a2880 0x7f9c300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:27.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.937+0000 7f9c4a800640 1 --2- 172.21.15.45:0/2802150859 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9c300a2880 0x7f9c300a4d40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c009bd0 tx=0x7f9c3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:27.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:27.941+0000 7f9c43400640 1 -- 172.21.15.45:0/2802150859 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9c4409be90 con 0x7f9c4c10db70 2024-03-28T17:48:28.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.139+0000 7f9c50a00640 1 -- 172.21.15.45:0/2802150859 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9c4c0622a0 con 0x7f9c4c10db70 2024-03-28T17:48:28.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.140+0000 7f9c43400640 1 -- 172.21.15.45:0/2802150859 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9c4409b790 con 0x7f9c4c10db70 2024-03-28T17:48:28.140 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-28T17:48:28.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.143+0000 7f9c50a00640 1 -- 172.21.15.45:0/2802150859 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9c300a2880 msgr2=0x7f9c300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:28.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.143+0000 7f9c50a00640 1 --2- 172.21.15.45:0/2802150859 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9c300a2880 0x7f9c300a4d40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c009bd0 tx=0x7f9c3c005f90 comp rx=0 tx=0).stop 2024-03-28T17:48:28.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.143+0000 7f9c50a00640 1 -- 172.21.15.45:0/2802150859 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c4c10db70 msgr2=0x7f9c4c1a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:28.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.143+0000 7f9c50a00640 1 --2- 172.21.15.45:0/2802150859 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c4c10db70 0x7f9c4c1a5490 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9c4400efd0 tx=0x7f9c4400c6a0 comp rx=0 tx=0).stop 2024-03-28T17:48:28.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.143+0000 7f9c50a00640 1 -- 172.21.15.45:0/2802150859 shutdown_connections 2024-03-28T17:48:28.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.143+0000 7f9c50a00640 1 --2- 172.21.15.45:0/2802150859 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9c300a2880 0x7f9c300a4d40 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:28.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.143+0000 7f9c50a00640 1 --2- 172.21.15.45:0/2802150859 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c4c10db70 0x7f9c4c1a5490 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:28.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.143+0000 7f9c50a00640 1 --2- 172.21.15.45:0/2802150859 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c4c0756e0 0x7f9c4c19e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:28.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.143+0000 7f9c50a00640 1 --2- 172.21.15.45:0/2802150859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c4c074100 0x7f9c4c19dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:28.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.143+0000 7f9c50a00640 1 -- 172.21.15.45:0/2802150859 >> 172.21.15.45:0/2802150859 conn(0x7f9c4c0fcb10 msgr2=0x7f9c4c100f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:28.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.143+0000 7f9c50a00640 1 -- 172.21.15.45:0/2802150859 shutdown_connections 2024-03-28T17:48:28.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.143+0000 7f9c50a00640 1 -- 172.21.15.45:0/2802150859 wait complete. 2024-03-28T17:48:28.164 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T17:48:28.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.388+0000 7fbc3e600640 1 -- 172.21.15.45:0/1657029703 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc38104ab0 msgr2=0x7fbc38104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:28.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.388+0000 7fbc3e600640 1 --2- 172.21.15.45:0/1657029703 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc38104ab0 0x7fbc38104f30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbc24009860 tx=0x7fbc2402bb70 comp rx=0 tx=0).stop 2024-03-28T17:48:28.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.389+0000 7fbc3e600640 1 -- 172.21.15.45:0/1657029703 shutdown_connections 2024-03-28T17:48:28.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.389+0000 7fbc3e600640 1 --2- 172.21.15.45:0/1657029703 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc38105470 0x7fbc3810bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:28.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.389+0000 7fbc3e600640 1 --2- 172.21.15.45:0/1657029703 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc38104ab0 0x7fbc38104f30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:28.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.389+0000 7fbc3e600640 1 --2- 172.21.15.45:0/1657029703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc381038c0 0x7fbc38103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:28.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.389+0000 7fbc3e600640 1 -- 172.21.15.45:0/1657029703 >> 172.21.15.45:0/1657029703 conn(0x7fbc380fee50 msgr2=0x7fbc38101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:28.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.389+0000 7fbc3e600640 1 -- 172.21.15.45:0/1657029703 shutdown_connections 2024-03-28T17:48:28.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.389+0000 7fbc3e600640 1 -- 172.21.15.45:0/1657029703 wait complete. 2024-03-28T17:48:28.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.390+0000 7fbc3e600640 1 Processor -- start 2024-03-28T17:48:28.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.390+0000 7fbc3e600640 1 -- start start 2024-03-28T17:48:28.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.390+0000 7fbc3e600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc381038c0 0x7fbc38199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:28.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.390+0000 7fbc3e600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc38104ab0 0x7fbc3819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:28.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.390+0000 7fbc3e600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc38105470 0x7fbc381a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:28.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.390+0000 7fbc3e600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3810f520 con 0x7fbc38104ab0 2024-03-28T17:48:28.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.390+0000 7fbc3e600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3810f3a0 con 0x7fbc381038c0 2024-03-28T17:48:28.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.390+0000 7fbc3e600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3810f6a0 con 0x7fbc38105470 2024-03-28T17:48:28.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.391+0000 7fbc37e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc38105470 0x7fbc381a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:28.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.391+0000 7fbc37400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc381038c0 0x7fbc38199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:28.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.391+0000 7fbc37e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc38105470 0x7fbc381a11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:41328/0 (socket says 172.21.15.45:41328) 2024-03-28T17:48:28.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.391+0000 7fbc37e00640 1 -- 172.21.15.45:0/93588854 learned_addr learned my addr 172.21.15.45:0/93588854 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:28.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.391+0000 7fbc36a00640 1 --2- 172.21.15.45:0/93588854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc38104ab0 0x7fbc3819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:28.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.391+0000 7fbc37400640 1 -- 172.21.15.45:0/93588854 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc38105470 msgr2=0x7fbc381a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:28.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.391+0000 7fbc37400640 1 --2- 172.21.15.45:0/93588854 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc38105470 0x7fbc381a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:28.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.391+0000 7fbc37400640 1 -- 172.21.15.45:0/93588854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc38104ab0 msgr2=0x7fbc3819a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:28.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.391+0000 7fbc37400640 1 --2- 172.21.15.45:0/93588854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc38104ab0 0x7fbc3819a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:28.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.391+0000 7fbc37400640 1 -- 172.21.15.45:0/93588854 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc381a1810 con 0x7fbc381038c0 2024-03-28T17:48:28.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.391+0000 7fbc37e00640 1 --2- 172.21.15.45:0/93588854 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc38105470 0x7fbc381a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:28.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.391+0000 7fbc36a00640 1 --2- 172.21.15.45:0/93588854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc38104ab0 0x7fbc3819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:28.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.392+0000 7fbc37400640 1 --2- 172.21.15.45:0/93588854 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc381038c0 0x7fbc38199bf0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbc38104910 tx=0x7fbc2800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:28.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.392+0000 7fbc2fe00640 1 -- 172.21.15.45:0/93588854 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28004440 con 0x7fbc381038c0 2024-03-28T17:48:28.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.392+0000 7fbc2fe00640 1 -- 172.21.15.45:0/93588854 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc2800cda0 con 0x7fbc381038c0 2024-03-28T17:48:28.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.392+0000 7fbc3e600640 1 -- 172.21.15.45:0/93588854 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc381a1b00 con 0x7fbc381038c0 2024-03-28T17:48:28.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.392+0000 7fbc2fe00640 1 -- 172.21.15.45:0/93588854 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc280196e0 con 0x7fbc381038c0 2024-03-28T17:48:28.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.392+0000 7fbc3e600640 1 -- 172.21.15.45:0/93588854 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc381a2070 con 0x7fbc381038c0 2024-03-28T17:48:28.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.393+0000 7fbc3e600640 1 -- 172.21.15.45:0/93588854 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc3810ac00 con 0x7fbc381038c0 2024-03-28T17:48:28.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.395+0000 7fbc2fe00640 1 -- 172.21.15.45:0/93588854 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbc28003780 con 0x7fbc381038c0 2024-03-28T17:48:28.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.396+0000 7fbc2fe00640 1 --2- 172.21.15.45:0/93588854 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fbc1c0a2880 0x7fbc1c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.397+0000 7fbc2fe00640 1 -- 172.21.15.45:0/93588854 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fbc280ceda0 con 0x7fbc381038c0 2024-03-28T17:48:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.397+0000 7fbc36a00640 1 --2- 172.21.15.45:0/93588854 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fbc1c0a2880 0x7fbc1c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.397+0000 7fbc36a00640 1 --2- 172.21.15.45:0/93588854 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fbc1c0a2880 0x7fbc1c0a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbc24009830 tx=0x7fbc2402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:28.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.404+0000 7fbc2fe00640 1 -- 172.21.15.45:0/93588854 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbc28003e20 con 0x7fbc381038c0 2024-03-28T17:48:28.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.605+0000 7fbc3e600640 1 -- 172.21.15.45:0/93588854 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbc38062340 con 0x7fbc381038c0 2024-03-28T17:48:28.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.606+0000 7fbc2fe00640 1 -- 172.21.15.45:0/93588854 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbc28009510 con 0x7fbc381038c0 2024-03-28T17:48:28.606 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-03-28T17:48:28.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.608+0000 7fbc3e600640 1 -- 172.21.15.45:0/93588854 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fbc1c0a2880 msgr2=0x7fbc1c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:28.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.608+0000 7fbc3e600640 1 --2- 172.21.15.45:0/93588854 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fbc1c0a2880 0x7fbc1c0a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbc24009830 tx=0x7fbc2402e040 comp rx=0 tx=0).stop 2024-03-28T17:48:28.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.608+0000 7fbc3e600640 1 -- 172.21.15.45:0/93588854 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc381038c0 msgr2=0x7fbc38199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:28.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.608+0000 7fbc3e600640 1 --2- 172.21.15.45:0/93588854 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc381038c0 0x7fbc38199bf0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbc38104910 tx=0x7fbc2800ee50 comp rx=0 tx=0).stop 2024-03-28T17:48:28.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.609+0000 7fbc3e600640 1 -- 172.21.15.45:0/93588854 shutdown_connections 2024-03-28T17:48:28.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.609+0000 7fbc3e600640 1 --2- 172.21.15.45:0/93588854 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fbc1c0a2880 0x7fbc1c0a4d40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:28.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.609+0000 7fbc3e600640 1 --2- 172.21.15.45:0/93588854 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc38105470 0x7fbc381a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:28.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.609+0000 7fbc3e600640 1 --2- 172.21.15.45:0/93588854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc38104ab0 0x7fbc3819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:28.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.609+0000 7fbc3e600640 1 --2- 172.21.15.45:0/93588854 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc381038c0 0x7fbc38199bf0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:28.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.609+0000 7fbc3e600640 1 -- 172.21.15.45:0/93588854 >> 172.21.15.45:0/93588854 conn(0x7fbc380fee50 msgr2=0x7fbc38107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:28.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.609+0000 7fbc3e600640 1 -- 172.21.15.45:0/93588854 shutdown_connections 2024-03-28T17:48:28.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.609+0000 7fbc3e600640 1 -- 172.21.15.45:0/93588854 wait complete. 2024-03-28T17:48:28.626 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T17:48:28.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:28 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1351026974' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:48:28.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:28 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/2802150859' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:48:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:28 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1351026974' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:48:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:28 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/2802150859' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:48:28.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.844+0000 7f5289a00640 1 -- 172.21.15.45:0/2104793489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5284107380 msgr2=0x7f52841097d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:28.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.844+0000 7f5289a00640 1 --2- 172.21.15.45:0/2104793489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5284107380 0x7f52841097d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f527c009ca0 tx=0x7f527c02c100 comp rx=0 tx=0).stop 2024-03-28T17:48:28.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.844+0000 7f5289a00640 1 -- 172.21.15.45:0/2104793489 shutdown_connections 2024-03-28T17:48:28.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.844+0000 7f5289a00640 1 --2- 172.21.15.45:0/2104793489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5284107380 0x7f52841097d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:28.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.844+0000 7f5289a00640 1 --2- 172.21.15.45:0/2104793489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5284104a50 0x7f5284106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:28.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.844+0000 7f5289a00640 1 --2- 172.21.15.45:0/2104793489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5284102120 0x7f5284104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:28.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.844+0000 7f5289a00640 1 -- 172.21.15.45:0/2104793489 >> 172.21.15.45:0/2104793489 conn(0x7f52840fbb10 msgr2=0x7f52840fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:28.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.845+0000 7f5289a00640 1 -- 172.21.15.45:0/2104793489 shutdown_connections 2024-03-28T17:48:28.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.845+0000 7f5289a00640 1 -- 172.21.15.45:0/2104793489 wait complete. 2024-03-28T17:48:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.845+0000 7f5289a00640 1 Processor -- start 2024-03-28T17:48:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.846+0000 7f5289a00640 1 -- start start 2024-03-28T17:48:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.846+0000 7f5289a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5284102120 0x7f5284197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.846+0000 7f5289a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5284104a50 0x7f5284197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.846+0000 7f5289a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5284107380 0x7f528419ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.846+0000 7f5289a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528410d340 con 0x7f5284104a50 2024-03-28T17:48:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.846+0000 7f5289a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528410d1c0 con 0x7f5284107380 2024-03-28T17:48:28.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.846+0000 7f5289a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528410d4c0 con 0x7f5284102120 2024-03-28T17:48:28.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.846+0000 7f5282a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5284102120 0x7f5284197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:28.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.846+0000 7f5282000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5284104a50 0x7f5284197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:28.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.846+0000 7f5282a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5284102120 0x7f5284197980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:41352/0 (socket says 172.21.15.45:41352) 2024-03-28T17:48:28.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.846+0000 7f5282a00640 1 -- 172.21.15.45:0/1785371988 learned_addr learned my addr 172.21.15.45:0/1785371988 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:28.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.846+0000 7f5283400640 1 --2- 172.21.15.45:0/1785371988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5284107380 0x7f528419ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:28.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.847+0000 7f5283400640 1 -- 172.21.15.45:0/1785371988 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5284102120 msgr2=0x7f5284197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:28.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.847+0000 7f5283400640 1 --2- 172.21.15.45:0/1785371988 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5284102120 0x7f5284197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:28.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.847+0000 7f5283400640 1 -- 172.21.15.45:0/1785371988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5284104a50 msgr2=0x7f5284197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:28.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.847+0000 7f5283400640 1 --2- 172.21.15.45:0/1785371988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5284104a50 0x7f5284197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:28.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.847+0000 7f5283400640 1 -- 172.21.15.45:0/1785371988 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528419f5a0 con 0x7f5284107380 2024-03-28T17:48:28.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.847+0000 7f5282a00640 1 --2- 172.21.15.45:0/1785371988 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5284102120 0x7f5284197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:28.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.847+0000 7f5282000640 1 --2- 172.21.15.45:0/1785371988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5284104a50 0x7f5284197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:28.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.847+0000 7f5283400640 1 --2- 172.21.15.45:0/1785371988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5284107380 0x7f528419ef30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f527c037040 tx=0x7f527c035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:28.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.847+0000 7f527b400640 1 -- 172.21.15.45:0/1785371988 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527c039040 con 0x7f5284107380 2024-03-28T17:48:28.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.847+0000 7f527b400640 1 -- 172.21.15.45:0/1785371988 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f527c044070 con 0x7f5284107380 2024-03-28T17:48:28.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.847+0000 7f5289a00640 1 -- 172.21.15.45:0/1785371988 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f528419f830 con 0x7f5284107380 2024-03-28T17:48:28.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.847+0000 7f527b400640 1 -- 172.21.15.45:0/1785371988 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527c004350 con 0x7f5284107380 2024-03-28T17:48:28.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.847+0000 7f5289a00640 1 -- 172.21.15.45:0/1785371988 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f528419fcf0 con 0x7f5284107380 2024-03-28T17:48:28.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.848+0000 7f5289a00640 1 -- 172.21.15.45:0/1785371988 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52840fd600 con 0x7f5284107380 2024-03-28T17:48:28.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.850+0000 7f527b400640 1 -- 172.21.15.45:0/1785371988 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f527c0044f0 con 0x7f5284107380 2024-03-28T17:48:28.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.851+0000 7f527b400640 1 --2- 172.21.15.45:0/1785371988 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f52680a2800 0x7f52680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:28.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.852+0000 7f527b400640 1 -- 172.21.15.45:0/1785371988 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f527c0f0fc0 con 0x7f5284107380 2024-03-28T17:48:28.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.852+0000 7f5282a00640 1 --2- 172.21.15.45:0/1785371988 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f52680a2800 0x7f52680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:28.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.852+0000 7f5282a00640 1 --2- 172.21.15.45:0/1785371988 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f52680a2800 0x7f52680a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5274011550 tx=0x7f5274005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:28.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:28.855+0000 7f527b400640 1 -- 172.21.15.45:0/1785371988 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f527c0be010 con 0x7f5284107380 2024-03-28T17:48:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:28 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1351026974' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:48:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:28 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/2802150859' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:48:29.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.059+0000 7f5289a00640 1 -- 172.21.15.45:0/1785371988 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f52840622a0 con 0x7f5284107380 2024-03-28T17:48:29.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.060+0000 7f527b400640 1 -- 172.21.15.45:0/1785371988 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f527c0098b0 con 0x7f5284107380 2024-03-28T17:48:29.060 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-03-28T17:48:29.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.063+0000 7f5289a00640 1 -- 172.21.15.45:0/1785371988 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f52680a2800 msgr2=0x7f52680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:29.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.063+0000 7f5289a00640 1 --2- 172.21.15.45:0/1785371988 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f52680a2800 0x7f52680a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5274011550 tx=0x7f5274005f90 comp rx=0 tx=0).stop 2024-03-28T17:48:29.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.063+0000 7f5289a00640 1 -- 172.21.15.45:0/1785371988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5284107380 msgr2=0x7f528419ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:29.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.063+0000 7f5289a00640 1 --2- 172.21.15.45:0/1785371988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5284107380 0x7f528419ef30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f527c037040 tx=0x7f527c035d10 comp rx=0 tx=0).stop 2024-03-28T17:48:29.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.063+0000 7f5289a00640 1 -- 172.21.15.45:0/1785371988 shutdown_connections 2024-03-28T17:48:29.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.063+0000 7f5289a00640 1 --2- 172.21.15.45:0/1785371988 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f52680a2800 0x7f52680a4cc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:29.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.063+0000 7f5289a00640 1 --2- 172.21.15.45:0/1785371988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5284107380 0x7f528419ef30 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:29.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.063+0000 7f5289a00640 1 --2- 172.21.15.45:0/1785371988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5284104a50 0x7f5284197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:29.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.063+0000 7f5289a00640 1 --2- 172.21.15.45:0/1785371988 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5284102120 0x7f5284197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:29.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.063+0000 7f5289a00640 1 -- 172.21.15.45:0/1785371988 >> 172.21.15.45:0/1785371988 conn(0x7f52840fbb10 msgr2=0x7f5284102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:29.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.063+0000 7f5289a00640 1 -- 172.21.15.45:0/1785371988 shutdown_connections 2024-03-28T17:48:29.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.063+0000 7f5289a00640 1 -- 172.21.15.45:0/1785371988 wait complete. 2024-03-28T17:48:29.084 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:48:29.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.300+0000 7f2e01800640 1 -- 172.21.15.45:0/3901359521 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2dfc1036f0 msgr2=0x7f2dfc103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:29.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.300+0000 7f2e01800640 1 --2- 172.21.15.45:0/3901359521 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2dfc1036f0 0x7f2dfc103b10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2dec009900 tx=0x7f2dec02bc30 comp rx=0 tx=0).stop 2024-03-28T17:48:29.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.300+0000 7f2e01800640 1 -- 172.21.15.45:0/3901359521 shutdown_connections 2024-03-28T17:48:29.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.300+0000 7f2e01800640 1 --2- 172.21.15.45:0/3901359521 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2dfc1052a0 0x7f2dfc10bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:29.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.300+0000 7f2e01800640 1 --2- 172.21.15.45:0/3901359521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dfc1048e0 0x7f2dfc104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:29.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.300+0000 7f2e01800640 1 --2- 172.21.15.45:0/3901359521 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2dfc1036f0 0x7f2dfc103b10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:29.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.300+0000 7f2e01800640 1 -- 172.21.15.45:0/3901359521 >> 172.21.15.45:0/3901359521 conn(0x7f2dfc0fec80 msgr2=0x7f2dfc1010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:29.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.300+0000 7f2e01800640 1 -- 172.21.15.45:0/3901359521 shutdown_connections 2024-03-28T17:48:29.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.300+0000 7f2e01800640 1 -- 172.21.15.45:0/3901359521 wait complete. 2024-03-28T17:48:29.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.301+0000 7f2e01800640 1 Processor -- start 2024-03-28T17:48:29.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.302+0000 7f2e01800640 1 -- start start 2024-03-28T17:48:29.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.302+0000 7f2e01800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2dfc1036f0 0x7f2dfc1999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:29.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.302+0000 7f2e01800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dfc1048e0 0x7f2dfc199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:29.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.302+0000 7f2e01800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2dfc1052a0 0x7f2dfc1a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:29.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.302+0000 7f2e01800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc10f3c0 con 0x7f2dfc1048e0 2024-03-28T17:48:29.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.302+0000 7f2e01800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc10f240 con 0x7f2dfc1052a0 2024-03-28T17:48:29.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.302+0000 7f2e01800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc10f540 con 0x7f2dfc1036f0 2024-03-28T17:48:29.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.302+0000 7f2dfbe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2dfc1036f0 0x7f2dfc1999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:29.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.302+0000 7f2df3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dfc1048e0 0x7f2dfc199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:29.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.302+0000 7f2dfbe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2dfc1036f0 0x7f2dfc1999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:41376/0 (socket says 172.21.15.45:41376) 2024-03-28T17:48:29.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.302+0000 7f2dfbe00640 1 -- 172.21.15.45:0/661021843 learned_addr learned my addr 172.21.15.45:0/661021843 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:29.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.302+0000 7f2e00e00640 1 --2- 172.21.15.45:0/661021843 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2dfc1052a0 0x7f2dfc1a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:29.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.303+0000 7f2dfbe00640 1 -- 172.21.15.45:0/661021843 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2dfc1052a0 msgr2=0x7f2dfc1a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:29.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.303+0000 7f2dfbe00640 1 --2- 172.21.15.45:0/661021843 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2dfc1052a0 0x7f2dfc1a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:29.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.303+0000 7f2dfbe00640 1 -- 172.21.15.45:0/661021843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dfc1048e0 msgr2=0x7f2dfc199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:29.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.303+0000 7f2dfbe00640 1 --2- 172.21.15.45:0/661021843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dfc1048e0 0x7f2dfc199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:29.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.303+0000 7f2dfbe00640 1 -- 172.21.15.45:0/661021843 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dfc1a15f0 con 0x7f2dfc1036f0 2024-03-28T17:48:29.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.303+0000 7f2dfbe00640 1 --2- 172.21.15.45:0/661021843 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2dfc1036f0 0x7f2dfc1999d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2dec000c00 tx=0x7f2dec02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:29.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.303+0000 7f2df9600640 1 -- 172.21.15.45:0/661021843 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec043020 con 0x7f2dfc1036f0 2024-03-28T17:48:29.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.303+0000 7f2df9600640 1 -- 172.21.15.45:0/661021843 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2dec0054e0 con 0x7f2dfc1036f0 2024-03-28T17:48:29.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.303+0000 7f2e01800640 1 -- 172.21.15.45:0/661021843 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dfc1a1880 con 0x7f2dfc1036f0 2024-03-28T17:48:29.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.303+0000 7f2df9600640 1 -- 172.21.15.45:0/661021843 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec003c40 con 0x7f2dfc1036f0 2024-03-28T17:48:29.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.303+0000 7f2e01800640 1 -- 172.21.15.45:0/661021843 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dfc1a1dd0 con 0x7f2dfc1036f0 2024-03-28T17:48:29.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.304+0000 7f2e01800640 1 -- 172.21.15.45:0/661021843 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dfc10a9b0 con 0x7f2dfc1036f0 2024-03-28T17:48:29.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.307+0000 7f2df9600640 1 -- 172.21.15.45:0/661021843 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2dec046020 con 0x7f2dfc1036f0 2024-03-28T17:48:29.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.308+0000 7f2df9600640 1 --2- 172.21.15.45:0/661021843 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2de00a2800 0x7f2de00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:29.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.308+0000 7f2df9600640 1 -- 172.21.15.45:0/661021843 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f2dec0f0e30 con 0x7f2dfc1036f0 2024-03-28T17:48:29.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.308+0000 7f2df3e00640 1 --2- 172.21.15.45:0/661021843 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2de00a2800 0x7f2de00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:29.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.309+0000 7f2df3e00640 1 --2- 172.21.15.45:0/661021843 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2de00a2800 0x7f2de00a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc19aee0 tx=0x7f2de4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:29.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.314+0000 7f2df9600640 1 -- 172.21.15.45:0/661021843 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2dec04d020 con 0x7f2dfc1036f0 2024-03-28T17:48:29.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.560+0000 7f2e01800640 1 -- 172.21.15.45:0/661021843 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2dfc02d070 con 0x7f2dfc1036f0 2024-03-28T17:48:29.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.561+0000 7f2df9600640 1 -- 172.21.15.45:0/661021843 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f2dec0bdee0 con 0x7f2dfc1036f0 2024-03-28T17:48:29.562 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:48:29.562 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6839/2313500150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":2313500150},{"type":"v1","addr":"172.21.15.45:6839","nonce":2313500150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6841/2647972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":2647972571},{"type":"v1","addr":"172.21.15.45:6841","nonce":2647972571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T17:46:12.261135+0000","modified":"2024-03-28T17:46:17.967524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24584,"mds_1":24530,"mds_2":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6835/2876354410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":2876354410},{"type":"v1","addr":"172.21.15.151:6835","nonce":2876354410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24500":{"gid":24500,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/4234998385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":4234998385},{"type":"v1","addr":"172.21.15.71:6835","nonce":4234998385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6837/1238964845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":1238964845},{"type":"v1","addr":"172.21.15.151:6837","nonce":1238964845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.151:6839/3003564901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6838","nonce":3003564901},{"type":"v1","addr":"172.21.15.151:6839","nonce":3003564901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6837/2008752655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2008752655},{"type":"v1","addr":"172.21.15.71:6837","nonce":2008752655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.71:6839/2709938800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":2709938800},{"type":"v1","addr":"172.21.15.71:6839","nonce":2709938800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24584,"qdb_cluster":[24529,24530,24584]},"id":1}]} 2024-03-28T17:48:29.562 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 22 2024-03-28T17:48:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.564+0000 7f2e01800640 1 -- 172.21.15.45:0/661021843 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2de00a2800 msgr2=0x7f2de00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.564+0000 7f2e01800640 1 --2- 172.21.15.45:0/661021843 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2de00a2800 0x7f2de00a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc19aee0 tx=0x7f2de4005f90 comp rx=0 tx=0).stop 2024-03-28T17:48:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.564+0000 7f2e01800640 1 -- 172.21.15.45:0/661021843 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2dfc1036f0 msgr2=0x7f2dfc1999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.564+0000 7f2e01800640 1 --2- 172.21.15.45:0/661021843 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2dfc1036f0 0x7f2dfc1999d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2dec000c00 tx=0x7f2dec02c650 comp rx=0 tx=0).stop 2024-03-28T17:48:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.564+0000 7f2e01800640 1 -- 172.21.15.45:0/661021843 shutdown_connections 2024-03-28T17:48:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.564+0000 7f2e01800640 1 --2- 172.21.15.45:0/661021843 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2de00a2800 0x7f2de00a4cc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.564+0000 7f2e01800640 1 --2- 172.21.15.45:0/661021843 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2dfc1052a0 0x7f2dfc1a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.564+0000 7f2e01800640 1 --2- 172.21.15.45:0/661021843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dfc1048e0 0x7f2dfc199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.564+0000 7f2e01800640 1 --2- 172.21.15.45:0/661021843 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2dfc1036f0 0x7f2dfc1999d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:29.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.565+0000 7f2e01800640 1 -- 172.21.15.45:0/661021843 >> 172.21.15.45:0/661021843 conn(0x7f2dfc0fec80 msgr2=0x7f2dfc107b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:29.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.565+0000 7f2e01800640 1 -- 172.21.15.45:0/661021843 shutdown_connections 2024-03-28T17:48:29.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.565+0000 7f2e01800640 1 -- 172.21.15.45:0/661021843 wait complete. 2024-03-28T17:48:29.583 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T17:48:29.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.805+0000 7ff147e00640 1 -- 172.21.15.45:0/1449874337 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1481096b0 msgr2=0x7ff14810ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:29.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.805+0000 7ff147e00640 1 --2- 172.21.15.45:0/1449874337 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1481096b0 0x7ff14810ff40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff140009ca0 tx=0x7ff14002c100 comp rx=0 tx=0).stop 2024-03-28T17:48:29.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.806+0000 7ff147e00640 1 -- 172.21.15.45:0/1449874337 shutdown_connections 2024-03-28T17:48:29.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.806+0000 7ff147e00640 1 --2- 172.21.15.45:0/1449874337 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1481096b0 0x7ff14810ff40 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:29.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.806+0000 7ff147e00640 1 --2- 172.21.15.45:0/1449874337 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff148108cf0 0x7ff148109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:29.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.806+0000 7ff147e00640 1 --2- 172.21.15.45:0/1449874337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff148107b00 0x7ff148107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:29.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.806+0000 7ff147e00640 1 -- 172.21.15.45:0/1449874337 >> 172.21.15.45:0/1449874337 conn(0x7ff148076ab0 msgr2=0x7ff148078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:29.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.806+0000 7ff147e00640 1 -- 172.21.15.45:0/1449874337 shutdown_connections 2024-03-28T17:48:29.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.806+0000 7ff147e00640 1 -- 172.21.15.45:0/1449874337 wait complete. 2024-03-28T17:48:29.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.807+0000 7ff147e00640 1 Processor -- start 2024-03-28T17:48:29.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.807+0000 7ff147e00640 1 -- start start 2024-03-28T17:48:29.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.808+0000 7ff147e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff148107b00 0x7ff14819dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:29.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.808+0000 7ff147e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff148108cf0 0x7ff14819e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:29.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.808+0000 7ff147e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1481096b0 0x7ff1481a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:29.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.808+0000 7ff147e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff148113780 con 0x7ff148107b00 2024-03-28T17:48:29.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.808+0000 7ff147e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff148113600 con 0x7ff1481096b0 2024-03-28T17:48:29.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.808+0000 7ff147e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff148113900 con 0x7ff148108cf0 2024-03-28T17:48:29.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.808+0000 7ff147400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1481096b0 0x7ff1481a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:29.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.808+0000 7ff13fe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff148108cf0 0x7ff14819e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:29.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.808+0000 7ff147400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1481096b0 0x7ff1481a52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53562/0 (socket says 172.21.15.45:53562) 2024-03-28T17:48:29.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.808+0000 7ff147400640 1 -- 172.21.15.45:0/37716806 learned_addr learned my addr 172.21.15.45:0/37716806 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:29.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.808+0000 7ff146a00640 1 --2- 172.21.15.45:0/37716806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff148107b00 0x7ff14819dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:29.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.808+0000 7ff147400640 1 -- 172.21.15.45:0/37716806 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff148108cf0 msgr2=0x7ff14819e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:29.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.808+0000 7ff147400640 1 --2- 172.21.15.45:0/37716806 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff148108cf0 0x7ff14819e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:29.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.808+0000 7ff147400640 1 -- 172.21.15.45:0/37716806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff148107b00 msgr2=0x7ff14819dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:29.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.808+0000 7ff147400640 1 --2- 172.21.15.45:0/37716806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff148107b00 0x7ff14819dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:29.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.808+0000 7ff147400640 1 -- 172.21.15.45:0/37716806 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1481a5940 con 0x7ff1481096b0 2024-03-28T17:48:29.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.808+0000 7ff146a00640 1 --2- 172.21.15.45:0/37716806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff148107b00 0x7ff14819dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T17:48:29.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.809+0000 7ff147400640 1 --2- 172.21.15.45:0/37716806 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1481096b0 0x7ff1481a52a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff14002c640 tx=0x7ff140035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:29.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.809+0000 7ff14ca00640 1 -- 172.21.15.45:0/37716806 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff140039040 con 0x7ff1481096b0 2024-03-28T17:48:29.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.809+0000 7ff14ca00640 1 -- 172.21.15.45:0/37716806 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff140044070 con 0x7ff1481096b0 2024-03-28T17:48:29.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.809+0000 7ff14ca00640 1 -- 172.21.15.45:0/37716806 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff140004460 con 0x7ff1481096b0 2024-03-28T17:48:29.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.809+0000 7ff147e00640 1 -- 172.21.15.45:0/37716806 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1481a5bd0 con 0x7ff1481096b0 2024-03-28T17:48:29.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.809+0000 7ff147e00640 1 -- 172.21.15.45:0/37716806 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1481a6110 con 0x7ff1481096b0 2024-03-28T17:48:29.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:29 smithi071 ceph-mon[36597]: pgmap v387: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:29.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:29 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/93588854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T17:48:29.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:29 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1785371988' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T17:48:29.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.810+0000 7ff147e00640 1 -- 172.21.15.45:0/37716806 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14810ed50 con 0x7ff1481096b0 2024-03-28T17:48:29.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.812+0000 7ff14ca00640 1 -- 172.21.15.45:0/37716806 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff140004600 con 0x7ff1481096b0 2024-03-28T17:48:29.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.813+0000 7ff14ca00640 1 --2- 172.21.15.45:0/37716806 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff12c0a2880 0x7ff12c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:29.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.813+0000 7ff14ca00640 1 -- 172.21.15.45:0/37716806 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff1400f11a0 con 0x7ff1481096b0 2024-03-28T17:48:29.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.814+0000 7ff146a00640 1 --2- 172.21.15.45:0/37716806 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff12c0a2880 0x7ff12c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:29.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.815+0000 7ff146a00640 1 --2- 172.21.15.45:0/37716806 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff12c0a2880 0x7ff12c0a4d40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff14819f200 tx=0x7ff138005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:29.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:29.817+0000 7ff14ca00640 1 -- 172.21.15.45:0/37716806 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff1400be1f0 con 0x7ff1481096b0 2024-03-28T17:48:29.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:29 smithi151 ceph-mon[36481]: pgmap v387: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:29.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:29 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/93588854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T17:48:29.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:29 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1785371988' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T17:48:29.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:29 smithi045 ceph-mon[31773]: pgmap v387: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:29.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:29 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/93588854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T17:48:29.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:29 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1785371988' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T17:48:30.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.023+0000 7ff147e00640 1 -- 172.21.15.45:0/37716806 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1480622a0 con 0x7ff1481096b0 2024-03-28T17:48:30.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.025+0000 7ff14ca00640 1 -- 172.21.15.45:0/37716806 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22583 (secure 0 0 0) 0x7ff1400098b0 con 0x7ff1481096b0 2024-03-28T17:48:30.025 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:48:30.026 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"9f4888ec-ed29-11ee-95d0-87774f69a715","created":"2024-03-28T17:36:51.623185+0000","modified":"2024-03-28T17:47:09.455843+0000","last_up_change":"2024-03-28T17:45:14.043432+0000","last_in_change":"2024-03-28T17:44:53.654489+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T17:40:54.208375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T17:46:10.564519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T17:46:11.734367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"74c56bc2-4169-4919-afd4-2ebd236f23e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6803","nonce":3795754023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6805","nonce":3795754023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6809","nonce":3795754023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6807","nonce":3795754023}]},"public_addr":"172.21.15.45:6803/3795754023","cluster_addr":"172.21.15.45:6805/3795754023","heartbeat_back_addr":"172.21.15.45:6809/3795754023","heartbeat_front_addr":"172.21.15.45:6807/3795754023","state":["exists","up"]},{"osd":1,"uuid":"78dcc21b-75b2-453d-b690-17b55b8ed1a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6801","nonce":2014529906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6803","nonce":2014529906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6807","nonce":2014529906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6805","nonce":2014529906}]},"public_addr":"172.21.15.71:6801/2014529906","cluster_addr":"172.21.15.71:6803/2014529906","heartbeat_back_addr":"172.21.15.71:6807/2014529906","heartbeat_front_addr":"172.21.15.71:6805/2014529906","state":["exists","up"]},{"osd":2,"uuid":"ca9f3f36-b71e-4962-a94f-cedf2f12eb7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6801","nonce":1850933775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6803","nonce":1850933775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6807","nonce":1850933775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6805","nonce":1850933775}]},"public_addr":"172.21.15.151:6801/1850933775","cluster_addr":"172.21.15.151:6803/1850933775","heartbeat_back_addr":"172.21.15.151:6807/1850933775","heartbeat_front_addr":"172.21.15.151:6805/1850933775","state":["exists","up"]},{"osd":3,"uuid":"d3f21f01-2973-413e-8daa-9974b8a44af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6811","nonce":3021569975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6813","nonce":3021569975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6817","nonce":3021569975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6815","nonce":3021569975}]},"public_addr":"172.21.15.45:6811/3021569975","cluster_addr":"172.21.15.45:6813/3021569975","heartbeat_back_addr":"172.21.15.45:6817/3021569975","heartbeat_front_addr":"172.21.15.45:6815/3021569975","state":["exists","up"]},{"osd":4,"uuid":"4a336f42-0605-492b-ad88-56cb02acdcf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6809","nonce":2069578333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6811","nonce":2069578333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6815","nonce":2069578333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6813","nonce":2069578333}]},"public_addr":"172.21.15.71:6809/2069578333","cluster_addr":"172.21.15.71:6811/2069578333","heartbeat_back_addr":"172.21.15.71:6815/2069578333","heartbeat_front_addr":"172.21.15.71:6813/2069578333","state":["exists","up"]},{"osd":5,"uuid":"c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6809","nonce":2629217227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6811","nonce":2629217227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6815","nonce":2629217227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6813","nonce":2629217227}]},"public_addr":"172.21.15.151:6809/2629217227","cluster_addr":"172.21.15.151:6811/2629217227","heartbeat_back_addr":"172.21.15.151:6815/2629217227","heartbeat_front_addr":"172.21.15.151:6813/2629217227","state":["exists","up"]},{"osd":6,"uuid":"58a6c5bd-a8b7-4892-b346-bcd7994bcb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6819","nonce":2077907039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6821","nonce":2077907039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6825","nonce":2077907039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6823","nonce":2077907039}]},"public_addr":"172.21.15.45:6819/2077907039","cluster_addr":"172.21.15.45:6821/2077907039","heartbeat_back_addr":"172.21.15.45:6825/2077907039","heartbeat_front_addr":"172.21.15.45:6823/2077907039","state":["exists","up"]},{"osd":7,"uuid":"f7a45d52-807b-4407-bb04-787b2a52a0ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6817","nonce":2000211251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6819","nonce":2000211251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6823","nonce":2000211251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6821","nonce":2000211251}]},"public_addr":"172.21.15.71:6817/2000211251","cluster_addr":"172.21.15.71:6819/2000211251","heartbeat_back_addr":"172.21.15.71:6823/2000211251","heartbeat_front_addr":"172.21.15.71:6821/2000211251","state":["exists","up"]},{"osd":8,"uuid":"88abcb0f-f189-49be-8315-c700c77369f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6817","nonce":3665460137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6819","nonce":3665460137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6823","nonce":3665460137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6821","nonce":3665460137}]},"public_addr":"172.21.15.151:6817/3665460137","cluster_addr":"172.21.15.151:6819/3665460137","heartbeat_back_addr":"172.21.15.151:6823/3665460137","heartbeat_front_addr":"172.21.15.151:6821/3665460137","state":["exists","up"]},{"osd":9,"uuid":"ed3fc096-961e-4392-abc2-c5aa4d050117","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6827","nonce":2122322684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6829","nonce":2122322684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6833","nonce":2122322684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6831","nonce":2122322684}]},"public_addr":"172.21.15.45:6827/2122322684","cluster_addr":"172.21.15.45:6829/2122322684","heartbeat_back_addr":"172.21.15.45:6833/2122322684","heartbeat_front_addr":"172.21.15.45:6831/2122322684","state":["exists","up"]},{"osd":10,"uuid":"ba94762b-bc5c-4c97-9cc1-17e5651c14b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6825","nonce":1099494037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6827","nonce":1099494037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6831","nonce":1099494037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6829","nonce":1099494037}]},"public_addr":"172.21.15.71:6825/1099494037","cluster_addr":"172.21.15.71:6827/1099494037","heartbeat_back_addr":"172.21.15.71:6831/1099494037","heartbeat_front_addr":"172.21.15.71:6829/1099494037","state":["exists","up"]},{"osd":11,"uuid":"1d178378-b9f5-40f6-99ef-3d3d445da640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6825","nonce":2941055364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6827","nonce":2941055364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6831","nonce":2941055364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6829","nonce":2941055364}]},"public_addr":"172.21.15.151:6825/2941055364","cluster_addr":"172.21.15.151:6827/2941055364","heartbeat_back_addr":"172.21.15.151:6831/2941055364","heartbeat_front_addr":"172.21.15.151:6829/2941055364","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:02.269586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:27.819386+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:51.552277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:25.052674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:51.873529+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:17.127867+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:51.588249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:18.381450+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:44.140534+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:17.634362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:45.469340+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:45:12.003433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3533611957":"2024-03-29T17:38:07.505611+0000","172.21.15.45:0/3625922430":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6801/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/482610468":"2024-03-29T17:37:32.424229+0000","172.21.15.45:6801/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3649804714":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/1198364079":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3211303958":"2024-03-29T17:38:07.505611+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T17:48:30.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.028+0000 7ff147e00640 1 -- 172.21.15.45:0/37716806 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff12c0a2880 msgr2=0x7ff12c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.028+0000 7ff147e00640 1 --2- 172.21.15.45:0/37716806 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff12c0a2880 0x7ff12c0a4d40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff14819f200 tx=0x7ff138005dd0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.028+0000 7ff147e00640 1 -- 172.21.15.45:0/37716806 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1481096b0 msgr2=0x7ff1481a52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.028+0000 7ff147e00640 1 --2- 172.21.15.45:0/37716806 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1481096b0 0x7ff1481a52a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff14002c640 tx=0x7ff140035d10 comp rx=0 tx=0).stop 2024-03-28T17:48:30.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.028+0000 7ff147e00640 1 -- 172.21.15.45:0/37716806 shutdown_connections 2024-03-28T17:48:30.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.028+0000 7ff147e00640 1 --2- 172.21.15.45:0/37716806 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff12c0a2880 0x7ff12c0a4d40 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.028+0000 7ff147e00640 1 --2- 172.21.15.45:0/37716806 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1481096b0 0x7ff1481a52a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.028+0000 7ff147e00640 1 --2- 172.21.15.45:0/37716806 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff148108cf0 0x7ff14819e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.028+0000 7ff147e00640 1 --2- 172.21.15.45:0/37716806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff148107b00 0x7ff14819dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.028+0000 7ff147e00640 1 -- 172.21.15.45:0/37716806 >> 172.21.15.45:0/37716806 conn(0x7ff148076ab0 msgr2=0x7ff14810bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:30.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.028+0000 7ff147e00640 1 -- 172.21.15.45:0/37716806 shutdown_connections 2024-03-28T17:48:30.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.028+0000 7ff147e00640 1 -- 172.21.15.45:0/37716806 wait complete. 2024-03-28T17:48:30.048 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T17:46:12.261135+0000', 'modified': '2024-03-28T17:46:17.967524+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24584, 'mds_1': 24530, 'mds_2': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24499': {'gid': 24499, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.151:6835/2876354410', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6834', 'nonce': 2876354410}, {'type': 'v1', 'addr': '172.21.15.151:6835', 'nonce': 2876354410}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24500': {'gid': 24500, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.71:6835/4234998385', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6834', 'nonce': 4234998385}, {'type': 'v1', 'addr': '172.21.15.71:6835', 'nonce': 4234998385}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24511': {'gid': 24511, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.151:6837/1238964845', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6836', 'nonce': 1238964845}, {'type': 'v1', 'addr': '172.21.15.151:6837', 'nonce': 1238964845}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24529': {'gid': 24529, 'name': 'i', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.151:6839/3003564901', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6838', 'nonce': 3003564901}, {'type': 'v1', 'addr': '172.21.15.151:6839', 'nonce': 3003564901}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24530': {'gid': 24530, 'name': 'h', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6837/2008752655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6836', 'nonce': 2008752655}, {'type': 'v1', 'addr': '172.21.15.71:6837', 'nonce': 2008752655}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24584': {'gid': 24584, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.71:6839/2709938800', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6838', 'nonce': 2709938800}, {'type': 'v1', 'addr': '172.21.15.71:6839', 'nonce': 2709938800}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24584, 'qdb_cluster': [24529, 24530, 24584]} 2024-03-28T17:48:30.048 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-28T17:48:30.048 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-28T17:48:30.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.269+0000 7f371f200640 1 -- 172.21.15.45:0/861689719 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37200ff860 msgr2=0x7f37200ffc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.269+0000 7f371f200640 1 --2- 172.21.15.45:0/861689719 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37200ff860 0x7f37200ffc80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3710009860 tx=0x7f371002bb70 comp rx=0 tx=0).stop 2024-03-28T17:48:30.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.269+0000 7f371f200640 1 -- 172.21.15.45:0/861689719 shutdown_connections 2024-03-28T17:48:30.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.269+0000 7f371f200640 1 --2- 172.21.15.45:0/861689719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3720100f80 0x7f3720101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.269+0000 7f371f200640 1 --2- 172.21.15.45:0/861689719 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37201005c0 0x7f3720100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.269+0000 7f371f200640 1 --2- 172.21.15.45:0/861689719 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37200ff860 0x7f37200ffc80 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.269+0000 7f371f200640 1 -- 172.21.15.45:0/861689719 >> 172.21.15.45:0/861689719 conn(0x7f37200fb190 msgr2=0x7f37200fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:30.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.269+0000 7f371f200640 1 -- 172.21.15.45:0/861689719 shutdown_connections 2024-03-28T17:48:30.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.269+0000 7f371f200640 1 -- 172.21.15.45:0/861689719 wait complete. 2024-03-28T17:48:30.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.270+0000 7f371f200640 1 Processor -- start 2024-03-28T17:48:30.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.270+0000 7f371f200640 1 -- start start 2024-03-28T17:48:30.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.271+0000 7f371f200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37200ff860 0x7f3720199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.271+0000 7f371f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37201005c0 0x7f372019a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.271+0000 7f371f200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3720100f80 0x7f37201a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.271+0000 7f371f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372010f410 con 0x7f37201005c0 2024-03-28T17:48:30.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.271+0000 7f371f200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372010f290 con 0x7f37200ff860 2024-03-28T17:48:30.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.271+0000 7f371f200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372010f590 con 0x7f3720100f80 2024-03-28T17:48:30.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.271+0000 7f371de00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37200ff860 0x7f3720199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.271+0000 7f371e800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3720100f80 0x7f37201a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.271+0000 7f371de00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37200ff860 0x7f3720199ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53576/0 (socket says 172.21.15.45:53576) 2024-03-28T17:48:30.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.271+0000 7f371de00640 1 -- 172.21.15.45:0/4153820627 learned_addr learned my addr 172.21.15.45:0/4153820627 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:30.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.271+0000 7f371d400640 1 --2- 172.21.15.45:0/4153820627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37201005c0 0x7f372019a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.271+0000 7f371de00640 1 -- 172.21.15.45:0/4153820627 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3720100f80 msgr2=0x7f37201a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.271+0000 7f371de00640 1 --2- 172.21.15.45:0/4153820627 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3720100f80 0x7f37201a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.271+0000 7f371de00640 1 -- 172.21.15.45:0/4153820627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37201005c0 msgr2=0x7f372019a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.271+0000 7f371de00640 1 --2- 172.21.15.45:0/4153820627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37201005c0 0x7f372019a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.271+0000 7f371de00640 1 -- 172.21.15.45:0/4153820627 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37201a16e0 con 0x7f37200ff860 2024-03-28T17:48:30.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.271+0000 7f371e800640 1 --2- 172.21.15.45:0/4153820627 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3720100f80 0x7f37201a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:30.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.272+0000 7f371de00640 1 --2- 172.21.15.45:0/4153820627 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37200ff860 0x7f3720199ac0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f37100349e0 tx=0x7f3710032450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:30.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.272+0000 7f3716a00640 1 -- 172.21.15.45:0/4153820627 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3710041030 con 0x7f37200ff860 2024-03-28T17:48:30.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.272+0000 7f3716a00640 1 -- 172.21.15.45:0/4153820627 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f371002c510 con 0x7f37200ff860 2024-03-28T17:48:30.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.272+0000 7f3716a00640 1 -- 172.21.15.45:0/4153820627 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3710039040 con 0x7f37200ff860 2024-03-28T17:48:30.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.272+0000 7f371f200640 1 -- 172.21.15.45:0/4153820627 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37201a1970 con 0x7f37200ff860 2024-03-28T17:48:30.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.272+0000 7f371f200640 1 -- 172.21.15.45:0/4153820627 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37201a1eb0 con 0x7f37200ff860 2024-03-28T17:48:30.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.277+0000 7f3716a00640 1 -- 172.21.15.45:0/4153820627 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3710044070 con 0x7f37200ff860 2024-03-28T17:48:30.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.277+0000 7f37277ad740 1 Processor -- start 2024-03-28T17:48:30.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.277+0000 7f37277ad740 1 -- start start 2024-03-28T17:48:30.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.278+0000 7f37277ad740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c3d80 0x55f4a19c41a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.278+0000 7f37277ad740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f4a19c4770 0x55f4a19c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.278+0000 7f37277ad740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c98e0 0x55f4a19cbd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.278+0000 7f37277ad740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4a1928c90 con 0x55f4a19c4770 2024-03-28T17:48:30.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.278+0000 7f37277ad740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4a18d1980 con 0x55f4a19c98e0 2024-03-28T17:48:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.278+0000 7f37277ad740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4a1924d30 con 0x55f4a19c3d80 2024-03-28T17:48:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.278+0000 7f371e800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c3d80 0x55f4a19c41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.278+0000 7f371e800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c3d80 0x55f4a19c41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:41404/0 (socket says 172.21.15.45:41404) 2024-03-28T17:48:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.278+0000 7f371e800640 1 -- 172.21.15.45:0/1170386431 learned_addr learned my addr 172.21.15.45:0/1170386431 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.278+0000 7f371d400640 1 --2- 172.21.15.45:0/1170386431 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c98e0 0x55f4a19cbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.278+0000 7f371de00640 1 --2- 172.21.15.45:0/1170386431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f4a19c4770 0x55f4a19c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.278+0000 7f371e800640 1 -- 172.21.15.45:0/1170386431 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c98e0 msgr2=0x55f4a19cbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.278+0000 7f371e800640 1 --2- 172.21.15.45:0/1170386431 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c98e0 0x55f4a19cbd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.278+0000 7f371e800640 1 -- 172.21.15.45:0/1170386431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f4a19c4770 msgr2=0x55f4a19c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.278+0000 7f371e800640 1 --2- 172.21.15.45:0/1170386431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f4a19c4770 0x55f4a19c9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.278+0000 7f371e800640 1 -- 172.21.15.45:0/1170386431 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f4a18e9510 con 0x55f4a19c3d80 2024-03-28T17:48:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.278+0000 7f371d400640 1 --2- 172.21.15.45:0/1170386431 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c98e0 0x55f4a19cbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.279+0000 7f371e800640 1 --2- 172.21.15.45:0/1170386431 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c3d80 0x55f4a19c41a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3718011810 tx=0x7f3718011cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edb4f19bcdf4710f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.279+0000 7f3716a00640 1 --2- 172.21.15.45:0/4153820627 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f37040a2880 0x7f37040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.279+0000 7f370b000640 1 -- 172.21.15.45:0/1170386431 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3718011310 con 0x55f4a19c3d80 2024-03-28T17:48:30.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.279+0000 7f370b000640 1 -- 172.21.15.45:0/1170386431 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f371800f9a0 con 0x55f4a19c3d80 2024-03-28T17:48:30.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.279+0000 7f370b000640 1 -- 172.21.15.45:0/1170386431 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3718005720 con 0x55f4a19c3d80 2024-03-28T17:48:30.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.279+0000 7f3716a00640 1 -- 172.21.15.45:0/4153820627 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f37100eff80 con 0x7f37200ff860 2024-03-28T17:48:30.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.279+0000 7f371de00640 1 --2- 172.21.15.45:0/4153820627 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f37040a2880 0x7f37040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.280+0000 7f37277ad740 1 -- 172.21.15.45:0/1170386431 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c3d80 msgr2=0x55f4a19c41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.280+0000 7f37277ad740 1 --2- 172.21.15.45:0/1170386431 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c3d80 0x55f4a19c41a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3718011810 tx=0x7f3718011cd0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.280+0000 7f37277ad740 1 -- 172.21.15.45:0/1170386431 shutdown_connections 2024-03-28T17:48:30.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.280+0000 7f37277ad740 1 --2- 172.21.15.45:0/1170386431 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c98e0 0x55f4a19cbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.280+0000 7f37277ad740 1 --2- 172.21.15.45:0/1170386431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f4a19c4770 0x55f4a19c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.280+0000 7f37277ad740 1 --2- 172.21.15.45:0/1170386431 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c3d80 0x55f4a19c41a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.280+0000 7f37277ad740 1 -- 172.21.15.45:0/1170386431 >> 172.21.15.45:0/1170386431 conn(0x55f4a19bf2f0 msgr2=0x55f4a19bf6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:30.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.280+0000 7f371de00640 1 --2- 172.21.15.45:0/4153820627 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f37040a2880 0x7f37040a4d40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f37100bc860 tx=0x7f3710005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:30.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.280+0000 7f37277ad740 1 -- 172.21.15.45:0/1170386431 shutdown_connections 2024-03-28T17:48:30.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.280+0000 7f37277ad740 1 -- 172.21.15.45:0/1170386431 wait complete. 2024-03-28T17:48:30.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.280+0000 7f37277ad740 1 -- start start 2024-03-28T17:48:30.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.280+0000 7f37277ad740 1 Processor -- start 2024-03-28T17:48:30.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.281+0000 7f37277ad740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c3d80 0x55f4a19ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.281+0000 7f37277ad740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c4770 0x55f4a19ce830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.281+0000 7f37277ad740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f4a19c98e0 0x55f4a19d5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.281+0000 7f37277ad740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4a19cc890 con 0x55f4a19c98e0 2024-03-28T17:48:30.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.281+0000 7f37277ad740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4a18e4dd0 con 0x55f4a19c4770 2024-03-28T17:48:30.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.281+0000 7f37277ad740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4a19cca10 con 0x55f4a19c3d80 2024-03-28T17:48:30.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.281+0000 7f37277ad740 10 client.0 _ll_register_callbacks cb 0x55f4a163cac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:48:30.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.281+0000 7f371e800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c3d80 0x55f4a19ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.281+0000 7f371d400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c4770 0x55f4a19ce830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.281+0000 7f371e800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c3d80 0x55f4a19ce2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:41414/0 (socket says 172.21.15.45:41414) 2024-03-28T17:48:30.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.281+0000 7f371e800640 1 -- 172.21.15.45:0/3147310893 learned_addr learned my addr 172.21.15.45:0/3147310893 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:30.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.281+0000 7f371e800640 1 --2- 172.21.15.45:0/3147310893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f4a19c98e0 0x55f4a19d5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.282+0000 7f371d400640 1 -- 172.21.15.45:0/3147310893 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c3d80 msgr2=0x55f4a19ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.282+0000 7f371d400640 1 --2- 172.21.15.45:0/3147310893 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c3d80 0x55f4a19ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.282+0000 7f371d400640 1 -- 172.21.15.45:0/3147310893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f4a19c98e0 msgr2=0x55f4a19d5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.282+0000 7f371d400640 1 --2- 172.21.15.45:0/3147310893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f4a19c98e0 0x55f4a19d5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.282+0000 7f371d400640 1 -- 172.21.15.45:0/3147310893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f4a163d2d0 con 0x55f4a19c4770 2024-03-28T17:48:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.282+0000 7f371e800640 1 --2- 172.21.15.45:0/3147310893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f4a19c98e0 0x55f4a19d5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:48:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.282+0000 7f371d400640 1 --2- 172.21.15.45:0/3147310893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c4770 0x55f4a19ce830 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f372019afd0 tx=0x7f370c011b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e20a50029dfb6ac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.282+0000 7f36ff400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:48:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.282+0000 7f36ff400640 1 -- 172.21.15.45:0/3147310893 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370c01e020 con 0x55f4a19c4770 2024-03-28T17:48:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.282+0000 7f36ff400640 1 -- 172.21.15.45:0/3147310893 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f370c00fbc0 con 0x55f4a19c4770 2024-03-28T17:48:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.282+0000 7f37277ad740 1 -- 172.21.15.45:0/3147310893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f371800f9a0 con 0x55f4a19c4770 2024-03-28T17:48:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.282+0000 7f36ff400640 1 -- 172.21.15.45:0/3147310893 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f370c00fdc0 con 0x55f4a19c4770 2024-03-28T17:48:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.282+0000 7f37277ad740 10 client.25151 fetch_fsmap learned FSMap version 22 2024-03-28T17:48:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.282+0000 7f37277ad740 1 -- 172.21.15.45:0/3147310893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55f4a17076b0 con 0x55f4a19c4770 2024-03-28T17:48:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.283+0000 7f36ff400640 1 -- 172.21.15.45:0/3147310893 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f371800f9a0 con 0x55f4a19c4770 2024-03-28T17:48:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.283+0000 7f36ff400640 1 -- 172.21.15.45:0/3147310893 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f370c019070 con 0x55f4a19c4770 2024-03-28T17:48:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.283+0000 7f37277ad740 10 client.25151 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T17:48:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.283+0000 7f37277ad740 10 client.25151 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T17:48:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.283+0000 7f37277ad740 20 client.25151 populate_metadata read hostname 'smithi045' 2024-03-28T17:48:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.283+0000 7f37277ad740 1 --2- 172.21.15.45:0/3147310893 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x55f4a19daa70 0x55f4a19d9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.283+0000 7f37277ad740 4 client.25151 mds_command: new command op to 24529 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:48:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.283+0000 7f37277ad740 1 -- 172.21.15.45:0/3147310893 --> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f4a18ca520 con 0x55f4a19daa70 2024-03-28T17:48:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.283+0000 7f371e800640 1 --2- 172.21.15.45:0/3147310893 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x55f4a19daa70 0x55f4a19d9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.284+0000 7f371e800640 1 --2- 172.21.15.45:0/3147310893 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x55f4a19daa70 0x55f4a19d9b30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cf580404f9831d9e server_cookie=c6408a0bd54f25ab in_seq=0 out_seq=0 2024-03-28T17:48:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.284+0000 7f36ff400640 10 client.25151 ms_handle_connect on v2:172.21.15.151:6838/3003564901 2024-03-28T17:48:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.285+0000 7f36ff400640 1 -- 172.21.15.45:0/3147310893 <== mds.2 v2:172.21.15.151:6838/3003564901 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f4a18ca520 con 0x55f4a19daa70 2024-03-28T17:48:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.285+0000 7f36ff400640 10 client.25151 handle_command_reply: tid=0 2024-03-28T17:48:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.285+0000 7f37277ad740 1 client.25151 shutdown 2024-03-28T17:48:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.285+0000 7f37277ad740 1 -- 172.21.15.45:0/3147310893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c4770 msgr2=0x55f4a19ce830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.285+0000 7f37277ad740 1 --2- 172.21.15.45:0/3147310893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c4770 0x55f4a19ce830 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f372019afd0 tx=0x7f370c011b30 comp rx=0 tx=0).stop 2024-03-28T17:48:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.285+0000 7f37277ad740 1 -- 172.21.15.45:0/3147310893 shutdown_connections 2024-03-28T17:48:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.285+0000 7f37277ad740 1 --2- 172.21.15.45:0/3147310893 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x55f4a19daa70 0x55f4a19d9b30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.285+0000 7f37277ad740 1 --2- 172.21.15.45:0/3147310893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f4a19c98e0 0x55f4a19d5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.285+0000 7f37277ad740 1 --2- 172.21.15.45:0/3147310893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c4770 0x55f4a19ce830 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.285+0000 7f36ff400640 0 client.25151 ms_handle_reset on v2:172.21.15.151:6838/3003564901 2024-03-28T17:48:30.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.285+0000 7f37277ad740 1 --2- 172.21.15.45:0/3147310893 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c3d80 0x55f4a19ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.285+0000 7f37277ad740 1 -- 172.21.15.45:0/3147310893 >> 172.21.15.45:0/3147310893 conn(0x55f4a19bf2f0 msgr2=0x55f4a19be4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:30.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.285+0000 7f37277ad740 1 -- 172.21.15.45:0/3147310893 shutdown_connections 2024-03-28T17:48:30.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.285+0000 7f37277ad740 1 -- 172.21.15.45:0/3147310893 wait complete. 2024-03-28T17:48:30.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.286+0000 7f37277ad740 20 client.25151 trim_cache size 0 max 16384 2024-03-28T17:48:30.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.304+0000 7f37277ad740 1 Processor -- start 2024-03-28T17:48:30.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.304+0000 7f37277ad740 1 -- start start 2024-03-28T17:48:30.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.304+0000 7f37277ad740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c3d80 0x55f4a19cbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.304+0000 7f37277ad740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f4a19c4770 0x55f4a19d5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.304+0000 7f37277ad740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c98e0 0x55f4a19bf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.304+0000 7f37277ad740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4a19da290 con 0x55f4a19c4770 2024-03-28T17:48:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.304+0000 7f37277ad740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4a19da8f0 con 0x55f4a19c98e0 2024-03-28T17:48:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.304+0000 7f37277ad740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4a19daf50 con 0x55f4a19c3d80 2024-03-28T17:48:30.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.305+0000 7f371e800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c3d80 0x55f4a19cbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.305+0000 7f371d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f4a19c4770 0x55f4a19d5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.305+0000 7f371e800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c3d80 0x55f4a19cbcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:41422/0 (socket says 172.21.15.45:41422) 2024-03-28T17:48:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.305+0000 7f371e800640 1 -- 172.21.15.45:0/1663716232 learned_addr learned my addr 172.21.15.45:0/1663716232 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.305+0000 7f371e800640 1 --2- 172.21.15.45:0/1663716232 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c98e0 0x55f4a19bf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.305+0000 7f371e800640 1 -- 172.21.15.45:0/1663716232 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c98e0 msgr2=0x55f4a19bf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.305+0000 7f371e800640 1 --2- 172.21.15.45:0/1663716232 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c98e0 0x55f4a19bf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.305+0000 7f371e800640 1 -- 172.21.15.45:0/1663716232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f4a19c4770 msgr2=0x55f4a19d5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.305+0000 7f371e800640 1 --2- 172.21.15.45:0/1663716232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f4a19c4770 0x55f4a19d5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.305+0000 7f371e800640 1 -- 172.21.15.45:0/1663716232 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f4a19c0060 con 0x55f4a19c3d80 2024-03-28T17:48:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.305+0000 7f371e800640 1 --2- 172.21.15.45:0/1663716232 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c3d80 0x55f4a19cbcd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f371800fe20 tx=0x7f3718009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc5890c903b4279c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.306+0000 7f370a600640 1 -- 172.21.15.45:0/1663716232 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371801e040 con 0x55f4a19c3d80 2024-03-28T17:48:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.306+0000 7f370a600640 1 -- 172.21.15.45:0/1663716232 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f371800f9a0 con 0x55f4a19c3d80 2024-03-28T17:48:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.306+0000 7f370a600640 1 -- 172.21.15.45:0/1663716232 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3718010430 con 0x55f4a19c3d80 2024-03-28T17:48:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.306+0000 7f37277ad740 1 -- 172.21.15.45:0/1663716232 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c3d80 msgr2=0x55f4a19cbcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.306+0000 7f37277ad740 1 --2- 172.21.15.45:0/1663716232 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c3d80 0x55f4a19cbcd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f371800fe20 tx=0x7f3718009500 comp rx=0 tx=0).stop 2024-03-28T17:48:30.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.306+0000 7f37277ad740 1 -- 172.21.15.45:0/1663716232 shutdown_connections 2024-03-28T17:48:30.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.306+0000 7f37277ad740 1 --2- 172.21.15.45:0/1663716232 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c98e0 0x55f4a19bf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.306+0000 7f37277ad740 1 --2- 172.21.15.45:0/1663716232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f4a19c4770 0x55f4a19d5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.306+0000 7f37277ad740 1 --2- 172.21.15.45:0/1663716232 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c3d80 0x55f4a19cbcd0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.306+0000 7f37277ad740 1 -- 172.21.15.45:0/1663716232 >> 172.21.15.45:0/1663716232 conn(0x55f4a19bf2f0 msgr2=0x55f4a19d9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:30.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.306+0000 7f37277ad740 1 -- 172.21.15.45:0/1663716232 shutdown_connections 2024-03-28T17:48:30.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.306+0000 7f37277ad740 1 -- 172.21.15.45:0/1663716232 wait complete. 2024-03-28T17:48:30.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.307+0000 7f37277ad740 1 -- start start 2024-03-28T17:48:30.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.307+0000 7f37277ad740 1 Processor -- start 2024-03-28T17:48:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.307+0000 7f37277ad740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c3d80 0x55f4a19d1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.307+0000 7f37277ad740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f4a19c4770 0x55f4a19d1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.307+0000 7f37277ad740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c98e0 0x55f4a19d4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.307+0000 7f37277ad740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4a19dbc10 con 0x55f4a19c4770 2024-03-28T17:48:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.307+0000 7f37277ad740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4a19db5b0 con 0x55f4a19c3d80 2024-03-28T17:48:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.307+0000 7f37277ad740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4a19dc270 con 0x55f4a19c98e0 2024-03-28T17:48:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.307+0000 7f37277ad740 10 client.0 _ll_register_callbacks cb 0x7f370c009890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:48:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.308+0000 7f371e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c3d80 0x55f4a19d1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.308+0000 7f371e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c3d80 0x55f4a19d1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53616/0 (socket says 172.21.15.45:53616) 2024-03-28T17:48:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.308+0000 7f371e800640 1 -- 172.21.15.45:0/402410131 learned_addr learned my addr 172.21.15.45:0/402410131 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.308+0000 7f371d400640 1 --2- 172.21.15.45:0/402410131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f4a19c4770 0x55f4a19d1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.308+0000 7f371e800640 1 --2- 172.21.15.45:0/402410131 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c98e0 0x55f4a19d4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.308+0000 7f371d400640 1 -- 172.21.15.45:0/402410131 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c98e0 msgr2=0x55f4a19d4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.308+0000 7f371d400640 1 --2- 172.21.15.45:0/402410131 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c98e0 0x55f4a19d4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.308+0000 7f371d400640 1 -- 172.21.15.45:0/402410131 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c3d80 msgr2=0x55f4a19d1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.308+0000 7f371d400640 1 --2- 172.21.15.45:0/402410131 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c3d80 0x55f4a19d1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.308+0000 7f371d400640 1 -- 172.21.15.45:0/402410131 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f4a19d54b0 con 0x55f4a19c4770 2024-03-28T17:48:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.308+0000 7f371e800640 1 --2- 172.21.15.45:0/402410131 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c3d80 0x55f4a19d1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.308+0000 7f371d400640 1 --2- 172.21.15.45:0/402410131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f4a19c4770 0x55f4a19d1ea0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f370c01a770 tx=0x7f370c01a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54cf53b91bc73c11 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.308+0000 7f36ff400640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-28T17:48:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.309+0000 7f36ff400640 1 -- 172.21.15.45:0/402410131 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370c032060 con 0x55f4a19c4770 2024-03-28T17:48:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.309+0000 7f37277ad740 1 -- 172.21.15.45:0/402410131 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f371800f9a0 con 0x55f4a19c4770 2024-03-28T17:48:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.309+0000 7f36ff400640 1 -- 172.21.15.45:0/402410131 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f370c00fbc0 con 0x55f4a19c4770 2024-03-28T17:48:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.309+0000 7f36ff400640 1 -- 172.21.15.45:0/402410131 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f370c01e070 con 0x55f4a19c4770 2024-03-28T17:48:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.309+0000 7f37277ad740 10 client.15024 fetch_fsmap learned FSMap version 22 2024-03-28T17:48:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.309+0000 7f37277ad740 1 -- 172.21.15.45:0/402410131 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55f4a1a36580 con 0x55f4a19c4770 2024-03-28T17:48:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.309+0000 7f36ff400640 1 -- 172.21.15.45:0/402410131 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f371800f9a0 con 0x55f4a19c4770 2024-03-28T17:48:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.310+0000 7f36ff400640 1 -- 172.21.15.45:0/402410131 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f370c019040 con 0x55f4a19c4770 2024-03-28T17:48:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.310+0000 7f37277ad740 10 client.15024 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T17:48:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.310+0000 7f37277ad740 10 client.15024 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T17:48:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.310+0000 7f37277ad740 20 client.15024 populate_metadata read hostname 'smithi045' 2024-03-28T17:48:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.310+0000 7f37277ad740 1 --2- 172.21.15.45:0/402410131 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x55f4a1a3ab90 0x55f4a1a367b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.310+0000 7f37277ad740 4 client.15024 mds_command: new command op to 24529 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T17:48:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.310+0000 7f37277ad740 1 -- 172.21.15.45:0/402410131 --> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f4a19361d0 con 0x55f4a1a3ab90 2024-03-28T17:48:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.310+0000 7f371e800640 1 --2- 172.21.15.45:0/402410131 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x55f4a1a3ab90 0x55f4a1a367b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.310+0000 7f371e800640 1 --2- 172.21.15.45:0/402410131 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x55f4a1a3ab90 0x55f4a1a367b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6afbdab14db78025 server_cookie=61108183010d24ad in_seq=0 out_seq=0 2024-03-28T17:48:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.310+0000 7f36ff400640 10 client.15024 ms_handle_connect on v2:172.21.15.151:6838/3003564901 2024-03-28T17:48:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.311+0000 7f36ff400640 1 -- 172.21.15.45:0/402410131 <== mds.2 v2:172.21.15.151:6838/3003564901 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55f4a19361d0 con 0x55f4a1a3ab90 2024-03-28T17:48:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.311+0000 7f36ff400640 10 client.15024 handle_command_reply: tid=0 2024-03-28T17:48:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.311+0000 7f37277ad740 1 client.15024 shutdown 2024-03-28T17:48:30.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.311+0000 7f37277ad740 1 -- 172.21.15.45:0/402410131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f4a19c4770 msgr2=0x55f4a19d1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.311+0000 7f37277ad740 1 --2- 172.21.15.45:0/402410131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f4a19c4770 0x55f4a19d1ea0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f370c01a770 tx=0x7f370c01a620 comp rx=0 tx=0).stop 2024-03-28T17:48:30.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.311+0000 7f37277ad740 1 -- 172.21.15.45:0/402410131 shutdown_connections 2024-03-28T17:48:30.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.311+0000 7f37277ad740 1 --2- 172.21.15.45:0/402410131 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x55f4a1a3ab90 0x55f4a1a367b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.311+0000 7f37277ad740 1 --2- 172.21.15.45:0/402410131 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f4a19c98e0 0x55f4a19d4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.311+0000 7f37277ad740 1 --2- 172.21.15.45:0/402410131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f4a19c4770 0x55f4a19d1ea0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.311+0000 7f37277ad740 1 --2- 172.21.15.45:0/402410131 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f4a19c3d80 0x55f4a19d1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.311+0000 7f36ff400640 0 client.15024 ms_handle_reset on v2:172.21.15.151:6838/3003564901 2024-03-28T17:48:30.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.311+0000 7f37277ad740 1 -- 172.21.15.45:0/402410131 >> 172.21.15.45:0/402410131 conn(0x55f4a19bf2f0 msgr2=0x55f4a19bf6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:30.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.311+0000 7f37277ad740 1 -- 172.21.15.45:0/402410131 shutdown_connections 2024-03-28T17:48:30.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.311+0000 7f37277ad740 1 -- 172.21.15.45:0/402410131 wait complete. 2024-03-28T17:48:30.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.311+0000 7f37277ad740 20 client.15024 trim_cache size 0 max 16384 2024-03-28T17:48:30.316 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:48:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "9f4888ec-ed29-11ee-95d0-87774f69a715", 2024-03-28T17:48:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 2, 2024-03-28T17:48:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 24529, 2024-03-28T17:48:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-03-28T17:48:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-03-28T17:48:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-03-28T17:48:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 133.34914560300001, 2024-03-28T17:48:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 22, 2024-03-28T17:48:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 87, 2024-03-28T17:48:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 85, 2024-03-28T17:48:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 155.652306452 2024-03-28T17:48:30.316 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:48:30.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.312+0000 7f371f200640 1 -- 172.21.15.45:0/4153820627 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f37040a2880 msgr2=0x7f37040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.312+0000 7f371f200640 1 --2- 172.21.15.45:0/4153820627 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f37040a2880 0x7f37040a4d40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f37100bc860 tx=0x7f3710005f90 comp rx=0 tx=0).stop 2024-03-28T17:48:30.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.312+0000 7f371f200640 1 -- 172.21.15.45:0/4153820627 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37200ff860 msgr2=0x7f3720199ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.312+0000 7f371f200640 1 --2- 172.21.15.45:0/4153820627 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37200ff860 0x7f3720199ac0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f37100349e0 tx=0x7f3710032450 comp rx=0 tx=0).stop 2024-03-28T17:48:30.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.312+0000 7f371f200640 1 -- 172.21.15.45:0/4153820627 shutdown_connections 2024-03-28T17:48:30.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.312+0000 7f371f200640 1 --2- 172.21.15.45:0/4153820627 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f37040a2880 0x7f37040a4d40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.312+0000 7f371f200640 1 --2- 172.21.15.45:0/4153820627 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3720100f80 0x7f37201a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.312+0000 7f371f200640 1 --2- 172.21.15.45:0/4153820627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37201005c0 0x7f372019a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.312+0000 7f371f200640 1 --2- 172.21.15.45:0/4153820627 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37200ff860 0x7f3720199ac0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.312+0000 7f371f200640 1 -- 172.21.15.45:0/4153820627 >> 172.21.15.45:0/4153820627 conn(0x7f37200fb190 msgr2=0x7f37200fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:30.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.312+0000 7f371f200640 1 -- 172.21.15.45:0/4153820627 shutdown_connections 2024-03-28T17:48:30.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.312+0000 7f371f200640 1 -- 172.21.15.45:0/4153820627 wait complete. 2024-03-28T17:48:30.329 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-28T17:48:30.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.545+0000 7f0004e00640 1 -- 172.21.15.45:0/3923789908 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0000105370 msgr2=0x7f000010bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.545+0000 7f0004e00640 1 --2- 172.21.15.45:0/3923789908 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0000105370 0x7f000010bc00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efff8009860 tx=0x7efff802bb70 comp rx=0 tx=0).stop 2024-03-28T17:48:30.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.545+0000 7f0004e00640 1 -- 172.21.15.45:0/3923789908 shutdown_connections 2024-03-28T17:48:30.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.545+0000 7f0004e00640 1 --2- 172.21.15.45:0/3923789908 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0000105370 0x7f000010bc00 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.545+0000 7f0004e00640 1 --2- 172.21.15.45:0/3923789908 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00001049b0 0x7f0000104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.545+0000 7f0004e00640 1 --2- 172.21.15.45:0/3923789908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00001037c0 0x7f0000103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.545+0000 7f0004e00640 1 -- 172.21.15.45:0/3923789908 >> 172.21.15.45:0/3923789908 conn(0x7f00000fed50 msgr2=0x7f0000101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:30.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.545+0000 7f0004e00640 1 -- 172.21.15.45:0/3923789908 shutdown_connections 2024-03-28T17:48:30.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.545+0000 7f0004e00640 1 -- 172.21.15.45:0/3923789908 wait complete. 2024-03-28T17:48:30.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.546+0000 7f0004e00640 1 Processor -- start 2024-03-28T17:48:30.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.546+0000 7f0004e00640 1 -- start start 2024-03-28T17:48:30.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.547+0000 7f0004e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00001037c0 0x7f0000199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.547+0000 7f0004e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00001049b0 0x7f0000199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.547+0000 7f0004e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0000105370 0x7f00001a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.547+0000 7f0004e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000010f420 con 0x7f0000105370 2024-03-28T17:48:30.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.547+0000 7f0004e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000010f2a0 con 0x7f00001049b0 2024-03-28T17:48:30.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.547+0000 7f0004e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000010f5a0 con 0x7f00001037c0 2024-03-28T17:48:30.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.547+0000 7effff400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00001037c0 0x7f0000199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.547+0000 7effff400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00001037c0 0x7f0000199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:41458/0 (socket says 172.21.15.45:41458) 2024-03-28T17:48:30.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.547+0000 7effff400640 1 -- 172.21.15.45:0/2895454868 learned_addr learned my addr 172.21.15.45:0/2895454868 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:30.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.547+0000 7effffe00640 1 --2- 172.21.15.45:0/2895454868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0000105370 0x7f00001a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.547+0000 7efff6a00640 1 --2- 172.21.15.45:0/2895454868 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00001049b0 0x7f0000199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.547+0000 7effff400640 1 -- 172.21.15.45:0/2895454868 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00001049b0 msgr2=0x7f0000199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.547+0000 7effff400640 1 --2- 172.21.15.45:0/2895454868 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00001049b0 0x7f0000199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.547+0000 7effff400640 1 -- 172.21.15.45:0/2895454868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0000105370 msgr2=0x7f00001a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.548+0000 7effff400640 1 --2- 172.21.15.45:0/2895454868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0000105370 0x7f00001a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.548+0000 7effff400640 1 -- 172.21.15.45:0/2895454868 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00001a16a0 con 0x7f00001037c0 2024-03-28T17:48:30.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.548+0000 7effff400640 1 --2- 172.21.15.45:0/2895454868 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00001037c0 0x7f0000199a80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efff0009fd0 tx=0x7efff000c270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:30.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.548+0000 7efffcc00640 1 -- 172.21.15.45:0/2895454868 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff0011470 con 0x7f00001037c0 2024-03-28T17:48:30.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.548+0000 7efffcc00640 1 -- 172.21.15.45:0/2895454868 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efff0011af0 con 0x7f00001037c0 2024-03-28T17:48:30.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.548+0000 7f0004e00640 1 -- 172.21.15.45:0/2895454868 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00001a1990 con 0x7f00001037c0 2024-03-28T17:48:30.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.548+0000 7efffcc00640 1 -- 172.21.15.45:0/2895454868 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff001ab90 con 0x7f00001037c0 2024-03-28T17:48:30.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.548+0000 7f0004e00640 1 -- 172.21.15.45:0/2895454868 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00001a1f30 con 0x7f00001037c0 2024-03-28T17:48:30.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.551+0000 7efffcc00640 1 -- 172.21.15.45:0/2895454868 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7efff0011610 con 0x7f00001037c0 2024-03-28T17:48:30.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.552+0000 7f00091c1740 1 Processor -- start 2024-03-28T17:48:30.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.552+0000 7f00091c1740 1 -- start start 2024-03-28T17:48:30.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.552+0000 7f00091c1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 0x556378b881a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.552+0000 7f00091c1740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b88770 0x556378b8d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.552+0000 7f00091c1740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x556378b8d8e0 0x556378b8fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.552+0000 7f00091c1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556378aecc90 con 0x556378b87d80 2024-03-28T17:48:30.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.552+0000 7f00091c1740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x556378a95980 con 0x556378b88770 2024-03-28T17:48:30.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.552+0000 7f00091c1740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x556378ae8d30 con 0x556378b8d8e0 2024-03-28T17:48:30.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.552+0000 7effff400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b88770 0x556378b8d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.552+0000 7effff400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b88770 0x556378b8d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53638/0 (socket says 172.21.15.45:53638) 2024-03-28T17:48:30.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.552+0000 7effff400640 1 -- 172.21.15.45:0/734041516 learned_addr learned my addr 172.21.15.45:0/734041516 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:30.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.552+0000 7effffe00640 1 --2- 172.21.15.45:0/734041516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 0x556378b881a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.552+0000 7efff6a00640 1 --2- 172.21.15.45:0/734041516 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x556378b8d8e0 0x556378b8fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.553+0000 7efffcc00640 1 --2- 172.21.15.45:0/2895454868 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7effe40a2880 0x7effe40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.553+0000 7effff400640 1 -- 172.21.15.45:0/734041516 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x556378b8d8e0 msgr2=0x556378b8fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.553+0000 7effff400640 1 --2- 172.21.15.45:0/734041516 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x556378b8d8e0 0x556378b8fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.553+0000 7effff400640 1 -- 172.21.15.45:0/734041516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 msgr2=0x556378b881a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.553+0000 7effff400640 1 --2- 172.21.15.45:0/734041516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 0x556378b881a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.553+0000 7efffcc00640 1 -- 172.21.15.45:0/2895454868 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7efff00cefa0 con 0x7f00001037c0 2024-03-28T17:48:30.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.553+0000 7effff400640 1 -- 172.21.15.45:0/734041516 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556378aad510 con 0x556378b88770 2024-03-28T17:48:30.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.553+0000 7effffe00640 1 --2- 172.21.15.45:0/734041516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 0x556378b881a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:48:30.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.553+0000 7effffe00640 1 --2- 172.21.15.45:0/2895454868 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7effe40a2880 0x7effe40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.553+0000 7effff400640 1 --2- 172.21.15.45:0/734041516 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b88770 0x556378b8d270 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7efff00cca50 tx=0x7efff0009520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ced49f72973227f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:30.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.553+0000 7efff4c00640 1 -- 172.21.15.45:0/734041516 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff009bff0 con 0x556378b88770 2024-03-28T17:48:30.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.553+0000 7efff4c00640 1 -- 172.21.15.45:0/734041516 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efff00a4090 con 0x556378b88770 2024-03-28T17:48:30.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.554+0000 7efff4c00640 1 -- 172.21.15.45:0/734041516 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff0011ca0 con 0x556378b88770 2024-03-28T17:48:30.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.554+0000 7f00091c1740 1 -- 172.21.15.45:0/734041516 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b88770 msgr2=0x556378b8d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.554+0000 7f00091c1740 1 --2- 172.21.15.45:0/734041516 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b88770 0x556378b8d270 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7efff00cca50 tx=0x7efff0009520 comp rx=0 tx=0).stop 2024-03-28T17:48:30.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.554+0000 7effffe00640 1 --2- 172.21.15.45:0/2895454868 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7effe40a2880 0x7effe40a4d40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efff8034a50 tx=0x7efff802e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:30.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.554+0000 7f00091c1740 1 -- 172.21.15.45:0/734041516 shutdown_connections 2024-03-28T17:48:30.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.554+0000 7f00091c1740 1 --2- 172.21.15.45:0/734041516 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x556378b8d8e0 0x556378b8fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.554+0000 7f00091c1740 1 --2- 172.21.15.45:0/734041516 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b88770 0x556378b8d270 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.554+0000 7f00091c1740 1 --2- 172.21.15.45:0/734041516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 0x556378b881a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.554+0000 7f00091c1740 1 -- 172.21.15.45:0/734041516 >> 172.21.15.45:0/734041516 conn(0x556378b832f0 msgr2=0x556378b836f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:30.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.554+0000 7f00091c1740 1 -- 172.21.15.45:0/734041516 shutdown_connections 2024-03-28T17:48:30.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.554+0000 7f00091c1740 1 -- 172.21.15.45:0/734041516 wait complete. 2024-03-28T17:48:30.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.555+0000 7f00091c1740 1 -- start start 2024-03-28T17:48:30.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.555+0000 7f00091c1740 1 Processor -- start 2024-03-28T17:48:30.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.555+0000 7f00091c1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 0x556378b921e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.556+0000 7f00091c1740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x556378b88770 0x556378b92720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.556+0000 7f00091c1740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b8d8e0 0x556378b99760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.556+0000 7f00091c1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556378b90830 con 0x556378b87d80 2024-03-28T17:48:30.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.556+0000 7f00091c1740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x556378aa8dd0 con 0x556378b8d8e0 2024-03-28T17:48:30.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.556+0000 7f00091c1740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x556378b909b0 con 0x556378b88770 2024-03-28T17:48:30.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.556+0000 7f00091c1740 10 client.0 _ll_register_callbacks cb 0x556378800ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:48:30.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.556+0000 7effff400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 0x556378b921e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.556+0000 7effffe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b8d8e0 0x556378b99760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.556+0000 7efff6a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x556378b88770 0x556378b92720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.556+0000 7effff400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 0x556378b921e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50644/0 (socket says 172.21.15.45:50644) 2024-03-28T17:48:30.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.556+0000 7efff6a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x556378b88770 0x556378b92720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:41474/0 (socket says 172.21.15.45:41474) 2024-03-28T17:48:30.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.556+0000 7effff400640 1 -- 172.21.15.45:0/3057338181 learned_addr learned my addr 172.21.15.45:0/3057338181 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.556+0000 7effffe00640 1 -- 172.21.15.45:0/3057338181 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x556378b88770 msgr2=0x556378b92720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.556+0000 7effffe00640 1 --2- 172.21.15.45:0/3057338181 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x556378b88770 0x556378b92720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.556+0000 7effffe00640 1 -- 172.21.15.45:0/3057338181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 msgr2=0x556378b921e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.556+0000 7effffe00640 1 --2- 172.21.15.45:0/3057338181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 0x556378b921e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.556+0000 7effffe00640 1 -- 172.21.15.45:0/3057338181 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5563788012d0 con 0x556378b8d8e0 2024-03-28T17:48:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.557+0000 7effffe00640 1 --2- 172.21.15.45:0/3057338181 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b8d8e0 0x556378b99760 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7efff802c0b0 tx=0x7efff80349e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39be36da870e6930 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.557+0000 7effed600640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:48:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.557+0000 7effed600640 1 -- 172.21.15.45:0/3057338181 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff804f020 con 0x556378b8d8e0 2024-03-28T17:48:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.557+0000 7effed600640 1 -- 172.21.15.45:0/3057338181 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7efff8004240 con 0x556378b8d8e0 2024-03-28T17:48:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.557+0000 7f00091c1740 1 -- 172.21.15.45:0/3057338181 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efff00a4090 con 0x556378b8d8e0 2024-03-28T17:48:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.557+0000 7effed600640 1 -- 172.21.15.45:0/3057338181 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7efff8039040 con 0x556378b8d8e0 2024-03-28T17:48:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.557+0000 7f00091c1740 10 client.25169 fetch_fsmap learned FSMap version 22 2024-03-28T17:48:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.557+0000 7f00091c1740 1 -- 172.21.15.45:0/3057338181 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5563788cb6b0 con 0x556378b8d8e0 2024-03-28T17:48:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.557+0000 7effed600640 1 -- 172.21.15.45:0/3057338181 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efff00a4090 con 0x556378b8d8e0 2024-03-28T17:48:30.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.558+0000 7effed600640 1 -- 172.21.15.45:0/3057338181 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7efff80329d0 con 0x556378b8d8e0 2024-03-28T17:48:30.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.558+0000 7f00091c1740 10 client.25169 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T17:48:30.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.558+0000 7f00091c1740 10 client.25169 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T17:48:30.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.558+0000 7f00091c1740 20 client.25169 populate_metadata read hostname 'smithi045' 2024-03-28T17:48:30.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.558+0000 7f00091c1740 1 --2- 172.21.15.45:0/3057338181 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x556378b9e9c0 0x556378b9da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.558+0000 7f00091c1740 4 client.25169 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:48:30.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.558+0000 7f00091c1740 1 -- 172.21.15.45:0/3057338181 --> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556378a8e520 con 0x556378b9e9c0 2024-03-28T17:48:30.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.558+0000 7effff400640 1 --2- 172.21.15.45:0/3057338181 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x556378b9e9c0 0x556378b9da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.559+0000 7effff400640 1 --2- 172.21.15.45:0/3057338181 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x556378b9e9c0 0x556378b9da40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eb1b90db911bf53f server_cookie=188da3aa13b1a7fa in_seq=0 out_seq=0 2024-03-28T17:48:30.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.559+0000 7effed600640 10 client.25169 ms_handle_connect on v2:172.21.15.71:6836/2008752655 2024-03-28T17:48:30.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.559+0000 7effed600640 1 -- 172.21.15.45:0/3057338181 <== mds.1 v2:172.21.15.71:6836/2008752655 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556378a8e520 con 0x556378b9e9c0 2024-03-28T17:48:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.559+0000 7effed600640 10 client.25169 handle_command_reply: tid=0 2024-03-28T17:48:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.560+0000 7f00091c1740 1 client.25169 shutdown 2024-03-28T17:48:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.560+0000 7f00091c1740 1 -- 172.21.15.45:0/3057338181 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b8d8e0 msgr2=0x556378b99760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.560+0000 7f00091c1740 1 --2- 172.21.15.45:0/3057338181 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b8d8e0 0x556378b99760 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7efff802c0b0 tx=0x7efff80349e0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.560+0000 7f00091c1740 1 -- 172.21.15.45:0/3057338181 shutdown_connections 2024-03-28T17:48:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.560+0000 7f00091c1740 1 --2- 172.21.15.45:0/3057338181 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b8d8e0 0x556378b99760 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.560+0000 7f00091c1740 1 --2- 172.21.15.45:0/3057338181 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x556378b9e9c0 0x556378b9da40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.560+0000 7f00091c1740 1 --2- 172.21.15.45:0/3057338181 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x556378b88770 0x556378b92720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.560+0000 7f00091c1740 1 --2- 172.21.15.45:0/3057338181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 0x556378b921e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.560+0000 7effed600640 0 client.25169 ms_handle_reset on v2:172.21.15.71:6836/2008752655 2024-03-28T17:48:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.560+0000 7f00091c1740 1 -- 172.21.15.45:0/3057338181 >> 172.21.15.45:0/3057338181 conn(0x556378b832f0 msgr2=0x556378b824f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.560+0000 7f00091c1740 1 -- 172.21.15.45:0/3057338181 shutdown_connections 2024-03-28T17:48:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.560+0000 7f00091c1740 1 -- 172.21.15.45:0/3057338181 wait complete. 2024-03-28T17:48:30.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.560+0000 7f00091c1740 20 client.25169 trim_cache size 0 max 16384 2024-03-28T17:48:30.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.579+0000 7f00091c1740 1 Processor -- start 2024-03-28T17:48:30.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.579+0000 7f00091c1740 1 -- start start 2024-03-28T17:48:30.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.579+0000 7f00091c1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 0x556378b8fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.579+0000 7f00091c1740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b88770 0x556378b99720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.579+0000 7f00091c1740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x556378b8d8e0 0x556378b836d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.579+0000 7f00091c1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556378b9e1e0 con 0x556378b87d80 2024-03-28T17:48:30.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.579+0000 7f00091c1740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x556378b9e840 con 0x556378b88770 2024-03-28T17:48:30.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.579+0000 7f00091c1740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x556378b9eea0 con 0x556378b8d8e0 2024-03-28T17:48:30.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.579+0000 7efff6a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b88770 0x556378b99720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.579+0000 7effff400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 0x556378b8fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.579+0000 7effffe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x556378b8d8e0 0x556378b836d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.579+0000 7efff6a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b88770 0x556378b99720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53650/0 (socket says 172.21.15.45:53650) 2024-03-28T17:48:30.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.579+0000 7efff6a00640 1 -- 172.21.15.45:0/1148062018 learned_addr learned my addr 172.21.15.45:0/1148062018 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:30.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.579+0000 7effff400640 1 --2- 172.21.15.45:0/1148062018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 0x556378b8fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50646/0 (socket says 172.21.15.45:50646) 2024-03-28T17:48:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.580+0000 7efff6a00640 1 -- 172.21.15.45:0/1148062018 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x556378b8d8e0 msgr2=0x556378b836d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.580+0000 7efff6a00640 1 --2- 172.21.15.45:0/1148062018 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x556378b8d8e0 0x556378b836d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.580+0000 7efff6a00640 1 -- 172.21.15.45:0/1148062018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 msgr2=0x556378b8fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.580+0000 7efff6a00640 1 --2- 172.21.15.45:0/1148062018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 0x556378b8fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.580+0000 7efff6a00640 1 -- 172.21.15.45:0/1148062018 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556378b84060 con 0x556378b88770 2024-03-28T17:48:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.580+0000 7effffe00640 1 --2- 172.21.15.45:0/1148062018 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x556378b8d8e0 0x556378b836d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.580+0000 7effff400640 1 --2- 172.21.15.45:0/1148062018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 0x556378b8fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.580+0000 7efff6a00640 1 --2- 172.21.15.45:0/1148062018 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b88770 0x556378b99720 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7effe8005fd0 tx=0x7effe8009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7565ddee9004f31b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.580+0000 7effefe00640 1 -- 172.21.15.45:0/1148062018 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe8004020 con 0x556378b88770 2024-03-28T17:48:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.580+0000 7effefe00640 1 -- 172.21.15.45:0/1148062018 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effe8005500 con 0x556378b88770 2024-03-28T17:48:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.580+0000 7effefe00640 1 -- 172.21.15.45:0/1148062018 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe80035f0 con 0x556378b88770 2024-03-28T17:48:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.581+0000 7f00091c1740 1 -- 172.21.15.45:0/1148062018 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b88770 msgr2=0x556378b99720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.581+0000 7f00091c1740 1 --2- 172.21.15.45:0/1148062018 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b88770 0x556378b99720 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7effe8005fd0 tx=0x7effe8009260 comp rx=0 tx=0).stop 2024-03-28T17:48:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.581+0000 7f00091c1740 1 -- 172.21.15.45:0/1148062018 shutdown_connections 2024-03-28T17:48:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.581+0000 7f00091c1740 1 --2- 172.21.15.45:0/1148062018 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x556378b8d8e0 0x556378b836d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.581+0000 7f00091c1740 1 --2- 172.21.15.45:0/1148062018 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b88770 0x556378b99720 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.581+0000 7f00091c1740 1 --2- 172.21.15.45:0/1148062018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 0x556378b8fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.581+0000 7f00091c1740 1 -- 172.21.15.45:0/1148062018 >> 172.21.15.45:0/1148062018 conn(0x556378b832f0 msgr2=0x556378b9da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.581+0000 7f00091c1740 1 -- 172.21.15.45:0/1148062018 shutdown_connections 2024-03-28T17:48:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.581+0000 7f00091c1740 1 -- 172.21.15.45:0/1148062018 wait complete. 2024-03-28T17:48:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.582+0000 7f00091c1740 1 -- start start 2024-03-28T17:48:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.582+0000 7f00091c1740 1 Processor -- start 2024-03-28T17:48:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.582+0000 7f00091c1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 0x556378b956c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.582+0000 7f00091c1740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x556378b88770 0x556378b95c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.582+0000 7f00091c1740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b8d8e0 0x556378b98c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.582+0000 7f00091c1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556378b9fb60 con 0x556378b87d80 2024-03-28T17:48:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.582+0000 7f00091c1740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x556378b9f500 con 0x556378b8d8e0 2024-03-28T17:48:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.582+0000 7f00091c1740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x556378ba01c0 con 0x556378b88770 2024-03-28T17:48:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.582+0000 7f00091c1740 10 client.0 _ll_register_callbacks cb 0x7effe8005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:48:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.582+0000 7effffe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b8d8e0 0x556378b98c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.582+0000 7effffe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b8d8e0 0x556378b98c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53652/0 (socket says 172.21.15.45:53652) 2024-03-28T17:48:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.583+0000 7effffe00640 1 -- 172.21.15.45:0/2939075839 learned_addr learned my addr 172.21.15.45:0/2939075839 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.583+0000 7effff400640 1 --2- 172.21.15.45:0/2939075839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 0x556378b956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.582+0000 7efff6a00640 1 --2- 172.21.15.45:0/2939075839 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x556378b88770 0x556378b95c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.583+0000 7effffe00640 1 -- 172.21.15.45:0/2939075839 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x556378b88770 msgr2=0x556378b95c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.583+0000 7effffe00640 1 --2- 172.21.15.45:0/2939075839 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x556378b88770 0x556378b95c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.583+0000 7effffe00640 1 -- 172.21.15.45:0/2939075839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 msgr2=0x556378b956c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.583+0000 7effffe00640 1 --2- 172.21.15.45:0/2939075839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 0x556378b956c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.583+0000 7effffe00640 1 -- 172.21.15.45:0/2939075839 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556378b99210 con 0x556378b8d8e0 2024-03-28T17:48:30.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.583+0000 7efff6a00640 1 --2- 172.21.15.45:0/2939075839 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x556378b88770 0x556378b95c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:48:30.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.583+0000 7effffe00640 1 --2- 172.21.15.45:0/2939075839 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b8d8e0 0x556378b98c20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efff8034fd0 tx=0x7efff80045d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fdce97fc1586156 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:30.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.583+0000 7effed600640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:48:30.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.583+0000 7effed600640 1 -- 172.21.15.45:0/2939075839 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff804f040 con 0x556378b8d8e0 2024-03-28T17:48:30.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.583+0000 7effed600640 1 -- 172.21.15.45:0/2939075839 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7efff8032a50 con 0x556378b8d8e0 2024-03-28T17:48:30.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.583+0000 7f00091c1740 1 -- 172.21.15.45:0/2939075839 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7effe8005500 con 0x556378b8d8e0 2024-03-28T17:48:30.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.584+0000 7effed600640 1 -- 172.21.15.45:0/2939075839 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7efff8003820 con 0x556378b8d8e0 2024-03-28T17:48:30.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.584+0000 7f00091c1740 10 client.25175 fetch_fsmap learned FSMap version 22 2024-03-28T17:48:30.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.584+0000 7f00091c1740 1 -- 172.21.15.45:0/2939075839 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x556378c02b90 con 0x556378b8d8e0 2024-03-28T17:48:30.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.584+0000 7effed600640 1 -- 172.21.15.45:0/2939075839 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effe8005500 con 0x556378b8d8e0 2024-03-28T17:48:30.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.584+0000 7effed600640 1 -- 172.21.15.45:0/2939075839 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7efff8067030 con 0x556378b8d8e0 2024-03-28T17:48:30.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.584+0000 7f00091c1740 10 client.25175 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T17:48:30.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.584+0000 7f00091c1740 10 client.25175 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T17:48:30.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.584+0000 7f00091c1740 20 client.25175 populate_metadata read hostname 'smithi045' 2024-03-28T17:48:30.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.584+0000 7f00091c1740 1 --2- 172.21.15.45:0/2939075839 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x556378c07250 0x556378c02dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.584+0000 7f00091c1740 4 client.25175 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T17:48:30.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.584+0000 7f00091c1740 1 -- 172.21.15.45:0/2939075839 --> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556378afa1d0 con 0x556378c07250 2024-03-28T17:48:30.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.584+0000 7effff400640 1 --2- 172.21.15.45:0/2939075839 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x556378c07250 0x556378c02dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.585+0000 7effff400640 1 --2- 172.21.15.45:0/2939075839 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x556378c07250 0x556378c02dc0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=13186c5f048445b3 server_cookie=fb753fe6af9b81d8 in_seq=0 out_seq=0 2024-03-28T17:48:30.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.585+0000 7effed600640 10 client.25175 ms_handle_connect on v2:172.21.15.71:6836/2008752655 2024-03-28T17:48:30.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.585+0000 7effed600640 1 -- 172.21.15.45:0/2939075839 <== mds.1 v2:172.21.15.71:6836/2008752655 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x556378afa1d0 con 0x556378c07250 2024-03-28T17:48:30.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.585+0000 7effed600640 10 client.25175 handle_command_reply: tid=0 2024-03-28T17:48:30.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.585+0000 7f00091c1740 1 client.25175 shutdown 2024-03-28T17:48:30.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.585+0000 7f00091c1740 1 -- 172.21.15.45:0/2939075839 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b8d8e0 msgr2=0x556378b98c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.585+0000 7f00091c1740 1 --2- 172.21.15.45:0/2939075839 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b8d8e0 0x556378b98c20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efff8034fd0 tx=0x7efff80045d0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.585+0000 7f00091c1740 1 -- 172.21.15.45:0/2939075839 shutdown_connections 2024-03-28T17:48:30.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.585+0000 7f00091c1740 1 --2- 172.21.15.45:0/2939075839 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556378b8d8e0 0x556378b98c20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.585+0000 7f00091c1740 1 --2- 172.21.15.45:0/2939075839 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x556378c07250 0x556378c02dc0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.585+0000 7f00091c1740 1 --2- 172.21.15.45:0/2939075839 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x556378b88770 0x556378b95c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.585+0000 7f00091c1740 1 --2- 172.21.15.45:0/2939075839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556378b87d80 0x556378b956c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.585+0000 7effed600640 0 client.25175 ms_handle_reset on v2:172.21.15.71:6836/2008752655 2024-03-28T17:48:30.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.585+0000 7f00091c1740 1 -- 172.21.15.45:0/2939075839 >> 172.21.15.45:0/2939075839 conn(0x556378b832f0 msgr2=0x556378b836d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:30.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.585+0000 7f00091c1740 1 -- 172.21.15.45:0/2939075839 shutdown_connections 2024-03-28T17:48:30.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.585+0000 7f00091c1740 1 -- 172.21.15.45:0/2939075839 wait complete. 2024-03-28T17:48:30.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.586+0000 7f00091c1740 20 client.25175 trim_cache size 0 max 16384 2024-03-28T17:48:30.590 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:48:30.591 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "9f4888ec-ed29-11ee-95d0-87774f69a715", 2024-03-28T17:48:30.591 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 1, 2024-03-28T17:48:30.591 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 24530, 2024-03-28T17:48:30.591 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-03-28T17:48:30.591 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-03-28T17:48:30.591 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-03-28T17:48:30.591 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 134.65693320700001, 2024-03-28T17:48:30.591 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 22, 2024-03-28T17:48:30.591 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 87, 2024-03-28T17:48:30.591 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 85, 2024-03-28T17:48:30.591 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 158.97980788300001 2024-03-28T17:48:30.591 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:48:30.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.586+0000 7f0004e00640 1 -- 172.21.15.45:0/2895454868 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7effe40a2880 msgr2=0x7effe40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.586+0000 7f0004e00640 1 --2- 172.21.15.45:0/2895454868 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7effe40a2880 0x7effe40a4d40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efff8034a50 tx=0x7efff802e040 comp rx=0 tx=0).stop 2024-03-28T17:48:30.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.586+0000 7f0004e00640 1 -- 172.21.15.45:0/2895454868 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00001037c0 msgr2=0x7f0000199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.586+0000 7f0004e00640 1 --2- 172.21.15.45:0/2895454868 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00001037c0 0x7f0000199a80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efff0009fd0 tx=0x7efff000c270 comp rx=0 tx=0).stop 2024-03-28T17:48:30.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.586+0000 7f0004e00640 1 -- 172.21.15.45:0/2895454868 shutdown_connections 2024-03-28T17:48:30.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.586+0000 7f0004e00640 1 --2- 172.21.15.45:0/2895454868 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7effe40a2880 0x7effe40a4d40 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.586+0000 7f0004e00640 1 --2- 172.21.15.45:0/2895454868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0000105370 0x7f00001a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.586+0000 7f0004e00640 1 --2- 172.21.15.45:0/2895454868 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00001049b0 0x7f0000199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.586+0000 7f0004e00640 1 --2- 172.21.15.45:0/2895454868 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00001037c0 0x7f0000199a80 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.586+0000 7f0004e00640 1 -- 172.21.15.45:0/2895454868 >> 172.21.15.45:0/2895454868 conn(0x7f00000fed50 msgr2=0x7f0000107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:30.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.586+0000 7f0004e00640 1 -- 172.21.15.45:0/2895454868 shutdown_connections 2024-03-28T17:48:30.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.586+0000 7f0004e00640 1 -- 172.21.15.45:0/2895454868 wait complete. 2024-03-28T17:48:30.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:30 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/661021843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:48:30.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:30 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/37716806' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:48:30.605 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T17:48:30.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:30 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/661021843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:48:30.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:30 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/37716806' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:48:30.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.813+0000 7f266d400640 1 -- 172.21.15.45:0/1722534964 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f26680772a0 msgr2=0x7f2668077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.813+0000 7f266d400640 1 --2- 172.21.15.45:0/1722534964 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f26680772a0 0x7f2668077700 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2650009900 tx=0x7f265002bc30 comp rx=0 tx=0).stop 2024-03-28T17:48:30.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.814+0000 7f266d400640 1 -- 172.21.15.45:0/1722534964 shutdown_connections 2024-03-28T17:48:30.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.814+0000 7f266d400640 1 --2- 172.21.15.45:0/1722534964 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2668077c40 0x7f26680784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.814+0000 7f266d400640 1 --2- 172.21.15.45:0/1722534964 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f26680772a0 0x7f2668077700 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.814+0000 7f266d400640 1 --2- 172.21.15.45:0/1722534964 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2668076060 0x7f2668076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.814+0000 7f266d400640 1 -- 172.21.15.45:0/1722534964 >> 172.21.15.45:0/1722534964 conn(0x7f26680fee50 msgr2=0x7f2668101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:30.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.814+0000 7f266d400640 1 -- 172.21.15.45:0/1722534964 shutdown_connections 2024-03-28T17:48:30.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.814+0000 7f266d400640 1 -- 172.21.15.45:0/1722534964 wait complete. 2024-03-28T17:48:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.815+0000 7f266d400640 1 Processor -- start 2024-03-28T17:48:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.815+0000 7f266d400640 1 -- start start 2024-03-28T17:48:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.815+0000 7f266d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2668076060 0x7f266819e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.815+0000 7f266d400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f26680772a0 0x7f266819e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.815+0000 7f266d400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2668077c40 0x7f26681a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.815+0000 7f266d400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2668113970 con 0x7f2668076060 2024-03-28T17:48:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.815+0000 7f266d400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26681137f0 con 0x7f26680772a0 2024-03-28T17:48:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.815+0000 7f266d400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2668113af0 con 0x7f2668077c40 2024-03-28T17:48:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.815+0000 7f2666800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2668076060 0x7f266819e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.815+0000 7f265de00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f26680772a0 0x7f266819e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.815+0000 7f2666800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2668076060 0x7f266819e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50672/0 (socket says 172.21.15.45:50672) 2024-03-28T17:48:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.815+0000 7f2667200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2668077c40 0x7f26681a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.816+0000 7f2666800640 1 -- 172.21.15.45:0/2001021032 learned_addr learned my addr 172.21.15.45:0/2001021032 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:30.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.816+0000 7f2667200640 1 -- 172.21.15.45:0/2001021032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f26680772a0 msgr2=0x7f266819e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.816+0000 7f2667200640 1 --2- 172.21.15.45:0/2001021032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f26680772a0 0x7f266819e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.816+0000 7f2667200640 1 -- 172.21.15.45:0/2001021032 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2668076060 msgr2=0x7f266819e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.816+0000 7f2667200640 1 --2- 172.21.15.45:0/2001021032 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2668076060 0x7f266819e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.816+0000 7f2667200640 1 -- 172.21.15.45:0/2001021032 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26681a5c40 con 0x7f2668077c40 2024-03-28T17:48:30.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.816+0000 7f2666800640 1 --2- 172.21.15.45:0/2001021032 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2668076060 0x7f266819e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:30.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.816+0000 7f2667200640 1 --2- 172.21.15.45:0/2001021032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2668077c40 0x7f26681a55d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2660009fd0 tx=0x7f266000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:30.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.816+0000 7f265fe00640 1 -- 172.21.15.45:0/2001021032 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f266000ce20 con 0x7f2668077c40 2024-03-28T17:48:30.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.816+0000 7f265fe00640 1 -- 172.21.15.45:0/2001021032 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2660005020 con 0x7f2668077c40 2024-03-28T17:48:30.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.816+0000 7f265fe00640 1 -- 172.21.15.45:0/2001021032 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2660010720 con 0x7f2668077c40 2024-03-28T17:48:30.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.816+0000 7f266d400640 1 -- 172.21.15.45:0/2001021032 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26681a5f30 con 0x7f2668077c40 2024-03-28T17:48:30.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.817+0000 7f266d400640 1 -- 172.21.15.45:0/2001021032 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26681a63f0 con 0x7f2668077c40 2024-03-28T17:48:30.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.821+0000 7f2670175740 1 Processor -- start 2024-03-28T17:48:30.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.821+0000 7f265fe00640 1 -- 172.21.15.45:0/2001021032 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2660005540 con 0x7f2668077c40 2024-03-28T17:48:30.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.821+0000 7f2670175740 1 -- start start 2024-03-28T17:48:30.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.822+0000 7f2670175740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019bdba0 0x55f2019bdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.822+0000 7f2670175740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 0x55f2019c3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.822+0000 7f2670175740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019c3700 0x55f2019c5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.822+0000 7f2670175740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f201922c90 con 0x55f2019bdba0 2024-03-28T17:48:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.822+0000 7f2670175740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2018cb980 con 0x55f2019c3700 2024-03-28T17:48:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.822+0000 7f2670175740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f20191ed30 con 0x55f2019be590 2024-03-28T17:48:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.822+0000 7f2666800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 0x55f2019c3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.822+0000 7f2666800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 0x55f2019c3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:41522/0 (socket says 172.21.15.45:41522) 2024-03-28T17:48:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.822+0000 7f2666800640 1 -- 172.21.15.45:0/2559905217 learned_addr learned my addr 172.21.15.45:0/2559905217 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.822+0000 7f265de00640 1 --2- 172.21.15.45:0/2559905217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019bdba0 0x55f2019bdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.822+0000 7f265de00640 1 --2- 172.21.15.45:0/2559905217 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019c3700 0x55f2019c5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.822+0000 7f2666800640 1 -- 172.21.15.45:0/2559905217 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019c3700 msgr2=0x55f2019c5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.822+0000 7f2666800640 1 --2- 172.21.15.45:0/2559905217 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019c3700 0x55f2019c5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.822+0000 7f2666800640 1 -- 172.21.15.45:0/2559905217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019bdba0 msgr2=0x55f2019bdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.823+0000 7f2666800640 1 --2- 172.21.15.45:0/2559905217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019bdba0 0x55f2019bdfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.823+0000 7f2666800640 1 -- 172.21.15.45:0/2559905217 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2018e3510 con 0x55f2019be590 2024-03-28T17:48:30.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.823+0000 7f265de00640 1 --2- 172.21.15.45:0/2559905217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019bdba0 0x55f2019bdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:48:30.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.823+0000 7f265fe00640 1 --2- 172.21.15.45:0/2001021032 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f264c0a2880 0x7f264c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.823+0000 7f2666800640 1 --2- 172.21.15.45:0/2559905217 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 0x55f2019c3090 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2658011650 tx=0x7f2658009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8edf1a41f6efbb2c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:30.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.823+0000 7f265fe00640 1 -- 172.21.15.45:0/2001021032 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f26600cf090 con 0x7f2668077c40 2024-03-28T17:48:30.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.823+0000 7f265de00640 1 --2- 172.21.15.45:0/2001021032 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f264c0a2880 0x7f264c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.823+0000 7f2657400640 1 -- 172.21.15.45:0/2559905217 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265801e070 con 0x55f2019be590 2024-03-28T17:48:30.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.823+0000 7f2657400640 1 -- 172.21.15.45:0/2559905217 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f265800fba0 con 0x55f2019be590 2024-03-28T17:48:30.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.823+0000 7f2657400640 1 -- 172.21.15.45:0/2559905217 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2658019cd0 con 0x55f2019be590 2024-03-28T17:48:30.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.824+0000 7f2670175740 1 -- 172.21.15.45:0/2559905217 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 msgr2=0x55f2019c3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.824+0000 7f265de00640 1 --2- 172.21.15.45:0/2001021032 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f264c0a2880 0x7f264c0a4d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f265002c240 tx=0x7f2650005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:30.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.824+0000 7f2670175740 1 --2- 172.21.15.45:0/2559905217 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 0x55f2019c3090 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2658011650 tx=0x7f2658009f90 comp rx=0 tx=0).stop 2024-03-28T17:48:30.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.824+0000 7f2670175740 1 -- 172.21.15.45:0/2559905217 shutdown_connections 2024-03-28T17:48:30.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.824+0000 7f2670175740 1 --2- 172.21.15.45:0/2559905217 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019c3700 0x55f2019c5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.824+0000 7f2670175740 1 --2- 172.21.15.45:0/2559905217 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 0x55f2019c3090 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.824+0000 7f2670175740 1 --2- 172.21.15.45:0/2559905217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019bdba0 0x55f2019bdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.824+0000 7f2670175740 1 -- 172.21.15.45:0/2559905217 >> 172.21.15.45:0/2559905217 conn(0x55f2019b9110 msgr2=0x55f2019b9510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:30.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.824+0000 7f2670175740 1 -- 172.21.15.45:0/2559905217 shutdown_connections 2024-03-28T17:48:30.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.824+0000 7f2670175740 1 -- 172.21.15.45:0/2559905217 wait complete. 2024-03-28T17:48:30.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.825+0000 7f2670175740 1 -- start start 2024-03-28T17:48:30.828 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:30 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/661021843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:48:30.828 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:30 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/37716806' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:48:30.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.825+0000 7f2670175740 1 Processor -- start 2024-03-28T17:48:30.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.825+0000 7f2670175740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019bdba0 0x55f2019c8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.825+0000 7f2670175740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 0x55f2019c85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.825+0000 7f2670175740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019c3700 0x55f2019cf600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.825+0000 7f2670175740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2019c66b0 con 0x55f2019c3700 2024-03-28T17:48:30.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.825+0000 7f2670175740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2018dedd0 con 0x55f2019bdba0 2024-03-28T17:48:30.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.825+0000 7f2670175740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2019c6830 con 0x55f2019be590 2024-03-28T17:48:30.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.825+0000 7f2670175740 10 client.0 _ll_register_callbacks cb 0x55f201636ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:48:30.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.826+0000 7f2666800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019bdba0 0x55f2019c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.826+0000 7f2666800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019bdba0 0x55f2019c8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53710/0 (socket says 172.21.15.45:53710) 2024-03-28T17:48:30.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.826+0000 7f2666800640 1 -- 172.21.15.45:0/1932279719 learned_addr learned my addr 172.21.15.45:0/1932279719 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:30.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.826+0000 7f265de00640 1 --2- 172.21.15.45:0/1932279719 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 0x55f2019c85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.826+0000 7f2667200640 1 --2- 172.21.15.45:0/1932279719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019c3700 0x55f2019cf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.826+0000 7f2666800640 1 -- 172.21.15.45:0/1932279719 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 msgr2=0x55f2019c85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.826+0000 7f2666800640 1 --2- 172.21.15.45:0/1932279719 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 0x55f2019c85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.826+0000 7f2666800640 1 -- 172.21.15.45:0/1932279719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019c3700 msgr2=0x55f2019cf600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.826+0000 7f2666800640 1 --2- 172.21.15.45:0/1932279719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019c3700 0x55f2019cf600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.826+0000 7f2666800640 1 -- 172.21.15.45:0/1932279719 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f2016372d0 con 0x55f2019bdba0 2024-03-28T17:48:30.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.826+0000 7f2667200640 1 --2- 172.21.15.45:0/1932279719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019c3700 0x55f2019cf600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:48:30.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.826+0000 7f2666800640 1 --2- 172.21.15.45:0/1932279719 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019bdba0 0x55f2019c8080 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f265801b000 tx=0x7f2658005cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=719529b0a9751b45 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:30.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.826+0000 7f264be00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:48:30.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.826+0000 7f264be00640 1 -- 172.21.15.45:0/1932279719 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265801e040 con 0x55f2019bdba0 2024-03-28T17:48:30.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.827+0000 7f264be00640 1 -- 172.21.15.45:0/1932279719 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f2658009cc0 con 0x55f2019bdba0 2024-03-28T17:48:30.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.827+0000 7f2670175740 1 -- 172.21.15.45:0/1932279719 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f265800fba0 con 0x55f2019bdba0 2024-03-28T17:48:30.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.827+0000 7f264be00640 1 -- 172.21.15.45:0/1932279719 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f265800fe50 con 0x55f2019bdba0 2024-03-28T17:48:30.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.827+0000 7f2670175740 10 client.25187 fetch_fsmap learned FSMap version 22 2024-03-28T17:48:30.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.827+0000 7f2670175740 1 -- 172.21.15.45:0/1932279719 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55f2017016b0 con 0x55f2019bdba0 2024-03-28T17:48:30.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.827+0000 7f264be00640 1 -- 172.21.15.45:0/1932279719 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f265800fba0 con 0x55f2019bdba0 2024-03-28T17:48:30.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.827+0000 7f264be00640 1 -- 172.21.15.45:0/1932279719 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f2658021070 con 0x55f2019bdba0 2024-03-28T17:48:30.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.827+0000 7f2670175740 10 client.25187 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T17:48:30.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.827+0000 7f2670175740 10 client.25187 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T17:48:30.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.828+0000 7f2670175740 20 client.25187 populate_metadata read hostname 'smithi045' 2024-03-28T17:48:30.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.828+0000 7f2670175740 1 --2- 172.21.15.45:0/1932279719 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f2019d4560 0x55f2019d37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.828+0000 7f2670175740 4 client.25187 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:48:30.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.828+0000 7f2670175740 1 -- 172.21.15.45:0/1932279719 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f2018c4520 con 0x55f2019d4560 2024-03-28T17:48:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.828+0000 7f265de00640 1 --2- 172.21.15.45:0/1932279719 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f2019d4560 0x55f2019d37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.828+0000 7f265de00640 1 --2- 172.21.15.45:0/1932279719 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f2019d4560 0x55f2019d37b0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fa3b075e89dc637 server_cookie=c4859a25ac831969 in_seq=0 out_seq=0 2024-03-28T17:48:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.828+0000 7f264be00640 10 client.25187 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:48:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.829+0000 7f264be00640 1 -- 172.21.15.45:0/1932279719 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f2018c4520 con 0x55f2019d4560 2024-03-28T17:48:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.829+0000 7f264be00640 10 client.25187 handle_command_reply: tid=0 2024-03-28T17:48:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.829+0000 7f2670175740 1 client.25187 shutdown 2024-03-28T17:48:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.829+0000 7f2670175740 1 -- 172.21.15.45:0/1932279719 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019bdba0 msgr2=0x55f2019c8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.829+0000 7f2670175740 1 --2- 172.21.15.45:0/1932279719 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019bdba0 0x55f2019c8080 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f265801b000 tx=0x7f2658005cd0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.829+0000 7f2670175740 1 -- 172.21.15.45:0/1932279719 shutdown_connections 2024-03-28T17:48:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.829+0000 7f2670175740 1 --2- 172.21.15.45:0/1932279719 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f2019d4560 0x55f2019d37b0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.829+0000 7f2670175740 1 --2- 172.21.15.45:0/1932279719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019c3700 0x55f2019cf600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.829+0000 7f2670175740 1 --2- 172.21.15.45:0/1932279719 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 0x55f2019c85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.829+0000 7f2670175740 1 --2- 172.21.15.45:0/1932279719 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019bdba0 0x55f2019c8080 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.829+0000 7f264be00640 0 client.25187 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:48:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.829+0000 7f2670175740 1 -- 172.21.15.45:0/1932279719 >> 172.21.15.45:0/1932279719 conn(0x55f2019b9110 msgr2=0x55f2019b8310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:30.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.829+0000 7f2670175740 1 -- 172.21.15.45:0/1932279719 shutdown_connections 2024-03-28T17:48:30.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.830+0000 7f2670175740 1 -- 172.21.15.45:0/1932279719 wait complete. 2024-03-28T17:48:30.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.830+0000 7f2670175740 20 client.25187 trim_cache size 0 max 16384 2024-03-28T17:48:30.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.848+0000 7f2670175740 1 Processor -- start 2024-03-28T17:48:30.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.848+0000 7f2670175740 1 -- start start 2024-03-28T17:48:30.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.848+0000 7f2670175740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019bdba0 0x55f2019b7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.848+0000 7f2670175740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 0x55f2019cf520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.848+0000 7f2670175740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019c3700 0x55f2019b7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.848+0000 7f2670175740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2019d3d80 con 0x55f2019c3700 2024-03-28T17:48:30.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.848+0000 7f2670175740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2019d43e0 con 0x55f2019bdba0 2024-03-28T17:48:30.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.848+0000 7f2670175740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2019d4a40 con 0x55f2019be590 2024-03-28T17:48:30.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.849+0000 7f265de00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 0x55f2019cf520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.849+0000 7f265de00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 0x55f2019cf520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:41534/0 (socket says 172.21.15.45:41534) 2024-03-28T17:48:30.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.849+0000 7f265de00640 1 -- 172.21.15.45:0/1734488334 learned_addr learned my addr 172.21.15.45:0/1734488334 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:30.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.849+0000 7f2667200640 1 --2- 172.21.15.45:0/1734488334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019c3700 0x55f2019b7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.849+0000 7f2666800640 1 --2- 172.21.15.45:0/1734488334 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019bdba0 0x55f2019b7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.849+0000 7f265de00640 1 -- 172.21.15.45:0/1734488334 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019bdba0 msgr2=0x55f2019b7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.849+0000 7f265de00640 1 --2- 172.21.15.45:0/1734488334 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019bdba0 0x55f2019b7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.849+0000 7f265de00640 1 -- 172.21.15.45:0/1734488334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019c3700 msgr2=0x55f2019b7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.849+0000 7f265de00640 1 --2- 172.21.15.45:0/1734488334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019c3700 0x55f2019b7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.849+0000 7f265de00640 1 -- 172.21.15.45:0/1734488334 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2019b9e80 con 0x55f2019be590 2024-03-28T17:48:30.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.849+0000 7f2666800640 1 --2- 172.21.15.45:0/1734488334 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019bdba0 0x55f2019b7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:48:30.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.849+0000 7f265de00640 1 --2- 172.21.15.45:0/1734488334 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 0x55f2019cf520 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2650009900 tx=0x7f2650043000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbdb45e9022b3e3c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:30.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.850+0000 7f2656a00640 1 -- 172.21.15.45:0/1734488334 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26500561e0 con 0x55f2019be590 2024-03-28T17:48:30.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.850+0000 7f2656a00640 1 -- 172.21.15.45:0/1734488334 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f265005b070 con 0x55f2019be590 2024-03-28T17:48:30.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.850+0000 7f2656a00640 1 -- 172.21.15.45:0/1734488334 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2650005110 con 0x55f2019be590 2024-03-28T17:48:30.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.850+0000 7f2670175740 1 -- 172.21.15.45:0/1734488334 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 msgr2=0x55f2019cf520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.850+0000 7f2670175740 1 --2- 172.21.15.45:0/1734488334 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 0x55f2019cf520 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2650009900 tx=0x7f2650043000 comp rx=0 tx=0).stop 2024-03-28T17:48:30.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.850+0000 7f2670175740 1 -- 172.21.15.45:0/1734488334 shutdown_connections 2024-03-28T17:48:30.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.850+0000 7f2670175740 1 --2- 172.21.15.45:0/1734488334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019c3700 0x55f2019b7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.850+0000 7f2670175740 1 --2- 172.21.15.45:0/1734488334 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 0x55f2019cf520 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.850+0000 7f2670175740 1 --2- 172.21.15.45:0/1734488334 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019bdba0 0x55f2019b7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.850+0000 7f2670175740 1 -- 172.21.15.45:0/1734488334 >> 172.21.15.45:0/1734488334 conn(0x55f2019b9110 msgr2=0x55f2019b94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:30.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.850+0000 7f2670175740 1 -- 172.21.15.45:0/1734488334 shutdown_connections 2024-03-28T17:48:30.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.850+0000 7f2670175740 1 -- 172.21.15.45:0/1734488334 wait complete. 2024-03-28T17:48:30.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.851+0000 7f2670175740 1 -- start start 2024-03-28T17:48:30.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.851+0000 7f2670175740 1 Processor -- start 2024-03-28T17:48:30.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.851+0000 7f2670175740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019bdba0 0x55f2019cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.851+0000 7f2670175740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 0x55f2019cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.851+0000 7f2670175740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019c3700 0x55f2019ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.851+0000 7f2670175740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2019d5700 con 0x55f2019c3700 2024-03-28T17:48:30.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.851+0000 7f2670175740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2019d50a0 con 0x55f2019bdba0 2024-03-28T17:48:30.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.851+0000 7f2670175740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2019d5d60 con 0x55f2019be590 2024-03-28T17:48:30.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.851+0000 7f2670175740 10 client.0 _ll_register_callbacks cb 0x7f265801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:48:30.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.852+0000 7f2667200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019c3700 0x55f2019ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.852+0000 7f2666800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019bdba0 0x55f2019cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.852+0000 7f265de00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 0x55f2019cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.852+0000 7f2666800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019bdba0 0x55f2019cb5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53722/0 (socket says 172.21.15.45:53722) 2024-03-28T17:48:30.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.852+0000 7f265de00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 0x55f2019cbae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:41546/0 (socket says 172.21.15.45:41546) 2024-03-28T17:48:30.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.852+0000 7f2667200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019c3700 0x55f2019ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50702/0 (socket says 172.21.15.45:50702) 2024-03-28T17:48:30.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.852+0000 7f265de00640 1 -- 172.21.15.45:0/2504737211 learned_addr learned my addr 172.21.15.45:0/2504737211 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:30.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.852+0000 7f2666800640 1 -- 172.21.15.45:0/2504737211 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 msgr2=0x55f2019cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.852+0000 7f2666800640 1 --2- 172.21.15.45:0/2504737211 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 0x55f2019cbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.852+0000 7f2666800640 1 -- 172.21.15.45:0/2504737211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019c3700 msgr2=0x55f2019ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:30.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.852+0000 7f2666800640 1 --2- 172.21.15.45:0/2504737211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019c3700 0x55f2019ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.852+0000 7f2666800640 1 -- 172.21.15.45:0/2504737211 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f2019cf0f0 con 0x55f2019bdba0 2024-03-28T17:48:30.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.852+0000 7f2667200640 1 --2- 172.21.15.45:0/2504737211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019c3700 0x55f2019ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:30.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.852+0000 7f2666800640 1 --2- 172.21.15.45:0/2504737211 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019bdba0 0x55f2019cb5a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f265801c040 tx=0x7f2658009530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=691bb8159137eac6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:30.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.852+0000 7f264be00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:48:30.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.852+0000 7f264be00640 1 -- 172.21.15.45:0/2504737211 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2658031030 con 0x55f2019bdba0 2024-03-28T17:48:30.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.853+0000 7f264be00640 1 -- 172.21.15.45:0/2504737211 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f265800f9d0 con 0x55f2019bdba0 2024-03-28T17:48:30.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.853+0000 7f2670175740 1 -- 172.21.15.45:0/2504737211 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f265005b070 con 0x55f2019bdba0 2024-03-28T17:48:30.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.853+0000 7f264be00640 1 -- 172.21.15.45:0/2504737211 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f265801e040 con 0x55f2019bdba0 2024-03-28T17:48:30.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.853+0000 7f2670175740 10 client.25193 fetch_fsmap learned FSMap version 22 2024-03-28T17:48:30.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.853+0000 7f2670175740 1 -- 172.21.15.45:0/2504737211 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55f201a2fe80 con 0x55f2019bdba0 2024-03-28T17:48:30.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.853+0000 7f264be00640 1 -- 172.21.15.45:0/2504737211 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f265005b070 con 0x55f2019bdba0 2024-03-28T17:48:30.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.853+0000 7f264be00640 1 -- 172.21.15.45:0/2504737211 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 22) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f2658021050 con 0x55f2019bdba0 2024-03-28T17:48:30.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.853+0000 7f2670175740 10 client.25193 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T17:48:30.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.853+0000 7f2670175740 10 client.25193 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T17:48:30.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.853+0000 7f2670175740 20 client.25193 populate_metadata read hostname 'smithi045' 2024-03-28T17:48:30.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.853+0000 7f2670175740 1 --2- 172.21.15.45:0/2504737211 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f201a34460 0x55f201a300b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:30.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.853+0000 7f2670175740 4 client.25193 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T17:48:30.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.853+0000 7f2670175740 1 -- 172.21.15.45:0/2504737211 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f2019301d0 con 0x55f201a34460 2024-03-28T17:48:30.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.854+0000 7f265de00640 1 --2- 172.21.15.45:0/2504737211 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f201a34460 0x55f201a300b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:30.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.854+0000 7f265de00640 1 --2- 172.21.15.45:0/2504737211 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f201a34460 0x55f201a300b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b58a79ccd65de3aa server_cookie=d1b3f00431d8d4bf in_seq=0 out_seq=0 2024-03-28T17:48:30.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.854+0000 7f264be00640 10 client.25193 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:48:30.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.854+0000 7f264be00640 1 -- 172.21.15.45:0/2504737211 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x55f2019301d0 con 0x55f201a34460 2024-03-28T17:48:30.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.854+0000 7f264be00640 10 client.25193 handle_command_reply: tid=0 2024-03-28T17:48:30.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.854+0000 7f2670175740 1 client.25193 shutdown 2024-03-28T17:48:30.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.854+0000 7f2670175740 1 -- 172.21.15.45:0/2504737211 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019bdba0 msgr2=0x55f2019cb5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.854+0000 7f2670175740 1 --2- 172.21.15.45:0/2504737211 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019bdba0 0x55f2019cb5a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f265801c040 tx=0x7f2658009530 comp rx=0 tx=0).stop 2024-03-28T17:48:30.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.854+0000 7f2670175740 1 -- 172.21.15.45:0/2504737211 shutdown_connections 2024-03-28T17:48:30.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.854+0000 7f2670175740 1 --2- 172.21.15.45:0/2504737211 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f201a34460 0x55f201a300b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.854+0000 7f2670175740 1 --2- 172.21.15.45:0/2504737211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f2019c3700 0x55f2019ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.854+0000 7f2670175740 1 --2- 172.21.15.45:0/2504737211 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f2019be590 0x55f2019cbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.854+0000 7f2670175740 1 --2- 172.21.15.45:0/2504737211 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f2019bdba0 0x55f2019cb5a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.854+0000 7f264be00640 0 client.25193 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:48:30.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.854+0000 7f2670175740 1 -- 172.21.15.45:0/2504737211 >> 172.21.15.45:0/2504737211 conn(0x55f2019b9110 msgr2=0x55f2019bc370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:30.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.855+0000 7f2670175740 1 -- 172.21.15.45:0/2504737211 shutdown_connections 2024-03-28T17:48:30.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.855+0000 7f2670175740 1 -- 172.21.15.45:0/2504737211 wait complete. 2024-03-28T17:48:30.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.855+0000 7f2670175740 20 client.25193 trim_cache size 0 max 16384 2024-03-28T17:48:30.860 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:48:30.860 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "9f4888ec-ed29-11ee-95d0-87774f69a715", 2024-03-28T17:48:30.860 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2024-03-28T17:48:30.860 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 24584, 2024-03-28T17:48:30.860 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-03-28T17:48:30.860 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-03-28T17:48:30.860 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-03-28T17:48:30.861 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 137.610681051, 2024-03-28T17:48:30.861 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 22, 2024-03-28T17:48:30.861 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 87, 2024-03-28T17:48:30.861 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 85, 2024-03-28T17:48:30.861 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 145.057733982 2024-03-28T17:48:30.861 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:48:30.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.855+0000 7f266d400640 1 -- 172.21.15.45:0/2001021032 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f264c0a2880 msgr2=0x7f264c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.855+0000 7f266d400640 1 --2- 172.21.15.45:0/2001021032 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f264c0a2880 0x7f264c0a4d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f265002c240 tx=0x7f2650005fd0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.855+0000 7f266d400640 1 -- 172.21.15.45:0/2001021032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2668077c40 msgr2=0x7f26681a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:30.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.855+0000 7f266d400640 1 --2- 172.21.15.45:0/2001021032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2668077c40 0x7f26681a55d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2660009fd0 tx=0x7f266000ef10 comp rx=0 tx=0).stop 2024-03-28T17:48:30.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.855+0000 7f266d400640 1 -- 172.21.15.45:0/2001021032 shutdown_connections 2024-03-28T17:48:30.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.855+0000 7f266d400640 1 --2- 172.21.15.45:0/2001021032 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f264c0a2880 0x7f264c0a4d40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.855+0000 7f266d400640 1 --2- 172.21.15.45:0/2001021032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2668077c40 0x7f26681a55d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.855+0000 7f266d400640 1 --2- 172.21.15.45:0/2001021032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f26680772a0 0x7f266819e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.855+0000 7f266d400640 1 --2- 172.21.15.45:0/2001021032 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2668076060 0x7f266819e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:30.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.855+0000 7f266d400640 1 -- 172.21.15.45:0/2001021032 >> 172.21.15.45:0/2001021032 conn(0x7f26680fee50 msgr2=0x7f2668100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:30.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.856+0000 7f266d400640 1 -- 172.21.15.45:0/2001021032 shutdown_connections 2024-03-28T17:48:30.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:30.856+0000 7f266d400640 1 -- 172.21.15.45:0/2001021032 wait complete. 2024-03-28T17:48:30.872 INFO:teuthology.run_tasks:Running task kclient... 2024-03-28T17:48:30.896 INFO:tasks.kclient:Mounting kernel clients... 2024-03-28T17:48:30.897 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-28T17:48:30.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T17:48:30.897 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-03-28T17:48:30.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T17:48:30.951 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link delete ceph-brx 2024-03-28T17:48:31.057 INFO:teuthology.orchestra.run.smithi045.stderr:Cannot find device "ceph-brx" 2024-03-28T17:48:31.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T17:48:31.062 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-03-28T17:48:31.062 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T17:48:31.062 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T17:48:31.063 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 9f4888ec-ed29-11ee-95d0-87774f69a715 -- ceph fs ls 2024-03-28T17:48:31.529 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T17:48:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:31 smithi071 ceph-mon[36597]: pgmap v388: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T17:48:31.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:31 smithi151 ceph-mon[36481]: pgmap v388: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T17:48:31.851 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:31 smithi045 ceph-mon[31773]: pgmap v388: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T17:48:33.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.704+0000 7f450e400700 1 -- 172.21.15.45:0/1552122666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4508101140 msgr2=0x7f450810f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:33.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.704+0000 7f450e400700 1 --2- 172.21.15.45:0/1552122666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4508101140 0x7f450810f0a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4500009f90 tx=0x7f450000b330 comp rx=0 tx=0).stop 2024-03-28T17:48:33.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.705+0000 7f450e400700 1 -- 172.21.15.45:0/1552122666 shutdown_connections 2024-03-28T17:48:33.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.705+0000 7f450e400700 1 --2- 172.21.15.45:0/1552122666 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4508107200 0x7f45081075e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:33.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.705+0000 7f450e400700 1 --2- 172.21.15.45:0/1552122666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4508101140 0x7f450810f0a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:33.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.705+0000 7f450e400700 1 --2- 172.21.15.45:0/1552122666 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4508100780 0x7f4508100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:33.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.705+0000 7f450e400700 1 -- 172.21.15.45:0/1552122666 >> 172.21.15.45:0/1552122666 conn(0x7f4508077690 msgr2=0x7f45080feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:33.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.705+0000 7f450e400700 1 -- 172.21.15.45:0/1552122666 shutdown_connections 2024-03-28T17:48:33.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.705+0000 7f450e400700 1 -- 172.21.15.45:0/1552122666 wait complete. 2024-03-28T17:48:33.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.706+0000 7f450e400700 1 Processor -- start 2024-03-28T17:48:33.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.706+0000 7f450e400700 1 -- start start 2024-03-28T17:48:33.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.707+0000 7f450e400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4508100780 0x7f450819e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:33.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.707+0000 7f450e400700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4508101140 0x7f450819ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:33.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.707+0000 7f450e400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4508107200 0x7f4508198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:33.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.707+0000 7f450e400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4508112610 con 0x7f4508107200 2024-03-28T17:48:33.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.707+0000 7f450e400700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4508112490 con 0x7f4508100780 2024-03-28T17:48:33.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.707+0000 7f450e400700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4508112790 con 0x7f4508101140 2024-03-28T17:48:33.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.707+0000 7f4507400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4508100780 0x7f450819e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:33.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.707+0000 7f4507400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4508100780 0x7f450819e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53746/0 (socket says 172.21.15.45:53746) 2024-03-28T17:48:33.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.707+0000 7f4507400700 1 -- 172.21.15.45:0/1851835245 learned_addr learned my addr 172.21.15.45:0/1851835245 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:33.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.707+0000 7f44ffe00700 1 --2- 172.21.15.45:0/1851835245 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4508101140 0x7f450819ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:33.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.707+0000 7f4507e00700 1 --2- 172.21.15.45:0/1851835245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4508107200 0x7f4508198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:33.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.708+0000 7f4507400700 1 -- 172.21.15.45:0/1851835245 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4508101140 msgr2=0x7f450819ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:33.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.708+0000 7f4507400700 1 --2- 172.21.15.45:0/1851835245 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4508101140 0x7f450819ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:33.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.708+0000 7f4507400700 1 -- 172.21.15.45:0/1851835245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4508107200 msgr2=0x7f4508198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:33.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.708+0000 7f4507400700 1 --2- 172.21.15.45:0/1851835245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4508107200 0x7f4508198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:33.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.708+0000 7f4507400700 1 -- 172.21.15.45:0/1851835245 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4508199010 con 0x7f4508100780 2024-03-28T17:48:33.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.708+0000 7f4507e00700 1 --2- 172.21.15.45:0/1851835245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4508107200 0x7f4508198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:48:33.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.708+0000 7f4507400700 1 --2- 172.21.15.45:0/1851835245 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4508100780 0x7f450819e6b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f44f800e9c0 tx=0x7f44f800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:33.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.708+0000 7f4504c00700 1 -- 172.21.15.45:0/1851835245 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f800cbd0 con 0x7f4508100780 2024-03-28T17:48:33.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.708+0000 7f4504c00700 1 -- 172.21.15.45:0/1851835245 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44f800cd70 con 0x7f4508100780 2024-03-28T17:48:33.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.708+0000 7f4504c00700 1 -- 172.21.15.45:0/1851835245 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f8005660 con 0x7f4508100780 2024-03-28T17:48:33.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.708+0000 7f450e400700 1 -- 172.21.15.45:0/1851835245 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4508199300 con 0x7f4508100780 2024-03-28T17:48:33.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.708+0000 7f450e400700 1 -- 172.21.15.45:0/1851835245 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4508101890 con 0x7f4508100780 2024-03-28T17:48:33.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.710+0000 7f450e400700 1 -- 172.21.15.45:0/1851835245 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4508068e50 con 0x7f4508100780 2024-03-28T17:48:33.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.711+0000 7f4504c00700 1 -- 172.21.15.45:0/1851835245 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f44f8004750 con 0x7f4508100780 2024-03-28T17:48:33.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.713+0000 7f4504c00700 1 --2- 172.21.15.45:0/1851835245 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f44ec0a2910 0x7f44ec0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:33.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.713+0000 7f4504c00700 1 -- 172.21.15.45:0/1851835245 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f44f80ceb60 con 0x7f4508100780 2024-03-28T17:48:33.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.713+0000 7f44ffe00700 1 --2- 172.21.15.45:0/1851835245 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f44ec0a2910 0x7f44ec0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:33.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.714+0000 7f44ffe00700 1 --2- 172.21.15.45:0/1851835245 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f44ec0a2910 0x7f44ec0a4dd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4508106a00 tx=0x7f44f000f7a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:33.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.717+0000 7f4504c00700 1 -- 172.21.15.45:0/1851835245 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f44f809bbb0 con 0x7f4508100780 2024-03-28T17:48:33.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:33 smithi071 ceph-mon[36597]: pgmap v389: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:48:33.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:33 smithi151 ceph-mon[36481]: pgmap v389: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:48:33.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:33 smithi045 ceph-mon[31773]: pgmap v389: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:48:33.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.994+0000 7f450e400700 1 -- 172.21.15.45:0/1851835245 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f450802da30 con 0x7f4508100780 2024-03-28T17:48:33.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.995+0000 7f4504c00700 1 -- 172.21.15.45:0/1851835245 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v22) v1 ==== 53+0+73 (secure 0 0 0) 0x7f44f809b4b0 con 0x7f4508100780 2024-03-28T17:48:33.995 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-28T17:48:33.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.998+0000 7f450e400700 1 -- 172.21.15.45:0/1851835245 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f44ec0a2910 msgr2=0x7f44ec0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:33.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.998+0000 7f450e400700 1 --2- 172.21.15.45:0/1851835245 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f44ec0a2910 0x7f44ec0a4dd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4508106a00 tx=0x7f44f000f7a0 comp rx=0 tx=0).stop 2024-03-28T17:48:33.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.998+0000 7f450e400700 1 -- 172.21.15.45:0/1851835245 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4508100780 msgr2=0x7f450819e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:33.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.998+0000 7f450e400700 1 --2- 172.21.15.45:0/1851835245 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4508100780 0x7f450819e6b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f44f800e9c0 tx=0x7f44f800ecd0 comp rx=0 tx=0).stop 2024-03-28T17:48:33.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.998+0000 7f450e400700 1 -- 172.21.15.45:0/1851835245 shutdown_connections 2024-03-28T17:48:33.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.998+0000 7f450e400700 1 --2- 172.21.15.45:0/1851835245 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f44ec0a2910 0x7f44ec0a4dd0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:33.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.998+0000 7f450e400700 1 --2- 172.21.15.45:0/1851835245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4508107200 0x7f4508198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:33.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.998+0000 7f450e400700 1 --2- 172.21.15.45:0/1851835245 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4508100780 0x7f450819e6b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:33.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.998+0000 7f450e400700 1 --2- 172.21.15.45:0/1851835245 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4508101140 0x7f450819ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:33.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.998+0000 7f450e400700 1 -- 172.21.15.45:0/1851835245 >> 172.21.15.45:0/1851835245 conn(0x7f4508077690 msgr2=0x7f4508105150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:33.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.998+0000 7f450e400700 1 -- 172.21.15.45:0/1851835245 shutdown_connections 2024-03-28T17:48:33.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:33.998+0000 7f450e400700 1 -- 172.21.15.45:0/1851835245 wait complete. 2024-03-28T17:48:34.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:34 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1851835245' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T17:48:34.793 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-28T17:48:34.793 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T17:48:34.793 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-03-28T17:48:34.793 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-28T17:48:34.793 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T17:48:34.793 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-28T17:48:34.794 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T17:48:34.794 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T17:48:34.794 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-28T17:48:34.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T17:48:34.794 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-03-28T17:48:34.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:34 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1851835245' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T17:48:34.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:34 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1851835245' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T17:48:34.846 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T17:48:34.846 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T17:48:34.846 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T17:48:34.847 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-03-28T17:48:34.847 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-03-28T17:48:34.847 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-03-28T17:48:34.847 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T17:48:34.847 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-03-28T17:48:34.847 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-03-28T17:48:34.847 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T17:48:34.847 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-03-28T17:48:34.847 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-03-28T17:48:34.847 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T17:48:34.847 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-03-28T17:48:34.847 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-03-28T17:48:34.847 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-28T17:48:34.847 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 41890sec preferred_lft 41890sec 2024-03-28T17:48:34.848 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2024-03-28T17:48:34.848 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-03-28T17:48:34.848 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T17:48:34.848 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-03-28T17:48:34.848 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-03-28T17:48:34.849 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T17:48:34.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T17:48:34.849 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2024-03-28T17:48:34.849 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-03-28T17:48:34.849 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link add name ceph-brx type bridge 2024-03-28T17:48:34.849 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip addr flush dev ceph-brx 2024-03-28T17:48:34.849 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set ceph-brx up 2024-03-28T17:48:34.849 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T17:48:34.850 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-03-28T17:48:35.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T17:48:35.287 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T17:48:35.366 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-03-28T17:48:35.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T17:48:35.370 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-03-28T17:48:35.423 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-03-28T17:48:35.423 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T17:48:35.423 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-28T17:48:35.423 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-28T17:48:35.423 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T17:48:35.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T17:48:35.424 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2024-03-28T17:48:35.424 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-03-28T17:48:35.424 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-28T17:48:35.424 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-28T17:48:35.424 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-28T17:48:35.424 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-03-28T17:48:35.769 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:35 smithi045 ceph-mon[31773]: pgmap v390: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:35.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:35 smithi071 ceph-mon[36597]: pgmap v390: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:35.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:35 smithi151 ceph-mon[36481]: pgmap v390: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:36.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T17:48:36.132 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-03-28T17:48:36.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T17:48:36.188 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list-id 2024-03-28T17:48:36.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T17:48:36.247 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2024-03-28T17:48:36.247 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-03-28T17:48:36.247 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T17:48:36.247 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-28T17:48:36.247 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-03-28T17:48:36.400 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-28T17:48:36.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T17:48:36.401 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2024-03-28T17:48:36.401 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-03-28T17:48:36.401 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-28T17:48:36.401 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T17:48:36.401 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-28T17:48:36.401 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-28T17:48:36.401 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-28T17:48:36.401 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-03-28T17:48:36.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T17:48:36.689 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2024-03-28T17:48:36.689 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-03-28T17:48:36.689 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-03-28T17:48:36.689 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T17:48:36.689 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-03-28T17:48:36.858 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T17:48:36.858 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-28T17:48:36.905 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-28T17:48:36.905 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T17:48:36.905 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-28T17:48:37.002 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-28T17:48:37.002 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-28T17:48:37.002 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/e789f540-3642-4eb0-b4d2-e1b55b454a1f 2024-03-28T17:48:37.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T17:48:37.002 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/e789f540-3642-4eb0-b4d2-e1b55b454a1f /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-03-28T17:48:37.148 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-03-28T17:48:37.148 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2024-03-28T17:48:37.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:37 smithi045 ceph-mon[31773]: pgmap v391: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:48:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:37 smithi071 ceph-mon[36597]: pgmap v391: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:48:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:37 smithi151 ceph-mon[36481]: pgmap v391: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T17:48:37.823 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-03-28T17:48:37.823 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2024-03-28T17:48:37.823 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:6789,172.21.15.71:6789,172.21.15.151:6789" 2024-03-28T17:48:37.824 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with new device syntax: 0@9f4888ec-ed29-11ee-95d0-87774f69a715.cephfs=/volumes/qa/sv_1/e789f540-3642-4eb0-b4d2-e1b55b454a1f 2024-03-28T17:48:37.824 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync,name=0,key=0,mon_addr=172.21.15.45:6789/172.21.15.71:6789/172.21.15.151:6789" will pass to kernel 2024-03-28T17:48:37.827 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-28T17:48:37.827 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2024-03-28T17:48:37.827 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-28T17:48:37.827 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2024-03-28T17:48:37.827 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-28T17:48:37.832 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-28T17:48:37.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T17:48:37.832 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-28T17:48:37.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T17:48:37.932 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T17:48:37.992 INFO:teuthology.orchestra.run.smithi045.stdout:client15069 2024-03-28T17:48:37.992 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-28T17:48:37.992 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T17:48:38.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T17:48:38.048 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T17:48:38.103 INFO:teuthology.orchestra.run.smithi045.stdout:client15069 2024-03-28T17:48:38.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T17:48:38.104 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T17:48:38.161 INFO:teuthology.orchestra.run.smithi045.stdout:client15069 2024-03-28T17:48:38.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T17:48:38.161 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715.client15069/status') 2024-03-28T17:48:38.262 INFO:teuthology.orchestra.run.smithi045.stdout:instance: client.15069 (3)192.168.0.1:0/466750684 2024-03-28T17:48:38.262 INFO:teuthology.orchestra.run.smithi045.stdout:blocklisted: false 2024-03-28T17:48:38.262 INFO:teuthology.orchestra.run.smithi045.stderr:0+1 records in 2024-03-28T17:48:38.262 INFO:teuthology.orchestra.run.smithi045.stderr:0+1 records out 2024-03-28T17:48:38.262 INFO:teuthology.orchestra.run.smithi045.stderr:69 bytes copied, 8.0372e-05 s, 859 kB/s 2024-03-28T17:48:38.267 INFO:teuthology.run_tasks:Running task exec... 2024-03-28T17:48:38.278 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T17:48:38.278 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi045.front.sepia.ceph.com 2024-03-28T17:48:38.278 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-28T17:48:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.501+0000 7fb3c9400640 1 -- 172.21.15.45:0/1706345462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3c41028a0 msgr2=0x7fb3c4104d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.501+0000 7fb3c9400640 1 --2- 172.21.15.45:0/1706345462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3c41028a0 0x7fb3c4104d10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac009900 tx=0x7fb3ac02bc50 comp rx=0 tx=0).stop 2024-03-28T17:48:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.502+0000 7fb3c9400640 1 -- 172.21.15.45:0/1706345462 shutdown_connections 2024-03-28T17:48:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.502+0000 7fb3c9400640 1 --2- 172.21.15.45:0/1706345462 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb3c4105250 0x7fb3c41056d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.502+0000 7fb3c9400640 1 --2- 172.21.15.45:0/1706345462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3c41028a0 0x7fb3c4104d10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.502+0000 7fb3c9400640 1 --2- 172.21.15.45:0/1706345462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3c40fff30 0x7fb3c4102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:38.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.502+0000 7fb3c9400640 1 -- 172.21.15.45:0/1706345462 >> 172.21.15.45:0/1706345462 conn(0x7fb3c40fbb30 msgr2=0x7fb3c40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:38.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.502+0000 7fb3c9400640 1 -- 172.21.15.45:0/1706345462 shutdown_connections 2024-03-28T17:48:38.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.502+0000 7fb3c9400640 1 -- 172.21.15.45:0/1706345462 wait complete. 2024-03-28T17:48:38.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.503+0000 7fb3c9400640 1 Processor -- start 2024-03-28T17:48:38.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.503+0000 7fb3c9400640 1 -- start start 2024-03-28T17:48:38.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.504+0000 7fb3c9400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3c40fff30 0x7fb3c419df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:38.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.504+0000 7fb3c9400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3c41028a0 0x7fb3c419e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:38.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.504+0000 7fb3c9400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb3c4105250 0x7fb3c41a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:38.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.504+0000 7fb3c9400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c4113980 con 0x7fb3c41028a0 2024-03-28T17:48:38.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.504+0000 7fb3c9400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c4113800 con 0x7fb3c4105250 2024-03-28T17:48:38.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.504+0000 7fb3c9400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c4113b00 con 0x7fb3c40fff30 2024-03-28T17:48:38.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.504+0000 7fb3c2800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3c40fff30 0x7fb3c419df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:38.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.504+0000 7fb3c3200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb3c4105250 0x7fb3c41a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:38.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.504+0000 7fb3c2800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3c40fff30 0x7fb3c419df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:49554/0 (socket says 172.21.15.45:49554) 2024-03-28T17:48:38.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.504+0000 7fb3c2800640 1 -- 172.21.15.45:0/647048792 learned_addr learned my addr 172.21.15.45:0/647048792 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:38.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.504+0000 7fb3b9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3c41028a0 0x7fb3c419e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:38.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.504+0000 7fb3c3200640 1 -- 172.21.15.45:0/647048792 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3c40fff30 msgr2=0x7fb3c419df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:38.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.504+0000 7fb3c3200640 1 --2- 172.21.15.45:0/647048792 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3c40fff30 0x7fb3c419df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:38.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.504+0000 7fb3c3200640 1 -- 172.21.15.45:0/647048792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3c41028a0 msgr2=0x7fb3c419e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:38.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.504+0000 7fb3c3200640 1 --2- 172.21.15.45:0/647048792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3c41028a0 0x7fb3c419e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:38.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.504+0000 7fb3c3200640 1 -- 172.21.15.45:0/647048792 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c41a5b00 con 0x7fb3c4105250 2024-03-28T17:48:38.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.504+0000 7fb3c2800640 1 --2- 172.21.15.45:0/647048792 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3c40fff30 0x7fb3c419df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:38.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.504+0000 7fb3b9e00640 1 --2- 172.21.15.45:0/647048792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3c41028a0 0x7fb3c419e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:48:38.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.505+0000 7fb3c3200640 1 --2- 172.21.15.45:0/647048792 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb3c4105250 0x7fb3c41a5490 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc00e970 tx=0x7fb3bc00ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:38.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.505+0000 7fb3bbe00640 1 -- 172.21.15.45:0/647048792 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc00ccc0 con 0x7fb3c4105250 2024-03-28T17:48:38.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.505+0000 7fb3bbe00640 1 -- 172.21.15.45:0/647048792 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3bc005020 con 0x7fb3c4105250 2024-03-28T17:48:38.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.505+0000 7fb3bbe00640 1 -- 172.21.15.45:0/647048792 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc004a00 con 0x7fb3c4105250 2024-03-28T17:48:38.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.505+0000 7fb3c9400640 1 -- 172.21.15.45:0/647048792 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c41a5df0 con 0x7fb3c4105250 2024-03-28T17:48:38.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.505+0000 7fb3c9400640 1 -- 172.21.15.45:0/647048792 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c41a6300 con 0x7fb3c4105250 2024-03-28T17:48:38.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.506+0000 7fb3c9400640 1 -- 172.21.15.45:0/647048792 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3c40fd620 con 0x7fb3c4105250 2024-03-28T17:48:38.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.508+0000 7fb3bbe00640 1 -- 172.21.15.45:0/647048792 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb3bc0051d0 con 0x7fb3c4105250 2024-03-28T17:48:38.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.509+0000 7fb3bbe00640 1 --2- 172.21.15.45:0/647048792 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb3a80a2920 0x7fb3a80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:38.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.509+0000 7fb3bbe00640 1 -- 172.21.15.45:0/647048792 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb3bc0ce990 con 0x7fb3c4105250 2024-03-28T17:48:38.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.510+0000 7fb3c2800640 1 --2- 172.21.15.45:0/647048792 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb3a80a2920 0x7fb3a80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:38.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.511+0000 7fb3c2800640 1 --2- 172.21.15.45:0/647048792 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb3a80a2920 0x7fb3a80a4de0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb3b400e840 tx=0x7fb3b4009800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:38.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.513+0000 7fb3bbe00640 1 -- 172.21.15.45:0/647048792 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb3bc09b9e0 con 0x7fb3c4105250 2024-03-28T17:48:38.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:38.760+0000 7fb3c9400640 1 -- 172.21.15.45:0/647048792 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) v1 -- 0x7fb3c4062340 con 0x7fb3c4105250 2024-03-28T17:48:39.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.531+0000 7fb3bbe00640 1 -- 172.21.15.45:0/647048792 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v23) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb3bc09b2e0 con 0x7fb3c4105250 2024-03-28T17:48:39.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.534+0000 7fb3c9400640 1 -- 172.21.15.45:0/647048792 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb3a80a2920 msgr2=0x7fb3a80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:39.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.534+0000 7fb3c9400640 1 --2- 172.21.15.45:0/647048792 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb3a80a2920 0x7fb3a80a4de0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb3b400e840 tx=0x7fb3b4009800 comp rx=0 tx=0).stop 2024-03-28T17:48:39.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.534+0000 7fb3c9400640 1 -- 172.21.15.45:0/647048792 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb3c4105250 msgr2=0x7fb3c41a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:39.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.534+0000 7fb3c9400640 1 --2- 172.21.15.45:0/647048792 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb3c4105250 0x7fb3c41a5490 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc00e970 tx=0x7fb3bc00ee30 comp rx=0 tx=0).stop 2024-03-28T17:48:39.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.535+0000 7fb3c9400640 1 -- 172.21.15.45:0/647048792 shutdown_connections 2024-03-28T17:48:39.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.535+0000 7fb3c9400640 1 --2- 172.21.15.45:0/647048792 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb3a80a2920 0x7fb3a80a4de0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:39.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.535+0000 7fb3c9400640 1 --2- 172.21.15.45:0/647048792 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb3c4105250 0x7fb3c41a5490 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:39.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.535+0000 7fb3c9400640 1 --2- 172.21.15.45:0/647048792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3c41028a0 0x7fb3c419e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:39.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.535+0000 7fb3c9400640 1 --2- 172.21.15.45:0/647048792 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3c40fff30 0x7fb3c419df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:39.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.535+0000 7fb3c9400640 1 -- 172.21.15.45:0/647048792 >> 172.21.15.45:0/647048792 conn(0x7fb3c40fbb30 msgr2=0x7fb3c41008d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:39.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.535+0000 7fb3c9400640 1 -- 172.21.15.45:0/647048792 shutdown_connections 2024-03-28T17:48:39.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.535+0000 7fb3c9400640 1 -- 172.21.15.45:0/647048792 wait complete. 2024-03-28T17:48:39.558 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-28T17:48:39.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.785+0000 7faffba00640 1 -- 172.21.15.45:0/2035029628 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faff40772a0 msgr2=0x7faff4077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:39.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.785+0000 7faffba00640 1 --2- 172.21.15.45:0/2035029628 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faff40772a0 0x7faff4077700 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fafe4009ca0 tx=0x7fafe402c100 comp rx=0 tx=0).stop 2024-03-28T17:48:39.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.785+0000 7faffba00640 1 -- 172.21.15.45:0/2035029628 shutdown_connections 2024-03-28T17:48:39.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.785+0000 7faffba00640 1 --2- 172.21.15.45:0/2035029628 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faff4077c40 0x7faff40784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:39.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.785+0000 7faffba00640 1 --2- 172.21.15.45:0/2035029628 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faff40772a0 0x7faff4077700 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:39.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.785+0000 7faffba00640 1 --2- 172.21.15.45:0/2035029628 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faff4076060 0x7faff4076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:39.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.785+0000 7faffba00640 1 -- 172.21.15.45:0/2035029628 >> 172.21.15.45:0/2035029628 conn(0x7faff40fee60 msgr2=0x7faff41012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:39.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.785+0000 7faffba00640 1 -- 172.21.15.45:0/2035029628 shutdown_connections 2024-03-28T17:48:39.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.785+0000 7faffba00640 1 -- 172.21.15.45:0/2035029628 wait complete. 2024-03-28T17:48:39.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.786+0000 7faffba00640 1 Processor -- start 2024-03-28T17:48:39.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.786+0000 7faffba00640 1 -- start start 2024-03-28T17:48:39.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.787+0000 7faffba00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faff4076060 0x7faff419df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:39.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.787+0000 7faffba00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faff40772a0 0x7faff419e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:39.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.787+0000 7faffba00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faff4077c40 0x7faff41a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:39.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.787+0000 7faffba00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff4113940 con 0x7faff40772a0 2024-03-28T17:48:39.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.787+0000 7faffba00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff41137c0 con 0x7faff4077c40 2024-03-28T17:48:39.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.787+0000 7faffba00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff4113ac0 con 0x7faff4076060 2024-03-28T17:48:39.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.787+0000 7faff9000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faff4076060 0x7faff419df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:39.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.787+0000 7faff9000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faff4076060 0x7faff419df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:49566/0 (socket says 172.21.15.45:49566) 2024-03-28T17:48:39.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.787+0000 7faff9000640 1 -- 172.21.15.45:0/446132609 learned_addr learned my addr 172.21.15.45:0/446132609 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:39.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.787+0000 7faff9a00640 1 --2- 172.21.15.45:0/446132609 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faff4077c40 0x7faff41a54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:39.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.787+0000 7faff3e00640 1 --2- 172.21.15.45:0/446132609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faff40772a0 0x7faff419e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:39.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.787+0000 7faff9000640 1 -- 172.21.15.45:0/446132609 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faff4077c40 msgr2=0x7faff41a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:39.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.787+0000 7faff9000640 1 --2- 172.21.15.45:0/446132609 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faff4077c40 0x7faff41a54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:39.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.787+0000 7faff9000640 1 -- 172.21.15.45:0/446132609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faff40772a0 msgr2=0x7faff419e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:39.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.787+0000 7faff9000640 1 --2- 172.21.15.45:0/446132609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faff40772a0 0x7faff419e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:39.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.787+0000 7faff9000640 1 -- 172.21.15.45:0/446132609 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff41a5b30 con 0x7faff4076060 2024-03-28T17:48:39.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.787+0000 7faff9a00640 1 --2- 172.21.15.45:0/446132609 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faff4077c40 0x7faff41a54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:39.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.787+0000 7faff3e00640 1 --2- 172.21.15.45:0/446132609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faff40772a0 0x7faff419e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:48:39.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.788+0000 7faff9000640 1 --2- 172.21.15.45:0/446132609 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faff4076060 0x7faff419df60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fafdc00e9f0 tx=0x7fafdc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:39.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.788+0000 7faff1600640 1 -- 172.21.15.45:0/446132609 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafdc00cd80 con 0x7faff4076060 2024-03-28T17:48:39.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.788+0000 7faff1600640 1 -- 172.21.15.45:0/446132609 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafdc010910 con 0x7faff4076060 2024-03-28T17:48:39.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.788+0000 7faff1600640 1 -- 172.21.15.45:0/446132609 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafdc018990 con 0x7faff4076060 2024-03-28T17:48:39.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.788+0000 7faffba00640 1 -- 172.21.15.45:0/446132609 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff41a5e20 con 0x7faff4076060 2024-03-28T17:48:39.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.788+0000 7faffba00640 1 -- 172.21.15.45:0/446132609 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faff41a6290 con 0x7faff4076060 2024-03-28T17:48:39.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.789+0000 7faffba00640 1 -- 172.21.15.45:0/446132609 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faff4110260 con 0x7faff4076060 2024-03-28T17:48:39.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.791+0000 7faff1600640 1 -- 172.21.15.45:0/446132609 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fafdc010ac0 con 0x7faff4076060 2024-03-28T17:48:39.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.792+0000 7faff1600640 1 --2- 172.21.15.45:0/446132609 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fafd80a2920 0x7fafd80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:39.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.792+0000 7faff1600640 1 -- 172.21.15.45:0/446132609 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fafdc0cee10 con 0x7faff4076060 2024-03-28T17:48:39.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.793+0000 7faff3e00640 1 --2- 172.21.15.45:0/446132609 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fafd80a2920 0x7fafd80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:39.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.793+0000 7faff3e00640 1 --2- 172.21.15.45:0/446132609 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fafd80a2920 0x7fafd80a4de0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fafe402c710 tx=0x7fafe4032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:39.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:39.796+0000 7faff1600640 1 -- 172.21.15.45:0/446132609 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fafdc0d2030 con 0x7faff4076060 2024-03-28T17:48:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:39 smithi071 ceph-mon[36597]: pgmap v392: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:39 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/647048792' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-28T17:48:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:39 smithi071 ceph-mon[36597]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-28T17:48:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:39 smithi151 ceph-mon[36481]: pgmap v392: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:39 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/647048792' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-28T17:48:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:39 smithi151 ceph-mon[36481]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-28T17:48:39.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:39 smithi045 ceph-mon[31773]: pgmap v392: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:48:39.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:39 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/647048792' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-28T17:48:39.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:39 smithi045 ceph-mon[31773]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-28T17:48:40.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.029+0000 7faffba00640 1 -- 172.21.15.45:0/446132609 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) v1 -- 0x7faff4061b80 con 0x7fafd80a2920 2024-03-28T17:48:40.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.051+0000 7faff1600640 1 -- 172.21.15.45:0/446132609 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3 (secure 0 0 0) 0x7faff4061b80 con 0x7fafd80a2920 2024-03-28T17:48:40.052 INFO:teuthology.orchestra.run.smithi045.stdout:{} 2024-03-28T17:48:40.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.054+0000 7faffba00640 1 -- 172.21.15.45:0/446132609 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fafd80a2920 msgr2=0x7fafd80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:40.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.054+0000 7faffba00640 1 --2- 172.21.15.45:0/446132609 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fafd80a2920 0x7fafd80a4de0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fafe402c710 tx=0x7fafe4032590 comp rx=0 tx=0).stop 2024-03-28T17:48:40.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.054+0000 7faffba00640 1 -- 172.21.15.45:0/446132609 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faff4076060 msgr2=0x7faff419df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:40.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.054+0000 7faffba00640 1 --2- 172.21.15.45:0/446132609 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faff4076060 0x7faff419df60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fafdc00e9f0 tx=0x7fafdc00eeb0 comp rx=0 tx=0).stop 2024-03-28T17:48:40.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.054+0000 7faffba00640 1 -- 172.21.15.45:0/446132609 shutdown_connections 2024-03-28T17:48:40.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.054+0000 7faffba00640 1 --2- 172.21.15.45:0/446132609 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fafd80a2920 0x7fafd80a4de0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:40.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.054+0000 7faffba00640 1 --2- 172.21.15.45:0/446132609 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faff4077c40 0x7faff41a54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:40.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.054+0000 7faffba00640 1 --2- 172.21.15.45:0/446132609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faff40772a0 0x7faff419e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:40.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.054+0000 7faffba00640 1 --2- 172.21.15.45:0/446132609 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faff4076060 0x7faff419df60 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:40.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.054+0000 7faffba00640 1 -- 172.21.15.45:0/446132609 >> 172.21.15.45:0/446132609 conn(0x7faff40fee60 msgr2=0x7faff4100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:40.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.054+0000 7faffba00640 1 -- 172.21.15.45:0/446132609 shutdown_connections 2024-03-28T17:48:40.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.054+0000 7faffba00640 1 -- 172.21.15.45:0/446132609 wait complete. 2024-03-28T17:48:40.074 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-28T17:48:40.090 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-28T17:48:40.124 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T17:48:40.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.333+0000 7f3afb200640 1 -- 172.21.15.45:0/852148557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3afc1048e0 msgr2=0x7f3afc104d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:40.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.333+0000 7f3afb200640 1 --2- 172.21.15.45:0/852148557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3afc1048e0 0x7f3afc104d60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4009900 tx=0x7f3ae402bc40 comp rx=0 tx=0).stop 2024-03-28T17:48:40.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.333+0000 7f3afb200640 1 -- 172.21.15.45:0/852148557 shutdown_connections 2024-03-28T17:48:40.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.333+0000 7f3afb200640 1 --2- 172.21.15.45:0/852148557 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3afc1052a0 0x7f3afc10bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:40.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.333+0000 7f3afb200640 1 --2- 172.21.15.45:0/852148557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3afc1048e0 0x7f3afc104d60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:40.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.333+0000 7f3afb200640 1 --2- 172.21.15.45:0/852148557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3afc1036f0 0x7f3afc103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:40.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.333+0000 7f3afb200640 1 -- 172.21.15.45:0/852148557 >> 172.21.15.45:0/852148557 conn(0x7f3afc0fec80 msgr2=0x7f3afc1010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:40.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.333+0000 7f3afb200640 1 -- 172.21.15.45:0/852148557 shutdown_connections 2024-03-28T17:48:40.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.333+0000 7f3afb200640 1 -- 172.21.15.45:0/852148557 wait complete. 2024-03-28T17:48:40.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.334+0000 7f3afb200640 1 Processor -- start 2024-03-28T17:48:40.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.335+0000 7f3afb200640 1 -- start start 2024-03-28T17:48:40.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.335+0000 7f3afb200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3afc1036f0 0x7f3afc1999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:40.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.335+0000 7f3afb200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3afc1048e0 0x7f3afc199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:40.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.335+0000 7f3afb200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3afc1052a0 0x7f3afc1a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:40.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.335+0000 7f3afb200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc10f3a0 con 0x7f3afc1052a0 2024-03-28T17:48:40.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.335+0000 7f3afb200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc10f220 con 0x7f3afc1036f0 2024-03-28T17:48:40.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.335+0000 7f3afb200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc10f520 con 0x7f3afc1048e0 2024-03-28T17:48:40.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.335+0000 7f3af9e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3afc1036f0 0x7f3afc1999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:40.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.335+0000 7f3af9e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3afc1036f0 0x7f3afc1999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:54830/0 (socket says 172.21.15.45:54830) 2024-03-28T17:48:40.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.335+0000 7f3af9e00640 1 -- 172.21.15.45:0/1413815400 learned_addr learned my addr 172.21.15.45:0/1413815400 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:40.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.335+0000 7f3af1400640 1 --2- 172.21.15.45:0/1413815400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3afc1048e0 0x7f3afc199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:40.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.336+0000 7f3afa800640 1 --2- 172.21.15.45:0/1413815400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3afc1052a0 0x7f3afc1a0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:40.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.336+0000 7f3af9e00640 1 -- 172.21.15.45:0/1413815400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3afc1048e0 msgr2=0x7f3afc199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:40.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.336+0000 7f3af9e00640 1 --2- 172.21.15.45:0/1413815400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3afc1048e0 0x7f3afc199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:40.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.336+0000 7f3af9e00640 1 -- 172.21.15.45:0/1413815400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3afc1052a0 msgr2=0x7f3afc1a0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:40.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.336+0000 7f3af9e00640 1 --2- 172.21.15.45:0/1413815400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3afc1052a0 0x7f3afc1a0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:40.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.336+0000 7f3af9e00640 1 -- 172.21.15.45:0/1413815400 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3afc1a15e0 con 0x7f3afc1036f0 2024-03-28T17:48:40.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.336+0000 7f3afa800640 1 --2- 172.21.15.45:0/1413815400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3afc1052a0 0x7f3afc1a0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T17:48:40.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.336+0000 7f3af9e00640 1 --2- 172.21.15.45:0/1413815400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3afc1036f0 0x7f3afc1999c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3aec00ebc0 tx=0x7f3aec00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:40.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.336+0000 7f3af3400640 1 -- 172.21.15.45:0/1413815400 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aec003b70 con 0x7f3afc1036f0 2024-03-28T17:48:40.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.336+0000 7f3af3400640 1 -- 172.21.15.45:0/1413815400 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3aec003d10 con 0x7f3afc1036f0 2024-03-28T17:48:40.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.336+0000 7f3af3400640 1 -- 172.21.15.45:0/1413815400 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aec019980 con 0x7f3afc1036f0 2024-03-28T17:48:40.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.336+0000 7f3afb200640 1 -- 172.21.15.45:0/1413815400 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3afc1a18d0 con 0x7f3afc1036f0 2024-03-28T17:48:40.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.337+0000 7f3afb200640 1 -- 172.21.15.45:0/1413815400 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3afc1a1da0 con 0x7f3afc1036f0 2024-03-28T17:48:40.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.338+0000 7f3afb200640 1 -- 172.21.15.45:0/1413815400 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3afc10aa10 con 0x7f3afc1036f0 2024-03-28T17:48:40.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.340+0000 7f3af3400640 1 -- 172.21.15.45:0/1413815400 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3aec019b20 con 0x7f3afc1036f0 2024-03-28T17:48:40.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.341+0000 7f3af3400640 1 --2- 172.21.15.45:0/1413815400 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3ae00a2880 0x7f3ae00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:40.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.341+0000 7f3af3400640 1 -- 172.21.15.45:0/1413815400 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3aec0cf2c0 con 0x7f3afc1036f0 2024-03-28T17:48:40.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.341+0000 7f3af1400640 1 --2- 172.21.15.45:0/1413815400 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3ae00a2880 0x7f3ae00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:40.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.342+0000 7f3af1400640 1 --2- 172.21.15.45:0/1413815400 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3ae00a2880 0x7f3ae00a4d40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3ae402c250 tx=0x7f3ae402cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:40.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.345+0000 7f3af3400640 1 -- 172.21.15.45:0/1413815400 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3aec09c310 con 0x7f3afc1036f0 2024-03-28T17:48:40.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:40.039+0000 7ffa32c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:48:40.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:40.039+0000 7ffa32c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:48:40.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:40.039+0000 7ffa32c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:48:40.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:40.039+0000 7ffa32c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:48:40.417 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:40.039+0000 7ffa32c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:48:40.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.554+0000 7f3afb200640 1 -- 172.21.15.45:0/1413815400 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3afc062370 con 0x7f3afc1036f0 2024-03-28T17:48:40.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.558+0000 7f3af3400640 1 -- 172.21.15.45:0/1413815400 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22583 (secure 0 0 0) 0x7f3aec09bc10 con 0x7f3afc1036f0 2024-03-28T17:48:40.558 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:48:40.558 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"9f4888ec-ed29-11ee-95d0-87774f69a715","created":"2024-03-28T17:36:51.623185+0000","modified":"2024-03-28T17:47:09.455843+0000","last_up_change":"2024-03-28T17:45:14.043432+0000","last_in_change":"2024-03-28T17:44:53.654489+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T17:40:54.208375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T17:46:10.564519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T17:46:11.734367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"74c56bc2-4169-4919-afd4-2ebd236f23e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6803","nonce":3795754023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6805","nonce":3795754023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6809","nonce":3795754023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6807","nonce":3795754023}]},"public_addr":"172.21.15.45:6803/3795754023","cluster_addr":"172.21.15.45:6805/3795754023","heartbeat_back_addr":"172.21.15.45:6809/3795754023","heartbeat_front_addr":"172.21.15.45:6807/3795754023","state":["exists","up"]},{"osd":1,"uuid":"78dcc21b-75b2-453d-b690-17b55b8ed1a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6801","nonce":2014529906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6803","nonce":2014529906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6807","nonce":2014529906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6805","nonce":2014529906}]},"public_addr":"172.21.15.71:6801/2014529906","cluster_addr":"172.21.15.71:6803/2014529906","heartbeat_back_addr":"172.21.15.71:6807/2014529906","heartbeat_front_addr":"172.21.15.71:6805/2014529906","state":["exists","up"]},{"osd":2,"uuid":"ca9f3f36-b71e-4962-a94f-cedf2f12eb7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6801","nonce":1850933775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6803","nonce":1850933775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6807","nonce":1850933775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6805","nonce":1850933775}]},"public_addr":"172.21.15.151:6801/1850933775","cluster_addr":"172.21.15.151:6803/1850933775","heartbeat_back_addr":"172.21.15.151:6807/1850933775","heartbeat_front_addr":"172.21.15.151:6805/1850933775","state":["exists","up"]},{"osd":3,"uuid":"d3f21f01-2973-413e-8daa-9974b8a44af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6811","nonce":3021569975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6813","nonce":3021569975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6817","nonce":3021569975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6815","nonce":3021569975}]},"public_addr":"172.21.15.45:6811/3021569975","cluster_addr":"172.21.15.45:6813/3021569975","heartbeat_back_addr":"172.21.15.45:6817/3021569975","heartbeat_front_addr":"172.21.15.45:6815/3021569975","state":["exists","up"]},{"osd":4,"uuid":"4a336f42-0605-492b-ad88-56cb02acdcf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6809","nonce":2069578333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6811","nonce":2069578333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6815","nonce":2069578333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6813","nonce":2069578333}]},"public_addr":"172.21.15.71:6809/2069578333","cluster_addr":"172.21.15.71:6811/2069578333","heartbeat_back_addr":"172.21.15.71:6815/2069578333","heartbeat_front_addr":"172.21.15.71:6813/2069578333","state":["exists","up"]},{"osd":5,"uuid":"c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6809","nonce":2629217227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6811","nonce":2629217227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6815","nonce":2629217227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6813","nonce":2629217227}]},"public_addr":"172.21.15.151:6809/2629217227","cluster_addr":"172.21.15.151:6811/2629217227","heartbeat_back_addr":"172.21.15.151:6815/2629217227","heartbeat_front_addr":"172.21.15.151:6813/2629217227","state":["exists","up"]},{"osd":6,"uuid":"58a6c5bd-a8b7-4892-b346-bcd7994bcb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6819","nonce":2077907039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6821","nonce":2077907039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6825","nonce":2077907039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6823","nonce":2077907039}]},"public_addr":"172.21.15.45:6819/2077907039","cluster_addr":"172.21.15.45:6821/2077907039","heartbeat_back_addr":"172.21.15.45:6825/2077907039","heartbeat_front_addr":"172.21.15.45:6823/2077907039","state":["exists","up"]},{"osd":7,"uuid":"f7a45d52-807b-4407-bb04-787b2a52a0ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6817","nonce":2000211251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6819","nonce":2000211251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6823","nonce":2000211251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6821","nonce":2000211251}]},"public_addr":"172.21.15.71:6817/2000211251","cluster_addr":"172.21.15.71:6819/2000211251","heartbeat_back_addr":"172.21.15.71:6823/2000211251","heartbeat_front_addr":"172.21.15.71:6821/2000211251","state":["exists","up"]},{"osd":8,"uuid":"88abcb0f-f189-49be-8315-c700c77369f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6817","nonce":3665460137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6819","nonce":3665460137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6823","nonce":3665460137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6821","nonce":3665460137}]},"public_addr":"172.21.15.151:6817/3665460137","cluster_addr":"172.21.15.151:6819/3665460137","heartbeat_back_addr":"172.21.15.151:6823/3665460137","heartbeat_front_addr":"172.21.15.151:6821/3665460137","state":["exists","up"]},{"osd":9,"uuid":"ed3fc096-961e-4392-abc2-c5aa4d050117","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6827","nonce":2122322684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6829","nonce":2122322684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6833","nonce":2122322684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6831","nonce":2122322684}]},"public_addr":"172.21.15.45:6827/2122322684","cluster_addr":"172.21.15.45:6829/2122322684","heartbeat_back_addr":"172.21.15.45:6833/2122322684","heartbeat_front_addr":"172.21.15.45:6831/2122322684","state":["exists","up"]},{"osd":10,"uuid":"ba94762b-bc5c-4c97-9cc1-17e5651c14b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6825","nonce":1099494037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6827","nonce":1099494037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6831","nonce":1099494037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6829","nonce":1099494037}]},"public_addr":"172.21.15.71:6825/1099494037","cluster_addr":"172.21.15.71:6827/1099494037","heartbeat_back_addr":"172.21.15.71:6831/1099494037","heartbeat_front_addr":"172.21.15.71:6829/1099494037","state":["exists","up"]},{"osd":11,"uuid":"1d178378-b9f5-40f6-99ef-3d3d445da640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6825","nonce":2941055364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6827","nonce":2941055364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6831","nonce":2941055364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6829","nonce":2941055364}]},"public_addr":"172.21.15.151:6825/2941055364","cluster_addr":"172.21.15.151:6827/2941055364","heartbeat_back_addr":"172.21.15.151:6831/2941055364","heartbeat_front_addr":"172.21.15.151:6829/2941055364","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:02.269586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:27.819386+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:51.552277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:25.052674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:51.873529+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:17.127867+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:51.588249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:18.381450+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:44.140534+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:17.634362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:45.469340+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:45:12.003433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3533611957":"2024-03-29T17:38:07.505611+0000","172.21.15.45:0/3625922430":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6801/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/482610468":"2024-03-29T17:37:32.424229+0000","172.21.15.45:6801/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3649804714":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/1198364079":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3211303958":"2024-03-29T17:38:07.505611+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T17:48:40.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.560+0000 7f3afb200640 1 -- 172.21.15.45:0/1413815400 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3ae00a2880 msgr2=0x7f3ae00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:40.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.560+0000 7f3afb200640 1 --2- 172.21.15.45:0/1413815400 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3ae00a2880 0x7f3ae00a4d40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3ae402c250 tx=0x7f3ae402cf90 comp rx=0 tx=0).stop 2024-03-28T17:48:40.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.560+0000 7f3afb200640 1 -- 172.21.15.45:0/1413815400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3afc1036f0 msgr2=0x7f3afc1999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:40.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.560+0000 7f3afb200640 1 --2- 172.21.15.45:0/1413815400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3afc1036f0 0x7f3afc1999c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3aec00ebc0 tx=0x7f3aec00c6a0 comp rx=0 tx=0).stop 2024-03-28T17:48:40.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.560+0000 7f3afb200640 1 -- 172.21.15.45:0/1413815400 shutdown_connections 2024-03-28T17:48:40.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.560+0000 7f3afb200640 1 --2- 172.21.15.45:0/1413815400 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f3ae00a2880 0x7f3ae00a4d40 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:40.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.560+0000 7f3afb200640 1 --2- 172.21.15.45:0/1413815400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3afc1052a0 0x7f3afc1a0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:40.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.560+0000 7f3afb200640 1 --2- 172.21.15.45:0/1413815400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3afc1048e0 0x7f3afc199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:40.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.560+0000 7f3afb200640 1 --2- 172.21.15.45:0/1413815400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3afc1036f0 0x7f3afc1999c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:40.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.560+0000 7f3afb200640 1 -- 172.21.15.45:0/1413815400 >> 172.21.15.45:0/1413815400 conn(0x7f3afc0fec80 msgr2=0x7f3afc107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:40.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.560+0000 7f3afb200640 1 -- 172.21.15.45:0/1413815400 shutdown_connections 2024-03-28T17:48:40.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.560+0000 7f3afb200640 1 -- 172.21.15.45:0/1413815400 wait complete. 2024-03-28T17:48:40.580 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T17:40:54.208375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T17:46:10.564519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T17:46:11.734367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T17:48:40.581 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T17:48:40.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.798+0000 7f2b24000640 1 -- 172.21.15.45:0/2324569188 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b1c105490 msgr2=0x7f2b1c10bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:40.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.798+0000 7f2b24000640 1 --2- 172.21.15.45:0/2324569188 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b1c105490 0x7f2b1c10bd20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2b14009ca0 tx=0x7f2b1402c100 comp rx=0 tx=0).stop 2024-03-28T17:48:40.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.798+0000 7f2b24000640 1 -- 172.21.15.45:0/2324569188 shutdown_connections 2024-03-28T17:48:40.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.798+0000 7f2b24000640 1 --2- 172.21.15.45:0/2324569188 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b1c105490 0x7f2b1c10bd20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:40.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.798+0000 7f2b24000640 1 --2- 172.21.15.45:0/2324569188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b1c104ad0 0x7f2b1c104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:40.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.798+0000 7f2b24000640 1 --2- 172.21.15.45:0/2324569188 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b1c1038e0 0x7f2b1c103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:40.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.798+0000 7f2b24000640 1 -- 172.21.15.45:0/2324569188 >> 172.21.15.45:0/2324569188 conn(0x7f2b1c0fee50 msgr2=0x7f2b1c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:40.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.798+0000 7f2b24000640 1 -- 172.21.15.45:0/2324569188 shutdown_connections 2024-03-28T17:48:40.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.798+0000 7f2b24000640 1 -- 172.21.15.45:0/2324569188 wait complete. 2024-03-28T17:48:40.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.799+0000 7f2b24000640 1 Processor -- start 2024-03-28T17:48:40.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.799+0000 7f2b24000640 1 -- start start 2024-03-28T17:48:40.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.799+0000 7f2b24000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b1c1038e0 0x7f2b1c199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:40.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.799+0000 7f2b24000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b1c104ad0 0x7f2b1c19a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:40.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.800+0000 7f2b24000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b1c105490 0x7f2b1c1a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:40.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.800+0000 7f2b24000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c10f560 con 0x7f2b1c104ad0 2024-03-28T17:48:40.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.800+0000 7f2b24000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c10f3e0 con 0x7f2b1c105490 2024-03-28T17:48:40.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.800+0000 7f2b24000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c10f6e0 con 0x7f2b1c1038e0 2024-03-28T17:48:40.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.800+0000 7f2b21600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b1c1038e0 0x7f2b1c199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:40.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.800+0000 7f2b22000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b1c105490 0x7f2b1c1a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:40.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.800+0000 7f2b21600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b1c1038e0 0x7f2b1c199b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:49622/0 (socket says 172.21.15.45:49622) 2024-03-28T17:48:40.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.800+0000 7f2b21600640 1 -- 172.21.15.45:0/2140119502 learned_addr learned my addr 172.21.15.45:0/2140119502 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:40.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.800+0000 7f2b20c00640 1 --2- 172.21.15.45:0/2140119502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b1c104ad0 0x7f2b1c19a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:40.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.800+0000 7f2b22000640 1 -- 172.21.15.45:0/2140119502 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b1c1038e0 msgr2=0x7f2b1c199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:40.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.800+0000 7f2b22000640 1 --2- 172.21.15.45:0/2140119502 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b1c1038e0 0x7f2b1c199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:40.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.800+0000 7f2b22000640 1 -- 172.21.15.45:0/2140119502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b1c104ad0 msgr2=0x7f2b1c19a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:40.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.800+0000 7f2b22000640 1 --2- 172.21.15.45:0/2140119502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b1c104ad0 0x7f2b1c19a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:40.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.800+0000 7f2b22000640 1 -- 172.21.15.45:0/2140119502 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1c1a1790 con 0x7f2b1c105490 2024-03-28T17:48:40.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.801+0000 7f2b22000640 1 --2- 172.21.15.45:0/2140119502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b1c105490 0x7f2b1c1a1120 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2b14037040 tx=0x7f2b14035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:40.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.801+0000 7f2b1a000640 1 -- 172.21.15.45:0/2140119502 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b14039040 con 0x7f2b1c105490 2024-03-28T17:48:40.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.801+0000 7f2b1a000640 1 -- 172.21.15.45:0/2140119502 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2b14044070 con 0x7f2b1c105490 2024-03-28T17:48:40.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.801+0000 7f2b24000640 1 -- 172.21.15.45:0/2140119502 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b1c1a1a20 con 0x7f2b1c105490 2024-03-28T17:48:40.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.801+0000 7f2b1a000640 1 -- 172.21.15.45:0/2140119502 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b14004350 con 0x7f2b1c105490 2024-03-28T17:48:40.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.801+0000 7f2b24000640 1 -- 172.21.15.45:0/2140119502 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b1c1a1ee0 con 0x7f2b1c105490 2024-03-28T17:48:40.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.802+0000 7f2b24000640 1 -- 172.21.15.45:0/2140119502 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b1c10ac00 con 0x7f2b1c105490 2024-03-28T17:48:40.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.804+0000 7f2b1a000640 1 -- 172.21.15.45:0/2140119502 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2b140044f0 con 0x7f2b1c105490 2024-03-28T17:48:40.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.805+0000 7f2b1a000640 1 --2- 172.21.15.45:0/2140119502 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2b040a2800 0x7f2b040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:40.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.805+0000 7f2b1a000640 1 -- 172.21.15.45:0/2140119502 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f2b140f0fc0 con 0x7f2b1c105490 2024-03-28T17:48:40.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.806+0000 7f2b21600640 1 --2- 172.21.15.45:0/2140119502 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2b040a2800 0x7f2b040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:40.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.807+0000 7f2b21600640 1 --2- 172.21.15.45:0/2140119502 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2b040a2800 0x7f2b040a4cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2b100058c0 tx=0x7f2b10005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:40.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:40.809+0000 7f2b1a000640 1 -- 172.21.15.45:0/2140119502 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2b140be010 con 0x7f2b1c105490 2024-03-28T17:48:40.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:40 smithi071 ceph-mon[36597]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-28T17:48:40.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:40 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:48:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:40 smithi151 ceph-mon[36481]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-28T17:48:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:40 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:48:40.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:40 smithi045 ceph-mon[31773]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-28T17:48:40.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:40 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:48:41.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.017+0000 7f2b24000640 1 -- 172.21.15.45:0/2140119502 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2b1c0622a0 con 0x7f2b1c105490 2024-03-28T17:48:41.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.018+0000 7f2b1a000640 1 -- 172.21.15.45:0/2140119502 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2b140098b0 con 0x7f2b1c105490 2024-03-28T17:48:41.018 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-28T17:48:41.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.020+0000 7f2b24000640 1 -- 172.21.15.45:0/2140119502 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2b040a2800 msgr2=0x7f2b040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:41.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.020+0000 7f2b24000640 1 --2- 172.21.15.45:0/2140119502 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2b040a2800 0x7f2b040a4cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2b100058c0 tx=0x7f2b10005f90 comp rx=0 tx=0).stop 2024-03-28T17:48:41.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.020+0000 7f2b24000640 1 -- 172.21.15.45:0/2140119502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b1c105490 msgr2=0x7f2b1c1a1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:41.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.020+0000 7f2b24000640 1 --2- 172.21.15.45:0/2140119502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b1c105490 0x7f2b1c1a1120 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2b14037040 tx=0x7f2b14035d10 comp rx=0 tx=0).stop 2024-03-28T17:48:41.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.021+0000 7f2b24000640 1 -- 172.21.15.45:0/2140119502 shutdown_connections 2024-03-28T17:48:41.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.021+0000 7f2b24000640 1 --2- 172.21.15.45:0/2140119502 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f2b040a2800 0x7f2b040a4cc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.021+0000 7f2b24000640 1 --2- 172.21.15.45:0/2140119502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b1c105490 0x7f2b1c1a1120 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.021+0000 7f2b24000640 1 --2- 172.21.15.45:0/2140119502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b1c104ad0 0x7f2b1c19a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.021+0000 7f2b24000640 1 --2- 172.21.15.45:0/2140119502 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b1c1038e0 0x7f2b1c199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.021+0000 7f2b24000640 1 -- 172.21.15.45:0/2140119502 >> 172.21.15.45:0/2140119502 conn(0x7f2b1c0fee50 msgr2=0x7f2b1c107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:41.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.021+0000 7f2b24000640 1 -- 172.21.15.45:0/2140119502 shutdown_connections 2024-03-28T17:48:41.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.021+0000 7f2b24000640 1 -- 172.21.15.45:0/2140119502 wait complete. 2024-03-28T17:48:41.040 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T17:48:41.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.265+0000 7f55f2000640 1 -- 172.21.15.45:0/1328460535 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55ec1015a0 msgr2=0x7f55ec1019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:41.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.265+0000 7f55f2000640 1 --2- 172.21.15.45:0/1328460535 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55ec1015a0 0x7f55ec1019c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f55dc009ca0 tx=0x7f55dc02c0f0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.265+0000 7f55f2000640 1 -- 172.21.15.45:0/1328460535 shutdown_connections 2024-03-28T17:48:41.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.265+0000 7f55f2000640 1 --2- 172.21.15.45:0/1328460535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55ec103150 0x7f55ec1099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.265+0000 7f55f2000640 1 --2- 172.21.15.45:0/1328460535 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55ec102790 0x7f55ec102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.265+0000 7f55f2000640 1 --2- 172.21.15.45:0/1328460535 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55ec1015a0 0x7f55ec1019c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.265+0000 7f55f2000640 1 -- 172.21.15.45:0/1328460535 >> 172.21.15.45:0/1328460535 conn(0x7f55ec0fcb10 msgr2=0x7f55ec0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:41.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.265+0000 7f55f2000640 1 -- 172.21.15.45:0/1328460535 shutdown_connections 2024-03-28T17:48:41.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.265+0000 7f55f2000640 1 -- 172.21.15.45:0/1328460535 wait complete. 2024-03-28T17:48:41.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.267+0000 7f55f2000640 1 Processor -- start 2024-03-28T17:48:41.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.267+0000 7f55f2000640 1 -- start start 2024-03-28T17:48:41.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.267+0000 7f55f2000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55ec1015a0 0x7f55ec197890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:41.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.267+0000 7f55f2000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55ec102790 0x7f55ec197dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:41.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.267+0000 7f55f2000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55ec103150 0x7f55ec19ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:41.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.268+0000 7f55f2000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ec10d230 con 0x7f55ec1015a0 2024-03-28T17:48:41.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.268+0000 7f55f2000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ec10d0b0 con 0x7f55ec102790 2024-03-28T17:48:41.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.268+0000 7f55f2000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ec10d3b0 con 0x7f55ec103150 2024-03-28T17:48:41.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.268+0000 7f55f0c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55ec1015a0 0x7f55ec197890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:41.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.268+0000 7f55f0c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55ec1015a0 0x7f55ec197890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44216/0 (socket says 172.21.15.45:44216) 2024-03-28T17:48:41.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.268+0000 7f55ebe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55ec102790 0x7f55ec197dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:41.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.268+0000 7f55f0c00640 1 -- 172.21.15.45:0/3391319388 learned_addr learned my addr 172.21.15.45:0/3391319388 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:41.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.268+0000 7f55f1600640 1 --2- 172.21.15.45:0/3391319388 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55ec103150 0x7f55ec19ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:41.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.268+0000 7f55ebe00640 1 -- 172.21.15.45:0/3391319388 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55ec103150 msgr2=0x7f55ec19ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:41.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.268+0000 7f55ebe00640 1 --2- 172.21.15.45:0/3391319388 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55ec103150 0x7f55ec19ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.268+0000 7f55ebe00640 1 -- 172.21.15.45:0/3391319388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55ec1015a0 msgr2=0x7f55ec197890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:41.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.268+0000 7f55ebe00640 1 --2- 172.21.15.45:0/3391319388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55ec1015a0 0x7f55ec197890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.268+0000 7f55ebe00640 1 -- 172.21.15.45:0/3391319388 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ec19f4b0 con 0x7f55ec102790 2024-03-28T17:48:41.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.268+0000 7f55f0c00640 1 --2- 172.21.15.45:0/3391319388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55ec1015a0 0x7f55ec197890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:41.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.269+0000 7f55ebe00640 1 --2- 172.21.15.45:0/3391319388 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55ec102790 0x7f55ec197dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f55d400e9f0 tx=0x7f55d400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:41.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.269+0000 7f55e9600640 1 -- 172.21.15.45:0/3391319388 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d400cd80 con 0x7f55ec102790 2024-03-28T17:48:41.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.269+0000 7f55e9600640 1 -- 172.21.15.45:0/3391319388 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f55d40035f0 con 0x7f55ec102790 2024-03-28T17:48:41.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.269+0000 7f55f2000640 1 -- 172.21.15.45:0/3391319388 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55ec19f7a0 con 0x7f55ec102790 2024-03-28T17:48:41.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.269+0000 7f55e9600640 1 -- 172.21.15.45:0/3391319388 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d4010670 con 0x7f55ec102790 2024-03-28T17:48:41.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.269+0000 7f55f2000640 1 -- 172.21.15.45:0/3391319388 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55ec19fd40 con 0x7f55ec102790 2024-03-28T17:48:41.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.270+0000 7f55f2000640 1 -- 172.21.15.45:0/3391319388 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55ec1088c0 con 0x7f55ec102790 2024-03-28T17:48:41.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.272+0000 7f55e9600640 1 -- 172.21.15.45:0/3391319388 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f55d40037a0 con 0x7f55ec102790 2024-03-28T17:48:41.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.273+0000 7f55e9600640 1 --2- 172.21.15.45:0/3391319388 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f55d00a2880 0x7f55d00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:41.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.273+0000 7f55e9600640 1 -- 172.21.15.45:0/3391319388 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f55d40cece0 con 0x7f55ec102790 2024-03-28T17:48:41.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.274+0000 7f55f0c00640 1 --2- 172.21.15.45:0/3391319388 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f55d00a2880 0x7f55d00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:41.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.274+0000 7f55f0c00640 1 --2- 172.21.15.45:0/3391319388 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f55d00a2880 0x7f55d00a4d40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f55dc005e30 tx=0x7f55dc004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:41.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.282+0000 7f55e9600640 1 -- 172.21.15.45:0/3391319388 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f55d409bd30 con 0x7f55ec102790 2024-03-28T17:48:41.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.481+0000 7f55f2000640 1 -- 172.21.15.45:0/3391319388 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f55ec0622a0 con 0x7f55ec102790 2024-03-28T17:48:41.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.482+0000 7f55e9600640 1 -- 172.21.15.45:0/3391319388 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f55d409b630 con 0x7f55ec102790 2024-03-28T17:48:41.482 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-03-28T17:48:41.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.485+0000 7f55f2000640 1 -- 172.21.15.45:0/3391319388 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f55d00a2880 msgr2=0x7f55d00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:41.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.485+0000 7f55f2000640 1 --2- 172.21.15.45:0/3391319388 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f55d00a2880 0x7f55d00a4d40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f55dc005e30 tx=0x7f55dc004a40 comp rx=0 tx=0).stop 2024-03-28T17:48:41.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.485+0000 7f55f2000640 1 -- 172.21.15.45:0/3391319388 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55ec102790 msgr2=0x7f55ec197dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:41.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.485+0000 7f55f2000640 1 --2- 172.21.15.45:0/3391319388 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55ec102790 0x7f55ec197dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f55d400e9f0 tx=0x7f55d400eeb0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.485+0000 7f55f2000640 1 -- 172.21.15.45:0/3391319388 shutdown_connections 2024-03-28T17:48:41.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.485+0000 7f55f2000640 1 --2- 172.21.15.45:0/3391319388 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f55d00a2880 0x7f55d00a4d40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.485+0000 7f55f2000640 1 --2- 172.21.15.45:0/3391319388 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f55ec103150 0x7f55ec19ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.485+0000 7f55f2000640 1 --2- 172.21.15.45:0/3391319388 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55ec102790 0x7f55ec197dd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.485+0000 7f55f2000640 1 --2- 172.21.15.45:0/3391319388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55ec1015a0 0x7f55ec197890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.485+0000 7f55f2000640 1 -- 172.21.15.45:0/3391319388 >> 172.21.15.45:0/3391319388 conn(0x7f55ec0fcb10 msgr2=0x7f55ec105a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:41.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.485+0000 7f55f2000640 1 -- 172.21.15.45:0/3391319388 shutdown_connections 2024-03-28T17:48:41.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.485+0000 7f55f2000640 1 -- 172.21.15.45:0/3391319388 wait complete. 2024-03-28T17:48:41.504 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T17:48:41.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.733+0000 7f5cce600640 1 -- 172.21.15.45:0/3753178742 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cc8075660 msgr2=0x7f5cc8073ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:41.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.733+0000 7f5cce600640 1 --2- 172.21.15.45:0/3753178742 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cc8075660 0x7f5cc8073ab0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8009ca0 tx=0x7f5cb802c190 comp rx=0 tx=0).stop 2024-03-28T17:48:41.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.733+0000 7f5cce600640 1 -- 172.21.15.45:0/3753178742 shutdown_connections 2024-03-28T17:48:41.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.733+0000 7f5cce600640 1 --2- 172.21.15.45:0/3753178742 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cc8109560 0x7f5cc810b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.733+0000 7f5cce600640 1 --2- 172.21.15.45:0/3753178742 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cc8074080 0x7f5cc80744e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.733+0000 7f5cce600640 1 --2- 172.21.15.45:0/3753178742 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cc8075660 0x7f5cc8073ab0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.733+0000 7f5cce600640 1 -- 172.21.15.45:0/3753178742 >> 172.21.15.45:0/3753178742 conn(0x7f5cc80fcc40 msgr2=0x7f5cc80ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:41.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.733+0000 7f5cce600640 1 -- 172.21.15.45:0/3753178742 shutdown_connections 2024-03-28T17:48:41.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.733+0000 7f5cce600640 1 -- 172.21.15.45:0/3753178742 wait complete. 2024-03-28T17:48:41.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.734+0000 7f5cce600640 1 Processor -- start 2024-03-28T17:48:41.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.735+0000 7f5cce600640 1 -- start start 2024-03-28T17:48:41.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.735+0000 7f5cce600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cc8074080 0x7f5cc8199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:41.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.735+0000 7f5cce600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cc8075660 0x7f5cc819a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:41.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.735+0000 7f5cce600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cc8109560 0x7f5cc81a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:41.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.735+0000 7f5cce600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc810f510 con 0x7f5cc8074080 2024-03-28T17:48:41.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.735+0000 7f5cce600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc810f390 con 0x7f5cc8075660 2024-03-28T17:48:41.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.735+0000 7f5cce600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc810f690 con 0x7f5cc8109560 2024-03-28T17:48:41.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.735+0000 7f5cc7400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cc8074080 0x7f5cc8199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:41.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.735+0000 7f5cbea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cc8075660 0x7f5cc819a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:41.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.735+0000 7f5cc7e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cc8109560 0x7f5cc81a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:41.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.735+0000 7f5cc7e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cc8109560 0x7f5cc81a1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:49646/0 (socket says 172.21.15.45:49646) 2024-03-28T17:48:41.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.735+0000 7f5cbea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cc8075660 0x7f5cc819a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:54870/0 (socket says 172.21.15.45:54870) 2024-03-28T17:48:41.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.735+0000 7f5cc7e00640 1 -- 172.21.15.45:0/3301102470 learned_addr learned my addr 172.21.15.45:0/3301102470 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:41.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.736+0000 7f5cc7e00640 1 -- 172.21.15.45:0/3301102470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cc8075660 msgr2=0x7f5cc819a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:41.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.736+0000 7f5cc7e00640 1 --2- 172.21.15.45:0/3301102470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cc8075660 0x7f5cc819a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.736+0000 7f5cc7e00640 1 -- 172.21.15.45:0/3301102470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cc8074080 msgr2=0x7f5cc8199b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:41.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.736+0000 7f5cc7e00640 1 --2- 172.21.15.45:0/3301102470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cc8074080 0x7f5cc8199b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.736+0000 7f5cc7e00640 1 -- 172.21.15.45:0/3301102470 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc81a1780 con 0x7f5cc8109560 2024-03-28T17:48:41.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.736+0000 7f5cbea00640 1 --2- 172.21.15.45:0/3301102470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cc8075660 0x7f5cc819a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:41.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.736+0000 7f5cc7e00640 1 --2- 172.21.15.45:0/3301102470 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cc8109560 0x7f5cc81a1110 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5cc000c970 tx=0x7f5cc000ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:41.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.736+0000 7f5cc4c00640 1 -- 172.21.15.45:0/3301102470 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc0007bb0 con 0x7f5cc8109560 2024-03-28T17:48:41.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.736+0000 7f5cc4c00640 1 -- 172.21.15.45:0/3301102470 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5cc0007d50 con 0x7f5cc8109560 2024-03-28T17:48:41.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.736+0000 7f5cce600640 1 -- 172.21.15.45:0/3301102470 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc81a1a70 con 0x7f5cc8109560 2024-03-28T17:48:41.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.736+0000 7f5cc4c00640 1 -- 172.21.15.45:0/3301102470 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc0018800 con 0x7f5cc8109560 2024-03-28T17:48:41.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.737+0000 7f5cce600640 1 -- 172.21.15.45:0/3301102470 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cc81a1fc0 con 0x7f5cc8109560 2024-03-28T17:48:41.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.738+0000 7f5cce600640 1 -- 172.21.15.45:0/3301102470 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cc8108320 con 0x7f5cc8109560 2024-03-28T17:48:41.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.740+0000 7f5cc4c00640 1 -- 172.21.15.45:0/3301102470 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5cc000fa50 con 0x7f5cc8109560 2024-03-28T17:48:41.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.741+0000 7f5cc4c00640 1 --2- 172.21.15.45:0/3301102470 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f5cac0a2800 0x7f5cac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:41.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.741+0000 7f5cc4c00640 1 -- 172.21.15.45:0/3301102470 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5cc00cea50 con 0x7f5cc8109560 2024-03-28T17:48:41.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.741+0000 7f5cc7400640 1 --2- 172.21.15.45:0/3301102470 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f5cac0a2800 0x7f5cac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:41.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.742+0000 7f5cc7400640 1 --2- 172.21.15.45:0/3301102470 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f5cac0a2800 0x7f5cac0a4cc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5cb802c7a0 tx=0x7f5cb8005f70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:41.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.745+0000 7f5cc4c00640 1 -- 172.21.15.45:0/3301102470 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5cc0014030 con 0x7f5cc8109560 2024-03-28T17:48:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:41 smithi071 ceph-mon[36597]: pgmap v393: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T17:48:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:41 smithi071 ceph-mon[36597]: from='client.25186 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:48:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:41 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1413815400' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:48:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:41 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/2140119502' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:48:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:41 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3391319388' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T17:48:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:41 smithi151 ceph-mon[36481]: pgmap v393: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T17:48:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:41 smithi151 ceph-mon[36481]: from='client.25186 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:48:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:41 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1413815400' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:48:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:41 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/2140119502' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:48:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:41 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3391319388' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T17:48:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:41 smithi045 ceph-mon[31773]: pgmap v393: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T17:48:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:41 smithi045 ceph-mon[31773]: from='client.25186 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:48:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:41 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1413815400' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:48:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:41 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/2140119502' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:48:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:41 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3391319388' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T17:48:41.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.952+0000 7f5cce600640 1 -- 172.21.15.45:0/3301102470 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5cc80622a0 con 0x7f5cc8109560 2024-03-28T17:48:41.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.953+0000 7f5cc4c00640 1 -- 172.21.15.45:0/3301102470 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5cc009baa0 con 0x7f5cc8109560 2024-03-28T17:48:41.953 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-03-28T17:48:41.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.956+0000 7f5cce600640 1 -- 172.21.15.45:0/3301102470 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f5cac0a2800 msgr2=0x7f5cac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:41.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.956+0000 7f5cce600640 1 --2- 172.21.15.45:0/3301102470 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f5cac0a2800 0x7f5cac0a4cc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5cb802c7a0 tx=0x7f5cb8005f70 comp rx=0 tx=0).stop 2024-03-28T17:48:41.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.956+0000 7f5cce600640 1 -- 172.21.15.45:0/3301102470 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cc8109560 msgr2=0x7f5cc81a1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:41.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.956+0000 7f5cce600640 1 --2- 172.21.15.45:0/3301102470 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cc8109560 0x7f5cc81a1110 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5cc000c970 tx=0x7f5cc000ce30 comp rx=0 tx=0).stop 2024-03-28T17:48:41.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.956+0000 7f5cce600640 1 -- 172.21.15.45:0/3301102470 shutdown_connections 2024-03-28T17:48:41.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.956+0000 7f5cce600640 1 --2- 172.21.15.45:0/3301102470 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f5cac0a2800 0x7f5cac0a4cc0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.956+0000 7f5cce600640 1 --2- 172.21.15.45:0/3301102470 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cc8109560 0x7f5cc81a1110 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.956+0000 7f5cce600640 1 --2- 172.21.15.45:0/3301102470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cc8075660 0x7f5cc819a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.956+0000 7f5cce600640 1 --2- 172.21.15.45:0/3301102470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cc8074080 0x7f5cc8199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:41.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.956+0000 7f5cce600640 1 -- 172.21.15.45:0/3301102470 >> 172.21.15.45:0/3301102470 conn(0x7f5cc80fcc40 msgr2=0x7f5cc81054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:41.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.956+0000 7f5cce600640 1 -- 172.21.15.45:0/3301102470 shutdown_connections 2024-03-28T17:48:41.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:41.956+0000 7f5cce600640 1 -- 172.21.15.45:0/3301102470 wait complete. 2024-03-28T17:48:41.977 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T17:48:42.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.201+0000 7f4132400640 1 -- 172.21.15.45:0/218718688 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c1048e0 msgr2=0x7f412c104d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:42.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.201+0000 7f4132400640 1 --2- 172.21.15.45:0/218718688 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c1048e0 0x7f412c104d60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f411c009860 tx=0x7f411c02bb70 comp rx=0 tx=0).stop 2024-03-28T17:48:42.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.201+0000 7f4132400640 1 -- 172.21.15.45:0/218718688 shutdown_connections 2024-03-28T17:48:42.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.201+0000 7f4132400640 1 --2- 172.21.15.45:0/218718688 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f412c1052a0 0x7f412c10bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:42.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.201+0000 7f4132400640 1 --2- 172.21.15.45:0/218718688 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c1048e0 0x7f412c104d60 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:42.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.201+0000 7f4132400640 1 --2- 172.21.15.45:0/218718688 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f412c1036f0 0x7f412c103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:42.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.201+0000 7f4132400640 1 -- 172.21.15.45:0/218718688 >> 172.21.15.45:0/218718688 conn(0x7f412c0fec80 msgr2=0x7f412c1010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:42.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.201+0000 7f4132400640 1 -- 172.21.15.45:0/218718688 shutdown_connections 2024-03-28T17:48:42.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.201+0000 7f4132400640 1 -- 172.21.15.45:0/218718688 wait complete. 2024-03-28T17:48:42.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.202+0000 7f4132400640 1 Processor -- start 2024-03-28T17:48:42.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.203+0000 7f4132400640 1 -- start start 2024-03-28T17:48:42.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.203+0000 7f4132400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f412c1036f0 0x7f412c1999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:42.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.203+0000 7f4132400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f412c1048e0 0x7f412c199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:42.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.203+0000 7f4132400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c1052a0 0x7f412c1a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:42.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.203+0000 7f4132400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c10f350 con 0x7f412c1048e0 2024-03-28T17:48:42.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.203+0000 7f4132400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c10f1d0 con 0x7f412c1052a0 2024-03-28T17:48:42.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.203+0000 7f4132400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c10f4d0 con 0x7f412c1036f0 2024-03-28T17:48:42.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.203+0000 7f4131a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c1052a0 0x7f412c1a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:42.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.203+0000 7f4131000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f412c1036f0 0x7f412c1999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:42.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.203+0000 7f4131a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c1052a0 0x7f412c1a0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:54880/0 (socket says 172.21.15.45:54880) 2024-03-28T17:48:42.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.203+0000 7f4131a00640 1 -- 172.21.15.45:0/4162442510 learned_addr learned my addr 172.21.15.45:0/4162442510 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:42.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.203+0000 7f4123e00640 1 --2- 172.21.15.45:0/4162442510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f412c1048e0 0x7f412c199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:42.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.204+0000 7f4131000640 1 -- 172.21.15.45:0/4162442510 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c1052a0 msgr2=0x7f412c1a0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:42.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.204+0000 7f4131000640 1 --2- 172.21.15.45:0/4162442510 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c1052a0 0x7f412c1a0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:42.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.204+0000 7f4131000640 1 -- 172.21.15.45:0/4162442510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f412c1048e0 msgr2=0x7f412c199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:42.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.204+0000 7f4131000640 1 --2- 172.21.15.45:0/4162442510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f412c1048e0 0x7f412c199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:42.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.204+0000 7f4131000640 1 -- 172.21.15.45:0/4162442510 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412c1a1610 con 0x7f412c1036f0 2024-03-28T17:48:42.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.204+0000 7f4131a00640 1 --2- 172.21.15.45:0/4162442510 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c1052a0 0x7f412c1a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:42.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.204+0000 7f4131000640 1 --2- 172.21.15.45:0/4162442510 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f412c1036f0 0x7f412c1999f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f411400e9c0 tx=0x7f411400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:42.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.204+0000 7f412a000640 1 -- 172.21.15.45:0/4162442510 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411400cd80 con 0x7f412c1036f0 2024-03-28T17:48:42.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.204+0000 7f4132400640 1 -- 172.21.15.45:0/4162442510 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f412c1a1900 con 0x7f412c1036f0 2024-03-28T17:48:42.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.204+0000 7f412a000640 1 -- 172.21.15.45:0/4162442510 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4114010910 con 0x7f412c1036f0 2024-03-28T17:48:42.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.204+0000 7f412a000640 1 -- 172.21.15.45:0/4162442510 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4114018980 con 0x7f412c1036f0 2024-03-28T17:48:42.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.204+0000 7f4132400640 1 -- 172.21.15.45:0/4162442510 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f412c1a1e70 con 0x7f412c1036f0 2024-03-28T17:48:42.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.206+0000 7f4132400640 1 -- 172.21.15.45:0/4162442510 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f412c10aa60 con 0x7f412c1036f0 2024-03-28T17:48:42.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.207+0000 7f412a000640 1 -- 172.21.15.45:0/4162442510 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f411401f020 con 0x7f412c1036f0 2024-03-28T17:48:42.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.208+0000 7f412a000640 1 --2- 172.21.15.45:0/4162442510 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f410c0a2880 0x7f410c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:42.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.209+0000 7f412a000640 1 -- 172.21.15.45:0/4162442510 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f41140ce9d0 con 0x7f412c1036f0 2024-03-28T17:48:42.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.209+0000 7f4123e00640 1 --2- 172.21.15.45:0/4162442510 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f410c0a2880 0x7f410c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:42.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.210+0000 7f4123e00640 1 --2- 172.21.15.45:0/4162442510 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f410c0a2880 0x7f410c0a4d40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f411c009830 tx=0x7f411c034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:42.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.212+0000 7f412a000640 1 -- 172.21.15.45:0/4162442510 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f411409ba20 con 0x7f412c1036f0 2024-03-28T17:48:42.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.419+0000 7f4132400640 1 -- 172.21.15.45:0/4162442510 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f412c0622a0 con 0x7f412c1036f0 2024-03-28T17:48:42.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.422+0000 7f412a000640 1 -- 172.21.15.45:0/4162442510 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22583 (secure 0 0 0) 0x7f41140b9e20 con 0x7f412c1036f0 2024-03-28T17:48:42.422 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:48:42.423 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"9f4888ec-ed29-11ee-95d0-87774f69a715","created":"2024-03-28T17:36:51.623185+0000","modified":"2024-03-28T17:47:09.455843+0000","last_up_change":"2024-03-28T17:45:14.043432+0000","last_in_change":"2024-03-28T17:44:53.654489+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T17:40:54.208375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T17:46:10.564519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T17:46:11.734367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"74c56bc2-4169-4919-afd4-2ebd236f23e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6803","nonce":3795754023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6805","nonce":3795754023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6809","nonce":3795754023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6807","nonce":3795754023}]},"public_addr":"172.21.15.45:6803/3795754023","cluster_addr":"172.21.15.45:6805/3795754023","heartbeat_back_addr":"172.21.15.45:6809/3795754023","heartbeat_front_addr":"172.21.15.45:6807/3795754023","state":["exists","up"]},{"osd":1,"uuid":"78dcc21b-75b2-453d-b690-17b55b8ed1a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6801","nonce":2014529906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6803","nonce":2014529906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6807","nonce":2014529906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6805","nonce":2014529906}]},"public_addr":"172.21.15.71:6801/2014529906","cluster_addr":"172.21.15.71:6803/2014529906","heartbeat_back_addr":"172.21.15.71:6807/2014529906","heartbeat_front_addr":"172.21.15.71:6805/2014529906","state":["exists","up"]},{"osd":2,"uuid":"ca9f3f36-b71e-4962-a94f-cedf2f12eb7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6801","nonce":1850933775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6803","nonce":1850933775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6807","nonce":1850933775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6805","nonce":1850933775}]},"public_addr":"172.21.15.151:6801/1850933775","cluster_addr":"172.21.15.151:6803/1850933775","heartbeat_back_addr":"172.21.15.151:6807/1850933775","heartbeat_front_addr":"172.21.15.151:6805/1850933775","state":["exists","up"]},{"osd":3,"uuid":"d3f21f01-2973-413e-8daa-9974b8a44af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6811","nonce":3021569975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6813","nonce":3021569975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6817","nonce":3021569975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6815","nonce":3021569975}]},"public_addr":"172.21.15.45:6811/3021569975","cluster_addr":"172.21.15.45:6813/3021569975","heartbeat_back_addr":"172.21.15.45:6817/3021569975","heartbeat_front_addr":"172.21.15.45:6815/3021569975","state":["exists","up"]},{"osd":4,"uuid":"4a336f42-0605-492b-ad88-56cb02acdcf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6809","nonce":2069578333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6811","nonce":2069578333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6815","nonce":2069578333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6813","nonce":2069578333}]},"public_addr":"172.21.15.71:6809/2069578333","cluster_addr":"172.21.15.71:6811/2069578333","heartbeat_back_addr":"172.21.15.71:6815/2069578333","heartbeat_front_addr":"172.21.15.71:6813/2069578333","state":["exists","up"]},{"osd":5,"uuid":"c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6809","nonce":2629217227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6811","nonce":2629217227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6815","nonce":2629217227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6813","nonce":2629217227}]},"public_addr":"172.21.15.151:6809/2629217227","cluster_addr":"172.21.15.151:6811/2629217227","heartbeat_back_addr":"172.21.15.151:6815/2629217227","heartbeat_front_addr":"172.21.15.151:6813/2629217227","state":["exists","up"]},{"osd":6,"uuid":"58a6c5bd-a8b7-4892-b346-bcd7994bcb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6819","nonce":2077907039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6821","nonce":2077907039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6825","nonce":2077907039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6823","nonce":2077907039}]},"public_addr":"172.21.15.45:6819/2077907039","cluster_addr":"172.21.15.45:6821/2077907039","heartbeat_back_addr":"172.21.15.45:6825/2077907039","heartbeat_front_addr":"172.21.15.45:6823/2077907039","state":["exists","up"]},{"osd":7,"uuid":"f7a45d52-807b-4407-bb04-787b2a52a0ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6817","nonce":2000211251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6819","nonce":2000211251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6823","nonce":2000211251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6821","nonce":2000211251}]},"public_addr":"172.21.15.71:6817/2000211251","cluster_addr":"172.21.15.71:6819/2000211251","heartbeat_back_addr":"172.21.15.71:6823/2000211251","heartbeat_front_addr":"172.21.15.71:6821/2000211251","state":["exists","up"]},{"osd":8,"uuid":"88abcb0f-f189-49be-8315-c700c77369f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6817","nonce":3665460137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6819","nonce":3665460137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6823","nonce":3665460137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6821","nonce":3665460137}]},"public_addr":"172.21.15.151:6817/3665460137","cluster_addr":"172.21.15.151:6819/3665460137","heartbeat_back_addr":"172.21.15.151:6823/3665460137","heartbeat_front_addr":"172.21.15.151:6821/3665460137","state":["exists","up"]},{"osd":9,"uuid":"ed3fc096-961e-4392-abc2-c5aa4d050117","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6827","nonce":2122322684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6829","nonce":2122322684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6833","nonce":2122322684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6831","nonce":2122322684}]},"public_addr":"172.21.15.45:6827/2122322684","cluster_addr":"172.21.15.45:6829/2122322684","heartbeat_back_addr":"172.21.15.45:6833/2122322684","heartbeat_front_addr":"172.21.15.45:6831/2122322684","state":["exists","up"]},{"osd":10,"uuid":"ba94762b-bc5c-4c97-9cc1-17e5651c14b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6825","nonce":1099494037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6827","nonce":1099494037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6831","nonce":1099494037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6829","nonce":1099494037}]},"public_addr":"172.21.15.71:6825/1099494037","cluster_addr":"172.21.15.71:6827/1099494037","heartbeat_back_addr":"172.21.15.71:6831/1099494037","heartbeat_front_addr":"172.21.15.71:6829/1099494037","state":["exists","up"]},{"osd":11,"uuid":"1d178378-b9f5-40f6-99ef-3d3d445da640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6825","nonce":2941055364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6827","nonce":2941055364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6831","nonce":2941055364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6829","nonce":2941055364}]},"public_addr":"172.21.15.151:6825/2941055364","cluster_addr":"172.21.15.151:6827/2941055364","heartbeat_back_addr":"172.21.15.151:6831/2941055364","heartbeat_front_addr":"172.21.15.151:6829/2941055364","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:02.269586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:27.819386+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:51.552277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:25.052674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:51.873529+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:17.127867+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:51.588249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:18.381450+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:44.140534+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:17.634362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:45.469340+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:45:12.003433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3533611957":"2024-03-29T17:38:07.505611+0000","172.21.15.45:0/3625922430":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6801/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/482610468":"2024-03-29T17:37:32.424229+0000","172.21.15.45:6801/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3649804714":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/1198364079":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3211303958":"2024-03-29T17:38:07.505611+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T17:48:42.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.424+0000 7f4132400640 1 -- 172.21.15.45:0/4162442510 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f410c0a2880 msgr2=0x7f410c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:42.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.424+0000 7f4132400640 1 --2- 172.21.15.45:0/4162442510 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f410c0a2880 0x7f410c0a4d40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f411c009830 tx=0x7f411c034ab0 comp rx=0 tx=0).stop 2024-03-28T17:48:42.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.424+0000 7f4132400640 1 -- 172.21.15.45:0/4162442510 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f412c1036f0 msgr2=0x7f412c1999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:42.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.424+0000 7f4132400640 1 --2- 172.21.15.45:0/4162442510 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f412c1036f0 0x7f412c1999f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f411400e9c0 tx=0x7f411400ee80 comp rx=0 tx=0).stop 2024-03-28T17:48:42.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.425+0000 7f4132400640 1 -- 172.21.15.45:0/4162442510 shutdown_connections 2024-03-28T17:48:42.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.425+0000 7f4132400640 1 --2- 172.21.15.45:0/4162442510 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f410c0a2880 0x7f410c0a4d40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:42.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.425+0000 7f4132400640 1 --2- 172.21.15.45:0/4162442510 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c1052a0 0x7f412c1a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:42.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.425+0000 7f4132400640 1 --2- 172.21.15.45:0/4162442510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f412c1048e0 0x7f412c199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:42.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.425+0000 7f4132400640 1 --2- 172.21.15.45:0/4162442510 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f412c1036f0 0x7f412c1999f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:42.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.425+0000 7f4132400640 1 -- 172.21.15.45:0/4162442510 >> 172.21.15.45:0/4162442510 conn(0x7f412c0fec80 msgr2=0x7f412c107c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:42.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.425+0000 7f4132400640 1 -- 172.21.15.45:0/4162442510 shutdown_connections 2024-03-28T17:48:42.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.425+0000 7f4132400640 1 -- 172.21.15.45:0/4162442510 wait complete. 2024-03-28T17:48:42.445 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T17:40:54.208375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T17:46:10.564519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T17:46:11.734367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T17:48:42.445 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T17:48:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.677+0000 7f6a55c00640 1 -- 172.21.15.45:0/3771780303 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a50109870 msgr2=0x7f6a50110100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.677+0000 7f6a55c00640 1 --2- 172.21.15.45:0/3771780303 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a50109870 0x7f6a50110100 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6a48009be0 tx=0x7f6a4802bef0 comp rx=0 tx=0).stop 2024-03-28T17:48:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.677+0000 7f6a55c00640 1 -- 172.21.15.45:0/3771780303 shutdown_connections 2024-03-28T17:48:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.677+0000 7f6a55c00640 1 --2- 172.21.15.45:0/3771780303 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a50109870 0x7f6a50110100 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.677+0000 7f6a55c00640 1 --2- 172.21.15.45:0/3771780303 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a50108eb0 0x7f6a50109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:42.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.677+0000 7f6a55c00640 1 --2- 172.21.15.45:0/3771780303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a50107cc0 0x7f6a501080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:42.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.678+0000 7f6a55c00640 1 -- 172.21.15.45:0/3771780303 >> 172.21.15.45:0/3771780303 conn(0x7f6a50076c60 msgr2=0x7f6a500790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:42.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.678+0000 7f6a55c00640 1 -- 172.21.15.45:0/3771780303 shutdown_connections 2024-03-28T17:48:42.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.678+0000 7f6a55c00640 1 -- 172.21.15.45:0/3771780303 wait complete. 2024-03-28T17:48:42.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.679+0000 7f6a55c00640 1 Processor -- start 2024-03-28T17:48:42.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.679+0000 7f6a55c00640 1 -- start start 2024-03-28T17:48:42.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.679+0000 7f6a55c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a50107cc0 0x7f6a5019df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:42.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.679+0000 7f6a55c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a50108eb0 0x7f6a5019e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:42.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.679+0000 7f6a55c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a50109870 0x7f6a501a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:42.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.679+0000 7f6a55c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a50113970 con 0x7f6a50107cc0 2024-03-28T17:48:42.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.679+0000 7f6a55c00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a501137f0 con 0x7f6a50108eb0 2024-03-28T17:48:42.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.679+0000 7f6a55c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a50113af0 con 0x7f6a50109870 2024-03-28T17:48:42.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.679+0000 7f6a4ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a50107cc0 0x7f6a5019df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:42.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.679+0000 7f6a4f400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a50109870 0x7f6a501a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:42.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.679+0000 7f6a47e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a50108eb0 0x7f6a5019e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:42.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.679+0000 7f6a4ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a50107cc0 0x7f6a5019df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44290/0 (socket says 172.21.15.45:44290) 2024-03-28T17:48:42.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.679+0000 7f6a4ea00640 1 -- 172.21.15.45:0/112426089 learned_addr learned my addr 172.21.15.45:0/112426089 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:42.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.680+0000 7f6a4f400640 1 -- 172.21.15.45:0/112426089 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a50108eb0 msgr2=0x7f6a5019e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:42.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.680+0000 7f6a4f400640 1 --2- 172.21.15.45:0/112426089 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a50108eb0 0x7f6a5019e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:42.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.680+0000 7f6a4f400640 1 -- 172.21.15.45:0/112426089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a50107cc0 msgr2=0x7f6a5019df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.680+0000 7f6a4f400640 1 --2- 172.21.15.45:0/112426089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a50107cc0 0x7f6a5019df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.680+0000 7f6a4f400640 1 -- 172.21.15.45:0/112426089 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a501a5b50 con 0x7f6a50109870 2024-03-28T17:48:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.680+0000 7f6a4f400640 1 --2- 172.21.15.45:0/112426089 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a50109870 0x7f6a501a54e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6a4802c430 tx=0x7f6a480337b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.680+0000 7f6a47400640 1 -- 172.21.15.45:0/112426089 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a48043020 con 0x7f6a50109870 2024-03-28T17:48:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.680+0000 7f6a47400640 1 -- 172.21.15.45:0/112426089 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a4802ca60 con 0x7f6a50109870 2024-03-28T17:48:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.680+0000 7f6a47400640 1 -- 172.21.15.45:0/112426089 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a48034690 con 0x7f6a50109870 2024-03-28T17:48:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.680+0000 7f6a55c00640 1 -- 172.21.15.45:0/112426089 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a501a5de0 con 0x7f6a50109870 2024-03-28T17:48:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.681+0000 7f6a55c00640 1 -- 172.21.15.45:0/112426089 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a501a62a0 con 0x7f6a50109870 2024-03-28T17:48:42.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.682+0000 7f6a55c00640 1 -- 172.21.15.45:0/112426089 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a5010ef80 con 0x7f6a50109870 2024-03-28T17:48:42.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.683+0000 7f6a47400640 1 -- 172.21.15.45:0/112426089 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f6a48046020 con 0x7f6a50109870 2024-03-28T17:48:42.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.685+0000 7f6a47400640 1 --2- 172.21.15.45:0/112426089 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6a340a2890 0x7f6a340a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:42.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.685+0000 7f6a47400640 1 -- 172.21.15.45:0/112426089 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6a480f1060 con 0x7f6a50109870 2024-03-28T17:48:42.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.685+0000 7f6a4ea00640 1 --2- 172.21.15.45:0/112426089 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6a340a2890 0x7f6a340a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:42.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.686+0000 7f6a4ea00640 1 --2- 172.21.15.45:0/112426089 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6a340a2890 0x7f6a340a4d50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6a400058c0 tx=0x7f6a40005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:42.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.689+0000 7f6a47400640 1 -- 172.21.15.45:0/112426089 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6a4804d020 con 0x7f6a50109870 2024-03-28T17:48:42.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:42 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3301102470' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T17:48:42.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:42 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/4162442510' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:48:42.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.891+0000 7f6a55c00640 1 -- 172.21.15.45:0/112426089 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6a5002d240 con 0x7f6a50109870 2024-03-28T17:48:42.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.892+0000 7f6a47400640 1 -- 172.21.15.45:0/112426089 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6a480be060 con 0x7f6a50109870 2024-03-28T17:48:42.892 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-28T17:48:42.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.895+0000 7f6a55c00640 1 -- 172.21.15.45:0/112426089 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6a340a2890 msgr2=0x7f6a340a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:42.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.895+0000 7f6a55c00640 1 --2- 172.21.15.45:0/112426089 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6a340a2890 0x7f6a340a4d50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6a400058c0 tx=0x7f6a40005f90 comp rx=0 tx=0).stop 2024-03-28T17:48:42.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.895+0000 7f6a55c00640 1 -- 172.21.15.45:0/112426089 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a50109870 msgr2=0x7f6a501a54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:42.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.895+0000 7f6a55c00640 1 --2- 172.21.15.45:0/112426089 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a50109870 0x7f6a501a54e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6a4802c430 tx=0x7f6a480337b0 comp rx=0 tx=0).stop 2024-03-28T17:48:42.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.895+0000 7f6a55c00640 1 -- 172.21.15.45:0/112426089 shutdown_connections 2024-03-28T17:48:42.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.895+0000 7f6a55c00640 1 --2- 172.21.15.45:0/112426089 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6a340a2890 0x7f6a340a4d50 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:42.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.895+0000 7f6a55c00640 1 --2- 172.21.15.45:0/112426089 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a50109870 0x7f6a501a54e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:42.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.895+0000 7f6a55c00640 1 --2- 172.21.15.45:0/112426089 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a50108eb0 0x7f6a5019e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:42.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.895+0000 7f6a55c00640 1 --2- 172.21.15.45:0/112426089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a50107cc0 0x7f6a5019df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:42.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.895+0000 7f6a55c00640 1 -- 172.21.15.45:0/112426089 >> 172.21.15.45:0/112426089 conn(0x7f6a50076c60 msgr2=0x7f6a5010c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:42.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.895+0000 7f6a55c00640 1 -- 172.21.15.45:0/112426089 shutdown_connections 2024-03-28T17:48:42.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:42.895+0000 7f6a55c00640 1 -- 172.21.15.45:0/112426089 wait complete. 2024-03-28T17:48:42.912 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:42 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3301102470' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T17:48:42.912 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:42 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/4162442510' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:48:42.916 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T17:48:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:42 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3301102470' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T17:48:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:42 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/4162442510' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:48:43.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.146+0000 7fa821400640 1 -- 172.21.15.45:0/3818059864 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa81c0ffdf0 msgr2=0x7fa81c102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:43.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.146+0000 7fa821400640 1 --2- 172.21.15.45:0/3818059864 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa81c0ffdf0 0x7fa81c102220 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa80c009900 tx=0x7fa80c02bc40 comp rx=0 tx=0).stop 2024-03-28T17:48:43.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.146+0000 7fa821400640 1 -- 172.21.15.45:0/3818059864 shutdown_connections 2024-03-28T17:48:43.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.146+0000 7fa821400640 1 --2- 172.21.15.45:0/3818059864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa81c105110 0x7fa81c105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:43.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.146+0000 7fa821400640 1 --2- 172.21.15.45:0/3818059864 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa81c102760 0x7fa81c104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:43.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.146+0000 7fa821400640 1 --2- 172.21.15.45:0/3818059864 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa81c0ffdf0 0x7fa81c102220 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:43.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.146+0000 7fa821400640 1 -- 172.21.15.45:0/3818059864 >> 172.21.15.45:0/3818059864 conn(0x7fa81c0fba10 msgr2=0x7fa81c0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:43.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.146+0000 7fa821400640 1 -- 172.21.15.45:0/3818059864 shutdown_connections 2024-03-28T17:48:43.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.146+0000 7fa821400640 1 -- 172.21.15.45:0/3818059864 wait complete. 2024-03-28T17:48:43.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.147+0000 7fa821400640 1 Processor -- start 2024-03-28T17:48:43.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.147+0000 7fa821400640 1 -- start start 2024-03-28T17:48:43.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.148+0000 7fa821400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa81c0ffdf0 0x7fa81c19de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:43.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.148+0000 7fa821400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa81c102760 0x7fa81c19e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:43.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.148+0000 7fa821400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa81c105110 0x7fa81c1a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:43.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.148+0000 7fa821400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c113840 con 0x7fa81c105110 2024-03-28T17:48:43.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.148+0000 7fa821400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c1136c0 con 0x7fa81c102760 2024-03-28T17:48:43.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.148+0000 7fa821400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c1139c0 con 0x7fa81c0ffdf0 2024-03-28T17:48:43.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.148+0000 7fa81be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa81c0ffdf0 0x7fa81c19de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:43.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.148+0000 7fa81be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa81c0ffdf0 0x7fa81c19de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:49700/0 (socket says 172.21.15.45:49700) 2024-03-28T17:48:43.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.148+0000 7fa81be00640 1 -- 172.21.15.45:0/3597514980 learned_addr learned my addr 172.21.15.45:0/3597514980 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:43.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.148+0000 7fa820a00640 1 --2- 172.21.15.45:0/3597514980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa81c105110 0x7fa81c1a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:43.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.148+0000 7fa81b400640 1 --2- 172.21.15.45:0/3597514980 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa81c102760 0x7fa81c19e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:43.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.148+0000 7fa81be00640 1 -- 172.21.15.45:0/3597514980 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa81c102760 msgr2=0x7fa81c19e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:43.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.148+0000 7fa81be00640 1 --2- 172.21.15.45:0/3597514980 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa81c102760 0x7fa81c19e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:43.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.149+0000 7fa81be00640 1 -- 172.21.15.45:0/3597514980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa81c105110 msgr2=0x7fa81c1a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:43.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.149+0000 7fa81be00640 1 --2- 172.21.15.45:0/3597514980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa81c105110 0x7fa81c1a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:43.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.149+0000 7fa81be00640 1 -- 172.21.15.45:0/3597514980 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81c1a5aa0 con 0x7fa81c0ffdf0 2024-03-28T17:48:43.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.149+0000 7fa81be00640 1 --2- 172.21.15.45:0/3597514980 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa81c0ffdf0 0x7fa81c19de80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa80c02c8f0 tx=0x7fa80c005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:43.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.149+0000 7fa818c00640 1 -- 172.21.15.45:0/3597514980 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80c041070 con 0x7fa81c0ffdf0 2024-03-28T17:48:43.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.149+0000 7fa818c00640 1 -- 172.21.15.45:0/3597514980 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa80c0044d0 con 0x7fa81c0ffdf0 2024-03-28T17:48:43.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.149+0000 7fa821400640 1 -- 172.21.15.45:0/3597514980 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa81c1a5d30 con 0x7fa81c0ffdf0 2024-03-28T17:48:43.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.149+0000 7fa818c00640 1 -- 172.21.15.45:0/3597514980 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80c034440 con 0x7fa81c0ffdf0 2024-03-28T17:48:43.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.149+0000 7fa821400640 1 -- 172.21.15.45:0/3597514980 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81c1a61c0 con 0x7fa81c0ffdf0 2024-03-28T17:48:43.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.150+0000 7fa821400640 1 -- 172.21.15.45:0/3597514980 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa81c0fd500 con 0x7fa81c0ffdf0 2024-03-28T17:48:43.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.152+0000 7fa818c00640 1 -- 172.21.15.45:0/3597514980 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fa80c044070 con 0x7fa81c0ffdf0 2024-03-28T17:48:43.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.153+0000 7fa818c00640 1 --2- 172.21.15.45:0/3597514980 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fa7fc0a2910 0x7fa7fc0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:43.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.154+0000 7fa818c00640 1 -- 172.21.15.45:0/3597514980 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa80c0f0610 con 0x7fa81c0ffdf0 2024-03-28T17:48:43.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.154+0000 7fa81b400640 1 --2- 172.21.15.45:0/3597514980 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fa7fc0a2910 0x7fa7fc0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:43.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.154+0000 7fa81b400640 1 --2- 172.21.15.45:0/3597514980 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fa7fc0a2910 0x7fa7fc0a4dd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa81c19f390 tx=0x7fa804009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:43.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.158+0000 7fa818c00640 1 -- 172.21.15.45:0/3597514980 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa80c0bcfc0 con 0x7fa81c0ffdf0 2024-03-28T17:48:43.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.363+0000 7fa821400640 1 -- 172.21.15.45:0/3597514980 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa81c02d280 con 0x7fa81c0ffdf0 2024-03-28T17:48:43.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.364+0000 7fa818c00640 1 -- 172.21.15.45:0/3597514980 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa80c0c1060 con 0x7fa81c0ffdf0 2024-03-28T17:48:43.364 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-03-28T17:48:43.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.367+0000 7fa821400640 1 -- 172.21.15.45:0/3597514980 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fa7fc0a2910 msgr2=0x7fa7fc0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:43.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.367+0000 7fa821400640 1 --2- 172.21.15.45:0/3597514980 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fa7fc0a2910 0x7fa7fc0a4dd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa81c19f390 tx=0x7fa804009500 comp rx=0 tx=0).stop 2024-03-28T17:48:43.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.367+0000 7fa821400640 1 -- 172.21.15.45:0/3597514980 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa81c0ffdf0 msgr2=0x7fa81c19de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:43.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.367+0000 7fa821400640 1 --2- 172.21.15.45:0/3597514980 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa81c0ffdf0 0x7fa81c19de80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa80c02c8f0 tx=0x7fa80c005d50 comp rx=0 tx=0).stop 2024-03-28T17:48:43.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.367+0000 7fa821400640 1 -- 172.21.15.45:0/3597514980 shutdown_connections 2024-03-28T17:48:43.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.367+0000 7fa821400640 1 --2- 172.21.15.45:0/3597514980 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fa7fc0a2910 0x7fa7fc0a4dd0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:43.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.367+0000 7fa821400640 1 --2- 172.21.15.45:0/3597514980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa81c105110 0x7fa81c1a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:43.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.367+0000 7fa821400640 1 --2- 172.21.15.45:0/3597514980 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa81c102760 0x7fa81c19e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:43.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.367+0000 7fa821400640 1 --2- 172.21.15.45:0/3597514980 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa81c0ffdf0 0x7fa81c19de80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:43.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.367+0000 7fa821400640 1 -- 172.21.15.45:0/3597514980 >> 172.21.15.45:0/3597514980 conn(0x7fa81c0fba10 msgr2=0x7fa81c103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:43.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.367+0000 7fa821400640 1 -- 172.21.15.45:0/3597514980 shutdown_connections 2024-03-28T17:48:43.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.367+0000 7fa821400640 1 -- 172.21.15.45:0/3597514980 wait complete. 2024-03-28T17:48:43.388 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T17:48:43.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.612+0000 7fcf7f200640 1 -- 172.21.15.45:0/1687423641 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf801015a0 msgr2=0x7fcf801019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:43.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.612+0000 7fcf7f200640 1 --2- 172.21.15.45:0/1687423641 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf801015a0 0x7fcf801019c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcf70009900 tx=0x7fcf7002bc30 comp rx=0 tx=0).stop 2024-03-28T17:48:43.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.613+0000 7fcf7f200640 1 -- 172.21.15.45:0/1687423641 shutdown_connections 2024-03-28T17:48:43.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.613+0000 7fcf7f200640 1 --2- 172.21.15.45:0/1687423641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf80103150 0x7fcf801099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:43.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.613+0000 7fcf7f200640 1 --2- 172.21.15.45:0/1687423641 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf80102790 0x7fcf80102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:43.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.613+0000 7fcf7f200640 1 --2- 172.21.15.45:0/1687423641 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf801015a0 0x7fcf801019c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:43.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.613+0000 7fcf7f200640 1 -- 172.21.15.45:0/1687423641 >> 172.21.15.45:0/1687423641 conn(0x7fcf800fcb10 msgr2=0x7fcf800fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:43.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.613+0000 7fcf7f200640 1 -- 172.21.15.45:0/1687423641 shutdown_connections 2024-03-28T17:48:43.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.613+0000 7fcf7f200640 1 -- 172.21.15.45:0/1687423641 wait complete. 2024-03-28T17:48:43.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.614+0000 7fcf7f200640 1 Processor -- start 2024-03-28T17:48:43.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.614+0000 7fcf7f200640 1 -- start start 2024-03-28T17:48:43.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.614+0000 7fcf7f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf801015a0 0x7fcf80075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:43.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.615+0000 7fcf7f200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf80102790 0x7fcf80073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:43.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.615+0000 7fcf7f200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf80103150 0x7fcf80073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:43.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.615+0000 7fcf7f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8010d260 con 0x7fcf801015a0 2024-03-28T17:48:43.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.615+0000 7fcf7f200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8010d0e0 con 0x7fcf80103150 2024-03-28T17:48:43.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.615+0000 7fcf7f200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8010d3e0 con 0x7fcf80102790 2024-03-28T17:48:43.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.615+0000 7fcf7e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf80103150 0x7fcf80073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:43.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.615+0000 7fcf7de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf801015a0 0x7fcf80075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:43.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.615+0000 7fcf7d400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf80102790 0x7fcf80073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:43.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.615+0000 7fcf7e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf80103150 0x7fcf80073fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:54936/0 (socket says 172.21.15.45:54936) 2024-03-28T17:48:43.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.615+0000 7fcf7e800640 1 -- 172.21.15.45:0/3208382511 learned_addr learned my addr 172.21.15.45:0/3208382511 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:43.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.615+0000 7fcf7e800640 1 -- 172.21.15.45:0/3208382511 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf80102790 msgr2=0x7fcf80073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:43.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.615+0000 7fcf7e800640 1 --2- 172.21.15.45:0/3208382511 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf80102790 0x7fcf80073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:43.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.615+0000 7fcf7e800640 1 -- 172.21.15.45:0/3208382511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf801015a0 msgr2=0x7fcf80075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:43.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.615+0000 7fcf7e800640 1 --2- 172.21.15.45:0/3208382511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf801015a0 0x7fcf80075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:43.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.615+0000 7fcf7e800640 1 -- 172.21.15.45:0/3208382511 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf80074860 con 0x7fcf80103150 2024-03-28T17:48:43.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.615+0000 7fcf7de00640 1 --2- 172.21.15.45:0/3208382511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf801015a0 0x7fcf80075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:43.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.616+0000 7fcf7e800640 1 --2- 172.21.15.45:0/3208382511 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf80103150 0x7fcf80073fd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcf7800eb10 tx=0x7fcf7800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:43.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.616+0000 7fcf76a00640 1 -- 172.21.15.45:0/3208382511 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf78004b20 con 0x7fcf80103150 2024-03-28T17:48:43.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.616+0000 7fcf76a00640 1 -- 172.21.15.45:0/3208382511 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf78004cc0 con 0x7fcf80103150 2024-03-28T17:48:43.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.616+0000 7fcf76a00640 1 -- 172.21.15.45:0/3208382511 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf78010760 con 0x7fcf80103150 2024-03-28T17:48:43.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.616+0000 7fcf7f200640 1 -- 172.21.15.45:0/3208382511 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf801a19c0 con 0x7fcf80103150 2024-03-28T17:48:43.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.616+0000 7fcf7f200640 1 -- 172.21.15.45:0/3208382511 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf801a1ec0 con 0x7fcf80103150 2024-03-28T17:48:43.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.617+0000 7fcf7f200640 1 -- 172.21.15.45:0/3208382511 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf801088c0 con 0x7fcf80103150 2024-03-28T17:48:43.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.621+0000 7fcf76a00640 1 -- 172.21.15.45:0/3208382511 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fcf78005020 con 0x7fcf80103150 2024-03-28T17:48:43.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.622+0000 7fcf76a00640 1 --2- 172.21.15.45:0/3208382511 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcf640a2890 0x7fcf640a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:43.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.623+0000 7fcf76a00640 1 -- 172.21.15.45:0/3208382511 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fcf780cf1c0 con 0x7fcf80103150 2024-03-28T17:48:43.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.623+0000 7fcf7de00640 1 --2- 172.21.15.45:0/3208382511 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcf640a2890 0x7fcf640a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:43.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.623+0000 7fcf7de00640 1 --2- 172.21.15.45:0/3208382511 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcf640a2890 0x7fcf640a4d50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcf7002c170 tx=0x7fcf7002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:43.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.627+0000 7fcf76a00640 1 -- 172.21.15.45:0/3208382511 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcf7809c110 con 0x7fcf80103150 2024-03-28T17:48:43.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:43 smithi151 ceph-mon[36481]: pgmap v394: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T17:48:43.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:43 smithi151 ceph-mon[36481]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-03-28T17:48:43.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:43 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/112426089' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:48:43.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:43 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3597514980' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T17:48:43.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.842+0000 7fcf7f200640 1 -- 172.21.15.45:0/3208382511 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcf8002d2c0 con 0x7fcf80103150 2024-03-28T17:48:43.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.842+0000 7fcf76a00640 1 -- 172.21.15.45:0/3208382511 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcf7809ba10 con 0x7fcf80103150 2024-03-28T17:48:43.843 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-03-28T17:48:43.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.845+0000 7fcf7f200640 1 -- 172.21.15.45:0/3208382511 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcf640a2890 msgr2=0x7fcf640a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:43.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.845+0000 7fcf7f200640 1 --2- 172.21.15.45:0/3208382511 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcf640a2890 0x7fcf640a4d50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcf7002c170 tx=0x7fcf7002df90 comp rx=0 tx=0).stop 2024-03-28T17:48:43.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.845+0000 7fcf7f200640 1 -- 172.21.15.45:0/3208382511 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf80103150 msgr2=0x7fcf80073fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:43.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.845+0000 7fcf7f200640 1 --2- 172.21.15.45:0/3208382511 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf80103150 0x7fcf80073fd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcf7800eb10 tx=0x7fcf7800c2d0 comp rx=0 tx=0).stop 2024-03-28T17:48:43.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.845+0000 7fcf7f200640 1 -- 172.21.15.45:0/3208382511 shutdown_connections 2024-03-28T17:48:43.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.845+0000 7fcf7f200640 1 --2- 172.21.15.45:0/3208382511 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcf640a2890 0x7fcf640a4d50 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:43.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.845+0000 7fcf7f200640 1 --2- 172.21.15.45:0/3208382511 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf80103150 0x7fcf80073fd0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:43.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.845+0000 7fcf7f200640 1 --2- 172.21.15.45:0/3208382511 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf80102790 0x7fcf80073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:43.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.845+0000 7fcf7f200640 1 --2- 172.21.15.45:0/3208382511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf801015a0 0x7fcf80075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:43.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.845+0000 7fcf7f200640 1 -- 172.21.15.45:0/3208382511 >> 172.21.15.45:0/3208382511 conn(0x7fcf800fcb10 msgr2=0x7fcf80105a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:43.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.846+0000 7fcf7f200640 1 -- 172.21.15.45:0/3208382511 shutdown_connections 2024-03-28T17:48:43.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:43.846+0000 7fcf7f200640 1 -- 172.21.15.45:0/3208382511 wait complete. 2024-03-28T17:48:43.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:43 smithi045 ceph-mon[31773]: pgmap v394: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T17:48:43.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:43 smithi045 ceph-mon[31773]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-03-28T17:48:43.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:43 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/112426089' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:48:43.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:43 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3597514980' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T17:48:43.864 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-28T17:48:43.864 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:48:44.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:43 smithi071 ceph-mon[36597]: pgmap v394: 97 pgs: 97 active+clean; 641 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T17:48:44.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:43 smithi071 ceph-mon[36597]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-03-28T17:48:44.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:43 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/112426089' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:48:44.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:43 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3597514980' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T17:48:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.078+0000 7fd701e00640 1 -- 172.21.15.45:0/3481118243 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6fc1026d0 msgr2=0x7fd6fc104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.078+0000 7fd701e00640 1 --2- 172.21.15.45:0/3481118243 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6fc1026d0 0x7fd6fc104b40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8009900 tx=0x7fd6e802bc30 comp rx=0 tx=0).stop 2024-03-28T17:48:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.078+0000 7fd701e00640 1 -- 172.21.15.45:0/3481118243 shutdown_connections 2024-03-28T17:48:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.078+0000 7fd701e00640 1 --2- 172.21.15.45:0/3481118243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6fc105080 0x7fd6fc105500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.078+0000 7fd701e00640 1 --2- 172.21.15.45:0/3481118243 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6fc1026d0 0x7fd6fc104b40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:44.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.078+0000 7fd701e00640 1 --2- 172.21.15.45:0/3481118243 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6fc0ffd60 0x7fd6fc102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:44.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.078+0000 7fd701e00640 1 -- 172.21.15.45:0/3481118243 >> 172.21.15.45:0/3481118243 conn(0x7fd6fc0fb980 msgr2=0x7fd6fc0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:44.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.078+0000 7fd701e00640 1 -- 172.21.15.45:0/3481118243 shutdown_connections 2024-03-28T17:48:44.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.078+0000 7fd701e00640 1 -- 172.21.15.45:0/3481118243 wait complete. 2024-03-28T17:48:44.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.079+0000 7fd701e00640 1 Processor -- start 2024-03-28T17:48:44.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.079+0000 7fd701e00640 1 -- start start 2024-03-28T17:48:44.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.079+0000 7fd701e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6fc0ffd60 0x7fd6fc19ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:44.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.079+0000 7fd701e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6fc1026d0 0x7fd6fc19e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:44.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.079+0000 7fd701e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6fc105080 0x7fd6fc1a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:44.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.079+0000 7fd701e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc1137b0 con 0x7fd6fc0ffd60 2024-03-28T17:48:44.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.079+0000 7fd701e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc113630 con 0x7fd6fc1026d0 2024-03-28T17:48:44.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.079+0000 7fd701e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc113930 con 0x7fd6fc105080 2024-03-28T17:48:44.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.079+0000 7fd6fbe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6fc1026d0 0x7fd6fc19e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:44.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.079+0000 7fd700a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6fc0ffd60 0x7fd6fc19ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:44.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.079+0000 7fd6fbe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6fc1026d0 0x7fd6fc19e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:54944/0 (socket says 172.21.15.45:54944) 2024-03-28T17:48:44.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.079+0000 7fd6fbe00640 1 -- 172.21.15.45:0/1098089609 learned_addr learned my addr 172.21.15.45:0/1098089609 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:44.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.080+0000 7fd701400640 1 --2- 172.21.15.45:0/1098089609 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6fc105080 0x7fd6fc1a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:44.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.080+0000 7fd6fbe00640 1 -- 172.21.15.45:0/1098089609 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6fc105080 msgr2=0x7fd6fc1a53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:44.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.080+0000 7fd6fbe00640 1 --2- 172.21.15.45:0/1098089609 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6fc105080 0x7fd6fc1a53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:44.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.080+0000 7fd6fbe00640 1 -- 172.21.15.45:0/1098089609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6fc0ffd60 msgr2=0x7fd6fc19ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:44.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.080+0000 7fd6fbe00640 1 --2- 172.21.15.45:0/1098089609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6fc0ffd60 0x7fd6fc19ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:44.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.080+0000 7fd6fbe00640 1 -- 172.21.15.45:0/1098089609 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6fc1a5a10 con 0x7fd6fc1026d0 2024-03-28T17:48:44.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.080+0000 7fd6fbe00640 1 --2- 172.21.15.45:0/1098089609 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6fc1026d0 0x7fd6fc19e330 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8000c00 tx=0x7fd6e802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:44.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.081+0000 7fd6f9600640 1 -- 172.21.15.45:0/1098089609 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8043020 con 0x7fd6fc1026d0 2024-03-28T17:48:44.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.081+0000 7fd6f9600640 1 -- 172.21.15.45:0/1098089609 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6e80054e0 con 0x7fd6fc1026d0 2024-03-28T17:48:44.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.081+0000 7fd701e00640 1 -- 172.21.15.45:0/1098089609 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6fc1a5ca0 con 0x7fd6fc1026d0 2024-03-28T17:48:44.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.081+0000 7fd6f9600640 1 -- 172.21.15.45:0/1098089609 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8003bd0 con 0x7fd6fc1026d0 2024-03-28T17:48:44.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.081+0000 7fd701e00640 1 -- 172.21.15.45:0/1098089609 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6fc1a61e0 con 0x7fd6fc1026d0 2024-03-28T17:48:44.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.082+0000 7fd701e00640 1 -- 172.21.15.45:0/1098089609 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6fc0fd470 con 0x7fd6fc1026d0 2024-03-28T17:48:44.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.084+0000 7fd6f9600640 1 -- 172.21.15.45:0/1098089609 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fd6e8046020 con 0x7fd6fc1026d0 2024-03-28T17:48:44.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.086+0000 7fd6f9600640 1 --2- 172.21.15.45:0/1098089609 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd6e40a2910 0x7fd6e40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:44.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.086+0000 7fd6f9600640 1 -- 172.21.15.45:0/1098089609 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd6e80f0b00 con 0x7fd6fc1026d0 2024-03-28T17:48:44.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.086+0000 7fd700a00640 1 --2- 172.21.15.45:0/1098089609 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd6e40a2910 0x7fd6e40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:44.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.087+0000 7fd700a00640 1 --2- 172.21.15.45:0/1098089609 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd6e40a2910 0x7fd6e40a4dd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0009bd0 tx=0x7fd6f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:44.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.089+0000 7fd6f9600640 1 -- 172.21.15.45:0/1098089609 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd6e804d020 con 0x7fd6fc1026d0 2024-03-28T17:48:44.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.355+0000 7fd701e00640 1 -- 172.21.15.45:0/1098089609 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6fc02d2c0 con 0x7fd6fc1026d0 2024-03-28T17:48:44.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.356+0000 7fd6f9600640 1 -- 172.21.15.45:0/1098089609 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7fd6e80bdb00 con 0x7fd6fc1026d0 2024-03-28T17:48:44.356 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:48:44.356 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6839/2313500150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":2313500150},{"type":"v1","addr":"172.21.15.45:6839","nonce":2313500150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6841/2647972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":2647972571},{"type":"v1","addr":"172.21.15.45:6841","nonce":2647972571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T17:46:12.261135+0000","modified":"2024-03-28T17:48:38.762497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24584,"mds_1":24530,"mds_2":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6835/2876354410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":2876354410},{"type":"v1","addr":"172.21.15.151:6835","nonce":2876354410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24500":{"gid":24500,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/4234998385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":4234998385},{"type":"v1","addr":"172.21.15.71:6835","nonce":4234998385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6837/1238964845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":1238964845},{"type":"v1","addr":"172.21.15.151:6837","nonce":1238964845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.151:6839/3003564901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6838","nonce":3003564901},{"type":"v1","addr":"172.21.15.151:6839","nonce":3003564901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6837/2008752655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2008752655},{"type":"v1","addr":"172.21.15.71:6837","nonce":2008752655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.71:6839/2709938800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":2709938800},{"type":"v1","addr":"172.21.15.71:6839","nonce":2709938800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24584,"qdb_cluster":[24584,24529,24530]},"id":1}]} 2024-03-28T17:48:44.357 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 23 2024-03-28T17:48:44.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.359+0000 7fd701e00640 1 -- 172.21.15.45:0/1098089609 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd6e40a2910 msgr2=0x7fd6e40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:44.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.359+0000 7fd701e00640 1 --2- 172.21.15.45:0/1098089609 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd6e40a2910 0x7fd6e40a4dd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0009bd0 tx=0x7fd6f0005f90 comp rx=0 tx=0).stop 2024-03-28T17:48:44.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.360+0000 7fd701e00640 1 -- 172.21.15.45:0/1098089609 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6fc1026d0 msgr2=0x7fd6fc19e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:44.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.360+0000 7fd701e00640 1 --2- 172.21.15.45:0/1098089609 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6fc1026d0 0x7fd6fc19e330 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8000c00 tx=0x7fd6e802c650 comp rx=0 tx=0).stop 2024-03-28T17:48:44.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.360+0000 7fd701e00640 1 -- 172.21.15.45:0/1098089609 shutdown_connections 2024-03-28T17:48:44.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.360+0000 7fd701e00640 1 --2- 172.21.15.45:0/1098089609 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fd6e40a2910 0x7fd6e40a4dd0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:44.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.360+0000 7fd701e00640 1 --2- 172.21.15.45:0/1098089609 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6fc105080 0x7fd6fc1a53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:44.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.360+0000 7fd701e00640 1 --2- 172.21.15.45:0/1098089609 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6fc1026d0 0x7fd6fc19e330 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:44.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.360+0000 7fd701e00640 1 --2- 172.21.15.45:0/1098089609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6fc0ffd60 0x7fd6fc19ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:44.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.360+0000 7fd701e00640 1 -- 172.21.15.45:0/1098089609 >> 172.21.15.45:0/1098089609 conn(0x7fd6fc0fb980 msgr2=0x7fd6fc100a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:44.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.360+0000 7fd701e00640 1 -- 172.21.15.45:0/1098089609 shutdown_connections 2024-03-28T17:48:44.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.360+0000 7fd701e00640 1 -- 172.21.15.45:0/1098089609 wait complete. 2024-03-28T17:48:44.381 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-28T17:48:44.381 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-28T17:48:44.382 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T17:48:44.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.583+0000 7f9cc8c00640 1 -- 172.21.15.45:0/3804710929 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9cc4104ad0 msgr2=0x7f9cc4104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:44.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.583+0000 7f9cc8c00640 1 --2- 172.21.15.45:0/3804710929 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9cc4104ad0 0x7f9cc4104f50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0009900 tx=0x7f9cb002bc30 comp rx=0 tx=0).stop 2024-03-28T17:48:44.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.583+0000 7f9cc8c00640 1 -- 172.21.15.45:0/3804710929 shutdown_connections 2024-03-28T17:48:44.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.583+0000 7f9cc8c00640 1 --2- 172.21.15.45:0/3804710929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cc4105490 0x7f9cc410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:44.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.583+0000 7f9cc8c00640 1 --2- 172.21.15.45:0/3804710929 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9cc4104ad0 0x7f9cc4104f50 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:44.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.583+0000 7f9cc8c00640 1 --2- 172.21.15.45:0/3804710929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cc41038e0 0x7f9cc4103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:44.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.583+0000 7f9cc8c00640 1 -- 172.21.15.45:0/3804710929 >> 172.21.15.45:0/3804710929 conn(0x7f9cc40fee50 msgr2=0x7f9cc4101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:44.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.584+0000 7f9cc8c00640 1 -- 172.21.15.45:0/3804710929 shutdown_connections 2024-03-28T17:48:44.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.584+0000 7f9cc8c00640 1 -- 172.21.15.45:0/3804710929 wait complete. 2024-03-28T17:48:44.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.585+0000 7f9cc8c00640 1 Processor -- start 2024-03-28T17:48:44.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.585+0000 7f9cc8c00640 1 -- start start 2024-03-28T17:48:44.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.585+0000 7f9cc8c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cc41038e0 0x7f9cc4199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:44.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.585+0000 7f9cc8c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cc4104ad0 0x7f9cc419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:44.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.585+0000 7f9cc8c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9cc4105490 0x7f9cc41a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:44.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.585+0000 7f9cc8c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc410f590 con 0x7f9cc4104ad0 2024-03-28T17:48:44.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.585+0000 7f9cc8c00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc410f410 con 0x7f9cc4105490 2024-03-28T17:48:44.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.585+0000 7f9cc8c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc410f710 con 0x7f9cc41038e0 2024-03-28T17:48:44.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.585+0000 7f9cc1e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cc41038e0 0x7f9cc4199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.585+0000 7f9cc1400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cc4104ad0 0x7f9cc419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.585+0000 7f9cc1e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cc41038e0 0x7f9cc4199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:49754/0 (socket says 172.21.15.45:49754) 2024-03-28T17:48:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.585+0000 7f9cc1e00640 1 -- 172.21.15.45:0/587739454 learned_addr learned my addr 172.21.15.45:0/587739454 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.585+0000 7f9cc2800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9cc4105490 0x7f9cc41a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.586+0000 7f9cc1e00640 1 -- 172.21.15.45:0/587739454 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9cc4105490 msgr2=0x7f9cc41a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.586+0000 7f9cc1e00640 1 --2- 172.21.15.45:0/587739454 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9cc4105490 0x7f9cc41a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.586+0000 7f9cc1e00640 1 -- 172.21.15.45:0/587739454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cc4104ad0 msgr2=0x7f9cc419a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.586+0000 7f9cc1e00640 1 --2- 172.21.15.45:0/587739454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cc4104ad0 0x7f9cc419a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.586+0000 7f9cc1e00640 1 -- 172.21.15.45:0/587739454 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc41a17f0 con 0x7f9cc41038e0 2024-03-28T17:48:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.586+0000 7f9cc1e00640 1 --2- 172.21.15.45:0/587739454 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cc41038e0 0x7f9cc4199bd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9cb400e9c0 tx=0x7f9cb400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.586+0000 7f9cbaa00640 1 -- 172.21.15.45:0/587739454 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb400cd80 con 0x7f9cc41038e0 2024-03-28T17:48:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.586+0000 7f9cbaa00640 1 -- 172.21.15.45:0/587739454 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9cb40035f0 con 0x7f9cc41038e0 2024-03-28T17:48:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.586+0000 7f9cc8c00640 1 -- 172.21.15.45:0/587739454 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc41a1ae0 con 0x7f9cc41038e0 2024-03-28T17:48:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.586+0000 7f9cbaa00640 1 -- 172.21.15.45:0/587739454 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb4010690 con 0x7f9cc41038e0 2024-03-28T17:48:44.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.587+0000 7f9cc8c00640 1 -- 172.21.15.45:0/587739454 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc41a2000 con 0x7f9cc41038e0 2024-03-28T17:48:44.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.588+0000 7f9cc8c00640 1 -- 172.21.15.45:0/587739454 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc410aba0 con 0x7f9cc41038e0 2024-03-28T17:48:44.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.590+0000 7f9cbaa00640 1 -- 172.21.15.45:0/587739454 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f9cb40037a0 con 0x7f9cc41038e0 2024-03-28T17:48:44.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.592+0000 7f9cbaa00640 1 --2- 172.21.15.45:0/587739454 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9ca80a2890 0x7f9ca80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:44.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.592+0000 7f9cbaa00640 1 -- 172.21.15.45:0/587739454 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9cb40cefd0 con 0x7f9cc41038e0 2024-03-28T17:48:44.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.592+0000 7f9cc1400640 1 --2- 172.21.15.45:0/587739454 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9ca80a2890 0x7f9ca80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:44.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.593+0000 7f9cc1400640 1 --2- 172.21.15.45:0/587739454 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9ca80a2890 0x7f9ca80a4d50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9cb002c240 tx=0x7f9cb002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:44.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.595+0000 7f9cbaa00640 1 -- 172.21.15.45:0/587739454 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9cb409bec0 con 0x7f9cc41038e0 2024-03-28T17:48:44.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:48:44.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:44 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3208382511' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T17:48:44.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:44.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:44.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:44 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1098089609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:48:44.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:44.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:44 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:44.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.803+0000 7f9cc8c00640 1 -- 172.21.15.45:0/587739454 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9cc4061ae0 con 0x7f9ca80a2890 2024-03-28T17:48:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:48:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:44 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3208382511' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T17:48:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:44 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1098089609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:48:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:44 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:44.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.828+0000 7f9cbaa00640 1 -- 172.21.15.45:0/587739454 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259043 (secure 0 0 0) 0x7f9cc4061ae0 con 0x7f9ca80a2890 2024-03-28T17:48:44.833 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:48:44.834 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-28T17:48:44.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.835+0000 7f9cc8c00640 1 -- 172.21.15.45:0/587739454 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9ca80a2890 msgr2=0x7f9ca80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:44.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.835+0000 7f9cc8c00640 1 --2- 172.21.15.45:0/587739454 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9ca80a2890 0x7f9ca80a4d50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9cb002c240 tx=0x7f9cb002df90 comp rx=0 tx=0).stop 2024-03-28T17:48:44.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.835+0000 7f9cc8c00640 1 -- 172.21.15.45:0/587739454 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cc41038e0 msgr2=0x7f9cc4199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:44.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.835+0000 7f9cc8c00640 1 --2- 172.21.15.45:0/587739454 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cc41038e0 0x7f9cc4199bd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9cb400e9c0 tx=0x7f9cb400ee80 comp rx=0 tx=0).stop 2024-03-28T17:48:44.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.835+0000 7f9cc8c00640 1 -- 172.21.15.45:0/587739454 shutdown_connections 2024-03-28T17:48:44.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.835+0000 7f9cc8c00640 1 --2- 172.21.15.45:0/587739454 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9ca80a2890 0x7f9ca80a4d50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:44.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.835+0000 7f9cc8c00640 1 --2- 172.21.15.45:0/587739454 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9cc4105490 0x7f9cc41a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:44.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.835+0000 7f9cc8c00640 1 --2- 172.21.15.45:0/587739454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cc4104ad0 0x7f9cc419a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:44.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.835+0000 7f9cc8c00640 1 --2- 172.21.15.45:0/587739454 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9cc41038e0 0x7f9cc4199bd0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:44.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.835+0000 7f9cc8c00640 1 -- 172.21.15.45:0/587739454 >> 172.21.15.45:0/587739454 conn(0x7f9cc40fee50 msgr2=0x7f9cc4107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:44.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.835+0000 7f9cc8c00640 1 -- 172.21.15.45:0/587739454 shutdown_connections 2024-03-28T17:48:44.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:44.835+0000 7f9cc8c00640 1 -- 172.21.15.45:0/587739454 wait complete. 2024-03-28T17:48:44.857 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-03-28T17:48:43.708770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664235,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":1032,"num_read_kb":648,"num_write":363,"num_write_kb":3073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180853,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":794,"ondisk_log_size":794,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":364876,"kb_used_data":11564,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576948,"statfs":{"total":1151940427776,"available":1151566794752,"internally_reserved":0,"allocated":11841536,"data_stored":7445635,"data_compressed":114219,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7837,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":78,"num_read_kb":50,"num_write":10,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004441"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683908+0000","last_change":"2024-03-28T17:46:12.783038+0000","last_active":"2024-03-28T17:47:37.683908+0000","last_peered":"2024-03-28T17:47:37.683908+0000","last_clean":"2024-03-28T17:47:37.683908+0000","last_became_active":"2024-03-28T17:46:12.782668+0000","last_became_peered":"2024-03-28T17:46:12.782668+0000","last_unstale":"2024-03-28T17:47:37.683908+0000","last_undegraded":"2024-03-28T17:47:37.683908+0000","last_fullsized":"2024-03-28T17:47:37.683908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:03.424959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683564+0000","last_change":"2024-03-28T17:46:12.781828+0000","last_active":"2024-03-28T17:47:37.683564+0000","last_peered":"2024-03-28T17:47:37.683564+0000","last_clean":"2024-03-28T17:47:37.683564+0000","last_became_active":"2024-03-28T17:46:12.781424+0000","last_became_peered":"2024-03-28T17:46:12.781424+0000","last_unstale":"2024-03-28T17:47:37.683564+0000","last_undegraded":"2024-03-28T17:47:37.683564+0000","last_fullsized":"2024-03-28T17:47:37.683564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:15:16.639473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683268+0000","last_change":"2024-03-28T17:46:12.785431+0000","last_active":"2024-03-28T17:47:37.683268+0000","last_peered":"2024-03-28T17:47:37.683268+0000","last_clean":"2024-03-28T17:47:37.683268+0000","last_became_active":"2024-03-28T17:46:12.784992+0000","last_became_peered":"2024-03-28T17:46:12.784992+0000","last_unstale":"2024-03-28T17:47:37.683268+0000","last_undegraded":"2024-03-28T17:47:37.683268+0000","last_fullsized":"2024-03-28T17:47:37.683268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:03:36.926613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.663175+0000","last_change":"2024-03-28T17:46:12.765274+0000","last_active":"2024-03-28T17:47:37.663175+0000","last_peered":"2024-03-28T17:47:37.663175+0000","last_clean":"2024-03-28T17:47:37.663175+0000","last_became_active":"2024-03-28T17:46:12.764890+0000","last_became_peered":"2024-03-28T17:46:12.764890+0000","last_unstale":"2024-03-28T17:47:37.663175+0000","last_undegraded":"2024-03-28T17:47:37.663175+0000","last_fullsized":"2024-03-28T17:47:37.663175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:58:37.149756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.670408+0000","last_change":"2024-03-28T17:46:12.782791+0000","last_active":"2024-03-28T17:47:37.670408+0000","last_peered":"2024-03-28T17:47:37.670408+0000","last_clean":"2024-03-28T17:47:37.670408+0000","last_became_active":"2024-03-28T17:46:12.782420+0000","last_became_peered":"2024-03-28T17:46:12.782420+0000","last_unstale":"2024-03-28T17:47:37.670408+0000","last_undegraded":"2024-03-28T17:47:37.670408+0000","last_fullsized":"2024-03-28T17:47:37.670408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:30.512863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.657353+0000","last_change":"2024-03-28T17:46:12.780713+0000","last_active":"2024-03-28T17:47:37.657353+0000","last_peered":"2024-03-28T17:47:37.657353+0000","last_clean":"2024-03-28T17:47:37.657353+0000","last_became_active":"2024-03-28T17:46:12.780305+0000","last_became_peered":"2024-03-28T17:46:12.780305+0000","last_unstale":"2024-03-28T17:47:37.657353+0000","last_undegraded":"2024-03-28T17:47:37.657353+0000","last_fullsized":"2024-03-28T17:47:37.657353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:08:09.840491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.653070+0000","last_change":"2024-03-28T17:46:12.778120+0000","last_active":"2024-03-28T17:47:37.653070+0000","last_peered":"2024-03-28T17:47:37.653070+0000","last_clean":"2024-03-28T17:47:37.653070+0000","last_became_active":"2024-03-28T17:46:12.777724+0000","last_became_peered":"2024-03-28T17:46:12.777724+0000","last_unstale":"2024-03-28T17:47:37.653070+0000","last_undegraded":"2024-03-28T17:47:37.653070+0000","last_fullsized":"2024-03-28T17:47:37.653070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:29:50.980366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687911+0000","last_change":"2024-03-28T17:46:12.793447+0000","last_active":"2024-03-28T17:47:37.687911+0000","last_peered":"2024-03-28T17:47:37.687911+0000","last_clean":"2024-03-28T17:47:37.687911+0000","last_became_active":"2024-03-28T17:46:12.793070+0000","last_became_peered":"2024-03-28T17:46:12.793070+0000","last_unstale":"2024-03-28T17:47:37.687911+0000","last_undegraded":"2024-03-28T17:47:37.687911+0000","last_fullsized":"2024-03-28T17:47:37.687911+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:01.364577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.668936+0000","last_change":"2024-03-28T17:46:12.763908+0000","last_active":"2024-03-28T17:47:37.668936+0000","last_peered":"2024-03-28T17:47:37.668936+0000","last_clean":"2024-03-28T17:47:37.668936+0000","last_became_active":"2024-03-28T17:46:12.763267+0000","last_became_peered":"2024-03-28T17:46:12.763267+0000","last_unstale":"2024-03-28T17:47:37.668936+0000","last_undegraded":"2024-03-28T17:47:37.668936+0000","last_fullsized":"2024-03-28T17:47:37.668936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:10.692974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.660790+0000","last_change":"2024-03-28T17:46:12.762648+0000","last_active":"2024-03-28T17:47:37.660790+0000","last_peered":"2024-03-28T17:47:37.660790+0000","last_clean":"2024-03-28T17:47:37.660790+0000","last_became_active":"2024-03-28T17:46:12.762300+0000","last_became_peered":"2024-03-28T17:46:12.762300+0000","last_unstale":"2024-03-28T17:47:37.660790+0000","last_undegraded":"2024-03-28T17:47:37.660790+0000","last_fullsized":"2024-03-28T17:47:37.660790+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:02.210828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.668940+0000","last_change":"2024-03-28T17:46:12.766482+0000","last_active":"2024-03-28T17:47:37.668940+0000","last_peered":"2024-03-28T17:47:37.668940+0000","last_clean":"2024-03-28T17:47:37.668940+0000","last_became_active":"2024-03-28T17:46:12.766155+0000","last_became_peered":"2024-03-28T17:46:12.766155+0000","last_unstale":"2024-03-28T17:47:37.668940+0000","last_undegraded":"2024-03-28T17:47:37.668940+0000","last_fullsized":"2024-03-28T17:47:37.668940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:46:24.261744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.643025+0000","last_change":"2024-03-28T17:46:12.766100+0000","last_active":"2024-03-28T17:47:37.643025+0000","last_peered":"2024-03-28T17:47:37.643025+0000","last_clean":"2024-03-28T17:47:37.643025+0000","last_became_active":"2024-03-28T17:46:12.765762+0000","last_became_peered":"2024-03-28T17:46:12.765762+0000","last_unstale":"2024-03-28T17:47:37.643025+0000","last_undegraded":"2024-03-28T17:47:37.643025+0000","last_fullsized":"2024-03-28T17:47:37.643025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:11.666653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.670019+0000","last_change":"2024-03-28T17:46:12.753465+0000","last_active":"2024-03-28T17:47:37.670019+0000","last_peered":"2024-03-28T17:47:37.670019+0000","last_clean":"2024-03-28T17:47:37.670019+0000","last_became_active":"2024-03-28T17:46:12.751931+0000","last_became_peered":"2024-03-28T17:46:12.751931+0000","last_unstale":"2024-03-28T17:47:37.670019+0000","last_undegraded":"2024-03-28T17:47:37.670019+0000","last_fullsized":"2024-03-28T17:47:37.670019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:39:38.757878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.684532+0000","last_change":"2024-03-28T17:46:12.784177+0000","last_active":"2024-03-28T17:47:37.684532+0000","last_peered":"2024-03-28T17:47:37.684532+0000","last_clean":"2024-03-28T17:47:37.684532+0000","last_became_active":"2024-03-28T17:46:12.783823+0000","last_became_peered":"2024-03-28T17:46:12.783823+0000","last_unstale":"2024-03-28T17:47:37.684532+0000","last_undegraded":"2024-03-28T17:47:37.684532+0000","last_fullsized":"2024-03-28T17:47:37.684532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:11:33.221095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685187+0000","last_change":"2024-03-28T17:46:12.777152+0000","last_active":"2024-03-28T17:47:37.685187+0000","last_peered":"2024-03-28T17:47:37.685187+0000","last_clean":"2024-03-28T17:47:37.685187+0000","last_became_active":"2024-03-28T17:46:12.776758+0000","last_became_peered":"2024-03-28T17:46:12.776758+0000","last_unstale":"2024-03-28T17:47:37.685187+0000","last_undegraded":"2024-03-28T17:47:37.685187+0000","last_fullsized":"2024-03-28T17:47:37.685187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:20:56.587470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.686334+0000","last_change":"2024-03-28T17:46:12.782384+0000","last_active":"2024-03-28T17:47:37.686334+0000","last_peered":"2024-03-28T17:47:37.686334+0000","last_clean":"2024-03-28T17:47:37.686334+0000","last_became_active":"2024-03-28T17:46:12.781971+0000","last_became_peered":"2024-03-28T17:46:12.781971+0000","last_unstale":"2024-03-28T17:47:37.686334+0000","last_undegraded":"2024-03-28T17:47:37.686334+0000","last_fullsized":"2024-03-28T17:47:37.686334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:19:32.652629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685920+0000","last_change":"2024-03-28T17:46:12.789364+0000","last_active":"2024-03-28T17:47:37.685920+0000","last_peered":"2024-03-28T17:47:37.685920+0000","last_clean":"2024-03-28T17:47:37.685920+0000","last_became_active":"2024-03-28T17:46:12.788976+0000","last_became_peered":"2024-03-28T17:46:12.788976+0000","last_unstale":"2024-03-28T17:47:37.685920+0000","last_undegraded":"2024-03-28T17:47:37.685920+0000","last_fullsized":"2024-03-28T17:47:37.685920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:43:37.183705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.663596+0000","last_change":"2024-03-28T17:46:12.782664+0000","last_active":"2024-03-28T17:47:37.663596+0000","last_peered":"2024-03-28T17:47:37.663596+0000","last_clean":"2024-03-28T17:47:37.663596+0000","last_became_active":"2024-03-28T17:46:12.782305+0000","last_became_peered":"2024-03-28T17:46:12.782305+0000","last_unstale":"2024-03-28T17:47:37.663596+0000","last_undegraded":"2024-03-28T17:47:37.663596+0000","last_fullsized":"2024-03-28T17:47:37.663596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:47:18.373517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.684368+0000","last_change":"2024-03-28T17:46:12.794494+0000","last_active":"2024-03-28T17:47:37.684368+0000","last_peered":"2024-03-28T17:47:37.684368+0000","last_clean":"2024-03-28T17:47:37.684368+0000","last_became_active":"2024-03-28T17:46:12.794151+0000","last_became_peered":"2024-03-28T17:46:12.794151+0000","last_unstale":"2024-03-28T17:47:37.684368+0000","last_undegraded":"2024-03-28T17:47:37.684368+0000","last_fullsized":"2024-03-28T17:47:37.684368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:17:24.596333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.475806+0000","last_change":"2024-03-28T17:46:11.759643+0000","last_active":"2024-03-28T17:47:09.475806+0000","last_peered":"2024-03-28T17:47:09.475806+0000","last_clean":"2024-03-28T17:47:09.475806+0000","last_became_active":"2024-03-28T17:46:11.759273+0000","last_became_peered":"2024-03-28T17:46:11.759273+0000","last_unstale":"2024-03-28T17:47:09.475806+0000","last_undegraded":"2024-03-28T17:47:09.475806+0000","last_fullsized":"2024-03-28T17:47:09.475806+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:09:51.764471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687168+0000","last_change":"2024-03-28T17:46:12.786503+0000","last_active":"2024-03-28T17:47:37.687168+0000","last_peered":"2024-03-28T17:47:37.687168+0000","last_clean":"2024-03-28T17:47:37.687168+0000","last_became_active":"2024-03-28T17:46:12.784651+0000","last_became_peered":"2024-03-28T17:46:12.784651+0000","last_unstale":"2024-03-28T17:47:37.687168+0000","last_undegraded":"2024-03-28T17:47:37.687168+0000","last_fullsized":"2024-03-28T17:47:37.687168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:30:22.401767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.480732+0000","last_change":"2024-03-28T17:46:11.780340+0000","last_active":"2024-03-28T17:47:09.480732+0000","last_peered":"2024-03-28T17:47:09.480732+0000","last_clean":"2024-03-28T17:47:09.480732+0000","last_became_active":"2024-03-28T17:46:11.779917+0000","last_became_peered":"2024-03-28T17:46:11.779917+0000","last_unstale":"2024-03-28T17:47:09.480732+0000","last_undegraded":"2024-03-28T17:47:09.480732+0000","last_fullsized":"2024-03-28T17:47:09.480732+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:22:45.617444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.686129+0000","last_change":"2024-03-28T17:46:12.795086+0000","last_active":"2024-03-28T17:47:37.686129+0000","last_peered":"2024-03-28T17:47:37.686129+0000","last_clean":"2024-03-28T17:47:37.686129+0000","last_became_active":"2024-03-28T17:46:12.794124+0000","last_became_peered":"2024-03-28T17:46:12.794124+0000","last_unstale":"2024-03-28T17:47:37.686129+0000","last_undegraded":"2024-03-28T17:47:37.686129+0000","last_fullsized":"2024-03-28T17:47:37.686129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:11:05.347047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.648328+0000","last_change":"2024-03-28T17:47:09.648328+0000","last_active":"2024-03-28T17:47:09.648328+0000","last_peered":"2024-03-28T17:47:09.648328+0000","last_clean":"2024-03-28T17:47:09.648328+0000","last_became_active":"2024-03-28T17:47:09.644797+0000","last_became_peered":"2024-03-28T17:47:09.644797+0000","last_unstale":"2024-03-28T17:47:09.648328+0000","last_undegraded":"2024-03-28T17:47:09.648328+0000","last_fullsized":"2024-03-28T17:47:09.648328+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:14:30.233156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.665594+0000","last_change":"2024-03-28T17:46:12.768118+0000","last_active":"2024-03-28T17:47:37.665594+0000","last_peered":"2024-03-28T17:47:37.665594+0000","last_clean":"2024-03-28T17:47:37.665594+0000","last_became_active":"2024-03-28T17:46:12.767765+0000","last_became_peered":"2024-03-28T17:46:12.767765+0000","last_unstale":"2024-03-28T17:47:37.665594+0000","last_undegraded":"2024-03-28T17:47:37.665594+0000","last_fullsized":"2024-03-28T17:47:37.665594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:16:12.268968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461621+0000","last_change":"2024-03-28T17:46:11.760204+0000","last_active":"2024-03-28T17:47:09.461621+0000","last_peered":"2024-03-28T17:47:09.461621+0000","last_clean":"2024-03-28T17:47:09.461621+0000","last_became_active":"2024-03-28T17:46:11.759829+0000","last_became_peered":"2024-03-28T17:46:11.759829+0000","last_unstale":"2024-03-28T17:47:09.461621+0000","last_undegraded":"2024-03-28T17:47:09.461621+0000","last_fullsized":"2024-03-28T17:47:09.461621+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:29:02.075424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.663263+0000","last_change":"2024-03-28T17:46:12.762229+0000","last_active":"2024-03-28T17:47:37.663263+0000","last_peered":"2024-03-28T17:47:37.663263+0000","last_clean":"2024-03-28T17:47:37.663263+0000","last_became_active":"2024-03-28T17:46:12.761889+0000","last_became_peered":"2024-03-28T17:46:12.761889+0000","last_unstale":"2024-03-28T17:47:37.663263+0000","last_undegraded":"2024-03-28T17:47:37.663263+0000","last_fullsized":"2024-03-28T17:47:37.663263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:15.023966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'8","reported_seq":173,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:48:38.124927+0000","last_change":"2024-03-28T17:46:11.763152+0000","last_active":"2024-03-28T17:48:38.124927+0000","last_peered":"2024-03-28T17:48:38.124927+0000","last_clean":"2024-03-28T17:48:38.124927+0000","last_became_active":"2024-03-28T17:46:11.762754+0000","last_became_peered":"2024-03-28T17:46:11.762754+0000","last_unstale":"2024-03-28T17:48:38.124927+0000","last_undegraded":"2024-03-28T17:48:38.124927+0000","last_fullsized":"2024-03-28T17:48:38.124927+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:17:06.659908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1460,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.649470+0000","last_change":"2024-03-28T17:46:12.780701+0000","last_active":"2024-03-28T17:47:37.649470+0000","last_peered":"2024-03-28T17:47:37.649470+0000","last_clean":"2024-03-28T17:47:37.649470+0000","last_became_active":"2024-03-28T17:46:12.780339+0000","last_became_peered":"2024-03-28T17:46:12.780339+0000","last_unstale":"2024-03-28T17:47:37.649470+0000","last_undegraded":"2024-03-28T17:47:37.649470+0000","last_fullsized":"2024-03-28T17:47:37.649470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:59.774888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.950734+0000","last_change":"2024-03-28T17:46:11.763387+0000","last_active":"2024-03-28T17:47:09.950734+0000","last_peered":"2024-03-28T17:47:09.950734+0000","last_clean":"2024-03-28T17:47:09.950734+0000","last_became_active":"2024-03-28T17:46:11.763028+0000","last_became_peered":"2024-03-28T17:46:11.763028+0000","last_unstale":"2024-03-28T17:47:09.950734+0000","last_undegraded":"2024-03-28T17:47:09.950734+0000","last_fullsized":"2024-03-28T17:47:09.950734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:21:31.585047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685166+0000","last_change":"2024-03-28T17:46:12.794167+0000","last_active":"2024-03-28T17:47:37.685166+0000","last_peered":"2024-03-28T17:47:37.685166+0000","last_clean":"2024-03-28T17:47:37.685166+0000","last_became_active":"2024-03-28T17:46:12.793830+0000","last_became_peered":"2024-03-28T17:46:12.793830+0000","last_unstale":"2024-03-28T17:47:37.685166+0000","last_undegraded":"2024-03-28T17:47:37.685166+0000","last_fullsized":"2024-03-28T17:47:37.685166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:20.907752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.470227+0000","last_change":"2024-03-28T17:46:11.769322+0000","last_active":"2024-03-28T17:47:09.470227+0000","last_peered":"2024-03-28T17:47:09.470227+0000","last_clean":"2024-03-28T17:47:09.470227+0000","last_became_active":"2024-03-28T17:46:11.768931+0000","last_became_peered":"2024-03-28T17:46:11.768931+0000","last_unstale":"2024-03-28T17:47:09.470227+0000","last_undegraded":"2024-03-28T17:47:09.470227+0000","last_fullsized":"2024-03-28T17:47:09.470227+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:29:34.372947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683503+0000","last_change":"2024-03-28T17:46:12.781306+0000","last_active":"2024-03-28T17:47:37.683503+0000","last_peered":"2024-03-28T17:47:37.683503+0000","last_clean":"2024-03-28T17:47:37.683503+0000","last_became_active":"2024-03-28T17:46:12.779161+0000","last_became_peered":"2024-03-28T17:46:12.779161+0000","last_unstale":"2024-03-28T17:47:37.683503+0000","last_undegraded":"2024-03-28T17:47:37.683503+0000","last_fullsized":"2024-03-28T17:47:37.683503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:06:10.302067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.464760+0000","last_change":"2024-03-28T17:46:11.762852+0000","last_active":"2024-03-28T17:47:09.464760+0000","last_peered":"2024-03-28T17:47:09.464760+0000","last_clean":"2024-03-28T17:47:09.464760+0000","last_became_active":"2024-03-28T17:46:11.762428+0000","last_became_peered":"2024-03-28T17:46:11.762428+0000","last_unstale":"2024-03-28T17:47:09.464760+0000","last_undegraded":"2024-03-28T17:47:09.464760+0000","last_fullsized":"2024-03-28T17:47:09.464760+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:38:20.433408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.686726+0000","last_change":"2024-03-28T17:46:12.786295+0000","last_active":"2024-03-28T17:47:37.686726+0000","last_peered":"2024-03-28T17:47:37.686726+0000","last_clean":"2024-03-28T17:47:37.686726+0000","last_became_active":"2024-03-28T17:46:12.785894+0000","last_became_peered":"2024-03-28T17:46:12.785894+0000","last_unstale":"2024-03-28T17:47:37.686726+0000","last_undegraded":"2024-03-28T17:47:37.686726+0000","last_fullsized":"2024-03-28T17:47:37.686726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:06.271660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'5","reported_seq":170,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:48:41.128735+0000","last_change":"2024-03-28T17:46:11.766787+0000","last_active":"2024-03-28T17:48:41.128735+0000","last_peered":"2024-03-28T17:48:41.128735+0000","last_clean":"2024-03-28T17:48:41.128735+0000","last_became_active":"2024-03-28T17:46:11.766352+0000","last_became_peered":"2024-03-28T17:46:11.766352+0000","last_unstale":"2024-03-28T17:48:41.128735+0000","last_undegraded":"2024-03-28T17:48:41.128735+0000","last_fullsized":"2024-03-28T17:48:41.128735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:54:48.157597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":145,"num_read_kb":145,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.666318+0000","last_change":"2024-03-28T17:46:12.785724+0000","last_active":"2024-03-28T17:47:37.666318+0000","last_peered":"2024-03-28T17:47:37.666318+0000","last_clean":"2024-03-28T17:47:37.666318+0000","last_became_active":"2024-03-28T17:46:12.785353+0000","last_became_peered":"2024-03-28T17:46:12.785353+0000","last_unstale":"2024-03-28T17:47:37.666318+0000","last_undegraded":"2024-03-28T17:47:37.666318+0000","last_fullsized":"2024-03-28T17:47:37.666318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:21:22.692899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.463127+0000","last_change":"2024-03-28T17:46:11.763507+0000","last_active":"2024-03-28T17:47:09.463127+0000","last_peered":"2024-03-28T17:47:09.463127+0000","last_clean":"2024-03-28T17:47:09.463127+0000","last_became_active":"2024-03-28T17:46:11.763177+0000","last_became_peered":"2024-03-28T17:46:11.763177+0000","last_unstale":"2024-03-28T17:47:09.463127+0000","last_undegraded":"2024-03-28T17:47:09.463127+0000","last_fullsized":"2024-03-28T17:47:09.463127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:56:39.696699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.686049+0000","last_change":"2024-03-28T17:47:09.479162+0000","last_active":"2024-03-28T17:47:37.686049+0000","last_peered":"2024-03-28T17:47:37.686049+0000","last_clean":"2024-03-28T17:47:37.686049+0000","last_became_active":"2024-03-28T17:47:09.478349+0000","last_became_peered":"2024-03-28T17:47:09.478349+0000","last_unstale":"2024-03-28T17:47:37.686049+0000","last_undegraded":"2024-03-28T17:47:37.686049+0000","last_fullsized":"2024-03-28T17:47:37.686049+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:41.056361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.465186+0000","last_change":"2024-03-28T17:46:11.774961+0000","last_active":"2024-03-28T17:47:09.465186+0000","last_peered":"2024-03-28T17:47:09.465186+0000","last_clean":"2024-03-28T17:47:09.465186+0000","last_became_active":"2024-03-28T17:46:11.774645+0000","last_became_peered":"2024-03-28T17:46:11.774645+0000","last_unstale":"2024-03-28T17:47:09.465186+0000","last_undegraded":"2024-03-28T17:47:09.465186+0000","last_fullsized":"2024-03-28T17:47:09.465186+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:38:23.983739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.667529+0000","last_change":"2024-03-28T17:46:12.763698+0000","last_active":"2024-03-28T17:47:37.667529+0000","last_peered":"2024-03-28T17:47:37.667529+0000","last_clean":"2024-03-28T17:47:37.667529+0000","last_became_active":"2024-03-28T17:46:12.763238+0000","last_became_peered":"2024-03-28T17:46:12.763238+0000","last_unstale":"2024-03-28T17:47:37.667529+0000","last_undegraded":"2024-03-28T17:47:37.667529+0000","last_fullsized":"2024-03-28T17:47:37.667529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:12:58.252855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'37","reported_seq":360,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:48:42.130390+0000","last_change":"2024-03-28T17:46:11.763096+0000","last_active":"2024-03-28T17:48:42.130390+0000","last_peered":"2024-03-28T17:48:42.130390+0000","last_clean":"2024-03-28T17:48:42.130390+0000","last_became_active":"2024-03-28T17:46:11.762730+0000","last_became_peered":"2024-03-28T17:46:11.762730+0000","last_unstale":"2024-03-28T17:48:42.130390+0000","last_undegraded":"2024-03-28T17:48:42.130390+0000","last_fullsized":"2024-03-28T17:48:42.130390+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:45:59.469096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68653,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":303,"num_read_kb":219,"num_write":38,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.651631+0000","last_change":"2024-03-28T17:46:12.760366+0000","last_active":"2024-03-28T17:47:37.651631+0000","last_peered":"2024-03-28T17:47:37.651631+0000","last_clean":"2024-03-28T17:47:37.651631+0000","last_became_active":"2024-03-28T17:46:12.759957+0000","last_became_peered":"2024-03-28T17:46:12.759957+0000","last_unstale":"2024-03-28T17:47:37.651631+0000","last_undegraded":"2024-03-28T17:47:37.651631+0000","last_fullsized":"2024-03-28T17:47:37.651631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:08:02.579659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.463545+0000","last_change":"2024-03-28T17:46:11.787108+0000","last_active":"2024-03-28T17:47:09.463545+0000","last_peered":"2024-03-28T17:47:09.463545+0000","last_clean":"2024-03-28T17:47:09.463545+0000","last_became_active":"2024-03-28T17:46:11.786734+0000","last_became_peered":"2024-03-28T17:46:11.786734+0000","last_unstale":"2024-03-28T17:47:09.463545+0000","last_undegraded":"2024-03-28T17:47:09.463545+0000","last_fullsized":"2024-03-28T17:47:09.463545+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:04:00.480545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.678776+0000","last_change":"2024-03-28T17:46:12.786362+0000","last_active":"2024-03-28T17:47:37.678776+0000","last_peered":"2024-03-28T17:47:37.678776+0000","last_clean":"2024-03-28T17:47:37.678776+0000","last_became_active":"2024-03-28T17:46:12.785946+0000","last_became_peered":"2024-03-28T17:46:12.785946+0000","last_unstale":"2024-03-28T17:47:37.678776+0000","last_undegraded":"2024-03-28T17:47:37.678776+0000","last_fullsized":"2024-03-28T17:47:37.678776+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:13:57.857818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.464899+0000","last_change":"2024-03-28T17:46:11.786292+0000","last_active":"2024-03-28T17:47:09.464899+0000","last_peered":"2024-03-28T17:47:09.464899+0000","last_clean":"2024-03-28T17:47:09.464899+0000","last_became_active":"2024-03-28T17:46:11.785953+0000","last_became_peered":"2024-03-28T17:46:11.785953+0000","last_unstale":"2024-03-28T17:47:09.464899+0000","last_undegraded":"2024-03-28T17:47:09.464899+0000","last_fullsized":"2024-03-28T17:47:09.464899+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:02:18.093585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.669899+0000","last_change":"2024-03-28T17:46:12.770669+0000","last_active":"2024-03-28T17:47:37.669899+0000","last_peered":"2024-03-28T17:47:37.669899+0000","last_clean":"2024-03-28T17:47:37.669899+0000","last_became_active":"2024-03-28T17:46:12.770301+0000","last_became_peered":"2024-03-28T17:46:12.770301+0000","last_unstale":"2024-03-28T17:47:37.669899+0000","last_undegraded":"2024-03-28T17:47:37.669899+0000","last_fullsized":"2024-03-28T17:47:37.669899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:59:00.601357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'5","reported_seq":171,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:48:38.226965+0000","last_change":"2024-03-28T17:46:11.759686+0000","last_active":"2024-03-28T17:48:38.226965+0000","last_peered":"2024-03-28T17:48:38.226965+0000","last_clean":"2024-03-28T17:48:38.226965+0000","last_became_active":"2024-03-28T17:46:11.759309+0000","last_became_peered":"2024-03-28T17:46:11.759309+0000","last_unstale":"2024-03-28T17:48:38.226965+0000","last_undegraded":"2024-03-28T17:48:38.226965+0000","last_fullsized":"2024-03-28T17:48:38.226965+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:31:09.999279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.653109+0000","last_change":"2024-03-28T17:46:12.760484+0000","last_active":"2024-03-28T17:47:37.653109+0000","last_peered":"2024-03-28T17:47:37.653109+0000","last_clean":"2024-03-28T17:47:37.653109+0000","last_became_active":"2024-03-28T17:46:12.760121+0000","last_became_peered":"2024-03-28T17:46:12.760121+0000","last_unstale":"2024-03-28T17:47:37.653109+0000","last_undegraded":"2024-03-28T17:47:37.653109+0000","last_fullsized":"2024-03-28T17:47:37.653109+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:33:51.457426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461309+0000","last_change":"2024-03-28T17:46:11.769133+0000","last_active":"2024-03-28T17:47:09.461309+0000","last_peered":"2024-03-28T17:47:09.461309+0000","last_clean":"2024-03-28T17:47:09.461309+0000","last_became_active":"2024-03-28T17:46:11.768857+0000","last_became_peered":"2024-03-28T17:46:11.768857+0000","last_unstale":"2024-03-28T17:47:09.461309+0000","last_undegraded":"2024-03-28T17:47:09.461309+0000","last_fullsized":"2024-03-28T17:47:09.461309+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:19:26.065368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.668453+0000","last_change":"2024-03-28T17:46:12.765246+0000","last_active":"2024-03-28T17:47:37.668453+0000","last_peered":"2024-03-28T17:47:37.668453+0000","last_clean":"2024-03-28T17:47:37.668453+0000","last_became_active":"2024-03-28T17:46:12.764788+0000","last_became_peered":"2024-03-28T17:46:12.764788+0000","last_unstale":"2024-03-28T17:47:37.668453+0000","last_undegraded":"2024-03-28T17:47:37.668453+0000","last_fullsized":"2024-03-28T17:47:37.668453+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:36:22.864337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461779+0000","last_change":"2024-03-28T17:46:11.761005+0000","last_active":"2024-03-28T17:47:09.461779+0000","last_peered":"2024-03-28T17:47:09.461779+0000","last_clean":"2024-03-28T17:47:09.461779+0000","last_became_active":"2024-03-28T17:46:11.760660+0000","last_became_peered":"2024-03-28T17:46:11.760660+0000","last_unstale":"2024-03-28T17:47:09.461779+0000","last_undegraded":"2024-03-28T17:47:09.461779+0000","last_fullsized":"2024-03-28T17:47:09.461779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:01:10.408853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.644551+0000","last_change":"2024-03-28T17:46:12.762851+0000","last_active":"2024-03-28T17:47:37.644551+0000","last_peered":"2024-03-28T17:47:37.644551+0000","last_clean":"2024-03-28T17:47:37.644551+0000","last_became_active":"2024-03-28T17:46:12.762508+0000","last_became_peered":"2024-03-28T17:46:12.762508+0000","last_unstale":"2024-03-28T17:47:37.644551+0000","last_undegraded":"2024-03-28T17:47:37.644551+0000","last_fullsized":"2024-03-28T17:47:37.644551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:55:29.389985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.955449+0000","last_change":"2024-03-28T17:47:09.953352+0000","last_active":"2024-03-28T17:47:09.955449+0000","last_peered":"2024-03-28T17:47:09.955449+0000","last_clean":"2024-03-28T17:47:09.955449+0000","last_became_active":"2024-03-28T17:47:09.952822+0000","last_became_peered":"2024-03-28T17:47:09.952822+0000","last_unstale":"2024-03-28T17:47:09.955449+0000","last_undegraded":"2024-03-28T17:47:09.955449+0000","last_fullsized":"2024-03-28T17:47:09.955449+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:22:48.635040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.684406+0000","last_change":"2024-03-28T17:46:12.794290+0000","last_active":"2024-03-28T17:47:37.684406+0000","last_peered":"2024-03-28T17:47:37.684406+0000","last_clean":"2024-03-28T17:47:37.684406+0000","last_became_active":"2024-03-28T17:46:12.793906+0000","last_became_peered":"2024-03-28T17:46:12.793906+0000","last_unstale":"2024-03-28T17:47:37.684406+0000","last_undegraded":"2024-03-28T17:47:37.684406+0000","last_fullsized":"2024-03-28T17:47:37.684406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:26:04.996019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.463067+0000","last_change":"2024-03-28T17:46:11.762676+0000","last_active":"2024-03-28T17:47:09.463067+0000","last_peered":"2024-03-28T17:47:09.463067+0000","last_clean":"2024-03-28T17:47:09.463067+0000","last_became_active":"2024-03-28T17:46:11.761458+0000","last_became_peered":"2024-03-28T17:46:11.761458+0000","last_unstale":"2024-03-28T17:47:09.463067+0000","last_undegraded":"2024-03-28T17:47:09.463067+0000","last_fullsized":"2024-03-28T17:47:09.463067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:22:26.642798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.686879+0000","last_change":"2024-03-28T17:46:12.787107+0000","last_active":"2024-03-28T17:47:37.686879+0000","last_peered":"2024-03-28T17:47:37.686879+0000","last_clean":"2024-03-28T17:47:37.686879+0000","last_became_active":"2024-03-28T17:46:12.786610+0000","last_became_peered":"2024-03-28T17:46:12.786610+0000","last_unstale":"2024-03-28T17:47:37.686879+0000","last_undegraded":"2024-03-28T17:47:37.686879+0000","last_fullsized":"2024-03-28T17:47:37.686879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:30:14.880483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.665610+0000","last_change":"2024-03-28T17:47:09.645023+0000","last_active":"2024-03-28T17:47:37.665610+0000","last_peered":"2024-03-28T17:47:37.665610+0000","last_clean":"2024-03-28T17:47:37.665610+0000","last_became_active":"2024-03-28T17:47:09.644507+0000","last_became_peered":"2024-03-28T17:47:09.644507+0000","last_unstale":"2024-03-28T17:47:37.665610+0000","last_undegraded":"2024-03-28T17:47:37.665610+0000","last_fullsized":"2024-03-28T17:47:37.665610+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:00:48.144635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"87'5","reported_seq":167,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:48:38.226417+0000","last_change":"2024-03-28T17:46:11.784261+0000","last_active":"2024-03-28T17:48:38.226417+0000","last_peered":"2024-03-28T17:48:38.226417+0000","last_clean":"2024-03-28T17:48:38.226417+0000","last_became_active":"2024-03-28T17:46:11.782957+0000","last_became_peered":"2024-03-28T17:46:11.782957+0000","last_unstale":"2024-03-28T17:48:38.226417+0000","last_undegraded":"2024-03-28T17:48:38.226417+0000","last_fullsized":"2024-03-28T17:48:38.226417+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:08:48.009774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":142,"num_read_kb":142,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687570+0000","last_change":"2024-03-28T17:46:12.784024+0000","last_active":"2024-03-28T17:47:37.687570+0000","last_peered":"2024-03-28T17:47:37.687570+0000","last_clean":"2024-03-28T17:47:37.687570+0000","last_became_active":"2024-03-28T17:46:12.782157+0000","last_became_peered":"2024-03-28T17:46:12.782157+0000","last_unstale":"2024-03-28T17:47:37.687570+0000","last_undegraded":"2024-03-28T17:47:37.687570+0000","last_fullsized":"2024-03-28T17:47:37.687570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:33.679894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.644104+0000","last_change":"2024-03-28T17:46:11.759602+0000","last_active":"2024-03-28T17:47:09.644104+0000","last_peered":"2024-03-28T17:47:09.644104+0000","last_clean":"2024-03-28T17:47:09.644104+0000","last_became_active":"2024-03-28T17:46:11.759248+0000","last_became_peered":"2024-03-28T17:46:11.759248+0000","last_unstale":"2024-03-28T17:47:09.644104+0000","last_undegraded":"2024-03-28T17:47:09.644104+0000","last_fullsized":"2024-03-28T17:47:09.644104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:15:59.997948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":352,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.462874+0000","last_change":"2024-03-28T17:44:51.172493+0000","last_active":"2024-03-28T17:47:09.462874+0000","last_peered":"2024-03-28T17:47:09.462874+0000","last_clean":"2024-03-28T17:47:09.462874+0000","last_became_active":"2024-03-28T17:44:51.172175+0000","last_became_peered":"2024-03-28T17:44:51.172175+0000","last_unstale":"2024-03-28T17:47:09.462874+0000","last_undegraded":"2024-03-28T17:47:09.462874+0000","last_fullsized":"2024-03-28T17:47:09.462874+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:40:54.580958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:40:54.580958+0000","last_clean_scrub_stamp":"2024-03-28T17:40:54.580958+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:34.942490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.669498+0000","last_change":"2024-03-28T17:46:12.766728+0000","last_active":"2024-03-28T17:47:37.669498+0000","last_peered":"2024-03-28T17:47:37.669498+0000","last_clean":"2024-03-28T17:47:37.669498+0000","last_became_active":"2024-03-28T17:46:12.763671+0000","last_became_peered":"2024-03-28T17:46:12.763671+0000","last_unstale":"2024-03-28T17:47:37.669498+0000","last_undegraded":"2024-03-28T17:47:37.669498+0000","last_fullsized":"2024-03-28T17:47:37.669498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:25:39.802551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.462989+0000","last_change":"2024-03-28T17:46:11.763361+0000","last_active":"2024-03-28T17:47:09.462989+0000","last_peered":"2024-03-28T17:47:09.462989+0000","last_clean":"2024-03-28T17:47:09.462989+0000","last_became_active":"2024-03-28T17:46:11.763049+0000","last_became_peered":"2024-03-28T17:46:11.763049+0000","last_unstale":"2024-03-28T17:47:09.462989+0000","last_undegraded":"2024-03-28T17:47:09.462989+0000","last_fullsized":"2024-03-28T17:47:09.462989+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:01.767964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461664+0000","last_change":"2024-03-28T17:46:11.774957+0000","last_active":"2024-03-28T17:47:09.461664+0000","last_peered":"2024-03-28T17:47:09.461664+0000","last_clean":"2024-03-28T17:47:09.461664+0000","last_became_active":"2024-03-28T17:46:11.774658+0000","last_became_peered":"2024-03-28T17:46:11.774658+0000","last_unstale":"2024-03-28T17:47:09.461664+0000","last_undegraded":"2024-03-28T17:47:09.461664+0000","last_fullsized":"2024-03-28T17:47:09.461664+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:40.254318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.675859+0000","last_change":"2024-03-28T17:46:12.778206+0000","last_active":"2024-03-28T17:47:37.675859+0000","last_peered":"2024-03-28T17:47:37.675859+0000","last_clean":"2024-03-28T17:47:37.675859+0000","last_became_active":"2024-03-28T17:46:12.777780+0000","last_became_peered":"2024-03-28T17:46:12.777780+0000","last_unstale":"2024-03-28T17:47:37.675859+0000","last_undegraded":"2024-03-28T17:47:37.675859+0000","last_fullsized":"2024-03-28T17:47:37.675859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:13.241794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'11","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:48:25.066011+0000","last_change":"2024-03-28T17:47:09.961000+0000","last_active":"2024-03-28T17:48:25.066011+0000","last_peered":"2024-03-28T17:48:25.066011+0000","last_clean":"2024-03-28T17:48:25.066011+0000","last_became_active":"2024-03-28T17:47:09.953155+0000","last_became_peered":"2024-03-28T17:47:09.953155+0000","last_unstale":"2024-03-28T17:48:25.066011+0000","last_undegraded":"2024-03-28T17:48:25.066011+0000","last_fullsized":"2024-03-28T17:48:25.066011+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:31:57.258152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683178+0000","last_change":"2024-03-28T17:46:12.790489+0000","last_active":"2024-03-28T17:47:37.683178+0000","last_peered":"2024-03-28T17:47:37.683178+0000","last_clean":"2024-03-28T17:47:37.683178+0000","last_became_active":"2024-03-28T17:46:12.790108+0000","last_became_peered":"2024-03-28T17:46:12.790108+0000","last_unstale":"2024-03-28T17:47:37.683178+0000","last_undegraded":"2024-03-28T17:47:37.683178+0000","last_fullsized":"2024-03-28T17:47:37.683178+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:16:21.210182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.644215+0000","last_change":"2024-03-28T17:46:11.762658+0000","last_active":"2024-03-28T17:47:09.644215+0000","last_peered":"2024-03-28T17:47:09.644215+0000","last_clean":"2024-03-28T17:47:09.644215+0000","last_became_active":"2024-03-28T17:46:11.762296+0000","last_became_peered":"2024-03-28T17:46:11.762296+0000","last_unstale":"2024-03-28T17:47:09.644215+0000","last_undegraded":"2024-03-28T17:47:09.644215+0000","last_fullsized":"2024-03-28T17:47:09.644215+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:42:52.768285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.686169+0000","last_change":"2024-03-28T17:47:09.488306+0000","last_active":"2024-03-28T17:47:37.686169+0000","last_peered":"2024-03-28T17:47:37.686169+0000","last_clean":"2024-03-28T17:47:37.686169+0000","last_became_active":"2024-03-28T17:47:09.487292+0000","last_became_peered":"2024-03-28T17:47:09.487292+0000","last_unstale":"2024-03-28T17:47:37.686169+0000","last_undegraded":"2024-03-28T17:47:37.686169+0000","last_fullsized":"2024-03-28T17:47:37.686169+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:55.612951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.951519+0000","last_change":"2024-03-28T17:46:11.769261+0000","last_active":"2024-03-28T17:47:09.951519+0000","last_peered":"2024-03-28T17:47:09.951519+0000","last_clean":"2024-03-28T17:47:09.951519+0000","last_became_active":"2024-03-28T17:46:11.768918+0000","last_became_peered":"2024-03-28T17:46:11.768918+0000","last_unstale":"2024-03-28T17:47:09.951519+0000","last_undegraded":"2024-03-28T17:47:09.951519+0000","last_fullsized":"2024-03-28T17:47:09.951519+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:26:38.276425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.655991+0000","last_change":"2024-03-28T17:46:12.762617+0000","last_active":"2024-03-28T17:47:37.655991+0000","last_peered":"2024-03-28T17:47:37.655991+0000","last_clean":"2024-03-28T17:47:37.655991+0000","last_became_active":"2024-03-28T17:46:12.762273+0000","last_became_peered":"2024-03-28T17:46:12.762273+0000","last_unstale":"2024-03-28T17:47:37.655991+0000","last_undegraded":"2024-03-28T17:47:37.655991+0000","last_fullsized":"2024-03-28T17:47:37.655991+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:45.848454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.476203+0000","last_change":"2024-03-28T17:46:11.770831+0000","last_active":"2024-03-28T17:47:09.476203+0000","last_peered":"2024-03-28T17:47:09.476203+0000","last_clean":"2024-03-28T17:47:09.476203+0000","last_became_active":"2024-03-28T17:46:11.770303+0000","last_became_peered":"2024-03-28T17:46:11.770303+0000","last_unstale":"2024-03-28T17:47:09.476203+0000","last_undegraded":"2024-03-28T17:47:09.476203+0000","last_fullsized":"2024-03-28T17:47:09.476203+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:48:25.566922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685855+0000","last_change":"2024-03-28T17:46:12.768885+0000","last_active":"2024-03-28T17:47:37.685855+0000","last_peered":"2024-03-28T17:47:37.685855+0000","last_clean":"2024-03-28T17:47:37.685855+0000","last_became_active":"2024-03-28T17:46:12.768525+0000","last_became_peered":"2024-03-28T17:46:12.768525+0000","last_unstale":"2024-03-28T17:47:37.685855+0000","last_undegraded":"2024-03-28T17:47:37.685855+0000","last_fullsized":"2024-03-28T17:47:37.685855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:35:30.779279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:48:24.591545+0000","last_change":"2024-03-28T17:46:11.786661+0000","last_active":"2024-03-28T17:48:24.591545+0000","last_peered":"2024-03-28T17:48:24.591545+0000","last_clean":"2024-03-28T17:48:24.591545+0000","last_became_active":"2024-03-28T17:46:11.786226+0000","last_became_peered":"2024-03-28T17:46:11.786226+0000","last_unstale":"2024-03-28T17:48:24.591545+0000","last_undegraded":"2024-03-28T17:48:24.591545+0000","last_fullsized":"2024-03-28T17:48:24.591545+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:51:24.781368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687652+0000","last_change":"2024-03-28T17:46:12.797531+0000","last_active":"2024-03-28T17:47:37.687652+0000","last_peered":"2024-03-28T17:47:37.687652+0000","last_clean":"2024-03-28T17:47:37.687652+0000","last_became_active":"2024-03-28T17:46:12.797161+0000","last_became_peered":"2024-03-28T17:46:12.797161+0000","last_unstale":"2024-03-28T17:47:37.687652+0000","last_undegraded":"2024-03-28T17:47:37.687652+0000","last_fullsized":"2024-03-28T17:47:37.687652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:01.534666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461493+0000","last_change":"2024-03-28T17:46:11.766589+0000","last_active":"2024-03-28T17:47:09.461493+0000","last_peered":"2024-03-28T17:47:09.461493+0000","last_clean":"2024-03-28T17:47:09.461493+0000","last_became_active":"2024-03-28T17:46:11.766269+0000","last_became_peered":"2024-03-28T17:46:11.766269+0000","last_unstale":"2024-03-28T17:47:09.461493+0000","last_undegraded":"2024-03-28T17:47:09.461493+0000","last_fullsized":"2024-03-28T17:47:09.461493+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:12:26.055262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.670254+0000","last_change":"2024-03-28T17:46:12.765723+0000","last_active":"2024-03-28T17:47:37.670254+0000","last_peered":"2024-03-28T17:47:37.670254+0000","last_clean":"2024-03-28T17:47:37.670254+0000","last_became_active":"2024-03-28T17:46:12.765366+0000","last_became_peered":"2024-03-28T17:46:12.765366+0000","last_unstale":"2024-03-28T17:47:37.670254+0000","last_undegraded":"2024-03-28T17:47:37.670254+0000","last_fullsized":"2024-03-28T17:47:37.670254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:39:55.103246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.463696+0000","last_change":"2024-03-28T17:46:11.767686+0000","last_active":"2024-03-28T17:47:09.463696+0000","last_peered":"2024-03-28T17:47:09.463696+0000","last_clean":"2024-03-28T17:47:09.463696+0000","last_became_active":"2024-03-28T17:46:11.767365+0000","last_became_peered":"2024-03-28T17:46:11.767365+0000","last_unstale":"2024-03-28T17:47:09.463696+0000","last_undegraded":"2024-03-28T17:47:09.463696+0000","last_fullsized":"2024-03-28T17:47:09.463696+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:49.909554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.669367+0000","last_change":"2024-03-28T17:46:12.764129+0000","last_active":"2024-03-28T17:47:37.669367+0000","last_peered":"2024-03-28T17:47:37.669367+0000","last_clean":"2024-03-28T17:47:37.669367+0000","last_became_active":"2024-03-28T17:46:12.763716+0000","last_became_peered":"2024-03-28T17:46:12.763716+0000","last_unstale":"2024-03-28T17:47:37.669367+0000","last_undegraded":"2024-03-28T17:47:37.669367+0000","last_fullsized":"2024-03-28T17:47:37.669367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:08.365578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.464975+0000","last_change":"2024-03-28T17:46:11.787374+0000","last_active":"2024-03-28T17:47:09.464975+0000","last_peered":"2024-03-28T17:47:09.464975+0000","last_clean":"2024-03-28T17:47:09.464975+0000","last_became_active":"2024-03-28T17:46:11.787047+0000","last_became_peered":"2024-03-28T17:46:11.787047+0000","last_unstale":"2024-03-28T17:47:09.464975+0000","last_undegraded":"2024-03-28T17:47:09.464975+0000","last_fullsized":"2024-03-28T17:47:09.464975+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:33.657488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.672808+0000","last_change":"2024-03-28T17:47:09.480465+0000","last_active":"2024-03-28T17:47:37.672808+0000","last_peered":"2024-03-28T17:47:37.672808+0000","last_clean":"2024-03-28T17:47:37.672808+0000","last_became_active":"2024-03-28T17:47:09.475758+0000","last_became_peered":"2024-03-28T17:47:09.475758+0000","last_unstale":"2024-03-28T17:47:37.672808+0000","last_undegraded":"2024-03-28T17:47:37.672808+0000","last_fullsized":"2024-03-28T17:47:37.672808+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:08.610192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.473252+0000","last_change":"2024-03-28T17:46:11.780388+0000","last_active":"2024-03-28T17:47:09.473252+0000","last_peered":"2024-03-28T17:47:09.473252+0000","last_clean":"2024-03-28T17:47:09.473252+0000","last_became_active":"2024-03-28T17:46:11.780016+0000","last_became_peered":"2024-03-28T17:46:11.780016+0000","last_unstale":"2024-03-28T17:47:09.473252+0000","last_undegraded":"2024-03-28T17:47:09.473252+0000","last_fullsized":"2024-03-28T17:47:09.473252+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:51:06.181772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.652005+0000","last_change":"2024-03-28T17:46:12.764748+0000","last_active":"2024-03-28T17:47:37.652005+0000","last_peered":"2024-03-28T17:47:37.652005+0000","last_clean":"2024-03-28T17:47:37.652005+0000","last_became_active":"2024-03-28T17:46:12.764407+0000","last_became_peered":"2024-03-28T17:46:12.764407+0000","last_unstale":"2024-03-28T17:47:37.652005+0000","last_undegraded":"2024-03-28T17:47:37.652005+0000","last_fullsized":"2024-03-28T17:47:37.652005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:41:54.185079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.652775+0000","last_change":"2024-03-28T17:46:12.762486+0000","last_active":"2024-03-28T17:47:37.652775+0000","last_peered":"2024-03-28T17:47:37.652775+0000","last_clean":"2024-03-28T17:47:37.652775+0000","last_became_active":"2024-03-28T17:46:12.762107+0000","last_became_peered":"2024-03-28T17:46:12.762107+0000","last_unstale":"2024-03-28T17:47:37.652775+0000","last_undegraded":"2024-03-28T17:47:37.652775+0000","last_fullsized":"2024-03-28T17:47:37.652775+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:31.062456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.664930+0000","last_change":"2024-03-28T17:46:12.764569+0000","last_active":"2024-03-28T17:47:37.664930+0000","last_peered":"2024-03-28T17:47:37.664930+0000","last_clean":"2024-03-28T17:47:37.664930+0000","last_became_active":"2024-03-28T17:46:12.764112+0000","last_became_peered":"2024-03-28T17:46:12.764112+0000","last_unstale":"2024-03-28T17:47:37.664930+0000","last_undegraded":"2024-03-28T17:47:37.664930+0000","last_fullsized":"2024-03-28T17:47:37.664930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:39:55.526432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.684961+0000","last_change":"2024-03-28T17:46:12.786031+0000","last_active":"2024-03-28T17:47:37.684961+0000","last_peered":"2024-03-28T17:47:37.684961+0000","last_clean":"2024-03-28T17:47:37.684961+0000","last_became_active":"2024-03-28T17:46:12.785679+0000","last_became_peered":"2024-03-28T17:46:12.785679+0000","last_unstale":"2024-03-28T17:47:37.684961+0000","last_undegraded":"2024-03-28T17:47:37.684961+0000","last_fullsized":"2024-03-28T17:47:37.684961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:41:22.537979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685335+0000","last_change":"2024-03-28T17:46:12.793285+0000","last_active":"2024-03-28T17:47:37.685335+0000","last_peered":"2024-03-28T17:47:37.685335+0000","last_clean":"2024-03-28T17:47:37.685335+0000","last_became_active":"2024-03-28T17:46:12.788935+0000","last_became_peered":"2024-03-28T17:46:12.788935+0000","last_unstale":"2024-03-28T17:47:37.685335+0000","last_undegraded":"2024-03-28T17:47:37.685335+0000","last_fullsized":"2024-03-28T17:47:37.685335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:53:58.886607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.688031+0000","last_change":"2024-03-28T17:47:09.486124+0000","last_active":"2024-03-28T17:47:37.688031+0000","last_peered":"2024-03-28T17:47:37.688031+0000","last_clean":"2024-03-28T17:47:37.688031+0000","last_became_active":"2024-03-28T17:47:09.481292+0000","last_became_peered":"2024-03-28T17:47:09.481292+0000","last_unstale":"2024-03-28T17:47:37.688031+0000","last_undegraded":"2024-03-28T17:47:37.688031+0000","last_fullsized":"2024-03-28T17:47:37.688031+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:18.656764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.645280+0000","last_change":"2024-03-28T17:46:12.765140+0000","last_active":"2024-03-28T17:47:37.645280+0000","last_peered":"2024-03-28T17:47:37.645280+0000","last_clean":"2024-03-28T17:47:37.645280+0000","last_became_active":"2024-03-28T17:46:12.764760+0000","last_became_peered":"2024-03-28T17:46:12.764760+0000","last_unstale":"2024-03-28T17:47:37.645280+0000","last_undegraded":"2024-03-28T17:47:37.645280+0000","last_fullsized":"2024-03-28T17:47:37.645280+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:41.576437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.658988+0000","last_change":"2024-03-28T17:46:12.760271+0000","last_active":"2024-03-28T17:47:37.658988+0000","last_peered":"2024-03-28T17:47:37.658988+0000","last_clean":"2024-03-28T17:47:37.658988+0000","last_became_active":"2024-03-28T17:46:12.759939+0000","last_became_peered":"2024-03-28T17:46:12.759939+0000","last_unstale":"2024-03-28T17:47:37.658988+0000","last_undegraded":"2024-03-28T17:47:37.658988+0000","last_fullsized":"2024-03-28T17:47:37.658988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:06:42.529826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.681262+0000","last_change":"2024-03-28T17:46:12.792360+0000","last_active":"2024-03-28T17:47:37.681262+0000","last_peered":"2024-03-28T17:47:37.681262+0000","last_clean":"2024-03-28T17:47:37.681262+0000","last_became_active":"2024-03-28T17:46:12.791979+0000","last_became_peered":"2024-03-28T17:46:12.791979+0000","last_unstale":"2024-03-28T17:47:37.681262+0000","last_undegraded":"2024-03-28T17:47:37.681262+0000","last_fullsized":"2024-03-28T17:47:37.681262+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:39.866969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685176+0000","last_change":"2024-03-28T17:46:12.784228+0000","last_active":"2024-03-28T17:47:37.685176+0000","last_peered":"2024-03-28T17:47:37.685176+0000","last_clean":"2024-03-28T17:47:37.685176+0000","last_became_active":"2024-03-28T17:46:12.783890+0000","last_became_peered":"2024-03-28T17:46:12.783890+0000","last_unstale":"2024-03-28T17:47:37.685176+0000","last_undegraded":"2024-03-28T17:47:37.685176+0000","last_fullsized":"2024-03-28T17:47:37.685176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:58:13.388705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685701+0000","last_change":"2024-03-28T17:46:12.782675+0000","last_active":"2024-03-28T17:47:37.685701+0000","last_peered":"2024-03-28T17:47:37.685701+0000","last_clean":"2024-03-28T17:47:37.685701+0000","last_became_active":"2024-03-28T17:46:12.779255+0000","last_became_peered":"2024-03-28T17:46:12.779255+0000","last_unstale":"2024-03-28T17:47:37.685701+0000","last_undegraded":"2024-03-28T17:47:37.685701+0000","last_fullsized":"2024-03-28T17:47:37.685701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:04:31.447512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.677034+0000","last_change":"2024-03-28T17:46:12.786308+0000","last_active":"2024-03-28T17:47:37.677034+0000","last_peered":"2024-03-28T17:47:37.677034+0000","last_clean":"2024-03-28T17:47:37.677034+0000","last_became_active":"2024-03-28T17:46:12.785948+0000","last_became_peered":"2024-03-28T17:46:12.785948+0000","last_unstale":"2024-03-28T17:47:37.677034+0000","last_undegraded":"2024-03-28T17:47:37.677034+0000","last_fullsized":"2024-03-28T17:47:37.677034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:43.292925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687492+0000","last_change":"2024-03-28T17:46:12.797278+0000","last_active":"2024-03-28T17:47:37.687492+0000","last_peered":"2024-03-28T17:47:37.687492+0000","last_clean":"2024-03-28T17:47:37.687492+0000","last_became_active":"2024-03-28T17:46:12.796593+0000","last_became_peered":"2024-03-28T17:46:12.796593+0000","last_unstale":"2024-03-28T17:47:37.687492+0000","last_undegraded":"2024-03-28T17:47:37.687492+0000","last_fullsized":"2024-03-28T17:47:37.687492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:07:34.433680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.638868+0000","last_change":"2024-03-28T17:46:12.764249+0000","last_active":"2024-03-28T17:47:37.638868+0000","last_peered":"2024-03-28T17:47:37.638868+0000","last_clean":"2024-03-28T17:47:37.638868+0000","last_became_active":"2024-03-28T17:46:12.763909+0000","last_became_peered":"2024-03-28T17:46:12.763909+0000","last_unstale":"2024-03-28T17:47:37.638868+0000","last_undegraded":"2024-03-28T17:47:37.638868+0000","last_fullsized":"2024-03-28T17:47:37.638868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:40:51.780187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250098,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73633,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":896,"num_read_kb":529,"num_write":180,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":211545,"data_compressed":5625,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383725,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":456197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645362,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":1116863,"data_compressed":38073,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808888,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":456550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070525,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":456353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332162,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":1049262,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495688,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":455822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574949,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":1003520,"data_stored":517043,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738475,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":457093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411424,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":519231,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182490,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":456016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920852,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":456791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757326,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":1048414,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34554,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34554,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":66133,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":63616,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":70763,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T17:48:44.862 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T17:48:45.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:48:45.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:44 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3208382511' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T17:48:45.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:45.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:45.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:44 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1098089609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:48:45.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:45.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:44 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:45.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.111+0000 7fb1f9200640 1 -- 172.21.15.45:0/1673438152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1f4103270 msgr2=0x7fb1f4109b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:45.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.111+0000 7fb1f9200640 1 --2- 172.21.15.45:0/1673438152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1f4103270 0x7fb1f4109b00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec009ca0 tx=0x7fb1ec02c0f0 comp rx=0 tx=0).stop 2024-03-28T17:48:45.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.112+0000 7fb1f9200640 1 -- 172.21.15.45:0/1673438152 shutdown_connections 2024-03-28T17:48:45.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.112+0000 7fb1f9200640 1 --2- 172.21.15.45:0/1673438152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1f4103270 0x7fb1f4109b00 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:45.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.112+0000 7fb1f9200640 1 --2- 172.21.15.45:0/1673438152 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f41028b0 0x7fb1f4102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:45.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.112+0000 7fb1f9200640 1 --2- 172.21.15.45:0/1673438152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1f41016c0 0x7fb1f4101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:45.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.112+0000 7fb1f9200640 1 -- 172.21.15.45:0/1673438152 >> 172.21.15.45:0/1673438152 conn(0x7fb1f40fcc50 msgr2=0x7fb1f40ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:45.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.112+0000 7fb1f9200640 1 -- 172.21.15.45:0/1673438152 shutdown_connections 2024-03-28T17:48:45.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.112+0000 7fb1f9200640 1 -- 172.21.15.45:0/1673438152 wait complete. 2024-03-28T17:48:45.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.113+0000 7fb1f9200640 1 Processor -- start 2024-03-28T17:48:45.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.113+0000 7fb1f9200640 1 -- start start 2024-03-28T17:48:45.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.113+0000 7fb1f9200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1f41016c0 0x7fb1f4199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:45.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.113+0000 7fb1f9200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1f41028b0 0x7fb1f419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:45.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.113+0000 7fb1f9200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f4103270 0x7fb1f41a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:45.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.113+0000 7fb1f9200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f410d340 con 0x7fb1f41016c0 2024-03-28T17:48:45.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.113+0000 7fb1f9200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f410d1c0 con 0x7fb1f41028b0 2024-03-28T17:48:45.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.113+0000 7fb1f9200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f410d4c0 con 0x7fb1f4103270 2024-03-28T17:48:45.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.113+0000 7fb1f1400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1f41028b0 0x7fb1f419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:45.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.113+0000 7fb1f1400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1f41028b0 0x7fb1f419a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:54966/0 (socket says 172.21.15.45:54966) 2024-03-28T17:48:45.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.114+0000 7fb1f1400640 1 -- 172.21.15.45:0/1034634117 learned_addr learned my addr 172.21.15.45:0/1034634117 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:45.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.114+0000 7fb1f2800640 1 --2- 172.21.15.45:0/1034634117 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f4103270 0x7fb1f41a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:45.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.113+0000 7fb1f1e00640 1 --2- 172.21.15.45:0/1034634117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1f41016c0 0x7fb1f4199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:45.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.114+0000 7fb1f1400640 1 -- 172.21.15.45:0/1034634117 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f4103270 msgr2=0x7fb1f41a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:45.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.114+0000 7fb1f1400640 1 --2- 172.21.15.45:0/1034634117 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f4103270 0x7fb1f41a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:45.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.114+0000 7fb1f1400640 1 -- 172.21.15.45:0/1034634117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1f41016c0 msgr2=0x7fb1f4199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:45.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.114+0000 7fb1f1400640 1 --2- 172.21.15.45:0/1034634117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1f41016c0 0x7fb1f4199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:45.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.114+0000 7fb1f1400640 1 -- 172.21.15.45:0/1034634117 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f41a17c0 con 0x7fb1f41028b0 2024-03-28T17:48:45.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.114+0000 7fb1f1e00640 1 --2- 172.21.15.45:0/1034634117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1f41016c0 0x7fb1f4199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:48:45.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.114+0000 7fb1f1400640 1 --2- 172.21.15.45:0/1034634117 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1f41028b0 0x7fb1f419a0e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb1e000e9f0 tx=0x7fb1e000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:45.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.114+0000 7fb1eaa00640 1 -- 172.21.15.45:0/1034634117 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e000cd80 con 0x7fb1f41028b0 2024-03-28T17:48:45.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.114+0000 7fb1eaa00640 1 -- 172.21.15.45:0/1034634117 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb1e0005500 con 0x7fb1f41028b0 2024-03-28T17:48:45.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.114+0000 7fb1f9200640 1 -- 172.21.15.45:0/1034634117 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f41a1ab0 con 0x7fb1f41028b0 2024-03-28T17:48:45.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.115+0000 7fb1eaa00640 1 -- 172.21.15.45:0/1034634117 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0010690 con 0x7fb1f41028b0 2024-03-28T17:48:45.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.115+0000 7fb1f9200640 1 -- 172.21.15.45:0/1034634117 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f41a1fb0 con 0x7fb1f41028b0 2024-03-28T17:48:45.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.116+0000 7fb1f9200640 1 -- 172.21.15.45:0/1034634117 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f41a2240 con 0x7fb1f41028b0 2024-03-28T17:48:45.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.118+0000 7fb1eaa00640 1 -- 172.21.15.45:0/1034634117 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fb1e00046c0 con 0x7fb1f41028b0 2024-03-28T17:48:45.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.120+0000 7fb1eaa00640 1 --2- 172.21.15.45:0/1034634117 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb1d80a2910 0x7fb1d80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:45.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.120+0000 7fb1eaa00640 1 -- 172.21.15.45:0/1034634117 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb1e00cf0e0 con 0x7fb1f41028b0 2024-03-28T17:48:45.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.120+0000 7fb1f1e00640 1 --2- 172.21.15.45:0/1034634117 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb1d80a2910 0x7fb1d80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:45.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.121+0000 7fb1f1e00640 1 --2- 172.21.15.45:0/1034634117 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb1d80a2910 0x7fb1d80a4dd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4102710 tx=0x7fb1e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:45.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.128+0000 7fb1eaa00640 1 -- 172.21.15.45:0/1034634117 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb1e009bfd0 con 0x7fb1f41028b0 2024-03-28T17:48:45.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.357+0000 7fb1f9200640 1 -- 172.21.15.45:0/1034634117 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb1f41a2410 con 0x7fb1d80a2910 2024-03-28T17:48:45.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.367+0000 7fb1eaa00640 1 -- 172.21.15.45:0/1034634117 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259043 (secure 0 0 0) 0x7fb1f41a2410 con 0x7fb1d80a2910 2024-03-28T17:48:45.371 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:48:45.372 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-28T17:48:45.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.372+0000 7fb1f9200640 1 -- 172.21.15.45:0/1034634117 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb1d80a2910 msgr2=0x7fb1d80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:45.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.372+0000 7fb1f9200640 1 --2- 172.21.15.45:0/1034634117 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb1d80a2910 0x7fb1d80a4dd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4102710 tx=0x7fb1e4005f90 comp rx=0 tx=0).stop 2024-03-28T17:48:45.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.372+0000 7fb1f9200640 1 -- 172.21.15.45:0/1034634117 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1f41028b0 msgr2=0x7fb1f419a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:45.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.372+0000 7fb1f9200640 1 --2- 172.21.15.45:0/1034634117 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1f41028b0 0x7fb1f419a0e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb1e000e9f0 tx=0x7fb1e000edf0 comp rx=0 tx=0).stop 2024-03-28T17:48:45.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.372+0000 7fb1f9200640 1 -- 172.21.15.45:0/1034634117 shutdown_connections 2024-03-28T17:48:45.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.372+0000 7fb1f9200640 1 --2- 172.21.15.45:0/1034634117 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fb1d80a2910 0x7fb1d80a4dd0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:45.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.372+0000 7fb1f9200640 1 --2- 172.21.15.45:0/1034634117 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f4103270 0x7fb1f41a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:45.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.372+0000 7fb1f9200640 1 --2- 172.21.15.45:0/1034634117 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1f41028b0 0x7fb1f419a0e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:45.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.372+0000 7fb1f9200640 1 --2- 172.21.15.45:0/1034634117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1f41016c0 0x7fb1f4199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:45.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.372+0000 7fb1f9200640 1 -- 172.21.15.45:0/1034634117 >> 172.21.15.45:0/1034634117 conn(0x7fb1f40fcc50 msgr2=0x7fb1f40738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:45.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.372+0000 7fb1f9200640 1 -- 172.21.15.45:0/1034634117 shutdown_connections 2024-03-28T17:48:45.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.372+0000 7fb1f9200640 1 -- 172.21.15.45:0/1034634117 wait complete. 2024-03-28T17:48:45.395 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-03-28T17:48:43.708770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664235,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":1032,"num_read_kb":648,"num_write":363,"num_write_kb":3073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180853,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":794,"ondisk_log_size":794,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":364876,"kb_used_data":11564,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576948,"statfs":{"total":1151940427776,"available":1151566794752,"internally_reserved":0,"allocated":11841536,"data_stored":7445635,"data_compressed":114219,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7837,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":78,"num_read_kb":50,"num_write":10,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004441"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683908+0000","last_change":"2024-03-28T17:46:12.783038+0000","last_active":"2024-03-28T17:47:37.683908+0000","last_peered":"2024-03-28T17:47:37.683908+0000","last_clean":"2024-03-28T17:47:37.683908+0000","last_became_active":"2024-03-28T17:46:12.782668+0000","last_became_peered":"2024-03-28T17:46:12.782668+0000","last_unstale":"2024-03-28T17:47:37.683908+0000","last_undegraded":"2024-03-28T17:47:37.683908+0000","last_fullsized":"2024-03-28T17:47:37.683908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:03.424959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683564+0000","last_change":"2024-03-28T17:46:12.781828+0000","last_active":"2024-03-28T17:47:37.683564+0000","last_peered":"2024-03-28T17:47:37.683564+0000","last_clean":"2024-03-28T17:47:37.683564+0000","last_became_active":"2024-03-28T17:46:12.781424+0000","last_became_peered":"2024-03-28T17:46:12.781424+0000","last_unstale":"2024-03-28T17:47:37.683564+0000","last_undegraded":"2024-03-28T17:47:37.683564+0000","last_fullsized":"2024-03-28T17:47:37.683564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:15:16.639473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683268+0000","last_change":"2024-03-28T17:46:12.785431+0000","last_active":"2024-03-28T17:47:37.683268+0000","last_peered":"2024-03-28T17:47:37.683268+0000","last_clean":"2024-03-28T17:47:37.683268+0000","last_became_active":"2024-03-28T17:46:12.784992+0000","last_became_peered":"2024-03-28T17:46:12.784992+0000","last_unstale":"2024-03-28T17:47:37.683268+0000","last_undegraded":"2024-03-28T17:47:37.683268+0000","last_fullsized":"2024-03-28T17:47:37.683268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:03:36.926613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.663175+0000","last_change":"2024-03-28T17:46:12.765274+0000","last_active":"2024-03-28T17:47:37.663175+0000","last_peered":"2024-03-28T17:47:37.663175+0000","last_clean":"2024-03-28T17:47:37.663175+0000","last_became_active":"2024-03-28T17:46:12.764890+0000","last_became_peered":"2024-03-28T17:46:12.764890+0000","last_unstale":"2024-03-28T17:47:37.663175+0000","last_undegraded":"2024-03-28T17:47:37.663175+0000","last_fullsized":"2024-03-28T17:47:37.663175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:58:37.149756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.670408+0000","last_change":"2024-03-28T17:46:12.782791+0000","last_active":"2024-03-28T17:47:37.670408+0000","last_peered":"2024-03-28T17:47:37.670408+0000","last_clean":"2024-03-28T17:47:37.670408+0000","last_became_active":"2024-03-28T17:46:12.782420+0000","last_became_peered":"2024-03-28T17:46:12.782420+0000","last_unstale":"2024-03-28T17:47:37.670408+0000","last_undegraded":"2024-03-28T17:47:37.670408+0000","last_fullsized":"2024-03-28T17:47:37.670408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:30.512863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.657353+0000","last_change":"2024-03-28T17:46:12.780713+0000","last_active":"2024-03-28T17:47:37.657353+0000","last_peered":"2024-03-28T17:47:37.657353+0000","last_clean":"2024-03-28T17:47:37.657353+0000","last_became_active":"2024-03-28T17:46:12.780305+0000","last_became_peered":"2024-03-28T17:46:12.780305+0000","last_unstale":"2024-03-28T17:47:37.657353+0000","last_undegraded":"2024-03-28T17:47:37.657353+0000","last_fullsized":"2024-03-28T17:47:37.657353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:08:09.840491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.653070+0000","last_change":"2024-03-28T17:46:12.778120+0000","last_active":"2024-03-28T17:47:37.653070+0000","last_peered":"2024-03-28T17:47:37.653070+0000","last_clean":"2024-03-28T17:47:37.653070+0000","last_became_active":"2024-03-28T17:46:12.777724+0000","last_became_peered":"2024-03-28T17:46:12.777724+0000","last_unstale":"2024-03-28T17:47:37.653070+0000","last_undegraded":"2024-03-28T17:47:37.653070+0000","last_fullsized":"2024-03-28T17:47:37.653070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:29:50.980366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687911+0000","last_change":"2024-03-28T17:46:12.793447+0000","last_active":"2024-03-28T17:47:37.687911+0000","last_peered":"2024-03-28T17:47:37.687911+0000","last_clean":"2024-03-28T17:47:37.687911+0000","last_became_active":"2024-03-28T17:46:12.793070+0000","last_became_peered":"2024-03-28T17:46:12.793070+0000","last_unstale":"2024-03-28T17:47:37.687911+0000","last_undegraded":"2024-03-28T17:47:37.687911+0000","last_fullsized":"2024-03-28T17:47:37.687911+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:01.364577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.668936+0000","last_change":"2024-03-28T17:46:12.763908+0000","last_active":"2024-03-28T17:47:37.668936+0000","last_peered":"2024-03-28T17:47:37.668936+0000","last_clean":"2024-03-28T17:47:37.668936+0000","last_became_active":"2024-03-28T17:46:12.763267+0000","last_became_peered":"2024-03-28T17:46:12.763267+0000","last_unstale":"2024-03-28T17:47:37.668936+0000","last_undegraded":"2024-03-28T17:47:37.668936+0000","last_fullsized":"2024-03-28T17:47:37.668936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:10.692974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.660790+0000","last_change":"2024-03-28T17:46:12.762648+0000","last_active":"2024-03-28T17:47:37.660790+0000","last_peered":"2024-03-28T17:47:37.660790+0000","last_clean":"2024-03-28T17:47:37.660790+0000","last_became_active":"2024-03-28T17:46:12.762300+0000","last_became_peered":"2024-03-28T17:46:12.762300+0000","last_unstale":"2024-03-28T17:47:37.660790+0000","last_undegraded":"2024-03-28T17:47:37.660790+0000","last_fullsized":"2024-03-28T17:47:37.660790+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:02.210828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.668940+0000","last_change":"2024-03-28T17:46:12.766482+0000","last_active":"2024-03-28T17:47:37.668940+0000","last_peered":"2024-03-28T17:47:37.668940+0000","last_clean":"2024-03-28T17:47:37.668940+0000","last_became_active":"2024-03-28T17:46:12.766155+0000","last_became_peered":"2024-03-28T17:46:12.766155+0000","last_unstale":"2024-03-28T17:47:37.668940+0000","last_undegraded":"2024-03-28T17:47:37.668940+0000","last_fullsized":"2024-03-28T17:47:37.668940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:46:24.261744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.643025+0000","last_change":"2024-03-28T17:46:12.766100+0000","last_active":"2024-03-28T17:47:37.643025+0000","last_peered":"2024-03-28T17:47:37.643025+0000","last_clean":"2024-03-28T17:47:37.643025+0000","last_became_active":"2024-03-28T17:46:12.765762+0000","last_became_peered":"2024-03-28T17:46:12.765762+0000","last_unstale":"2024-03-28T17:47:37.643025+0000","last_undegraded":"2024-03-28T17:47:37.643025+0000","last_fullsized":"2024-03-28T17:47:37.643025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:11.666653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.670019+0000","last_change":"2024-03-28T17:46:12.753465+0000","last_active":"2024-03-28T17:47:37.670019+0000","last_peered":"2024-03-28T17:47:37.670019+0000","last_clean":"2024-03-28T17:47:37.670019+0000","last_became_active":"2024-03-28T17:46:12.751931+0000","last_became_peered":"2024-03-28T17:46:12.751931+0000","last_unstale":"2024-03-28T17:47:37.670019+0000","last_undegraded":"2024-03-28T17:47:37.670019+0000","last_fullsized":"2024-03-28T17:47:37.670019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:39:38.757878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.684532+0000","last_change":"2024-03-28T17:46:12.784177+0000","last_active":"2024-03-28T17:47:37.684532+0000","last_peered":"2024-03-28T17:47:37.684532+0000","last_clean":"2024-03-28T17:47:37.684532+0000","last_became_active":"2024-03-28T17:46:12.783823+0000","last_became_peered":"2024-03-28T17:46:12.783823+0000","last_unstale":"2024-03-28T17:47:37.684532+0000","last_undegraded":"2024-03-28T17:47:37.684532+0000","last_fullsized":"2024-03-28T17:47:37.684532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:11:33.221095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685187+0000","last_change":"2024-03-28T17:46:12.777152+0000","last_active":"2024-03-28T17:47:37.685187+0000","last_peered":"2024-03-28T17:47:37.685187+0000","last_clean":"2024-03-28T17:47:37.685187+0000","last_became_active":"2024-03-28T17:46:12.776758+0000","last_became_peered":"2024-03-28T17:46:12.776758+0000","last_unstale":"2024-03-28T17:47:37.685187+0000","last_undegraded":"2024-03-28T17:47:37.685187+0000","last_fullsized":"2024-03-28T17:47:37.685187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:20:56.587470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.686334+0000","last_change":"2024-03-28T17:46:12.782384+0000","last_active":"2024-03-28T17:47:37.686334+0000","last_peered":"2024-03-28T17:47:37.686334+0000","last_clean":"2024-03-28T17:47:37.686334+0000","last_became_active":"2024-03-28T17:46:12.781971+0000","last_became_peered":"2024-03-28T17:46:12.781971+0000","last_unstale":"2024-03-28T17:47:37.686334+0000","last_undegraded":"2024-03-28T17:47:37.686334+0000","last_fullsized":"2024-03-28T17:47:37.686334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:19:32.652629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685920+0000","last_change":"2024-03-28T17:46:12.789364+0000","last_active":"2024-03-28T17:47:37.685920+0000","last_peered":"2024-03-28T17:47:37.685920+0000","last_clean":"2024-03-28T17:47:37.685920+0000","last_became_active":"2024-03-28T17:46:12.788976+0000","last_became_peered":"2024-03-28T17:46:12.788976+0000","last_unstale":"2024-03-28T17:47:37.685920+0000","last_undegraded":"2024-03-28T17:47:37.685920+0000","last_fullsized":"2024-03-28T17:47:37.685920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:43:37.183705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.663596+0000","last_change":"2024-03-28T17:46:12.782664+0000","last_active":"2024-03-28T17:47:37.663596+0000","last_peered":"2024-03-28T17:47:37.663596+0000","last_clean":"2024-03-28T17:47:37.663596+0000","last_became_active":"2024-03-28T17:46:12.782305+0000","last_became_peered":"2024-03-28T17:46:12.782305+0000","last_unstale":"2024-03-28T17:47:37.663596+0000","last_undegraded":"2024-03-28T17:47:37.663596+0000","last_fullsized":"2024-03-28T17:47:37.663596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:47:18.373517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.684368+0000","last_change":"2024-03-28T17:46:12.794494+0000","last_active":"2024-03-28T17:47:37.684368+0000","last_peered":"2024-03-28T17:47:37.684368+0000","last_clean":"2024-03-28T17:47:37.684368+0000","last_became_active":"2024-03-28T17:46:12.794151+0000","last_became_peered":"2024-03-28T17:46:12.794151+0000","last_unstale":"2024-03-28T17:47:37.684368+0000","last_undegraded":"2024-03-28T17:47:37.684368+0000","last_fullsized":"2024-03-28T17:47:37.684368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:17:24.596333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.475806+0000","last_change":"2024-03-28T17:46:11.759643+0000","last_active":"2024-03-28T17:47:09.475806+0000","last_peered":"2024-03-28T17:47:09.475806+0000","last_clean":"2024-03-28T17:47:09.475806+0000","last_became_active":"2024-03-28T17:46:11.759273+0000","last_became_peered":"2024-03-28T17:46:11.759273+0000","last_unstale":"2024-03-28T17:47:09.475806+0000","last_undegraded":"2024-03-28T17:47:09.475806+0000","last_fullsized":"2024-03-28T17:47:09.475806+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:09:51.764471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687168+0000","last_change":"2024-03-28T17:46:12.786503+0000","last_active":"2024-03-28T17:47:37.687168+0000","last_peered":"2024-03-28T17:47:37.687168+0000","last_clean":"2024-03-28T17:47:37.687168+0000","last_became_active":"2024-03-28T17:46:12.784651+0000","last_became_peered":"2024-03-28T17:46:12.784651+0000","last_unstale":"2024-03-28T17:47:37.687168+0000","last_undegraded":"2024-03-28T17:47:37.687168+0000","last_fullsized":"2024-03-28T17:47:37.687168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:30:22.401767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.480732+0000","last_change":"2024-03-28T17:46:11.780340+0000","last_active":"2024-03-28T17:47:09.480732+0000","last_peered":"2024-03-28T17:47:09.480732+0000","last_clean":"2024-03-28T17:47:09.480732+0000","last_became_active":"2024-03-28T17:46:11.779917+0000","last_became_peered":"2024-03-28T17:46:11.779917+0000","last_unstale":"2024-03-28T17:47:09.480732+0000","last_undegraded":"2024-03-28T17:47:09.480732+0000","last_fullsized":"2024-03-28T17:47:09.480732+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:22:45.617444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.686129+0000","last_change":"2024-03-28T17:46:12.795086+0000","last_active":"2024-03-28T17:47:37.686129+0000","last_peered":"2024-03-28T17:47:37.686129+0000","last_clean":"2024-03-28T17:47:37.686129+0000","last_became_active":"2024-03-28T17:46:12.794124+0000","last_became_peered":"2024-03-28T17:46:12.794124+0000","last_unstale":"2024-03-28T17:47:37.686129+0000","last_undegraded":"2024-03-28T17:47:37.686129+0000","last_fullsized":"2024-03-28T17:47:37.686129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:11:05.347047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.648328+0000","last_change":"2024-03-28T17:47:09.648328+0000","last_active":"2024-03-28T17:47:09.648328+0000","last_peered":"2024-03-28T17:47:09.648328+0000","last_clean":"2024-03-28T17:47:09.648328+0000","last_became_active":"2024-03-28T17:47:09.644797+0000","last_became_peered":"2024-03-28T17:47:09.644797+0000","last_unstale":"2024-03-28T17:47:09.648328+0000","last_undegraded":"2024-03-28T17:47:09.648328+0000","last_fullsized":"2024-03-28T17:47:09.648328+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:14:30.233156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.665594+0000","last_change":"2024-03-28T17:46:12.768118+0000","last_active":"2024-03-28T17:47:37.665594+0000","last_peered":"2024-03-28T17:47:37.665594+0000","last_clean":"2024-03-28T17:47:37.665594+0000","last_became_active":"2024-03-28T17:46:12.767765+0000","last_became_peered":"2024-03-28T17:46:12.767765+0000","last_unstale":"2024-03-28T17:47:37.665594+0000","last_undegraded":"2024-03-28T17:47:37.665594+0000","last_fullsized":"2024-03-28T17:47:37.665594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:16:12.268968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461621+0000","last_change":"2024-03-28T17:46:11.760204+0000","last_active":"2024-03-28T17:47:09.461621+0000","last_peered":"2024-03-28T17:47:09.461621+0000","last_clean":"2024-03-28T17:47:09.461621+0000","last_became_active":"2024-03-28T17:46:11.759829+0000","last_became_peered":"2024-03-28T17:46:11.759829+0000","last_unstale":"2024-03-28T17:47:09.461621+0000","last_undegraded":"2024-03-28T17:47:09.461621+0000","last_fullsized":"2024-03-28T17:47:09.461621+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:29:02.075424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.663263+0000","last_change":"2024-03-28T17:46:12.762229+0000","last_active":"2024-03-28T17:47:37.663263+0000","last_peered":"2024-03-28T17:47:37.663263+0000","last_clean":"2024-03-28T17:47:37.663263+0000","last_became_active":"2024-03-28T17:46:12.761889+0000","last_became_peered":"2024-03-28T17:46:12.761889+0000","last_unstale":"2024-03-28T17:47:37.663263+0000","last_undegraded":"2024-03-28T17:47:37.663263+0000","last_fullsized":"2024-03-28T17:47:37.663263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:15.023966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'8","reported_seq":173,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:48:38.124927+0000","last_change":"2024-03-28T17:46:11.763152+0000","last_active":"2024-03-28T17:48:38.124927+0000","last_peered":"2024-03-28T17:48:38.124927+0000","last_clean":"2024-03-28T17:48:38.124927+0000","last_became_active":"2024-03-28T17:46:11.762754+0000","last_became_peered":"2024-03-28T17:46:11.762754+0000","last_unstale":"2024-03-28T17:48:38.124927+0000","last_undegraded":"2024-03-28T17:48:38.124927+0000","last_fullsized":"2024-03-28T17:48:38.124927+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:17:06.659908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1460,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.649470+0000","last_change":"2024-03-28T17:46:12.780701+0000","last_active":"2024-03-28T17:47:37.649470+0000","last_peered":"2024-03-28T17:47:37.649470+0000","last_clean":"2024-03-28T17:47:37.649470+0000","last_became_active":"2024-03-28T17:46:12.780339+0000","last_became_peered":"2024-03-28T17:46:12.780339+0000","last_unstale":"2024-03-28T17:47:37.649470+0000","last_undegraded":"2024-03-28T17:47:37.649470+0000","last_fullsized":"2024-03-28T17:47:37.649470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:59.774888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.950734+0000","last_change":"2024-03-28T17:46:11.763387+0000","last_active":"2024-03-28T17:47:09.950734+0000","last_peered":"2024-03-28T17:47:09.950734+0000","last_clean":"2024-03-28T17:47:09.950734+0000","last_became_active":"2024-03-28T17:46:11.763028+0000","last_became_peered":"2024-03-28T17:46:11.763028+0000","last_unstale":"2024-03-28T17:47:09.950734+0000","last_undegraded":"2024-03-28T17:47:09.950734+0000","last_fullsized":"2024-03-28T17:47:09.950734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:21:31.585047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685166+0000","last_change":"2024-03-28T17:46:12.794167+0000","last_active":"2024-03-28T17:47:37.685166+0000","last_peered":"2024-03-28T17:47:37.685166+0000","last_clean":"2024-03-28T17:47:37.685166+0000","last_became_active":"2024-03-28T17:46:12.793830+0000","last_became_peered":"2024-03-28T17:46:12.793830+0000","last_unstale":"2024-03-28T17:47:37.685166+0000","last_undegraded":"2024-03-28T17:47:37.685166+0000","last_fullsized":"2024-03-28T17:47:37.685166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:20.907752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.470227+0000","last_change":"2024-03-28T17:46:11.769322+0000","last_active":"2024-03-28T17:47:09.470227+0000","last_peered":"2024-03-28T17:47:09.470227+0000","last_clean":"2024-03-28T17:47:09.470227+0000","last_became_active":"2024-03-28T17:46:11.768931+0000","last_became_peered":"2024-03-28T17:46:11.768931+0000","last_unstale":"2024-03-28T17:47:09.470227+0000","last_undegraded":"2024-03-28T17:47:09.470227+0000","last_fullsized":"2024-03-28T17:47:09.470227+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:29:34.372947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683503+0000","last_change":"2024-03-28T17:46:12.781306+0000","last_active":"2024-03-28T17:47:37.683503+0000","last_peered":"2024-03-28T17:47:37.683503+0000","last_clean":"2024-03-28T17:47:37.683503+0000","last_became_active":"2024-03-28T17:46:12.779161+0000","last_became_peered":"2024-03-28T17:46:12.779161+0000","last_unstale":"2024-03-28T17:47:37.683503+0000","last_undegraded":"2024-03-28T17:47:37.683503+0000","last_fullsized":"2024-03-28T17:47:37.683503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:06:10.302067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.464760+0000","last_change":"2024-03-28T17:46:11.762852+0000","last_active":"2024-03-28T17:47:09.464760+0000","last_peered":"2024-03-28T17:47:09.464760+0000","last_clean":"2024-03-28T17:47:09.464760+0000","last_became_active":"2024-03-28T17:46:11.762428+0000","last_became_peered":"2024-03-28T17:46:11.762428+0000","last_unstale":"2024-03-28T17:47:09.464760+0000","last_undegraded":"2024-03-28T17:47:09.464760+0000","last_fullsized":"2024-03-28T17:47:09.464760+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:38:20.433408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.686726+0000","last_change":"2024-03-28T17:46:12.786295+0000","last_active":"2024-03-28T17:47:37.686726+0000","last_peered":"2024-03-28T17:47:37.686726+0000","last_clean":"2024-03-28T17:47:37.686726+0000","last_became_active":"2024-03-28T17:46:12.785894+0000","last_became_peered":"2024-03-28T17:46:12.785894+0000","last_unstale":"2024-03-28T17:47:37.686726+0000","last_undegraded":"2024-03-28T17:47:37.686726+0000","last_fullsized":"2024-03-28T17:47:37.686726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:06.271660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'5","reported_seq":170,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:48:41.128735+0000","last_change":"2024-03-28T17:46:11.766787+0000","last_active":"2024-03-28T17:48:41.128735+0000","last_peered":"2024-03-28T17:48:41.128735+0000","last_clean":"2024-03-28T17:48:41.128735+0000","last_became_active":"2024-03-28T17:46:11.766352+0000","last_became_peered":"2024-03-28T17:46:11.766352+0000","last_unstale":"2024-03-28T17:48:41.128735+0000","last_undegraded":"2024-03-28T17:48:41.128735+0000","last_fullsized":"2024-03-28T17:48:41.128735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:54:48.157597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":145,"num_read_kb":145,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.666318+0000","last_change":"2024-03-28T17:46:12.785724+0000","last_active":"2024-03-28T17:47:37.666318+0000","last_peered":"2024-03-28T17:47:37.666318+0000","last_clean":"2024-03-28T17:47:37.666318+0000","last_became_active":"2024-03-28T17:46:12.785353+0000","last_became_peered":"2024-03-28T17:46:12.785353+0000","last_unstale":"2024-03-28T17:47:37.666318+0000","last_undegraded":"2024-03-28T17:47:37.666318+0000","last_fullsized":"2024-03-28T17:47:37.666318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:21:22.692899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.463127+0000","last_change":"2024-03-28T17:46:11.763507+0000","last_active":"2024-03-28T17:47:09.463127+0000","last_peered":"2024-03-28T17:47:09.463127+0000","last_clean":"2024-03-28T17:47:09.463127+0000","last_became_active":"2024-03-28T17:46:11.763177+0000","last_became_peered":"2024-03-28T17:46:11.763177+0000","last_unstale":"2024-03-28T17:47:09.463127+0000","last_undegraded":"2024-03-28T17:47:09.463127+0000","last_fullsized":"2024-03-28T17:47:09.463127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:56:39.696699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.686049+0000","last_change":"2024-03-28T17:47:09.479162+0000","last_active":"2024-03-28T17:47:37.686049+0000","last_peered":"2024-03-28T17:47:37.686049+0000","last_clean":"2024-03-28T17:47:37.686049+0000","last_became_active":"2024-03-28T17:47:09.478349+0000","last_became_peered":"2024-03-28T17:47:09.478349+0000","last_unstale":"2024-03-28T17:47:37.686049+0000","last_undegraded":"2024-03-28T17:47:37.686049+0000","last_fullsized":"2024-03-28T17:47:37.686049+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:41.056361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.465186+0000","last_change":"2024-03-28T17:46:11.774961+0000","last_active":"2024-03-28T17:47:09.465186+0000","last_peered":"2024-03-28T17:47:09.465186+0000","last_clean":"2024-03-28T17:47:09.465186+0000","last_became_active":"2024-03-28T17:46:11.774645+0000","last_became_peered":"2024-03-28T17:46:11.774645+0000","last_unstale":"2024-03-28T17:47:09.465186+0000","last_undegraded":"2024-03-28T17:47:09.465186+0000","last_fullsized":"2024-03-28T17:47:09.465186+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:38:23.983739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.667529+0000","last_change":"2024-03-28T17:46:12.763698+0000","last_active":"2024-03-28T17:47:37.667529+0000","last_peered":"2024-03-28T17:47:37.667529+0000","last_clean":"2024-03-28T17:47:37.667529+0000","last_became_active":"2024-03-28T17:46:12.763238+0000","last_became_peered":"2024-03-28T17:46:12.763238+0000","last_unstale":"2024-03-28T17:47:37.667529+0000","last_undegraded":"2024-03-28T17:47:37.667529+0000","last_fullsized":"2024-03-28T17:47:37.667529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:12:58.252855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'37","reported_seq":360,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:48:42.130390+0000","last_change":"2024-03-28T17:46:11.763096+0000","last_active":"2024-03-28T17:48:42.130390+0000","last_peered":"2024-03-28T17:48:42.130390+0000","last_clean":"2024-03-28T17:48:42.130390+0000","last_became_active":"2024-03-28T17:46:11.762730+0000","last_became_peered":"2024-03-28T17:46:11.762730+0000","last_unstale":"2024-03-28T17:48:42.130390+0000","last_undegraded":"2024-03-28T17:48:42.130390+0000","last_fullsized":"2024-03-28T17:48:42.130390+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:45:59.469096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68653,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":303,"num_read_kb":219,"num_write":38,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.651631+0000","last_change":"2024-03-28T17:46:12.760366+0000","last_active":"2024-03-28T17:47:37.651631+0000","last_peered":"2024-03-28T17:47:37.651631+0000","last_clean":"2024-03-28T17:47:37.651631+0000","last_became_active":"2024-03-28T17:46:12.759957+0000","last_became_peered":"2024-03-28T17:46:12.759957+0000","last_unstale":"2024-03-28T17:47:37.651631+0000","last_undegraded":"2024-03-28T17:47:37.651631+0000","last_fullsized":"2024-03-28T17:47:37.651631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:08:02.579659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.463545+0000","last_change":"2024-03-28T17:46:11.787108+0000","last_active":"2024-03-28T17:47:09.463545+0000","last_peered":"2024-03-28T17:47:09.463545+0000","last_clean":"2024-03-28T17:47:09.463545+0000","last_became_active":"2024-03-28T17:46:11.786734+0000","last_became_peered":"2024-03-28T17:46:11.786734+0000","last_unstale":"2024-03-28T17:47:09.463545+0000","last_undegraded":"2024-03-28T17:47:09.463545+0000","last_fullsized":"2024-03-28T17:47:09.463545+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:04:00.480545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.678776+0000","last_change":"2024-03-28T17:46:12.786362+0000","last_active":"2024-03-28T17:47:37.678776+0000","last_peered":"2024-03-28T17:47:37.678776+0000","last_clean":"2024-03-28T17:47:37.678776+0000","last_became_active":"2024-03-28T17:46:12.785946+0000","last_became_peered":"2024-03-28T17:46:12.785946+0000","last_unstale":"2024-03-28T17:47:37.678776+0000","last_undegraded":"2024-03-28T17:47:37.678776+0000","last_fullsized":"2024-03-28T17:47:37.678776+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:13:57.857818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.464899+0000","last_change":"2024-03-28T17:46:11.786292+0000","last_active":"2024-03-28T17:47:09.464899+0000","last_peered":"2024-03-28T17:47:09.464899+0000","last_clean":"2024-03-28T17:47:09.464899+0000","last_became_active":"2024-03-28T17:46:11.785953+0000","last_became_peered":"2024-03-28T17:46:11.785953+0000","last_unstale":"2024-03-28T17:47:09.464899+0000","last_undegraded":"2024-03-28T17:47:09.464899+0000","last_fullsized":"2024-03-28T17:47:09.464899+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:02:18.093585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.669899+0000","last_change":"2024-03-28T17:46:12.770669+0000","last_active":"2024-03-28T17:47:37.669899+0000","last_peered":"2024-03-28T17:47:37.669899+0000","last_clean":"2024-03-28T17:47:37.669899+0000","last_became_active":"2024-03-28T17:46:12.770301+0000","last_became_peered":"2024-03-28T17:46:12.770301+0000","last_unstale":"2024-03-28T17:47:37.669899+0000","last_undegraded":"2024-03-28T17:47:37.669899+0000","last_fullsized":"2024-03-28T17:47:37.669899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:59:00.601357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'5","reported_seq":171,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:48:38.226965+0000","last_change":"2024-03-28T17:46:11.759686+0000","last_active":"2024-03-28T17:48:38.226965+0000","last_peered":"2024-03-28T17:48:38.226965+0000","last_clean":"2024-03-28T17:48:38.226965+0000","last_became_active":"2024-03-28T17:46:11.759309+0000","last_became_peered":"2024-03-28T17:46:11.759309+0000","last_unstale":"2024-03-28T17:48:38.226965+0000","last_undegraded":"2024-03-28T17:48:38.226965+0000","last_fullsized":"2024-03-28T17:48:38.226965+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:31:09.999279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.653109+0000","last_change":"2024-03-28T17:46:12.760484+0000","last_active":"2024-03-28T17:47:37.653109+0000","last_peered":"2024-03-28T17:47:37.653109+0000","last_clean":"2024-03-28T17:47:37.653109+0000","last_became_active":"2024-03-28T17:46:12.760121+0000","last_became_peered":"2024-03-28T17:46:12.760121+0000","last_unstale":"2024-03-28T17:47:37.653109+0000","last_undegraded":"2024-03-28T17:47:37.653109+0000","last_fullsized":"2024-03-28T17:47:37.653109+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:33:51.457426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461309+0000","last_change":"2024-03-28T17:46:11.769133+0000","last_active":"2024-03-28T17:47:09.461309+0000","last_peered":"2024-03-28T17:47:09.461309+0000","last_clean":"2024-03-28T17:47:09.461309+0000","last_became_active":"2024-03-28T17:46:11.768857+0000","last_became_peered":"2024-03-28T17:46:11.768857+0000","last_unstale":"2024-03-28T17:47:09.461309+0000","last_undegraded":"2024-03-28T17:47:09.461309+0000","last_fullsized":"2024-03-28T17:47:09.461309+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:19:26.065368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.668453+0000","last_change":"2024-03-28T17:46:12.765246+0000","last_active":"2024-03-28T17:47:37.668453+0000","last_peered":"2024-03-28T17:47:37.668453+0000","last_clean":"2024-03-28T17:47:37.668453+0000","last_became_active":"2024-03-28T17:46:12.764788+0000","last_became_peered":"2024-03-28T17:46:12.764788+0000","last_unstale":"2024-03-28T17:47:37.668453+0000","last_undegraded":"2024-03-28T17:47:37.668453+0000","last_fullsized":"2024-03-28T17:47:37.668453+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:36:22.864337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461779+0000","last_change":"2024-03-28T17:46:11.761005+0000","last_active":"2024-03-28T17:47:09.461779+0000","last_peered":"2024-03-28T17:47:09.461779+0000","last_clean":"2024-03-28T17:47:09.461779+0000","last_became_active":"2024-03-28T17:46:11.760660+0000","last_became_peered":"2024-03-28T17:46:11.760660+0000","last_unstale":"2024-03-28T17:47:09.461779+0000","last_undegraded":"2024-03-28T17:47:09.461779+0000","last_fullsized":"2024-03-28T17:47:09.461779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:01:10.408853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.644551+0000","last_change":"2024-03-28T17:46:12.762851+0000","last_active":"2024-03-28T17:47:37.644551+0000","last_peered":"2024-03-28T17:47:37.644551+0000","last_clean":"2024-03-28T17:47:37.644551+0000","last_became_active":"2024-03-28T17:46:12.762508+0000","last_became_peered":"2024-03-28T17:46:12.762508+0000","last_unstale":"2024-03-28T17:47:37.644551+0000","last_undegraded":"2024-03-28T17:47:37.644551+0000","last_fullsized":"2024-03-28T17:47:37.644551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:55:29.389985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.955449+0000","last_change":"2024-03-28T17:47:09.953352+0000","last_active":"2024-03-28T17:47:09.955449+0000","last_peered":"2024-03-28T17:47:09.955449+0000","last_clean":"2024-03-28T17:47:09.955449+0000","last_became_active":"2024-03-28T17:47:09.952822+0000","last_became_peered":"2024-03-28T17:47:09.952822+0000","last_unstale":"2024-03-28T17:47:09.955449+0000","last_undegraded":"2024-03-28T17:47:09.955449+0000","last_fullsized":"2024-03-28T17:47:09.955449+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:22:48.635040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.684406+0000","last_change":"2024-03-28T17:46:12.794290+0000","last_active":"2024-03-28T17:47:37.684406+0000","last_peered":"2024-03-28T17:47:37.684406+0000","last_clean":"2024-03-28T17:47:37.684406+0000","last_became_active":"2024-03-28T17:46:12.793906+0000","last_became_peered":"2024-03-28T17:46:12.793906+0000","last_unstale":"2024-03-28T17:47:37.684406+0000","last_undegraded":"2024-03-28T17:47:37.684406+0000","last_fullsized":"2024-03-28T17:47:37.684406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:26:04.996019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.463067+0000","last_change":"2024-03-28T17:46:11.762676+0000","last_active":"2024-03-28T17:47:09.463067+0000","last_peered":"2024-03-28T17:47:09.463067+0000","last_clean":"2024-03-28T17:47:09.463067+0000","last_became_active":"2024-03-28T17:46:11.761458+0000","last_became_peered":"2024-03-28T17:46:11.761458+0000","last_unstale":"2024-03-28T17:47:09.463067+0000","last_undegraded":"2024-03-28T17:47:09.463067+0000","last_fullsized":"2024-03-28T17:47:09.463067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:22:26.642798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.686879+0000","last_change":"2024-03-28T17:46:12.787107+0000","last_active":"2024-03-28T17:47:37.686879+0000","last_peered":"2024-03-28T17:47:37.686879+0000","last_clean":"2024-03-28T17:47:37.686879+0000","last_became_active":"2024-03-28T17:46:12.786610+0000","last_became_peered":"2024-03-28T17:46:12.786610+0000","last_unstale":"2024-03-28T17:47:37.686879+0000","last_undegraded":"2024-03-28T17:47:37.686879+0000","last_fullsized":"2024-03-28T17:47:37.686879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:30:14.880483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.665610+0000","last_change":"2024-03-28T17:47:09.645023+0000","last_active":"2024-03-28T17:47:37.665610+0000","last_peered":"2024-03-28T17:47:37.665610+0000","last_clean":"2024-03-28T17:47:37.665610+0000","last_became_active":"2024-03-28T17:47:09.644507+0000","last_became_peered":"2024-03-28T17:47:09.644507+0000","last_unstale":"2024-03-28T17:47:37.665610+0000","last_undegraded":"2024-03-28T17:47:37.665610+0000","last_fullsized":"2024-03-28T17:47:37.665610+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:00:48.144635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"87'5","reported_seq":167,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:48:38.226417+0000","last_change":"2024-03-28T17:46:11.784261+0000","last_active":"2024-03-28T17:48:38.226417+0000","last_peered":"2024-03-28T17:48:38.226417+0000","last_clean":"2024-03-28T17:48:38.226417+0000","last_became_active":"2024-03-28T17:46:11.782957+0000","last_became_peered":"2024-03-28T17:46:11.782957+0000","last_unstale":"2024-03-28T17:48:38.226417+0000","last_undegraded":"2024-03-28T17:48:38.226417+0000","last_fullsized":"2024-03-28T17:48:38.226417+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:08:48.009774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":142,"num_read_kb":142,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687570+0000","last_change":"2024-03-28T17:46:12.784024+0000","last_active":"2024-03-28T17:47:37.687570+0000","last_peered":"2024-03-28T17:47:37.687570+0000","last_clean":"2024-03-28T17:47:37.687570+0000","last_became_active":"2024-03-28T17:46:12.782157+0000","last_became_peered":"2024-03-28T17:46:12.782157+0000","last_unstale":"2024-03-28T17:47:37.687570+0000","last_undegraded":"2024-03-28T17:47:37.687570+0000","last_fullsized":"2024-03-28T17:47:37.687570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:33.679894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.644104+0000","last_change":"2024-03-28T17:46:11.759602+0000","last_active":"2024-03-28T17:47:09.644104+0000","last_peered":"2024-03-28T17:47:09.644104+0000","last_clean":"2024-03-28T17:47:09.644104+0000","last_became_active":"2024-03-28T17:46:11.759248+0000","last_became_peered":"2024-03-28T17:46:11.759248+0000","last_unstale":"2024-03-28T17:47:09.644104+0000","last_undegraded":"2024-03-28T17:47:09.644104+0000","last_fullsized":"2024-03-28T17:47:09.644104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:15:59.997948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":352,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.462874+0000","last_change":"2024-03-28T17:44:51.172493+0000","last_active":"2024-03-28T17:47:09.462874+0000","last_peered":"2024-03-28T17:47:09.462874+0000","last_clean":"2024-03-28T17:47:09.462874+0000","last_became_active":"2024-03-28T17:44:51.172175+0000","last_became_peered":"2024-03-28T17:44:51.172175+0000","last_unstale":"2024-03-28T17:47:09.462874+0000","last_undegraded":"2024-03-28T17:47:09.462874+0000","last_fullsized":"2024-03-28T17:47:09.462874+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:40:54.580958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:40:54.580958+0000","last_clean_scrub_stamp":"2024-03-28T17:40:54.580958+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:34.942490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.669498+0000","last_change":"2024-03-28T17:46:12.766728+0000","last_active":"2024-03-28T17:47:37.669498+0000","last_peered":"2024-03-28T17:47:37.669498+0000","last_clean":"2024-03-28T17:47:37.669498+0000","last_became_active":"2024-03-28T17:46:12.763671+0000","last_became_peered":"2024-03-28T17:46:12.763671+0000","last_unstale":"2024-03-28T17:47:37.669498+0000","last_undegraded":"2024-03-28T17:47:37.669498+0000","last_fullsized":"2024-03-28T17:47:37.669498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:25:39.802551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.462989+0000","last_change":"2024-03-28T17:46:11.763361+0000","last_active":"2024-03-28T17:47:09.462989+0000","last_peered":"2024-03-28T17:47:09.462989+0000","last_clean":"2024-03-28T17:47:09.462989+0000","last_became_active":"2024-03-28T17:46:11.763049+0000","last_became_peered":"2024-03-28T17:46:11.763049+0000","last_unstale":"2024-03-28T17:47:09.462989+0000","last_undegraded":"2024-03-28T17:47:09.462989+0000","last_fullsized":"2024-03-28T17:47:09.462989+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:01.767964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461664+0000","last_change":"2024-03-28T17:46:11.774957+0000","last_active":"2024-03-28T17:47:09.461664+0000","last_peered":"2024-03-28T17:47:09.461664+0000","last_clean":"2024-03-28T17:47:09.461664+0000","last_became_active":"2024-03-28T17:46:11.774658+0000","last_became_peered":"2024-03-28T17:46:11.774658+0000","last_unstale":"2024-03-28T17:47:09.461664+0000","last_undegraded":"2024-03-28T17:47:09.461664+0000","last_fullsized":"2024-03-28T17:47:09.461664+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:40.254318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.675859+0000","last_change":"2024-03-28T17:46:12.778206+0000","last_active":"2024-03-28T17:47:37.675859+0000","last_peered":"2024-03-28T17:47:37.675859+0000","last_clean":"2024-03-28T17:47:37.675859+0000","last_became_active":"2024-03-28T17:46:12.777780+0000","last_became_peered":"2024-03-28T17:46:12.777780+0000","last_unstale":"2024-03-28T17:47:37.675859+0000","last_undegraded":"2024-03-28T17:47:37.675859+0000","last_fullsized":"2024-03-28T17:47:37.675859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:13.241794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'11","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:48:25.066011+0000","last_change":"2024-03-28T17:47:09.961000+0000","last_active":"2024-03-28T17:48:25.066011+0000","last_peered":"2024-03-28T17:48:25.066011+0000","last_clean":"2024-03-28T17:48:25.066011+0000","last_became_active":"2024-03-28T17:47:09.953155+0000","last_became_peered":"2024-03-28T17:47:09.953155+0000","last_unstale":"2024-03-28T17:48:25.066011+0000","last_undegraded":"2024-03-28T17:48:25.066011+0000","last_fullsized":"2024-03-28T17:48:25.066011+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:31:57.258152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683178+0000","last_change":"2024-03-28T17:46:12.790489+0000","last_active":"2024-03-28T17:47:37.683178+0000","last_peered":"2024-03-28T17:47:37.683178+0000","last_clean":"2024-03-28T17:47:37.683178+0000","last_became_active":"2024-03-28T17:46:12.790108+0000","last_became_peered":"2024-03-28T17:46:12.790108+0000","last_unstale":"2024-03-28T17:47:37.683178+0000","last_undegraded":"2024-03-28T17:47:37.683178+0000","last_fullsized":"2024-03-28T17:47:37.683178+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:16:21.210182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.644215+0000","last_change":"2024-03-28T17:46:11.762658+0000","last_active":"2024-03-28T17:47:09.644215+0000","last_peered":"2024-03-28T17:47:09.644215+0000","last_clean":"2024-03-28T17:47:09.644215+0000","last_became_active":"2024-03-28T17:46:11.762296+0000","last_became_peered":"2024-03-28T17:46:11.762296+0000","last_unstale":"2024-03-28T17:47:09.644215+0000","last_undegraded":"2024-03-28T17:47:09.644215+0000","last_fullsized":"2024-03-28T17:47:09.644215+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:42:52.768285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.686169+0000","last_change":"2024-03-28T17:47:09.488306+0000","last_active":"2024-03-28T17:47:37.686169+0000","last_peered":"2024-03-28T17:47:37.686169+0000","last_clean":"2024-03-28T17:47:37.686169+0000","last_became_active":"2024-03-28T17:47:09.487292+0000","last_became_peered":"2024-03-28T17:47:09.487292+0000","last_unstale":"2024-03-28T17:47:37.686169+0000","last_undegraded":"2024-03-28T17:47:37.686169+0000","last_fullsized":"2024-03-28T17:47:37.686169+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:55.612951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.951519+0000","last_change":"2024-03-28T17:46:11.769261+0000","last_active":"2024-03-28T17:47:09.951519+0000","last_peered":"2024-03-28T17:47:09.951519+0000","last_clean":"2024-03-28T17:47:09.951519+0000","last_became_active":"2024-03-28T17:46:11.768918+0000","last_became_peered":"2024-03-28T17:46:11.768918+0000","last_unstale":"2024-03-28T17:47:09.951519+0000","last_undegraded":"2024-03-28T17:47:09.951519+0000","last_fullsized":"2024-03-28T17:47:09.951519+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:26:38.276425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.655991+0000","last_change":"2024-03-28T17:46:12.762617+0000","last_active":"2024-03-28T17:47:37.655991+0000","last_peered":"2024-03-28T17:47:37.655991+0000","last_clean":"2024-03-28T17:47:37.655991+0000","last_became_active":"2024-03-28T17:46:12.762273+0000","last_became_peered":"2024-03-28T17:46:12.762273+0000","last_unstale":"2024-03-28T17:47:37.655991+0000","last_undegraded":"2024-03-28T17:47:37.655991+0000","last_fullsized":"2024-03-28T17:47:37.655991+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:45.848454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.476203+0000","last_change":"2024-03-28T17:46:11.770831+0000","last_active":"2024-03-28T17:47:09.476203+0000","last_peered":"2024-03-28T17:47:09.476203+0000","last_clean":"2024-03-28T17:47:09.476203+0000","last_became_active":"2024-03-28T17:46:11.770303+0000","last_became_peered":"2024-03-28T17:46:11.770303+0000","last_unstale":"2024-03-28T17:47:09.476203+0000","last_undegraded":"2024-03-28T17:47:09.476203+0000","last_fullsized":"2024-03-28T17:47:09.476203+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:48:25.566922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685855+0000","last_change":"2024-03-28T17:46:12.768885+0000","last_active":"2024-03-28T17:47:37.685855+0000","last_peered":"2024-03-28T17:47:37.685855+0000","last_clean":"2024-03-28T17:47:37.685855+0000","last_became_active":"2024-03-28T17:46:12.768525+0000","last_became_peered":"2024-03-28T17:46:12.768525+0000","last_unstale":"2024-03-28T17:47:37.685855+0000","last_undegraded":"2024-03-28T17:47:37.685855+0000","last_fullsized":"2024-03-28T17:47:37.685855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:35:30.779279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:48:24.591545+0000","last_change":"2024-03-28T17:46:11.786661+0000","last_active":"2024-03-28T17:48:24.591545+0000","last_peered":"2024-03-28T17:48:24.591545+0000","last_clean":"2024-03-28T17:48:24.591545+0000","last_became_active":"2024-03-28T17:46:11.786226+0000","last_became_peered":"2024-03-28T17:46:11.786226+0000","last_unstale":"2024-03-28T17:48:24.591545+0000","last_undegraded":"2024-03-28T17:48:24.591545+0000","last_fullsized":"2024-03-28T17:48:24.591545+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:51:24.781368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687652+0000","last_change":"2024-03-28T17:46:12.797531+0000","last_active":"2024-03-28T17:47:37.687652+0000","last_peered":"2024-03-28T17:47:37.687652+0000","last_clean":"2024-03-28T17:47:37.687652+0000","last_became_active":"2024-03-28T17:46:12.797161+0000","last_became_peered":"2024-03-28T17:46:12.797161+0000","last_unstale":"2024-03-28T17:47:37.687652+0000","last_undegraded":"2024-03-28T17:47:37.687652+0000","last_fullsized":"2024-03-28T17:47:37.687652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:01.534666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461493+0000","last_change":"2024-03-28T17:46:11.766589+0000","last_active":"2024-03-28T17:47:09.461493+0000","last_peered":"2024-03-28T17:47:09.461493+0000","last_clean":"2024-03-28T17:47:09.461493+0000","last_became_active":"2024-03-28T17:46:11.766269+0000","last_became_peered":"2024-03-28T17:46:11.766269+0000","last_unstale":"2024-03-28T17:47:09.461493+0000","last_undegraded":"2024-03-28T17:47:09.461493+0000","last_fullsized":"2024-03-28T17:47:09.461493+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:12:26.055262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.670254+0000","last_change":"2024-03-28T17:46:12.765723+0000","last_active":"2024-03-28T17:47:37.670254+0000","last_peered":"2024-03-28T17:47:37.670254+0000","last_clean":"2024-03-28T17:47:37.670254+0000","last_became_active":"2024-03-28T17:46:12.765366+0000","last_became_peered":"2024-03-28T17:46:12.765366+0000","last_unstale":"2024-03-28T17:47:37.670254+0000","last_undegraded":"2024-03-28T17:47:37.670254+0000","last_fullsized":"2024-03-28T17:47:37.670254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:39:55.103246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.463696+0000","last_change":"2024-03-28T17:46:11.767686+0000","last_active":"2024-03-28T17:47:09.463696+0000","last_peered":"2024-03-28T17:47:09.463696+0000","last_clean":"2024-03-28T17:47:09.463696+0000","last_became_active":"2024-03-28T17:46:11.767365+0000","last_became_peered":"2024-03-28T17:46:11.767365+0000","last_unstale":"2024-03-28T17:47:09.463696+0000","last_undegraded":"2024-03-28T17:47:09.463696+0000","last_fullsized":"2024-03-28T17:47:09.463696+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:49.909554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.669367+0000","last_change":"2024-03-28T17:46:12.764129+0000","last_active":"2024-03-28T17:47:37.669367+0000","last_peered":"2024-03-28T17:47:37.669367+0000","last_clean":"2024-03-28T17:47:37.669367+0000","last_became_active":"2024-03-28T17:46:12.763716+0000","last_became_peered":"2024-03-28T17:46:12.763716+0000","last_unstale":"2024-03-28T17:47:37.669367+0000","last_undegraded":"2024-03-28T17:47:37.669367+0000","last_fullsized":"2024-03-28T17:47:37.669367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:08.365578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.464975+0000","last_change":"2024-03-28T17:46:11.787374+0000","last_active":"2024-03-28T17:47:09.464975+0000","last_peered":"2024-03-28T17:47:09.464975+0000","last_clean":"2024-03-28T17:47:09.464975+0000","last_became_active":"2024-03-28T17:46:11.787047+0000","last_became_peered":"2024-03-28T17:46:11.787047+0000","last_unstale":"2024-03-28T17:47:09.464975+0000","last_undegraded":"2024-03-28T17:47:09.464975+0000","last_fullsized":"2024-03-28T17:47:09.464975+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:33.657488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.672808+0000","last_change":"2024-03-28T17:47:09.480465+0000","last_active":"2024-03-28T17:47:37.672808+0000","last_peered":"2024-03-28T17:47:37.672808+0000","last_clean":"2024-03-28T17:47:37.672808+0000","last_became_active":"2024-03-28T17:47:09.475758+0000","last_became_peered":"2024-03-28T17:47:09.475758+0000","last_unstale":"2024-03-28T17:47:37.672808+0000","last_undegraded":"2024-03-28T17:47:37.672808+0000","last_fullsized":"2024-03-28T17:47:37.672808+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:08.610192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.473252+0000","last_change":"2024-03-28T17:46:11.780388+0000","last_active":"2024-03-28T17:47:09.473252+0000","last_peered":"2024-03-28T17:47:09.473252+0000","last_clean":"2024-03-28T17:47:09.473252+0000","last_became_active":"2024-03-28T17:46:11.780016+0000","last_became_peered":"2024-03-28T17:46:11.780016+0000","last_unstale":"2024-03-28T17:47:09.473252+0000","last_undegraded":"2024-03-28T17:47:09.473252+0000","last_fullsized":"2024-03-28T17:47:09.473252+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:51:06.181772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.652005+0000","last_change":"2024-03-28T17:46:12.764748+0000","last_active":"2024-03-28T17:47:37.652005+0000","last_peered":"2024-03-28T17:47:37.652005+0000","last_clean":"2024-03-28T17:47:37.652005+0000","last_became_active":"2024-03-28T17:46:12.764407+0000","last_became_peered":"2024-03-28T17:46:12.764407+0000","last_unstale":"2024-03-28T17:47:37.652005+0000","last_undegraded":"2024-03-28T17:47:37.652005+0000","last_fullsized":"2024-03-28T17:47:37.652005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:41:54.185079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.652775+0000","last_change":"2024-03-28T17:46:12.762486+0000","last_active":"2024-03-28T17:47:37.652775+0000","last_peered":"2024-03-28T17:47:37.652775+0000","last_clean":"2024-03-28T17:47:37.652775+0000","last_became_active":"2024-03-28T17:46:12.762107+0000","last_became_peered":"2024-03-28T17:46:12.762107+0000","last_unstale":"2024-03-28T17:47:37.652775+0000","last_undegraded":"2024-03-28T17:47:37.652775+0000","last_fullsized":"2024-03-28T17:47:37.652775+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:31.062456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.664930+0000","last_change":"2024-03-28T17:46:12.764569+0000","last_active":"2024-03-28T17:47:37.664930+0000","last_peered":"2024-03-28T17:47:37.664930+0000","last_clean":"2024-03-28T17:47:37.664930+0000","last_became_active":"2024-03-28T17:46:12.764112+0000","last_became_peered":"2024-03-28T17:46:12.764112+0000","last_unstale":"2024-03-28T17:47:37.664930+0000","last_undegraded":"2024-03-28T17:47:37.664930+0000","last_fullsized":"2024-03-28T17:47:37.664930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:39:55.526432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.684961+0000","last_change":"2024-03-28T17:46:12.786031+0000","last_active":"2024-03-28T17:47:37.684961+0000","last_peered":"2024-03-28T17:47:37.684961+0000","last_clean":"2024-03-28T17:47:37.684961+0000","last_became_active":"2024-03-28T17:46:12.785679+0000","last_became_peered":"2024-03-28T17:46:12.785679+0000","last_unstale":"2024-03-28T17:47:37.684961+0000","last_undegraded":"2024-03-28T17:47:37.684961+0000","last_fullsized":"2024-03-28T17:47:37.684961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:41:22.537979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685335+0000","last_change":"2024-03-28T17:46:12.793285+0000","last_active":"2024-03-28T17:47:37.685335+0000","last_peered":"2024-03-28T17:47:37.685335+0000","last_clean":"2024-03-28T17:47:37.685335+0000","last_became_active":"2024-03-28T17:46:12.788935+0000","last_became_peered":"2024-03-28T17:46:12.788935+0000","last_unstale":"2024-03-28T17:47:37.685335+0000","last_undegraded":"2024-03-28T17:47:37.685335+0000","last_fullsized":"2024-03-28T17:47:37.685335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:53:58.886607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.688031+0000","last_change":"2024-03-28T17:47:09.486124+0000","last_active":"2024-03-28T17:47:37.688031+0000","last_peered":"2024-03-28T17:47:37.688031+0000","last_clean":"2024-03-28T17:47:37.688031+0000","last_became_active":"2024-03-28T17:47:09.481292+0000","last_became_peered":"2024-03-28T17:47:09.481292+0000","last_unstale":"2024-03-28T17:47:37.688031+0000","last_undegraded":"2024-03-28T17:47:37.688031+0000","last_fullsized":"2024-03-28T17:47:37.688031+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:18.656764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.645280+0000","last_change":"2024-03-28T17:46:12.765140+0000","last_active":"2024-03-28T17:47:37.645280+0000","last_peered":"2024-03-28T17:47:37.645280+0000","last_clean":"2024-03-28T17:47:37.645280+0000","last_became_active":"2024-03-28T17:46:12.764760+0000","last_became_peered":"2024-03-28T17:46:12.764760+0000","last_unstale":"2024-03-28T17:47:37.645280+0000","last_undegraded":"2024-03-28T17:47:37.645280+0000","last_fullsized":"2024-03-28T17:47:37.645280+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:41.576437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.658988+0000","last_change":"2024-03-28T17:46:12.760271+0000","last_active":"2024-03-28T17:47:37.658988+0000","last_peered":"2024-03-28T17:47:37.658988+0000","last_clean":"2024-03-28T17:47:37.658988+0000","last_became_active":"2024-03-28T17:46:12.759939+0000","last_became_peered":"2024-03-28T17:46:12.759939+0000","last_unstale":"2024-03-28T17:47:37.658988+0000","last_undegraded":"2024-03-28T17:47:37.658988+0000","last_fullsized":"2024-03-28T17:47:37.658988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:06:42.529826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.681262+0000","last_change":"2024-03-28T17:46:12.792360+0000","last_active":"2024-03-28T17:47:37.681262+0000","last_peered":"2024-03-28T17:47:37.681262+0000","last_clean":"2024-03-28T17:47:37.681262+0000","last_became_active":"2024-03-28T17:46:12.791979+0000","last_became_peered":"2024-03-28T17:46:12.791979+0000","last_unstale":"2024-03-28T17:47:37.681262+0000","last_undegraded":"2024-03-28T17:47:37.681262+0000","last_fullsized":"2024-03-28T17:47:37.681262+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:39.866969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685176+0000","last_change":"2024-03-28T17:46:12.784228+0000","last_active":"2024-03-28T17:47:37.685176+0000","last_peered":"2024-03-28T17:47:37.685176+0000","last_clean":"2024-03-28T17:47:37.685176+0000","last_became_active":"2024-03-28T17:46:12.783890+0000","last_became_peered":"2024-03-28T17:46:12.783890+0000","last_unstale":"2024-03-28T17:47:37.685176+0000","last_undegraded":"2024-03-28T17:47:37.685176+0000","last_fullsized":"2024-03-28T17:47:37.685176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:58:13.388705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685701+0000","last_change":"2024-03-28T17:46:12.782675+0000","last_active":"2024-03-28T17:47:37.685701+0000","last_peered":"2024-03-28T17:47:37.685701+0000","last_clean":"2024-03-28T17:47:37.685701+0000","last_became_active":"2024-03-28T17:46:12.779255+0000","last_became_peered":"2024-03-28T17:46:12.779255+0000","last_unstale":"2024-03-28T17:47:37.685701+0000","last_undegraded":"2024-03-28T17:47:37.685701+0000","last_fullsized":"2024-03-28T17:47:37.685701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:04:31.447512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.677034+0000","last_change":"2024-03-28T17:46:12.786308+0000","last_active":"2024-03-28T17:47:37.677034+0000","last_peered":"2024-03-28T17:47:37.677034+0000","last_clean":"2024-03-28T17:47:37.677034+0000","last_became_active":"2024-03-28T17:46:12.785948+0000","last_became_peered":"2024-03-28T17:46:12.785948+0000","last_unstale":"2024-03-28T17:47:37.677034+0000","last_undegraded":"2024-03-28T17:47:37.677034+0000","last_fullsized":"2024-03-28T17:47:37.677034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:43.292925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687492+0000","last_change":"2024-03-28T17:46:12.797278+0000","last_active":"2024-03-28T17:47:37.687492+0000","last_peered":"2024-03-28T17:47:37.687492+0000","last_clean":"2024-03-28T17:47:37.687492+0000","last_became_active":"2024-03-28T17:46:12.796593+0000","last_became_peered":"2024-03-28T17:46:12.796593+0000","last_unstale":"2024-03-28T17:47:37.687492+0000","last_undegraded":"2024-03-28T17:47:37.687492+0000","last_fullsized":"2024-03-28T17:47:37.687492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:07:34.433680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.638868+0000","last_change":"2024-03-28T17:46:12.764249+0000","last_active":"2024-03-28T17:47:37.638868+0000","last_peered":"2024-03-28T17:47:37.638868+0000","last_clean":"2024-03-28T17:47:37.638868+0000","last_became_active":"2024-03-28T17:46:12.763909+0000","last_became_peered":"2024-03-28T17:46:12.763909+0000","last_unstale":"2024-03-28T17:47:37.638868+0000","last_undegraded":"2024-03-28T17:47:37.638868+0000","last_fullsized":"2024-03-28T17:47:37.638868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:40:51.780187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250098,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73633,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":896,"num_read_kb":529,"num_write":180,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":211545,"data_compressed":5625,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383725,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":456197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645362,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":1116863,"data_compressed":38073,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808888,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":456550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070525,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":456353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332162,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":1049262,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495688,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":455822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574949,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":1003520,"data_stored":517043,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738475,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":457093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411424,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":519231,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182490,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":456016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920852,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":456791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757326,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":1048414,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34554,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34554,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":66133,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":63616,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":70763,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T17:48:45.399 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-28T17:48:45.399 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T17:48:45.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.640+0000 7fc0fae00640 1 -- 172.21.15.45:0/4265720090 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0f41048a0 msgr2=0x7fc0f4106c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:45.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.640+0000 7fc0fae00640 1 --2- 172.21.15.45:0/4265720090 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0f41048a0 0x7fc0f4106c90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0009900 tx=0x7fc0e002bc30 comp rx=0 tx=0).stop 2024-03-28T17:48:45.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.640+0000 7fc0fae00640 1 -- 172.21.15.45:0/4265720090 shutdown_connections 2024-03-28T17:48:45.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.640+0000 7fc0fae00640 1 --2- 172.21.15.45:0/4265720090 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f41071d0 0x7fc0f4109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:45.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.640+0000 7fc0fae00640 1 --2- 172.21.15.45:0/4265720090 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0f41048a0 0x7fc0f4106c90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:45.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.640+0000 7fc0fae00640 1 --2- 172.21.15.45:0/4265720090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0f4101f70 0x7fc0f4104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:45.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.640+0000 7fc0fae00640 1 -- 172.21.15.45:0/4265720090 >> 172.21.15.45:0/4265720090 conn(0x7fc0f40fb980 msgr2=0x7fc0f40fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:45.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.640+0000 7fc0fae00640 1 -- 172.21.15.45:0/4265720090 shutdown_connections 2024-03-28T17:48:45.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.640+0000 7fc0fae00640 1 -- 172.21.15.45:0/4265720090 wait complete. 2024-03-28T17:48:45.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.641+0000 7fc0fae00640 1 Processor -- start 2024-03-28T17:48:45.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.641+0000 7fc0fae00640 1 -- start start 2024-03-28T17:48:45.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.642+0000 7fc0fae00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0f4101f70 0x7fc0f41977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:45.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.642+0000 7fc0fae00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f41048a0 0x7fc0f4197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:45.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.642+0000 7fc0fae00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0f41071d0 0x7fc0f419eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:45.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.642+0000 7fc0fae00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f410d140 con 0x7fc0f41071d0 2024-03-28T17:48:45.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.642+0000 7fc0fae00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f410cfc0 con 0x7fc0f41048a0 2024-03-28T17:48:45.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.642+0000 7fc0fae00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f410d2c0 con 0x7fc0f4101f70 2024-03-28T17:48:45.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.642+0000 7fc0f9a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0f4101f70 0x7fc0f41977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:45.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.642+0000 7fc0f9a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0f4101f70 0x7fc0f41977f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:49796/0 (socket says 172.21.15.45:49796) 2024-03-28T17:48:45.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.642+0000 7fc0f9a00640 1 -- 172.21.15.45:0/1951184275 learned_addr learned my addr 172.21.15.45:0/1951184275 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:45.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.642+0000 7fc0f9000640 1 --2- 172.21.15.45:0/1951184275 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f41048a0 0x7fc0f4197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:45.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.642+0000 7fc0fa400640 1 --2- 172.21.15.45:0/1951184275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0f41071d0 0x7fc0f419eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:45.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.642+0000 7fc0f9a00640 1 -- 172.21.15.45:0/1951184275 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f41048a0 msgr2=0x7fc0f4197d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:45.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.642+0000 7fc0f9a00640 1 --2- 172.21.15.45:0/1951184275 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f41048a0 0x7fc0f4197d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:45.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.642+0000 7fc0f9a00640 1 -- 172.21.15.45:0/1951184275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0f41071d0 msgr2=0x7fc0f419eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:45.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.642+0000 7fc0f9a00640 1 --2- 172.21.15.45:0/1951184275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0f41071d0 0x7fc0f419eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:45.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.642+0000 7fc0f9a00640 1 -- 172.21.15.45:0/1951184275 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f419f410 con 0x7fc0f4101f70 2024-03-28T17:48:45.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.643+0000 7fc0f9a00640 1 --2- 172.21.15.45:0/1951184275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0f4101f70 0x7fc0f41977f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc0e800e9f0 tx=0x7fc0e800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:45.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.643+0000 7fc0f2000640 1 -- 172.21.15.45:0/1951184275 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e800cd80 con 0x7fc0f4101f70 2024-03-28T17:48:45.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.643+0000 7fc0f2000640 1 -- 172.21.15.45:0/1951184275 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0e8010910 con 0x7fc0f4101f70 2024-03-28T17:48:45.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.643+0000 7fc0f2000640 1 -- 172.21.15.45:0/1951184275 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e8018990 con 0x7fc0f4101f70 2024-03-28T17:48:45.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.643+0000 7fc0fae00640 1 -- 172.21.15.45:0/1951184275 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f419f700 con 0x7fc0f4101f70 2024-03-28T17:48:45.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.643+0000 7fc0fae00640 1 -- 172.21.15.45:0/1951184275 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f419fc70 con 0x7fc0f4101f70 2024-03-28T17:48:45.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.644+0000 7fc0fae00640 1 -- 172.21.15.45:0/1951184275 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0f40fd470 con 0x7fc0f4101f70 2024-03-28T17:48:45.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.646+0000 7fc0f2000640 1 -- 172.21.15.45:0/1951184275 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fc0e8010ac0 con 0x7fc0f4101f70 2024-03-28T17:48:45.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.647+0000 7fc0f2000640 1 --2- 172.21.15.45:0/1951184275 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc0dc0a2910 0x7fc0dc0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:45.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.648+0000 7fc0f2000640 1 -- 172.21.15.45:0/1951184275 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc0e80ceef0 con 0x7fc0f4101f70 2024-03-28T17:48:45.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.648+0000 7fc0f9000640 1 --2- 172.21.15.45:0/1951184275 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc0dc0a2910 0x7fc0dc0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:45.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.648+0000 7fc0f9000640 1 --2- 172.21.15.45:0/1951184275 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc0dc0a2910 0x7fc0dc0a4dd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0000c00 tx=0x7fc0e0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:45.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.656+0000 7fc0f2000640 1 -- 172.21.15.45:0/1951184275 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc0e809b790 con 0x7fc0f4101f70 2024-03-28T17:48:45.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:45 smithi045 ceph-mon[31773]: pgmap v395: 97 pgs: 97 active+clean; 649 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-28T17:48:45.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:45 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:45.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:45 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:45.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:45 smithi151 ceph-mon[36481]: pgmap v395: 97 pgs: 97 active+clean; 649 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-28T17:48:45.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:45 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:45.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:45 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:45.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.863+0000 7fc0fae00640 1 -- 172.21.15.45:0/1951184275 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc0f4061ae0 con 0x7fc0dc0a2910 2024-03-28T17:48:45.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.874+0000 7fc0f2000640 1 -- 172.21.15.45:0/1951184275 <== mgr.14152 v2:172.21.15.45:6800/1246060167 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259043 (secure 0 0 0) 0x7fc0f4061ae0 con 0x7fc0dc0a2910 2024-03-28T17:48:45.878 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:48:45.879 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-28T17:48:45.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.879+0000 7fc0fae00640 1 -- 172.21.15.45:0/1951184275 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc0dc0a2910 msgr2=0x7fc0dc0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:45.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.879+0000 7fc0fae00640 1 --2- 172.21.15.45:0/1951184275 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc0dc0a2910 0x7fc0dc0a4dd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0000c00 tx=0x7fc0e0005fd0 comp rx=0 tx=0).stop 2024-03-28T17:48:45.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.880+0000 7fc0fae00640 1 -- 172.21.15.45:0/1951184275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0f4101f70 msgr2=0x7fc0f41977f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:45.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.880+0000 7fc0fae00640 1 --2- 172.21.15.45:0/1951184275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0f4101f70 0x7fc0f41977f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc0e800e9f0 tx=0x7fc0e800eeb0 comp rx=0 tx=0).stop 2024-03-28T17:48:45.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.880+0000 7fc0fae00640 1 -- 172.21.15.45:0/1951184275 shutdown_connections 2024-03-28T17:48:45.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.880+0000 7fc0fae00640 1 --2- 172.21.15.45:0/1951184275 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fc0dc0a2910 0x7fc0dc0a4dd0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:45.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.880+0000 7fc0fae00640 1 --2- 172.21.15.45:0/1951184275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0f41071d0 0x7fc0f419eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:45.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.880+0000 7fc0fae00640 1 --2- 172.21.15.45:0/1951184275 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f41048a0 0x7fc0f4197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:45.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.880+0000 7fc0fae00640 1 --2- 172.21.15.45:0/1951184275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0f4101f70 0x7fc0f41977f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:45.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.880+0000 7fc0fae00640 1 -- 172.21.15.45:0/1951184275 >> 172.21.15.45:0/1951184275 conn(0x7fc0f40fb980 msgr2=0x7fc0f41029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:45.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.880+0000 7fc0fae00640 1 -- 172.21.15.45:0/1951184275 shutdown_connections 2024-03-28T17:48:45.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:45.880+0000 7fc0fae00640 1 -- 172.21.15.45:0/1951184275 wait complete. 2024-03-28T17:48:45.904 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2024-03-28T17:48:45.709209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664626,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":1048,"num_read_kb":654,"num_write":368,"num_write_kb":3076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180853,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":797,"ondisk_log_size":797,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":364884,"kb_used_data":11572,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576940,"statfs":{"total":1151940427776,"available":1151566786560,"internally_reserved":0,"allocated":11849728,"data_stored":7454033,"data_compressed":114219,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8228,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":79,"num_read_kb":46,"num_write":15,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004000"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683908+0000","last_change":"2024-03-28T17:46:12.783038+0000","last_active":"2024-03-28T17:47:37.683908+0000","last_peered":"2024-03-28T17:47:37.683908+0000","last_clean":"2024-03-28T17:47:37.683908+0000","last_became_active":"2024-03-28T17:46:12.782668+0000","last_became_peered":"2024-03-28T17:46:12.782668+0000","last_unstale":"2024-03-28T17:47:37.683908+0000","last_undegraded":"2024-03-28T17:47:37.683908+0000","last_fullsized":"2024-03-28T17:47:37.683908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:03.424959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683564+0000","last_change":"2024-03-28T17:46:12.781828+0000","last_active":"2024-03-28T17:47:37.683564+0000","last_peered":"2024-03-28T17:47:37.683564+0000","last_clean":"2024-03-28T17:47:37.683564+0000","last_became_active":"2024-03-28T17:46:12.781424+0000","last_became_peered":"2024-03-28T17:46:12.781424+0000","last_unstale":"2024-03-28T17:47:37.683564+0000","last_undegraded":"2024-03-28T17:47:37.683564+0000","last_fullsized":"2024-03-28T17:47:37.683564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:15:16.639473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683268+0000","last_change":"2024-03-28T17:46:12.785431+0000","last_active":"2024-03-28T17:47:37.683268+0000","last_peered":"2024-03-28T17:47:37.683268+0000","last_clean":"2024-03-28T17:47:37.683268+0000","last_became_active":"2024-03-28T17:46:12.784992+0000","last_became_peered":"2024-03-28T17:46:12.784992+0000","last_unstale":"2024-03-28T17:47:37.683268+0000","last_undegraded":"2024-03-28T17:47:37.683268+0000","last_fullsized":"2024-03-28T17:47:37.683268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:03:36.926613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.663175+0000","last_change":"2024-03-28T17:46:12.765274+0000","last_active":"2024-03-28T17:47:37.663175+0000","last_peered":"2024-03-28T17:47:37.663175+0000","last_clean":"2024-03-28T17:47:37.663175+0000","last_became_active":"2024-03-28T17:46:12.764890+0000","last_became_peered":"2024-03-28T17:46:12.764890+0000","last_unstale":"2024-03-28T17:47:37.663175+0000","last_undegraded":"2024-03-28T17:47:37.663175+0000","last_fullsized":"2024-03-28T17:47:37.663175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:58:37.149756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.670545+0000","last_change":"2024-03-28T17:46:12.782791+0000","last_active":"2024-03-28T17:47:37.670545+0000","last_peered":"2024-03-28T17:47:37.670545+0000","last_clean":"2024-03-28T17:47:37.670545+0000","last_became_active":"2024-03-28T17:46:12.782420+0000","last_became_peered":"2024-03-28T17:46:12.782420+0000","last_unstale":"2024-03-28T17:47:37.670545+0000","last_undegraded":"2024-03-28T17:47:37.670545+0000","last_fullsized":"2024-03-28T17:47:37.670545+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:30.512863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.657353+0000","last_change":"2024-03-28T17:46:12.780713+0000","last_active":"2024-03-28T17:47:37.657353+0000","last_peered":"2024-03-28T17:47:37.657353+0000","last_clean":"2024-03-28T17:47:37.657353+0000","last_became_active":"2024-03-28T17:46:12.780305+0000","last_became_peered":"2024-03-28T17:46:12.780305+0000","last_unstale":"2024-03-28T17:47:37.657353+0000","last_undegraded":"2024-03-28T17:47:37.657353+0000","last_fullsized":"2024-03-28T17:47:37.657353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:08:09.840491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.653244+0000","last_change":"2024-03-28T17:46:12.778120+0000","last_active":"2024-03-28T17:47:37.653244+0000","last_peered":"2024-03-28T17:47:37.653244+0000","last_clean":"2024-03-28T17:47:37.653244+0000","last_became_active":"2024-03-28T17:46:12.777724+0000","last_became_peered":"2024-03-28T17:46:12.777724+0000","last_unstale":"2024-03-28T17:47:37.653244+0000","last_undegraded":"2024-03-28T17:47:37.653244+0000","last_fullsized":"2024-03-28T17:47:37.653244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:29:50.980366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687911+0000","last_change":"2024-03-28T17:46:12.793447+0000","last_active":"2024-03-28T17:47:37.687911+0000","last_peered":"2024-03-28T17:47:37.687911+0000","last_clean":"2024-03-28T17:47:37.687911+0000","last_became_active":"2024-03-28T17:46:12.793070+0000","last_became_peered":"2024-03-28T17:46:12.793070+0000","last_unstale":"2024-03-28T17:47:37.687911+0000","last_undegraded":"2024-03-28T17:47:37.687911+0000","last_fullsized":"2024-03-28T17:47:37.687911+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:01.364577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.668936+0000","last_change":"2024-03-28T17:46:12.763908+0000","last_active":"2024-03-28T17:47:37.668936+0000","last_peered":"2024-03-28T17:47:37.668936+0000","last_clean":"2024-03-28T17:47:37.668936+0000","last_became_active":"2024-03-28T17:46:12.763267+0000","last_became_peered":"2024-03-28T17:46:12.763267+0000","last_unstale":"2024-03-28T17:47:37.668936+0000","last_undegraded":"2024-03-28T17:47:37.668936+0000","last_fullsized":"2024-03-28T17:47:37.668936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:10.692974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.660790+0000","last_change":"2024-03-28T17:46:12.762648+0000","last_active":"2024-03-28T17:47:37.660790+0000","last_peered":"2024-03-28T17:47:37.660790+0000","last_clean":"2024-03-28T17:47:37.660790+0000","last_became_active":"2024-03-28T17:46:12.762300+0000","last_became_peered":"2024-03-28T17:46:12.762300+0000","last_unstale":"2024-03-28T17:47:37.660790+0000","last_undegraded":"2024-03-28T17:47:37.660790+0000","last_fullsized":"2024-03-28T17:47:37.660790+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:02.210828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.668940+0000","last_change":"2024-03-28T17:46:12.766482+0000","last_active":"2024-03-28T17:47:37.668940+0000","last_peered":"2024-03-28T17:47:37.668940+0000","last_clean":"2024-03-28T17:47:37.668940+0000","last_became_active":"2024-03-28T17:46:12.766155+0000","last_became_peered":"2024-03-28T17:46:12.766155+0000","last_unstale":"2024-03-28T17:47:37.668940+0000","last_undegraded":"2024-03-28T17:47:37.668940+0000","last_fullsized":"2024-03-28T17:47:37.668940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:46:24.261744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.643025+0000","last_change":"2024-03-28T17:46:12.766100+0000","last_active":"2024-03-28T17:47:37.643025+0000","last_peered":"2024-03-28T17:47:37.643025+0000","last_clean":"2024-03-28T17:47:37.643025+0000","last_became_active":"2024-03-28T17:46:12.765762+0000","last_became_peered":"2024-03-28T17:46:12.765762+0000","last_unstale":"2024-03-28T17:47:37.643025+0000","last_undegraded":"2024-03-28T17:47:37.643025+0000","last_fullsized":"2024-03-28T17:47:37.643025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:11.666653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.670019+0000","last_change":"2024-03-28T17:46:12.753465+0000","last_active":"2024-03-28T17:47:37.670019+0000","last_peered":"2024-03-28T17:47:37.670019+0000","last_clean":"2024-03-28T17:47:37.670019+0000","last_became_active":"2024-03-28T17:46:12.751931+0000","last_became_peered":"2024-03-28T17:46:12.751931+0000","last_unstale":"2024-03-28T17:47:37.670019+0000","last_undegraded":"2024-03-28T17:47:37.670019+0000","last_fullsized":"2024-03-28T17:47:37.670019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:39:38.757878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.684532+0000","last_change":"2024-03-28T17:46:12.784177+0000","last_active":"2024-03-28T17:47:37.684532+0000","last_peered":"2024-03-28T17:47:37.684532+0000","last_clean":"2024-03-28T17:47:37.684532+0000","last_became_active":"2024-03-28T17:46:12.783823+0000","last_became_peered":"2024-03-28T17:46:12.783823+0000","last_unstale":"2024-03-28T17:47:37.684532+0000","last_undegraded":"2024-03-28T17:47:37.684532+0000","last_fullsized":"2024-03-28T17:47:37.684532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:11:33.221095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685187+0000","last_change":"2024-03-28T17:46:12.777152+0000","last_active":"2024-03-28T17:47:37.685187+0000","last_peered":"2024-03-28T17:47:37.685187+0000","last_clean":"2024-03-28T17:47:37.685187+0000","last_became_active":"2024-03-28T17:46:12.776758+0000","last_became_peered":"2024-03-28T17:46:12.776758+0000","last_unstale":"2024-03-28T17:47:37.685187+0000","last_undegraded":"2024-03-28T17:47:37.685187+0000","last_fullsized":"2024-03-28T17:47:37.685187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:20:56.587470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.686334+0000","last_change":"2024-03-28T17:46:12.782384+0000","last_active":"2024-03-28T17:47:37.686334+0000","last_peered":"2024-03-28T17:47:37.686334+0000","last_clean":"2024-03-28T17:47:37.686334+0000","last_became_active":"2024-03-28T17:46:12.781971+0000","last_became_peered":"2024-03-28T17:46:12.781971+0000","last_unstale":"2024-03-28T17:47:37.686334+0000","last_undegraded":"2024-03-28T17:47:37.686334+0000","last_fullsized":"2024-03-28T17:47:37.686334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:19:32.652629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685920+0000","last_change":"2024-03-28T17:46:12.789364+0000","last_active":"2024-03-28T17:47:37.685920+0000","last_peered":"2024-03-28T17:47:37.685920+0000","last_clean":"2024-03-28T17:47:37.685920+0000","last_became_active":"2024-03-28T17:46:12.788976+0000","last_became_peered":"2024-03-28T17:46:12.788976+0000","last_unstale":"2024-03-28T17:47:37.685920+0000","last_undegraded":"2024-03-28T17:47:37.685920+0000","last_fullsized":"2024-03-28T17:47:37.685920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:43:37.183705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.663596+0000","last_change":"2024-03-28T17:46:12.782664+0000","last_active":"2024-03-28T17:47:37.663596+0000","last_peered":"2024-03-28T17:47:37.663596+0000","last_clean":"2024-03-28T17:47:37.663596+0000","last_became_active":"2024-03-28T17:46:12.782305+0000","last_became_peered":"2024-03-28T17:46:12.782305+0000","last_unstale":"2024-03-28T17:47:37.663596+0000","last_undegraded":"2024-03-28T17:47:37.663596+0000","last_fullsized":"2024-03-28T17:47:37.663596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:47:18.373517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.684368+0000","last_change":"2024-03-28T17:46:12.794494+0000","last_active":"2024-03-28T17:47:37.684368+0000","last_peered":"2024-03-28T17:47:37.684368+0000","last_clean":"2024-03-28T17:47:37.684368+0000","last_became_active":"2024-03-28T17:46:12.794151+0000","last_became_peered":"2024-03-28T17:46:12.794151+0000","last_unstale":"2024-03-28T17:47:37.684368+0000","last_undegraded":"2024-03-28T17:47:37.684368+0000","last_fullsized":"2024-03-28T17:47:37.684368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:17:24.596333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.475806+0000","last_change":"2024-03-28T17:46:11.759643+0000","last_active":"2024-03-28T17:47:09.475806+0000","last_peered":"2024-03-28T17:47:09.475806+0000","last_clean":"2024-03-28T17:47:09.475806+0000","last_became_active":"2024-03-28T17:46:11.759273+0000","last_became_peered":"2024-03-28T17:46:11.759273+0000","last_unstale":"2024-03-28T17:47:09.475806+0000","last_undegraded":"2024-03-28T17:47:09.475806+0000","last_fullsized":"2024-03-28T17:47:09.475806+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:09:51.764471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687168+0000","last_change":"2024-03-28T17:46:12.786503+0000","last_active":"2024-03-28T17:47:37.687168+0000","last_peered":"2024-03-28T17:47:37.687168+0000","last_clean":"2024-03-28T17:47:37.687168+0000","last_became_active":"2024-03-28T17:46:12.784651+0000","last_became_peered":"2024-03-28T17:46:12.784651+0000","last_unstale":"2024-03-28T17:47:37.687168+0000","last_undegraded":"2024-03-28T17:47:37.687168+0000","last_fullsized":"2024-03-28T17:47:37.687168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:30:22.401767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.480732+0000","last_change":"2024-03-28T17:46:11.780340+0000","last_active":"2024-03-28T17:47:09.480732+0000","last_peered":"2024-03-28T17:47:09.480732+0000","last_clean":"2024-03-28T17:47:09.480732+0000","last_became_active":"2024-03-28T17:46:11.779917+0000","last_became_peered":"2024-03-28T17:46:11.779917+0000","last_unstale":"2024-03-28T17:47:09.480732+0000","last_undegraded":"2024-03-28T17:47:09.480732+0000","last_fullsized":"2024-03-28T17:47:09.480732+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:22:45.617444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.686129+0000","last_change":"2024-03-28T17:46:12.795086+0000","last_active":"2024-03-28T17:47:37.686129+0000","last_peered":"2024-03-28T17:47:37.686129+0000","last_clean":"2024-03-28T17:47:37.686129+0000","last_became_active":"2024-03-28T17:46:12.794124+0000","last_became_peered":"2024-03-28T17:46:12.794124+0000","last_unstale":"2024-03-28T17:47:37.686129+0000","last_undegraded":"2024-03-28T17:47:37.686129+0000","last_fullsized":"2024-03-28T17:47:37.686129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:11:05.347047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.648659+0000","last_change":"2024-03-28T17:47:09.648328+0000","last_active":"2024-03-28T17:47:09.648659+0000","last_peered":"2024-03-28T17:47:09.648659+0000","last_clean":"2024-03-28T17:47:09.648659+0000","last_became_active":"2024-03-28T17:47:09.644797+0000","last_became_peered":"2024-03-28T17:47:09.644797+0000","last_unstale":"2024-03-28T17:47:09.648659+0000","last_undegraded":"2024-03-28T17:47:09.648659+0000","last_fullsized":"2024-03-28T17:47:09.648659+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:14:30.233156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.665594+0000","last_change":"2024-03-28T17:46:12.768118+0000","last_active":"2024-03-28T17:47:37.665594+0000","last_peered":"2024-03-28T17:47:37.665594+0000","last_clean":"2024-03-28T17:47:37.665594+0000","last_became_active":"2024-03-28T17:46:12.767765+0000","last_became_peered":"2024-03-28T17:46:12.767765+0000","last_unstale":"2024-03-28T17:47:37.665594+0000","last_undegraded":"2024-03-28T17:47:37.665594+0000","last_fullsized":"2024-03-28T17:47:37.665594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:16:12.268968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461675+0000","last_change":"2024-03-28T17:46:11.760204+0000","last_active":"2024-03-28T17:47:09.461675+0000","last_peered":"2024-03-28T17:47:09.461675+0000","last_clean":"2024-03-28T17:47:09.461675+0000","last_became_active":"2024-03-28T17:46:11.759829+0000","last_became_peered":"2024-03-28T17:46:11.759829+0000","last_unstale":"2024-03-28T17:47:09.461675+0000","last_undegraded":"2024-03-28T17:47:09.461675+0000","last_fullsized":"2024-03-28T17:47:09.461675+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:29:02.075424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.668433+0000","last_change":"2024-03-28T17:46:12.762229+0000","last_active":"2024-03-28T17:47:37.668433+0000","last_peered":"2024-03-28T17:47:37.668433+0000","last_clean":"2024-03-28T17:47:37.668433+0000","last_became_active":"2024-03-28T17:46:12.761889+0000","last_became_peered":"2024-03-28T17:46:12.761889+0000","last_unstale":"2024-03-28T17:47:37.668433+0000","last_undegraded":"2024-03-28T17:47:37.668433+0000","last_fullsized":"2024-03-28T17:47:37.668433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:15.023966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'9","reported_seq":181,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:48:43.131178+0000","last_change":"2024-03-28T17:46:11.763152+0000","last_active":"2024-03-28T17:48:43.131178+0000","last_peered":"2024-03-28T17:48:43.131178+0000","last_clean":"2024-03-28T17:48:43.131178+0000","last_became_active":"2024-03-28T17:46:11.762754+0000","last_became_peered":"2024-03-28T17:46:11.762754+0000","last_unstale":"2024-03-28T17:48:43.131178+0000","last_undegraded":"2024-03-28T17:48:43.131178+0000","last_fullsized":"2024-03-28T17:48:43.131178+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:17:06.659908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":152,"num_read_kb":6,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.650191+0000","last_change":"2024-03-28T17:46:12.780701+0000","last_active":"2024-03-28T17:47:37.650191+0000","last_peered":"2024-03-28T17:47:37.650191+0000","last_clean":"2024-03-28T17:47:37.650191+0000","last_became_active":"2024-03-28T17:46:12.780339+0000","last_became_peered":"2024-03-28T17:46:12.780339+0000","last_unstale":"2024-03-28T17:47:37.650191+0000","last_undegraded":"2024-03-28T17:47:37.650191+0000","last_fullsized":"2024-03-28T17:47:37.650191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:59.774888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.950734+0000","last_change":"2024-03-28T17:46:11.763387+0000","last_active":"2024-03-28T17:47:09.950734+0000","last_peered":"2024-03-28T17:47:09.950734+0000","last_clean":"2024-03-28T17:47:09.950734+0000","last_became_active":"2024-03-28T17:46:11.763028+0000","last_became_peered":"2024-03-28T17:46:11.763028+0000","last_unstale":"2024-03-28T17:47:09.950734+0000","last_undegraded":"2024-03-28T17:47:09.950734+0000","last_fullsized":"2024-03-28T17:47:09.950734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:21:31.585047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685257+0000","last_change":"2024-03-28T17:46:12.794167+0000","last_active":"2024-03-28T17:47:37.685257+0000","last_peered":"2024-03-28T17:47:37.685257+0000","last_clean":"2024-03-28T17:47:37.685257+0000","last_became_active":"2024-03-28T17:46:12.793830+0000","last_became_peered":"2024-03-28T17:46:12.793830+0000","last_unstale":"2024-03-28T17:47:37.685257+0000","last_undegraded":"2024-03-28T17:47:37.685257+0000","last_fullsized":"2024-03-28T17:47:37.685257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:20.907752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.470227+0000","last_change":"2024-03-28T17:46:11.769322+0000","last_active":"2024-03-28T17:47:09.470227+0000","last_peered":"2024-03-28T17:47:09.470227+0000","last_clean":"2024-03-28T17:47:09.470227+0000","last_became_active":"2024-03-28T17:46:11.768931+0000","last_became_peered":"2024-03-28T17:46:11.768931+0000","last_unstale":"2024-03-28T17:47:09.470227+0000","last_undegraded":"2024-03-28T17:47:09.470227+0000","last_fullsized":"2024-03-28T17:47:09.470227+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:29:34.372947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683503+0000","last_change":"2024-03-28T17:46:12.781306+0000","last_active":"2024-03-28T17:47:37.683503+0000","last_peered":"2024-03-28T17:47:37.683503+0000","last_clean":"2024-03-28T17:47:37.683503+0000","last_became_active":"2024-03-28T17:46:12.779161+0000","last_became_peered":"2024-03-28T17:46:12.779161+0000","last_unstale":"2024-03-28T17:47:37.683503+0000","last_undegraded":"2024-03-28T17:47:37.683503+0000","last_fullsized":"2024-03-28T17:47:37.683503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:06:10.302067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.464760+0000","last_change":"2024-03-28T17:46:11.762852+0000","last_active":"2024-03-28T17:47:09.464760+0000","last_peered":"2024-03-28T17:47:09.464760+0000","last_clean":"2024-03-28T17:47:09.464760+0000","last_became_active":"2024-03-28T17:46:11.762428+0000","last_became_peered":"2024-03-28T17:46:11.762428+0000","last_unstale":"2024-03-28T17:47:09.464760+0000","last_undegraded":"2024-03-28T17:47:09.464760+0000","last_fullsized":"2024-03-28T17:47:09.464760+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:38:20.433408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.686726+0000","last_change":"2024-03-28T17:46:12.786295+0000","last_active":"2024-03-28T17:47:37.686726+0000","last_peered":"2024-03-28T17:47:37.686726+0000","last_clean":"2024-03-28T17:47:37.686726+0000","last_became_active":"2024-03-28T17:46:12.785894+0000","last_became_peered":"2024-03-28T17:46:12.785894+0000","last_unstale":"2024-03-28T17:47:37.686726+0000","last_undegraded":"2024-03-28T17:47:37.686726+0000","last_fullsized":"2024-03-28T17:47:37.686726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:06.271660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'5","reported_seq":170,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:48:41.128735+0000","last_change":"2024-03-28T17:46:11.766787+0000","last_active":"2024-03-28T17:48:41.128735+0000","last_peered":"2024-03-28T17:48:41.128735+0000","last_clean":"2024-03-28T17:48:41.128735+0000","last_became_active":"2024-03-28T17:46:11.766352+0000","last_became_peered":"2024-03-28T17:46:11.766352+0000","last_unstale":"2024-03-28T17:48:41.128735+0000","last_undegraded":"2024-03-28T17:48:41.128735+0000","last_fullsized":"2024-03-28T17:48:41.128735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:54:48.157597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":145,"num_read_kb":145,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.666318+0000","last_change":"2024-03-28T17:46:12.785724+0000","last_active":"2024-03-28T17:47:37.666318+0000","last_peered":"2024-03-28T17:47:37.666318+0000","last_clean":"2024-03-28T17:47:37.666318+0000","last_became_active":"2024-03-28T17:46:12.785353+0000","last_became_peered":"2024-03-28T17:46:12.785353+0000","last_unstale":"2024-03-28T17:47:37.666318+0000","last_undegraded":"2024-03-28T17:47:37.666318+0000","last_fullsized":"2024-03-28T17:47:37.666318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:21:22.692899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.463127+0000","last_change":"2024-03-28T17:46:11.763507+0000","last_active":"2024-03-28T17:47:09.463127+0000","last_peered":"2024-03-28T17:47:09.463127+0000","last_clean":"2024-03-28T17:47:09.463127+0000","last_became_active":"2024-03-28T17:46:11.763177+0000","last_became_peered":"2024-03-28T17:46:11.763177+0000","last_unstale":"2024-03-28T17:47:09.463127+0000","last_undegraded":"2024-03-28T17:47:09.463127+0000","last_fullsized":"2024-03-28T17:47:09.463127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:56:39.696699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.686049+0000","last_change":"2024-03-28T17:47:09.479162+0000","last_active":"2024-03-28T17:47:37.686049+0000","last_peered":"2024-03-28T17:47:37.686049+0000","last_clean":"2024-03-28T17:47:37.686049+0000","last_became_active":"2024-03-28T17:47:09.478349+0000","last_became_peered":"2024-03-28T17:47:09.478349+0000","last_unstale":"2024-03-28T17:47:37.686049+0000","last_undegraded":"2024-03-28T17:47:37.686049+0000","last_fullsized":"2024-03-28T17:47:37.686049+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:41.056361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.465186+0000","last_change":"2024-03-28T17:46:11.774961+0000","last_active":"2024-03-28T17:47:09.465186+0000","last_peered":"2024-03-28T17:47:09.465186+0000","last_clean":"2024-03-28T17:47:09.465186+0000","last_became_active":"2024-03-28T17:46:11.774645+0000","last_became_peered":"2024-03-28T17:46:11.774645+0000","last_unstale":"2024-03-28T17:47:09.465186+0000","last_undegraded":"2024-03-28T17:47:09.465186+0000","last_fullsized":"2024-03-28T17:47:09.465186+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:38:23.983739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.667529+0000","last_change":"2024-03-28T17:46:12.763698+0000","last_active":"2024-03-28T17:47:37.667529+0000","last_peered":"2024-03-28T17:47:37.667529+0000","last_clean":"2024-03-28T17:47:37.667529+0000","last_became_active":"2024-03-28T17:46:12.763238+0000","last_became_peered":"2024-03-28T17:46:12.763238+0000","last_unstale":"2024-03-28T17:47:37.667529+0000","last_undegraded":"2024-03-28T17:47:37.667529+0000","last_fullsized":"2024-03-28T17:47:37.667529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:12:58.252855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'37","reported_seq":360,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:48:42.130390+0000","last_change":"2024-03-28T17:46:11.763096+0000","last_active":"2024-03-28T17:48:42.130390+0000","last_peered":"2024-03-28T17:48:42.130390+0000","last_clean":"2024-03-28T17:48:42.130390+0000","last_became_active":"2024-03-28T17:46:11.762730+0000","last_became_peered":"2024-03-28T17:46:11.762730+0000","last_unstale":"2024-03-28T17:48:42.130390+0000","last_undegraded":"2024-03-28T17:48:42.130390+0000","last_fullsized":"2024-03-28T17:48:42.130390+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:45:59.469096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68653,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":303,"num_read_kb":219,"num_write":38,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.651631+0000","last_change":"2024-03-28T17:46:12.760366+0000","last_active":"2024-03-28T17:47:37.651631+0000","last_peered":"2024-03-28T17:47:37.651631+0000","last_clean":"2024-03-28T17:47:37.651631+0000","last_became_active":"2024-03-28T17:46:12.759957+0000","last_became_peered":"2024-03-28T17:46:12.759957+0000","last_unstale":"2024-03-28T17:47:37.651631+0000","last_undegraded":"2024-03-28T17:47:37.651631+0000","last_fullsized":"2024-03-28T17:47:37.651631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:08:02.579659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.463545+0000","last_change":"2024-03-28T17:46:11.787108+0000","last_active":"2024-03-28T17:47:09.463545+0000","last_peered":"2024-03-28T17:47:09.463545+0000","last_clean":"2024-03-28T17:47:09.463545+0000","last_became_active":"2024-03-28T17:46:11.786734+0000","last_became_peered":"2024-03-28T17:46:11.786734+0000","last_unstale":"2024-03-28T17:47:09.463545+0000","last_undegraded":"2024-03-28T17:47:09.463545+0000","last_fullsized":"2024-03-28T17:47:09.463545+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:04:00.480545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.678776+0000","last_change":"2024-03-28T17:46:12.786362+0000","last_active":"2024-03-28T17:47:37.678776+0000","last_peered":"2024-03-28T17:47:37.678776+0000","last_clean":"2024-03-28T17:47:37.678776+0000","last_became_active":"2024-03-28T17:46:12.785946+0000","last_became_peered":"2024-03-28T17:46:12.785946+0000","last_unstale":"2024-03-28T17:47:37.678776+0000","last_undegraded":"2024-03-28T17:47:37.678776+0000","last_fullsized":"2024-03-28T17:47:37.678776+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:13:57.857818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.464899+0000","last_change":"2024-03-28T17:46:11.786292+0000","last_active":"2024-03-28T17:47:09.464899+0000","last_peered":"2024-03-28T17:47:09.464899+0000","last_clean":"2024-03-28T17:47:09.464899+0000","last_became_active":"2024-03-28T17:46:11.785953+0000","last_became_peered":"2024-03-28T17:46:11.785953+0000","last_unstale":"2024-03-28T17:47:09.464899+0000","last_undegraded":"2024-03-28T17:47:09.464899+0000","last_fullsized":"2024-03-28T17:47:09.464899+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:02:18.093585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.669899+0000","last_change":"2024-03-28T17:46:12.770669+0000","last_active":"2024-03-28T17:47:37.669899+0000","last_peered":"2024-03-28T17:47:37.669899+0000","last_clean":"2024-03-28T17:47:37.669899+0000","last_became_active":"2024-03-28T17:46:12.770301+0000","last_became_peered":"2024-03-28T17:46:12.770301+0000","last_unstale":"2024-03-28T17:47:37.669899+0000","last_undegraded":"2024-03-28T17:47:37.669899+0000","last_fullsized":"2024-03-28T17:47:37.669899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:59:00.601357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'5","reported_seq":176,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:48:43.236550+0000","last_change":"2024-03-28T17:46:11.759686+0000","last_active":"2024-03-28T17:48:43.236550+0000","last_peered":"2024-03-28T17:48:43.236550+0000","last_clean":"2024-03-28T17:48:43.236550+0000","last_became_active":"2024-03-28T17:46:11.759309+0000","last_became_peered":"2024-03-28T17:46:11.759309+0000","last_unstale":"2024-03-28T17:48:43.236550+0000","last_undegraded":"2024-03-28T17:48:43.236550+0000","last_fullsized":"2024-03-28T17:48:43.236550+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:31:09.999279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":152,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.653109+0000","last_change":"2024-03-28T17:46:12.760484+0000","last_active":"2024-03-28T17:47:37.653109+0000","last_peered":"2024-03-28T17:47:37.653109+0000","last_clean":"2024-03-28T17:47:37.653109+0000","last_became_active":"2024-03-28T17:46:12.760121+0000","last_became_peered":"2024-03-28T17:46:12.760121+0000","last_unstale":"2024-03-28T17:47:37.653109+0000","last_undegraded":"2024-03-28T17:47:37.653109+0000","last_fullsized":"2024-03-28T17:47:37.653109+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:33:51.457426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461309+0000","last_change":"2024-03-28T17:46:11.769133+0000","last_active":"2024-03-28T17:47:09.461309+0000","last_peered":"2024-03-28T17:47:09.461309+0000","last_clean":"2024-03-28T17:47:09.461309+0000","last_became_active":"2024-03-28T17:46:11.768857+0000","last_became_peered":"2024-03-28T17:46:11.768857+0000","last_unstale":"2024-03-28T17:47:09.461309+0000","last_undegraded":"2024-03-28T17:47:09.461309+0000","last_fullsized":"2024-03-28T17:47:09.461309+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:19:26.065368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.668453+0000","last_change":"2024-03-28T17:46:12.765246+0000","last_active":"2024-03-28T17:47:37.668453+0000","last_peered":"2024-03-28T17:47:37.668453+0000","last_clean":"2024-03-28T17:47:37.668453+0000","last_became_active":"2024-03-28T17:46:12.764788+0000","last_became_peered":"2024-03-28T17:46:12.764788+0000","last_unstale":"2024-03-28T17:47:37.668453+0000","last_undegraded":"2024-03-28T17:47:37.668453+0000","last_fullsized":"2024-03-28T17:47:37.668453+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:36:22.864337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461779+0000","last_change":"2024-03-28T17:46:11.761005+0000","last_active":"2024-03-28T17:47:09.461779+0000","last_peered":"2024-03-28T17:47:09.461779+0000","last_clean":"2024-03-28T17:47:09.461779+0000","last_became_active":"2024-03-28T17:46:11.760660+0000","last_became_peered":"2024-03-28T17:46:11.760660+0000","last_unstale":"2024-03-28T17:47:09.461779+0000","last_undegraded":"2024-03-28T17:47:09.461779+0000","last_fullsized":"2024-03-28T17:47:09.461779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:01:10.408853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.644551+0000","last_change":"2024-03-28T17:46:12.762851+0000","last_active":"2024-03-28T17:47:37.644551+0000","last_peered":"2024-03-28T17:47:37.644551+0000","last_clean":"2024-03-28T17:47:37.644551+0000","last_became_active":"2024-03-28T17:46:12.762508+0000","last_became_peered":"2024-03-28T17:46:12.762508+0000","last_unstale":"2024-03-28T17:47:37.644551+0000","last_undegraded":"2024-03-28T17:47:37.644551+0000","last_fullsized":"2024-03-28T17:47:37.644551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:55:29.389985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.955449+0000","last_change":"2024-03-28T17:47:09.953352+0000","last_active":"2024-03-28T17:47:09.955449+0000","last_peered":"2024-03-28T17:47:09.955449+0000","last_clean":"2024-03-28T17:47:09.955449+0000","last_became_active":"2024-03-28T17:47:09.952822+0000","last_became_peered":"2024-03-28T17:47:09.952822+0000","last_unstale":"2024-03-28T17:47:09.955449+0000","last_undegraded":"2024-03-28T17:47:09.955449+0000","last_fullsized":"2024-03-28T17:47:09.955449+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:22:48.635040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.684406+0000","last_change":"2024-03-28T17:46:12.794290+0000","last_active":"2024-03-28T17:47:37.684406+0000","last_peered":"2024-03-28T17:47:37.684406+0000","last_clean":"2024-03-28T17:47:37.684406+0000","last_became_active":"2024-03-28T17:46:12.793906+0000","last_became_peered":"2024-03-28T17:46:12.793906+0000","last_unstale":"2024-03-28T17:47:37.684406+0000","last_undegraded":"2024-03-28T17:47:37.684406+0000","last_fullsized":"2024-03-28T17:47:37.684406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:26:04.996019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.463067+0000","last_change":"2024-03-28T17:46:11.762676+0000","last_active":"2024-03-28T17:47:09.463067+0000","last_peered":"2024-03-28T17:47:09.463067+0000","last_clean":"2024-03-28T17:47:09.463067+0000","last_became_active":"2024-03-28T17:46:11.761458+0000","last_became_peered":"2024-03-28T17:46:11.761458+0000","last_unstale":"2024-03-28T17:47:09.463067+0000","last_undegraded":"2024-03-28T17:47:09.463067+0000","last_fullsized":"2024-03-28T17:47:09.463067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:22:26.642798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.686879+0000","last_change":"2024-03-28T17:46:12.787107+0000","last_active":"2024-03-28T17:47:37.686879+0000","last_peered":"2024-03-28T17:47:37.686879+0000","last_clean":"2024-03-28T17:47:37.686879+0000","last_became_active":"2024-03-28T17:46:12.786610+0000","last_became_peered":"2024-03-28T17:46:12.786610+0000","last_unstale":"2024-03-28T17:47:37.686879+0000","last_undegraded":"2024-03-28T17:47:37.686879+0000","last_fullsized":"2024-03-28T17:47:37.686879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:30:14.880483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.665610+0000","last_change":"2024-03-28T17:47:09.645023+0000","last_active":"2024-03-28T17:47:37.665610+0000","last_peered":"2024-03-28T17:47:37.665610+0000","last_clean":"2024-03-28T17:47:37.665610+0000","last_became_active":"2024-03-28T17:47:09.644507+0000","last_became_peered":"2024-03-28T17:47:09.644507+0000","last_unstale":"2024-03-28T17:47:37.665610+0000","last_undegraded":"2024-03-28T17:47:37.665610+0000","last_fullsized":"2024-03-28T17:47:37.665610+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:00:48.144635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"87'5","reported_seq":172,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:48:43.235933+0000","last_change":"2024-03-28T17:46:11.784261+0000","last_active":"2024-03-28T17:48:43.235933+0000","last_peered":"2024-03-28T17:48:43.235933+0000","last_clean":"2024-03-28T17:48:43.235933+0000","last_became_active":"2024-03-28T17:46:11.782957+0000","last_became_peered":"2024-03-28T17:46:11.782957+0000","last_unstale":"2024-03-28T17:48:43.235933+0000","last_undegraded":"2024-03-28T17:48:43.235933+0000","last_fullsized":"2024-03-28T17:48:43.235933+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:08:48.009774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":147,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687570+0000","last_change":"2024-03-28T17:46:12.784024+0000","last_active":"2024-03-28T17:47:37.687570+0000","last_peered":"2024-03-28T17:47:37.687570+0000","last_clean":"2024-03-28T17:47:37.687570+0000","last_became_active":"2024-03-28T17:46:12.782157+0000","last_became_peered":"2024-03-28T17:46:12.782157+0000","last_unstale":"2024-03-28T17:47:37.687570+0000","last_undegraded":"2024-03-28T17:47:37.687570+0000","last_fullsized":"2024-03-28T17:47:37.687570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:33.679894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.644104+0000","last_change":"2024-03-28T17:46:11.759602+0000","last_active":"2024-03-28T17:47:09.644104+0000","last_peered":"2024-03-28T17:47:09.644104+0000","last_clean":"2024-03-28T17:47:09.644104+0000","last_became_active":"2024-03-28T17:46:11.759248+0000","last_became_peered":"2024-03-28T17:46:11.759248+0000","last_unstale":"2024-03-28T17:47:09.644104+0000","last_undegraded":"2024-03-28T17:47:09.644104+0000","last_fullsized":"2024-03-28T17:47:09.644104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:15:59.997948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":352,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.462874+0000","last_change":"2024-03-28T17:44:51.172493+0000","last_active":"2024-03-28T17:47:09.462874+0000","last_peered":"2024-03-28T17:47:09.462874+0000","last_clean":"2024-03-28T17:47:09.462874+0000","last_became_active":"2024-03-28T17:44:51.172175+0000","last_became_peered":"2024-03-28T17:44:51.172175+0000","last_unstale":"2024-03-28T17:47:09.462874+0000","last_undegraded":"2024-03-28T17:47:09.462874+0000","last_fullsized":"2024-03-28T17:47:09.462874+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:40:54.580958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:40:54.580958+0000","last_clean_scrub_stamp":"2024-03-28T17:40:54.580958+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:34.942490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.669498+0000","last_change":"2024-03-28T17:46:12.766728+0000","last_active":"2024-03-28T17:47:37.669498+0000","last_peered":"2024-03-28T17:47:37.669498+0000","last_clean":"2024-03-28T17:47:37.669498+0000","last_became_active":"2024-03-28T17:46:12.763671+0000","last_became_peered":"2024-03-28T17:46:12.763671+0000","last_unstale":"2024-03-28T17:47:37.669498+0000","last_undegraded":"2024-03-28T17:47:37.669498+0000","last_fullsized":"2024-03-28T17:47:37.669498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:25:39.802551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.462989+0000","last_change":"2024-03-28T17:46:11.763361+0000","last_active":"2024-03-28T17:47:09.462989+0000","last_peered":"2024-03-28T17:47:09.462989+0000","last_clean":"2024-03-28T17:47:09.462989+0000","last_became_active":"2024-03-28T17:46:11.763049+0000","last_became_peered":"2024-03-28T17:46:11.763049+0000","last_unstale":"2024-03-28T17:47:09.462989+0000","last_undegraded":"2024-03-28T17:47:09.462989+0000","last_fullsized":"2024-03-28T17:47:09.462989+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:01.767964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461664+0000","last_change":"2024-03-28T17:46:11.774957+0000","last_active":"2024-03-28T17:47:09.461664+0000","last_peered":"2024-03-28T17:47:09.461664+0000","last_clean":"2024-03-28T17:47:09.461664+0000","last_became_active":"2024-03-28T17:46:11.774658+0000","last_became_peered":"2024-03-28T17:46:11.774658+0000","last_unstale":"2024-03-28T17:47:09.461664+0000","last_undegraded":"2024-03-28T17:47:09.461664+0000","last_fullsized":"2024-03-28T17:47:09.461664+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:40.254318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.675859+0000","last_change":"2024-03-28T17:46:12.778206+0000","last_active":"2024-03-28T17:47:37.675859+0000","last_peered":"2024-03-28T17:47:37.675859+0000","last_clean":"2024-03-28T17:47:37.675859+0000","last_became_active":"2024-03-28T17:46:12.777780+0000","last_became_peered":"2024-03-28T17:46:12.777780+0000","last_unstale":"2024-03-28T17:47:37.675859+0000","last_undegraded":"2024-03-28T17:47:37.675859+0000","last_fullsized":"2024-03-28T17:47:37.675859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:13.241794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'12","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:48:38.262650+0000","last_change":"2024-03-28T17:47:09.961000+0000","last_active":"2024-03-28T17:48:38.262650+0000","last_peered":"2024-03-28T17:48:38.262650+0000","last_clean":"2024-03-28T17:48:38.262650+0000","last_became_active":"2024-03-28T17:47:09.953155+0000","last_became_peered":"2024-03-28T17:47:09.953155+0000","last_unstale":"2024-03-28T17:48:38.262650+0000","last_undegraded":"2024-03-28T17:48:38.262650+0000","last_fullsized":"2024-03-28T17:48:38.262650+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:31:57.258152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.683178+0000","last_change":"2024-03-28T17:46:12.790489+0000","last_active":"2024-03-28T17:47:37.683178+0000","last_peered":"2024-03-28T17:47:37.683178+0000","last_clean":"2024-03-28T17:47:37.683178+0000","last_became_active":"2024-03-28T17:46:12.790108+0000","last_became_peered":"2024-03-28T17:46:12.790108+0000","last_unstale":"2024-03-28T17:47:37.683178+0000","last_undegraded":"2024-03-28T17:47:37.683178+0000","last_fullsized":"2024-03-28T17:47:37.683178+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:16:21.210182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.644215+0000","last_change":"2024-03-28T17:46:11.762658+0000","last_active":"2024-03-28T17:47:09.644215+0000","last_peered":"2024-03-28T17:47:09.644215+0000","last_clean":"2024-03-28T17:47:09.644215+0000","last_became_active":"2024-03-28T17:46:11.762296+0000","last_became_peered":"2024-03-28T17:46:11.762296+0000","last_unstale":"2024-03-28T17:47:09.644215+0000","last_undegraded":"2024-03-28T17:47:09.644215+0000","last_fullsized":"2024-03-28T17:47:09.644215+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:42:52.768285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.686169+0000","last_change":"2024-03-28T17:47:09.488306+0000","last_active":"2024-03-28T17:47:37.686169+0000","last_peered":"2024-03-28T17:47:37.686169+0000","last_clean":"2024-03-28T17:47:37.686169+0000","last_became_active":"2024-03-28T17:47:09.487292+0000","last_became_peered":"2024-03-28T17:47:09.487292+0000","last_unstale":"2024-03-28T17:47:37.686169+0000","last_undegraded":"2024-03-28T17:47:37.686169+0000","last_fullsized":"2024-03-28T17:47:37.686169+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:55.612951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.951519+0000","last_change":"2024-03-28T17:46:11.769261+0000","last_active":"2024-03-28T17:47:09.951519+0000","last_peered":"2024-03-28T17:47:09.951519+0000","last_clean":"2024-03-28T17:47:09.951519+0000","last_became_active":"2024-03-28T17:46:11.768918+0000","last_became_peered":"2024-03-28T17:46:11.768918+0000","last_unstale":"2024-03-28T17:47:09.951519+0000","last_undegraded":"2024-03-28T17:47:09.951519+0000","last_fullsized":"2024-03-28T17:47:09.951519+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:26:38.276425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.655991+0000","last_change":"2024-03-28T17:46:12.762617+0000","last_active":"2024-03-28T17:47:37.655991+0000","last_peered":"2024-03-28T17:47:37.655991+0000","last_clean":"2024-03-28T17:47:37.655991+0000","last_became_active":"2024-03-28T17:46:12.762273+0000","last_became_peered":"2024-03-28T17:46:12.762273+0000","last_unstale":"2024-03-28T17:47:37.655991+0000","last_undegraded":"2024-03-28T17:47:37.655991+0000","last_fullsized":"2024-03-28T17:47:37.655991+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:45.848454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.476203+0000","last_change":"2024-03-28T17:46:11.770831+0000","last_active":"2024-03-28T17:47:09.476203+0000","last_peered":"2024-03-28T17:47:09.476203+0000","last_clean":"2024-03-28T17:47:09.476203+0000","last_became_active":"2024-03-28T17:46:11.770303+0000","last_became_peered":"2024-03-28T17:46:11.770303+0000","last_unstale":"2024-03-28T17:47:09.476203+0000","last_undegraded":"2024-03-28T17:47:09.476203+0000","last_fullsized":"2024-03-28T17:47:09.476203+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:48:25.566922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685855+0000","last_change":"2024-03-28T17:46:12.768885+0000","last_active":"2024-03-28T17:47:37.685855+0000","last_peered":"2024-03-28T17:47:37.685855+0000","last_clean":"2024-03-28T17:47:37.685855+0000","last_became_active":"2024-03-28T17:46:12.768525+0000","last_became_peered":"2024-03-28T17:46:12.768525+0000","last_unstale":"2024-03-28T17:47:37.685855+0000","last_undegraded":"2024-03-28T17:47:37.685855+0000","last_fullsized":"2024-03-28T17:47:37.685855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:35:30.779279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:48:37.945382+0000","last_change":"2024-03-28T17:46:11.786661+0000","last_active":"2024-03-28T17:48:37.945382+0000","last_peered":"2024-03-28T17:48:37.945382+0000","last_clean":"2024-03-28T17:48:37.945382+0000","last_became_active":"2024-03-28T17:46:11.786226+0000","last_became_peered":"2024-03-28T17:46:11.786226+0000","last_unstale":"2024-03-28T17:48:37.945382+0000","last_undegraded":"2024-03-28T17:48:37.945382+0000","last_fullsized":"2024-03-28T17:48:37.945382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:51:24.781368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687652+0000","last_change":"2024-03-28T17:46:12.797531+0000","last_active":"2024-03-28T17:47:37.687652+0000","last_peered":"2024-03-28T17:47:37.687652+0000","last_clean":"2024-03-28T17:47:37.687652+0000","last_became_active":"2024-03-28T17:46:12.797161+0000","last_became_peered":"2024-03-28T17:46:12.797161+0000","last_unstale":"2024-03-28T17:47:37.687652+0000","last_undegraded":"2024-03-28T17:47:37.687652+0000","last_fullsized":"2024-03-28T17:47:37.687652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:01.534666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.461493+0000","last_change":"2024-03-28T17:46:11.766589+0000","last_active":"2024-03-28T17:47:09.461493+0000","last_peered":"2024-03-28T17:47:09.461493+0000","last_clean":"2024-03-28T17:47:09.461493+0000","last_became_active":"2024-03-28T17:46:11.766269+0000","last_became_peered":"2024-03-28T17:46:11.766269+0000","last_unstale":"2024-03-28T17:47:09.461493+0000","last_undegraded":"2024-03-28T17:47:09.461493+0000","last_fullsized":"2024-03-28T17:47:09.461493+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:12:26.055262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.670254+0000","last_change":"2024-03-28T17:46:12.765723+0000","last_active":"2024-03-28T17:47:37.670254+0000","last_peered":"2024-03-28T17:47:37.670254+0000","last_clean":"2024-03-28T17:47:37.670254+0000","last_became_active":"2024-03-28T17:46:12.765366+0000","last_became_peered":"2024-03-28T17:46:12.765366+0000","last_unstale":"2024-03-28T17:47:37.670254+0000","last_undegraded":"2024-03-28T17:47:37.670254+0000","last_fullsized":"2024-03-28T17:47:37.670254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:39:55.103246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.463696+0000","last_change":"2024-03-28T17:46:11.767686+0000","last_active":"2024-03-28T17:47:09.463696+0000","last_peered":"2024-03-28T17:47:09.463696+0000","last_clean":"2024-03-28T17:47:09.463696+0000","last_became_active":"2024-03-28T17:46:11.767365+0000","last_became_peered":"2024-03-28T17:46:11.767365+0000","last_unstale":"2024-03-28T17:47:09.463696+0000","last_undegraded":"2024-03-28T17:47:09.463696+0000","last_fullsized":"2024-03-28T17:47:09.463696+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:49.909554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.669367+0000","last_change":"2024-03-28T17:46:12.764129+0000","last_active":"2024-03-28T17:47:37.669367+0000","last_peered":"2024-03-28T17:47:37.669367+0000","last_clean":"2024-03-28T17:47:37.669367+0000","last_became_active":"2024-03-28T17:46:12.763716+0000","last_became_peered":"2024-03-28T17:46:12.763716+0000","last_unstale":"2024-03-28T17:47:37.669367+0000","last_undegraded":"2024-03-28T17:47:37.669367+0000","last_fullsized":"2024-03-28T17:47:37.669367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:08.365578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.464975+0000","last_change":"2024-03-28T17:46:11.787374+0000","last_active":"2024-03-28T17:47:09.464975+0000","last_peered":"2024-03-28T17:47:09.464975+0000","last_clean":"2024-03-28T17:47:09.464975+0000","last_became_active":"2024-03-28T17:46:11.787047+0000","last_became_peered":"2024-03-28T17:46:11.787047+0000","last_unstale":"2024-03-28T17:47:09.464975+0000","last_undegraded":"2024-03-28T17:47:09.464975+0000","last_fullsized":"2024-03-28T17:47:09.464975+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:33.657488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.672808+0000","last_change":"2024-03-28T17:47:09.480465+0000","last_active":"2024-03-28T17:47:37.672808+0000","last_peered":"2024-03-28T17:47:37.672808+0000","last_clean":"2024-03-28T17:47:37.672808+0000","last_became_active":"2024-03-28T17:47:09.475758+0000","last_became_peered":"2024-03-28T17:47:09.475758+0000","last_unstale":"2024-03-28T17:47:37.672808+0000","last_undegraded":"2024-03-28T17:47:37.672808+0000","last_fullsized":"2024-03-28T17:47:37.672808+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:08.610192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:09.473252+0000","last_change":"2024-03-28T17:46:11.780388+0000","last_active":"2024-03-28T17:47:09.473252+0000","last_peered":"2024-03-28T17:47:09.473252+0000","last_clean":"2024-03-28T17:47:09.473252+0000","last_became_active":"2024-03-28T17:46:11.780016+0000","last_became_peered":"2024-03-28T17:46:11.780016+0000","last_unstale":"2024-03-28T17:47:09.473252+0000","last_undegraded":"2024-03-28T17:47:09.473252+0000","last_fullsized":"2024-03-28T17:47:09.473252+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:10.635615+0000","last_clean_scrub_stamp":"2024-03-28T17:46:10.635615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:51:06.181772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.652005+0000","last_change":"2024-03-28T17:46:12.764748+0000","last_active":"2024-03-28T17:47:37.652005+0000","last_peered":"2024-03-28T17:47:37.652005+0000","last_clean":"2024-03-28T17:47:37.652005+0000","last_became_active":"2024-03-28T17:46:12.764407+0000","last_became_peered":"2024-03-28T17:46:12.764407+0000","last_unstale":"2024-03-28T17:47:37.652005+0000","last_undegraded":"2024-03-28T17:47:37.652005+0000","last_fullsized":"2024-03-28T17:47:37.652005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:41:54.185079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.652775+0000","last_change":"2024-03-28T17:46:12.762486+0000","last_active":"2024-03-28T17:47:37.652775+0000","last_peered":"2024-03-28T17:47:37.652775+0000","last_clean":"2024-03-28T17:47:37.652775+0000","last_became_active":"2024-03-28T17:46:12.762107+0000","last_became_peered":"2024-03-28T17:46:12.762107+0000","last_unstale":"2024-03-28T17:47:37.652775+0000","last_undegraded":"2024-03-28T17:47:37.652775+0000","last_fullsized":"2024-03-28T17:47:37.652775+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:31.062456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.664930+0000","last_change":"2024-03-28T17:46:12.764569+0000","last_active":"2024-03-28T17:47:37.664930+0000","last_peered":"2024-03-28T17:47:37.664930+0000","last_clean":"2024-03-28T17:47:37.664930+0000","last_became_active":"2024-03-28T17:46:12.764112+0000","last_became_peered":"2024-03-28T17:46:12.764112+0000","last_unstale":"2024-03-28T17:47:37.664930+0000","last_undegraded":"2024-03-28T17:47:37.664930+0000","last_fullsized":"2024-03-28T17:47:37.664930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:39:55.526432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.684961+0000","last_change":"2024-03-28T17:46:12.786031+0000","last_active":"2024-03-28T17:47:37.684961+0000","last_peered":"2024-03-28T17:47:37.684961+0000","last_clean":"2024-03-28T17:47:37.684961+0000","last_became_active":"2024-03-28T17:46:12.785679+0000","last_became_peered":"2024-03-28T17:46:12.785679+0000","last_unstale":"2024-03-28T17:47:37.684961+0000","last_undegraded":"2024-03-28T17:47:37.684961+0000","last_fullsized":"2024-03-28T17:47:37.684961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:41:22.537979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685335+0000","last_change":"2024-03-28T17:46:12.793285+0000","last_active":"2024-03-28T17:47:37.685335+0000","last_peered":"2024-03-28T17:47:37.685335+0000","last_clean":"2024-03-28T17:47:37.685335+0000","last_became_active":"2024-03-28T17:46:12.788935+0000","last_became_peered":"2024-03-28T17:46:12.788935+0000","last_unstale":"2024-03-28T17:47:37.685335+0000","last_undegraded":"2024-03-28T17:47:37.685335+0000","last_fullsized":"2024-03-28T17:47:37.685335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:53:58.886607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.688031+0000","last_change":"2024-03-28T17:47:09.486124+0000","last_active":"2024-03-28T17:47:37.688031+0000","last_peered":"2024-03-28T17:47:37.688031+0000","last_clean":"2024-03-28T17:47:37.688031+0000","last_became_active":"2024-03-28T17:47:09.481292+0000","last_became_peered":"2024-03-28T17:47:09.481292+0000","last_unstale":"2024-03-28T17:47:37.688031+0000","last_undegraded":"2024-03-28T17:47:37.688031+0000","last_fullsized":"2024-03-28T17:47:37.688031+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:18.656764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.645280+0000","last_change":"2024-03-28T17:46:12.765140+0000","last_active":"2024-03-28T17:47:37.645280+0000","last_peered":"2024-03-28T17:47:37.645280+0000","last_clean":"2024-03-28T17:47:37.645280+0000","last_became_active":"2024-03-28T17:46:12.764760+0000","last_became_peered":"2024-03-28T17:46:12.764760+0000","last_unstale":"2024-03-28T17:47:37.645280+0000","last_undegraded":"2024-03-28T17:47:37.645280+0000","last_fullsized":"2024-03-28T17:47:37.645280+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:41.576437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.658988+0000","last_change":"2024-03-28T17:46:12.760271+0000","last_active":"2024-03-28T17:47:37.658988+0000","last_peered":"2024-03-28T17:47:37.658988+0000","last_clean":"2024-03-28T17:47:37.658988+0000","last_became_active":"2024-03-28T17:46:12.759939+0000","last_became_peered":"2024-03-28T17:46:12.759939+0000","last_unstale":"2024-03-28T17:47:37.658988+0000","last_undegraded":"2024-03-28T17:47:37.658988+0000","last_fullsized":"2024-03-28T17:47:37.658988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:06:42.529826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.681262+0000","last_change":"2024-03-28T17:46:12.792360+0000","last_active":"2024-03-28T17:47:37.681262+0000","last_peered":"2024-03-28T17:47:37.681262+0000","last_clean":"2024-03-28T17:47:37.681262+0000","last_became_active":"2024-03-28T17:46:12.791979+0000","last_became_peered":"2024-03-28T17:46:12.791979+0000","last_unstale":"2024-03-28T17:47:37.681262+0000","last_undegraded":"2024-03-28T17:47:37.681262+0000","last_fullsized":"2024-03-28T17:47:37.681262+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:39.866969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685176+0000","last_change":"2024-03-28T17:46:12.784228+0000","last_active":"2024-03-28T17:47:37.685176+0000","last_peered":"2024-03-28T17:47:37.685176+0000","last_clean":"2024-03-28T17:47:37.685176+0000","last_became_active":"2024-03-28T17:46:12.783890+0000","last_became_peered":"2024-03-28T17:46:12.783890+0000","last_unstale":"2024-03-28T17:47:37.685176+0000","last_undegraded":"2024-03-28T17:47:37.685176+0000","last_fullsized":"2024-03-28T17:47:37.685176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:58:13.388705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.685701+0000","last_change":"2024-03-28T17:46:12.782675+0000","last_active":"2024-03-28T17:47:37.685701+0000","last_peered":"2024-03-28T17:47:37.685701+0000","last_clean":"2024-03-28T17:47:37.685701+0000","last_became_active":"2024-03-28T17:46:12.779255+0000","last_became_peered":"2024-03-28T17:46:12.779255+0000","last_unstale":"2024-03-28T17:47:37.685701+0000","last_undegraded":"2024-03-28T17:47:37.685701+0000","last_fullsized":"2024-03-28T17:47:37.685701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:04:31.447512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.677034+0000","last_change":"2024-03-28T17:46:12.786308+0000","last_active":"2024-03-28T17:47:37.677034+0000","last_peered":"2024-03-28T17:47:37.677034+0000","last_clean":"2024-03-28T17:47:37.677034+0000","last_became_active":"2024-03-28T17:46:12.785948+0000","last_became_peered":"2024-03-28T17:46:12.785948+0000","last_unstale":"2024-03-28T17:47:37.677034+0000","last_undegraded":"2024-03-28T17:47:37.677034+0000","last_fullsized":"2024-03-28T17:47:37.677034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:43.292925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.687492+0000","last_change":"2024-03-28T17:46:12.797278+0000","last_active":"2024-03-28T17:47:37.687492+0000","last_peered":"2024-03-28T17:47:37.687492+0000","last_clean":"2024-03-28T17:47:37.687492+0000","last_became_active":"2024-03-28T17:46:12.796593+0000","last_became_peered":"2024-03-28T17:46:12.796593+0000","last_unstale":"2024-03-28T17:47:37.687492+0000","last_undegraded":"2024-03-28T17:47:37.687492+0000","last_fullsized":"2024-03-28T17:47:37.687492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:07:34.433680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T17:47:37.638868+0000","last_change":"2024-03-28T17:46:12.764249+0000","last_active":"2024-03-28T17:47:37.638868+0000","last_peered":"2024-03-28T17:47:37.638868+0000","last_clean":"2024-03-28T17:47:37.638868+0000","last_became_active":"2024-03-28T17:46:12.763909+0000","last_became_peered":"2024-03-28T17:46:12.763909+0000","last_unstale":"2024-03-28T17:47:37.638868+0000","last_undegraded":"2024-03-28T17:47:37.638868+0000","last_fullsized":"2024-03-28T17:47:37.638868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T17:46:11.735186+0000","last_clean_scrub_stamp":"2024-03-28T17:46:11.735186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:40:51.780187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250098,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74024,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":912,"num_read_kb":535,"num_write":185,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":219552,"data_compressed":5625,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383726,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":456197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645362,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":1116863,"data_compressed":38073,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808889,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":456550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070526,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":456353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332163,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":1049653,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495689,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":455822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574949,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":1003520,"data_stored":517043,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738476,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":457093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411425,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":526847,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182490,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":456016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920853,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":456791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757327,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":1048805,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34554,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34554,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36198,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":66133,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71232,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":70763,"data_compressed":1875,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T17:48:45.909 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T17:48:46.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:45 smithi071 ceph-mon[36597]: pgmap v395: 97 pgs: 97 active+clean; 649 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-28T17:48:46.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:45 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:46.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:45 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:46.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.166+0000 7f9d15600640 1 -- 172.21.15.45:0/2541160725 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d100fff30 msgr2=0x7f9d10102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:46.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.166+0000 7f9d15600640 1 --2- 172.21.15.45:0/2541160725 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d100fff30 0x7f9d10102360 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9d00009900 tx=0x7f9d0002bc30 comp rx=0 tx=0).stop 2024-03-28T17:48:46.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.166+0000 7f9d15600640 1 -- 172.21.15.45:0/2541160725 shutdown_connections 2024-03-28T17:48:46.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.166+0000 7f9d15600640 1 --2- 172.21.15.45:0/2541160725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d10105250 0x7f9d101056d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:46.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.166+0000 7f9d15600640 1 --2- 172.21.15.45:0/2541160725 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d101028a0 0x7f9d10104d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:46.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.166+0000 7f9d15600640 1 --2- 172.21.15.45:0/2541160725 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d100fff30 0x7f9d10102360 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:46.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.166+0000 7f9d15600640 1 -- 172.21.15.45:0/2541160725 >> 172.21.15.45:0/2541160725 conn(0x7f9d100fbb30 msgr2=0x7f9d100fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:46.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.166+0000 7f9d15600640 1 -- 172.21.15.45:0/2541160725 shutdown_connections 2024-03-28T17:48:46.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.166+0000 7f9d15600640 1 -- 172.21.15.45:0/2541160725 wait complete. 2024-03-28T17:48:46.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.167+0000 7f9d15600640 1 Processor -- start 2024-03-28T17:48:46.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.167+0000 7f9d15600640 1 -- start start 2024-03-28T17:48:46.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.167+0000 7f9d15600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d100fff30 0x7f9d1019dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:46.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.167+0000 7f9d15600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d101028a0 0x7f9d1019e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:46.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.168+0000 7f9d15600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d10105250 0x7f9d101a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:46.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.168+0000 7f9d15600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d10113960 con 0x7f9d10105250 2024-03-28T17:48:46.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.168+0000 7f9d15600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d101137e0 con 0x7f9d100fff30 2024-03-28T17:48:46.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.168+0000 7f9d15600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d10113ae0 con 0x7f9d101028a0 2024-03-28T17:48:46.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.168+0000 7f9d0ea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d100fff30 0x7f9d1019dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:46.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.168+0000 7f9d0ea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d100fff30 0x7f9d1019dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:46376/0 (socket says 172.21.15.45:46376) 2024-03-28T17:48:46.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.168+0000 7f9d0ea00640 1 -- 172.21.15.45:0/3832216186 learned_addr learned my addr 172.21.15.45:0/3832216186 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:46.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.168+0000 7f9d07e00640 1 --2- 172.21.15.45:0/3832216186 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d101028a0 0x7f9d1019e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:46.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.168+0000 7f9d0f400640 1 --2- 172.21.15.45:0/3832216186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d10105250 0x7f9d101a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:46.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.168+0000 7f9d0ea00640 1 -- 172.21.15.45:0/3832216186 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d101028a0 msgr2=0x7f9d1019e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:46.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.168+0000 7f9d0ea00640 1 --2- 172.21.15.45:0/3832216186 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d101028a0 0x7f9d1019e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:46.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.168+0000 7f9d0ea00640 1 -- 172.21.15.45:0/3832216186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d10105250 msgr2=0x7f9d101a5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:46.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.168+0000 7f9d0ea00640 1 --2- 172.21.15.45:0/3832216186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d10105250 0x7f9d101a5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:46.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.168+0000 7f9d0ea00640 1 -- 172.21.15.45:0/3832216186 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d101a5bf0 con 0x7f9d100fff30 2024-03-28T17:48:46.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.168+0000 7f9d07e00640 1 --2- 172.21.15.45:0/3832216186 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d101028a0 0x7f9d1019e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:46.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.169+0000 7f9d0ea00640 1 --2- 172.21.15.45:0/3832216186 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d100fff30 0x7f9d1019dfd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9d00000c00 tx=0x7f9d0002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:46.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.169+0000 7f9d07400640 1 -- 172.21.15.45:0/3832216186 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d00043020 con 0x7f9d100fff30 2024-03-28T17:48:46.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.169+0000 7f9d07400640 1 -- 172.21.15.45:0/3832216186 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d000054e0 con 0x7f9d100fff30 2024-03-28T17:48:46.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.169+0000 7f9d07400640 1 -- 172.21.15.45:0/3832216186 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d00003c40 con 0x7f9d100fff30 2024-03-28T17:48:46.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.169+0000 7f9d15600640 1 -- 172.21.15.45:0/3832216186 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d101a5e80 con 0x7f9d100fff30 2024-03-28T17:48:46.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.169+0000 7f9d15600640 1 -- 172.21.15.45:0/3832216186 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d101a6230 con 0x7f9d100fff30 2024-03-28T17:48:46.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.170+0000 7f9d15600640 1 -- 172.21.15.45:0/3832216186 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d100fd620 con 0x7f9d100fff30 2024-03-28T17:48:46.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.172+0000 7f9d07400640 1 -- 172.21.15.45:0/3832216186 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f9d00046020 con 0x7f9d100fff30 2024-03-28T17:48:46.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.174+0000 7f9d07400640 1 --2- 172.21.15.45:0/3832216186 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9cf40a2890 0x7f9cf40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:46.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.174+0000 7f9d07400640 1 -- 172.21.15.45:0/3832216186 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9d000f0f60 con 0x7f9d100fff30 2024-03-28T17:48:46.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.174+0000 7f9d07e00640 1 --2- 172.21.15.45:0/3832216186 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9cf40a2890 0x7f9cf40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:46.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.175+0000 7f9d07e00640 1 --2- 172.21.15.45:0/3832216186 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9cf40a2890 0x7f9cf40a4d50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9cf80095f0 tx=0x7f9cf8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:46.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.177+0000 7f9d07400640 1 -- 172.21.15.45:0/3832216186 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9d0004d020 con 0x7f9d100fff30 2024-03-28T17:48:46.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.389+0000 7f9d15600640 1 -- 172.21.15.45:0/3832216186 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d1002d240 con 0x7f9d100fff30 2024-03-28T17:48:46.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.391+0000 7f9d07400640 1 -- 172.21.15.45:0/3832216186 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22583 (secure 0 0 0) 0x7f9d000bdf60 con 0x7f9d100fff30 2024-03-28T17:48:46.392 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:48:46.392 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"9f4888ec-ed29-11ee-95d0-87774f69a715","created":"2024-03-28T17:36:51.623185+0000","modified":"2024-03-28T17:47:09.455843+0000","last_up_change":"2024-03-28T17:45:14.043432+0000","last_in_change":"2024-03-28T17:44:53.654489+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T17:40:54.208375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T17:46:10.564519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T17:46:11.734367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"74c56bc2-4169-4919-afd4-2ebd236f23e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6803","nonce":3795754023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6805","nonce":3795754023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6809","nonce":3795754023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6807","nonce":3795754023}]},"public_addr":"172.21.15.45:6803/3795754023","cluster_addr":"172.21.15.45:6805/3795754023","heartbeat_back_addr":"172.21.15.45:6809/3795754023","heartbeat_front_addr":"172.21.15.45:6807/3795754023","state":["exists","up"]},{"osd":1,"uuid":"78dcc21b-75b2-453d-b690-17b55b8ed1a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6801","nonce":2014529906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6803","nonce":2014529906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6807","nonce":2014529906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6805","nonce":2014529906}]},"public_addr":"172.21.15.71:6801/2014529906","cluster_addr":"172.21.15.71:6803/2014529906","heartbeat_back_addr":"172.21.15.71:6807/2014529906","heartbeat_front_addr":"172.21.15.71:6805/2014529906","state":["exists","up"]},{"osd":2,"uuid":"ca9f3f36-b71e-4962-a94f-cedf2f12eb7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6801","nonce":1850933775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6803","nonce":1850933775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6807","nonce":1850933775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6805","nonce":1850933775}]},"public_addr":"172.21.15.151:6801/1850933775","cluster_addr":"172.21.15.151:6803/1850933775","heartbeat_back_addr":"172.21.15.151:6807/1850933775","heartbeat_front_addr":"172.21.15.151:6805/1850933775","state":["exists","up"]},{"osd":3,"uuid":"d3f21f01-2973-413e-8daa-9974b8a44af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6811","nonce":3021569975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6813","nonce":3021569975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6817","nonce":3021569975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6815","nonce":3021569975}]},"public_addr":"172.21.15.45:6811/3021569975","cluster_addr":"172.21.15.45:6813/3021569975","heartbeat_back_addr":"172.21.15.45:6817/3021569975","heartbeat_front_addr":"172.21.15.45:6815/3021569975","state":["exists","up"]},{"osd":4,"uuid":"4a336f42-0605-492b-ad88-56cb02acdcf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6809","nonce":2069578333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6811","nonce":2069578333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6815","nonce":2069578333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6813","nonce":2069578333}]},"public_addr":"172.21.15.71:6809/2069578333","cluster_addr":"172.21.15.71:6811/2069578333","heartbeat_back_addr":"172.21.15.71:6815/2069578333","heartbeat_front_addr":"172.21.15.71:6813/2069578333","state":["exists","up"]},{"osd":5,"uuid":"c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6809","nonce":2629217227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6811","nonce":2629217227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6815","nonce":2629217227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6813","nonce":2629217227}]},"public_addr":"172.21.15.151:6809/2629217227","cluster_addr":"172.21.15.151:6811/2629217227","heartbeat_back_addr":"172.21.15.151:6815/2629217227","heartbeat_front_addr":"172.21.15.151:6813/2629217227","state":["exists","up"]},{"osd":6,"uuid":"58a6c5bd-a8b7-4892-b346-bcd7994bcb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6819","nonce":2077907039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6821","nonce":2077907039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6825","nonce":2077907039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6823","nonce":2077907039}]},"public_addr":"172.21.15.45:6819/2077907039","cluster_addr":"172.21.15.45:6821/2077907039","heartbeat_back_addr":"172.21.15.45:6825/2077907039","heartbeat_front_addr":"172.21.15.45:6823/2077907039","state":["exists","up"]},{"osd":7,"uuid":"f7a45d52-807b-4407-bb04-787b2a52a0ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6817","nonce":2000211251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6819","nonce":2000211251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6823","nonce":2000211251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6821","nonce":2000211251}]},"public_addr":"172.21.15.71:6817/2000211251","cluster_addr":"172.21.15.71:6819/2000211251","heartbeat_back_addr":"172.21.15.71:6823/2000211251","heartbeat_front_addr":"172.21.15.71:6821/2000211251","state":["exists","up"]},{"osd":8,"uuid":"88abcb0f-f189-49be-8315-c700c77369f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6817","nonce":3665460137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6819","nonce":3665460137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6823","nonce":3665460137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6821","nonce":3665460137}]},"public_addr":"172.21.15.151:6817/3665460137","cluster_addr":"172.21.15.151:6819/3665460137","heartbeat_back_addr":"172.21.15.151:6823/3665460137","heartbeat_front_addr":"172.21.15.151:6821/3665460137","state":["exists","up"]},{"osd":9,"uuid":"ed3fc096-961e-4392-abc2-c5aa4d050117","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6827","nonce":2122322684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6829","nonce":2122322684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6833","nonce":2122322684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6831","nonce":2122322684}]},"public_addr":"172.21.15.45:6827/2122322684","cluster_addr":"172.21.15.45:6829/2122322684","heartbeat_back_addr":"172.21.15.45:6833/2122322684","heartbeat_front_addr":"172.21.15.45:6831/2122322684","state":["exists","up"]},{"osd":10,"uuid":"ba94762b-bc5c-4c97-9cc1-17e5651c14b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6825","nonce":1099494037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6827","nonce":1099494037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6831","nonce":1099494037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6829","nonce":1099494037}]},"public_addr":"172.21.15.71:6825/1099494037","cluster_addr":"172.21.15.71:6827/1099494037","heartbeat_back_addr":"172.21.15.71:6831/1099494037","heartbeat_front_addr":"172.21.15.71:6829/1099494037","state":["exists","up"]},{"osd":11,"uuid":"1d178378-b9f5-40f6-99ef-3d3d445da640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6825","nonce":2941055364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6827","nonce":2941055364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6831","nonce":2941055364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6829","nonce":2941055364}]},"public_addr":"172.21.15.151:6825/2941055364","cluster_addr":"172.21.15.151:6827/2941055364","heartbeat_back_addr":"172.21.15.151:6831/2941055364","heartbeat_front_addr":"172.21.15.151:6829/2941055364","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:02.269586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:27.819386+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:51.552277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:25.052674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:51.873529+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:17.127867+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:51.588249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:18.381450+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:44.140534+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:17.634362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:45.469340+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:45:12.003433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3533611957":"2024-03-29T17:38:07.505611+0000","172.21.15.45:0/3625922430":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6801/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/482610468":"2024-03-29T17:37:32.424229+0000","172.21.15.45:6801/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3649804714":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/1198364079":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3211303958":"2024-03-29T17:38:07.505611+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T17:48:46.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.394+0000 7f9d15600640 1 -- 172.21.15.45:0/3832216186 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9cf40a2890 msgr2=0x7f9cf40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:46.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.394+0000 7f9d15600640 1 --2- 172.21.15.45:0/3832216186 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9cf40a2890 0x7f9cf40a4d50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9cf80095f0 tx=0x7f9cf8005f90 comp rx=0 tx=0).stop 2024-03-28T17:48:46.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.394+0000 7f9d15600640 1 -- 172.21.15.45:0/3832216186 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d100fff30 msgr2=0x7f9d1019dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:46.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.394+0000 7f9d15600640 1 --2- 172.21.15.45:0/3832216186 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d100fff30 0x7f9d1019dfd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9d00000c00 tx=0x7f9d0002c650 comp rx=0 tx=0).stop 2024-03-28T17:48:46.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.394+0000 7f9d15600640 1 -- 172.21.15.45:0/3832216186 shutdown_connections 2024-03-28T17:48:46.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.394+0000 7f9d15600640 1 --2- 172.21.15.45:0/3832216186 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9cf40a2890 0x7f9cf40a4d50 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:46.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.394+0000 7f9d15600640 1 --2- 172.21.15.45:0/3832216186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d10105250 0x7f9d101a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:46.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.394+0000 7f9d15600640 1 --2- 172.21.15.45:0/3832216186 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d101028a0 0x7f9d1019e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:46.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.394+0000 7f9d15600640 1 --2- 172.21.15.45:0/3832216186 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d100fff30 0x7f9d1019dfd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:46.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.394+0000 7f9d15600640 1 -- 172.21.15.45:0/3832216186 >> 172.21.15.45:0/3832216186 conn(0x7f9d100fbb30 msgr2=0x7f9d10103230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:46.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.394+0000 7f9d15600640 1 -- 172.21.15.45:0/3832216186 shutdown_connections 2024-03-28T17:48:46.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.394+0000 7f9d15600640 1 -- 172.21.15.45:0/3832216186 wait complete. 2024-03-28T17:48:46.413 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T17:40:54.208375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T17:46:10.564519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T17:46:11.734367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T17:48:46.414 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T17:48:46.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.606+0000 7f9c09800640 1 -- 172.21.15.45:0/3657797270 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c04074100 msgr2=0x7f9c04074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:46.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.606+0000 7f9c09800640 1 --2- 172.21.15.45:0/3657797270 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c04074100 0x7f9c04074580 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9bec009900 tx=0x7f9bec02bc30 comp rx=0 tx=0).stop 2024-03-28T17:48:46.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.607+0000 7f9c09800640 1 -- 172.21.15.45:0/3657797270 shutdown_connections 2024-03-28T17:48:46.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.607+0000 7f9c09800640 1 --2- 172.21.15.45:0/3657797270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c04109460 0x7f9c0410b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:46.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.607+0000 7f9c09800640 1 --2- 172.21.15.45:0/3657797270 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c04074100 0x7f9c04074580 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:46.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.607+0000 7f9c09800640 1 --2- 172.21.15.45:0/3657797270 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c040756e0 0x7f9c04073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:46.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.607+0000 7f9c09800640 1 -- 172.21.15.45:0/3657797270 >> 172.21.15.45:0/3657797270 conn(0x7f9c040fcb10 msgr2=0x7f9c040fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:46.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.607+0000 7f9c09800640 1 -- 172.21.15.45:0/3657797270 shutdown_connections 2024-03-28T17:48:46.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.607+0000 7f9c09800640 1 -- 172.21.15.45:0/3657797270 wait complete. 2024-03-28T17:48:46.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.608+0000 7f9c09800640 1 Processor -- start 2024-03-28T17:48:46.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.608+0000 7f9c09800640 1 -- start start 2024-03-28T17:48:46.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.608+0000 7f9c09800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c04074100 0x7f9c04199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:46.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.608+0000 7f9c09800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c040756e0 0x7f9c04199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:46.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.608+0000 7f9c09800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c04109460 0x7f9c041a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:46.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.608+0000 7f9c09800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0410f430 con 0x7f9c040756e0 2024-03-28T17:48:46.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.608+0000 7f9c09800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0410f2b0 con 0x7f9c04074100 2024-03-28T17:48:46.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.608+0000 7f9c09800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0410f5b0 con 0x7f9c04109460 2024-03-28T17:48:46.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.608+0000 7f9c03e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c04074100 0x7f9c04199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:46.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.608+0000 7f9c03400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c040756e0 0x7f9c04199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:46.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.608+0000 7f9c03e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c04074100 0x7f9c04199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:46394/0 (socket says 172.21.15.45:46394) 2024-03-28T17:48:46.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.608+0000 7f9c03e00640 1 -- 172.21.15.45:0/3168134071 learned_addr learned my addr 172.21.15.45:0/3168134071 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:46.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.608+0000 7f9c08e00640 1 --2- 172.21.15.45:0/3168134071 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c04109460 0x7f9c041a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:46.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.609+0000 7f9c03e00640 1 -- 172.21.15.45:0/3168134071 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c04109460 msgr2=0x7f9c041a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:46.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.609+0000 7f9c03e00640 1 --2- 172.21.15.45:0/3168134071 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c04109460 0x7f9c041a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:46.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.609+0000 7f9c03e00640 1 -- 172.21.15.45:0/3168134071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c040756e0 msgr2=0x7f9c04199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:46.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.609+0000 7f9c03e00640 1 --2- 172.21.15.45:0/3168134071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c040756e0 0x7f9c04199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:46.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.609+0000 7f9c03e00640 1 -- 172.21.15.45:0/3168134071 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c041a16a0 con 0x7f9c04074100 2024-03-28T17:48:46.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.609+0000 7f9c03400640 1 --2- 172.21.15.45:0/3168134071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c040756e0 0x7f9c04199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:46.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.609+0000 7f9c03e00640 1 --2- 172.21.15.45:0/3168134071 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c04074100 0x7f9c04199a80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9bf400ebc0 tx=0x7f9bf400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:46.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.609+0000 7f9c00c00640 1 -- 172.21.15.45:0/3168134071 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf4010630 con 0x7f9c04074100 2024-03-28T17:48:46.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.609+0000 7f9c00c00640 1 -- 172.21.15.45:0/3168134071 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9bf4010cb0 con 0x7f9c04074100 2024-03-28T17:48:46.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.609+0000 7f9c09800640 1 -- 172.21.15.45:0/3168134071 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c041a1990 con 0x7f9c04074100 2024-03-28T17:48:46.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.609+0000 7f9c00c00640 1 -- 172.21.15.45:0/3168134071 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf4018bd0 con 0x7f9c04074100 2024-03-28T17:48:46.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.610+0000 7f9c09800640 1 -- 172.21.15.45:0/3168134071 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c041a1f00 con 0x7f9c04074100 2024-03-28T17:48:46.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.611+0000 7f9c09800640 1 -- 172.21.15.45:0/3168134071 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c04108190 con 0x7f9c04074100 2024-03-28T17:48:46.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.613+0000 7f9c00c00640 1 -- 172.21.15.45:0/3168134071 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f9bf40107d0 con 0x7f9c04074100 2024-03-28T17:48:46.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.614+0000 7f9c00c00640 1 --2- 172.21.15.45:0/3168134071 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9be80a2910 0x7f9be80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:46.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.615+0000 7f9c00c00640 1 -- 172.21.15.45:0/3168134071 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9bf40cf1a0 con 0x7f9c04074100 2024-03-28T17:48:46.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.615+0000 7f9c03400640 1 --2- 172.21.15.45:0/3168134071 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9be80a2910 0x7f9be80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:46.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.616+0000 7f9c03400640 1 --2- 172.21.15.45:0/3168134071 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9be80a2910 0x7f9be80a4dd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9bec02c240 tx=0x7f9bec02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:46.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.618+0000 7f9c00c00640 1 -- 172.21.15.45:0/3168134071 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9bf409ba40 con 0x7f9c04074100 2024-03-28T17:48:46.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:46 smithi045 ceph-mon[31773]: from='client.25252 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:48:46.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:46 smithi045 ceph-mon[31773]: from='client.25283 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:48:46.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:46 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:46.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:46 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:46.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:46 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3832216186' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:48:46.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.830+0000 7f9c09800640 1 -- 172.21.15.45:0/3168134071 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9c040622a0 con 0x7f9c04074100 2024-03-28T17:48:46.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.830+0000 7f9c00c00640 1 -- 172.21.15.45:0/3168134071 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9bf409fae0 con 0x7f9c04074100 2024-03-28T17:48:46.831 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-28T17:48:46.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.833+0000 7f9c09800640 1 -- 172.21.15.45:0/3168134071 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9be80a2910 msgr2=0x7f9be80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:46.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.833+0000 7f9c09800640 1 --2- 172.21.15.45:0/3168134071 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9be80a2910 0x7f9be80a4dd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9bec02c240 tx=0x7f9bec02df90 comp rx=0 tx=0).stop 2024-03-28T17:48:46.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.833+0000 7f9c09800640 1 -- 172.21.15.45:0/3168134071 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c04074100 msgr2=0x7f9c04199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:46.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.833+0000 7f9c09800640 1 --2- 172.21.15.45:0/3168134071 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c04074100 0x7f9c04199a80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9bf400ebc0 tx=0x7f9bf400c6a0 comp rx=0 tx=0).stop 2024-03-28T17:48:46.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.834+0000 7f9c09800640 1 -- 172.21.15.45:0/3168134071 shutdown_connections 2024-03-28T17:48:46.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.834+0000 7f9c09800640 1 --2- 172.21.15.45:0/3168134071 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f9be80a2910 0x7f9be80a4dd0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:46.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.834+0000 7f9c09800640 1 --2- 172.21.15.45:0/3168134071 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9c04109460 0x7f9c041a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:46.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.834+0000 7f9c09800640 1 --2- 172.21.15.45:0/3168134071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c040756e0 0x7f9c04199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:46.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.834+0000 7f9c09800640 1 --2- 172.21.15.45:0/3168134071 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c04074100 0x7f9c04199a80 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:46.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.834+0000 7f9c09800640 1 -- 172.21.15.45:0/3168134071 >> 172.21.15.45:0/3168134071 conn(0x7f9c040fcb10 msgr2=0x7f9c04105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:46.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.834+0000 7f9c09800640 1 -- 172.21.15.45:0/3168134071 shutdown_connections 2024-03-28T17:48:46.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:46.834+0000 7f9c09800640 1 -- 172.21.15.45:0/3168134071 wait complete. 2024-03-28T17:48:46.854 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T17:48:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:46 smithi071 ceph-mon[36597]: from='client.25252 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:48:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:46 smithi071 ceph-mon[36597]: from='client.25283 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:48:47.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:46 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:47.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:46 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:47.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:46 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3832216186' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:48:47.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.061+0000 7f6b6e000640 1 -- 172.21.15.45:0/3966043557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b68103260 msgr2=0x7f6b68109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:47.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.061+0000 7f6b6e000640 1 --2- 172.21.15.45:0/3966043557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b68103260 0x7f6b68109af0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6b60009ca0 tx=0x7f6b6002c100 comp rx=0 tx=0).stop 2024-03-28T17:48:47.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.061+0000 7f6b6e000640 1 -- 172.21.15.45:0/3966043557 shutdown_connections 2024-03-28T17:48:47.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.061+0000 7f6b6e000640 1 --2- 172.21.15.45:0/3966043557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b68103260 0x7f6b68109af0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.061+0000 7f6b6e000640 1 --2- 172.21.15.45:0/3966043557 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b681028a0 0x7f6b68102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.061+0000 7f6b6e000640 1 --2- 172.21.15.45:0/3966043557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b681016b0 0x7f6b68101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.061+0000 7f6b6e000640 1 -- 172.21.15.45:0/3966043557 >> 172.21.15.45:0/3966043557 conn(0x7f6b680fcc20 msgr2=0x7f6b680ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:47.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.061+0000 7f6b6e000640 1 -- 172.21.15.45:0/3966043557 shutdown_connections 2024-03-28T17:48:47.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.061+0000 7f6b6e000640 1 -- 172.21.15.45:0/3966043557 wait complete. 2024-03-28T17:48:47.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.062+0000 7f6b6e000640 1 Processor -- start 2024-03-28T17:48:47.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.063+0000 7f6b6e000640 1 -- start start 2024-03-28T17:48:47.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.063+0000 7f6b6e000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b681016b0 0x7f6b68199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:47.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.063+0000 7f6b6e000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b681028a0 0x7f6b6819a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:47.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.063+0000 7f6b6e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b68103260 0x7f6b681a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:47.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.063+0000 7f6b6e000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6810d310 con 0x7f6b68103260 2024-03-28T17:48:47.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.063+0000 7f6b6e000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6810d190 con 0x7f6b681016b0 2024-03-28T17:48:47.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.063+0000 7f6b6e000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6810d490 con 0x7f6b681028a0 2024-03-28T17:48:47.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.063+0000 7f6b67400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b681016b0 0x7f6b68199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:47.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.063+0000 7f6b67e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b68103260 0x7f6b681a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:47.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.063+0000 7f6b66a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b681028a0 0x7f6b6819a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:47.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.063+0000 7f6b67400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b681016b0 0x7f6b68199b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:46416/0 (socket says 172.21.15.45:46416) 2024-03-28T17:48:47.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.063+0000 7f6b67400640 1 -- 172.21.15.45:0/462374682 learned_addr learned my addr 172.21.15.45:0/462374682 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:47.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.064+0000 7f6b66a00640 1 -- 172.21.15.45:0/462374682 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b681016b0 msgr2=0x7f6b68199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:47.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.064+0000 7f6b66a00640 1 --2- 172.21.15.45:0/462374682 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b681016b0 0x7f6b68199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.064+0000 7f6b66a00640 1 -- 172.21.15.45:0/462374682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b68103260 msgr2=0x7f6b681a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:47.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.064+0000 7f6b66a00640 1 --2- 172.21.15.45:0/462374682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b68103260 0x7f6b681a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.064+0000 7f6b66a00640 1 -- 172.21.15.45:0/462374682 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b681a1790 con 0x7f6b681028a0 2024-03-28T17:48:47.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.064+0000 7f6b67400640 1 --2- 172.21.15.45:0/462374682 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b681016b0 0x7f6b68199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:47.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.064+0000 7f6b66a00640 1 --2- 172.21.15.45:0/462374682 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b681028a0 0x7f6b6819a0b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6b5400e9f0 tx=0x7f6b5400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:47.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.064+0000 7f6b5fe00640 1 -- 172.21.15.45:0/462374682 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5400ccc0 con 0x7f6b681028a0 2024-03-28T17:48:47.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.064+0000 7f6b5fe00640 1 -- 172.21.15.45:0/462374682 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b54010910 con 0x7f6b681028a0 2024-03-28T17:48:47.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.064+0000 7f6b5fe00640 1 -- 172.21.15.45:0/462374682 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b54018960 con 0x7f6b681028a0 2024-03-28T17:48:47.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.065+0000 7f6b6e000640 1 -- 172.21.15.45:0/462374682 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b681a1a80 con 0x7f6b681028a0 2024-03-28T17:48:47.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.065+0000 7f6b6e000640 1 -- 172.21.15.45:0/462374682 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b681a1f80 con 0x7f6b681028a0 2024-03-28T17:48:47.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.066+0000 7f6b6e000640 1 -- 172.21.15.45:0/462374682 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b681a21b0 con 0x7f6b681028a0 2024-03-28T17:48:47.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.070+0000 7f6b5fe00640 1 -- 172.21.15.45:0/462374682 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f6b54010ac0 con 0x7f6b681028a0 2024-03-28T17:48:47.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.071+0000 7f6b5fe00640 1 --2- 172.21.15.45:0/462374682 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6b4c0a2890 0x7f6b4c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:47.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.072+0000 7f6b5fe00640 1 -- 172.21.15.45:0/462374682 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6b540cef90 con 0x7f6b681028a0 2024-03-28T17:48:47.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.072+0000 7f6b67400640 1 --2- 172.21.15.45:0/462374682 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6b4c0a2890 0x7f6b4c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:47.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.072+0000 7f6b67400640 1 --2- 172.21.15.45:0/462374682 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6b4c0a2890 0x7f6b4c0a4d50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6b580058c0 tx=0x7f6b58005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:46 smithi151 ceph-mon[36481]: from='client.25252 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:48:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:46 smithi151 ceph-mon[36481]: from='client.25283 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:48:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:46 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:46 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:46 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3832216186' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:48:47.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.075+0000 7f6b5fe00640 1 -- 172.21.15.45:0/462374682 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6b5409b890 con 0x7f6b681028a0 2024-03-28T17:48:47.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.292+0000 7f6b6e000640 1 -- 172.21.15.45:0/462374682 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6b681a2540 con 0x7f6b681028a0 2024-03-28T17:48:47.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.293+0000 7f6b5fe00640 1 -- 172.21.15.45:0/462374682 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6b54084e20 con 0x7f6b681028a0 2024-03-28T17:48:47.293 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-03-28T17:48:47.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.296+0000 7f6b6e000640 1 -- 172.21.15.45:0/462374682 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6b4c0a2890 msgr2=0x7f6b4c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:47.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.296+0000 7f6b6e000640 1 --2- 172.21.15.45:0/462374682 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6b4c0a2890 0x7f6b4c0a4d50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6b580058c0 tx=0x7f6b58005f90 comp rx=0 tx=0).stop 2024-03-28T17:48:47.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.296+0000 7f6b6e000640 1 -- 172.21.15.45:0/462374682 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b681028a0 msgr2=0x7f6b6819a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:47.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.296+0000 7f6b6e000640 1 --2- 172.21.15.45:0/462374682 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b681028a0 0x7f6b6819a0b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6b5400e9f0 tx=0x7f6b5400eeb0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.296+0000 7f6b6e000640 1 -- 172.21.15.45:0/462374682 shutdown_connections 2024-03-28T17:48:47.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.296+0000 7f6b6e000640 1 --2- 172.21.15.45:0/462374682 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6b4c0a2890 0x7f6b4c0a4d50 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.296+0000 7f6b6e000640 1 --2- 172.21.15.45:0/462374682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b68103260 0x7f6b681a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.296+0000 7f6b6e000640 1 --2- 172.21.15.45:0/462374682 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b681028a0 0x7f6b6819a0b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.296+0000 7f6b6e000640 1 --2- 172.21.15.45:0/462374682 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b681016b0 0x7f6b68199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.296+0000 7f6b6e000640 1 -- 172.21.15.45:0/462374682 >> 172.21.15.45:0/462374682 conn(0x7f6b680fcc20 msgr2=0x7f6b680738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:47.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.296+0000 7f6b6e000640 1 -- 172.21.15.45:0/462374682 shutdown_connections 2024-03-28T17:48:47.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.296+0000 7f6b6e000640 1 -- 172.21.15.45:0/462374682 wait complete. 2024-03-28T17:48:47.315 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T17:48:47.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.521+0000 7fcfcca00640 1 -- 172.21.15.45:0/1031519475 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfc8102130 msgr2=0x7fcfc8104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:47.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.521+0000 7fcfcca00640 1 --2- 172.21.15.45:0/1031519475 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfc8102130 0x7fcfc8104520 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8009ca0 tx=0x7fcfb802c100 comp rx=0 tx=0).stop 2024-03-28T17:48:47.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.521+0000 7fcfcca00640 1 -- 172.21.15.45:0/1031519475 shutdown_connections 2024-03-28T17:48:47.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.521+0000 7fcfcca00640 1 --2- 172.21.15.45:0/1031519475 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfc8107390 0x7fcfc81097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.521+0000 7fcfcca00640 1 --2- 172.21.15.45:0/1031519475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfc8104a60 0x7fcfc8106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.521+0000 7fcfcca00640 1 --2- 172.21.15.45:0/1031519475 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfc8102130 0x7fcfc8104520 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.521+0000 7fcfcca00640 1 -- 172.21.15.45:0/1031519475 >> 172.21.15.45:0/1031519475 conn(0x7fcfc80fbb40 msgr2=0x7fcfc80fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:47.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.521+0000 7fcfcca00640 1 -- 172.21.15.45:0/1031519475 shutdown_connections 2024-03-28T17:48:47.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.522+0000 7fcfcca00640 1 -- 172.21.15.45:0/1031519475 wait complete. 2024-03-28T17:48:47.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.523+0000 7fcfcca00640 1 Processor -- start 2024-03-28T17:48:47.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.523+0000 7fcfcca00640 1 -- start start 2024-03-28T17:48:47.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.523+0000 7fcfcca00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfc8102130 0x7fcfc8197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:47.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.523+0000 7fcfcca00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfc8104a60 0x7fcfc8197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:47.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.523+0000 7fcfcca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfc8107390 0x7fcfc819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:47.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.523+0000 7fcfcca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc810d340 con 0x7fcfc8107390 2024-03-28T17:48:47.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.523+0000 7fcfcca00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc810d1c0 con 0x7fcfc8102130 2024-03-28T17:48:47.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.523+0000 7fcfcca00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc810d4c0 con 0x7fcfc8104a60 2024-03-28T17:48:47.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.523+0000 7fcfc5e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfc8102130 0x7fcfc8197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:47.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.523+0000 7fcfc5400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfc8104a60 0x7fcfc8197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:47.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.523+0000 7fcfc5e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfc8102130 0x7fcfc8197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:46436/0 (socket says 172.21.15.45:46436) 2024-03-28T17:48:47.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.523+0000 7fcfc5e00640 1 -- 172.21.15.45:0/1603345402 learned_addr learned my addr 172.21.15.45:0/1603345402 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:47.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.524+0000 7fcfc6800640 1 --2- 172.21.15.45:0/1603345402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfc8107390 0x7fcfc819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:47.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.524+0000 7fcfc5e00640 1 -- 172.21.15.45:0/1603345402 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfc8104a60 msgr2=0x7fcfc8197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:47.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.524+0000 7fcfc5e00640 1 --2- 172.21.15.45:0/1603345402 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfc8104a60 0x7fcfc8197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.524+0000 7fcfc5e00640 1 -- 172.21.15.45:0/1603345402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfc8107390 msgr2=0x7fcfc819efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:47.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.524+0000 7fcfc5e00640 1 --2- 172.21.15.45:0/1603345402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfc8107390 0x7fcfc819efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.524+0000 7fcfc5e00640 1 -- 172.21.15.45:0/1603345402 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc819f630 con 0x7fcfc8102130 2024-03-28T17:48:47.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.524+0000 7fcfc5400640 1 --2- 172.21.15.45:0/1603345402 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfc8104a60 0x7fcfc8197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:47.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.524+0000 7fcfc6800640 1 --2- 172.21.15.45:0/1603345402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfc8107390 0x7fcfc819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:48:47.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.524+0000 7fcfc5e00640 1 --2- 172.21.15.45:0/1603345402 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfc8102130 0x7fcfc8197a10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcfb80325f0 tx=0x7fcfb8035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:47.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.525+0000 7fcfbea00640 1 -- 172.21.15.45:0/1603345402 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8039040 con 0x7fcfc8102130 2024-03-28T17:48:47.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.525+0000 7fcfbea00640 1 -- 172.21.15.45:0/1603345402 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfb8044070 con 0x7fcfc8102130 2024-03-28T17:48:47.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.525+0000 7fcfbea00640 1 -- 172.21.15.45:0/1603345402 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb80044d0 con 0x7fcfc8102130 2024-03-28T17:48:47.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.525+0000 7fcfcca00640 1 -- 172.21.15.45:0/1603345402 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc819f8c0 con 0x7fcfc8102130 2024-03-28T17:48:47.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.525+0000 7fcfcca00640 1 -- 172.21.15.45:0/1603345402 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc819fd30 con 0x7fcfc8102130 2024-03-28T17:48:47.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.526+0000 7fcfcca00640 1 -- 172.21.15.45:0/1603345402 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc80fd630 con 0x7fcfc8102130 2024-03-28T17:48:47.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.529+0000 7fcfbea00640 1 -- 172.21.15.45:0/1603345402 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fcfb8004670 con 0x7fcfc8102130 2024-03-28T17:48:47.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.530+0000 7fcfbea00640 1 --2- 172.21.15.45:0/1603345402 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcfac0a2910 0x7fcfac0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:47.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.531+0000 7fcfbea00640 1 -- 172.21.15.45:0/1603345402 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fcfb80f1060 con 0x7fcfc8102130 2024-03-28T17:48:47.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.531+0000 7fcfc5400640 1 --2- 172.21.15.45:0/1603345402 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcfac0a2910 0x7fcfac0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:47.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.531+0000 7fcfc5400640 1 --2- 172.21.15.45:0/1603345402 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcfac0a2910 0x7fcfac0a4dd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8198f20 tx=0x7fcfb0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:47.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.534+0000 7fcfbea00640 1 -- 172.21.15.45:0/1603345402 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcfb80be0c0 con 0x7fcfc8102130 2024-03-28T17:48:47.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.743+0000 7fcfcca00640 1 -- 172.21.15.45:0/1603345402 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcfc80622a0 con 0x7fcfc8102130 2024-03-28T17:48:47.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.744+0000 7fcfbea00640 1 -- 172.21.15.45:0/1603345402 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcfb80bd9c0 con 0x7fcfc8102130 2024-03-28T17:48:47.744 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-03-28T17:48:47.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.746+0000 7fcfcca00640 1 -- 172.21.15.45:0/1603345402 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcfac0a2910 msgr2=0x7fcfac0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:47.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.746+0000 7fcfcca00640 1 --2- 172.21.15.45:0/1603345402 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcfac0a2910 0x7fcfac0a4dd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8198f20 tx=0x7fcfb0005f90 comp rx=0 tx=0).stop 2024-03-28T17:48:47.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.746+0000 7fcfcca00640 1 -- 172.21.15.45:0/1603345402 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfc8102130 msgr2=0x7fcfc8197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:47.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.746+0000 7fcfcca00640 1 --2- 172.21.15.45:0/1603345402 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfc8102130 0x7fcfc8197a10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcfb80325f0 tx=0x7fcfb8035d10 comp rx=0 tx=0).stop 2024-03-28T17:48:47.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.746+0000 7fcfcca00640 1 -- 172.21.15.45:0/1603345402 shutdown_connections 2024-03-28T17:48:47.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.746+0000 7fcfcca00640 1 --2- 172.21.15.45:0/1603345402 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fcfac0a2910 0x7fcfac0a4dd0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.746+0000 7fcfcca00640 1 --2- 172.21.15.45:0/1603345402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfc8107390 0x7fcfc819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.746+0000 7fcfcca00640 1 --2- 172.21.15.45:0/1603345402 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcfc8104a60 0x7fcfc8197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.746+0000 7fcfcca00640 1 --2- 172.21.15.45:0/1603345402 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfc8102130 0x7fcfc8197a10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.746+0000 7fcfcca00640 1 -- 172.21.15.45:0/1603345402 >> 172.21.15.45:0/1603345402 conn(0x7fcfc80fbb40 msgr2=0x7fcfc8102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:47.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.747+0000 7fcfcca00640 1 -- 172.21.15.45:0/1603345402 shutdown_connections 2024-03-28T17:48:47.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.747+0000 7fcfcca00640 1 -- 172.21.15.45:0/1603345402 wait complete. 2024-03-28T17:48:47.766 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:48:47.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:47 smithi045 ceph-mon[31773]: pgmap v396: 97 pgs: 97 active+clean; 649 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-28T17:48:47.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:47 smithi045 ceph-mon[31773]: from='client.25264 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:48:47.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:47 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3168134071' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:48:47.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:47 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:47.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:47 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:47.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:47 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/462374682' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T17:48:47.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.991+0000 7f6ccdc00640 1 -- 172.21.15.45:0/1150710262 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6cc8107b00 msgr2=0x7f6cc8107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:47.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.991+0000 7f6ccdc00640 1 --2- 172.21.15.45:0/1150710262 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6cc8107b00 0x7f6cc8107f20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8009be0 tx=0x7f6cb802bef0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.991+0000 7f6ccdc00640 1 -- 172.21.15.45:0/1150710262 shutdown_connections 2024-03-28T17:48:47.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.991+0000 7f6ccdc00640 1 --2- 172.21.15.45:0/1150710262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cc81096b0 0x7f6cc810ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.991+0000 7f6ccdc00640 1 --2- 172.21.15.45:0/1150710262 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cc8108cf0 0x7f6cc8109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.991+0000 7f6ccdc00640 1 --2- 172.21.15.45:0/1150710262 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6cc8107b00 0x7f6cc8107f20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.991+0000 7f6ccdc00640 1 -- 172.21.15.45:0/1150710262 >> 172.21.15.45:0/1150710262 conn(0x7f6cc8076ab0 msgr2=0x7f6cc8078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:47.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.992+0000 7f6ccdc00640 1 -- 172.21.15.45:0/1150710262 shutdown_connections 2024-03-28T17:48:47.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.992+0000 7f6ccdc00640 1 -- 172.21.15.45:0/1150710262 wait complete. 2024-03-28T17:48:47.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.992+0000 7f6ccdc00640 1 Processor -- start 2024-03-28T17:48:47.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.993+0000 7f6ccdc00640 1 -- start start 2024-03-28T17:48:47.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.993+0000 7f6ccdc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cc8107b00 0x7f6cc819de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:47.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.993+0000 7f6ccdc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cc8108cf0 0x7f6cc819e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:47.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.993+0000 7f6ccdc00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6cc81096b0 0x7f6cc81a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:47.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.993+0000 7f6ccdc00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc81137a0 con 0x7f6cc8108cf0 2024-03-28T17:48:47.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.993+0000 7f6ccdc00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc8113620 con 0x7f6cc8107b00 2024-03-28T17:48:47.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.993+0000 7f6ccdc00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc8113920 con 0x7f6cc81096b0 2024-03-28T17:48:47.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.993+0000 7f6cbf400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cc8108cf0 0x7f6cc819e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:47.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.993+0000 7f6cc7e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cc8107b00 0x7f6cc819de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:47.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.993+0000 7f6cc7e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cc8107b00 0x7f6cc819de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:46452/0 (socket says 172.21.15.45:46452) 2024-03-28T17:48:47.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.993+0000 7f6cc7e00640 1 -- 172.21.15.45:0/3732346279 learned_addr learned my addr 172.21.15.45:0/3732346279 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:47.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.993+0000 7f6ccd200640 1 --2- 172.21.15.45:0/3732346279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6cc81096b0 0x7f6cc81a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:47.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.994+0000 7f6cc7e00640 1 -- 172.21.15.45:0/3732346279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6cc81096b0 msgr2=0x7f6cc81a53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:47.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.994+0000 7f6cc7e00640 1 --2- 172.21.15.45:0/3732346279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6cc81096b0 0x7f6cc81a53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.994+0000 7f6cc7e00640 1 -- 172.21.15.45:0/3732346279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cc8108cf0 msgr2=0x7f6cc819e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:47.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.994+0000 7f6cc7e00640 1 --2- 172.21.15.45:0/3732346279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cc8108cf0 0x7f6cc819e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:47.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.994+0000 7f6cc7e00640 1 -- 172.21.15.45:0/3732346279 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc81a5a30 con 0x7f6cc8107b00 2024-03-28T17:48:47.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.994+0000 7f6ccd200640 1 --2- 172.21.15.45:0/3732346279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6cc81096b0 0x7f6cc81a53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:47.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.994+0000 7f6cc7e00640 1 --2- 172.21.15.45:0/3732346279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cc8107b00 0x7f6cc819de10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6cb802c500 tx=0x7f6cb8033830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:47.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.995+0000 7f6cc5600640 1 -- 172.21.15.45:0/3732346279 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8043020 con 0x7f6cc8107b00 2024-03-28T17:48:47.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.995+0000 7f6cc5600640 1 -- 172.21.15.45:0/3732346279 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6cb802ca60 con 0x7f6cc8107b00 2024-03-28T17:48:47.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.995+0000 7f6cc5600640 1 -- 172.21.15.45:0/3732346279 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8034690 con 0x7f6cc8107b00 2024-03-28T17:48:47.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.995+0000 7f6ccdc00640 1 -- 172.21.15.45:0/3732346279 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cc81a5cc0 con 0x7f6cc8107b00 2024-03-28T17:48:47.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.995+0000 7f6ccdc00640 1 -- 172.21.15.45:0/3732346279 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cc81a61c0 con 0x7f6cc8107b00 2024-03-28T17:48:47.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.996+0000 7f6ccdc00640 1 -- 172.21.15.45:0/3732346279 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cc810ee20 con 0x7f6cc8107b00 2024-03-28T17:48:47.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.998+0000 7f6cc5600640 1 -- 172.21.15.45:0/3732346279 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f6cb8046020 con 0x7f6cc8107b00 2024-03-28T17:48:47.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.999+0000 7f6cc5600640 1 --2- 172.21.15.45:0/3732346279 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6cac0a2910 0x7f6cac0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:47.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.999+0000 7f6cc5600640 1 -- 172.21.15.45:0/3732346279 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6cb80f1270 con 0x7f6cc8107b00 2024-03-28T17:48:47.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:47.999+0000 7f6cbf400640 1 --2- 172.21.15.45:0/3732346279 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6cac0a2910 0x7f6cac0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:48.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.000+0000 7f6cbf400640 1 --2- 172.21.15.45:0/3732346279 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6cac0a2910 0x7f6cac0a4dd0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6cc819f320 tx=0x7f6cb0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:48.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.003+0000 7f6cc5600640 1 -- 172.21.15.45:0/3732346279 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6cb804d020 con 0x7f6cc8107b00 2024-03-28T17:48:48.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:47 smithi071 ceph-mon[36597]: pgmap v396: 97 pgs: 97 active+clean; 649 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-28T17:48:48.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:47 smithi071 ceph-mon[36597]: from='client.25264 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:48:48.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:47 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3168134071' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:48:48.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:47 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:48.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:47 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:48.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:47 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/462374682' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T17:48:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:47 smithi151 ceph-mon[36481]: pgmap v396: 97 pgs: 97 active+clean; 649 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-28T17:48:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:47 smithi151 ceph-mon[36481]: from='client.25264 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T17:48:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:47 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3168134071' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T17:48:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:47 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:47 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:47 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/462374682' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T17:48:48.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.251+0000 7f6ccdc00640 1 -- 172.21.15.45:0/3732346279 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6cc802d070 con 0x7f6cc8107b00 2024-03-28T17:48:48.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.252+0000 7f6cc5600640 1 -- 172.21.15.45:0/3732346279 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f6cb80be2d0 con 0x7f6cc8107b00 2024-03-28T17:48:48.253 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:48:48.253 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6839/2313500150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":2313500150},{"type":"v1","addr":"172.21.15.45:6839","nonce":2313500150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6841/2647972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":2647972571},{"type":"v1","addr":"172.21.15.45:6841","nonce":2647972571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T17:46:12.261135+0000","modified":"2024-03-28T17:48:38.762497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24584,"mds_1":24530,"mds_2":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6835/2876354410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":2876354410},{"type":"v1","addr":"172.21.15.151:6835","nonce":2876354410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24500":{"gid":24500,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/4234998385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":4234998385},{"type":"v1","addr":"172.21.15.71:6835","nonce":4234998385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6837/1238964845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":1238964845},{"type":"v1","addr":"172.21.15.151:6837","nonce":1238964845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.151:6839/3003564901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6838","nonce":3003564901},{"type":"v1","addr":"172.21.15.151:6839","nonce":3003564901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6837/2008752655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2008752655},{"type":"v1","addr":"172.21.15.71:6837","nonce":2008752655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.71:6839/2709938800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":2709938800},{"type":"v1","addr":"172.21.15.71:6839","nonce":2709938800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24584,"qdb_cluster":[24584,24529,24530]},"id":1}]} 2024-03-28T17:48:48.254 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 23 2024-03-28T17:48:48.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.254+0000 7f6ccdc00640 1 -- 172.21.15.45:0/3732346279 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6cac0a2910 msgr2=0x7f6cac0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:48.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.254+0000 7f6ccdc00640 1 --2- 172.21.15.45:0/3732346279 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6cac0a2910 0x7f6cac0a4dd0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6cc819f320 tx=0x7f6cb0009260 comp rx=0 tx=0).stop 2024-03-28T17:48:48.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.255+0000 7f6ccdc00640 1 -- 172.21.15.45:0/3732346279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cc8107b00 msgr2=0x7f6cc819de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:48.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.255+0000 7f6ccdc00640 1 --2- 172.21.15.45:0/3732346279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cc8107b00 0x7f6cc819de10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6cb802c500 tx=0x7f6cb8033830 comp rx=0 tx=0).stop 2024-03-28T17:48:48.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.255+0000 7f6ccdc00640 1 -- 172.21.15.45:0/3732346279 shutdown_connections 2024-03-28T17:48:48.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.255+0000 7f6ccdc00640 1 --2- 172.21.15.45:0/3732346279 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f6cac0a2910 0x7f6cac0a4dd0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.255+0000 7f6ccdc00640 1 --2- 172.21.15.45:0/3732346279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6cc81096b0 0x7f6cc81a53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.255+0000 7f6ccdc00640 1 --2- 172.21.15.45:0/3732346279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cc8108cf0 0x7f6cc819e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.255+0000 7f6ccdc00640 1 --2- 172.21.15.45:0/3732346279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cc8107b00 0x7f6cc819de10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.255+0000 7f6ccdc00640 1 -- 172.21.15.45:0/3732346279 >> 172.21.15.45:0/3732346279 conn(0x7f6cc8076ab0 msgr2=0x7f6cc810bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:48.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.255+0000 7f6ccdc00640 1 -- 172.21.15.45:0/3732346279 shutdown_connections 2024-03-28T17:48:48.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.255+0000 7f6ccdc00640 1 -- 172.21.15.45:0/3732346279 wait complete. 2024-03-28T17:48:48.274 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T17:48:48.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.487+0000 7fdcb2200640 1 -- 172.21.15.45:0/4203178676 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcac105490 msgr2=0x7fdcac10bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:48.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.487+0000 7fdcb2200640 1 --2- 172.21.15.45:0/4203178676 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcac105490 0x7fdcac10bd20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdca4009900 tx=0x7fdca402bc30 comp rx=0 tx=0).stop 2024-03-28T17:48:48.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.488+0000 7fdcb2200640 1 -- 172.21.15.45:0/4203178676 shutdown_connections 2024-03-28T17:48:48.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.488+0000 7fdcb2200640 1 --2- 172.21.15.45:0/4203178676 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcac105490 0x7fdcac10bd20 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.488+0000 7fdcb2200640 1 --2- 172.21.15.45:0/4203178676 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcac104ad0 0x7fdcac104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.488+0000 7fdcb2200640 1 --2- 172.21.15.45:0/4203178676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcac1038e0 0x7fdcac103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.488+0000 7fdcb2200640 1 -- 172.21.15.45:0/4203178676 >> 172.21.15.45:0/4203178676 conn(0x7fdcac0fee90 msgr2=0x7fdcac1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:48.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.488+0000 7fdcb2200640 1 -- 172.21.15.45:0/4203178676 shutdown_connections 2024-03-28T17:48:48.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.488+0000 7fdcb2200640 1 -- 172.21.15.45:0/4203178676 wait complete. 2024-03-28T17:48:48.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.489+0000 7fdcb2200640 1 Processor -- start 2024-03-28T17:48:48.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.489+0000 7fdcb2200640 1 -- start start 2024-03-28T17:48:48.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.489+0000 7fdcb2200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcac1038e0 0x7fdcac199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:48.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.489+0000 7fdcb2200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcac104ad0 0x7fdcac19a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:48.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.489+0000 7fdcb2200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcac105490 0x7fdcac1a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:48.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.489+0000 7fdcb2200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac10f590 con 0x7fdcac105490 2024-03-28T17:48:48.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.489+0000 7fdcb2200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac10f410 con 0x7fdcac104ad0 2024-03-28T17:48:48.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.489+0000 7fdcb2200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac10f710 con 0x7fdcac1038e0 2024-03-28T17:48:48.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.490+0000 7fdcaaa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcac104ad0 0x7fdcac19a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:48.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.490+0000 7fdcab400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcac1038e0 0x7fdcac199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:48.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.490+0000 7fdcab400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcac1038e0 0x7fdcac199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:60788/0 (socket says 172.21.15.45:60788) 2024-03-28T17:48:48.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.490+0000 7fdcab400640 1 -- 172.21.15.45:0/81568504 learned_addr learned my addr 172.21.15.45:0/81568504 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:48.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.490+0000 7fdcabe00640 1 --2- 172.21.15.45:0/81568504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcac105490 0x7fdcac1a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:48.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.490+0000 7fdcab400640 1 -- 172.21.15.45:0/81568504 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcac104ad0 msgr2=0x7fdcac19a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:48.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.490+0000 7fdcab400640 1 --2- 172.21.15.45:0/81568504 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcac104ad0 0x7fdcac19a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.490+0000 7fdcab400640 1 -- 172.21.15.45:0/81568504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcac105490 msgr2=0x7fdcac1a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:48.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.490+0000 7fdcab400640 1 --2- 172.21.15.45:0/81568504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcac105490 0x7fdcac1a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.490+0000 7fdcab400640 1 -- 172.21.15.45:0/81568504 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcac1a1760 con 0x7fdcac1038e0 2024-03-28T17:48:48.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.490+0000 7fdcabe00640 1 --2- 172.21.15.45:0/81568504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcac105490 0x7fdcac1a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:48:48.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.490+0000 7fdcaaa00640 1 --2- 172.21.15.45:0/81568504 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcac104ad0 0x7fdcac19a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:48.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.490+0000 7fdcab400640 1 --2- 172.21.15.45:0/81568504 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcac1038e0 0x7fdcac199b40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c00e9c0 tx=0x7fdc9c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:48.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.490+0000 7fdca3e00640 1 -- 172.21.15.45:0/81568504 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c00cd80 con 0x7fdcac1038e0 2024-03-28T17:48:48.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.491+0000 7fdca3e00640 1 -- 172.21.15.45:0/81568504 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc9c0035f0 con 0x7fdcac1038e0 2024-03-28T17:48:48.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.491+0000 7fdca3e00640 1 -- 172.21.15.45:0/81568504 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c010690 con 0x7fdcac1038e0 2024-03-28T17:48:48.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.491+0000 7fdcb2200640 1 -- 172.21.15.45:0/81568504 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcac1a1a50 con 0x7fdcac1038e0 2024-03-28T17:48:48.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.491+0000 7fdcb2200640 1 -- 172.21.15.45:0/81568504 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcac1a1f50 con 0x7fdcac1038e0 2024-03-28T17:48:48.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.492+0000 7fdcb2200640 1 -- 172.21.15.45:0/81568504 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcac10aba0 con 0x7fdcac1038e0 2024-03-28T17:48:48.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.494+0000 7fdca3e00640 1 -- 172.21.15.45:0/81568504 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 19) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fdc9c0037a0 con 0x7fdcac1038e0 2024-03-28T17:48:48.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.495+0000 7fdca3e00640 1 --2- 172.21.15.45:0/81568504 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdc8c0a2890 0x7fdc8c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:48.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.495+0000 7fdca3e00640 1 -- 172.21.15.45:0/81568504 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fdc9c0cf040 con 0x7fdcac1038e0 2024-03-28T17:48:48.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.496+0000 7fdcaaa00640 1 --2- 172.21.15.45:0/81568504 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdc8c0a2890 0x7fdc8c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:48.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.496+0000 7fdcaaa00640 1 --2- 172.21.15.45:0/81568504 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdc8c0a2890 0x7fdc8c0a4d50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdcac19b050 tx=0x7fdc94009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:48.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.499+0000 7fdca3e00640 1 -- 172.21.15.45:0/81568504 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdc9c09bf90 con 0x7fdcac1038e0 2024-03-28T17:48:48.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.709+0000 7fdcb2200640 1 -- 172.21.15.45:0/81568504 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcac02d260 con 0x7fdcac1038e0 2024-03-28T17:48:48.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.711+0000 7fdca3e00640 1 -- 172.21.15.45:0/81568504 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22583 (secure 0 0 0) 0x7fdc9c084e20 con 0x7fdcac1038e0 2024-03-28T17:48:48.712 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:48:48.712 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"9f4888ec-ed29-11ee-95d0-87774f69a715","created":"2024-03-28T17:36:51.623185+0000","modified":"2024-03-28T17:47:09.455843+0000","last_up_change":"2024-03-28T17:45:14.043432+0000","last_in_change":"2024-03-28T17:44:53.654489+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T17:40:54.208375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T17:46:10.564519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T17:46:11.734367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"74c56bc2-4169-4919-afd4-2ebd236f23e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6803","nonce":3795754023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6805","nonce":3795754023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6809","nonce":3795754023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3795754023},{"type":"v1","addr":"172.21.15.45:6807","nonce":3795754023}]},"public_addr":"172.21.15.45:6803/3795754023","cluster_addr":"172.21.15.45:6805/3795754023","heartbeat_back_addr":"172.21.15.45:6809/3795754023","heartbeat_front_addr":"172.21.15.45:6807/3795754023","state":["exists","up"]},{"osd":1,"uuid":"78dcc21b-75b2-453d-b690-17b55b8ed1a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6801","nonce":2014529906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6803","nonce":2014529906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6807","nonce":2014529906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2014529906},{"type":"v1","addr":"172.21.15.71:6805","nonce":2014529906}]},"public_addr":"172.21.15.71:6801/2014529906","cluster_addr":"172.21.15.71:6803/2014529906","heartbeat_back_addr":"172.21.15.71:6807/2014529906","heartbeat_front_addr":"172.21.15.71:6805/2014529906","state":["exists","up"]},{"osd":2,"uuid":"ca9f3f36-b71e-4962-a94f-cedf2f12eb7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6801","nonce":1850933775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6803","nonce":1850933775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6807","nonce":1850933775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1850933775},{"type":"v1","addr":"172.21.15.151:6805","nonce":1850933775}]},"public_addr":"172.21.15.151:6801/1850933775","cluster_addr":"172.21.15.151:6803/1850933775","heartbeat_back_addr":"172.21.15.151:6807/1850933775","heartbeat_front_addr":"172.21.15.151:6805/1850933775","state":["exists","up"]},{"osd":3,"uuid":"d3f21f01-2973-413e-8daa-9974b8a44af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6811","nonce":3021569975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6813","nonce":3021569975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6817","nonce":3021569975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3021569975},{"type":"v1","addr":"172.21.15.45:6815","nonce":3021569975}]},"public_addr":"172.21.15.45:6811/3021569975","cluster_addr":"172.21.15.45:6813/3021569975","heartbeat_back_addr":"172.21.15.45:6817/3021569975","heartbeat_front_addr":"172.21.15.45:6815/3021569975","state":["exists","up"]},{"osd":4,"uuid":"4a336f42-0605-492b-ad88-56cb02acdcf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6809","nonce":2069578333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6811","nonce":2069578333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6815","nonce":2069578333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2069578333},{"type":"v1","addr":"172.21.15.71:6813","nonce":2069578333}]},"public_addr":"172.21.15.71:6809/2069578333","cluster_addr":"172.21.15.71:6811/2069578333","heartbeat_back_addr":"172.21.15.71:6815/2069578333","heartbeat_front_addr":"172.21.15.71:6813/2069578333","state":["exists","up"]},{"osd":5,"uuid":"c3ff0c0a-7db8-4aa0-ae8f-1e2b5be345a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6809","nonce":2629217227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6811","nonce":2629217227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6815","nonce":2629217227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":2629217227},{"type":"v1","addr":"172.21.15.151:6813","nonce":2629217227}]},"public_addr":"172.21.15.151:6809/2629217227","cluster_addr":"172.21.15.151:6811/2629217227","heartbeat_back_addr":"172.21.15.151:6815/2629217227","heartbeat_front_addr":"172.21.15.151:6813/2629217227","state":["exists","up"]},{"osd":6,"uuid":"58a6c5bd-a8b7-4892-b346-bcd7994bcb6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6819","nonce":2077907039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6821","nonce":2077907039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6825","nonce":2077907039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2077907039},{"type":"v1","addr":"172.21.15.45:6823","nonce":2077907039}]},"public_addr":"172.21.15.45:6819/2077907039","cluster_addr":"172.21.15.45:6821/2077907039","heartbeat_back_addr":"172.21.15.45:6825/2077907039","heartbeat_front_addr":"172.21.15.45:6823/2077907039","state":["exists","up"]},{"osd":7,"uuid":"f7a45d52-807b-4407-bb04-787b2a52a0ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6817","nonce":2000211251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6819","nonce":2000211251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6823","nonce":2000211251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2000211251},{"type":"v1","addr":"172.21.15.71:6821","nonce":2000211251}]},"public_addr":"172.21.15.71:6817/2000211251","cluster_addr":"172.21.15.71:6819/2000211251","heartbeat_back_addr":"172.21.15.71:6823/2000211251","heartbeat_front_addr":"172.21.15.71:6821/2000211251","state":["exists","up"]},{"osd":8,"uuid":"88abcb0f-f189-49be-8315-c700c77369f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6817","nonce":3665460137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6819","nonce":3665460137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6823","nonce":3665460137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3665460137},{"type":"v1","addr":"172.21.15.151:6821","nonce":3665460137}]},"public_addr":"172.21.15.151:6817/3665460137","cluster_addr":"172.21.15.151:6819/3665460137","heartbeat_back_addr":"172.21.15.151:6823/3665460137","heartbeat_front_addr":"172.21.15.151:6821/3665460137","state":["exists","up"]},{"osd":9,"uuid":"ed3fc096-961e-4392-abc2-c5aa4d050117","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6827","nonce":2122322684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6829","nonce":2122322684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6833","nonce":2122322684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2122322684},{"type":"v1","addr":"172.21.15.45:6831","nonce":2122322684}]},"public_addr":"172.21.15.45:6827/2122322684","cluster_addr":"172.21.15.45:6829/2122322684","heartbeat_back_addr":"172.21.15.45:6833/2122322684","heartbeat_front_addr":"172.21.15.45:6831/2122322684","state":["exists","up"]},{"osd":10,"uuid":"ba94762b-bc5c-4c97-9cc1-17e5651c14b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6825","nonce":1099494037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6827","nonce":1099494037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6831","nonce":1099494037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1099494037},{"type":"v1","addr":"172.21.15.71:6829","nonce":1099494037}]},"public_addr":"172.21.15.71:6825/1099494037","cluster_addr":"172.21.15.71:6827/1099494037","heartbeat_back_addr":"172.21.15.71:6831/1099494037","heartbeat_front_addr":"172.21.15.71:6829/1099494037","state":["exists","up"]},{"osd":11,"uuid":"1d178378-b9f5-40f6-99ef-3d3d445da640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6825","nonce":2941055364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6827","nonce":2941055364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6831","nonce":2941055364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2941055364},{"type":"v1","addr":"172.21.15.151:6829","nonce":2941055364}]},"public_addr":"172.21.15.151:6825/2941055364","cluster_addr":"172.21.15.151:6827/2941055364","heartbeat_back_addr":"172.21.15.151:6831/2941055364","heartbeat_front_addr":"172.21.15.151:6829/2941055364","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:02.269586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:27.819386+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:40:51.552277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:25.052674+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:41:51.873529+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:17.127867+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:42:51.588249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:18.381450+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:43:44.140534+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:17.634362+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:44:45.469340+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T17:45:12.003433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3533611957":"2024-03-29T17:38:07.505611+0000","172.21.15.45:0/3625922430":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6801/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/3951447106":"2024-03-29T17:38:07.505611+0000","172.21.15.45:6800/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/482610468":"2024-03-29T17:37:32.424229+0000","172.21.15.45:6801/4186185610":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3649804714":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/1198364079":"2024-03-29T17:37:32.424229+0000","172.21.15.45:0/3211303958":"2024-03-29T17:38:07.505611+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T17:48:48.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.714+0000 7fdcb2200640 1 -- 172.21.15.45:0/81568504 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdc8c0a2890 msgr2=0x7fdc8c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:48.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.714+0000 7fdcb2200640 1 --2- 172.21.15.45:0/81568504 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdc8c0a2890 0x7fdc8c0a4d50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdcac19b050 tx=0x7fdc94009500 comp rx=0 tx=0).stop 2024-03-28T17:48:48.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.714+0000 7fdcb2200640 1 -- 172.21.15.45:0/81568504 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcac1038e0 msgr2=0x7fdcac199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:48.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.714+0000 7fdcb2200640 1 --2- 172.21.15.45:0/81568504 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcac1038e0 0x7fdcac199b40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c00e9c0 tx=0x7fdc9c00ee80 comp rx=0 tx=0).stop 2024-03-28T17:48:48.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.714+0000 7fdcb2200640 1 -- 172.21.15.45:0/81568504 shutdown_connections 2024-03-28T17:48:48.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.714+0000 7fdcb2200640 1 --2- 172.21.15.45:0/81568504 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fdc8c0a2890 0x7fdc8c0a4d50 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.714+0000 7fdcb2200640 1 --2- 172.21.15.45:0/81568504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcac105490 0x7fdcac1a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.714+0000 7fdcb2200640 1 --2- 172.21.15.45:0/81568504 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcac104ad0 0x7fdcac19a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.714+0000 7fdcb2200640 1 --2- 172.21.15.45:0/81568504 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdcac1038e0 0x7fdcac199b40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.714+0000 7fdcb2200640 1 -- 172.21.15.45:0/81568504 >> 172.21.15.45:0/81568504 conn(0x7fdcac0fee90 msgr2=0x7fdcac107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:48.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.714+0000 7fdcb2200640 1 -- 172.21.15.45:0/81568504 shutdown_connections 2024-03-28T17:48:48.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.714+0000 7fdcb2200640 1 -- 172.21.15.45:0/81568504 wait complete. 2024-03-28T17:48:48.735 DEBUG:tasks.fwd_scrub:Yielding 2024-03-28T17:48:48.735 INFO:teuthology.run_tasks:Running task exec... 2024-03-28T17:48:48.744 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T17:48:48.745 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi045.front.sepia.ceph.com 2024-03-28T17:48:48.745 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-28T17:48:48.745 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-28T17:48:48.745 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T17:48:48.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T17:48:48.746 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T17:48:48.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:48 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1603345402' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T17:48:48.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:48 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3732346279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:48:48.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:48 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:48:48.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:48 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:48:48.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:48 smithi045 ceph-mon[31773]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:48.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.946+0000 7fe510a00640 1 -- 172.21.15.45:0/2518646967 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe50c1054a0 msgr2=0x7fe50c10bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:48.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.946+0000 7fe510a00640 1 --2- 172.21.15.45:0/2518646967 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe50c1054a0 0x7fe50c10bd30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe504009ca0 tx=0x7fe50402c0f0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.946+0000 7fe510a00640 1 -- 172.21.15.45:0/2518646967 shutdown_connections 2024-03-28T17:48:48.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.946+0000 7fe510a00640 1 --2- 172.21.15.45:0/2518646967 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe50c1054a0 0x7fe50c10bd30 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.946+0000 7fe510a00640 1 --2- 172.21.15.45:0/2518646967 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe50c104ae0 0x7fe50c104f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.946+0000 7fe510a00640 1 --2- 172.21.15.45:0/2518646967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe50c1038f0 0x7fe50c103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.946+0000 7fe510a00640 1 -- 172.21.15.45:0/2518646967 >> 172.21.15.45:0/2518646967 conn(0x7fe50c0fee60 msgr2=0x7fe50c1012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:48.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.946+0000 7fe510a00640 1 -- 172.21.15.45:0/2518646967 shutdown_connections 2024-03-28T17:48:48.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.946+0000 7fe510a00640 1 -- 172.21.15.45:0/2518646967 wait complete. 2024-03-28T17:48:48.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.947+0000 7fe510a00640 1 Processor -- start 2024-03-28T17:48:48.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.947+0000 7fe510a00640 1 -- start start 2024-03-28T17:48:48.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.948+0000 7fe510a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe50c1038f0 0x7fe50c199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:48.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.948+0000 7fe510a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe50c104ae0 0x7fe50c19a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:48.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.948+0000 7fe510a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe50c1054a0 0x7fe50c1a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:48.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.948+0000 7fe510a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe50c10f550 con 0x7fe50c104ae0 2024-03-28T17:48:48.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.948+0000 7fe510a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe50c10f3d0 con 0x7fe50c1038f0 2024-03-28T17:48:48.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.948+0000 7fe510a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe50c10f6d0 con 0x7fe50c1054a0 2024-03-28T17:48:48.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.948+0000 7fe509e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe50c1038f0 0x7fe50c199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:48.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.948+0000 7fe509400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe50c104ae0 0x7fe50c19a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:48.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.948+0000 7fe509e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe50c1038f0 0x7fe50c199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:46480/0 (socket says 172.21.15.45:46480) 2024-03-28T17:48:48.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.948+0000 7fe509e00640 1 -- 172.21.15.45:0/3043207053 learned_addr learned my addr 172.21.15.45:0/3043207053 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:48.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.948+0000 7fe50a800640 1 --2- 172.21.15.45:0/3043207053 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe50c1054a0 0x7fe50c1a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:48.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.948+0000 7fe509e00640 1 -- 172.21.15.45:0/3043207053 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe50c1054a0 msgr2=0x7fe50c1a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:48.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.948+0000 7fe509e00640 1 --2- 172.21.15.45:0/3043207053 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe50c1054a0 0x7fe50c1a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.948+0000 7fe509e00640 1 -- 172.21.15.45:0/3043207053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe50c104ae0 msgr2=0x7fe50c19a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:48.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.948+0000 7fe509e00640 1 --2- 172.21.15.45:0/3043207053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe50c104ae0 0x7fe50c19a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.948+0000 7fe509e00640 1 -- 172.21.15.45:0/3043207053 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe50c1a1760 con 0x7fe50c1038f0 2024-03-28T17:48:48.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.949+0000 7fe509e00640 1 --2- 172.21.15.45:0/3043207053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe50c1038f0 0x7fe50c199b40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc00e9c0 tx=0x7fe4fc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:48.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.949+0000 7fe502a00640 1 -- 172.21.15.45:0/3043207053 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc00cd80 con 0x7fe50c1038f0 2024-03-28T17:48:48.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.949+0000 7fe502a00640 1 -- 172.21.15.45:0/3043207053 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4fc0035f0 con 0x7fe50c1038f0 2024-03-28T17:48:48.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.949+0000 7fe510a00640 1 -- 172.21.15.45:0/3043207053 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe50c1a1a50 con 0x7fe50c1038f0 2024-03-28T17:48:48.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.949+0000 7fe502a00640 1 -- 172.21.15.45:0/3043207053 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc010690 con 0x7fe50c1038f0 2024-03-28T17:48:48.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.949+0000 7fe510a00640 1 -- 172.21.15.45:0/3043207053 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe50c1a1ff0 con 0x7fe50c1038f0 2024-03-28T17:48:48.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.950+0000 7fe510a00640 1 -- 172.21.15.45:0/3043207053 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe50c04f400 con 0x7fe50c1038f0 2024-03-28T17:48:48.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.952+0000 7fe502a00640 1 -- 172.21.15.45:0/3043207053 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fe4fc0037a0 con 0x7fe50c1038f0 2024-03-28T17:48:48.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.953+0000 7fe502a00640 1 --2- 172.21.15.45:0/3043207053 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe4f00a29c0 0x7fe4f00a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:48.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.954+0000 7fe502a00640 1 -- 172.21.15.45:0/3043207053 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe4fc0cef70 con 0x7fe50c1038f0 2024-03-28T17:48:48.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.954+0000 7fe509400640 1 --2- 172.21.15.45:0/3043207053 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe4f00a29c0 0x7fe4f00a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:48.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.954+0000 7fe509400640 1 --2- 172.21.15.45:0/3043207053 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe4f00a29c0 0x7fe4f00a4e80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe50c19b050 tx=0x7fe4f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:48.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.957+0000 7fe502a00640 1 -- 172.21.15.45:0/3043207053 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe4fc09be60 con 0x7fe50c1038f0 2024-03-28T17:48:48.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.961+0000 7f89d1000640 1 -- 172.21.15.45:0/1116713364 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89cc1028a0 msgr2=0x7f89cc102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:48.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.961+0000 7f89d1000640 1 --2- 172.21.15.45:0/1116713364 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89cc1028a0 0x7f89cc102d20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f89b8009900 tx=0x7f89b802bc30 comp rx=0 tx=0).stop 2024-03-28T17:48:48.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.961+0000 7f89d1000640 1 -- 172.21.15.45:0/1116713364 shutdown_connections 2024-03-28T17:48:48.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.961+0000 7f89d1000640 1 --2- 172.21.15.45:0/1116713364 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89cc103260 0x7f89cc109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.961+0000 7f89d1000640 1 --2- 172.21.15.45:0/1116713364 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89cc1028a0 0x7f89cc102d20 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.961+0000 7f89d1000640 1 --2- 172.21.15.45:0/1116713364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89cc1016b0 0x7f89cc101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.961+0000 7f89d1000640 1 -- 172.21.15.45:0/1116713364 >> 172.21.15.45:0/1116713364 conn(0x7f89cc0fcc40 msgr2=0x7f89cc0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:48.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.961+0000 7f89d1000640 1 -- 172.21.15.45:0/1116713364 shutdown_connections 2024-03-28T17:48:48.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.961+0000 7f89d1000640 1 -- 172.21.15.45:0/1116713364 wait complete. 2024-03-28T17:48:48.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.962+0000 7f89d1000640 1 Processor -- start 2024-03-28T17:48:48.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.962+0000 7f89d1000640 1 -- start start 2024-03-28T17:48:48.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.962+0000 7f89d1000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89cc1016b0 0x7f89cc199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:48.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.962+0000 7f89d1000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89cc1028a0 0x7f89cc19a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:48.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.962+0000 7f89d1000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89cc103260 0x7f89cc1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:48.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.962+0000 7f89d1000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc10d380 con 0x7f89cc103260 2024-03-28T17:48:48.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.962+0000 7f89d1000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc10d200 con 0x7f89cc1028a0 2024-03-28T17:48:48.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.962+0000 7f89d1000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc10d500 con 0x7f89cc1016b0 2024-03-28T17:48:48.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.963+0000 7f89c9400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89cc1028a0 0x7f89cc19a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:48.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.963+0000 7f89ca800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89cc103260 0x7f89cc1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:48.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.963+0000 7f89c9e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89cc1016b0 0x7f89cc199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:48.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.963+0000 7f89c9400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89cc1028a0 0x7f89cc19a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:46504/0 (socket says 172.21.15.45:46504) 2024-03-28T17:48:48.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.963+0000 7f89c9400640 1 -- 172.21.15.45:0/868166487 learned_addr learned my addr 172.21.15.45:0/868166487 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:48.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.963+0000 7f89c9400640 1 -- 172.21.15.45:0/868166487 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89cc1016b0 msgr2=0x7f89cc199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:48.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.963+0000 7f89c9400640 1 --2- 172.21.15.45:0/868166487 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89cc1016b0 0x7f89cc199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.963+0000 7f89c9400640 1 -- 172.21.15.45:0/868166487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89cc103260 msgr2=0x7f89cc1a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:48.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.963+0000 7f89c9400640 1 --2- 172.21.15.45:0/868166487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89cc103260 0x7f89cc1a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.963+0000 7f89c9400640 1 -- 172.21.15.45:0/868166487 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89cc1a1830 con 0x7f89cc1028a0 2024-03-28T17:48:48.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.963+0000 7f89c9400640 1 --2- 172.21.15.45:0/868166487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89cc1028a0 0x7f89cc19a160 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f89b8005ab0 tx=0x7f89b802c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:48.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.964+0000 7f89c2a00640 1 -- 172.21.15.45:0/868166487 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b8043020 con 0x7f89cc1028a0 2024-03-28T17:48:48.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.964+0000 7f89c2a00640 1 -- 172.21.15.45:0/868166487 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f89b80054e0 con 0x7f89cc1028a0 2024-03-28T17:48:48.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.964+0000 7f89c2a00640 1 -- 172.21.15.45:0/868166487 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b8003ce0 con 0x7f89cc1028a0 2024-03-28T17:48:48.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.964+0000 7f89d1000640 1 -- 172.21.15.45:0/868166487 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89cc1a1ac0 con 0x7f89cc1028a0 2024-03-28T17:48:48.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.964+0000 7f89d1000640 1 -- 172.21.15.45:0/868166487 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89cc1a2010 con 0x7f89cc1028a0 2024-03-28T17:48:48.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.967+0000 7f89c2a00640 1 -- 172.21.15.45:0/868166487 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f89b8046020 con 0x7f89cc1028a0 2024-03-28T17:48:48.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.967+0000 7f89d3c29740 1 Processor -- start 2024-03-28T17:48:48.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.967+0000 7f89d3c29740 1 -- start start 2024-03-28T17:48:48.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.968+0000 7f89d3c29740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555750cab0a0 0x555750cab4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:48.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.968+0000 7f89d3c29740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750caba90 0x555750cb0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:48.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.968+0000 7f89d3c29740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555750cb0c00 0x555750cb3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:48.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.968+0000 7f89d3c29740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555750c10eb0 con 0x555750cab0a0 2024-03-28T17:48:48.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.968+0000 7f89d3c29740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x555750bb9a60 con 0x555750caba90 2024-03-28T17:48:48.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.968+0000 7f89d3c29740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x555750c0cf50 con 0x555750cb0c00 2024-03-28T17:48:48.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.968+0000 7f89c9e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750caba90 0x555750cb0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:48.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.968+0000 7f89ca800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555750cab0a0 0x555750cab4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:48.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.968+0000 7f89c9e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750caba90 0x555750cb0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:46514/0 (socket says 172.21.15.45:46514) 2024-03-28T17:48:48.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.968+0000 7f89c9e00640 1 -- 172.21.15.45:0/1552825543 learned_addr learned my addr 172.21.15.45:0/1552825543 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:48.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.968+0000 7f89c9400640 1 --2- 172.21.15.45:0/1552825543 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555750cb0c00 0x555750cb3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:48.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.969+0000 7f89c9e00640 1 -- 172.21.15.45:0/1552825543 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555750cb0c00 msgr2=0x555750cb3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:48.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.969+0000 7f89c9e00640 1 --2- 172.21.15.45:0/1552825543 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555750cb0c00 0x555750cb3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.969+0000 7f89c9e00640 1 -- 172.21.15.45:0/1552825543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555750cab0a0 msgr2=0x555750cab4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:48.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.969+0000 7f89c9e00640 1 --2- 172.21.15.45:0/1552825543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555750cab0a0 0x555750cab4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.969+0000 7f89c9e00640 1 -- 172.21.15.45:0/1552825543 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557509ed7e0 con 0x555750caba90 2024-03-28T17:48:48.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.969+0000 7f89c2a00640 1 --2- 172.21.15.45:0/868166487 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f89b00a2890 0x7f89b00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:48.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.969+0000 7f89c9e00640 1 --2- 172.21.15.45:0/1552825543 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750caba90 0x555750cb0590 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f89bc009c60 tx=0x7f89bc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1d6db425a512c55 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:48.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.969+0000 7f89c2a00640 1 -- 172.21.15.45:0/868166487 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f89b80f0c70 con 0x7f89cc1028a0 2024-03-28T17:48:48.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.969+0000 7f89ca800640 1 --2- 172.21.15.45:0/868166487 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f89b00a2890 0x7f89b00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:48.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.969+0000 7f89b7400640 1 -- 172.21.15.45:0/1552825543 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89bc004220 con 0x555750caba90 2024-03-28T17:48:48.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.969+0000 7f89b7400640 1 -- 172.21.15.45:0/1552825543 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f89bc0043c0 con 0x555750caba90 2024-03-28T17:48:48.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.969+0000 7f89b7400640 1 -- 172.21.15.45:0/1552825543 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89bc0198d0 con 0x555750caba90 2024-03-28T17:48:48.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.970+0000 7f89d3c29740 1 -- 172.21.15.45:0/1552825543 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750caba90 msgr2=0x555750cb0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:48.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.970+0000 7f89d3c29740 1 --2- 172.21.15.45:0/1552825543 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750caba90 0x555750cb0590 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f89bc009c60 tx=0x7f89bc011810 comp rx=0 tx=0).stop 2024-03-28T17:48:48.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.970+0000 7f89ca800640 1 --2- 172.21.15.45:0/868166487 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f89b00a2890 0x7f89b00a4d50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x555750cac2f0 tx=0x7f89c4009600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:48.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.970+0000 7f89d3c29740 1 -- 172.21.15.45:0/1552825543 shutdown_connections 2024-03-28T17:48:48.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.970+0000 7f89d3c29740 1 --2- 172.21.15.45:0/1552825543 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555750cb0c00 0x555750cb3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.970+0000 7f89d3c29740 1 --2- 172.21.15.45:0/1552825543 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750caba90 0x555750cb0590 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.970+0000 7f89d3c29740 1 --2- 172.21.15.45:0/1552825543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555750cab0a0 0x555750cab4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.970+0000 7f89d3c29740 1 -- 172.21.15.45:0/1552825543 >> 172.21.15.45:0/1552825543 conn(0x555750ca6610 msgr2=0x555750ca6a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:48.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.970+0000 7f89d3c29740 1 -- 172.21.15.45:0/1552825543 shutdown_connections 2024-03-28T17:48:48.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.970+0000 7f89d3c29740 1 -- 172.21.15.45:0/1552825543 wait complete. 2024-03-28T17:48:48.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.971+0000 7f89d3c29740 1 -- start start 2024-03-28T17:48:48.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.971+0000 7f89d3c29740 1 Processor -- start 2024-03-28T17:48:48.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.971+0000 7f89d3c29740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750cab0a0 0x555750cb52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:48.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.971+0000 7f89d3c29740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555750caba90 0x555750cb5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:48.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.971+0000 7f89d3c29740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555750cb0c00 0x555750cbc860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:48.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.971+0000 7f89d3c29740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555750cb3bb0 con 0x555750cb0c00 2024-03-28T17:48:48.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.971+0000 7f89d3c29740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x555750bcceb0 con 0x555750cab0a0 2024-03-28T17:48:48.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.971+0000 7f89d3c29740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x555750cb3d30 con 0x555750caba90 2024-03-28T17:48:48.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.972+0000 7f89c9e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750cab0a0 0x555750cb52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:48.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.972+0000 7f89d3c29740 10 client.0 _ll_register_callbacks cb 0x555750bc1150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:48:48.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.972+0000 7f89c9e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750cab0a0 0x555750cb52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:46528/0 (socket says 172.21.15.45:46528) 2024-03-28T17:48:48.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.972+0000 7f89c9e00640 1 -- 172.21.15.45:0/4008754914 learned_addr learned my addr 172.21.15.45:0/4008754914 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:48.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.972+0000 7f89c9400640 1 --2- 172.21.15.45:0/4008754914 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555750caba90 0x555750cb5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:48.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.972+0000 7f89ca800640 1 --2- 172.21.15.45:0/4008754914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555750cb0c00 0x555750cbc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:48.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.972+0000 7f89c9e00640 1 -- 172.21.15.45:0/4008754914 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555750caba90 msgr2=0x555750cb5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:48.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.972+0000 7f89c9e00640 1 --2- 172.21.15.45:0/4008754914 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555750caba90 0x555750cb5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.972+0000 7f89c9e00640 1 -- 172.21.15.45:0/4008754914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555750cb0c00 msgr2=0x555750cbc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:48.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.972+0000 7f89c9e00640 1 --2- 172.21.15.45:0/4008754914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555750cb0c00 0x555750cbc860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.972+0000 7f89c9e00640 1 -- 172.21.15.45:0/4008754914 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555750a4df70 con 0x555750cab0a0 2024-03-28T17:48:48.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.972+0000 7f89ca800640 1 --2- 172.21.15.45:0/4008754914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555750cb0c00 0x555750cbc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:48:48.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.972+0000 7f89c9e00640 1 --2- 172.21.15.45:0/4008754914 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750cab0a0 0x555750cb52e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f89bc01b040 tx=0x7f89bc003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ceae1e06d7b5a784 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:48.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.972+0000 7f89abe00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:48:48.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.973+0000 7f89abe00640 1 -- 172.21.15.45:0/4008754914 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89bc028070 con 0x555750cab0a0 2024-03-28T17:48:48.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.973+0000 7f89abe00640 1 -- 172.21.15.45:0/4008754914 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f89bc005040 con 0x555750cab0a0 2024-03-28T17:48:48.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.973+0000 7f89d3c29740 1 -- 172.21.15.45:0/4008754914 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f89bc0043c0 con 0x555750cab0a0 2024-03-28T17:48:48.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.973+0000 7f89abe00640 1 -- 172.21.15.45:0/4008754914 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89bc034050 con 0x555750cab0a0 2024-03-28T17:48:48.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.973+0000 7f89d3c29740 10 client.25343 fetch_fsmap learned FSMap version 23 2024-03-28T17:48:48.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.973+0000 7f89d3c29740 1 -- 172.21.15.45:0/4008754914 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5557508e5170 con 0x555750cab0a0 2024-03-28T17:48:48.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.973+0000 7f89abe00640 1 -- 172.21.15.45:0/4008754914 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f89bc0043c0 con 0x555750cab0a0 2024-03-28T17:48:48.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.974+0000 7f89abe00640 1 -- 172.21.15.45:0/4008754914 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f89bc019b30 con 0x555750cab0a0 2024-03-28T17:48:48.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.974+0000 7f89d3c29740 10 client.25343 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:48:48.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.974+0000 7f89d3c29740 10 client.25343 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:48:48.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.974+0000 7f89d3c29740 20 client.25343 populate_metadata read hostname 'smithi045' 2024-03-28T17:48:48.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.974+0000 7f89d3c29740 1 --2- 172.21.15.45:0/4008754914 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555750cc0e70 0x555750cc32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:48.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.974+0000 7f89d3c29740 4 client.25343 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:48:48.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.974+0000 7f89d3c29740 1 -- 172.21.15.45:0/4008754914 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555750c1e3f0 con 0x555750cc0e70 2024-03-28T17:48:48.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.974+0000 7f89c9400640 1 --2- 172.21.15.45:0/4008754914 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555750cc0e70 0x555750cc32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:48.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.975+0000 7f89abe00640 10 client.25343 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:48:48.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.975+0000 7f89c9400640 1 --2- 172.21.15.45:0/4008754914 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555750cc0e70 0x555750cc32c0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3c55d31f5c152a3 server_cookie=25050939c72dc064 in_seq=0 out_seq=0 2024-03-28T17:48:48.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.975+0000 7f89abe00640 1 -- 172.21.15.45:0/4008754914 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555750c1e3f0 con 0x555750cc0e70 2024-03-28T17:48:48.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.975+0000 7f89abe00640 10 client.25343 handle_command_reply: tid=0 2024-03-28T17:48:48.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.975+0000 7f89d3c29740 1 client.25343 shutdown 2024-03-28T17:48:48.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.976+0000 7f89d3c29740 1 -- 172.21.15.45:0/4008754914 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750cab0a0 msgr2=0x555750cb52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:48.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.976+0000 7f89d3c29740 1 --2- 172.21.15.45:0/4008754914 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750cab0a0 0x555750cb52e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f89bc01b040 tx=0x7f89bc003820 comp rx=0 tx=0).stop 2024-03-28T17:48:48.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.976+0000 7f89d3c29740 1 -- 172.21.15.45:0/4008754914 shutdown_connections 2024-03-28T17:48:48.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.976+0000 7f89d3c29740 1 --2- 172.21.15.45:0/4008754914 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555750cc0e70 0x555750cc32c0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.976+0000 7f89d3c29740 1 --2- 172.21.15.45:0/4008754914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555750cb0c00 0x555750cbc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.976+0000 7f89abe00640 0 client.25343 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:48:48.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.976+0000 7f89d3c29740 1 --2- 172.21.15.45:0/4008754914 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555750caba90 0x555750cb5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.976+0000 7f89d3c29740 1 --2- 172.21.15.45:0/4008754914 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750cab0a0 0x555750cb52e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.976+0000 7f89d3c29740 1 -- 172.21.15.45:0/4008754914 >> 172.21.15.45:0/4008754914 conn(0x555750ca6610 msgr2=0x555750ca5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:48.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.976+0000 7f89d3c29740 1 -- 172.21.15.45:0/4008754914 shutdown_connections 2024-03-28T17:48:48.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.976+0000 7f89d3c29740 1 -- 172.21.15.45:0/4008754914 wait complete. 2024-03-28T17:48:48.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.976+0000 7f89d3c29740 20 client.25343 trim_cache size 0 max 16384 2024-03-28T17:48:48.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.994+0000 7f89d3c29740 1 Processor -- start 2024-03-28T17:48:48.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.994+0000 7f89d3c29740 1 -- start start 2024-03-28T17:48:48.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.994+0000 7f89d3c29740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555750cab0a0 0x555750cc7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:48.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.994+0000 7f89d3c29740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555750caba90 0x555750cbc820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:48.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.994+0000 7f89d3c29740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750cb0c00 0x555750cc8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:48.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.994+0000 7f89d3c29740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc1acff0 con 0x555750caba90 2024-03-28T17:48:48.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.994+0000 7f89d3c29740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc1aaaf0 con 0x555750cb0c00 2024-03-28T17:48:48.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.994+0000 7f89d3c29740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc1a8b90 con 0x555750cab0a0 2024-03-28T17:48:48.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.995+0000 7f89ca800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750cb0c00 0x555750cc8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:48.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.995+0000 7f89c9e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555750cab0a0 0x555750cc7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:48.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.995+0000 7f89ca800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750cb0c00 0x555750cc8100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:46534/0 (socket says 172.21.15.45:46534) 2024-03-28T17:48:48.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.995+0000 7f89ca800640 1 -- 172.21.15.45:0/2420378895 learned_addr learned my addr 172.21.15.45:0/2420378895 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:48.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.995+0000 7f89c9400640 1 --2- 172.21.15.45:0/2420378895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555750caba90 0x555750cbc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:48.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.995+0000 7f89ca800640 1 -- 172.21.15.45:0/2420378895 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555750cab0a0 msgr2=0x555750cc7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:48.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.995+0000 7f89ca800640 1 --2- 172.21.15.45:0/2420378895 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555750cab0a0 0x555750cc7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.995+0000 7f89ca800640 1 -- 172.21.15.45:0/2420378895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555750caba90 msgr2=0x555750cbc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:48.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.995+0000 7f89ca800640 1 --2- 172.21.15.45:0/2420378895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555750caba90 0x555750cbc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:48.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.995+0000 7f89ca800640 1 -- 172.21.15.45:0/2420378895 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555750cb3a90 con 0x555750cb0c00 2024-03-28T17:48:48.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.995+0000 7f89c9e00640 1 --2- 172.21.15.45:0/2420378895 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555750cab0a0 0x555750cc7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:48.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.995+0000 7f89ca800640 1 --2- 172.21.15.45:0/2420378895 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750cb0c00 0x555750cc8100 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f89c4003c80 tx=0x7f89c4016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64d30efe25afff16 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:49.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.996+0000 7f89b6a00640 1 -- 172.21.15.45:0/2420378895 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c40047b0 con 0x555750cb0c00 2024-03-28T17:48:49.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.996+0000 7f89b6a00640 1 -- 172.21.15.45:0/2420378895 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f89c4026030 con 0x555750cb0c00 2024-03-28T17:48:49.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.996+0000 7f89b6a00640 1 -- 172.21.15.45:0/2420378895 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c4022870 con 0x555750cb0c00 2024-03-28T17:48:49.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.996+0000 7f89d3c29740 1 -- 172.21.15.45:0/2420378895 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750cb0c00 msgr2=0x555750cc8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:49.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.996+0000 7f89d3c29740 1 --2- 172.21.15.45:0/2420378895 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750cb0c00 0x555750cc8100 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f89c4003c80 tx=0x7f89c4016000 comp rx=0 tx=0).stop 2024-03-28T17:48:49.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.996+0000 7f89d3c29740 1 -- 172.21.15.45:0/2420378895 shutdown_connections 2024-03-28T17:48:49.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.996+0000 7f89d3c29740 1 --2- 172.21.15.45:0/2420378895 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750cb0c00 0x555750cc8100 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.996+0000 7f89d3c29740 1 --2- 172.21.15.45:0/2420378895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555750caba90 0x555750cbc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.996+0000 7f89d3c29740 1 --2- 172.21.15.45:0/2420378895 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555750cab0a0 0x555750cc7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.996+0000 7f89d3c29740 1 -- 172.21.15.45:0/2420378895 >> 172.21.15.45:0/2420378895 conn(0x555750ca6610 msgr2=0x555750ca5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:49.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.997+0000 7f89d3c29740 1 -- 172.21.15.45:0/2420378895 shutdown_connections 2024-03-28T17:48:49.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.997+0000 7f89d3c29740 1 -- 172.21.15.45:0/2420378895 wait complete. 2024-03-28T17:48:49.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.997+0000 7f89d3c29740 1 -- start start 2024-03-28T17:48:49.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.997+0000 7f89d3c29740 1 Processor -- start 2024-03-28T17:48:49.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.997+0000 7f89d3c29740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555750cab0a0 0x555750cbc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.997+0000 7f89d3c29740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555750caba90 0x555750cbc540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.997+0000 7f89d3c29740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750cb0c00 0x555750cba610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.997+0000 7f89d3c29740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc1a9190 con 0x555750cab0a0 2024-03-28T17:48:49.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.997+0000 7f89d3c29740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc1aa490 con 0x555750cb0c00 2024-03-28T17:48:49.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.997+0000 7f89d3c29740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc1a97d0 con 0x555750caba90 2024-03-28T17:48:49.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.998+0000 7f89d3c29740 10 client.0 _ll_register_callbacks cb 0x7f89c4009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:48:49.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.998+0000 7f89ca800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750cb0c00 0x555750cba610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.998+0000 7f89c9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555750cab0a0 0x555750cbc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.998+0000 7f89c9400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555750caba90 0x555750cbc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.998+0000 7f89ca800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750cb0c00 0x555750cba610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:46544/0 (socket says 172.21.15.45:46544) 2024-03-28T17:48:49.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.998+0000 7f89ca800640 1 -- 172.21.15.45:0/1076525495 learned_addr learned my addr 172.21.15.45:0/1076525495 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:49.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.998+0000 7f89ca800640 1 -- 172.21.15.45:0/1076525495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555750caba90 msgr2=0x555750cbc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:49.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.998+0000 7f89ca800640 1 --2- 172.21.15.45:0/1076525495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555750caba90 0x555750cbc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.998+0000 7f89ca800640 1 -- 172.21.15.45:0/1076525495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555750cab0a0 msgr2=0x555750cbc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:49.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.998+0000 7f89ca800640 1 --2- 172.21.15.45:0/1076525495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555750cab0a0 0x555750cbc000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.998+0000 7f89ca800640 1 -- 172.21.15.45:0/1076525495 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555750cbac50 con 0x555750cb0c00 2024-03-28T17:48:49.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.998+0000 7f89c9400640 1 --2- 172.21.15.45:0/1076525495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555750caba90 0x555750cbc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:49.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.999+0000 7f89ca800640 1 --2- 172.21.15.45:0/1076525495 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750cb0c00 0x555750cba610 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f89c4009870 tx=0x7f89c4005930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dc0b6e4774c8e31 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:49.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.999+0000 7f89abe00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:48:49.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.999+0000 7f89abe00640 1 -- 172.21.15.45:0/1076525495 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c403a060 con 0x555750cb0c00 2024-03-28T17:48:49.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.999+0000 7f89d3c29740 1 -- 172.21.15.45:0/1076525495 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f89c4026030 con 0x555750cb0c00 2024-03-28T17:48:49.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.999+0000 7f89abe00640 1 -- 172.21.15.45:0/1076525495 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f89c40119f0 con 0x555750cb0c00 2024-03-28T17:48:49.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.999+0000 7f89abe00640 1 -- 172.21.15.45:0/1076525495 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89c4011bf0 con 0x555750cb0c00 2024-03-28T17:48:49.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.999+0000 7f89d3c29740 10 client.25349 fetch_fsmap learned FSMap version 23 2024-03-28T17:48:49.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.999+0000 7f89d3c29740 1 -- 172.21.15.45:0/1076525495 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555750d1d160 con 0x555750cb0c00 2024-03-28T17:48:49.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.999+0000 7f89abe00640 1 -- 172.21.15.45:0/1076525495 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f89c4026030 con 0x555750cb0c00 2024-03-28T17:48:49.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:48.999+0000 7f89abe00640 1 -- 172.21.15.45:0/1076525495 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f89c4034070 con 0x555750cb0c00 2024-03-28T17:48:49.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.000+0000 7f89d3c29740 10 client.25349 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:48:49.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.000+0000 7f89d3c29740 10 client.25349 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:48:49.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.000+0000 7f89d3c29740 20 client.25349 populate_metadata read hostname 'smithi045' 2024-03-28T17:48:49.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.000+0000 7f89d3c29740 1 --2- 172.21.15.45:0/1076525495 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555750cc0e70 0x555750d1fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.000+0000 7f89d3c29740 4 client.25349 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T17:48:49.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.000+0000 7f89d3c29740 1 -- 172.21.15.45:0/1076525495 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x555750c18740 con 0x555750cc0e70 2024-03-28T17:48:49.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.000+0000 7f89c9e00640 1 --2- 172.21.15.45:0/1076525495 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555750cc0e70 0x555750d1fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.000+0000 7f89c9e00640 1 --2- 172.21.15.45:0/1076525495 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555750cc0e70 0x555750d1fa30 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=136fbfe6cd81035d server_cookie=a84286ae2e28b214 in_seq=0 out_seq=0 2024-03-28T17:48:49.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.000+0000 7f89abe00640 10 client.25349 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:48:49.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.002+0000 7f89abe00640 1 -- 172.21.15.45:0/1076525495 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x555750c18740 con 0x555750cc0e70 2024-03-28T17:48:49.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.002+0000 7f89abe00640 10 client.25349 handle_command_reply: tid=0 2024-03-28T17:48:49.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.002+0000 7f89d3c29740 1 client.25349 shutdown 2024-03-28T17:48:49.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.002+0000 7f89d3c29740 1 -- 172.21.15.45:0/1076525495 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750cb0c00 msgr2=0x555750cba610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:49.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.002+0000 7f89d3c29740 1 --2- 172.21.15.45:0/1076525495 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750cb0c00 0x555750cba610 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f89c4009870 tx=0x7f89c4005930 comp rx=0 tx=0).stop 2024-03-28T17:48:49.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.002+0000 7f89d3c29740 1 -- 172.21.15.45:0/1076525495 shutdown_connections 2024-03-28T17:48:49.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.002+0000 7f89d3c29740 1 --2- 172.21.15.45:0/1076525495 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555750cc0e70 0x555750d1fa30 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.002+0000 7f89d3c29740 1 --2- 172.21.15.45:0/1076525495 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555750cb0c00 0x555750cba610 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.002+0000 7f89d3c29740 1 --2- 172.21.15.45:0/1076525495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555750caba90 0x555750cbc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.002+0000 7f89d3c29740 1 --2- 172.21.15.45:0/1076525495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555750cab0a0 0x555750cbc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.002+0000 7f89abe00640 0 client.25349 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:48:49.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.002+0000 7f89d3c29740 1 -- 172.21.15.45:0/1076525495 >> 172.21.15.45:0/1076525495 conn(0x555750ca6610 msgr2=0x555750cac320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:49.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.002+0000 7f89d3c29740 1 -- 172.21.15.45:0/1076525495 shutdown_connections 2024-03-28T17:48:49.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.002+0000 7f89d3c29740 1 -- 172.21.15.45:0/1076525495 wait complete. 2024-03-28T17:48:49.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.002+0000 7f89d3c29740 20 client.25349 trim_cache size 0 max 16384 2024-03-28T17:48:49.006 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:48:49.006 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-28T17:48:49.006 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "8f73244b-7b63-4ac8-9fb9-84711717049d", 2024-03-28T17:48:49.006 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-28T17:48:49.006 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:48:49.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.003+0000 7f89d1000640 1 -- 172.21.15.45:0/868166487 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f89b00a2890 msgr2=0x7f89b00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:49.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.003+0000 7f89d1000640 1 --2- 172.21.15.45:0/868166487 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f89b00a2890 0x7f89b00a4d50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x555750cac2f0 tx=0x7f89c4009600 comp rx=0 tx=0).stop 2024-03-28T17:48:49.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.003+0000 7f89d1000640 1 -- 172.21.15.45:0/868166487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89cc1028a0 msgr2=0x7f89cc19a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:49.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.003+0000 7f89d1000640 1 --2- 172.21.15.45:0/868166487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89cc1028a0 0x7f89cc19a160 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f89b8005ab0 tx=0x7f89b802c6c0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.003+0000 7f89d1000640 1 -- 172.21.15.45:0/868166487 shutdown_connections 2024-03-28T17:48:49.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.003+0000 7f89d1000640 1 --2- 172.21.15.45:0/868166487 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f89b00a2890 0x7f89b00a4d50 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.003+0000 7f89d1000640 1 --2- 172.21.15.45:0/868166487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89cc103260 0x7f89cc1a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.003+0000 7f89d1000640 1 --2- 172.21.15.45:0/868166487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89cc1028a0 0x7f89cc19a160 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.003+0000 7f89d1000640 1 --2- 172.21.15.45:0/868166487 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89cc1016b0 0x7f89cc199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.003+0000 7f89d1000640 1 -- 172.21.15.45:0/868166487 >> 172.21.15.45:0/868166487 conn(0x7f89cc0fcc40 msgr2=0x7f89cc0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:49.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.003+0000 7f89d1000640 1 -- 172.21.15.45:0/868166487 shutdown_connections 2024-03-28T17:48:49.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.003+0000 7f89d1000640 1 -- 172.21.15.45:0/868166487 wait complete. 2024-03-28T17:48:49.030 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:48:49.030 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:48:49.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:48 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1603345402' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T17:48:49.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:48 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3732346279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:48:49.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:48 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:48:49.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:48 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:48:49.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:48 smithi071 ceph-mon[36597]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:49.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:48 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1603345402' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T17:48:49.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:48 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3732346279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:48:49.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:48 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:48:49.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:48 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:48:49.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:48 smithi151 ceph-mon[36481]: from='mgr.14152 172.21.15.45:0/3517204993' entity='mgr.x' 2024-03-28T17:48:49.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.233+0000 7fe510a00640 1 -- 172.21.15.45:0/3043207053 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fe50c02d070 con 0x7fe50c1038f0 2024-03-28T17:48:49.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.242+0000 7fbd29c00640 1 -- 172.21.15.45:0/3448689369 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd240ffe50 msgr2=0x7fbd24100270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:49.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.242+0000 7fbd29c00640 1 --2- 172.21.15.45:0/3448689369 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd240ffe50 0x7fbd24100270 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbd14009ca0 tx=0x7fbd1402c100 comp rx=0 tx=0).stop 2024-03-28T17:48:49.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.242+0000 7fbd29c00640 1 -- 172.21.15.45:0/3448689369 shutdown_connections 2024-03-28T17:48:49.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.242+0000 7fbd29c00640 1 --2- 172.21.15.45:0/3448689369 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd24101150 0x7fbd241015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.242+0000 7fbd29c00640 1 --2- 172.21.15.45:0/3448689369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd241007b0 0x7fbd24100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.242+0000 7fbd29c00640 1 --2- 172.21.15.45:0/3448689369 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd240ffe50 0x7fbd24100270 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.242+0000 7fbd29c00640 1 -- 172.21.15.45:0/3448689369 >> 172.21.15.45:0/3448689369 conn(0x7fbd240fba10 msgr2=0x7fbd240fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:49.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.243+0000 7fbd29c00640 1 -- 172.21.15.45:0/3448689369 shutdown_connections 2024-03-28T17:48:49.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.243+0000 7fbd29c00640 1 -- 172.21.15.45:0/3448689369 wait complete. 2024-03-28T17:48:49.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.243+0000 7fbd29c00640 1 Processor -- start 2024-03-28T17:48:49.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.244+0000 7fbd29c00640 1 -- start start 2024-03-28T17:48:49.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.244+0000 7fbd29c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd240ffe50 0x7fbd24199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.244+0000 7fbd29c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd241007b0 0x7fbd24199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.244+0000 7fbd29c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd24101150 0x7fbd241a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.244+0000 7fbd29c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2410f3e0 con 0x7fbd24101150 2024-03-28T17:48:49.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.244+0000 7fbd29c00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2410f260 con 0x7fbd240ffe50 2024-03-28T17:48:49.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.244+0000 7fbd29c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2410f560 con 0x7fbd241007b0 2024-03-28T17:48:49.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.244+0000 7fbd23e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd240ffe50 0x7fbd24199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.244+0000 7fbd23e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd240ffe50 0x7fbd24199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:46558/0 (socket says 172.21.15.45:46558) 2024-03-28T17:48:49.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.244+0000 7fbd1b400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd241007b0 0x7fbd24199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.244+0000 7fbd23e00640 1 -- 172.21.15.45:0/3847022714 learned_addr learned my addr 172.21.15.45:0/3847022714 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:49.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.244+0000 7fbd29200640 1 --2- 172.21.15.45:0/3847022714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd24101150 0x7fbd241a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.245+0000 7fbd23e00640 1 -- 172.21.15.45:0/3847022714 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd241007b0 msgr2=0x7fbd24199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:49.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.245+0000 7fbd23e00640 1 --2- 172.21.15.45:0/3847022714 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd241007b0 0x7fbd24199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.245+0000 7fbd23e00640 1 -- 172.21.15.45:0/3847022714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd24101150 msgr2=0x7fbd241a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.245+0000 7fbd23e00640 1 --2- 172.21.15.45:0/3847022714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd24101150 0x7fbd241a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.245+0000 7fbd23e00640 1 -- 172.21.15.45:0/3847022714 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd241a1670 con 0x7fbd240ffe50 2024-03-28T17:48:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.245+0000 7fbd29200640 1 --2- 172.21.15.45:0/3847022714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd24101150 0x7fbd241a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:48:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.245+0000 7fbd23e00640 1 --2- 172.21.15.45:0/3847022714 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd240ffe50 0x7fbd24199a50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbd140325f0 tx=0x7fbd14035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.245+0000 7fbd21600640 1 -- 172.21.15.45:0/3847022714 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd14039040 con 0x7fbd240ffe50 2024-03-28T17:48:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.245+0000 7fbd21600640 1 -- 172.21.15.45:0/3847022714 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd14044070 con 0x7fbd240ffe50 2024-03-28T17:48:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.245+0000 7fbd21600640 1 -- 172.21.15.45:0/3847022714 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd140044d0 con 0x7fbd240ffe50 2024-03-28T17:48:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.245+0000 7fbd29c00640 1 -- 172.21.15.45:0/3847022714 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd241a1900 con 0x7fbd240ffe50 2024-03-28T17:48:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.246+0000 7fbd29c00640 1 -- 172.21.15.45:0/3847022714 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd241a1d70 con 0x7fbd240ffe50 2024-03-28T17:48:49.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.249+0000 7fbd21600640 1 -- 172.21.15.45:0/3847022714 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fbd14004670 con 0x7fbd240ffe50 2024-03-28T17:48:49.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.249+0000 7fbd2de67740 1 Processor -- start 2024-03-28T17:48:49.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.249+0000 7fbd2de67740 1 -- start start 2024-03-28T17:48:49.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.249+0000 7fbd2de67740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56035000d050 0x56035000d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.249+0000 7fbd2de67740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56035000da40 0x560350012540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.249+0000 7fbd2de67740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560350012bb0 0x560350015040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.249+0000 7fbd2de67740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56034ff72250 con 0x56035000d050 2024-03-28T17:48:49.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.249+0000 7fbd2de67740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x56034ff1aa30 con 0x56035000da40 2024-03-28T17:48:49.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.249+0000 7fbd2de67740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x56034ff6e2f0 con 0x560350012bb0 2024-03-28T17:48:49.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.249+0000 7fbd23e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56035000da40 0x560350012540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.250+0000 7fbd1b400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560350012bb0 0x560350015040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.250+0000 7fbd29200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56035000d050 0x56035000d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.250+0000 7fbd23e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56035000da40 0x560350012540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:46572/0 (socket says 172.21.15.45:46572) 2024-03-28T17:48:49.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.250+0000 7fbd23e00640 1 -- 172.21.15.45:0/724777449 learned_addr learned my addr 172.21.15.45:0/724777449 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:49.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.250+0000 7fbd23e00640 1 -- 172.21.15.45:0/724777449 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560350012bb0 msgr2=0x560350015040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:49.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.250+0000 7fbd23e00640 1 --2- 172.21.15.45:0/724777449 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560350012bb0 0x560350015040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.250+0000 7fbd23e00640 1 -- 172.21.15.45:0/724777449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56035000d050 msgr2=0x56035000d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:49.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.250+0000 7fbd23e00640 1 --2- 172.21.15.45:0/724777449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56035000d050 0x56035000d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.250+0000 7fbd23e00640 1 -- 172.21.15.45:0/724777449 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56034ff05690 con 0x56035000da40 2024-03-28T17:48:49.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.250+0000 7fbd23e00640 1 --2- 172.21.15.45:0/724777449 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56035000da40 0x560350012540 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbd140eea60 tx=0x7fbd140f9220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc5f6eca494a7473 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:49.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.250+0000 7fbd21600640 1 --2- 172.21.15.45:0/3847022714 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fbd080a2910 0x7fbd080a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.251+0000 7fbd19600640 1 -- 172.21.15.45:0/724777449 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd140b9c50 con 0x56035000da40 2024-03-28T17:48:49.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.251+0000 7fbd29200640 1 --2- 172.21.15.45:0/3847022714 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fbd080a2910 0x7fbd080a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.251+0000 7fbd19600640 1 -- 172.21.15.45:0/724777449 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd140fe030 con 0x56035000da40 2024-03-28T17:48:49.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.251+0000 7fbd19600640 1 -- 172.21.15.45:0/724777449 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd140bce00 con 0x56035000da40 2024-03-28T17:48:49.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.251+0000 7fbd21600640 1 -- 172.21.15.45:0/3847022714 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fbd140f1050 con 0x7fbd240ffe50 2024-03-28T17:48:49.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.251+0000 7fbd2de67740 1 -- 172.21.15.45:0/724777449 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56035000da40 msgr2=0x560350012540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:49.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.251+0000 7fbd2de67740 1 --2- 172.21.15.45:0/724777449 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56035000da40 0x560350012540 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbd140eea60 tx=0x7fbd140f9220 comp rx=0 tx=0).stop 2024-03-28T17:48:49.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.251+0000 7fbd29200640 1 --2- 172.21.15.45:0/3847022714 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fbd080a2910 0x7fbd080a4dd0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x56035000e2a0 tx=0x7fbd1c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:49.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.251+0000 7fbd2de67740 1 -- 172.21.15.45:0/724777449 shutdown_connections 2024-03-28T17:48:49.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.251+0000 7fbd2de67740 1 --2- 172.21.15.45:0/724777449 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560350012bb0 0x560350015040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.252+0000 7fbd2de67740 1 --2- 172.21.15.45:0/724777449 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56035000da40 0x560350012540 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.252+0000 7fbd2de67740 1 --2- 172.21.15.45:0/724777449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56035000d050 0x56035000d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.252+0000 7fbd2de67740 1 -- 172.21.15.45:0/724777449 >> 172.21.15.45:0/724777449 conn(0x56034ffd5200 msgr2=0x5603500089c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:49.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.252+0000 7fbd2de67740 1 -- 172.21.15.45:0/724777449 shutdown_connections 2024-03-28T17:48:49.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.252+0000 7fbd2de67740 1 -- 172.21.15.45:0/724777449 wait complete. 2024-03-28T17:48:49.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.252+0000 7fbd2de67740 1 -- start start 2024-03-28T17:48:49.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.252+0000 7fbd2de67740 1 Processor -- start 2024-03-28T17:48:49.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.253+0000 7fbd2de67740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56035000d050 0x560350016cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.253+0000 7fbd2de67740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56035000da40 0x560350017200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.253+0000 7fbd2de67740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560350012bb0 0x56035001e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.253+0000 7fbd2de67740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x560350015920 con 0x56035000d050 2024-03-28T17:48:49.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.253+0000 7fbd2de67740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x56034ff2de80 con 0x560350012bb0 2024-03-28T17:48:49.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.253+0000 7fbd2de67740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x560350015aa0 con 0x56035000da40 2024-03-28T17:48:49.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.253+0000 7fbd2de67740 10 client.0 _ll_register_callbacks cb 0x56034ff40130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:48:49.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.253+0000 7fbd29200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560350012bb0 0x56035001e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.253+0000 7fbd1b400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56035000da40 0x560350017200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.253+0000 7fbd23e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56035000d050 0x560350016cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.253+0000 7fbd29200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560350012bb0 0x56035001e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:46574/0 (socket says 172.21.15.45:46574) 2024-03-28T17:48:49.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.253+0000 7fbd1b400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56035000da40 0x560350017200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:60924/0 (socket says 172.21.15.45:60924) 2024-03-28T17:48:49.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.253+0000 7fbd29200640 1 -- 172.21.15.45:0/1242390563 learned_addr learned my addr 172.21.15.45:0/1242390563 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:49.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.253+0000 7fbd29200640 1 -- 172.21.15.45:0/1242390563 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56035000da40 msgr2=0x560350017200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:49.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.253+0000 7fbd29200640 1 --2- 172.21.15.45:0/1242390563 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56035000da40 0x560350017200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.253+0000 7fbd29200640 1 -- 172.21.15.45:0/1242390563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56035000d050 msgr2=0x560350016cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:49.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.253+0000 7fbd29200640 1 --2- 172.21.15.45:0/1242390563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56035000d050 0x560350016cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.253+0000 7fbd29200640 1 -- 172.21.15.45:0/1242390563 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56034fd507d0 con 0x560350012bb0 2024-03-28T17:48:49.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.253+0000 7fbd23e00640 1 --2- 172.21.15.45:0/1242390563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56035000d050 0x560350016cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:49.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.254+0000 7fbd29200640 1 --2- 172.21.15.45:0/1242390563 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560350012bb0 0x56035001e240 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c0209e0 tx=0x7fbd1c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e43a95e4f3a73f70 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:49.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.254+0000 7fbd12000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:48:49.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.254+0000 7fbd12000640 1 -- 172.21.15.45:0/1242390563 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c02d060 con 0x560350012bb0 2024-03-28T17:48:49.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.254+0000 7fbd12000640 1 -- 172.21.15.45:0/1242390563 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fbd1c018030 con 0x560350012bb0 2024-03-28T17:48:49.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.254+0000 7fbd2de67740 1 -- 172.21.15.45:0/1242390563 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbd140fe030 con 0x560350012bb0 2024-03-28T17:48:49.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.254+0000 7fbd12000640 1 -- 172.21.15.45:0/1242390563 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd1c0119c0 con 0x560350012bb0 2024-03-28T17:48:49.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.254+0000 7fbd2de67740 10 client.25361 fetch_fsmap learned FSMap version 23 2024-03-28T17:48:49.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.254+0000 7fbd2de67740 1 -- 172.21.15.45:0/1242390563 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56034fc863f0 con 0x560350012bb0 2024-03-28T17:48:49.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.255+0000 7fbd12000640 1 -- 172.21.15.45:0/1242390563 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd140fe030 con 0x560350012bb0 2024-03-28T17:48:49.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.255+0000 7fbd12000640 1 -- 172.21.15.45:0/1242390563 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fbd1c01d3f0 con 0x560350012bb0 2024-03-28T17:48:49.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.255+0000 7fbd2de67740 10 client.25361 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:48:49.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.255+0000 7fbd2de67740 10 client.25361 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:48:49.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.255+0000 7fbd2de67740 20 client.25361 populate_metadata read hostname 'smithi045' 2024-03-28T17:48:49.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.255+0000 7fbd2de67740 1 --2- 172.21.15.45:0/1242390563 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x560350022b00 0x560350024fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.255+0000 7fbd2de67740 4 client.25361 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:48:49.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.255+0000 7fbd2de67740 1 -- 172.21.15.45:0/1242390563 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56034ff7f790 con 0x560350022b00 2024-03-28T17:48:49.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.255+0000 7fbd23e00640 1 --2- 172.21.15.45:0/1242390563 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x560350022b00 0x560350024fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.256+0000 7fbd23e00640 1 --2- 172.21.15.45:0/1242390563 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x560350022b00 0x560350024fa0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4700610137c0dcb7 server_cookie=1b451c3abf6cb7ae in_seq=0 out_seq=0 2024-03-28T17:48:49.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.256+0000 7fbd12000640 10 client.25361 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:48:49.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.256+0000 7fbd12000640 1 -- 172.21.15.45:0/1242390563 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56034ff7f790 con 0x560350022b00 2024-03-28T17:48:49.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.256+0000 7fbd12000640 10 client.25361 handle_command_reply: tid=0 2024-03-28T17:48:49.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.256+0000 7fbd2de67740 1 client.25361 shutdown 2024-03-28T17:48:49.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.257+0000 7fbd2de67740 1 -- 172.21.15.45:0/1242390563 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560350012bb0 msgr2=0x56035001e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:49.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.257+0000 7fbd2de67740 1 --2- 172.21.15.45:0/1242390563 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560350012bb0 0x56035001e240 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c0209e0 tx=0x7fbd1c004730 comp rx=0 tx=0).stop 2024-03-28T17:48:49.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.257+0000 7fbd2de67740 1 -- 172.21.15.45:0/1242390563 shutdown_connections 2024-03-28T17:48:49.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.257+0000 7fbd2de67740 1 --2- 172.21.15.45:0/1242390563 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x560350022b00 0x560350024fa0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.257+0000 7fbd2de67740 1 --2- 172.21.15.45:0/1242390563 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560350012bb0 0x56035001e240 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.257+0000 7fbd2de67740 1 --2- 172.21.15.45:0/1242390563 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56035000da40 0x560350017200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.257+0000 7fbd2de67740 1 --2- 172.21.15.45:0/1242390563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56035000d050 0x560350016cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.257+0000 7fbd12000640 0 client.25361 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:48:49.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.257+0000 7fbd2de67740 1 -- 172.21.15.45:0/1242390563 >> 172.21.15.45:0/1242390563 conn(0x56034ffd5200 msgr2=0x560350007bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:49.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.257+0000 7fbd2de67740 1 -- 172.21.15.45:0/1242390563 shutdown_connections 2024-03-28T17:48:49.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.257+0000 7fbd2de67740 1 -- 172.21.15.45:0/1242390563 wait complete. 2024-03-28T17:48:49.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.257+0000 7fbd2de67740 20 client.25361 trim_cache size 0 max 16384 2024-03-28T17:48:49.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.274+0000 7fbd2de67740 1 Processor -- start 2024-03-28T17:48:49.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.275+0000 7fbd2de67740 1 -- start start 2024-03-28T17:48:49.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.275+0000 7fbd2de67740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56035000d050 0x560350007a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.275+0000 7fbd2de67740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56035000da40 0x560350007f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.275+0000 7fbd2de67740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560350012bb0 0x56035001e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.275+0000 7fbd2de67740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd241acf10 con 0x56035000d050 2024-03-28T17:48:49.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.275+0000 7fbd2de67740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd241aaa10 con 0x56035000da40 2024-03-28T17:48:49.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.275+0000 7fbd2de67740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd241a8ab0 con 0x560350012bb0 2024-03-28T17:48:49.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.275+0000 7fbd1b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56035000da40 0x560350007f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.275+0000 7fbd1b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56035000da40 0x560350007f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:46578/0 (socket says 172.21.15.45:46578) 2024-03-28T17:48:49.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.275+0000 7fbd1b400640 1 -- 172.21.15.45:0/2820942885 learned_addr learned my addr 172.21.15.45:0/2820942885 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:49.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.275+0000 7fbd29200640 1 --2- 172.21.15.45:0/2820942885 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560350012bb0 0x56035001e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.275+0000 7fbd23e00640 1 --2- 172.21.15.45:0/2820942885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56035000d050 0x560350007a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.276+0000 7fbd1b400640 1 -- 172.21.15.45:0/2820942885 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560350012bb0 msgr2=0x56035001e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:49.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.276+0000 7fbd1b400640 1 --2- 172.21.15.45:0/2820942885 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560350012bb0 0x56035001e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.276+0000 7fbd1b400640 1 -- 172.21.15.45:0/2820942885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56035000d050 msgr2=0x560350007a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:49.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.276+0000 7fbd1b400640 1 --2- 172.21.15.45:0/2820942885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56035000d050 0x560350007a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.276+0000 7fbd1b400640 1 -- 172.21.15.45:0/2820942885 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56034fc460f0 con 0x56035000da40 2024-03-28T17:48:49.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.276+0000 7fbd29200640 1 --2- 172.21.15.45:0/2820942885 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560350012bb0 0x56035001e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:49.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.276+0000 7fbd1b400640 1 --2- 172.21.15.45:0/2820942885 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56035000da40 0x560350007f80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c009e90 tx=0x7fbd0c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8b21fdae9ce0721 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:49.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.276+0000 7fbd18c00640 1 -- 172.21.15.45:0/2820942885 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c022070 con 0x56035000da40 2024-03-28T17:48:49.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.276+0000 7fbd18c00640 1 -- 172.21.15.45:0/2820942885 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd0c0035f0 con 0x56035000da40 2024-03-28T17:48:49.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.276+0000 7fbd18c00640 1 -- 172.21.15.45:0/2820942885 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c01d6c0 con 0x56035000da40 2024-03-28T17:48:49.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.277+0000 7fbd2de67740 1 -- 172.21.15.45:0/2820942885 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56035000da40 msgr2=0x560350007f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:49.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.277+0000 7fbd2de67740 1 --2- 172.21.15.45:0/2820942885 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56035000da40 0x560350007f80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c009e90 tx=0x7fbd0c005f90 comp rx=0 tx=0).stop 2024-03-28T17:48:49.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.277+0000 7fbd2de67740 1 -- 172.21.15.45:0/2820942885 shutdown_connections 2024-03-28T17:48:49.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.277+0000 7fbd2de67740 1 --2- 172.21.15.45:0/2820942885 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560350012bb0 0x56035001e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.277+0000 7fbd2de67740 1 --2- 172.21.15.45:0/2820942885 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56035000da40 0x560350007f80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.277+0000 7fbd2de67740 1 --2- 172.21.15.45:0/2820942885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56035000d050 0x560350007a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.277+0000 7fbd2de67740 1 -- 172.21.15.45:0/2820942885 >> 172.21.15.45:0/2820942885 conn(0x56034ffd5200 msgr2=0x560350007330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:49.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.277+0000 7fbd2de67740 1 -- 172.21.15.45:0/2820942885 shutdown_connections 2024-03-28T17:48:49.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.277+0000 7fbd2de67740 1 -- 172.21.15.45:0/2820942885 wait complete. 2024-03-28T17:48:49.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.278+0000 7fbd2de67740 1 -- start start 2024-03-28T17:48:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.278+0000 7fbd2de67740 1 Processor -- start 2024-03-28T17:48:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.278+0000 7fbd2de67740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56035000d050 0x56035001c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.278+0000 7fbd2de67740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56035000da40 0x56035001ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.278+0000 7fbd2de67740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560350012bb0 0x56035001d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.278+0000 7fbd2de67740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd241a90b0 con 0x56035000d050 2024-03-28T17:48:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.278+0000 7fbd2de67740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd241aa3b0 con 0x56035000da40 2024-03-28T17:48:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.278+0000 7fbd2de67740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd241a96f0 con 0x560350012bb0 2024-03-28T17:48:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.278+0000 7fbd2de67740 10 client.0 _ll_register_callbacks cb 0x7fbd1c0096c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:48:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.278+0000 7fbd1b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56035000da40 0x56035001ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.278+0000 7fbd23e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56035000d050 0x56035001c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.279+0000 7fbd1b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56035000da40 0x56035001ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:46584/0 (socket says 172.21.15.45:46584) 2024-03-28T17:48:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.279+0000 7fbd1b400640 1 -- 172.21.15.45:0/1056111577 learned_addr learned my addr 172.21.15.45:0/1056111577 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.279+0000 7fbd29200640 1 --2- 172.21.15.45:0/1056111577 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560350012bb0 0x56035001d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.279+0000 7fbd1b400640 1 -- 172.21.15.45:0/1056111577 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560350012bb0 msgr2=0x56035001d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:49.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.279+0000 7fbd1b400640 1 --2- 172.21.15.45:0/1056111577 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560350012bb0 0x56035001d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.279+0000 7fbd1b400640 1 -- 172.21.15.45:0/1056111577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56035000d050 msgr2=0x56035001c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:48:49.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.279+0000 7fbd1b400640 1 --2- 172.21.15.45:0/1056111577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56035000d050 0x56035001c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.279+0000 7fbd1b400640 1 -- 172.21.15.45:0/1056111577 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56035001dea0 con 0x56035000da40 2024-03-28T17:48:49.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.279+0000 7fbd29200640 1 --2- 172.21.15.45:0/1056111577 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560350012bb0 0x56035001d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:49.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.279+0000 7fbd23e00640 1 --2- 172.21.15.45:0/1056111577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56035000d050 0x56035001c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:49.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.279+0000 7fbd1b400640 1 --2- 172.21.15.45:0/1056111577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56035000da40 0x56035001ca70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c020040 tx=0x7fbd0c004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70a5137f08b35b80 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:49.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.279+0000 7fbd12000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:48:49.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.279+0000 7fbd12000640 1 -- 172.21.15.45:0/1056111577 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c022070 con 0x56035000da40 2024-03-28T17:48:49.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.280+0000 7fbd12000640 1 -- 172.21.15.45:0/1056111577 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fbd0c005030 con 0x56035000da40 2024-03-28T17:48:49.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.280+0000 7fbd2de67740 1 -- 172.21.15.45:0/1056111577 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbd0c0035f0 con 0x56035000da40 2024-03-28T17:48:49.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.280+0000 7fbd12000640 1 -- 172.21.15.45:0/1056111577 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd0c005230 con 0x56035000da40 2024-03-28T17:48:49.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.280+0000 7fbd2de67740 10 client.25367 fetch_fsmap learned FSMap version 23 2024-03-28T17:48:49.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.280+0000 7fbd2de67740 1 -- 172.21.15.45:0/1056111577 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56035007ee30 con 0x56035000da40 2024-03-28T17:48:49.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.280+0000 7fbd12000640 1 -- 172.21.15.45:0/1056111577 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd0c0035f0 con 0x56035000da40 2024-03-28T17:48:49.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.280+0000 7fbd12000640 1 -- 172.21.15.45:0/1056111577 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fbd0c030030 con 0x56035000da40 2024-03-28T17:48:49.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.280+0000 7fbd2de67740 10 client.25367 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:48:49.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.280+0000 7fbd2de67740 10 client.25367 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:48:49.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.280+0000 7fbd2de67740 20 client.25367 populate_metadata read hostname 'smithi045' 2024-03-28T17:48:49.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.280+0000 7fbd2de67740 1 --2- 172.21.15.45:0/1056111577 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x560350022b00 0x5603500817b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.280+0000 7fbd2de67740 4 client.25367 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:48:49.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.280+0000 7fbd2de67740 1 -- 172.21.15.45:0/1056111577 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56034ff79ae0 con 0x560350022b00 2024-03-28T17:48:49.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.281+0000 7fbd23e00640 1 --2- 172.21.15.45:0/1056111577 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x560350022b00 0x5603500817b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.281+0000 7fbd23e00640 1 --2- 172.21.15.45:0/1056111577 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x560350022b00 0x5603500817b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6faba95d9f4b8851 server_cookie=dfb05f36e9b820d6 in_seq=0 out_seq=0 2024-03-28T17:48:49.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.281+0000 7fbd12000640 10 client.25367 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:48:49.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.281+0000 7fbd12000640 1 -- 172.21.15.45:0/1056111577 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56034ff79ae0 con 0x560350022b00 2024-03-28T17:48:49.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.281+0000 7fbd12000640 10 client.25367 handle_command_reply: tid=0 2024-03-28T17:48:49.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.281+0000 7fbd2de67740 1 client.25367 shutdown 2024-03-28T17:48:49.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.281+0000 7fbd2de67740 1 -- 172.21.15.45:0/1056111577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56035000da40 msgr2=0x56035001ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:49.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.281+0000 7fbd2de67740 1 --2- 172.21.15.45:0/1056111577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56035000da40 0x56035001ca70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c020040 tx=0x7fbd0c004610 comp rx=0 tx=0).stop 2024-03-28T17:48:49.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.281+0000 7fbd2de67740 1 -- 172.21.15.45:0/1056111577 shutdown_connections 2024-03-28T17:48:49.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.281+0000 7fbd2de67740 1 --2- 172.21.15.45:0/1056111577 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x560350022b00 0x5603500817b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.281+0000 7fbd2de67740 1 --2- 172.21.15.45:0/1056111577 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560350012bb0 0x56035001d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.281+0000 7fbd2de67740 1 --2- 172.21.15.45:0/1056111577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56035000da40 0x56035001ca70 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.281+0000 7fbd12000640 0 client.25367 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:48:49.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.281+0000 7fbd2de67740 1 --2- 172.21.15.45:0/1056111577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56035000d050 0x56035001c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.281+0000 7fbd2de67740 1 -- 172.21.15.45:0/1056111577 >> 172.21.15.45:0/1056111577 conn(0x56034ffd5200 msgr2=0x560350007bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:49.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.282+0000 7fbd2de67740 1 -- 172.21.15.45:0/1056111577 shutdown_connections 2024-03-28T17:48:49.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.282+0000 7fbd2de67740 1 -- 172.21.15.45:0/1056111577 wait complete. 2024-03-28T17:48:49.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.282+0000 7fbd2de67740 20 client.25367 trim_cache size 0 max 16384 2024-03-28T17:48:49.287 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:48:49.287 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T17:48:49.287 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T17:48:49.287 INFO:teuthology.orchestra.run.smithi045.stdout: "8f73244b-7b63-4ac8-9fb9-84711717049d": { 2024-03-28T17:48:49.287 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T17:48:49.287 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "8f73244b-7b63-4ac8-9fb9-84711717049d", 2024-03-28T17:48:49.287 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T17:48:49.287 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:48:49.287 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:48:49.287 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:48:49.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.282+0000 7fbd29c00640 1 -- 172.21.15.45:0/3847022714 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fbd080a2910 msgr2=0x7fbd080a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:49.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.282+0000 7fbd29c00640 1 --2- 172.21.15.45:0/3847022714 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fbd080a2910 0x7fbd080a4dd0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x56035000e2a0 tx=0x7fbd1c005f90 comp rx=0 tx=0).stop 2024-03-28T17:48:49.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.282+0000 7fbd29c00640 1 -- 172.21.15.45:0/3847022714 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd240ffe50 msgr2=0x7fbd24199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:49.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.282+0000 7fbd29c00640 1 --2- 172.21.15.45:0/3847022714 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd240ffe50 0x7fbd24199a50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbd140325f0 tx=0x7fbd14035d10 comp rx=0 tx=0).stop 2024-03-28T17:48:49.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.282+0000 7fbd29c00640 1 -- 172.21.15.45:0/3847022714 shutdown_connections 2024-03-28T17:48:49.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.282+0000 7fbd29c00640 1 --2- 172.21.15.45:0/3847022714 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fbd080a2910 0x7fbd080a4dd0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.282+0000 7fbd29c00640 1 --2- 172.21.15.45:0/3847022714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd24101150 0x7fbd241a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.282+0000 7fbd29c00640 1 --2- 172.21.15.45:0/3847022714 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd241007b0 0x7fbd24199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.282+0000 7fbd29c00640 1 --2- 172.21.15.45:0/3847022714 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd240ffe50 0x7fbd24199a50 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.282+0000 7fbd29c00640 1 -- 172.21.15.45:0/3847022714 >> 172.21.15.45:0/3847022714 conn(0x7fbd240fba10 msgr2=0x7fbd240fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:49.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.283+0000 7fbd29c00640 1 -- 172.21.15.45:0/3847022714 shutdown_connections 2024-03-28T17:48:49.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.283+0000 7fbd29c00640 1 -- 172.21.15.45:0/3847022714 wait complete. 2024-03-28T17:48:49.301 INFO:tasks.cephfs.filesystem:scrub status for tag:8f73244b-7b63-4ac8-9fb9-84711717049d - {'path': '/', 'tag': '8f73244b-7b63-4ac8-9fb9-84711717049d', 'options': 'recursive,force'} 2024-03-28T17:48:49.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.612+0000 7fe502a00640 1 -- 172.21.15.45:0/3043207053 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mgrmap(e 20) v1 ==== 137358+0+0 (secure 0 0 0) 0x7fe4fc097dd0 con 0x7fe50c1038f0 2024-03-28T17:48:49.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.614+0000 7fe502a00640 1 -- 172.21.15.45:0/3043207053 <== mon.2 v2:172.21.15.71:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v20) v1 ==== 92+0+0 (secure 0 0 0) 0x7fe4fc077880 con 0x7fe50c1038f0 2024-03-28T17:48:49.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.618+0000 7fe510a00640 1 -- 172.21.15.45:0/3043207053 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe4f00a29c0 msgr2=0x7fe4f00a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:49.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.618+0000 7fe510a00640 1 --2- 172.21.15.45:0/3043207053 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe4f00a29c0 0x7fe4f00a4e80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe50c19b050 tx=0x7fe4f8005f90 comp rx=0 tx=0).stop 2024-03-28T17:48:49.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.618+0000 7fe510a00640 1 -- 172.21.15.45:0/3043207053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe50c1038f0 msgr2=0x7fe50c199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:49.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.618+0000 7fe510a00640 1 --2- 172.21.15.45:0/3043207053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe50c1038f0 0x7fe50c199b40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc00e9c0 tx=0x7fe4fc00ee80 comp rx=0 tx=0).stop 2024-03-28T17:48:49.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.618+0000 7fe510a00640 1 -- 172.21.15.45:0/3043207053 shutdown_connections 2024-03-28T17:48:49.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.618+0000 7fe510a00640 1 --2- 172.21.15.45:0/3043207053 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7fe4f00a29c0 0x7fe4f00a4e80 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.618+0000 7fe510a00640 1 --2- 172.21.15.45:0/3043207053 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe50c1054a0 0x7fe50c1a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.618+0000 7fe510a00640 1 --2- 172.21.15.45:0/3043207053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe50c104ae0 0x7fe50c19a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.618+0000 7fe510a00640 1 --2- 172.21.15.45:0/3043207053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe50c1038f0 0x7fe50c199b40 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.618+0000 7fe510a00640 1 -- 172.21.15.45:0/3043207053 >> 172.21.15.45:0/3043207053 conn(0x7fe50c0fee60 msgr2=0x7fe50c107d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:49.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.618+0000 7fe510a00640 1 -- 172.21.15.45:0/3043207053 shutdown_connections 2024-03-28T17:48:49.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.618+0000 7fe510a00640 1 -- 172.21.15.45:0/3043207053 wait complete. 2024-03-28T17:48:49.639 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-28T17:48:49.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.832+0000 7ff2a3c00640 1 -- 172.21.15.45:0/2646290451 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff29c102140 msgr2=0x7ff29c104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:49.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.832+0000 7ff2a3c00640 1 --2- 172.21.15.45:0/2646290451 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff29c102140 0x7ff29c104530 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff290009860 tx=0x7ff29002bb70 comp rx=0 tx=0).stop 2024-03-28T17:48:49.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.832+0000 7ff2a3c00640 1 -- 172.21.15.45:0/2646290451 shutdown_connections 2024-03-28T17:48:49.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.832+0000 7ff2a3c00640 1 --2- 172.21.15.45:0/2646290451 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff29c1073a0 0x7ff29c1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.832+0000 7ff2a3c00640 1 --2- 172.21.15.45:0/2646290451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff29c104a70 0x7ff29c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.832+0000 7ff2a3c00640 1 --2- 172.21.15.45:0/2646290451 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff29c102140 0x7ff29c104530 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.833+0000 7ff2a3c00640 1 -- 172.21.15.45:0/2646290451 >> 172.21.15.45:0/2646290451 conn(0x7ff29c0fbb30 msgr2=0x7ff29c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:49.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.833+0000 7ff2a3c00640 1 -- 172.21.15.45:0/2646290451 shutdown_connections 2024-03-28T17:48:49.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.833+0000 7ff2a3c00640 1 -- 172.21.15.45:0/2646290451 wait complete. 2024-03-28T17:48:49.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.834+0000 7ff2a3c00640 1 Processor -- start 2024-03-28T17:48:49.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.834+0000 7ff2a3c00640 1 -- start start 2024-03-28T17:48:49.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.834+0000 7ff2a3c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff29c102140 0x7ff29c199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.834+0000 7ff2a3c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff29c104a70 0x7ff29c19a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.835+0000 7ff2a3c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff29c1073a0 0x7ff29c1a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.835+0000 7ff2a3c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c10d350 con 0x7ff29c104a70 2024-03-28T17:48:49.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.835+0000 7ff2a3c00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c10d1d0 con 0x7ff29c1073a0 2024-03-28T17:48:49.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.835+0000 7ff2a3c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c10d4d0 con 0x7ff29c102140 2024-03-28T17:48:49.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.835+0000 7ff2a1200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff29c102140 0x7ff29c199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.835+0000 7ff2a1c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff29c1073a0 0x7ff29c1a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.835+0000 7ff29be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff29c104a70 0x7ff29c19a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:49.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.835+0000 7ff2a1c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff29c1073a0 0x7ff29c1a1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:46602/0 (socket says 172.21.15.45:46602) 2024-03-28T17:48:49.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.835+0000 7ff2a1200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff29c102140 0x7ff29c199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:60964/0 (socket says 172.21.15.45:60964) 2024-03-28T17:48:49.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.835+0000 7ff2a1c00640 1 -- 172.21.15.45:0/678310328 learned_addr learned my addr 172.21.15.45:0/678310328 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:49.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.835+0000 7ff2a1c00640 1 -- 172.21.15.45:0/678310328 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff29c102140 msgr2=0x7ff29c199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:49.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.835+0000 7ff2a1c00640 1 --2- 172.21.15.45:0/678310328 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff29c102140 0x7ff29c199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.835+0000 7ff2a1c00640 1 -- 172.21.15.45:0/678310328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff29c104a70 msgr2=0x7ff29c19a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:49.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.835+0000 7ff2a1c00640 1 --2- 172.21.15.45:0/678310328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff29c104a70 0x7ff29c19a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:49.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.835+0000 7ff2a1c00640 1 -- 172.21.15.45:0/678310328 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff29c1a17a0 con 0x7ff29c1073a0 2024-03-28T17:48:49.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.835+0000 7ff2a1200640 1 --2- 172.21.15.45:0/678310328 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff29c102140 0x7ff29c199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:48:49.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.836+0000 7ff2a1c00640 1 --2- 172.21.15.45:0/678310328 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff29c1073a0 0x7ff29c1a1130 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff29400eb10 tx=0x7ff29400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:49.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.836+0000 7ff299600640 1 -- 172.21.15.45:0/678310328 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29400ce20 con 0x7ff29c1073a0 2024-03-28T17:48:49.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.836+0000 7ff299600640 1 -- 172.21.15.45:0/678310328 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff294005020 con 0x7ff29c1073a0 2024-03-28T17:48:49.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.836+0000 7ff2a3c00640 1 -- 172.21.15.45:0/678310328 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff29c1a1a90 con 0x7ff29c1073a0 2024-03-28T17:48:49.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.836+0000 7ff299600640 1 -- 172.21.15.45:0/678310328 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff294010690 con 0x7ff29c1073a0 2024-03-28T17:48:49.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.836+0000 7ff2a3c00640 1 -- 172.21.15.45:0/678310328 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff29c1a1f90 con 0x7ff29c1073a0 2024-03-28T17:48:49.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.837+0000 7ff2a3c00640 1 -- 172.21.15.45:0/678310328 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff29c108ef0 con 0x7ff29c1073a0 2024-03-28T17:48:49.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.839+0000 7ff299600640 1 -- 172.21.15.45:0/678310328 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) v1 ==== 137358+0+0 (secure 0 0 0) 0x7ff294005540 con 0x7ff29c1073a0 2024-03-28T17:48:49.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.840+0000 7ff299600640 1 --2- 172.21.15.45:0/678310328 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff2840a2990 0x7ff2840a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:49.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.840+0000 7ff2a1200640 1 -- 172.21.15.45:0/678310328 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff2840a2990 msgr2=0x7ff2840a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1246060167 2024-03-28T17:48:49.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.840+0000 7ff299600640 1 -- 172.21.15.45:0/678310328 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff2940cf250 con 0x7ff29c1073a0 2024-03-28T17:48:49.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.841+0000 7ff2a1200640 1 --2- 172.21.15.45:0/678310328 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff2840a2990 0x7ff2840a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T17:48:49.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:49.848+0000 7ff299600640 1 -- 172.21.15.45:0/678310328 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff29409c140 con 0x7ff29c1073a0 2024-03-28T17:48:49.916 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:49 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: ignoring --setuser ceph since I am not root 2024-03-28T17:48:49.916 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:49 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: ignoring --setgroup ceph since I am not root 2024-03-28T17:48:49.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:49 smithi045 ceph-mon[31773]: pgmap v397: 97 pgs: 97 active+clean; 649 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-28T17:48:49.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:49 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/81568504' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:48:49.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:49 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3043207053' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T17:48:49.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:49 smithi045 ceph-mon[31773]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T17:48:50.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.042+0000 7ff2a1200640 1 -- 172.21.15.45:0/678310328 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff2840a2990 msgr2=0x7ff2840a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1246060167 2024-03-28T17:48:50.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.042+0000 7ff2a1200640 1 --2- 172.21.15.45:0/678310328 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff2840a2990 0x7ff2840a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T17:48:50.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.055+0000 7ff2a3c00640 1 -- 172.21.15.45:0/678310328 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7ff29c0622a0 con 0x7ff29c1073a0 2024-03-28T17:48:50.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:49 smithi071 ceph-mon[36597]: pgmap v397: 97 pgs: 97 active+clean; 649 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-28T17:48:50.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:49 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/81568504' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:48:50.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:49 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3043207053' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T17:48:50.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:49 smithi071 ceph-mon[36597]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T17:48:50.061 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:49 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: ignoring --setuser ceph since I am not root 2024-03-28T17:48:50.061 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:49 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: ignoring --setgroup ceph since I am not root 2024-03-28T17:48:50.061 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:49 smithi071 ceph-mgr[37978]: -- 172.21.15.71:0/2452089568 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5627a3c7a5a0 con 0x5627a3c4e800 2024-03-28T17:48:50.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.066+0000 7ff299600640 1 -- 172.21.15.45:0/678310328 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff29409ba40 con 0x7ff29c1073a0 2024-03-28T17:48:50.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.069+0000 7ff2a3c00640 1 -- 172.21.15.45:0/678310328 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff2840a2990 msgr2=0x7ff2840a4e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T17:48:50.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.069+0000 7ff2a3c00640 1 --2- 172.21.15.45:0/678310328 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff2840a2990 0x7ff2840a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:50.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.069+0000 7ff2a3c00640 1 -- 172.21.15.45:0/678310328 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff29c1073a0 msgr2=0x7ff29c1a1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:50.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.069+0000 7ff2a3c00640 1 --2- 172.21.15.45:0/678310328 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff29c1073a0 0x7ff29c1a1130 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff29400eb10 tx=0x7ff29400ef10 comp rx=0 tx=0).stop 2024-03-28T17:48:50.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.069+0000 7ff2a3c00640 1 -- 172.21.15.45:0/678310328 shutdown_connections 2024-03-28T17:48:50.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.069+0000 7ff2a3c00640 1 --2- 172.21.15.45:0/678310328 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7ff2840a2990 0x7ff2840a4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:50.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.069+0000 7ff2a3c00640 1 --2- 172.21.15.45:0/678310328 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff29c1073a0 0x7ff29c1a1130 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:50.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.069+0000 7ff2a3c00640 1 --2- 172.21.15.45:0/678310328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff29c104a70 0x7ff29c19a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:50.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.069+0000 7ff2a3c00640 1 --2- 172.21.15.45:0/678310328 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff29c102140 0x7ff29c199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:50.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.069+0000 7ff2a3c00640 1 -- 172.21.15.45:0/678310328 >> 172.21.15.45:0/678310328 conn(0x7ff29c0fbb30 msgr2=0x7ff29c1018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:50.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.069+0000 7ff2a3c00640 1 -- 172.21.15.45:0/678310328 shutdown_connections 2024-03-28T17:48:50.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.069+0000 7ff2a3c00640 1 -- 172.21.15.45:0/678310328 wait complete. 2024-03-28T17:48:50.073 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:49 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: ignoring --setuser ceph since I am not root 2024-03-28T17:48:50.073 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:49 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: ignoring --setgroup ceph since I am not root 2024-03-28T17:48:50.073 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:49 smithi151 ceph-mgr[38352]: -- 172.21.15.151:0/2300286254 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a637cb45a0 con 0x55a637c88800 2024-03-28T17:48:50.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:49 smithi151 ceph-mon[36481]: pgmap v397: 97 pgs: 97 active+clean; 649 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-28T17:48:50.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:49 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/81568504' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T17:48:50.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:49 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3043207053' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T17:48:50.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:49 smithi151 ceph-mon[36481]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T17:48:50.089 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-28T17:48:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.288+0000 7f84f9200640 1 -- 172.21.15.45:0/108024814 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84f4101700 msgr2=0x7f84f4101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:50.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.288+0000 7f84f9200640 1 --2- 172.21.15.45:0/108024814 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84f4101700 0x7f84f4101b20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f84e4009900 tx=0x7f84e402bc40 comp rx=0 tx=0).stop 2024-03-28T17:48:50.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.288+0000 7f84f9200640 1 -- 172.21.15.45:0/108024814 shutdown_connections 2024-03-28T17:48:50.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.288+0000 7f84f9200640 1 --2- 172.21.15.45:0/108024814 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84f41032b0 0x7f84f4109b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:50.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.288+0000 7f84f9200640 1 --2- 172.21.15.45:0/108024814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84f41028f0 0x7f84f4102d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:50.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.288+0000 7f84f9200640 1 --2- 172.21.15.45:0/108024814 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84f4101700 0x7f84f4101b20 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:50.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.288+0000 7f84f9200640 1 -- 172.21.15.45:0/108024814 >> 172.21.15.45:0/108024814 conn(0x7f84f40fcc70 msgr2=0x7f84f40ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:50.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.288+0000 7f84f9200640 1 -- 172.21.15.45:0/108024814 shutdown_connections 2024-03-28T17:48:50.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.288+0000 7f84f9200640 1 -- 172.21.15.45:0/108024814 wait complete. 2024-03-28T17:48:50.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.289+0000 7f84f9200640 1 Processor -- start 2024-03-28T17:48:50.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.289+0000 7f84f9200640 1 -- start start 2024-03-28T17:48:50.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.289+0000 7f84f9200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84f4101700 0x7f84f4199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:50.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.290+0000 7f84f9200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84f41028f0 0x7f84f419a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:50.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.290+0000 7f84f9200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84f41032b0 0x7f84f41a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.290+0000 7f84f9200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f410d3a0 con 0x7f84f4101700 2024-03-28T17:48:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.290+0000 7f84f9200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f410d220 con 0x7f84f41032b0 2024-03-28T17:48:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.290+0000 7f84f9200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f410d520 con 0x7f84f41028f0 2024-03-28T17:48:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.290+0000 7f84f1400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84f41028f0 0x7f84f419a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.290+0000 7f84f2800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84f41032b0 0x7f84f41a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.290+0000 7f84f1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84f4101700 0x7f84f4199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.290+0000 7f84f2800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84f41032b0 0x7f84f41a1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:46614/0 (socket says 172.21.15.45:46614) 2024-03-28T17:48:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.290+0000 7f84f1400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84f41028f0 0x7f84f419a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:60980/0 (socket says 172.21.15.45:60980) 2024-03-28T17:48:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.290+0000 7f84f2800640 1 -- 172.21.15.45:0/4029361999 learned_addr learned my addr 172.21.15.45:0/4029361999 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.290+0000 7f84f2800640 1 -- 172.21.15.45:0/4029361999 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84f41028f0 msgr2=0x7f84f419a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.290+0000 7f84f2800640 1 --2- 172.21.15.45:0/4029361999 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84f41028f0 0x7f84f419a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.290+0000 7f84f2800640 1 -- 172.21.15.45:0/4029361999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84f4101700 msgr2=0x7f84f4199b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.290+0000 7f84f2800640 1 --2- 172.21.15.45:0/4029361999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84f4101700 0x7f84f4199b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:50.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.290+0000 7f84f2800640 1 -- 172.21.15.45:0/4029361999 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f41a1770 con 0x7f84f41032b0 2024-03-28T17:48:50.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.291+0000 7f84f2800640 1 --2- 172.21.15.45:0/4029361999 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84f41032b0 0x7f84f41a1100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f84ec00eb10 tx=0x7f84ec00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:50.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.291+0000 7f84eaa00640 1 -- 172.21.15.45:0/4029361999 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec004b20 con 0x7f84f41032b0 2024-03-28T17:48:50.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.291+0000 7f84eaa00640 1 -- 172.21.15.45:0/4029361999 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84ec004cc0 con 0x7f84f41032b0 2024-03-28T17:48:50.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.291+0000 7f84eaa00640 1 -- 172.21.15.45:0/4029361999 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec010760 con 0x7f84f41032b0 2024-03-28T17:48:50.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.291+0000 7f84f9200640 1 -- 172.21.15.45:0/4029361999 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84f41a1a60 con 0x7f84f41032b0 2024-03-28T17:48:50.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.291+0000 7f84f9200640 1 -- 172.21.15.45:0/4029361999 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f41a1f60 con 0x7f84f41032b0 2024-03-28T17:48:50.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.292+0000 7f84f9200640 1 -- 172.21.15.45:0/4029361999 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84f41a2190 con 0x7f84f41032b0 2024-03-28T17:48:50.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.294+0000 7f84eaa00640 1 -- 172.21.15.45:0/4029361999 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) v1 ==== 137358+0+0 (secure 0 0 0) 0x7f84ec005020 con 0x7f84f41032b0 2024-03-28T17:48:50.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.295+0000 7f84eaa00640 1 --2- 172.21.15.45:0/4029361999 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f84d80a2990 0x7f84d80a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:50.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.296+0000 7f84f1e00640 1 -- 172.21.15.45:0/4029361999 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f84d80a2990 msgr2=0x7f84d80a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1246060167 2024-03-28T17:48:50.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.296+0000 7f84eaa00640 1 -- 172.21.15.45:0/4029361999 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f84ec0cf210 con 0x7f84f41032b0 2024-03-28T17:48:50.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.296+0000 7f84f1e00640 1 --2- 172.21.15.45:0/4029361999 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f84d80a2990 0x7f84d80a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T17:48:50.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.303+0000 7f84eaa00640 1 -- 172.21.15.45:0/4029361999 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f84ec09c100 con 0x7f84f41032b0 2024-03-28T17:48:50.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.497+0000 7f84f1e00640 1 -- 172.21.15.45:0/4029361999 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f84d80a2990 msgr2=0x7f84d80a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1246060167 2024-03-28T17:48:50.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.497+0000 7f84f1e00640 1 --2- 172.21.15.45:0/4029361999 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f84d80a2990 0x7f84d80a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T17:48:50.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.507+0000 7f84f9200640 1 -- 172.21.15.45:0/4029361999 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f84f41a2360 con 0x7f84f41032b0 2024-03-28T17:48:50.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.517+0000 7f84eaa00640 1 -- 172.21.15.45:0/4029361999 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) v1 ==== 133+0+0 (secure 0 0 0) 0x7f84ec09ba00 con 0x7f84f41032b0 2024-03-28T17:48:50.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.520+0000 7f84f9200640 1 -- 172.21.15.45:0/4029361999 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f84d80a2990 msgr2=0x7f84d80a4e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T17:48:50.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.520+0000 7f84f9200640 1 --2- 172.21.15.45:0/4029361999 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f84d80a2990 0x7f84d80a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:50.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.520+0000 7f84f9200640 1 -- 172.21.15.45:0/4029361999 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84f41032b0 msgr2=0x7f84f41a1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:50.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.520+0000 7f84f9200640 1 --2- 172.21.15.45:0/4029361999 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84f41032b0 0x7f84f41a1100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f84ec00eb10 tx=0x7f84ec00c2d0 comp rx=0 tx=0).stop 2024-03-28T17:48:50.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.520+0000 7f84f9200640 1 -- 172.21.15.45:0/4029361999 shutdown_connections 2024-03-28T17:48:50.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.520+0000 7f84f9200640 1 --2- 172.21.15.45:0/4029361999 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f84d80a2990 0x7f84d80a4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:50.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.520+0000 7f84f9200640 1 --2- 172.21.15.45:0/4029361999 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84f41032b0 0x7f84f41a1100 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:50.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.520+0000 7f84f9200640 1 --2- 172.21.15.45:0/4029361999 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84f41028f0 0x7f84f419a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:50.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.520+0000 7f84f9200640 1 --2- 172.21.15.45:0/4029361999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84f4101700 0x7f84f4199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:50.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.520+0000 7f84f9200640 1 -- 172.21.15.45:0/4029361999 >> 172.21.15.45:0/4029361999 conn(0x7f84f40fcc70 msgr2=0x7f84f40738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:50.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.520+0000 7f84f9200640 1 -- 172.21.15.45:0/4029361999 shutdown_connections 2024-03-28T17:48:50.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.520+0000 7f84f9200640 1 -- 172.21.15.45:0/4029361999 wait complete. 2024-03-28T17:48:50.537 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-28T17:48:50.560 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:50 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:50.199+0000 7fea5d03a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T17:48:50.560 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:50 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:50.294+0000 7fea5d03a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T17:48:50.573 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:50 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:50.264+0000 7fa0c4058200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T17:48:50.573 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:50 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:50.359+0000 7fa0c4058200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T17:48:50.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.724+0000 7f0a49000640 1 -- 172.21.15.45:0/1268812094 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a44104a70 msgr2=0x7f0a44106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:50.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.724+0000 7f0a49000640 1 --2- 172.21.15.45:0/1268812094 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a44104a70 0x7f0a44106e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0a30009900 tx=0x7f0a3002bc30 comp rx=0 tx=0).stop 2024-03-28T17:48:50.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.724+0000 7f0a49000640 1 -- 172.21.15.45:0/1268812094 shutdown_connections 2024-03-28T17:48:50.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.724+0000 7f0a49000640 1 --2- 172.21.15.45:0/1268812094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a441073a0 0x7f0a441097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:50.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.724+0000 7f0a49000640 1 --2- 172.21.15.45:0/1268812094 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a44104a70 0x7f0a44106e60 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:50.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.724+0000 7f0a49000640 1 --2- 172.21.15.45:0/1268812094 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a44102140 0x7f0a44104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:50.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.724+0000 7f0a49000640 1 -- 172.21.15.45:0/1268812094 >> 172.21.15.45:0/1268812094 conn(0x7f0a440fbb30 msgr2=0x7f0a440fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:50.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.724+0000 7f0a49000640 1 -- 172.21.15.45:0/1268812094 shutdown_connections 2024-03-28T17:48:50.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.724+0000 7f0a49000640 1 -- 172.21.15.45:0/1268812094 wait complete. 2024-03-28T17:48:50.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.725+0000 7f0a49000640 1 Processor -- start 2024-03-28T17:48:50.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.725+0000 7f0a49000640 1 -- start start 2024-03-28T17:48:50.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.725+0000 7f0a49000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a44102140 0x7f0a44101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:50.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.725+0000 7f0a49000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a44104a70 0x7f0a441000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:50.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.725+0000 7f0a49000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a441073a0 0x7f0a44100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:50.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.725+0000 7f0a49000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4410d3a0 con 0x7f0a44102140 2024-03-28T17:48:50.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.725+0000 7f0a49000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4410d220 con 0x7f0a441073a0 2024-03-28T17:48:50.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.725+0000 7f0a49000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4410d520 con 0x7f0a44104a70 2024-03-28T17:48:50.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.725+0000 7f0a41e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a44102140 0x7f0a44101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:50.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.725+0000 7f0a41400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a44104a70 0x7f0a441000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:50.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.725+0000 7f0a41e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a44102140 0x7f0a44101ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55190/0 (socket says 172.21.15.45:55190) 2024-03-28T17:48:50.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.725+0000 7f0a41e00640 1 -- 172.21.15.45:0/1691910450 learned_addr learned my addr 172.21.15.45:0/1691910450 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:48:50.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.725+0000 7f0a42800640 1 --2- 172.21.15.45:0/1691910450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a441073a0 0x7f0a44100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:50.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.726+0000 7f0a41e00640 1 -- 172.21.15.45:0/1691910450 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a44104a70 msgr2=0x7f0a441000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:50.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.726+0000 7f0a41e00640 1 --2- 172.21.15.45:0/1691910450 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a44104a70 0x7f0a441000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:50.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.726+0000 7f0a41e00640 1 -- 172.21.15.45:0/1691910450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a441073a0 msgr2=0x7f0a44100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:50.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.726+0000 7f0a41e00640 1 --2- 172.21.15.45:0/1691910450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a441073a0 0x7f0a44100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:50.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.726+0000 7f0a41e00640 1 -- 172.21.15.45:0/1691910450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a44100ec0 con 0x7f0a44102140 2024-03-28T17:48:50.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.726+0000 7f0a42800640 1 --2- 172.21.15.45:0/1691910450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a441073a0 0x7f0a44100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:48:50.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.726+0000 7f0a41e00640 1 --2- 172.21.15.45:0/1691910450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a44102140 0x7f0a44101ab0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0a3400eb90 tx=0x7f0a3400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:50.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.726+0000 7f0a3aa00640 1 -- 172.21.15.45:0/1691910450 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a34004d20 con 0x7f0a44102140 2024-03-28T17:48:50.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.726+0000 7f0a49000640 1 -- 172.21.15.45:0/1691910450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a441a1a70 con 0x7f0a44102140 2024-03-28T17:48:50.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.726+0000 7f0a3aa00640 1 -- 172.21.15.45:0/1691910450 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a3400ee90 con 0x7f0a44102140 2024-03-28T17:48:50.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.726+0000 7f0a3aa00640 1 -- 172.21.15.45:0/1691910450 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a34010830 con 0x7f0a44102140 2024-03-28T17:48:50.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.727+0000 7f0a49000640 1 -- 172.21.15.45:0/1691910450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a441a1f40 con 0x7f0a44102140 2024-03-28T17:48:50.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.728+0000 7f0a49000640 1 -- 172.21.15.45:0/1691910450 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a4404f400 con 0x7f0a44102140 2024-03-28T17:48:50.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.730+0000 7f0a3aa00640 1 -- 172.21.15.45:0/1691910450 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) v1 ==== 137358+0+0 (secure 0 0 0) 0x7f0a34005020 con 0x7f0a44102140 2024-03-28T17:48:50.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.731+0000 7f0a3aa00640 1 --2- 172.21.15.45:0/1691910450 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f0a280a2990 0x7f0a280a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:50.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.731+0000 7f0a41400640 1 -- 172.21.15.45:0/1691910450 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f0a280a2990 msgr2=0x7f0a280a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1246060167 2024-03-28T17:48:50.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.731+0000 7f0a3aa00640 1 -- 172.21.15.45:0/1691910450 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f0a340cf3c0 con 0x7f0a44102140 2024-03-28T17:48:50.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.731+0000 7f0a41400640 1 --2- 172.21.15.45:0/1691910450 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f0a280a2990 0x7f0a280a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T17:48:50.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.735+0000 7f0a3aa00640 1 -- 172.21.15.45:0/1691910450 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0a3409c2b0 con 0x7f0a44102140 2024-03-28T17:48:50.915 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:50.467+0000 7f53abd1b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T17:48:50.916 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:50.593+0000 7f53abd1b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T17:48:50.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:50 smithi045 ceph-mon[31773]: scrub queued for path: / 2024-03-28T17:48:50.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:50 smithi045 ceph-mon[31773]: scrub summary: idle+waiting paths [/] 2024-03-28T17:48:50.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:50 smithi045 ceph-mon[31773]: scrub summary: active paths [/] 2024-03-28T17:48:50.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:50 smithi045 ceph-mon[31773]: scrub summary: idle+waiting paths [/] 2024-03-28T17:48:50.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:50 smithi045 ceph-mon[31773]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T17:48:50.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:50 smithi045 ceph-mon[31773]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-28T17:48:50.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:50 smithi045 ceph-mon[31773]: from='client.? ' entity='client.admin' 2024-03-28T17:48:50.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:50 smithi045 ceph-mon[31773]: from='client.? ' entity='client.admin' 2024-03-28T17:48:50.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.932+0000 7f0a41400640 1 -- 172.21.15.45:0/1691910450 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f0a280a2990 msgr2=0x7f0a280a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1246060167 2024-03-28T17:48:50.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.932+0000 7f0a41400640 1 --2- 172.21.15.45:0/1691910450 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f0a280a2990 0x7f0a280a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T17:48:50.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:50.960+0000 7f0a49000640 1 -- 172.21.15.45:0/1691910450 --> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a44061ae0 con 0x7f0a280a2990 2024-03-28T17:48:50.994 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:50 smithi071 ceph-mon[36597]: scrub queued for path: / 2024-03-28T17:48:50.994 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:50 smithi071 ceph-mon[36597]: scrub summary: idle+waiting paths [/] 2024-03-28T17:48:50.994 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:50 smithi071 ceph-mon[36597]: scrub summary: active paths [/] 2024-03-28T17:48:50.994 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:50 smithi071 ceph-mon[36597]: scrub summary: idle+waiting paths [/] 2024-03-28T17:48:50.994 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:50 smithi071 ceph-mon[36597]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T17:48:50.994 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:50 smithi071 ceph-mon[36597]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-28T17:48:50.994 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:50 smithi071 ceph-mon[36597]: from='client.? ' entity='client.admin' 2024-03-28T17:48:50.994 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:50 smithi071 ceph-mon[36597]: from='client.? ' entity='client.admin' 2024-03-28T17:48:51.013 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:50 smithi151 ceph-mon[36481]: scrub queued for path: / 2024-03-28T17:48:51.013 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:50 smithi151 ceph-mon[36481]: scrub summary: idle+waiting paths [/] 2024-03-28T17:48:51.013 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:50 smithi151 ceph-mon[36481]: scrub summary: active paths [/] 2024-03-28T17:48:51.013 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:50 smithi151 ceph-mon[36481]: scrub summary: idle+waiting paths [/] 2024-03-28T17:48:51.013 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:50 smithi151 ceph-mon[36481]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T17:48:51.013 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:50 smithi151 ceph-mon[36481]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-28T17:48:51.013 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:50 smithi151 ceph-mon[36481]: from='client.? ' entity='client.admin' 2024-03-28T17:48:51.013 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:50 smithi151 ceph-mon[36481]: from='client.? ' entity='client.admin' 2024-03-28T17:48:51.266 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:50 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:50.993+0000 7fea5d03a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T17:48:51.266 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:51.122+0000 7fea5d03a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T17:48:51.266 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:51.265+0000 7fea5d03a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T17:48:51.323 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:51.012+0000 7fa0c4058200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T17:48:51.323 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:51.107+0000 7fa0c4058200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T17:48:51.323 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:51.254+0000 7fa0c4058200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T17:48:51.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:51.333+0000 7f0a41400640 1 -- 172.21.15.45:0/1691910450 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f0a280a2990 msgr2=0x7f0a280a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1246060167 2024-03-28T17:48:51.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:51.333+0000 7f0a41400640 1 --2- 172.21.15.45:0/1691910450 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f0a280a2990 0x7f0a280a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T17:48:51.560 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:51.363+0000 7fea5d03a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T17:48:51.561 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:51.462+0000 7fea5d03a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T17:48:51.719 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:51.364+0000 7fa0c4058200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T17:48:51.720 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:51.449+0000 7fa0c4058200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T17:48:51.734 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:51.421+0000 7f53abd1b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T17:48:51.734 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:51.548+0000 7f53abd1b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T17:48:51.734 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:51.733+0000 7f53abd1b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T17:48:52.020 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:51.719+0000 7fa0c4058200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T17:48:52.020 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:51.835+0000 7fa0c4058200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T17:48:52.020 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:51.922+0000 7fa0c4058200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T17:48:52.060 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:51.783+0000 7fea5d03a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T17:48:52.060 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:51.896+0000 7fea5d03a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T17:48:52.061 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:52 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:52.016+0000 7fea5d03a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T17:48:52.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:52.134+0000 7f0a41400640 1 -- 172.21.15.45:0/1691910450 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f0a280a2990 msgr2=0x7f0a280a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1246060167 2024-03-28T17:48:52.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:52.134+0000 7f0a41400640 1 --2- 172.21.15.45:0/1691910450 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f0a280a2990 0x7f0a280a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T17:48:52.165 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:51.867+0000 7f53abd1b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T17:48:52.165 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:51.983+0000 7f53abd1b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T17:48:52.323 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:52 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:52.019+0000 7fa0c4058200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T17:48:52.323 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:52 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:52.198+0000 7fa0c4058200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T17:48:52.560 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:52 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:52.128+0000 7fea5d03a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T17:48:52.560 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:52 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:52.344+0000 7fea5d03a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T17:48:52.591 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:52 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:52.327+0000 7f53abd1b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T17:48:52.591 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:52 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:52.473+0000 7f53abd1b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T17:48:52.592 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:52 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:52.590+0000 7f53abd1b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T17:48:52.915 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:52 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:52.720+0000 7f53abd1b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T17:48:53.227 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:52 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:52.950+0000 7f53abd1b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T17:48:53.503 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:53 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:53.196+0000 7fa0c4058200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T17:48:53.503 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:53 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:53.333+0000 7fa0c4058200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T17:48:53.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:53.736+0000 7f0a41400640 1 -- 172.21.15.45:0/1691910450 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f0a280a2990 msgr2=0x7f0a280a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1246060167 2024-03-28T17:48:53.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:53.736+0000 7f0a41400640 1 --2- 172.21.15.45:0/1691910450 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f0a280a2990 0x7f0a280a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T17:48:53.789 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:53 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:53.683+0000 7fa0c4058200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T17:48:53.810 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:53 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:53.531+0000 7fea5d03a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T17:48:53.810 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:53 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:53.683+0000 7fea5d03a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T17:48:54.055 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:53 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:53.788+0000 7fa0c4058200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T17:48:54.323 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:54 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:54.054+0000 7fa0c4058200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T17:48:54.323 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:54 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:54.245+0000 7fa0c4058200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T17:48:54.379 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:54 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:54.070+0000 7fea5d03a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T17:48:54.379 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:54 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:54.177+0000 7fea5d03a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T17:48:54.379 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:54 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:54.378+0000 7fea5d03a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T17:48:54.665 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:54 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:54.287+0000 7f53abd1b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T17:48:54.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:54 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:54.468+0000 7f53abd1b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T17:48:54.810 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:54 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:54.566+0000 7fea5d03a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T17:48:54.818 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:54 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:54.536+0000 7fa0c4058200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T17:48:55.073 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:54 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:54.818+0000 7fa0c4058200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T17:48:55.165 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:54 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:54.909+0000 7f53abd1b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T17:48:55.165 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:55 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:55.033+0000 7f53abd1b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T17:48:55.179 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:54 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:54.897+0000 7fea5d03a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T17:48:55.179 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:55 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:55.179+0000 7fea5d03a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T17:48:55.559 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:55 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:55.301+0000 7f53abd1b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T17:48:55.559 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:55 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:55.558+0000 7f53abd1b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T17:48:55.560 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:55 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:55.426+0000 7fea5d03a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T17:48:55.573 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:55.104+0000 7fa0c4058200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T17:48:56.282 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:55 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:55.952+0000 7f53abd1b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T17:48:56.323 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:55.924+0000 7fa0c4058200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T17:48:56.560 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:56 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:56.284+0000 7fea5d03a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T17:48:56.598 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:56.281+0000 7f53abd1b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T17:48:56.823 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:56 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:56.507+0000 7fa0c4058200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T17:48:56.823 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:56 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:56.607+0000 7fa0c4058200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T17:48:56.915 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:56.597+0000 7f53abd1b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T17:48:56.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:56.939+0000 7f0a41400640 1 -- 172.21.15.45:0/1691910450 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f0a280a2990 msgr2=0x7f0a280a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1246060167 2024-03-28T17:48:56.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:56.939+0000 7f0a41400640 1 --2- 172.21.15.45:0/1691910450 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f0a280a2990 0x7f0a280a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-28T17:48:57.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:57.054+0000 7f0a3aa00640 1 -- 172.21.15.45:0/1691910450 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 21) v1 ==== 137358+0+0 (secure 0 0 0) 0x7f0a34098220 con 0x7f0a44102140 2024-03-28T17:48:57.265 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:57 smithi071 ceph-mon[36597]: Standby manager daemon z restarted 2024-03-28T17:48:57.265 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:57 smithi071 ceph-mon[36597]: Standby manager daemon z started 2024-03-28T17:48:57.266 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:57 smithi071 ceph-mon[36597]: from='mgr.? 172.21.15.151:0/1729237123' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T17:48:57.266 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:57 smithi071 ceph-mon[36597]: from='mgr.? 172.21.15.151:0/1729237123' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T17:48:57.266 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:57 smithi071 ceph-mon[36597]: from='mgr.? 172.21.15.151:0/1729237123' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T17:48:57.266 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:57 smithi071 ceph-mon[36597]: from='mgr.? 172.21.15.151:0/1729237123' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T17:48:57.266 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:56 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:56.921+0000 7fea5d03a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T17:48:57.266 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:57 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:57.019+0000 7fea5d03a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T17:48:57.266 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Mar 28 17:48:57 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-y[37974]: 2024-03-28T17:48:57.265+0000 7fea5d03a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T17:48:57.323 INFO:journalctl@ceph.mgr.z.smithi151.stdout:Mar 28 17:48:56 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-z[38348]: 2024-03-28T17:48:56.827+0000 7fa0c4058200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T17:48:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:57 smithi151 ceph-mon[36481]: Standby manager daemon z restarted 2024-03-28T17:48:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:57 smithi151 ceph-mon[36481]: Standby manager daemon z started 2024-03-28T17:48:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:57 smithi151 ceph-mon[36481]: from='mgr.? 172.21.15.151:0/1729237123' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T17:48:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:57 smithi151 ceph-mon[36481]: from='mgr.? 172.21.15.151:0/1729237123' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T17:48:57.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:57 smithi151 ceph-mon[36481]: from='mgr.? 172.21.15.151:0/1729237123' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T17:48:57.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:57 smithi151 ceph-mon[36481]: from='mgr.? 172.21.15.151:0/1729237123' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T17:48:57.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:57 smithi045 ceph-mon[31773]: Standby manager daemon z restarted 2024-03-28T17:48:57.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:57 smithi045 ceph-mon[31773]: Standby manager daemon z started 2024-03-28T17:48:57.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:57 smithi045 ceph-mon[31773]: from='mgr.? 172.21.15.151:0/1729237123' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T17:48:57.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:57 smithi045 ceph-mon[31773]: from='mgr.? 172.21.15.151:0/1729237123' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T17:48:57.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:57 smithi045 ceph-mon[31773]: from='mgr.? 172.21.15.151:0/1729237123' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T17:48:57.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:57 smithi045 ceph-mon[31773]: from='mgr.? 172.21.15.151:0/1729237123' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T17:48:58.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:58.057+0000 7f0a3aa00640 1 -- 172.21.15.45:0/1691910450 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 22) v1 ==== 137358+0+0 (secure 0 0 0) 0x7f0a340ccf70 con 0x7f0a44102140 2024-03-28T17:48:58.059 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:57 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:57.682+0000 7f53abd1b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T17:48:58.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:58 smithi071 ceph-mon[36597]: mgrmap e21: x(active, since 10m), standbys: y, z 2024-03-28T17:48:58.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:58 smithi071 ceph-mon[36597]: Standby manager daemon y restarted 2024-03-28T17:48:58.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:58 smithi071 ceph-mon[36597]: Standby manager daemon y started 2024-03-28T17:48:58.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:58 smithi071 ceph-mon[36597]: from='mgr.? 172.21.15.71:0/3845032077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T17:48:58.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:58 smithi071 ceph-mon[36597]: from='mgr.? 172.21.15.71:0/3845032077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T17:48:58.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:58 smithi071 ceph-mon[36597]: from='mgr.? 172.21.15.71:0/3845032077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T17:48:58.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:58 smithi071 ceph-mon[36597]: from='mgr.? 172.21.15.71:0/3845032077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T17:48:58.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:58 smithi151 ceph-mon[36481]: mgrmap e21: x(active, since 10m), standbys: y, z 2024-03-28T17:48:58.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:58 smithi151 ceph-mon[36481]: Standby manager daemon y restarted 2024-03-28T17:48:58.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:58 smithi151 ceph-mon[36481]: Standby manager daemon y started 2024-03-28T17:48:58.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:58 smithi151 ceph-mon[36481]: from='mgr.? 172.21.15.71:0/3845032077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T17:48:58.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:58 smithi151 ceph-mon[36481]: from='mgr.? 172.21.15.71:0/3845032077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T17:48:58.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:58 smithi151 ceph-mon[36481]: from='mgr.? 172.21.15.71:0/3845032077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T17:48:58.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:58 smithi151 ceph-mon[36481]: from='mgr.? 172.21.15.71:0/3845032077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T17:48:58.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:58 smithi045 ceph-mon[31773]: mgrmap e21: x(active, since 10m), standbys: y, z 2024-03-28T17:48:58.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:58 smithi045 ceph-mon[31773]: Standby manager daemon y restarted 2024-03-28T17:48:58.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:58 smithi045 ceph-mon[31773]: Standby manager daemon y started 2024-03-28T17:48:58.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:58 smithi045 ceph-mon[31773]: from='mgr.? 172.21.15.71:0/3845032077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T17:48:58.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:58 smithi045 ceph-mon[31773]: from='mgr.? 172.21.15.71:0/3845032077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T17:48:58.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:58 smithi045 ceph-mon[31773]: from='mgr.? 172.21.15.71:0/3845032077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T17:48:58.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:58 smithi045 ceph-mon[31773]: from='mgr.? 172.21.15.71:0/3845032077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T17:48:58.857 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:58.421+0000 7f53abd1b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T17:48:58.857 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:58.557+0000 7f53abd1b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T17:48:58.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:58.885+0000 7f0a3aa00640 1 -- 172.21.15.45:0/1691910450 <== mon.0 v2:172.21.15.45:3300/0 9 ==== mgrmap(e 23) v1 ==== 136975+0+0 (secure 0 0 0) 0x7f0a3409be10 con 0x7f0a44102140 2024-03-28T17:48:58.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:58.885+0000 7f0a3aa00640 1 -- 172.21.15.45:0/1691910450 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f0a280a2990 msgr2=0x7f0a280a4e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T17:48:58.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:58.885+0000 7f0a3aa00640 1 --2- 172.21.15.45:0/1691910450 >> [v2:172.21.15.45:6800/1246060167,v1:172.21.15.45:6801/1246060167] conn(0x7f0a280a2990 0x7f0a280a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:48:59.164 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:58.856+0000 7f53abd1b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T17:48:59.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: mgrmap e22: x(active, since 10m), standbys: z, y 2024-03-28T17:48:59.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: Active manager daemon x restarted 2024-03-28T17:48:59.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: Activating manager daemon x 2024-03-28T17:48:59.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T17:48:59.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T17:48:59.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:48:59.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:48:59.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: mgrmap e23: x(active, starting, since 0.024964s), standbys: z, y 2024-03-28T17:48:59.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T17:48:59.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T17:48:59.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T17:48:59.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T17:48:59.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T17:48:59.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T17:48:59.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T17:48:59.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T17:48:59.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T17:48:59.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T17:48:59.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T17:48:59.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T17:48:59.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T17:48:59.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T17:48:59.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T17:48:59.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T17:48:59.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T17:48:59.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T17:48:59.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T17:48:59.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T17:48:59.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T17:48:59.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T17:48:59.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T17:48:59.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T17:48:59.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T17:48:59.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T17:48:59.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T17:48:59.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T17:48:59.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T17:48:59.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T17:48:59.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: Manager daemon x is now available 2024-03-28T17:48:59.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:48:59.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:48:59.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T17:48:59.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T17:48:59.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T17:48:59.168 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T17:48:59.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: mgrmap e22: x(active, since 10m), standbys: z, y 2024-03-28T17:48:59.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: Active manager daemon x restarted 2024-03-28T17:48:59.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: Activating manager daemon x 2024-03-28T17:48:59.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T17:48:59.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T17:48:59.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:48:59.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:48:59.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: mgrmap e23: x(active, starting, since 0.024964s), standbys: z, y 2024-03-28T17:48:59.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T17:48:59.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T17:48:59.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T17:48:59.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T17:48:59.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T17:48:59.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T17:48:59.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T17:48:59.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T17:48:59.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T17:48:59.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T17:48:59.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T17:48:59.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T17:48:59.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T17:48:59.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T17:48:59.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T17:48:59.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T17:48:59.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T17:48:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T17:48:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T17:48:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T17:48:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T17:48:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T17:48:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T17:48:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T17:48:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T17:48:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T17:48:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T17:48:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T17:48:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T17:48:59.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T17:48:59.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: Manager daemon x is now available 2024-03-28T17:48:59.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:48:59.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:48:59.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T17:48:59.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T17:48:59.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T17:48:59.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:48:59 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T17:48:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: mgrmap e22: x(active, since 10m), standbys: z, y 2024-03-28T17:48:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: Active manager daemon x restarted 2024-03-28T17:48:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: Activating manager daemon x 2024-03-28T17:48:59.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T17:48:59.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T17:48:59.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T17:48:59.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T17:48:59.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: mgrmap e23: x(active, starting, since 0.024964s), standbys: z, y 2024-03-28T17:48:59.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T17:48:59.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T17:48:59.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T17:48:59.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T17:48:59.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T17:48:59.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T17:48:59.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T17:48:59.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T17:48:59.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T17:48:59.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T17:48:59.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T17:48:59.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T17:48:59.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T17:48:59.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T17:48:59.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T17:48:59.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T17:48:59.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T17:48:59.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T17:48:59.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T17:48:59.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T17:48:59.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T17:48:59.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T17:48:59.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T17:48:59.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T17:48:59.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T17:48:59.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T17:48:59.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T17:48:59.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T17:48:59.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T17:48:59.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T17:48:59.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: Manager daemon x is now available 2024-03-28T17:48:59.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:48:59.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:48:59.327 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T17:48:59.327 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T17:48:59.327 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T17:48:59.327 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:48:59 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T17:48:59.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:59.162+0000 7f52fd200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:48:59.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:59.162+0000 7f52fd200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:48:59.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:59.162+0000 7f52fd200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:48:59.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:59.162+0000 7f52fd200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:48:59.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:59.162+0000 7f52fd200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:48:59.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:59.163+0000 7f5301800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:48:59.417 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:59.163+0000 7f5301800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:48:59.417 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:59.163+0000 7f5301800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:48:59.417 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:59.163+0000 7f5301800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:48:59.417 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:48:59 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:48:59.163+0000 7f5301800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:48:59.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:59.897+0000 7f0a3aa00640 1 -- 172.21.15.45:0/1691910450 <== mon.0 v2:172.21.15.45:3300/0 10 ==== mgrmap(e 24) v1 ==== 137102+0+0 (secure 0 0 0) 0x7f0a3409b4d0 con 0x7f0a44102140 2024-03-28T17:48:59.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:59.898+0000 7f0a3aa00640 1 --2- 172.21.15.45:0/1691910450 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f0a28068990 0x7f0a28068d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:48:59.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:59.898+0000 7f0a3aa00640 1 -- 172.21.15.45:0/1691910450 --> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a3401fe80 con 0x7f0a28068990 2024-03-28T17:48:59.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:59.899+0000 7f0a41400640 1 --2- 172.21.15.45:0/1691910450 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f0a28068990 0x7f0a28068d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:48:59.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:59.905+0000 7f0a41400640 1 --2- 172.21.15.45:0/1691910450 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f0a28068990 0x7f0a28068d70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0a3002c170 tx=0x7f0a3002df90 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:48:59.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:59.986+0000 7f0a3aa00640 1 -- 172.21.15.45:0/1691910450 <== mgr.25354 v2:172.21.15.45:6800/1309597278 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f0a3401fe80 con 0x7f0a28068990 2024-03-28T17:48:59.988 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path / 2024-03-28T17:48:59.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:59.990+0000 7f0a49000640 1 -- 172.21.15.45:0/1691910450 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f0a28068990 msgr2=0x7f0a28068d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:59.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:59.990+0000 7f0a49000640 1 --2- 172.21.15.45:0/1691910450 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f0a28068990 0x7f0a28068d70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0a3002c170 tx=0x7f0a3002df90 comp rx=0 tx=0).stop 2024-03-28T17:48:59.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:59.990+0000 7f0a49000640 1 -- 172.21.15.45:0/1691910450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a44102140 msgr2=0x7f0a44101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:48:59.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:59.990+0000 7f0a49000640 1 --2- 172.21.15.45:0/1691910450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a44102140 0x7f0a44101ab0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0a3400eb90 tx=0x7f0a3400c6a0 comp rx=0 tx=0).stop 2024-03-28T17:48:59.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:59.990+0000 7f0a42800640 1 -- 172.21.15.45:0/1691910450 reap_dead start 2024-03-28T17:48:59.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:59.990+0000 7f0a49000640 1 -- 172.21.15.45:0/1691910450 shutdown_connections 2024-03-28T17:48:59.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:59.990+0000 7f0a49000640 1 -- 172.21.15.45:0/1691910450 >> 172.21.15.45:0/1691910450 conn(0x7f0a440fbb30 msgr2=0x7f0a44102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:48:59.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:59.991+0000 7f0a49000640 1 -- 172.21.15.45:0/1691910450 shutdown_connections 2024-03-28T17:48:59.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:48:59.991+0000 7f0a49000640 1 -- 172.21.15.45:0/1691910450 wait complete. 2024-03-28T17:49:00.018 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-28T17:49:00.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.209+0000 7f5fbc000640 1 -- 172.21.15.45:0/3609219624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5fb41038f0 msgr2=0x7f5fb4103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:00.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.209+0000 7f5fbc000640 1 --2- 172.21.15.45:0/3609219624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5fb41038f0 0x7f5fb4103d10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009900 tx=0x7f5fa802bc30 comp rx=0 tx=0).stop 2024-03-28T17:49:00.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.210+0000 7f5fbc000640 1 -- 172.21.15.45:0/3609219624 shutdown_connections 2024-03-28T17:49:00.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.210+0000 7f5fbc000640 1 --2- 172.21.15.45:0/3609219624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fb41054a0 0x7f5fb410bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:00.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.210+0000 7f5fbc000640 1 --2- 172.21.15.45:0/3609219624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5fb4104ae0 0x7f5fb4104f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:00.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.210+0000 7f5fbc000640 1 --2- 172.21.15.45:0/3609219624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5fb41038f0 0x7f5fb4103d10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:00.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.210+0000 7f5fbc000640 1 -- 172.21.15.45:0/3609219624 >> 172.21.15.45:0/3609219624 conn(0x7f5fb40fee60 msgr2=0x7f5fb41012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:00.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.210+0000 7f5fbc000640 1 -- 172.21.15.45:0/3609219624 shutdown_connections 2024-03-28T17:49:00.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.210+0000 7f5fbc000640 1 -- 172.21.15.45:0/3609219624 wait complete. 2024-03-28T17:49:00.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.211+0000 7f5fbc000640 1 Processor -- start 2024-03-28T17:49:00.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.211+0000 7f5fbc000640 1 -- start start 2024-03-28T17:49:00.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.211+0000 7f5fbc000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fb41038f0 0x7f5fb4199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:00.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.211+0000 7f5fbc000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5fb4104ae0 0x7f5fb419a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:00.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.211+0000 7f5fbc000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5fb41054a0 0x7f5fb41a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:00.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.211+0000 7f5fbc000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb410f590 con 0x7f5fb41038f0 2024-03-28T17:49:00.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.211+0000 7f5fbc000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb410f410 con 0x7f5fb41054a0 2024-03-28T17:49:00.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.211+0000 7f5fbc000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb410f710 con 0x7f5fb4104ae0 2024-03-28T17:49:00.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.211+0000 7f5fb9600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fb41038f0 0x7f5fb4199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:00.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.211+0000 7f5fb8c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5fb4104ae0 0x7f5fb419a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.211+0000 7f5fba000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5fb41054a0 0x7f5fb41a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.211+0000 7f5fb9600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fb41038f0 0x7f5fb4199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43160/0 (socket says 172.21.15.45:43160) 2024-03-28T17:49:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.211+0000 7f5fb9600640 1 -- 172.21.15.45:0/1284926896 learned_addr learned my addr 172.21.15.45:0/1284926896 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.212+0000 7f5fb9600640 1 -- 172.21.15.45:0/1284926896 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5fb4104ae0 msgr2=0x7f5fb419a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.212+0000 7f5fb9600640 1 --2- 172.21.15.45:0/1284926896 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5fb4104ae0 0x7f5fb419a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.212+0000 7f5fb9600640 1 -- 172.21.15.45:0/1284926896 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5fb41054a0 msgr2=0x7f5fb41a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.212+0000 7f5fb9600640 1 --2- 172.21.15.45:0/1284926896 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5fb41054a0 0x7f5fb41a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.212+0000 7f5fb9600640 1 -- 172.21.15.45:0/1284926896 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb41a1760 con 0x7f5fb41038f0 2024-03-28T17:49:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.212+0000 7f5fba000640 1 --2- 172.21.15.45:0/1284926896 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5fb41054a0 0x7f5fb41a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:49:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.212+0000 7f5fb9600640 1 --2- 172.21.15.45:0/1284926896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fb41038f0 0x7f5fb4199b40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8000c00 tx=0x7f5fa802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.213+0000 7f5fb2000640 1 -- 172.21.15.45:0/1284926896 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8043020 con 0x7f5fb41038f0 2024-03-28T17:49:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.213+0000 7f5fbc000640 1 -- 172.21.15.45:0/1284926896 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb41a19f0 con 0x7f5fb41038f0 2024-03-28T17:49:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.213+0000 7f5fb2000640 1 -- 172.21.15.45:0/1284926896 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fa80054e0 con 0x7f5fb41038f0 2024-03-28T17:49:00.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.213+0000 7f5fb2000640 1 -- 172.21.15.45:0/1284926896 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8003c40 con 0x7f5fb41038f0 2024-03-28T17:49:00.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.213+0000 7f5fbc000640 1 -- 172.21.15.45:0/1284926896 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb41a1f90 con 0x7f5fb41038f0 2024-03-28T17:49:00.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.214+0000 7f5fbc000640 1 -- 172.21.15.45:0/1284926896 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb410ac10 con 0x7f5fb41038f0 2024-03-28T17:49:00.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.218+0000 7f5fb2000640 1 -- 172.21.15.45:0/1284926896 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 24) v1 ==== 137102+0+0 (secure 0 0 0) 0x7f5fa8046020 con 0x7f5fb41038f0 2024-03-28T17:49:00.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.220+0000 7f5fb2000640 1 --2- 172.21.15.45:0/1284926896 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f5f9c0a26b0 0x7f5f9c0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:00.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.220+0000 7f5fb2000640 1 -- 172.21.15.45:0/1284926896 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f5fa80f0b50 con 0x7f5fb41038f0 2024-03-28T17:49:00.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.220+0000 7f5fb8c00640 1 --2- 172.21.15.45:0/1284926896 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f5f9c0a26b0 0x7f5f9c0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:00.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.221+0000 7f5fb8c00640 1 --2- 172.21.15.45:0/1284926896 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f5f9c0a26b0 0x7f5f9c0a4b70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5fb419b050 tx=0x7f5fa0005f90 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:00.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.222+0000 7f5fb2000640 1 -- 172.21.15.45:0/1284926896 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5fa804d020 con 0x7f5fb41038f0 2024-03-28T17:49:00.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.464+0000 7f5fbc000640 1 -- 172.21.15.45:0/1284926896 --> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) v1 -- 0x7f5fb4061b80 con 0x7f5f9c0a26b0 2024-03-28T17:49:00.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.550+0000 7f5fb2000640 1 -- 172.21.15.45:0/1284926896 <== mgr.25354 v2:172.21.15.45:6800/1309597278 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f5fb4061b80 con 0x7f5f9c0a26b0 2024-03-28T17:49:00.552 INFO:teuthology.orchestra.run.smithi045.stdout:Retention added to path / 2024-03-28T17:49:00.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.553+0000 7f5fbc000640 1 -- 172.21.15.45:0/1284926896 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f5f9c0a26b0 msgr2=0x7f5f9c0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:00.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.553+0000 7f5fbc000640 1 --2- 172.21.15.45:0/1284926896 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f5f9c0a26b0 0x7f5f9c0a4b70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5fb419b050 tx=0x7f5fa0005f90 comp rx=0 tx=0).stop 2024-03-28T17:49:00.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.553+0000 7f5fbc000640 1 -- 172.21.15.45:0/1284926896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fb41038f0 msgr2=0x7f5fb4199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:00.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.553+0000 7f5fbc000640 1 --2- 172.21.15.45:0/1284926896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fb41038f0 0x7f5fb4199b40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8000c00 tx=0x7f5fa802c650 comp rx=0 tx=0).stop 2024-03-28T17:49:00.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.553+0000 7f5fbc000640 1 -- 172.21.15.45:0/1284926896 shutdown_connections 2024-03-28T17:49:00.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.553+0000 7f5fbc000640 1 --2- 172.21.15.45:0/1284926896 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f5f9c0a26b0 0x7f5f9c0a4b70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:00.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.553+0000 7f5fbc000640 1 --2- 172.21.15.45:0/1284926896 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5fb41054a0 0x7f5fb41a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:00.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.553+0000 7f5fbc000640 1 --2- 172.21.15.45:0/1284926896 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5fb4104ae0 0x7f5fb419a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:00.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.553+0000 7f5fbc000640 1 --2- 172.21.15.45:0/1284926896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fb41038f0 0x7f5fb4199b40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:00.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.553+0000 7f5fbc000640 1 -- 172.21.15.45:0/1284926896 >> 172.21.15.45:0/1284926896 conn(0x7f5fb40fee60 msgr2=0x7f5fb4107de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:00.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.553+0000 7f5fbc000640 1 -- 172.21.15.45:0/1284926896 shutdown_connections 2024-03-28T17:49:00.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.553+0000 7f5fbc000640 1 -- 172.21.15.45:0/1284926896 wait complete. 2024-03-28T17:49:00.572 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-28T17:49:00.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.766+0000 7f827b000640 1 -- 172.21.15.45:0/2211833691 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8274104ae0 msgr2=0x7f8274104f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:00.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.766+0000 7f827b000640 1 --2- 172.21.15.45:0/2211833691 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8274104ae0 0x7f8274104f60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8260009860 tx=0x7f826002bb70 comp rx=0 tx=0).stop 2024-03-28T17:49:00.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.767+0000 7f827b000640 1 -- 172.21.15.45:0/2211833691 shutdown_connections 2024-03-28T17:49:00.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.767+0000 7f827b000640 1 --2- 172.21.15.45:0/2211833691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82741054a0 0x7f827410bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:00.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.767+0000 7f827b000640 1 --2- 172.21.15.45:0/2211833691 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8274104ae0 0x7f8274104f60 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:00.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.767+0000 7f827b000640 1 --2- 172.21.15.45:0/2211833691 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82741038f0 0x7f8274103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:00.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.767+0000 7f827b000640 1 -- 172.21.15.45:0/2211833691 >> 172.21.15.45:0/2211833691 conn(0x7f82740fee60 msgr2=0x7f82741012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:00.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.767+0000 7f827b000640 1 -- 172.21.15.45:0/2211833691 shutdown_connections 2024-03-28T17:49:00.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.767+0000 7f827b000640 1 -- 172.21.15.45:0/2211833691 wait complete. 2024-03-28T17:49:00.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.768+0000 7f827b000640 1 Processor -- start 2024-03-28T17:49:00.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.768+0000 7f827b000640 1 -- start start 2024-03-28T17:49:00.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.768+0000 7f827b000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82741038f0 0x7f8274199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:00.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.768+0000 7f827b000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8274104ae0 0x7f827419a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:00.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.768+0000 7f827b000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82741054a0 0x7f82741a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.768+0000 7f827b000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827410f570 con 0x7f82741038f0 2024-03-28T17:49:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.768+0000 7f827b000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827410f3f0 con 0x7f82741054a0 2024-03-28T17:49:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.768+0000 7f827b000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827410f6f0 con 0x7f8274104ae0 2024-03-28T17:49:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.768+0000 7f8273400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8274104ae0 0x7f827419a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.769+0000 7f8273e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82741038f0 0x7f8274199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.769+0000 7f8273400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8274104ae0 0x7f827419a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:58264/0 (socket says 172.21.15.45:58264) 2024-03-28T17:49:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.769+0000 7f8273400640 1 -- 172.21.15.45:0/325150620 learned_addr learned my addr 172.21.15.45:0/325150620 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.769+0000 7f8279000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82741054a0 0x7f82741a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.769+0000 7f8273400640 1 -- 172.21.15.45:0/325150620 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82741054a0 msgr2=0x7f82741a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.769+0000 7f8273400640 1 --2- 172.21.15.45:0/325150620 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82741054a0 0x7f82741a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.769+0000 7f8273400640 1 -- 172.21.15.45:0/325150620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82741038f0 msgr2=0x7f8274199b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.769+0000 7f8273400640 1 --2- 172.21.15.45:0/325150620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82741038f0 0x7f8274199b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.769+0000 7f8273400640 1 -- 172.21.15.45:0/325150620 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82741a1770 con 0x7f8274104ae0 2024-03-28T17:49:00.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.769+0000 7f8273400640 1 --2- 172.21.15.45:0/325150620 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8274104ae0 0x7f827419a090 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8260005940 tx=0x7f82600042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:00.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.770+0000 7f8270c00640 1 -- 172.21.15.45:0/325150620 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8260041020 con 0x7f8274104ae0 2024-03-28T17:49:00.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.770+0000 7f8270c00640 1 -- 172.21.15.45:0/325150620 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82600323f0 con 0x7f8274104ae0 2024-03-28T17:49:00.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.770+0000 7f827b000640 1 -- 172.21.15.45:0/325150620 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82741a1a00 con 0x7f8274104ae0 2024-03-28T17:49:00.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.770+0000 7f8270c00640 1 -- 172.21.15.45:0/325150620 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8260039040 con 0x7f8274104ae0 2024-03-28T17:49:00.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.770+0000 7f827b000640 1 -- 172.21.15.45:0/325150620 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82741a1ef0 con 0x7f8274104ae0 2024-03-28T17:49:00.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.771+0000 7f827b000640 1 -- 172.21.15.45:0/325150620 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f827410ac10 con 0x7f8274104ae0 2024-03-28T17:49:00.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.774+0000 7f8270c00640 1 -- 172.21.15.45:0/325150620 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 24) v1 ==== 137102+0+0 (secure 0 0 0) 0x7f8260044020 con 0x7f8274104ae0 2024-03-28T17:49:00.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.776+0000 7f8270c00640 1 --2- 172.21.15.45:0/325150620 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f82580a26b0 0x7f82580a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:00.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.776+0000 7f8270c00640 1 -- 172.21.15.45:0/325150620 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f82600f0190 con 0x7f8274104ae0 2024-03-28T17:49:00.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.776+0000 7f8273e00640 1 --2- 172.21.15.45:0/325150620 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f82580a26b0 0x7f82580a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:00.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.777+0000 7f8273e00640 1 --2- 172.21.15.45:0/325150620 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f82580a26b0 0x7f82580a4b70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8274104940 tx=0x7f8264009260 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:00.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:00.777+0000 7f8270c00640 1 -- 172.21.15.45:0/325150620 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f826004b020 con 0x7f8274104ae0 2024-03-28T17:49:01.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.024+0000 7f827b000640 1 -- 172.21.15.45:0/325150620 --> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f8274061ae0 con 0x7f82580a26b0 2024-03-28T17:49:01.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.027+0000 7f8270c00640 1 -- 172.21.15.45:0/325150620 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mgrmap(e 25) v1 ==== 137410+0+0 (secure 0 0 0) 0x7f82600b9360 con 0x7f8274104ae0 2024-03-28T17:49:01.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.036+0000 7f8270c00640 1 -- 172.21.15.45:0/325150620 <== mgr.25354 v2:172.21.15.45:6800/1309597278 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+298 (secure 0 0 0) 0x7f8274061ae0 con 0x7f82580a26b0 2024-03-28T17:49:01.037 INFO:teuthology.orchestra.run.smithi045.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-28T00:00:00", "created": "2024-03-28T17:48:59", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-28T17:49:01.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.039+0000 7f827b000640 1 -- 172.21.15.45:0/325150620 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f82580a26b0 msgr2=0x7f82580a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:01.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.039+0000 7f827b000640 1 --2- 172.21.15.45:0/325150620 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f82580a26b0 0x7f82580a4b70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8274104940 tx=0x7f8264009260 comp rx=0 tx=0).stop 2024-03-28T17:49:01.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.039+0000 7f827b000640 1 -- 172.21.15.45:0/325150620 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8274104ae0 msgr2=0x7f827419a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:01.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.039+0000 7f827b000640 1 --2- 172.21.15.45:0/325150620 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8274104ae0 0x7f827419a090 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8260005940 tx=0x7f82600042f0 comp rx=0 tx=0).stop 2024-03-28T17:49:01.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.039+0000 7f827b000640 1 -- 172.21.15.45:0/325150620 shutdown_connections 2024-03-28T17:49:01.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.039+0000 7f827b000640 1 --2- 172.21.15.45:0/325150620 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f82580a26b0 0x7f82580a4b70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:01.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.039+0000 7f827b000640 1 --2- 172.21.15.45:0/325150620 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82741054a0 0x7f82741a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:01.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.039+0000 7f827b000640 1 --2- 172.21.15.45:0/325150620 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8274104ae0 0x7f827419a090 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:01.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.039+0000 7f827b000640 1 --2- 172.21.15.45:0/325150620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82741038f0 0x7f8274199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.039+0000 7f827b000640 1 -- 172.21.15.45:0/325150620 >> 172.21.15.45:0/325150620 conn(0x7f82740fee60 msgr2=0x7f8274107de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.039+0000 7f827b000640 1 -- 172.21.15.45:0/325150620 shutdown_connections 2024-03-28T17:49:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.039+0000 7f827b000640 1 -- 172.21.15.45:0/325150620 wait complete. 2024-03-28T17:49:01.057 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-28T17:49:01.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:00 smithi045 ceph-mon[31773]: mgrmap e24: x(active, since 1.03783s), standbys: z, y 2024-03-28T17:49:01.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:00 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:01.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:00 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:01.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:00 smithi045 ceph-mon[31773]: [28/Mar/2024:17:49:00] ENGINE Bus STARTING 2024-03-28T17:49:01.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:00 smithi045 ceph-mon[31773]: [28/Mar/2024:17:49:00] ENGINE Serving on https://172.21.15.45:7150 2024-03-28T17:49:01.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:00 smithi045 ceph-mon[31773]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:49:01.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:00 smithi045 ceph-mon[31773]: [28/Mar/2024:17:49:00] ENGINE Serving on http://172.21.15.45:8765 2024-03-28T17:49:01.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:00 smithi045 ceph-mon[31773]: [28/Mar/2024:17:49:00] ENGINE Bus STARTED 2024-03-28T17:49:01.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:00 smithi045 ceph-mon[31773]: scrub summary: idle 2024-03-28T17:49:01.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.254+0000 7f4eaec00640 1 -- 172.21.15.45:0/3252611817 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ea8104910 msgr2=0x7f4ea8104d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:01.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.254+0000 7f4eaec00640 1 --2- 172.21.15.45:0/3252611817 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ea8104910 0x7f4ea8104d90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4e94009860 tx=0x7f4e9402bb70 comp rx=0 tx=0).stop 2024-03-28T17:49:01.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.254+0000 7f4eaec00640 1 -- 172.21.15.45:0/3252611817 shutdown_connections 2024-03-28T17:49:01.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.254+0000 7f4eaec00640 1 --2- 172.21.15.45:0/3252611817 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ea81052d0 0x7f4ea810bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:01.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.254+0000 7f4eaec00640 1 --2- 172.21.15.45:0/3252611817 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ea8104910 0x7f4ea8104d90 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:01.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.254+0000 7f4eaec00640 1 --2- 172.21.15.45:0/3252611817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ea8103720 0x7f4ea8103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:01.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.254+0000 7f4eaec00640 1 -- 172.21.15.45:0/3252611817 >> 172.21.15.45:0/3252611817 conn(0x7f4ea80fecb0 msgr2=0x7f4ea81010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:01.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.254+0000 7f4eaec00640 1 -- 172.21.15.45:0/3252611817 shutdown_connections 2024-03-28T17:49:01.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.254+0000 7f4eaec00640 1 -- 172.21.15.45:0/3252611817 wait complete. 2024-03-28T17:49:01.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.255+0000 7f4eaec00640 1 Processor -- start 2024-03-28T17:49:01.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.255+0000 7f4eaec00640 1 -- start start 2024-03-28T17:49:01.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.256+0000 7f4eaec00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ea8103720 0x7f4ea81999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:01.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.256+0000 7f4eaec00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ea8104910 0x7f4ea8199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:01.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.256+0000 7f4eaec00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ea81052d0 0x7f4ea81a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:01.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.256+0000 7f4eaec00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea810f380 con 0x7f4ea81052d0 2024-03-28T17:49:01.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.256+0000 7f4eaec00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea810f200 con 0x7f4ea8103720 2024-03-28T17:49:01.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.256+0000 7f4eaec00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea810f500 con 0x7f4ea8104910 2024-03-28T17:49:01.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.256+0000 7f4ead800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ea8103720 0x7f4ea81999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:01.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.256+0000 7f4ead800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ea8103720 0x7f4ea81999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:36218/0 (socket says 172.21.15.45:36218) 2024-03-28T17:49:01.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.256+0000 7f4eace00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ea8104910 0x7f4ea8199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:01.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.256+0000 7f4ead800640 1 -- 172.21.15.45:0/914882847 learned_addr learned my addr 172.21.15.45:0/914882847 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:01.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.256+0000 7f4eae200640 1 --2- 172.21.15.45:0/914882847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ea81052d0 0x7f4ea81a0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:01.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.256+0000 7f4ead800640 1 -- 172.21.15.45:0/914882847 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ea8104910 msgr2=0x7f4ea8199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:01.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.256+0000 7f4ead800640 1 --2- 172.21.15.45:0/914882847 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ea8104910 0x7f4ea8199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:01.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.256+0000 7f4ead800640 1 -- 172.21.15.45:0/914882847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ea81052d0 msgr2=0x7f4ea81a0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:01.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.256+0000 7f4ead800640 1 --2- 172.21.15.45:0/914882847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ea81052d0 0x7f4ea81a0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:01.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.256+0000 7f4ead800640 1 -- 172.21.15.45:0/914882847 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea81a15e0 con 0x7f4ea8103720 2024-03-28T17:49:01.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.257+0000 7f4ead800640 1 --2- 172.21.15.45:0/914882847 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ea8103720 0x7f4ea81999c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00e9c0 tx=0x7f4e9c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:01.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.258+0000 7f4ea6000640 1 -- 172.21.15.45:0/914882847 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c00cde0 con 0x7f4ea8103720 2024-03-28T17:49:01.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.258+0000 7f4ea6000640 1 -- 172.21.15.45:0/914882847 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e9c010910 con 0x7f4ea8103720 2024-03-28T17:49:01.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.258+0000 7f4ea6000640 1 -- 172.21.15.45:0/914882847 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c018a00 con 0x7f4ea8103720 2024-03-28T17:49:01.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.258+0000 7f4eaec00640 1 -- 172.21.15.45:0/914882847 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea81a18d0 con 0x7f4ea8103720 2024-03-28T17:49:01.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.258+0000 7f4eaec00640 1 -- 172.21.15.45:0/914882847 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea81a1da0 con 0x7f4ea8103720 2024-03-28T17:49:01.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.259+0000 7f4eaec00640 1 -- 172.21.15.45:0/914882847 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea810aa40 con 0x7f4ea8103720 2024-03-28T17:49:01.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.262+0000 7f4ea6000640 1 -- 172.21.15.45:0/914882847 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 25) v1 ==== 137410+0+0 (secure 0 0 0) 0x7f4e9c01f020 con 0x7f4ea8103720 2024-03-28T17:49:01.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.263+0000 7f4ea6000640 1 --2- 172.21.15.45:0/914882847 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f4e900a29e0 0x7f4e900a4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:01.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.263+0000 7f4ea6000640 1 -- 172.21.15.45:0/914882847 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f4e9c0cf090 con 0x7f4ea8103720 2024-03-28T17:49:01.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.264+0000 7f4eace00640 1 --2- 172.21.15.45:0/914882847 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f4e900a29e0 0x7f4e900a4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:01.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.264+0000 7f4eace00640 1 --2- 172.21.15.45:0/914882847 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f4e900a29e0 0x7f4e900a4ea0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4e94009830 tx=0x7f4e94034ab0 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:01.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.267+0000 7f4ea6000640 1 -- 172.21.15.45:0/914882847 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4e9c09bdc0 con 0x7f4ea8103720 2024-03-28T17:49:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:00 smithi071 ceph-mon[36597]: mgrmap e24: x(active, since 1.03783s), standbys: z, y 2024-03-28T17:49:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:00 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:00 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:00 smithi071 ceph-mon[36597]: [28/Mar/2024:17:49:00] ENGINE Bus STARTING 2024-03-28T17:49:01.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:00 smithi071 ceph-mon[36597]: [28/Mar/2024:17:49:00] ENGINE Serving on https://172.21.15.45:7150 2024-03-28T17:49:01.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:00 smithi071 ceph-mon[36597]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:49:01.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:00 smithi071 ceph-mon[36597]: [28/Mar/2024:17:49:00] ENGINE Serving on http://172.21.15.45:8765 2024-03-28T17:49:01.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:00 smithi071 ceph-mon[36597]: [28/Mar/2024:17:49:00] ENGINE Bus STARTED 2024-03-28T17:49:01.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:00 smithi071 ceph-mon[36597]: scrub summary: idle 2024-03-28T17:49:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:00 smithi151 ceph-mon[36481]: mgrmap e24: x(active, since 1.03783s), standbys: z, y 2024-03-28T17:49:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:00 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:00 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:01.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:00 smithi151 ceph-mon[36481]: [28/Mar/2024:17:49:00] ENGINE Bus STARTING 2024-03-28T17:49:01.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:00 smithi151 ceph-mon[36481]: [28/Mar/2024:17:49:00] ENGINE Serving on https://172.21.15.45:7150 2024-03-28T17:49:01.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:00 smithi151 ceph-mon[36481]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:49:01.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:00 smithi151 ceph-mon[36481]: [28/Mar/2024:17:49:00] ENGINE Serving on http://172.21.15.45:8765 2024-03-28T17:49:01.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:00 smithi151 ceph-mon[36481]: [28/Mar/2024:17:49:00] ENGINE Bus STARTED 2024-03-28T17:49:01.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:00 smithi151 ceph-mon[36481]: scrub summary: idle 2024-03-28T17:49:01.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.513+0000 7f4eaec00640 1 -- 172.21.15.45:0/914882847 --> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f4ea8061ae0 con 0x7f4e900a29e0 2024-03-28T17:49:01.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.524+0000 7f4ea6000640 1 -- 172.21.15.45:0/914882847 <== mgr.25354 v2:172.21.15.45:6800/1309597278 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f4ea8061ae0 con 0x7f4e900a29e0 2024-03-28T17:49:01.526 INFO:teuthology.orchestra.run.smithi045.stdout:/ 1m 6m3h 2024-03-28T17:49:01.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.527+0000 7f4eaec00640 1 -- 172.21.15.45:0/914882847 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f4e900a29e0 msgr2=0x7f4e900a4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:01.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.527+0000 7f4eaec00640 1 --2- 172.21.15.45:0/914882847 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f4e900a29e0 0x7f4e900a4ea0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4e94009830 tx=0x7f4e94034ab0 comp rx=0 tx=0).stop 2024-03-28T17:49:01.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.527+0000 7f4eaec00640 1 -- 172.21.15.45:0/914882847 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ea8103720 msgr2=0x7f4ea81999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:01.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.527+0000 7f4eaec00640 1 --2- 172.21.15.45:0/914882847 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ea8103720 0x7f4ea81999c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00e9c0 tx=0x7f4e9c00ee80 comp rx=0 tx=0).stop 2024-03-28T17:49:01.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.527+0000 7f4eaec00640 1 -- 172.21.15.45:0/914882847 shutdown_connections 2024-03-28T17:49:01.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.527+0000 7f4eaec00640 1 --2- 172.21.15.45:0/914882847 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f4e900a29e0 0x7f4e900a4ea0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:01.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.527+0000 7f4eaec00640 1 --2- 172.21.15.45:0/914882847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ea81052d0 0x7f4ea81a0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:01.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.527+0000 7f4eaec00640 1 --2- 172.21.15.45:0/914882847 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ea8104910 0x7f4ea8199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:01.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.527+0000 7f4eaec00640 1 --2- 172.21.15.45:0/914882847 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ea8103720 0x7f4ea81999c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:01.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.527+0000 7f4eaec00640 1 -- 172.21.15.45:0/914882847 >> 172.21.15.45:0/914882847 conn(0x7f4ea80fecb0 msgr2=0x7f4ea8107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:01.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.527+0000 7f4eaec00640 1 -- 172.21.15.45:0/914882847 shutdown_connections 2024-03-28T17:49:01.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:01.527+0000 7f4eaec00640 1 -- 172.21.15.45:0/914882847 wait complete. 2024-03-28T17:49:01.547 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-28T17:49:01.618 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-28T17:49:01.628 INFO:teuthology.run_tasks:Running task workunit... 2024-03-28T17:49:01.673 INFO:tasks.workunit:Pulling workunits from ref 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:49:01.674 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-28T17:49:01.674 INFO:tasks.workunit:timeout=3h 2024-03-28T17:49:01.674 INFO:tasks.workunit:cleanup=True 2024-03-28T17:49:01.674 DEBUG:teuthology.orchestra.run.smithi045:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-28T17:49:01.715 INFO:teuthology.orchestra.run.smithi045.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-28T17:49:01.715 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-28T17:49:01.715 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 76h/118d Inode: 1099511628283 Links: 2 2024-03-28T17:49:01.715 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T17:49:01.715 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-28T17:49:01.715 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-03-28 17:46:20.037393791 +0000 2024-03-28T17:49:01.715 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-03-28 17:46:20.037393791 +0000 2024-03-28T17:49:01.715 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-03-28 17:48:37.921328662 +0000 2024-03-28T17:49:01.715 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-03-28 17:46:20.037393791 +0000 2024-03-28T17:49:01.716 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-28T17:49:01.716 DEBUG:teuthology.orchestra.run.smithi045:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-28T17:49:01.833 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T17:49:01.876 INFO:tasks.workunit.client.0.smithi045.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-28T17:49:02.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:02 smithi045 ceph-mon[31773]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:49:02.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:02 smithi045 ceph-mon[31773]: from='client.25420 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:49:02.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:02 smithi045 ceph-mon[31773]: mgrmap e25: x(active, since 2s), standbys: z, y 2024-03-28T17:49:02.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:02 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:02.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:02 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:02.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:02 smithi045 ceph-mon[31773]: from='client.25457 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:49:02.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:02 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:02.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:02 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:02.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:02 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:02.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:02 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:02 smithi071 ceph-mon[36597]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:49:02.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:02 smithi071 ceph-mon[36597]: from='client.25420 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:49:02.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:02 smithi071 ceph-mon[36597]: mgrmap e25: x(active, since 2s), standbys: z, y 2024-03-28T17:49:02.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:02 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:02.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:02 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:02.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:02 smithi071 ceph-mon[36597]: from='client.25457 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:49:02.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:02 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:02.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:02 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:02.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:02 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:02.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:02 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:02 smithi151 ceph-mon[36481]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:49:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:02 smithi151 ceph-mon[36481]: from='client.25420 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:49:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:02 smithi151 ceph-mon[36481]: mgrmap e25: x(active, since 2s), standbys: z, y 2024-03-28T17:49:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:02 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:02 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:02 smithi151 ceph-mon[36481]: from='client.25457 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T17:49:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:02 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:02 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:02.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:02 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:02.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:02 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:03.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:03 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:03.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:03 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:03.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:03 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:03 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:03 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:03 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:03 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:03 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:03 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:03 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:03 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T17:49:03.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:03 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T17:49:03.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:03 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:03.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:03 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:03.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:03 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:03 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:03 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:03 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:03 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:03 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:03 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:03 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:03 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:03.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:03 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:03.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:03 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:03 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:03 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:03 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:03 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:03 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:03 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:03 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:03 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T17:49:03.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:03 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T17:49:03.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:03 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:03.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:03 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:03.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:03 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:03 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:03 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:03 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:03 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:03 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:03 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:03 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:03 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:03.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:03 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:03.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:03 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:03 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:03 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:03 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:03 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:03 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:03 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:03 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:03 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T17:49:03.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:03 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T17:49:03.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:03 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:03.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:03 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:03.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:03 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:03 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:03 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:03 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:03 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:03 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:03 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:03.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:03 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.544 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:04 smithi071 ceph-mon[36597]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:49:04.544 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:04 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T17:49:04.544 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:04 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T17:49:04.545 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:04 smithi071 ceph-mon[36597]: mgrmap e26: x(active, since 4s), standbys: z, y 2024-03-28T17:49:04.545 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:04 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:04.545 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:04 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:04.545 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:04 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.545 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:04 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.545 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:04 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.545 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:04 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.545 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:04 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.545 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:04 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.545 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:04 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.545 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:04 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.545 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:04 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:49:04.545 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:04 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:49:04.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:04 smithi151 ceph-mon[36481]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:49:04.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:04 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T17:49:04.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:04 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T17:49:04.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:04 smithi151 ceph-mon[36481]: mgrmap e26: x(active, since 4s), standbys: z, y 2024-03-28T17:49:04.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:04 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:04.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:04 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:04.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:04 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:04 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:04 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:04 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:04 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:04 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:04 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:04 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:04 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:49:04.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:04 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:49:04.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:04 smithi045 ceph-mon[31773]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:49:04.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:04 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T17:49:04.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:04 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T17:49:04.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:04 smithi045 ceph-mon[31773]: mgrmap e26: x(active, since 4s), standbys: z, y 2024-03-28T17:49:04.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:04 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:04.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:04 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:04.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:04 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:04 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:04 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:04 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:04 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:04 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:04 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:04 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:49:04.919 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:04 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:49:04.919 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:04 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:49:05.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:05 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T17:49:05.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:05 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T17:49:05.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:05 smithi071 ceph-mon[36597]: Updating smithi045:/etc/ceph/ceph.conf 2024-03-28T17:49:05.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:05 smithi071 ceph-mon[36597]: Updating smithi071:/etc/ceph/ceph.conf 2024-03-28T17:49:05.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:05 smithi071 ceph-mon[36597]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-28T17:49:05.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:05 smithi071 ceph-mon[36597]: Updating smithi151:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:49:05.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:05 smithi071 ceph-mon[36597]: Updating smithi071:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:49:05.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:05 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:05.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:05 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:05.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:05 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:05.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:05 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:05 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T17:49:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:05 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T17:49:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:05 smithi151 ceph-mon[36481]: Updating smithi045:/etc/ceph/ceph.conf 2024-03-28T17:49:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:05 smithi151 ceph-mon[36481]: Updating smithi071:/etc/ceph/ceph.conf 2024-03-28T17:49:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:05 smithi151 ceph-mon[36481]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-28T17:49:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:05 smithi151 ceph-mon[36481]: Updating smithi151:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:49:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:05 smithi151 ceph-mon[36481]: Updating smithi071:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:49:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:05 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:05 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:05 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:05 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:05.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:05 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T17:49:05.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:05 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T17:49:05.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:05 smithi045 ceph-mon[31773]: Updating smithi045:/etc/ceph/ceph.conf 2024-03-28T17:49:05.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:05 smithi045 ceph-mon[31773]: Updating smithi071:/etc/ceph/ceph.conf 2024-03-28T17:49:05.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:05 smithi045 ceph-mon[31773]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-28T17:49:05.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:05 smithi045 ceph-mon[31773]: Updating smithi151:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:49:05.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:05 smithi045 ceph-mon[31773]: Updating smithi071:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:49:05.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:05 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:05.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:05 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:05.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:05 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:05.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:05 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:06 smithi071 ceph-mon[36597]: Updating smithi071:/etc/ceph/ceph.client.admin.keyring 2024-03-28T17:49:06.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:06 smithi071 ceph-mon[36597]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-03-28T17:49:06.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:06 smithi071 ceph-mon[36597]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:49:06.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:06 smithi071 ceph-mon[36597]: Updating smithi045:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:49:06.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:06 smithi071 ceph-mon[36597]: Updating smithi071:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T17:49:06.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:06 smithi071 ceph-mon[36597]: Updating smithi151:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T17:49:06.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:06 smithi071 ceph-mon[36597]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-03-28T17:49:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:06 smithi151 ceph-mon[36481]: Updating smithi071:/etc/ceph/ceph.client.admin.keyring 2024-03-28T17:49:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:06 smithi151 ceph-mon[36481]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-03-28T17:49:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:06 smithi151 ceph-mon[36481]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:49:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:06 smithi151 ceph-mon[36481]: Updating smithi045:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:49:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:06 smithi151 ceph-mon[36481]: Updating smithi071:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T17:49:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:06 smithi151 ceph-mon[36481]: Updating smithi151:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T17:49:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:06 smithi151 ceph-mon[36481]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-03-28T17:49:06.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:06 smithi045 ceph-mon[31773]: Updating smithi071:/etc/ceph/ceph.client.admin.keyring 2024-03-28T17:49:06.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:06 smithi045 ceph-mon[31773]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-03-28T17:49:06.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:06 smithi045 ceph-mon[31773]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T17:49:06.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:06 smithi045 ceph-mon[31773]: Updating smithi045:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T17:49:06.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:06 smithi045 ceph-mon[31773]: Updating smithi071:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T17:49:06.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:06 smithi045 ceph-mon[31773]: Updating smithi151:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T17:49:06.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:06 smithi045 ceph-mon[31773]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-03-28T17:49:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:07 smithi071 ceph-mon[36597]: Updating smithi045:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T17:49:07.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:07 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:07.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:07 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:07.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:07 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:07.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:07 smithi151 ceph-mon[36481]: Updating smithi045:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T17:49:07.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:07 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:07.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:07 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:07.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:07 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:07.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:07 smithi045 ceph-mon[31773]: Updating smithi045:/var/lib/ceph/9f4888ec-ed29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T17:49:07.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:07 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:07.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:07 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:07.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:07 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:08.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:08 smithi071 ceph-mon[36597]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-03-28T17:49:08.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:08 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:49:08.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:08 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:49:08.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:08 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:49:08.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:08 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:08 smithi151 ceph-mon[36481]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-03-28T17:49:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:08 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:49:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:08 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:49:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:08 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:49:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:08 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:08.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:08 smithi045 ceph-mon[31773]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-03-28T17:49:08.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:08 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:49:08.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:08 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:49:08.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:08 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:49:08.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:08 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:49:10.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:10 smithi071 ceph-mon[36597]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 103 KiB/s wr, 60 op/s 2024-03-28T17:49:10.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:10 smithi151 ceph-mon[36481]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 103 KiB/s wr, 60 op/s 2024-03-28T17:49:10.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:10 smithi045 ceph-mon[31773]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 103 KiB/s wr, 60 op/s 2024-03-28T17:49:12.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:12 smithi151 ceph-mon[36481]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-03-28T17:49:12.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:12 smithi045 ceph-mon[31773]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-03-28T17:49:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:12 smithi071 ceph-mon[36597]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-03-28T17:49:14.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:14 smithi151 ceph-mon[36481]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-28T17:49:14.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:14 smithi045 ceph-mon[31773]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-28T17:49:15.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:14 smithi071 ceph-mon[36597]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-28T17:49:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:16 smithi045 ceph-mon[31773]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-03-28T17:49:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:16 smithi071 ceph-mon[36597]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-03-28T17:49:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:16 smithi151 ceph-mon[36481]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-03-28T17:49:18.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:18 smithi045 ceph-mon[31773]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-28T17:49:19.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:18 smithi071 ceph-mon[36597]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-28T17:49:19.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:18 smithi151 ceph-mon[36481]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-28T17:49:19.302 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:49:19.302 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:49:19.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.545+0000 7f869f200640 1 -- 172.21.15.45:0/3053597306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86a01040e0 msgr2=0x7f86a01105f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:19.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.545+0000 7f869f200640 1 --2- 172.21.15.45:0/3053597306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86a01040e0 0x7f86a01105f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8698009ca0 tx=0x7f869802c100 comp rx=0 tx=0).stop 2024-03-28T17:49:19.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.545+0000 7f869f200640 1 -- 172.21.15.45:0/3053597306 shutdown_connections 2024-03-28T17:49:19.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.545+0000 7f869f200640 1 --2- 172.21.15.45:0/3053597306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86a01040e0 0x7f86a01105f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.545+0000 7f869f200640 1 --2- 172.21.15.45:0/3053597306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86a0103720 0x7f86a0103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.545+0000 7f869f200640 1 --2- 172.21.15.45:0/3053597306 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86a0109730 0x7f86a0109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.545+0000 7f869f200640 1 -- 172.21.15.45:0/3053597306 >> 172.21.15.45:0/3053597306 conn(0x7f86a00ff1d0 msgr2=0x7f86a01015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:19.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.546+0000 7f869f200640 1 -- 172.21.15.45:0/3053597306 shutdown_connections 2024-03-28T17:49:19.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.546+0000 7f869f200640 1 -- 172.21.15.45:0/3053597306 wait complete. 2024-03-28T17:49:19.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.547+0000 7f869f200640 1 Processor -- start 2024-03-28T17:49:19.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.547+0000 7f869f200640 1 -- start start 2024-03-28T17:49:19.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.547+0000 7f869f200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86a0103720 0x7f86a019a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:19.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.547+0000 7f869f200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86a01040e0 0x7f86a019a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:19.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.547+0000 7f869f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86a0109730 0x7f86a019e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:19.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.547+0000 7f869f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a0113b40 con 0x7f86a0109730 2024-03-28T17:49:19.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.547+0000 7f869f200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a01139c0 con 0x7f86a0103720 2024-03-28T17:49:19.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.547+0000 7f869f200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a0113cc0 con 0x7f86a01040e0 2024-03-28T17:49:19.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.548+0000 7f8697e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86a01040e0 0x7f86a019a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:19.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.548+0000 7f869de00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86a0103720 0x7f86a019a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:19.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.548+0000 7f8697e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86a01040e0 0x7f86a019a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:56450/0 (socket says 172.21.15.45:56450) 2024-03-28T17:49:19.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.548+0000 7f8697e00640 1 -- 172.21.15.45:0/3694495514 learned_addr learned my addr 172.21.15.45:0/3694495514 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:19.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.548+0000 7f869e800640 1 --2- 172.21.15.45:0/3694495514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86a0109730 0x7f86a019e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:19.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.548+0000 7f869de00640 1 -- 172.21.15.45:0/3694495514 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86a01040e0 msgr2=0x7f86a019a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:19.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.548+0000 7f869de00640 1 --2- 172.21.15.45:0/3694495514 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86a01040e0 0x7f86a019a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.548+0000 7f869de00640 1 -- 172.21.15.45:0/3694495514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86a0109730 msgr2=0x7f86a019e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:19.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.548+0000 7f869de00640 1 --2- 172.21.15.45:0/3694495514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86a0109730 0x7f86a019e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.548+0000 7f869de00640 1 -- 172.21.15.45:0/3694495514 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a019efa0 con 0x7f86a0103720 2024-03-28T17:49:19.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.548+0000 7f869de00640 1 --2- 172.21.15.45:0/3694495514 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86a0103720 0x7f86a019a000 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f869000e9f0 tx=0x7f869000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:19.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.548+0000 7f8696a00640 1 -- 172.21.15.45:0/3694495514 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869000ccc0 con 0x7f86a0103720 2024-03-28T17:49:19.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.549+0000 7f8696a00640 1 -- 172.21.15.45:0/3694495514 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8690010910 con 0x7f86a0103720 2024-03-28T17:49:19.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.549+0000 7f869f200640 1 -- 172.21.15.45:0/3694495514 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a019f290 con 0x7f86a0103720 2024-03-28T17:49:19.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.549+0000 7f8696a00640 1 -- 172.21.15.45:0/3694495514 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8690018960 con 0x7f86a0103720 2024-03-28T17:49:19.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.549+0000 7f869f200640 1 -- 172.21.15.45:0/3694495514 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a01a6b20 con 0x7f86a0103720 2024-03-28T17:49:19.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.552+0000 7f8696a00640 1 -- 172.21.15.45:0/3694495514 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 26) v1 ==== 137466+0+0 (secure 0 0 0) 0x7f8690010ac0 con 0x7f86a0103720 2024-03-28T17:49:19.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.552+0000 7f86a7d64740 1 Processor -- start 2024-03-28T17:49:19.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.552+0000 7f86a7d64740 1 -- start start 2024-03-28T17:49:19.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.553+0000 7f86a7d64740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9ae0c2dd0 0x55f9ae0c1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:19.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.553+0000 7f86a7d64740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0c20f0 0x55f9ae0c2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:19.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.553+0000 7f86a7d64740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0d3c70 0x55f9ae0d6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:19.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.553+0000 7f86a7d64740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9ae026250 con 0x55f9ae0c2dd0 2024-03-28T17:49:19.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.553+0000 7f86a7d64740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9adfcea30 con 0x55f9ae0d3c70 2024-03-28T17:49:19.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.553+0000 7f86a7d64740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9ae0222f0 con 0x55f9ae0c20f0 2024-03-28T17:49:19.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.553+0000 7f869de00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0c20f0 0x55f9ae0c2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:19.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.553+0000 7f869e800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9ae0c2dd0 0x55f9ae0c1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:19.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.553+0000 7f8697e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0d3c70 0x55f9ae0d6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:19.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.553+0000 7f8697e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0d3c70 0x55f9ae0d6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40398/0 (socket says 172.21.15.45:40398) 2024-03-28T17:49:19.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.553+0000 7f8697e00640 1 -- 172.21.15.45:0/2775723869 learned_addr learned my addr 172.21.15.45:0/2775723869 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:19.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.553+0000 7f8697e00640 1 -- 172.21.15.45:0/2775723869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0c20f0 msgr2=0x55f9ae0c2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:19.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.553+0000 7f8697e00640 1 --2- 172.21.15.45:0/2775723869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0c20f0 0x55f9ae0c2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.553+0000 7f8697e00640 1 -- 172.21.15.45:0/2775723869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9ae0c2dd0 msgr2=0x55f9ae0c1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:19.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.553+0000 7f8697e00640 1 --2- 172.21.15.45:0/2775723869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9ae0c2dd0 0x55f9ae0c1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.553+0000 7f8697e00640 1 -- 172.21.15.45:0/2775723869 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9adfb9690 con 0x55f9ae0d3c70 2024-03-28T17:49:19.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.554+0000 7f8696a00640 1 --2- 172.21.15.45:0/3694495514 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f86840a2a80 0x7f86840a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:19.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.554+0000 7f8697e00640 1 --2- 172.21.15.45:0/2775723869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0d3c70 0x55f9ae0d6060 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8688004630 tx=0x7f868800bc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea941cec19a0e2f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:19.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.554+0000 7f8696a00640 1 -- 172.21.15.45:0/3694495514 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f86900cf390 con 0x7f86a0103720 2024-03-28T17:49:19.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.554+0000 7f868f400640 1 -- 172.21.15.45:0/2775723869 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8688003dd0 con 0x55f9ae0d3c70 2024-03-28T17:49:19.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.554+0000 7f868f400640 1 -- 172.21.15.45:0/2775723869 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8688015070 con 0x55f9ae0d3c70 2024-03-28T17:49:19.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.554+0000 7f869e800640 1 --2- 172.21.15.45:0/3694495514 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f86840a2a80 0x7f86840a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:19.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.554+0000 7f868f400640 1 -- 172.21.15.45:0/2775723869 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8688011750 con 0x55f9ae0d3c70 2024-03-28T17:49:19.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.555+0000 7f86a7d64740 1 -- 172.21.15.45:0/2775723869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0d3c70 msgr2=0x55f9ae0d6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:19.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.555+0000 7f86a7d64740 1 --2- 172.21.15.45:0/2775723869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0d3c70 0x55f9ae0d6060 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8688004630 tx=0x7f868800bc70 comp rx=0 tx=0).stop 2024-03-28T17:49:19.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.555+0000 7f86a7d64740 1 -- 172.21.15.45:0/2775723869 shutdown_connections 2024-03-28T17:49:19.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.555+0000 7f86a7d64740 1 --2- 172.21.15.45:0/2775723869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0d3c70 0x55f9ae0d6060 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.555+0000 7f86a7d64740 1 --2- 172.21.15.45:0/2775723869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0c20f0 0x55f9ae0c2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.555+0000 7f86a7d64740 1 --2- 172.21.15.45:0/2775723869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9ae0c2dd0 0x55f9ae0c1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.555+0000 7f86a7d64740 1 -- 172.21.15.45:0/2775723869 >> 172.21.15.45:0/2775723869 conn(0x55f9ae089200 msgr2=0x55f9ae0bcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:19.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.555+0000 7f869e800640 1 --2- 172.21.15.45:0/3694495514 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f86840a2a80 0x7f86840a4f40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8698006010 tx=0x7f8698004a40 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:19.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.555+0000 7f86a7d64740 1 -- 172.21.15.45:0/2775723869 shutdown_connections 2024-03-28T17:49:19.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.555+0000 7f86a7d64740 1 -- 172.21.15.45:0/2775723869 wait complete. 2024-03-28T17:49:19.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.555+0000 7f86a7d64740 1 -- start start 2024-03-28T17:49:19.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.556+0000 7f86a7d64740 1 Processor -- start 2024-03-28T17:49:19.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.556+0000 7f86a7d64740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0c20f0 0x55f9ae0d3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:19.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.556+0000 7f86a7d64740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9ae0c2dd0 0x55f9ae0d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:19.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.556+0000 7f86a7d64740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0d3c70 0x55f9ae0d0c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:19.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.556+0000 7f86a7d64740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9ae0d6ae0 con 0x55f9ae0c2dd0 2024-03-28T17:49:19.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.556+0000 7f86a7d64740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9adfe1e80 con 0x55f9ae0d3c70 2024-03-28T17:49:19.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.556+0000 7f86a7d64740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9ae0d6c60 con 0x55f9ae0c20f0 2024-03-28T17:49:19.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.556+0000 7f86a7d64740 10 client.0 _ll_register_callbacks cb 0x55f9adff4130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:19.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.556+0000 7f869de00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0c20f0 0x55f9ae0d3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:19.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.556+0000 7f869e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0d3c70 0x55f9ae0d0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:19.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.556+0000 7f869de00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0c20f0 0x55f9ae0d3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:56472/0 (socket says 172.21.15.45:56472) 2024-03-28T17:49:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.556+0000 7f869de00640 1 -- 172.21.15.45:0/2348001149 learned_addr learned my addr 172.21.15.45:0/2348001149 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.557+0000 7f8697e00640 1 --2- 172.21.15.45:0/2348001149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9ae0c2dd0 0x55f9ae0d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.557+0000 7f869de00640 1 -- 172.21.15.45:0/2348001149 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0d3c70 msgr2=0x55f9ae0d0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.557+0000 7f869de00640 1 --2- 172.21.15.45:0/2348001149 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0d3c70 0x55f9ae0d0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.557+0000 7f869de00640 1 -- 172.21.15.45:0/2348001149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9ae0c2dd0 msgr2=0x55f9ae0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.557+0000 7f869de00640 1 --2- 172.21.15.45:0/2348001149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9ae0c2dd0 0x55f9ae0d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.557+0000 7f869de00640 1 -- 172.21.15.45:0/2348001149 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f9ade047d0 con 0x55f9ae0c20f0 2024-03-28T17:49:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.557+0000 7f8697e00640 1 --2- 172.21.15.45:0/2348001149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9ae0c2dd0 0x55f9ae0d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:49:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.557+0000 7f869de00640 1 --2- 172.21.15.45:0/2348001149 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0c20f0 0x55f9ae0d3350 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f869000e9c0 tx=0x7f8690010ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd36cb2d7bd2770d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.557+0000 7f8683e00640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:49:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.557+0000 7f8683e00640 1 -- 172.21.15.45:0/2348001149 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869009b390 con 0x55f9ae0c20f0 2024-03-28T17:49:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.557+0000 7f86a7d64740 1 -- 172.21.15.45:0/2348001149 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8688015070 con 0x55f9ae0c20f0 2024-03-28T17:49:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.557+0000 7f8683e00640 1 -- 172.21.15.45:0/2348001149 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f86900ccea0 con 0x55f9ae0c20f0 2024-03-28T17:49:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.557+0000 7f8683e00640 1 -- 172.21.15.45:0/2348001149 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f869009b1b0 con 0x55f9ae0c20f0 2024-03-28T17:49:19.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.557+0000 7f86a7d64740 10 client.25444 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:19.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.558+0000 7f86a7d64740 1 -- 172.21.15.45:0/2348001149 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f9add3a3f0 con 0x55f9ae0c20f0 2024-03-28T17:49:19.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.558+0000 7f8683e00640 1 -- 172.21.15.45:0/2348001149 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8688015070 con 0x55f9ae0c20f0 2024-03-28T17:49:19.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.558+0000 7f8683e00640 1 -- 172.21.15.45:0/2348001149 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f869009adc0 con 0x55f9ae0c20f0 2024-03-28T17:49:19.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.558+0000 7f86a7d64740 10 client.25444 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:19.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.558+0000 7f86a7d64740 10 client.25444 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:49:19.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.558+0000 7f86a7d64740 20 client.25444 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:19.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.558+0000 7f86a7d64740 1 --2- 172.21.15.45:0/2348001149 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f9ae0e3ad0 0x55f9ae0e5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:19.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.558+0000 7f86a7d64740 4 client.25444 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:49:19.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.558+0000 7f86a7d64740 1 -- 172.21.15.45:0/2348001149 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f9ae033790 con 0x55f9ae0e3ad0 2024-03-28T17:49:19.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.558+0000 7f8697e00640 1 --2- 172.21.15.45:0/2348001149 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f9ae0e3ad0 0x55f9ae0e5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:19.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.559+0000 7f8697e00640 1 --2- 172.21.15.45:0/2348001149 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f9ae0e3ad0 0x55f9ae0e5ee0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce0fcf15a79012c server_cookie=6f460e4c839301ff in_seq=0 out_seq=0 2024-03-28T17:49:19.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.559+0000 7f8683e00640 10 client.25444 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:19.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.559+0000 7f8683e00640 1 -- 172.21.15.45:0/2348001149 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f9ae033790 con 0x55f9ae0e3ad0 2024-03-28T17:49:19.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.560+0000 7f8683e00640 10 client.25444 handle_command_reply: tid=0 2024-03-28T17:49:19.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.560+0000 7f86a7d64740 1 client.25444 shutdown 2024-03-28T17:49:19.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.560+0000 7f86a7d64740 1 -- 172.21.15.45:0/2348001149 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0c20f0 msgr2=0x55f9ae0d3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:19.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.560+0000 7f86a7d64740 1 --2- 172.21.15.45:0/2348001149 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0c20f0 0x55f9ae0d3350 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f869000e9c0 tx=0x7f8690010ef0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.560+0000 7f86a7d64740 1 -- 172.21.15.45:0/2348001149 shutdown_connections 2024-03-28T17:49:19.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.560+0000 7f86a7d64740 1 --2- 172.21.15.45:0/2348001149 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f9ae0e3ad0 0x55f9ae0e5ee0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.560+0000 7f86a7d64740 1 --2- 172.21.15.45:0/2348001149 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0d3c70 0x55f9ae0d0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.560+0000 7f86a7d64740 1 --2- 172.21.15.45:0/2348001149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9ae0c2dd0 0x55f9ae0d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.560+0000 7f8683e00640 0 client.25444 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:19.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.560+0000 7f86a7d64740 1 --2- 172.21.15.45:0/2348001149 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0c20f0 0x55f9ae0d3350 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.560+0000 7f86a7d64740 1 -- 172.21.15.45:0/2348001149 >> 172.21.15.45:0/2348001149 conn(0x55f9ae089200 msgr2=0x55f9ae0be890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:19.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.560+0000 7f86a7d64740 1 -- 172.21.15.45:0/2348001149 shutdown_connections 2024-03-28T17:49:19.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.560+0000 7f86a7d64740 1 -- 172.21.15.45:0/2348001149 wait complete. 2024-03-28T17:49:19.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.560+0000 7f86a7d64740 20 client.25444 trim_cache size 0 max 16384 2024-03-28T17:49:19.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.578+0000 7f86a7d64740 1 Processor -- start 2024-03-28T17:49:19.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.578+0000 7f86a7d64740 1 -- start start 2024-03-28T17:49:19.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.578+0000 7f86a7d64740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9ae0c20f0 0x55f9ae0d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:19.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.578+0000 7f86a7d64740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0c2dd0 0x55f9ae0d3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:19.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.578+0000 7f86a7d64740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0d3c70 0x55f9ae0d5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:19.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.578+0000 7f86a7d64740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a01ad400 con 0x55f9ae0c20f0 2024-03-28T17:49:19.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.578+0000 7f86a7d64740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a01aaf00 con 0x55f9ae0c2dd0 2024-03-28T17:49:19.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.578+0000 7f86a7d64740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a01a8f80 con 0x55f9ae0d3c70 2024-03-28T17:49:19.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.578+0000 7f869de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9ae0c20f0 0x55f9ae0d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:19.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.578+0000 7f8697e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0c2dd0 0x55f9ae0d3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:19.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.579+0000 7f869de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9ae0c20f0 0x55f9ae0d2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46480/0 (socket says 172.21.15.45:46480) 2024-03-28T17:49:19.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.579+0000 7f869de00640 1 -- 172.21.15.45:0/2514319568 learned_addr learned my addr 172.21.15.45:0/2514319568 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:19.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.579+0000 7f869e800640 1 --2- 172.21.15.45:0/2514319568 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0d3c70 0x55f9ae0d5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:19.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.579+0000 7f869de00640 1 -- 172.21.15.45:0/2514319568 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0d3c70 msgr2=0x55f9ae0d5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:19.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.579+0000 7f869de00640 1 --2- 172.21.15.45:0/2514319568 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0d3c70 0x55f9ae0d5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.579+0000 7f869de00640 1 -- 172.21.15.45:0/2514319568 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0c2dd0 msgr2=0x55f9ae0d3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:19.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.579+0000 7f869de00640 1 --2- 172.21.15.45:0/2514319568 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0c2dd0 0x55f9ae0d3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.579+0000 7f869de00640 1 -- 172.21.15.45:0/2514319568 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9adcfa0f0 con 0x55f9ae0c20f0 2024-03-28T17:49:19.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.579+0000 7f8697e00640 1 --2- 172.21.15.45:0/2514319568 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0c2dd0 0x55f9ae0d3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:49:19.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.579+0000 7f869de00640 1 --2- 172.21.15.45:0/2514319568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9ae0c20f0 0x55f9ae0d2e10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f869009b550 tx=0x7f8690097bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35769592a2d1edfc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:19.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.583+0000 7f868ea00640 1 -- 172.21.15.45:0/2514319568 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869009bf50 con 0x55f9ae0c20f0 2024-03-28T17:49:19.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.583+0000 7f868ea00640 1 -- 172.21.15.45:0/2514319568 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8690097670 con 0x55f9ae0c20f0 2024-03-28T17:49:19.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.583+0000 7f868ea00640 1 -- 172.21.15.45:0/2514319568 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869009aca0 con 0x55f9ae0c20f0 2024-03-28T17:49:19.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.584+0000 7f86a7d64740 1 -- 172.21.15.45:0/2514319568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9ae0c20f0 msgr2=0x55f9ae0d2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:19.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.584+0000 7f86a7d64740 1 --2- 172.21.15.45:0/2514319568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9ae0c20f0 0x55f9ae0d2e10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f869009b550 tx=0x7f8690097bb0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.584+0000 7f86a7d64740 1 -- 172.21.15.45:0/2514319568 shutdown_connections 2024-03-28T17:49:19.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.584+0000 7f86a7d64740 1 --2- 172.21.15.45:0/2514319568 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0d3c70 0x55f9ae0d5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.584+0000 7f86a7d64740 1 --2- 172.21.15.45:0/2514319568 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0c2dd0 0x55f9ae0d3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.584+0000 7f86a7d64740 1 --2- 172.21.15.45:0/2514319568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9ae0c20f0 0x55f9ae0d2e10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.584+0000 7f86a7d64740 1 -- 172.21.15.45:0/2514319568 >> 172.21.15.45:0/2514319568 conn(0x55f9ae089200 msgr2=0x55f9ae0bc670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:19.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.584+0000 7f86a7d64740 1 -- 172.21.15.45:0/2514319568 shutdown_connections 2024-03-28T17:49:19.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.584+0000 7f86a7d64740 1 -- 172.21.15.45:0/2514319568 wait complete. 2024-03-28T17:49:19.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.585+0000 7f86a7d64740 1 -- start start 2024-03-28T17:49:19.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.585+0000 7f86a7d64740 1 Processor -- start 2024-03-28T17:49:19.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.585+0000 7f86a7d64740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0c20f0 0x55f9ae0cd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:19.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.585+0000 7f86a7d64740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9ae0c2dd0 0x55f9ae0cd990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:19.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.585+0000 7f86a7d64740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0d3c70 0x55f9ae13e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:19.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.585+0000 7f86a7d64740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a01a9580 con 0x55f9ae0c2dd0 2024-03-28T17:49:19.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.585+0000 7f86a7d64740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a01aa8a0 con 0x55f9ae0d3c70 2024-03-28T17:49:19.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.585+0000 7f86a7d64740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a01a9be0 con 0x55f9ae0c20f0 2024-03-28T17:49:19.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.585+0000 7f86a7d64740 10 client.0 _ll_register_callbacks cb 0x7f869000ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:19.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.586+0000 7f869e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0d3c70 0x55f9ae13e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:19.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.586+0000 7f869de00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0c20f0 0x55f9ae0cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:19.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.586+0000 7f8697e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9ae0c2dd0 0x55f9ae0cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:19.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.586+0000 7f869de00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0c20f0 0x55f9ae0cd450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:56494/0 (socket says 172.21.15.45:56494) 2024-03-28T17:49:19.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.586+0000 7f869e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0d3c70 0x55f9ae13e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40436/0 (socket says 172.21.15.45:40436) 2024-03-28T17:49:19.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.586+0000 7f869de00640 1 -- 172.21.15.45:0/3798267461 learned_addr learned my addr 172.21.15.45:0/3798267461 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:19.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.586+0000 7f869e800640 1 -- 172.21.15.45:0/3798267461 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0c20f0 msgr2=0x55f9ae0cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:19.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.586+0000 7f869e800640 1 --2- 172.21.15.45:0/3798267461 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0c20f0 0x55f9ae0cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.586+0000 7f869e800640 1 -- 172.21.15.45:0/3798267461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9ae0c2dd0 msgr2=0x55f9ae0cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:19.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.586+0000 7f869e800640 1 --2- 172.21.15.45:0/3798267461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9ae0c2dd0 0x55f9ae0cd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.586+0000 7f869e800640 1 -- 172.21.15.45:0/3798267461 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f9ae13e8f0 con 0x55f9ae0d3c70 2024-03-28T17:49:19.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.586+0000 7f869de00640 1 --2- 172.21.15.45:0/3798267461 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0c20f0 0x55f9ae0cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:49:19.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.586+0000 7f869e800640 1 --2- 172.21.15.45:0/3798267461 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0d3c70 0x55f9ae13e3b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8698009bf0 tx=0x7f869802c100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba630d899092129f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:19.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.586+0000 7f8683e00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:49:19.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.587+0000 7f8683e00640 1 -- 172.21.15.45:0/3798267461 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8698052030 con 0x55f9ae0d3c70 2024-03-28T17:49:19.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.587+0000 7f86a7d64740 1 -- 172.21.15.45:0/3798267461 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8690097670 con 0x55f9ae0d3c70 2024-03-28T17:49:19.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.587+0000 7f8683e00640 1 -- 172.21.15.45:0/3798267461 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f8698040460 con 0x55f9ae0d3c70 2024-03-28T17:49:19.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.587+0000 7f8683e00640 1 -- 172.21.15.45:0/3798267461 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8698039040 con 0x55f9ae0d3c70 2024-03-28T17:49:19.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.587+0000 7f86a7d64740 10 client.25475 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:19.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.587+0000 7f86a7d64740 1 -- 172.21.15.45:0/3798267461 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f9ae13fb30 con 0x55f9ae0d3c70 2024-03-28T17:49:19.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.587+0000 7f8683e00640 1 -- 172.21.15.45:0/3798267461 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8690097670 con 0x55f9ae0d3c70 2024-03-28T17:49:19.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.587+0000 7f8683e00640 1 -- 172.21.15.45:0/3798267461 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f8698034940 con 0x55f9ae0d3c70 2024-03-28T17:49:19.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.587+0000 7f86a7d64740 10 client.25475 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:19.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.587+0000 7f86a7d64740 10 client.25475 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:49:19.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.587+0000 7f86a7d64740 20 client.25475 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:19.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.587+0000 7f86a7d64740 1 --2- 172.21.15.45:0/3798267461 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f9ae140260 0x55f9ae142680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:19.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.587+0000 7f86a7d64740 4 client.25475 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:49:19.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.587+0000 7f86a7d64740 1 -- 172.21.15.45:0/3798267461 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f9ae02dae0 con 0x55f9ae140260 2024-03-28T17:49:19.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.588+0000 7f869de00640 1 --2- 172.21.15.45:0/3798267461 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f9ae140260 0x55f9ae142680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:19.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.588+0000 7f869de00640 1 --2- 172.21.15.45:0/3798267461 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f9ae140260 0x55f9ae142680 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35276eab0b083840 server_cookie=528c6c1b37eaadb9 in_seq=0 out_seq=0 2024-03-28T17:49:19.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.588+0000 7f8683e00640 10 client.25475 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:19.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.588+0000 7f8683e00640 1 -- 172.21.15.45:0/3798267461 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55f9ae02dae0 con 0x55f9ae140260 2024-03-28T17:49:19.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.588+0000 7f8683e00640 10 client.25475 handle_command_reply: tid=0 2024-03-28T17:49:19.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.588+0000 7f86a7d64740 1 client.25475 shutdown 2024-03-28T17:49:19.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.588+0000 7f86a7d64740 1 -- 172.21.15.45:0/3798267461 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0d3c70 msgr2=0x55f9ae13e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:19.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.588+0000 7f86a7d64740 1 --2- 172.21.15.45:0/3798267461 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0d3c70 0x55f9ae13e3b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8698009bf0 tx=0x7f869802c100 comp rx=0 tx=0).stop 2024-03-28T17:49:19.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.588+0000 7f86a7d64740 1 -- 172.21.15.45:0/3798267461 shutdown_connections 2024-03-28T17:49:19.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.588+0000 7f86a7d64740 1 --2- 172.21.15.45:0/3798267461 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f9ae140260 0x55f9ae142680 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.588+0000 7f86a7d64740 1 --2- 172.21.15.45:0/3798267461 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f9ae0d3c70 0x55f9ae13e3b0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.588+0000 7f86a7d64740 1 --2- 172.21.15.45:0/3798267461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9ae0c2dd0 0x55f9ae0cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.588+0000 7f86a7d64740 1 --2- 172.21.15.45:0/3798267461 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f9ae0c20f0 0x55f9ae0cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.588+0000 7f8683e00640 0 client.25475 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:19.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.588+0000 7f86a7d64740 1 -- 172.21.15.45:0/3798267461 >> 172.21.15.45:0/3798267461 conn(0x55f9ae089200 msgr2=0x55f9ae0bb330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:19.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.589+0000 7f86a7d64740 1 -- 172.21.15.45:0/3798267461 shutdown_connections 2024-03-28T17:49:19.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.589+0000 7f86a7d64740 1 -- 172.21.15.45:0/3798267461 wait complete. 2024-03-28T17:49:19.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.589+0000 7f86a7d64740 20 client.25475 trim_cache size 0 max 16384 2024-03-28T17:49:19.593 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:49:19.593 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-28T17:49:19.593 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-28T17:49:19.593 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:49:19.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.589+0000 7f869f200640 1 -- 172.21.15.45:0/3694495514 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f86840a2a80 msgr2=0x7f86840a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:19.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.589+0000 7f869f200640 1 --2- 172.21.15.45:0/3694495514 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f86840a2a80 0x7f86840a4f40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8698006010 tx=0x7f8698004a40 comp rx=0 tx=0).stop 2024-03-28T17:49:19.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.589+0000 7f869f200640 1 -- 172.21.15.45:0/3694495514 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86a0103720 msgr2=0x7f86a019a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:19.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.589+0000 7f869f200640 1 --2- 172.21.15.45:0/3694495514 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86a0103720 0x7f86a019a000 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f869000e9f0 tx=0x7f869000eeb0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.589+0000 7f869f200640 1 -- 172.21.15.45:0/3694495514 shutdown_connections 2024-03-28T17:49:19.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.590+0000 7f869f200640 1 --2- 172.21.15.45:0/3694495514 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f86840a2a80 0x7f86840a4f40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.590+0000 7f869f200640 1 --2- 172.21.15.45:0/3694495514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86a0109730 0x7f86a019e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.590+0000 7f869f200640 1 --2- 172.21.15.45:0/3694495514 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86a01040e0 0x7f86a019a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.590+0000 7f869f200640 1 --2- 172.21.15.45:0/3694495514 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86a0103720 0x7f86a019a000 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.590+0000 7f869f200640 1 -- 172.21.15.45:0/3694495514 >> 172.21.15.45:0/3694495514 conn(0x7f86a00ff1d0 msgr2=0x7f86a00ff8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.590+0000 7f869f200640 1 -- 172.21.15.45:0/3694495514 shutdown_connections 2024-03-28T17:49:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.590+0000 7f869f200640 1 -- 172.21.15.45:0/3694495514 wait complete. 2024-03-28T17:49:19.608 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T17:49:19.608 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:49:19.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.816+0000 7f950be00640 1 -- 172.21.15.45:0/321134084 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f950c104030 msgr2=0x7f950c110540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:19.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.816+0000 7f950be00640 1 --2- 172.21.15.45:0/321134084 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f950c104030 0x7f950c110540 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9504009cd0 tx=0x7f950402c0e0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.816+0000 7f950be00640 1 -- 172.21.15.45:0/321134084 shutdown_connections 2024-03-28T17:49:19.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.816+0000 7f950be00640 1 --2- 172.21.15.45:0/321134084 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f950c104030 0x7f950c110540 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.816+0000 7f950be00640 1 --2- 172.21.15.45:0/321134084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f950c103670 0x7f950c103af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.816+0000 7f950be00640 1 --2- 172.21.15.45:0/321134084 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f950c109680 0x7f950c109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.816+0000 7f950be00640 1 -- 172.21.15.45:0/321134084 >> 172.21.15.45:0/321134084 conn(0x7f950c0ff140 msgr2=0x7f950c101560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:19.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.816+0000 7f950be00640 1 -- 172.21.15.45:0/321134084 shutdown_connections 2024-03-28T17:49:19.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.817+0000 7f950be00640 1 -- 172.21.15.45:0/321134084 wait complete. 2024-03-28T17:49:19.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.817+0000 7f950be00640 1 Processor -- start 2024-03-28T17:49:19.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.818+0000 7f950be00640 1 -- start start 2024-03-28T17:49:19.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.818+0000 7f950be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f950c103670 0x7f950c199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:19.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.818+0000 7f950be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f950c104030 0x7f950c19a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:19.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.818+0000 7f950be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f950c109680 0x7f950c19e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:19.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.818+0000 7f950be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c113a90 con 0x7f950c104030 2024-03-28T17:49:19.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.818+0000 7f950be00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c113910 con 0x7f950c103670 2024-03-28T17:49:19.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.818+0000 7f950be00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c113c10 con 0x7f950c109680 2024-03-28T17:49:19.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.818+0000 7f950aa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f950c103670 0x7f950c199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:19.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.818+0000 7f950aa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f950c103670 0x7f950c199e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40450/0 (socket says 172.21.15.45:40450) 2024-03-28T17:49:19.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.818+0000 7f950aa00640 1 -- 172.21.15.45:0/2690843274 learned_addr learned my addr 172.21.15.45:0/2690843274 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:19.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.818+0000 7f950b400640 1 --2- 172.21.15.45:0/2690843274 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f950c109680 0x7f950c19e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:19.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.818+0000 7f950a000640 1 --2- 172.21.15.45:0/2690843274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f950c104030 0x7f950c19a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:19.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.819+0000 7f950aa00640 1 -- 172.21.15.45:0/2690843274 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f950c109680 msgr2=0x7f950c19e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:19.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.819+0000 7f950aa00640 1 --2- 172.21.15.45:0/2690843274 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f950c109680 0x7f950c19e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.819+0000 7f950aa00640 1 -- 172.21.15.45:0/2690843274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f950c104030 msgr2=0x7f950c19a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:19.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.819+0000 7f950aa00640 1 --2- 172.21.15.45:0/2690843274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f950c104030 0x7f950c19a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:19.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.819+0000 7f950aa00640 1 -- 172.21.15.45:0/2690843274 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950c19ee60 con 0x7f950c103670 2024-03-28T17:49:19.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.819+0000 7f950aa00640 1 --2- 172.21.15.45:0/2690843274 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f950c103670 0x7f950c199e70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f94fc00e9f0 tx=0x7f94fc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:19.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.819+0000 7f9510a00640 1 -- 172.21.15.45:0/2690843274 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94fc00cd80 con 0x7f950c103670 2024-03-28T17:49:19.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.819+0000 7f9510a00640 1 -- 172.21.15.45:0/2690843274 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94fc005500 con 0x7f950c103670 2024-03-28T17:49:19.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.819+0000 7f950be00640 1 -- 172.21.15.45:0/2690843274 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f950c19f150 con 0x7f950c103670 2024-03-28T17:49:19.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.819+0000 7f9510a00640 1 -- 172.21.15.45:0/2690843274 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94fc010690 con 0x7f950c103670 2024-03-28T17:49:19.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.819+0000 7f950be00640 1 -- 172.21.15.45:0/2690843274 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950c1a69e0 con 0x7f950c103670 2024-03-28T17:49:19.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.820+0000 7f950be00640 1 -- 172.21.15.45:0/2690843274 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f950c068800 con 0x7f950c103670 2024-03-28T17:49:19.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.822+0000 7f9510a00640 1 -- 172.21.15.45:0/2690843274 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 26) v1 ==== 137466+0+0 (secure 0 0 0) 0x7f94fc0046c0 con 0x7f950c103670 2024-03-28T17:49:19.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.823+0000 7f9510a00640 1 --2- 172.21.15.45:0/2690843274 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f94f00a2a00 0x7f94f00a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:19.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.824+0000 7f9510a00640 1 -- 172.21.15.45:0/2690843274 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f94fc0cf470 con 0x7f950c103670 2024-03-28T17:49:19.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.824+0000 7f950a000640 1 --2- 172.21.15.45:0/2690843274 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f94f00a2a00 0x7f94f00a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:19.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.825+0000 7f950a000640 1 --2- 172.21.15.45:0/2690843274 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f94f00a2a00 0x7f94f00a4ec0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f94f8005e10 tx=0x7f94f8005da0 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:19.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:19.832+0000 7f9510a00640 1 -- 172.21.15.45:0/2690843274 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f94fc09c030 con 0x7f950c103670 2024-03-28T17:49:20.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.076+0000 7f950be00640 1 -- 172.21.15.45:0/2690843274 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f950c19b1c0 con 0x7f950c103670 2024-03-28T17:49:20.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.077+0000 7f9510a00640 1 -- 172.21.15.45:0/2690843274 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f94fc09b930 con 0x7f950c103670 2024-03-28T17:49:20.078 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:49:20.078 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6839/2313500150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":2313500150},{"type":"v1","addr":"172.21.15.45:6839","nonce":2313500150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6841/2647972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":2647972571},{"type":"v1","addr":"172.21.15.45:6841","nonce":2647972571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T17:46:12.261135+0000","modified":"2024-03-28T17:48:38.762497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24584,"mds_1":24530,"mds_2":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6835/2876354410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":2876354410},{"type":"v1","addr":"172.21.15.151:6835","nonce":2876354410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24500":{"gid":24500,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/4234998385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":4234998385},{"type":"v1","addr":"172.21.15.71:6835","nonce":4234998385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6837/1238964845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":1238964845},{"type":"v1","addr":"172.21.15.151:6837","nonce":1238964845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.151:6839/3003564901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6838","nonce":3003564901},{"type":"v1","addr":"172.21.15.151:6839","nonce":3003564901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6837/2008752655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2008752655},{"type":"v1","addr":"172.21.15.71:6837","nonce":2008752655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.71:6839/2709938800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":2709938800},{"type":"v1","addr":"172.21.15.71:6839","nonce":2709938800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24584,"qdb_cluster":[24584,24529,24530]},"id":1}]} 2024-03-28T17:49:20.078 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 23 2024-03-28T17:49:20.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.080+0000 7f950be00640 1 -- 172.21.15.45:0/2690843274 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f94f00a2a00 msgr2=0x7f94f00a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.080+0000 7f950be00640 1 --2- 172.21.15.45:0/2690843274 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f94f00a2a00 0x7f94f00a4ec0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f94f8005e10 tx=0x7f94f8005da0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.080+0000 7f950be00640 1 -- 172.21.15.45:0/2690843274 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f950c103670 msgr2=0x7f950c199e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.080+0000 7f950be00640 1 --2- 172.21.15.45:0/2690843274 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f950c103670 0x7f950c199e70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f94fc00e9f0 tx=0x7f94fc00edf0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.080+0000 7f950be00640 1 -- 172.21.15.45:0/2690843274 shutdown_connections 2024-03-28T17:49:20.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.080+0000 7f950be00640 1 --2- 172.21.15.45:0/2690843274 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f94f00a2a00 0x7f94f00a4ec0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.080+0000 7f950be00640 1 --2- 172.21.15.45:0/2690843274 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f950c109680 0x7f950c19e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.080+0000 7f950be00640 1 --2- 172.21.15.45:0/2690843274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f950c104030 0x7f950c19a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.080+0000 7f950be00640 1 --2- 172.21.15.45:0/2690843274 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f950c103670 0x7f950c199e70 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.080+0000 7f950be00640 1 -- 172.21.15.45:0/2690843274 >> 172.21.15.45:0/2690843274 conn(0x7f950c0ff140 msgr2=0x7f950c0ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:20.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.080+0000 7f950be00640 1 -- 172.21.15.45:0/2690843274 shutdown_connections 2024-03-28T17:49:20.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.080+0000 7f950be00640 1 -- 172.21.15.45:0/2690843274 wait complete. 2024-03-28T17:49:20.100 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T17:49:20.101 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T17:49:20.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.326+0000 7f6116000640 1 -- 172.21.15.45:0/797336775 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61101124e0 msgr2=0x7f61101148d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.326+0000 7f6116000640 1 --2- 172.21.15.45:0/797336775 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61101124e0 0x7f61101148d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6108009cd0 tx=0x7f610802c0f0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.327+0000 7f6116000640 1 -- 172.21.15.45:0/797336775 shutdown_connections 2024-03-28T17:49:20.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.327+0000 7f6116000640 1 --2- 172.21.15.45:0/797336775 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61101124e0 0x7f61101148d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.327+0000 7f6116000640 1 --2- 172.21.15.45:0/797336775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61100764f0 0x7f6110111fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.327+0000 7f6116000640 1 --2- 172.21.15.45:0/797336775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6110075bd0 0x7f6110075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.327+0000 7f6116000640 1 -- 172.21.15.45:0/797336775 >> 172.21.15.45:0/797336775 conn(0x7f61100ff2f0 msgr2=0x7f6110101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:20.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.327+0000 7f6116000640 1 -- 172.21.15.45:0/797336775 shutdown_connections 2024-03-28T17:49:20.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.327+0000 7f6116000640 1 -- 172.21.15.45:0/797336775 wait complete. 2024-03-28T17:49:20.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.328+0000 7f6116000640 1 Processor -- start 2024-03-28T17:49:20.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.328+0000 7f6116000640 1 -- start start 2024-03-28T17:49:20.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.328+0000 7f6116000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6110075bd0 0x7f611019e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.328+0000 7f6116000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61100764f0 0x7f611019eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.328+0000 7f6116000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61101124e0 0x7f61101a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.328+0000 7f6116000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6110118040 con 0x7f61100764f0 2024-03-28T17:49:20.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.328+0000 7f6116000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6110117ec0 con 0x7f6110075bd0 2024-03-28T17:49:20.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.328+0000 7f6116000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61101181c0 con 0x7f61101124e0 2024-03-28T17:49:20.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.328+0000 7f610f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6110075bd0 0x7f611019e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.328+0000 7f610f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6110075bd0 0x7f611019e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40460/0 (socket says 172.21.15.45:40460) 2024-03-28T17:49:20.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.328+0000 7f610f400640 1 -- 172.21.15.45:0/511512961 learned_addr learned my addr 172.21.15.45:0/511512961 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:20.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.328+0000 7f6107e00640 1 --2- 172.21.15.45:0/511512961 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61100764f0 0x7f611019eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.328+0000 7f610fe00640 1 --2- 172.21.15.45:0/511512961 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61101124e0 0x7f61101a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.329+0000 7f610f400640 1 -- 172.21.15.45:0/511512961 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61101124e0 msgr2=0x7f61101a2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.329+0000 7f610f400640 1 --2- 172.21.15.45:0/511512961 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61101124e0 0x7f61101a2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.329+0000 7f610f400640 1 -- 172.21.15.45:0/511512961 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61100764f0 msgr2=0x7f611019eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.329+0000 7f610f400640 1 --2- 172.21.15.45:0/511512961 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61100764f0 0x7f611019eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.329+0000 7f610f400640 1 -- 172.21.15.45:0/511512961 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61101a3570 con 0x7f6110075bd0 2024-03-28T17:49:20.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.329+0000 7f610f400640 1 --2- 172.21.15.45:0/511512961 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6110075bd0 0x7f611019e580 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f610000e9f0 tx=0x7f610000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:20.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.329+0000 7f610cc00640 1 -- 172.21.15.45:0/511512961 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610000cd80 con 0x7f6110075bd0 2024-03-28T17:49:20.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.329+0000 7f610cc00640 1 -- 172.21.15.45:0/511512961 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6100010910 con 0x7f6110075bd0 2024-03-28T17:49:20.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.329+0000 7f6116000640 1 -- 172.21.15.45:0/511512961 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61101a3860 con 0x7f6110075bd0 2024-03-28T17:49:20.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.329+0000 7f610cc00640 1 -- 172.21.15.45:0/511512961 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6100018990 con 0x7f6110075bd0 2024-03-28T17:49:20.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.329+0000 7f6116000640 1 -- 172.21.15.45:0/511512961 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61101ab0f0 con 0x7f6110075bd0 2024-03-28T17:49:20.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.332+0000 7f610cc00640 1 -- 172.21.15.45:0/511512961 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 26) v1 ==== 137466+0+0 (secure 0 0 0) 0x7f6100010ac0 con 0x7f6110075bd0 2024-03-28T17:49:20.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.332+0000 7f6118c3d740 1 Processor -- start 2024-03-28T17:49:20.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.333+0000 7f6118c3d740 1 -- start start 2024-03-28T17:49:20.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.333+0000 7f6118c3d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa05c6d0 0x55dcfa05cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.333+0000 7f6118c3d740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05d080 0x55dcfa065630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.333+0000 7f6118c3d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa065c00 0x55dcfa068050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.333+0000 7f6118c3d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcf9fc02c0 con 0x55dcfa05c6d0 2024-03-28T17:49:20.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.333+0000 7f6118c3d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcf9f68c90 con 0x55dcfa065c00 2024-03-28T17:49:20.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.333+0000 7f6118c3d740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcf9fbc360 con 0x55dcfa05d080 2024-03-28T17:49:20.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.333+0000 7f610f400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05d080 0x55dcfa065630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.333+0000 7f610fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa05c6d0 0x55dcfa05cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.333+0000 7f610f400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05d080 0x55dcfa065630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:56540/0 (socket says 172.21.15.45:56540) 2024-03-28T17:49:20.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.333+0000 7f610fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa05c6d0 0x55dcfa05cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46534/0 (socket says 172.21.15.45:46534) 2024-03-28T17:49:20.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.333+0000 7f610f400640 1 -- 172.21.15.45:0/95697578 learned_addr learned my addr 172.21.15.45:0/95697578 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:20.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.333+0000 7f6107e00640 1 --2- 172.21.15.45:0/95697578 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa065c00 0x55dcfa068050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.334+0000 7f610f400640 1 -- 172.21.15.45:0/95697578 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa065c00 msgr2=0x55dcfa068050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.334+0000 7f610f400640 1 --2- 172.21.15.45:0/95697578 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa065c00 0x55dcfa068050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.334+0000 7f610f400640 1 -- 172.21.15.45:0/95697578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa05c6d0 msgr2=0x55dcfa05cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.334+0000 7f610f400640 1 --2- 172.21.15.45:0/95697578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa05c6d0 0x55dcfa05cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.334+0000 7f610f400640 1 -- 172.21.15.45:0/95697578 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dcf9f8d080 con 0x55dcfa05d080 2024-03-28T17:49:20.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.334+0000 7f610f400640 1 --2- 172.21.15.45:0/95697578 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05d080 0x55dcfa065630 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f61000cc940 tx=0x7f61000d7910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=caaf849a679b064 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:20.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.334+0000 7f610cc00640 1 --2- 172.21.15.45:0/511512961 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f60f40a2a00 0x7f60f40a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.334+0000 7f60ffe00640 1 -- 172.21.15.45:0/95697578 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61000e2070 con 0x55dcfa05d080 2024-03-28T17:49:20.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.334+0000 7f60ffe00640 1 -- 172.21.15.45:0/95697578 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f610009b8e0 con 0x55dcfa05d080 2024-03-28T17:49:20.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.334+0000 7f60ffe00640 1 -- 172.21.15.45:0/95697578 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610001d210 con 0x55dcfa05d080 2024-03-28T17:49:20.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.334+0000 7f610fe00640 1 --2- 172.21.15.45:0/511512961 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f60f40a2a00 0x7f60f40a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.334+0000 7f610cc00640 1 -- 172.21.15.45:0/511512961 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f61000cf370 con 0x7f6110075bd0 2024-03-28T17:49:20.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.335+0000 7f6118c3d740 1 -- 172.21.15.45:0/95697578 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05d080 msgr2=0x55dcfa065630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.335+0000 7f6118c3d740 1 --2- 172.21.15.45:0/95697578 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05d080 0x55dcfa065630 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f61000cc940 tx=0x7f61000d7910 comp rx=0 tx=0).stop 2024-03-28T17:49:20.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.335+0000 7f610fe00640 1 --2- 172.21.15.45:0/511512961 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f60f40a2a00 0x7f60f40a4ec0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6108009bf0 tx=0x7f6108005d80 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:20.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.335+0000 7f6118c3d740 1 -- 172.21.15.45:0/95697578 shutdown_connections 2024-03-28T17:49:20.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.335+0000 7f6118c3d740 1 --2- 172.21.15.45:0/95697578 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa065c00 0x55dcfa068050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.335+0000 7f6118c3d740 1 --2- 172.21.15.45:0/95697578 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05d080 0x55dcfa065630 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.335+0000 7f6118c3d740 1 --2- 172.21.15.45:0/95697578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa05c6d0 0x55dcfa05cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.335+0000 7f6118c3d740 1 -- 172.21.15.45:0/95697578 >> 172.21.15.45:0/95697578 conn(0x55dcfa056da0 msgr2=0x55dcfa057180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:20.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.336+0000 7f6118c3d740 1 -- 172.21.15.45:0/95697578 shutdown_connections 2024-03-28T17:49:20.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.336+0000 7f6118c3d740 1 -- 172.21.15.45:0/95697578 wait complete. 2024-03-28T17:49:20.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.336+0000 7f6118c3d740 1 -- start start 2024-03-28T17:49:20.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.336+0000 7f6118c3d740 1 Processor -- start 2024-03-28T17:49:20.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.337+0000 7f6118c3d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa05c6d0 0x55dcfa065500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.337+0000 7f6118c3d740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05d080 0x55dcfa062890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.337+0000 7f6118c3d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa065c00 0x55dcfa062e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.337+0000 7f6118c3d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcfa068b70 con 0x55dcfa065c00 2024-03-28T17:49:20.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.337+0000 7f6118c3d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcf9f7c0d0 con 0x55dcfa05c6d0 2024-03-28T17:49:20.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.337+0000 7f6118c3d740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcfa068cf0 con 0x55dcfa05d080 2024-03-28T17:49:20.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.337+0000 7f6118c3d740 10 client.0 _ll_register_callbacks cb 0x55dcf9f80e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:20.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.337+0000 7f610f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa05c6d0 0x55dcfa065500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.337+0000 7f6107e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05d080 0x55dcfa062890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.337+0000 7f610f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa05c6d0 0x55dcfa065500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40472/0 (socket says 172.21.15.45:40472) 2024-03-28T17:49:20.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.337+0000 7f610f400640 1 -- 172.21.15.45:0/1555513452 learned_addr learned my addr 172.21.15.45:0/1555513452 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:20.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.337+0000 7f610fe00640 1 --2- 172.21.15.45:0/1555513452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa065c00 0x55dcfa062e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.337+0000 7f610f400640 1 -- 172.21.15.45:0/1555513452 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05d080 msgr2=0x55dcfa062890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.337+0000 7f610f400640 1 --2- 172.21.15.45:0/1555513452 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05d080 0x55dcfa062890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.337+0000 7f610f400640 1 -- 172.21.15.45:0/1555513452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa065c00 msgr2=0x55dcfa062e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.337+0000 7f610f400640 1 --2- 172.21.15.45:0/1555513452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa065c00 0x55dcfa062e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.337+0000 7f610f400640 1 -- 172.21.15.45:0/1555513452 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dcf9cd43f0 con 0x55dcfa05c6d0 2024-03-28T17:49:20.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.337+0000 7f610fe00640 1 --2- 172.21.15.45:0/1555513452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa065c00 0x55dcfa062e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:49:20.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.337+0000 7f6107e00640 1 --2- 172.21.15.45:0/1555513452 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05d080 0x55dcfa062890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:49:20.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.338+0000 7f610f400640 1 --2- 172.21.15.45:0/1555513452 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa05c6d0 0x55dcfa065500 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f610009b870 tx=0x7f610001ff70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a98311c591fdeea8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:20.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.338+0000 7f60fcc00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:49:20.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.338+0000 7f60fcc00640 1 -- 172.21.15.45:0/1555513452 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610004b0c0 con 0x55dcfa05c6d0 2024-03-28T17:49:20.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.338+0000 7f60fcc00640 1 -- 172.21.15.45:0/1555513452 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f610009bb40 con 0x55dcfa05c6d0 2024-03-28T17:49:20.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.338+0000 7f6118c3d740 1 -- 172.21.15.45:0/1555513452 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f610009b8e0 con 0x55dcfa05c6d0 2024-03-28T17:49:20.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.338+0000 7f60fcc00640 1 -- 172.21.15.45:0/1555513452 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f610004a3e0 con 0x55dcfa05c6d0 2024-03-28T17:49:20.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.338+0000 7f6118c3d740 10 client.25493 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:20.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.338+0000 7f6118c3d740 1 -- 172.21.15.45:0/1555513452 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55dcf9d9e7d0 con 0x55dcfa05c6d0 2024-03-28T17:49:20.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.338+0000 7f60fcc00640 1 -- 172.21.15.45:0/1555513452 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f610009b8e0 con 0x55dcfa05c6d0 2024-03-28T17:49:20.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.339+0000 7f60fcc00640 1 -- 172.21.15.45:0/1555513452 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f61000a3b40 con 0x55dcfa05c6d0 2024-03-28T17:49:20.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.339+0000 7f6118c3d740 10 client.25493 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:20.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.339+0000 7f6118c3d740 10 client.25493 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-28T17:49:20.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.339+0000 7f6118c3d740 20 client.25493 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:20.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.339+0000 7f6118c3d740 1 --2- 172.21.15.45:0/1555513452 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x55dcfa0758f0 0x55dcfa077ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.339+0000 7f6118c3d740 4 client.25493 mds_command: new command op to 24529 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:49:20.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.339+0000 7f6118c3d740 1 -- 172.21.15.45:0/1555513452 --> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dcf9fcd800 con 0x55dcfa0758f0 2024-03-28T17:49:20.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.339+0000 7f6107e00640 1 --2- 172.21.15.45:0/1555513452 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x55dcfa0758f0 0x55dcfa077ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.340+0000 7f6107e00640 1 --2- 172.21.15.45:0/1555513452 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x55dcfa0758f0 0x55dcfa077ce0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dc67c114cdcf01a4 server_cookie=ea5fb3b5b8a39582 in_seq=0 out_seq=0 2024-03-28T17:49:20.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.340+0000 7f60fcc00640 10 client.25493 ms_handle_connect on v2:172.21.15.151:6838/3003564901 2024-03-28T17:49:20.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.340+0000 7f60fcc00640 1 -- 172.21.15.45:0/1555513452 <== mds.2 v2:172.21.15.151:6838/3003564901 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55dcf9fcd800 con 0x55dcfa0758f0 2024-03-28T17:49:20.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.340+0000 7f60fcc00640 10 client.25493 handle_command_reply: tid=0 2024-03-28T17:49:20.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.340+0000 7f6118c3d740 1 client.25493 shutdown 2024-03-28T17:49:20.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.341+0000 7f6118c3d740 1 -- 172.21.15.45:0/1555513452 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa05c6d0 msgr2=0x55dcfa065500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.341+0000 7f6118c3d740 1 --2- 172.21.15.45:0/1555513452 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa05c6d0 0x55dcfa065500 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f610009b870 tx=0x7f610001ff70 comp rx=0 tx=0).stop 2024-03-28T17:49:20.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.341+0000 7f6118c3d740 1 -- 172.21.15.45:0/1555513452 shutdown_connections 2024-03-28T17:49:20.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.341+0000 7f6118c3d740 1 --2- 172.21.15.45:0/1555513452 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x55dcfa0758f0 0x55dcfa077ce0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.341+0000 7f6118c3d740 1 --2- 172.21.15.45:0/1555513452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa065c00 0x55dcfa062e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.341+0000 7f6118c3d740 1 --2- 172.21.15.45:0/1555513452 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05d080 0x55dcfa062890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.341+0000 7f6118c3d740 1 --2- 172.21.15.45:0/1555513452 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa05c6d0 0x55dcfa065500 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.341+0000 7f60fcc00640 0 client.25493 ms_handle_reset on v2:172.21.15.151:6838/3003564901 2024-03-28T17:49:20.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.341+0000 7f6118c3d740 1 -- 172.21.15.45:0/1555513452 >> 172.21.15.45:0/1555513452 conn(0x55dcfa056da0 msgr2=0x55dcfa058af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:20.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.341+0000 7f6118c3d740 1 -- 172.21.15.45:0/1555513452 shutdown_connections 2024-03-28T17:49:20.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.341+0000 7f6118c3d740 1 -- 172.21.15.45:0/1555513452 wait complete. 2024-03-28T17:49:20.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.341+0000 7f6118c3d740 20 client.25493 trim_cache size 0 max 16384 2024-03-28T17:49:20.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.355+0000 7f6118c3d740 1 Processor -- start 2024-03-28T17:49:20.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.355+0000 7f6118c3d740 1 -- start start 2024-03-28T17:49:20.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.355+0000 7f6118c3d740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05c6d0 0x55dcfa055d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.355+0000 7f6118c3d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa05d080 0x55dcfa064fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.355+0000 7f6118c3d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa065c00 0x55dcfa065500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.356+0000 7f6118c3d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61101b5f00 con 0x55dcfa05d080 2024-03-28T17:49:20.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.356+0000 7f6118c3d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61101b3a00 con 0x55dcfa065c00 2024-03-28T17:49:20.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.356+0000 7f6118c3d740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61101b1a80 con 0x55dcfa05c6d0 2024-03-28T17:49:20.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.356+0000 7f610fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa065c00 0x55dcfa065500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.356+0000 7f610fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa065c00 0x55dcfa065500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40476/0 (socket says 172.21.15.45:40476) 2024-03-28T17:49:20.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.356+0000 7f610fe00640 1 -- 172.21.15.45:0/1070936871 learned_addr learned my addr 172.21.15.45:0/1070936871 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:20.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.356+0000 7f610f400640 1 --2- 172.21.15.45:0/1070936871 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05c6d0 0x55dcfa055d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.356+0000 7f6107e00640 1 --2- 172.21.15.45:0/1070936871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa05d080 0x55dcfa064fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.356+0000 7f610fe00640 1 -- 172.21.15.45:0/1070936871 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05c6d0 msgr2=0x55dcfa055d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.356+0000 7f610fe00640 1 --2- 172.21.15.45:0/1070936871 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05c6d0 0x55dcfa055d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.356+0000 7f610fe00640 1 -- 172.21.15.45:0/1070936871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa05d080 msgr2=0x55dcfa064fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.356+0000 7f610fe00640 1 --2- 172.21.15.45:0/1070936871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa05d080 0x55dcfa064fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.356+0000 7f610fe00640 1 -- 172.21.15.45:0/1070936871 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dcfa063180 con 0x55dcfa065c00 2024-03-28T17:49:20.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.356+0000 7f610f400640 1 --2- 172.21.15.45:0/1070936871 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05c6d0 0x55dcfa055d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:49:20.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.357+0000 7f610fe00640 1 --2- 172.21.15.45:0/1070936871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa065c00 0x55dcfa065500 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f611006aa60 tx=0x7f6108041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9299147eb5f35a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:20.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.357+0000 7f60ff400640 1 -- 172.21.15.45:0/1070936871 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6108039040 con 0x55dcfa065c00 2024-03-28T17:49:20.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.357+0000 7f60ff400640 1 -- 172.21.15.45:0/1070936871 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6108003e50 con 0x55dcfa065c00 2024-03-28T17:49:20.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.357+0000 7f60ff400640 1 -- 172.21.15.45:0/1070936871 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610804e460 con 0x55dcfa065c00 2024-03-28T17:49:20.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.358+0000 7f6118c3d740 1 -- 172.21.15.45:0/1070936871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa065c00 msgr2=0x55dcfa065500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.358+0000 7f6118c3d740 1 --2- 172.21.15.45:0/1070936871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa065c00 0x55dcfa065500 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f611006aa60 tx=0x7f6108041000 comp rx=0 tx=0).stop 2024-03-28T17:49:20.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.358+0000 7f6118c3d740 1 -- 172.21.15.45:0/1070936871 shutdown_connections 2024-03-28T17:49:20.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.358+0000 7f6118c3d740 1 --2- 172.21.15.45:0/1070936871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa065c00 0x55dcfa065500 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.358+0000 7f6118c3d740 1 --2- 172.21.15.45:0/1070936871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa05d080 0x55dcfa064fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.358+0000 7f6118c3d740 1 --2- 172.21.15.45:0/1070936871 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05c6d0 0x55dcfa055d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.358+0000 7f6118c3d740 1 -- 172.21.15.45:0/1070936871 >> 172.21.15.45:0/1070936871 conn(0x55dcfa056da0 msgr2=0x55dcfa055620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:20.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.358+0000 7f6118c3d740 1 -- 172.21.15.45:0/1070936871 shutdown_connections 2024-03-28T17:49:20.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.358+0000 7f6118c3d740 1 -- 172.21.15.45:0/1070936871 wait complete. 2024-03-28T17:49:20.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.358+0000 7f6118c3d740 1 -- start start 2024-03-28T17:49:20.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.358+0000 7f6118c3d740 1 Processor -- start 2024-03-28T17:49:20.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.359+0000 7f6118c3d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa05c6d0 0x55dcfa078870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.359+0000 7f6118c3d740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05d080 0x55dcfa078db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.359+0000 7f6118c3d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa065c00 0x55dcfa0cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.359+0000 7f6118c3d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61101b2080 con 0x55dcfa05c6d0 2024-03-28T17:49:20.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.359+0000 7f6118c3d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61101b33a0 con 0x55dcfa065c00 2024-03-28T17:49:20.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.359+0000 7f6118c3d740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61101b26e0 con 0x55dcfa05d080 2024-03-28T17:49:20.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.359+0000 7f6118c3d740 10 client.0 _ll_register_callbacks cb 0x7f61000e0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:20.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.359+0000 7f610f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa05c6d0 0x55dcfa078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.359+0000 7f6107e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05d080 0x55dcfa078db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.359+0000 7f610fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa065c00 0x55dcfa0cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.359+0000 7f610f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa05c6d0 0x55dcfa078870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46562/0 (socket says 172.21.15.45:46562) 2024-03-28T17:49:20.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.359+0000 7f610f400640 1 -- 172.21.15.45:0/1444688286 learned_addr learned my addr 172.21.15.45:0/1444688286 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:20.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.359+0000 7f6107e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05d080 0x55dcfa078db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:56566/0 (socket says 172.21.15.45:56566) 2024-03-28T17:49:20.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.360+0000 7f610fe00640 1 -- 172.21.15.45:0/1444688286 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05d080 msgr2=0x55dcfa078db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.360+0000 7f610fe00640 1 --2- 172.21.15.45:0/1444688286 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05d080 0x55dcfa078db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.360+0000 7f610fe00640 1 -- 172.21.15.45:0/1444688286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa05c6d0 msgr2=0x55dcfa078870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.360+0000 7f610fe00640 1 --2- 172.21.15.45:0/1444688286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa05c6d0 0x55dcfa078870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.360+0000 7f610fe00640 1 -- 172.21.15.45:0/1444688286 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dcfa0d0390 con 0x55dcfa065c00 2024-03-28T17:49:20.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.360+0000 7f6107e00640 1 --2- 172.21.15.45:0/1444688286 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05d080 0x55dcfa078db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:49:20.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.360+0000 7f610f400640 1 --2- 172.21.15.45:0/1444688286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa05c6d0 0x55dcfa078870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:49:20.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.360+0000 7f610fe00640 1 --2- 172.21.15.45:0/1444688286 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa065c00 0x55dcfa0cfe20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f610802c700 tx=0x7f6108005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e0ef3b6b70a2ceb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:20.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.360+0000 7f60fcc00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:49:20.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.360+0000 7f60fcc00640 1 -- 172.21.15.45:0/1444688286 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610805d020 con 0x55dcfa065c00 2024-03-28T17:49:20.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.360+0000 7f60fcc00640 1 -- 172.21.15.45:0/1444688286 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f6108003970 con 0x55dcfa065c00 2024-03-28T17:49:20.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.360+0000 7f6118c3d740 1 -- 172.21.15.45:0/1444688286 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6108003e50 con 0x55dcfa065c00 2024-03-28T17:49:20.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.361+0000 7f60fcc00640 1 -- 172.21.15.45:0/1444688286 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6108068050 con 0x55dcfa065c00 2024-03-28T17:49:20.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.361+0000 7f6118c3d740 10 client.25499 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:20.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.361+0000 7f6118c3d740 1 -- 172.21.15.45:0/1444688286 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55dcfa0d1690 con 0x55dcfa065c00 2024-03-28T17:49:20.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.361+0000 7f60fcc00640 1 -- 172.21.15.45:0/1444688286 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6108003e50 con 0x55dcfa065c00 2024-03-28T17:49:20.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.361+0000 7f60fcc00640 1 -- 172.21.15.45:0/1444688286 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f610804e840 con 0x55dcfa065c00 2024-03-28T17:49:20.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.361+0000 7f6118c3d740 10 client.25499 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:20.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.361+0000 7f6118c3d740 10 client.25499 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-28T17:49:20.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.361+0000 7f6118c3d740 20 client.25499 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:20.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.361+0000 7f6118c3d740 1 --2- 172.21.15.45:0/1444688286 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x55dcfa0d1cf0 0x55dcfa0d40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.361+0000 7f6118c3d740 4 client.25499 mds_command: new command op to 24529 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T17:49:20.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.361+0000 7f6118c3d740 1 -- 172.21.15.45:0/1444688286 --> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55dcf9fc7b50 con 0x55dcfa0d1cf0 2024-03-28T17:49:20.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.361+0000 7f610f400640 1 --2- 172.21.15.45:0/1444688286 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x55dcfa0d1cf0 0x55dcfa0d40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.362+0000 7f610f400640 1 --2- 172.21.15.45:0/1444688286 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x55dcfa0d1cf0 0x55dcfa0d40e0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=73fc43981d841127 server_cookie=83a6159d0b13e8f7 in_seq=0 out_seq=0 2024-03-28T17:49:20.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.362+0000 7f60fcc00640 10 client.25499 ms_handle_connect on v2:172.21.15.151:6838/3003564901 2024-03-28T17:49:20.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.362+0000 7f60fcc00640 1 -- 172.21.15.45:0/1444688286 <== mds.2 v2:172.21.15.151:6838/3003564901 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55dcf9fc7b50 con 0x55dcfa0d1cf0 2024-03-28T17:49:20.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.362+0000 7f60fcc00640 10 client.25499 handle_command_reply: tid=0 2024-03-28T17:49:20.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.362+0000 7f6118c3d740 1 client.25499 shutdown 2024-03-28T17:49:20.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.362+0000 7f6118c3d740 1 -- 172.21.15.45:0/1444688286 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa065c00 msgr2=0x55dcfa0cfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.362+0000 7f6118c3d740 1 --2- 172.21.15.45:0/1444688286 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa065c00 0x55dcfa0cfe20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f610802c700 tx=0x7f6108005ef0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.362+0000 7f6118c3d740 1 -- 172.21.15.45:0/1444688286 shutdown_connections 2024-03-28T17:49:20.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.362+0000 7f6118c3d740 1 --2- 172.21.15.45:0/1444688286 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x55dcfa0d1cf0 0x55dcfa0d40e0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.362+0000 7f6118c3d740 1 --2- 172.21.15.45:0/1444688286 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55dcfa065c00 0x55dcfa0cfe20 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.362+0000 7f6118c3d740 1 --2- 172.21.15.45:0/1444688286 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55dcfa05d080 0x55dcfa078db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.362+0000 7f6118c3d740 1 --2- 172.21.15.45:0/1444688286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dcfa05c6d0 0x55dcfa078870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.362+0000 7f6118c3d740 1 -- 172.21.15.45:0/1444688286 >> 172.21.15.45:0/1444688286 conn(0x55dcfa056da0 msgr2=0x55dcfa055eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:20.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.362+0000 7f60fcc00640 0 client.25499 ms_handle_reset on v2:172.21.15.151:6838/3003564901 2024-03-28T17:49:20.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.362+0000 7f6118c3d740 1 -- 172.21.15.45:0/1444688286 shutdown_connections 2024-03-28T17:49:20.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.362+0000 7f6118c3d740 1 -- 172.21.15.45:0/1444688286 wait complete. 2024-03-28T17:49:20.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.362+0000 7f6118c3d740 20 client.25499 trim_cache size 0 max 16384 2024-03-28T17:49:20.368 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-28T17:49:20.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.363+0000 7f6116000640 1 -- 172.21.15.45:0/511512961 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f60f40a2a00 msgr2=0x7f60f40a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.363+0000 7f6116000640 1 --2- 172.21.15.45:0/511512961 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f60f40a2a00 0x7f60f40a4ec0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6108009bf0 tx=0x7f6108005d80 comp rx=0 tx=0).stop 2024-03-28T17:49:20.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.363+0000 7f6116000640 1 -- 172.21.15.45:0/511512961 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6110075bd0 msgr2=0x7f611019e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.363+0000 7f6116000640 1 --2- 172.21.15.45:0/511512961 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6110075bd0 0x7f611019e580 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f610000e9f0 tx=0x7f610000eeb0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.363+0000 7f6116000640 1 -- 172.21.15.45:0/511512961 shutdown_connections 2024-03-28T17:49:20.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.363+0000 7f6116000640 1 --2- 172.21.15.45:0/511512961 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f60f40a2a00 0x7f60f40a4ec0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.363+0000 7f6116000640 1 --2- 172.21.15.45:0/511512961 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61101124e0 0x7f61101a2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.363+0000 7f6116000640 1 --2- 172.21.15.45:0/511512961 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61100764f0 0x7f611019eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.363+0000 7f6116000640 1 --2- 172.21.15.45:0/511512961 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6110075bd0 0x7f611019e580 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.363+0000 7f6116000640 1 -- 172.21.15.45:0/511512961 >> 172.21.15.45:0/511512961 conn(0x7f61100ff2f0 msgr2=0x7f6110113150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:20.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.363+0000 7f6116000640 1 -- 172.21.15.45:0/511512961 shutdown_connections 2024-03-28T17:49:20.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.363+0000 7f6116000640 1 -- 172.21.15.45:0/511512961 wait complete. 2024-03-28T17:49:20.380 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T17:49:20.381 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T17:49:20.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.591+0000 7fd8c9000640 1 -- 172.21.15.45:0/2928051738 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8c41040e0 msgr2=0x7fd8c41105f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.591+0000 7fd8c9000640 1 --2- 172.21.15.45:0/2928051738 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8c41040e0 0x7fd8c41105f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc009ca0 tx=0x7fd8bc02c100 comp rx=0 tx=0).stop 2024-03-28T17:49:20.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.591+0000 7fd8c9000640 1 -- 172.21.15.45:0/2928051738 shutdown_connections 2024-03-28T17:49:20.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.591+0000 7fd8c9000640 1 --2- 172.21.15.45:0/2928051738 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8c41040e0 0x7fd8c41105f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.591+0000 7fd8c9000640 1 --2- 172.21.15.45:0/2928051738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8c4103720 0x7fd8c4103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.591+0000 7fd8c9000640 1 --2- 172.21.15.45:0/2928051738 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8c4109730 0x7fd8c4109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.591+0000 7fd8c9000640 1 -- 172.21.15.45:0/2928051738 >> 172.21.15.45:0/2928051738 conn(0x7fd8c40ff1d0 msgr2=0x7fd8c41015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:20.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.591+0000 7fd8c9000640 1 -- 172.21.15.45:0/2928051738 shutdown_connections 2024-03-28T17:49:20.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.591+0000 7fd8c9000640 1 -- 172.21.15.45:0/2928051738 wait complete. 2024-03-28T17:49:20.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.592+0000 7fd8c9000640 1 Processor -- start 2024-03-28T17:49:20.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.592+0000 7fd8c9000640 1 -- start start 2024-03-28T17:49:20.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.593+0000 7fd8c9000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8c4103720 0x7fd8c4199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.593+0000 7fd8c9000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8c41040e0 0x7fd8c419a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.593+0000 7fd8c9000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8c4109730 0x7fd8c419e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.593+0000 7fd8c9000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c4113b00 con 0x7fd8c4103720 2024-03-28T17:49:20.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.593+0000 7fd8c9000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c4113980 con 0x7fd8c4109730 2024-03-28T17:49:20.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.593+0000 7fd8c9000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c4113c80 con 0x7fd8c41040e0 2024-03-28T17:49:20.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.593+0000 7fd8c2a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8c41040e0 0x7fd8c419a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.593+0000 7fd8c3e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8c4109730 0x7fd8c419e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.593+0000 7fd8c2a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8c41040e0 0x7fd8c419a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:56574/0 (socket says 172.21.15.45:56574) 2024-03-28T17:49:20.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.593+0000 7fd8c2a00640 1 -- 172.21.15.45:0/271631631 learned_addr learned my addr 172.21.15.45:0/271631631 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:20.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.593+0000 7fd8c3400640 1 --2- 172.21.15.45:0/271631631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8c4103720 0x7fd8c4199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.594+0000 7fd8c3e00640 1 -- 172.21.15.45:0/271631631 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8c41040e0 msgr2=0x7fd8c419a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.594+0000 7fd8c3e00640 1 --2- 172.21.15.45:0/271631631 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8c41040e0 0x7fd8c419a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.594+0000 7fd8c3e00640 1 -- 172.21.15.45:0/271631631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8c4103720 msgr2=0x7fd8c4199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.594+0000 7fd8c3e00640 1 --2- 172.21.15.45:0/271631631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8c4103720 0x7fd8c4199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.594+0000 7fd8c3e00640 1 -- 172.21.15.45:0/271631631 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c419ef70 con 0x7fd8c4109730 2024-03-28T17:49:20.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.594+0000 7fd8c2a00640 1 --2- 172.21.15.45:0/271631631 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8c41040e0 0x7fd8c419a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:49:20.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.594+0000 7fd8c3400640 1 --2- 172.21.15.45:0/271631631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8c4103720 0x7fd8c4199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:49:20.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.594+0000 7fd8c3e00640 1 --2- 172.21.15.45:0/271631631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8c4109730 0x7fd8c419e840 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc009bf0 tx=0x7fd8bc034670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:20.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.594+0000 7fd8bbe00640 1 -- 172.21.15.45:0/271631631 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc039050 con 0x7fd8c4109730 2024-03-28T17:49:20.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.594+0000 7fd8bbe00640 1 -- 172.21.15.45:0/271631631 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8bc045070 con 0x7fd8c4109730 2024-03-28T17:49:20.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.594+0000 7fd8c9000640 1 -- 172.21.15.45:0/271631631 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c419f200 con 0x7fd8c4109730 2024-03-28T17:49:20.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.594+0000 7fd8bbe00640 1 -- 172.21.15.45:0/271631631 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc004610 con 0x7fd8c4109730 2024-03-28T17:49:20.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.594+0000 7fd8c9000640 1 -- 172.21.15.45:0/271631631 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c41a6af0 con 0x7fd8c4109730 2024-03-28T17:49:20.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.597+0000 7fd8bbe00640 1 -- 172.21.15.45:0/271631631 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 26) v1 ==== 137466+0+0 (secure 0 0 0) 0x7fd8bc003970 con 0x7fd8c4109730 2024-03-28T17:49:20.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.597+0000 7fd8cd3d3740 1 Processor -- start 2024-03-28T17:49:20.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.598+0000 7fd8cd3d3740 1 -- start start 2024-03-28T17:49:20.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.598+0000 7fd8cd3d3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5613361228b0 0x561336122c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.598+0000 7fd8cd3d3740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561336123260 0x56133612b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.598+0000 7fd8cd3d3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56133612bde0 0x56133612e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.598+0000 7fd8cd3d3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613360862c0 con 0x56133612bde0 2024-03-28T17:49:20.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.598+0000 7fd8cd3d3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x56133602ec90 con 0x5613361228b0 2024-03-28T17:49:20.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.598+0000 7fd8cd3d3740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x561336082360 con 0x561336123260 2024-03-28T17:49:20.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.598+0000 7fd8c2a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5613361228b0 0x561336122c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.598+0000 7fd8c2a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56133612bde0 0x56133612e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.598+0000 7fd8c3400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561336123260 0x56133612b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.598+0000 7fd8c2a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56133612bde0 0x56133612e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46586/0 (socket says 172.21.15.45:46586) 2024-03-28T17:49:20.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.598+0000 7fd8c2a00640 1 -- 172.21.15.45:0/2098406910 learned_addr learned my addr 172.21.15.45:0/2098406910 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:20.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.599+0000 7fd8c2a00640 1 -- 172.21.15.45:0/2098406910 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561336123260 msgr2=0x56133612b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.599+0000 7fd8c2a00640 1 --2- 172.21.15.45:0/2098406910 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561336123260 0x56133612b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.599+0000 7fd8c2a00640 1 -- 172.21.15.45:0/2098406910 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5613361228b0 msgr2=0x561336122c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.599+0000 7fd8c2a00640 1 --2- 172.21.15.45:0/2098406910 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5613361228b0 0x561336122c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.599+0000 7fd8c2a00640 1 -- 172.21.15.45:0/2098406910 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561336053080 con 0x56133612bde0 2024-03-28T17:49:20.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.599+0000 7fd8c2a00640 1 --2- 172.21.15.45:0/2098406910 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56133612bde0 0x56133612e230 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0012c40 tx=0x7fd8b000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76b67d90b30340ee server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:20.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.599+0000 7fd8bbe00640 1 --2- 172.21.15.45:0/271631631 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd8a80a2a80 0x7fd8a80a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.599+0000 7fd8b8c00640 1 -- 172.21.15.45:0/2098406910 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b000fbf0 con 0x56133612bde0 2024-03-28T17:49:20.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.599+0000 7fd8bbe00640 1 -- 172.21.15.45:0/271631631 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fd8bc0f16d0 con 0x7fd8c4109730 2024-03-28T17:49:20.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.599+0000 7fd8b8c00640 1 -- 172.21.15.45:0/2098406910 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8b000fd90 con 0x56133612bde0 2024-03-28T17:49:20.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.599+0000 7fd8b8c00640 1 -- 172.21.15.45:0/2098406910 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b00049a0 con 0x56133612bde0 2024-03-28T17:49:20.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.599+0000 7fd8c3400640 1 --2- 172.21.15.45:0/271631631 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd8a80a2a80 0x7fd8a80a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.600+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/2098406910 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56133612bde0 msgr2=0x56133612e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.600+0000 7fd8cd3d3740 1 --2- 172.21.15.45:0/2098406910 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56133612bde0 0x56133612e230 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0012c40 tx=0x7fd8b000b480 comp rx=0 tx=0).stop 2024-03-28T17:49:20.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.600+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/2098406910 shutdown_connections 2024-03-28T17:49:20.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.600+0000 7fd8cd3d3740 1 --2- 172.21.15.45:0/2098406910 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56133612bde0 0x56133612e230 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.600+0000 7fd8cd3d3740 1 --2- 172.21.15.45:0/2098406910 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x561336123260 0x56133612b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.600+0000 7fd8c3400640 1 --2- 172.21.15.45:0/271631631 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd8a80a2a80 0x7fd8a80a4f40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x56133611dff0 tx=0x7fd8b400b550 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:20.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.600+0000 7fd8cd3d3740 1 --2- 172.21.15.45:0/2098406910 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5613361228b0 0x561336122c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.600+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/2098406910 >> 172.21.15.45:0/2098406910 conn(0x56133611cf80 msgr2=0x56133611d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:20.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.600+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/2098406910 shutdown_connections 2024-03-28T17:49:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.600+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/2098406910 wait complete. 2024-03-28T17:49:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.601+0000 7fd8cd3d3740 1 -- start start 2024-03-28T17:49:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.601+0000 7fd8cd3d3740 1 Processor -- start 2024-03-28T17:49:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.601+0000 7fd8cd3d3740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5613361228b0 0x56133612b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.601+0000 7fd8cd3d3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561336123260 0x561336128980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.601+0000 7fd8cd3d3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56133612bde0 0x561336128ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.601+0000 7fd8cd3d3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56133612ed50 con 0x56133612bde0 2024-03-28T17:49:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.601+0000 7fd8cd3d3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613360420d0 con 0x561336123260 2024-03-28T17:49:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.601+0000 7fd8cd3d3740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x56133612eed0 con 0x5613361228b0 2024-03-28T17:49:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.602+0000 7fd8cd3d3740 10 client.0 _ll_register_callbacks cb 0x561336046e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.602+0000 7fd8c2a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561336123260 0x561336128980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.602+0000 7fd8c3400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5613361228b0 0x56133612b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.602+0000 7fd8c2a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561336123260 0x561336128980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40514/0 (socket says 172.21.15.45:40514) 2024-03-28T17:49:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.602+0000 7fd8c2a00640 1 -- 172.21.15.45:0/733949657 learned_addr learned my addr 172.21.15.45:0/733949657 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:20.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.602+0000 7fd8c3e00640 1 --2- 172.21.15.45:0/733949657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56133612bde0 0x561336128ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.602+0000 7fd8c2a00640 1 -- 172.21.15.45:0/733949657 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5613361228b0 msgr2=0x56133612b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.602+0000 7fd8c2a00640 1 --2- 172.21.15.45:0/733949657 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5613361228b0 0x56133612b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.602+0000 7fd8c2a00640 1 -- 172.21.15.45:0/733949657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56133612bde0 msgr2=0x561336128ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.602+0000 7fd8c2a00640 1 --2- 172.21.15.45:0/733949657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56133612bde0 0x561336128ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.602+0000 7fd8c2a00640 1 -- 172.21.15.45:0/733949657 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561335d9a3f0 con 0x561336123260 2024-03-28T17:49:20.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.602+0000 7fd8c2a00640 1 --2- 172.21.15.45:0/733949657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561336123260 0x561336128980 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0005490 tx=0x7fd8b00051a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2f4fd23d85e0149 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:20.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.602+0000 7fd8ad600640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:49:20.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.603+0000 7fd8ad600640 1 -- 172.21.15.45:0/733949657 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0016050 con 0x561336123260 2024-03-28T17:49:20.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.603+0000 7fd8ad600640 1 -- 172.21.15.45:0/733949657 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fd8b0004ad0 con 0x561336123260 2024-03-28T17:49:20.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.603+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/733949657 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd8b000fd90 con 0x561336123260 2024-03-28T17:49:20.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.603+0000 7fd8ad600640 1 -- 172.21.15.45:0/733949657 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8b000fbc0 con 0x561336123260 2024-03-28T17:49:20.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.603+0000 7fd8cd3d3740 10 client.25511 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:20.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.603+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/733949657 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561335e647d0 con 0x561336123260 2024-03-28T17:49:20.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.603+0000 7fd8ad600640 1 -- 172.21.15.45:0/733949657 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8b000fd90 con 0x561336123260 2024-03-28T17:49:20.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.603+0000 7fd8ad600640 1 -- 172.21.15.45:0/733949657 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fd8b0029960 con 0x561336123260 2024-03-28T17:49:20.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.603+0000 7fd8cd3d3740 10 client.25511 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:20.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.603+0000 7fd8cd3d3740 10 client.25511 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T17:49:20.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.603+0000 7fd8cd3d3740 20 client.25511 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:20.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.603+0000 7fd8cd3d3740 1 --2- 172.21.15.45:0/733949657 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x56133613ba50 0x56133613de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.603+0000 7fd8cd3d3740 4 client.25511 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:49:20.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.603+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/733949657 --> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561336093800 con 0x56133613ba50 2024-03-28T17:49:20.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.604+0000 7fd8c3400640 1 --2- 172.21.15.45:0/733949657 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x56133613ba50 0x56133613de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.604+0000 7fd8c3400640 1 --2- 172.21.15.45:0/733949657 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x56133613ba50 0x56133613de40 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=976582a235335150 server_cookie=dc0984b6f9ad7417 in_seq=0 out_seq=0 2024-03-28T17:49:20.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.604+0000 7fd8ad600640 10 client.25511 ms_handle_connect on v2:172.21.15.71:6836/2008752655 2024-03-28T17:49:20.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.605+0000 7fd8ad600640 1 -- 172.21.15.45:0/733949657 <== mds.1 v2:172.21.15.71:6836/2008752655 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561336093800 con 0x56133613ba50 2024-03-28T17:49:20.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.605+0000 7fd8ad600640 10 client.25511 handle_command_reply: tid=0 2024-03-28T17:49:20.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.605+0000 7fd8cd3d3740 1 client.25511 shutdown 2024-03-28T17:49:20.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.605+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/733949657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561336123260 msgr2=0x561336128980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.605+0000 7fd8cd3d3740 1 --2- 172.21.15.45:0/733949657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561336123260 0x561336128980 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0005490 tx=0x7fd8b00051a0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.605+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/733949657 shutdown_connections 2024-03-28T17:49:20.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.605+0000 7fd8cd3d3740 1 --2- 172.21.15.45:0/733949657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56133612bde0 0x561336128ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.605+0000 7fd8cd3d3740 1 --2- 172.21.15.45:0/733949657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561336123260 0x561336128980 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.605+0000 7fd8cd3d3740 1 --2- 172.21.15.45:0/733949657 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x56133613ba50 0x56133613de40 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.605+0000 7fd8ad600640 0 client.25511 ms_handle_reset on v2:172.21.15.71:6836/2008752655 2024-03-28T17:49:20.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.605+0000 7fd8cd3d3740 1 --2- 172.21.15.45:0/733949657 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5613361228b0 0x56133612b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.605+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/733949657 >> 172.21.15.45:0/733949657 conn(0x56133611cf80 msgr2=0x56133611ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:20.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.605+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/733949657 shutdown_connections 2024-03-28T17:49:20.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.605+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/733949657 wait complete. 2024-03-28T17:49:20.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.605+0000 7fd8cd3d3740 20 client.25511 trim_cache size 0 max 16384 2024-03-28T17:49:20.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.619+0000 7fd8cd3d3740 1 Processor -- start 2024-03-28T17:49:20.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.619+0000 7fd8cd3d3740 1 -- start start 2024-03-28T17:49:20.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.619+0000 7fd8cd3d3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5613361228b0 0x56133612b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.619+0000 7fd8cd3d3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561336123260 0x56133612b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.619+0000 7fd8cd3d3740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56133612bde0 0x561336125c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.619+0000 7fd8cd3d3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c41b1620 con 0x561336123260 2024-03-28T17:49:20.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.619+0000 7fd8cd3d3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c41af120 con 0x5613361228b0 2024-03-28T17:49:20.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.619+0000 7fd8cd3d3740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c41ad1a0 con 0x56133612bde0 2024-03-28T17:49:20.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.620+0000 7fd8c3400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5613361228b0 0x56133612b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.620+0000 7fd8c3400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5613361228b0 0x56133612b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40530/0 (socket says 172.21.15.45:40530) 2024-03-28T17:49:20.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.620+0000 7fd8c3400640 1 -- 172.21.15.45:0/1017788317 learned_addr learned my addr 172.21.15.45:0/1017788317 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:20.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.620+0000 7fd8c3e00640 1 --2- 172.21.15.45:0/1017788317 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56133612bde0 0x561336125c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.620+0000 7fd8c2a00640 1 --2- 172.21.15.45:0/1017788317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561336123260 0x56133612b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.620+0000 7fd8c3400640 1 -- 172.21.15.45:0/1017788317 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56133612bde0 msgr2=0x561336125c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.620+0000 7fd8c3400640 1 --2- 172.21.15.45:0/1017788317 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56133612bde0 0x561336125c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.620+0000 7fd8c3400640 1 -- 172.21.15.45:0/1017788317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561336123260 msgr2=0x56133612b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.620+0000 7fd8c3400640 1 --2- 172.21.15.45:0/1017788317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561336123260 0x56133612b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.620+0000 7fd8c3400640 1 -- 172.21.15.45:0/1017788317 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56133612e1d0 con 0x5613361228b0 2024-03-28T17:49:20.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.620+0000 7fd8c3400640 1 --2- 172.21.15.45:0/1017788317 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5613361228b0 0x56133612b0b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x561336122380 tx=0x7fd8b4015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9199bdc742905310 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:20.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.620+0000 7fd8afe00640 1 -- 172.21.15.45:0/1017788317 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4021c60 con 0x5613361228b0 2024-03-28T17:49:20.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.620+0000 7fd8afe00640 1 -- 172.21.15.45:0/1017788317 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8b402e070 con 0x5613361228b0 2024-03-28T17:49:20.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.620+0000 7fd8afe00640 1 -- 172.21.15.45:0/1017788317 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b401f6e0 con 0x5613361228b0 2024-03-28T17:49:20.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.621+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/1017788317 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5613361228b0 msgr2=0x56133612b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.621+0000 7fd8cd3d3740 1 --2- 172.21.15.45:0/1017788317 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5613361228b0 0x56133612b0b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x561336122380 tx=0x7fd8b4015000 comp rx=0 tx=0).stop 2024-03-28T17:49:20.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.621+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/1017788317 shutdown_connections 2024-03-28T17:49:20.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.621+0000 7fd8cd3d3740 1 --2- 172.21.15.45:0/1017788317 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56133612bde0 0x561336125c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.621+0000 7fd8cd3d3740 1 --2- 172.21.15.45:0/1017788317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561336123260 0x56133612b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.621+0000 7fd8cd3d3740 1 --2- 172.21.15.45:0/1017788317 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5613361228b0 0x56133612b0b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.621+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/1017788317 >> 172.21.15.45:0/1017788317 conn(0x56133611cf80 msgr2=0x561336123f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:20.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.621+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/1017788317 shutdown_connections 2024-03-28T17:49:20.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.621+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/1017788317 wait complete. 2024-03-28T17:49:20.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.622+0000 7fd8cd3d3740 1 -- start start 2024-03-28T17:49:20.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.622+0000 7fd8cd3d3740 1 Processor -- start 2024-03-28T17:49:20.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.622+0000 7fd8cd3d3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613361228b0 0x56133612f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.622+0000 7fd8cd3d3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561336123260 0x56133612fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.622+0000 7fd8cd3d3740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56133612bde0 0x5613361301a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.622+0000 7fd8cd3d3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c41ad7a0 con 0x5613361228b0 2024-03-28T17:49:20.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.622+0000 7fd8cd3d3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c41aeac0 con 0x561336123260 2024-03-28T17:49:20.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.622+0000 7fd8cd3d3740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c41ade00 con 0x56133612bde0 2024-03-28T17:49:20.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.622+0000 7fd8cd3d3740 10 client.0 _ll_register_callbacks cb 0x7fd8b0012b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:20.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.622+0000 7fd8c2a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561336123260 0x56133612fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.622+0000 7fd8c2a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561336123260 0x56133612fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40542/0 (socket says 172.21.15.45:40542) 2024-03-28T17:49:20.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.622+0000 7fd8c2a00640 1 -- 172.21.15.45:0/1368102547 learned_addr learned my addr 172.21.15.45:0/1368102547 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:20.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.622+0000 7fd8c3400640 1 --2- 172.21.15.45:0/1368102547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613361228b0 0x56133612f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.623+0000 7fd8c3e00640 1 --2- 172.21.15.45:0/1368102547 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56133612bde0 0x5613361301a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.623+0000 7fd8c2a00640 1 -- 172.21.15.45:0/1368102547 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56133612bde0 msgr2=0x5613361301a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.623+0000 7fd8c2a00640 1 --2- 172.21.15.45:0/1368102547 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56133612bde0 0x5613361301a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.623+0000 7fd8c2a00640 1 -- 172.21.15.45:0/1368102547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613361228b0 msgr2=0x56133612f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.623+0000 7fd8c2a00640 1 --2- 172.21.15.45:0/1368102547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613361228b0 0x56133612f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.623+0000 7fd8c2a00640 1 -- 172.21.15.45:0/1368102547 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56133619e8d0 con 0x561336123260 2024-03-28T17:49:20.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.623+0000 7fd8c2a00640 1 --2- 172.21.15.45:0/1368102547 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561336123260 0x56133612fc60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x561336129f50 tx=0x7fd8b00324c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82195778931a509 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:20.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.623+0000 7fd8ad600640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:49:20.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.623+0000 7fd8ad600640 1 -- 172.21.15.45:0/1368102547 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0037030 con 0x561336123260 2024-03-28T17:49:20.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.623+0000 7fd8ad600640 1 -- 172.21.15.45:0/1368102547 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fd8b000fd60 con 0x561336123260 2024-03-28T17:49:20.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.623+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/1368102547 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd8b402e070 con 0x561336123260 2024-03-28T17:49:20.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.624+0000 7fd8ad600640 1 -- 172.21.15.45:0/1368102547 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8b0041050 con 0x561336123260 2024-03-28T17:49:20.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.624+0000 7fd8cd3d3740 10 client.25517 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:20.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.624+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/1368102547 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56133619fb80 con 0x561336123260 2024-03-28T17:49:20.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.624+0000 7fd8ad600640 1 -- 172.21.15.45:0/1368102547 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8b402e070 con 0x561336123260 2024-03-28T17:49:20.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.624+0000 7fd8ad600640 1 -- 172.21.15.45:0/1368102547 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fd8b00048d0 con 0x561336123260 2024-03-28T17:49:20.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.624+0000 7fd8cd3d3740 10 client.25517 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:20.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.624+0000 7fd8cd3d3740 10 client.25517 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T17:49:20.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.624+0000 7fd8cd3d3740 20 client.25517 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:20.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.624+0000 7fd8cd3d3740 1 --2- 172.21.15.45:0/1368102547 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x56133613ba50 0x5613361a22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.624+0000 7fd8cd3d3740 4 client.25517 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T17:49:20.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.624+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/1368102547 --> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56133608db50 con 0x56133613ba50 2024-03-28T17:49:20.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.624+0000 7fd8c3400640 1 --2- 172.21.15.45:0/1368102547 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x56133613ba50 0x5613361a22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.625+0000 7fd8c3400640 1 --2- 172.21.15.45:0/1368102547 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x56133613ba50 0x5613361a22c0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b25e9229742440a6 server_cookie=b1e6b5a27ba56c25 in_seq=0 out_seq=0 2024-03-28T17:49:20.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.625+0000 7fd8ad600640 10 client.25517 ms_handle_connect on v2:172.21.15.71:6836/2008752655 2024-03-28T17:49:20.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.625+0000 7fd8ad600640 1 -- 172.21.15.45:0/1368102547 <== mds.1 v2:172.21.15.71:6836/2008752655 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56133608db50 con 0x56133613ba50 2024-03-28T17:49:20.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.625+0000 7fd8ad600640 10 client.25517 handle_command_reply: tid=0 2024-03-28T17:49:20.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.625+0000 7fd8cd3d3740 1 client.25517 shutdown 2024-03-28T17:49:20.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.625+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/1368102547 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561336123260 msgr2=0x56133612fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.625+0000 7fd8cd3d3740 1 --2- 172.21.15.45:0/1368102547 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561336123260 0x56133612fc60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x561336129f50 tx=0x7fd8b00324c0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.625+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/1368102547 shutdown_connections 2024-03-28T17:49:20.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.625+0000 7fd8cd3d3740 1 --2- 172.21.15.45:0/1368102547 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x56133613ba50 0x5613361a22c0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.625+0000 7fd8cd3d3740 1 --2- 172.21.15.45:0/1368102547 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56133612bde0 0x5613361301a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.625+0000 7fd8cd3d3740 1 --2- 172.21.15.45:0/1368102547 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561336123260 0x56133612fc60 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.625+0000 7fd8cd3d3740 1 --2- 172.21.15.45:0/1368102547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613361228b0 0x56133612f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.625+0000 7fd8ad600640 0 client.25517 ms_handle_reset on v2:172.21.15.71:6836/2008752655 2024-03-28T17:49:20.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.625+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/1368102547 >> 172.21.15.45:0/1368102547 conn(0x56133611cf80 msgr2=0x561336123640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:20.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.625+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/1368102547 shutdown_connections 2024-03-28T17:49:20.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.625+0000 7fd8cd3d3740 1 -- 172.21.15.45:0/1368102547 wait complete. 2024-03-28T17:49:20.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.626+0000 7fd8cd3d3740 20 client.25517 trim_cache size 0 max 16384 2024-03-28T17:49:20.631 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-28T17:49:20.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.626+0000 7fd8c9000640 1 -- 172.21.15.45:0/271631631 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd8a80a2a80 msgr2=0x7fd8a80a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.626+0000 7fd8c9000640 1 --2- 172.21.15.45:0/271631631 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd8a80a2a80 0x7fd8a80a4f40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x56133611dff0 tx=0x7fd8b400b550 comp rx=0 tx=0).stop 2024-03-28T17:49:20.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.626+0000 7fd8c9000640 1 -- 172.21.15.45:0/271631631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8c4109730 msgr2=0x7fd8c419e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.626+0000 7fd8c9000640 1 --2- 172.21.15.45:0/271631631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8c4109730 0x7fd8c419e840 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc009bf0 tx=0x7fd8bc034670 comp rx=0 tx=0).stop 2024-03-28T17:49:20.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.626+0000 7fd8c9000640 1 -- 172.21.15.45:0/271631631 shutdown_connections 2024-03-28T17:49:20.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.626+0000 7fd8c9000640 1 --2- 172.21.15.45:0/271631631 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd8a80a2a80 0x7fd8a80a4f40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.626+0000 7fd8c9000640 1 --2- 172.21.15.45:0/271631631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8c4109730 0x7fd8c419e840 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.626+0000 7fd8c9000640 1 --2- 172.21.15.45:0/271631631 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd8c41040e0 0x7fd8c419a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.626+0000 7fd8c9000640 1 --2- 172.21.15.45:0/271631631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8c4103720 0x7fd8c4199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.626+0000 7fd8c9000640 1 -- 172.21.15.45:0/271631631 >> 172.21.15.45:0/271631631 conn(0x7fd8c40ff1d0 msgr2=0x7fd8c40ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:20.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.626+0000 7fd8c9000640 1 -- 172.21.15.45:0/271631631 shutdown_connections 2024-03-28T17:49:20.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.626+0000 7fd8c9000640 1 -- 172.21.15.45:0/271631631 wait complete. 2024-03-28T17:49:20.641 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:20 smithi045 ceph-mon[31773]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:49:20.641 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:20 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/2690843274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:49:20.645 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T17:49:20.645 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T17:49:20.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.870+0000 7fced8800640 1 -- 172.21.15.45:0/3911484188 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fced0101ff0 msgr2=0x7fced010e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.870+0000 7fced8800640 1 --2- 172.21.15.45:0/3911484188 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fced0101ff0 0x7fced010e500 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcecc009be0 tx=0x7fcecc02bef0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.870+0000 7fced8800640 1 -- 172.21.15.45:0/3911484188 shutdown_connections 2024-03-28T17:49:20.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.870+0000 7fced8800640 1 --2- 172.21.15.45:0/3911484188 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fced0101ff0 0x7fced010e500 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.870+0000 7fced8800640 1 --2- 172.21.15.45:0/3911484188 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fced0101630 0x7fced0101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.870+0000 7fced8800640 1 --2- 172.21.15.45:0/3911484188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fced0107640 0x7fced0107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.870+0000 7fced8800640 1 -- 172.21.15.45:0/3911484188 >> 172.21.15.45:0/3911484188 conn(0x7fced0078aa0 msgr2=0x7fced0078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:20.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.870+0000 7fced8800640 1 -- 172.21.15.45:0/3911484188 shutdown_connections 2024-03-28T17:49:20.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.871+0000 7fced8800640 1 -- 172.21.15.45:0/3911484188 wait complete. 2024-03-28T17:49:20.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.872+0000 7fced8800640 1 Processor -- start 2024-03-28T17:49:20.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.872+0000 7fced8800640 1 -- start start 2024-03-28T17:49:20.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.872+0000 7fced8800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fced0101630 0x7fced019a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.872+0000 7fced8800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fced0101ff0 0x7fced019a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.872+0000 7fced8800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fced0107640 0x7fced019e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.872+0000 7fced8800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced0113c20 con 0x7fced0101ff0 2024-03-28T17:49:20.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.872+0000 7fced8800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced0113aa0 con 0x7fced0107640 2024-03-28T17:49:20.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.872+0000 7fced8800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced0113da0 con 0x7fced0101630 2024-03-28T17:49:20.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.873+0000 7fced5e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fced0101630 0x7fced019a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.873+0000 7fced5e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fced0101630 0x7fced019a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:56624/0 (socket says 172.21.15.45:56624) 2024-03-28T17:49:20.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.873+0000 7fced5e00640 1 -- 172.21.15.45:0/3962710056 learned_addr learned my addr 172.21.15.45:0/3962710056 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:20.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.873+0000 7fced5400640 1 --2- 172.21.15.45:0/3962710056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fced0101ff0 0x7fced019a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.873+0000 7fced6800640 1 --2- 172.21.15.45:0/3962710056 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fced0107640 0x7fced019e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.873+0000 7fced5e00640 1 -- 172.21.15.45:0/3962710056 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fced0107640 msgr2=0x7fced019e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.873+0000 7fced5e00640 1 --2- 172.21.15.45:0/3962710056 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fced0107640 0x7fced019e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.873+0000 7fced5e00640 1 -- 172.21.15.45:0/3962710056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fced0101ff0 msgr2=0x7fced019a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.873+0000 7fced5e00640 1 --2- 172.21.15.45:0/3962710056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fced0101ff0 0x7fced019a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.873+0000 7fced5e00640 1 -- 172.21.15.45:0/3962710056 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced019f070 con 0x7fced0101630 2024-03-28T17:49:20.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.873+0000 7fced5e00640 1 --2- 172.21.15.45:0/3962710056 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fced0101630 0x7fced019a080 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcec400d950 tx=0x7fcec400dd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:20.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.874+0000 7fcecaa00640 1 -- 172.21.15.45:0/3962710056 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec4017070 con 0x7fced0101630 2024-03-28T17:49:20.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.874+0000 7fcecaa00640 1 -- 172.21.15.45:0/3962710056 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcec400de80 con 0x7fced0101630 2024-03-28T17:49:20.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.874+0000 7fcecaa00640 1 -- 172.21.15.45:0/3962710056 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec400f620 con 0x7fced0101630 2024-03-28T17:49:20.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.874+0000 7fced8800640 1 -- 172.21.15.45:0/3962710056 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced019f360 con 0x7fced0101630 2024-03-28T17:49:20.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.874+0000 7fced8800640 1 -- 172.21.15.45:0/3962710056 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced01a6bf0 con 0x7fced0101630 2024-03-28T17:49:20.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.877+0000 7fcecaa00640 1 -- 172.21.15.45:0/3962710056 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 26) v1 ==== 137466+0+0 (secure 0 0 0) 0x7fcec4009c30 con 0x7fced0101630 2024-03-28T17:49:20.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.877+0000 7fcedb463740 1 Processor -- start 2024-03-28T17:49:20.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.877+0000 7fcedb463740 1 -- start start 2024-03-28T17:49:20.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.877+0000 7fcedb463740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748d6d0 0x56137748dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.878+0000 7fcedb463740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56137748e080 0x561377496630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.878+0000 7fcedb463740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561377496c00 0x561377499050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.878+0000 7fcedb463740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613773f12c0 con 0x561377496c00 2024-03-28T17:49:20.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.878+0000 7fcedb463740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x561377399c90 con 0x56137748e080 2024-03-28T17:49:20.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.878+0000 7fcedb463740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613773ed360 con 0x56137748d6d0 2024-03-28T17:49:20.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.878+0000 7fced6800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748d6d0 0x56137748dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.878+0000 7fced5e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56137748e080 0x561377496630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.878+0000 7fced6800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748d6d0 0x56137748dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:56628/0 (socket says 172.21.15.45:56628) 2024-03-28T17:49:20.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.878+0000 7fced6800640 1 -- 172.21.15.45:0/666706282 learned_addr learned my addr 172.21.15.45:0/666706282 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:20.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.878+0000 7fced5400640 1 --2- 172.21.15.45:0/666706282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561377496c00 0x561377499050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.878+0000 7fcecaa00640 1 --2- 172.21.15.45:0/3962710056 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fceb80a2a80 0x7fceb80a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.878+0000 7fced5e00640 1 -- 172.21.15.45:0/666706282 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748d6d0 msgr2=0x56137748dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.878+0000 7fced5e00640 1 --2- 172.21.15.45:0/666706282 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748d6d0 0x56137748dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.878+0000 7fced5e00640 1 -- 172.21.15.45:0/666706282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561377496c00 msgr2=0x561377499050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.878+0000 7fced5e00640 1 --2- 172.21.15.45:0/666706282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561377496c00 0x561377499050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.878+0000 7fced5e00640 1 -- 172.21.15.45:0/666706282 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5613773be080 con 0x56137748e080 2024-03-28T17:49:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.879+0000 7fced6800640 1 --2- 172.21.15.45:0/3962710056 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fceb80a2a80 0x7fceb80a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.879+0000 7fcecaa00640 1 -- 172.21.15.45:0/3962710056 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fcec40ceb20 con 0x7fced0101630 2024-03-28T17:49:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.879+0000 7fced5e00640 1 --2- 172.21.15.45:0/666706282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56137748e080 0x561377496630 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcec40cc180 tx=0x7fcec4022000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68f695e705fe3daf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.879+0000 7fcec3000640 1 -- 172.21.15.45:0/666706282 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec409ab20 con 0x56137748e080 2024-03-28T17:49:20.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.879+0000 7fcec3000640 1 -- 172.21.15.45:0/666706282 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcec4084520 con 0x56137748e080 2024-03-28T17:49:20.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.879+0000 7fcec3000640 1 -- 172.21.15.45:0/666706282 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec409a940 con 0x56137748e080 2024-03-28T17:49:20.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.880+0000 7fcedb463740 1 -- 172.21.15.45:0/666706282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56137748e080 msgr2=0x561377496630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.880+0000 7fcedb463740 1 --2- 172.21.15.45:0/666706282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56137748e080 0x561377496630 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcec40cc180 tx=0x7fcec4022000 comp rx=0 tx=0).stop 2024-03-28T17:49:20.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.880+0000 7fced6800640 1 --2- 172.21.15.45:0/3962710056 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fceb80a2a80 0x7fceb80a4f40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcecc02c500 tx=0x7fcecc009500 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:20.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.880+0000 7fcedb463740 1 -- 172.21.15.45:0/666706282 shutdown_connections 2024-03-28T17:49:20.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.880+0000 7fcedb463740 1 --2- 172.21.15.45:0/666706282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561377496c00 0x561377499050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.880+0000 7fcedb463740 1 --2- 172.21.15.45:0/666706282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56137748e080 0x561377496630 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.880+0000 7fcedb463740 1 --2- 172.21.15.45:0/666706282 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748d6d0 0x56137748dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.880+0000 7fcedb463740 1 -- 172.21.15.45:0/666706282 >> 172.21.15.45:0/666706282 conn(0x561377487da0 msgr2=0x561377488180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:20.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.880+0000 7fcedb463740 1 -- 172.21.15.45:0/666706282 shutdown_connections 2024-03-28T17:49:20.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.880+0000 7fcedb463740 1 -- 172.21.15.45:0/666706282 wait complete. 2024-03-28T17:49:20.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.880+0000 7fcedb463740 1 -- start start 2024-03-28T17:49:20.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.881+0000 7fcedb463740 1 Processor -- start 2024-03-28T17:49:20.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.881+0000 7fcedb463740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56137748d6d0 0x561377496430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.881+0000 7fcedb463740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748e080 0x5613774937c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.881+0000 7fcedb463740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561377496c00 0x561377493d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.881+0000 7fcedb463740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561377499ab0 con 0x56137748d6d0 2024-03-28T17:49:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.881+0000 7fcedb463740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613773ad0d0 con 0x561377496c00 2024-03-28T17:49:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.881+0000 7fcedb463740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x561377499c30 con 0x56137748e080 2024-03-28T17:49:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.881+0000 7fcedb463740 10 client.0 _ll_register_callbacks cb 0x5613773b1e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.881+0000 7fced6800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561377496c00 0x561377493d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.881+0000 7fced5400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748e080 0x5613774937c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.881+0000 7fced5400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748e080 0x5613774937c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:56632/0 (socket says 172.21.15.45:56632) 2024-03-28T17:49:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.881+0000 7fced5400640 1 -- 172.21.15.45:0/996452729 learned_addr learned my addr 172.21.15.45:0/996452729 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.881+0000 7fced5e00640 1 --2- 172.21.15.45:0/996452729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56137748d6d0 0x561377496430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.882+0000 7fced6800640 1 -- 172.21.15.45:0/996452729 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748e080 msgr2=0x5613774937c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.882+0000 7fced6800640 1 --2- 172.21.15.45:0/996452729 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748e080 0x5613774937c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.882+0000 7fced6800640 1 -- 172.21.15.45:0/996452729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56137748d6d0 msgr2=0x561377496430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.882+0000 7fced6800640 1 --2- 172.21.15.45:0/996452729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56137748d6d0 0x561377496430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.882+0000 7fced6800640 1 -- 172.21.15.45:0/996452729 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5613771053f0 con 0x561377496c00 2024-03-28T17:49:20.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.882+0000 7fced5400640 1 --2- 172.21.15.45:0/996452729 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748e080 0x5613774937c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:49:20.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.882+0000 7fced6800640 1 --2- 172.21.15.45:0/996452729 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561377496c00 0x561377493d00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcecc036040 tx=0x7fcecc005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41aad41f65ee77ea server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:20.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.882+0000 7fceb7400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:49:20.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.882+0000 7fceb7400640 1 -- 172.21.15.45:0/996452729 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc050070 con 0x561377496c00 2024-03-28T17:49:20.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.883+0000 7fcedb463740 1 -- 172.21.15.45:0/996452729 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcec4084520 con 0x561377496c00 2024-03-28T17:49:20.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.883+0000 7fceb7400640 1 -- 172.21.15.45:0/996452729 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fcecc041980 con 0x561377496c00 2024-03-28T17:49:20.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.883+0000 7fceb7400640 1 -- 172.21.15.45:0/996452729 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcecc05c050 con 0x561377496c00 2024-03-28T17:49:20.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.883+0000 7fcedb463740 10 client.25529 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:20.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.883+0000 7fcedb463740 1 -- 172.21.15.45:0/996452729 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5613771cf7d0 con 0x561377496c00 2024-03-28T17:49:20.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.883+0000 7fceb7400640 1 -- 172.21.15.45:0/996452729 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcec4084520 con 0x561377496c00 2024-03-28T17:49:20.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.883+0000 7fceb7400640 1 -- 172.21.15.45:0/996452729 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fcecc0043e0 con 0x561377496c00 2024-03-28T17:49:20.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.883+0000 7fcedb463740 10 client.25529 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:20.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.883+0000 7fcedb463740 10 client.25529 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:49:20.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.883+0000 7fcedb463740 20 client.25529 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:20.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.883+0000 7fcedb463740 1 --2- 172.21.15.45:0/996452729 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5613774a6880 0x5613774a8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.883+0000 7fcedb463740 4 client.25529 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:49:20.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.883+0000 7fcedb463740 1 -- 172.21.15.45:0/996452729 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5613773fe800 con 0x5613774a6880 2024-03-28T17:49:20.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.884+0000 7fced5e00640 1 --2- 172.21.15.45:0/996452729 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5613774a6880 0x5613774a8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.884+0000 7fced5e00640 1 --2- 172.21.15.45:0/996452729 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5613774a6880 0x5613774a8ca0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20c2852d02612ae9 server_cookie=2921bc52271f1331 in_seq=0 out_seq=0 2024-03-28T17:49:20.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.884+0000 7fceb7400640 10 client.25529 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.885+0000 7fceb7400640 1 -- 172.21.15.45:0/996452729 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5613773fe800 con 0x5613774a6880 2024-03-28T17:49:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.885+0000 7fceb7400640 10 client.25529 handle_command_reply: tid=0 2024-03-28T17:49:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.885+0000 7fcedb463740 1 client.25529 shutdown 2024-03-28T17:49:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.885+0000 7fcedb463740 1 -- 172.21.15.45:0/996452729 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561377496c00 msgr2=0x561377493d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.885+0000 7fcedb463740 1 --2- 172.21.15.45:0/996452729 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561377496c00 0x561377493d00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcecc036040 tx=0x7fcecc005fd0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.885+0000 7fcedb463740 1 -- 172.21.15.45:0/996452729 shutdown_connections 2024-03-28T17:49:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.885+0000 7fcedb463740 1 --2- 172.21.15.45:0/996452729 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5613774a6880 0x5613774a8ca0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.885+0000 7fcedb463740 1 --2- 172.21.15.45:0/996452729 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561377496c00 0x561377493d00 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.885+0000 7fcedb463740 1 --2- 172.21.15.45:0/996452729 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748e080 0x5613774937c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.885+0000 7fcedb463740 1 --2- 172.21.15.45:0/996452729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56137748d6d0 0x561377496430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.885+0000 7fceb7400640 0 client.25529 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.885+0000 7fcedb463740 1 -- 172.21.15.45:0/996452729 >> 172.21.15.45:0/996452729 conn(0x561377487da0 msgr2=0x561377489af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.885+0000 7fcedb463740 1 -- 172.21.15.45:0/996452729 shutdown_connections 2024-03-28T17:49:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.885+0000 7fcedb463740 1 -- 172.21.15.45:0/996452729 wait complete. 2024-03-28T17:49:20.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.885+0000 7fcedb463740 20 client.25529 trim_cache size 0 max 16384 2024-03-28T17:49:20.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.899+0000 7fcedb463740 1 Processor -- start 2024-03-28T17:49:20.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.899+0000 7fcedb463740 1 -- start start 2024-03-28T17:49:20.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.899+0000 7fcedb463740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748d6d0 0x56137748eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.899+0000 7fcedb463740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56137748e080 0x561377495ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.899+0000 7fcedb463740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561377496c00 0x561377496430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.899+0000 7fcedb463740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced01b18d0 con 0x561377496c00 2024-03-28T17:49:20.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.899+0000 7fcedb463740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced01af3d0 con 0x56137748e080 2024-03-28T17:49:20.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.899+0000 7fcedb463740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced01ad450 con 0x56137748d6d0 2024-03-28T17:49:20.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.900+0000 7fced5400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56137748e080 0x561377495ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.900+0000 7fced5e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748d6d0 0x56137748eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.900+0000 7fced5400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56137748e080 0x561377495ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40580/0 (socket says 172.21.15.45:40580) 2024-03-28T17:49:20.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.900+0000 7fced5400640 1 -- 172.21.15.45:0/1236118309 learned_addr learned my addr 172.21.15.45:0/1236118309 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:20.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.900+0000 7fced6800640 1 --2- 172.21.15.45:0/1236118309 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561377496c00 0x561377496430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.900+0000 7fced5e00640 1 -- 172.21.15.45:0/1236118309 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56137748e080 msgr2=0x561377495ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.900+0000 7fced5e00640 1 --2- 172.21.15.45:0/1236118309 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56137748e080 0x561377495ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.900+0000 7fced5e00640 1 -- 172.21.15.45:0/1236118309 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561377496c00 msgr2=0x561377496430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.900+0000 7fced5e00640 1 --2- 172.21.15.45:0/1236118309 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561377496c00 0x561377496430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.900+0000 7fced5e00640 1 -- 172.21.15.45:0/1236118309 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561377491d30 con 0x56137748d6d0 2024-03-28T17:49:20.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.900+0000 7fced5e00640 1 --2- 172.21.15.45:0/1236118309 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748d6d0 0x56137748eb70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcec40cc1b0 tx=0x7fcec4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc31dd26ea97b081 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:20.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.900+0000 7fcec2600640 1 -- 172.21.15.45:0/1236118309 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec40d4070 con 0x56137748d6d0 2024-03-28T17:49:20.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.901+0000 7fcec2600640 1 -- 172.21.15.45:0/1236118309 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcec409b6e0 con 0x56137748d6d0 2024-03-28T17:49:20.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.901+0000 7fcec2600640 1 -- 172.21.15.45:0/1236118309 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec409b240 con 0x56137748d6d0 2024-03-28T17:49:20.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.901+0000 7fcedb463740 1 -- 172.21.15.45:0/1236118309 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748d6d0 msgr2=0x56137748eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.901+0000 7fcedb463740 1 --2- 172.21.15.45:0/1236118309 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748d6d0 0x56137748eb70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcec40cc1b0 tx=0x7fcec4000bc0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.901+0000 7fcedb463740 1 -- 172.21.15.45:0/1236118309 shutdown_connections 2024-03-28T17:49:20.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.901+0000 7fcedb463740 1 --2- 172.21.15.45:0/1236118309 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561377496c00 0x561377496430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.901+0000 7fcedb463740 1 --2- 172.21.15.45:0/1236118309 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56137748e080 0x561377495ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.901+0000 7fcedb463740 1 --2- 172.21.15.45:0/1236118309 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748d6d0 0x56137748eb70 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.901+0000 7fcedb463740 1 -- 172.21.15.45:0/1236118309 >> 172.21.15.45:0/1236118309 conn(0x561377487da0 msgr2=0x56137748e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:20.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.901+0000 7fcedb463740 1 -- 172.21.15.45:0/1236118309 shutdown_connections 2024-03-28T17:49:20.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.901+0000 7fcedb463740 1 -- 172.21.15.45:0/1236118309 wait complete. 2024-03-28T17:49:20.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.902+0000 7fcedb463740 1 -- start start 2024-03-28T17:49:20.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.902+0000 7fcedb463740 1 Processor -- start 2024-03-28T17:49:20.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.902+0000 7fcedb463740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748d6d0 0x56137749aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.902+0000 7fcedb463740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56137748e080 0x56137749afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.902+0000 7fcedb463740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561377496c00 0x561377501130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.902+0000 7fcedb463740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced01ada50 con 0x56137748e080 2024-03-28T17:49:20.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.903+0000 7fcedb463740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced01aed70 con 0x561377496c00 2024-03-28T17:49:20.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.903+0000 7fcedb463740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced01ae0b0 con 0x56137748d6d0 2024-03-28T17:49:20.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.903+0000 7fcedb463740 10 client.0 _ll_register_callbacks cb 0x7fcecc033eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:20.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.903+0000 7fced5e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748d6d0 0x56137749aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.903+0000 7fced5e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748d6d0 0x56137749aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:56654/0 (socket says 172.21.15.45:56654) 2024-03-28T17:49:20.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.903+0000 7fced5e00640 1 -- 172.21.15.45:0/2979675062 learned_addr learned my addr 172.21.15.45:0/2979675062 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:20.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.903+0000 7fced6800640 1 --2- 172.21.15.45:0/2979675062 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561377496c00 0x561377501130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.903+0000 7fced5400640 1 --2- 172.21.15.45:0/2979675062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56137748e080 0x56137749afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.903+0000 7fced5e00640 1 -- 172.21.15.45:0/2979675062 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561377496c00 msgr2=0x561377501130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.903+0000 7fced5e00640 1 --2- 172.21.15.45:0/2979675062 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561377496c00 0x561377501130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.903+0000 7fced5e00640 1 -- 172.21.15.45:0/2979675062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56137748e080 msgr2=0x56137749afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:20.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.903+0000 7fced5e00640 1 --2- 172.21.15.45:0/2979675062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56137748e080 0x56137749afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.903+0000 7fced5e00640 1 -- 172.21.15.45:0/2979675062 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561377501740 con 0x56137748d6d0 2024-03-28T17:49:20.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.903+0000 7fced6800640 1 --2- 172.21.15.45:0/2979675062 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561377496c00 0x561377501130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:49:20.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.903+0000 7fced5e00640 1 --2- 172.21.15.45:0/2979675062 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748d6d0 0x56137749aa60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcec408d3d0 tx=0x7fcec409aee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=898263f678da6159 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:20.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.903+0000 7fceb7400640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:49:20.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.904+0000 7fceb7400640 1 -- 172.21.15.45:0/2979675062 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec40d4070 con 0x56137748d6d0 2024-03-28T17:49:20.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.904+0000 7fceb7400640 1 -- 172.21.15.45:0/2979675062 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fcec40992e0 con 0x56137748d6d0 2024-03-28T17:49:20.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.904+0000 7fcedb463740 1 -- 172.21.15.45:0/2979675062 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcec409b6e0 con 0x56137748d6d0 2024-03-28T17:49:20.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.904+0000 7fceb7400640 1 -- 172.21.15.45:0/2979675062 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcec4020040 con 0x56137748d6d0 2024-03-28T17:49:20.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.904+0000 7fcedb463740 10 client.25510 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:20.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.904+0000 7fcedb463740 1 -- 172.21.15.45:0/2979675062 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561377502a40 con 0x56137748d6d0 2024-03-28T17:49:20.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.904+0000 7fceb7400640 1 -- 172.21.15.45:0/2979675062 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcec40b7030 con 0x56137748d6d0 2024-03-28T17:49:20.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.904+0000 7fceb7400640 1 -- 172.21.15.45:0/2979675062 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fcec40a32f0 con 0x56137748d6d0 2024-03-28T17:49:20.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.904+0000 7fcedb463740 10 client.25510 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:20.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.904+0000 7fcedb463740 10 client.25510 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:49:20.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.904+0000 7fcedb463740 20 client.25510 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:20.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.904+0000 7fcedb463740 1 --2- 172.21.15.45:0/2979675062 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x561377502cd0 0x5613775050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:20.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.904+0000 7fcedb463740 4 client.25510 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T17:49:20.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.904+0000 7fcedb463740 1 -- 172.21.15.45:0/2979675062 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561377494090 con 0x561377502cd0 2024-03-28T17:49:20.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.905+0000 7fced5400640 1 --2- 172.21.15.45:0/2979675062 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x561377502cd0 0x5613775050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:20.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.905+0000 7fced5400640 1 --2- 172.21.15.45:0/2979675062 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x561377502cd0 0x5613775050c0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e19befb582c0e485 server_cookie=2689b95e9dab9ba5 in_seq=0 out_seq=0 2024-03-28T17:49:20.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.905+0000 7fceb7400640 10 client.25510 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:20.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.905+0000 7fceb7400640 1 -- 172.21.15.45:0/2979675062 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561377494090 con 0x561377502cd0 2024-03-28T17:49:20.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.905+0000 7fceb7400640 10 client.25510 handle_command_reply: tid=0 2024-03-28T17:49:20.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.905+0000 7fcedb463740 1 client.25510 shutdown 2024-03-28T17:49:20.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.905+0000 7fcedb463740 1 -- 172.21.15.45:0/2979675062 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748d6d0 msgr2=0x56137749aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.905+0000 7fcedb463740 1 --2- 172.21.15.45:0/2979675062 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748d6d0 0x56137749aa60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcec408d3d0 tx=0x7fcec409aee0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.905+0000 7fcedb463740 1 -- 172.21.15.45:0/2979675062 shutdown_connections 2024-03-28T17:49:20.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.906+0000 7fcedb463740 1 --2- 172.21.15.45:0/2979675062 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x561377502cd0 0x5613775050c0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.906+0000 7fcedb463740 1 --2- 172.21.15.45:0/2979675062 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561377496c00 0x561377501130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.906+0000 7fcedb463740 1 --2- 172.21.15.45:0/2979675062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56137748e080 0x56137749afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.906+0000 7fcedb463740 1 --2- 172.21.15.45:0/2979675062 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56137748d6d0 0x56137749aa60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.906+0000 7fceb7400640 0 client.25510 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:20.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.906+0000 7fcedb463740 1 -- 172.21.15.45:0/2979675062 >> 172.21.15.45:0/2979675062 conn(0x561377487da0 msgr2=0x56137748ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:20.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.906+0000 7fcedb463740 1 -- 172.21.15.45:0/2979675062 shutdown_connections 2024-03-28T17:49:20.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.906+0000 7fcedb463740 1 -- 172.21.15.45:0/2979675062 wait complete. 2024-03-28T17:49:20.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.906+0000 7fcedb463740 20 client.25510 trim_cache size 0 max 16384 2024-03-28T17:49:20.911 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-28T17:49:20.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.906+0000 7fced8800640 1 -- 172.21.15.45:0/3962710056 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fceb80a2a80 msgr2=0x7fceb80a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.906+0000 7fced8800640 1 --2- 172.21.15.45:0/3962710056 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fceb80a2a80 0x7fceb80a4f40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcecc02c500 tx=0x7fcecc009500 comp rx=0 tx=0).stop 2024-03-28T17:49:20.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.906+0000 7fced8800640 1 -- 172.21.15.45:0/3962710056 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fced0101630 msgr2=0x7fced019a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:20.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.906+0000 7fced8800640 1 --2- 172.21.15.45:0/3962710056 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fced0101630 0x7fced019a080 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcec400d950 tx=0x7fcec400dd50 comp rx=0 tx=0).stop 2024-03-28T17:49:20.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.907+0000 7fced8800640 1 -- 172.21.15.45:0/3962710056 shutdown_connections 2024-03-28T17:49:20.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.907+0000 7fced8800640 1 --2- 172.21.15.45:0/3962710056 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fceb80a2a80 0x7fceb80a4f40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.907+0000 7fced8800640 1 --2- 172.21.15.45:0/3962710056 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fced0107640 0x7fced019e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.907+0000 7fced8800640 1 --2- 172.21.15.45:0/3962710056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fced0101ff0 0x7fced019a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.907+0000 7fced8800640 1 --2- 172.21.15.45:0/3962710056 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fced0101630 0x7fced019a080 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:20.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.907+0000 7fced8800640 1 -- 172.21.15.45:0/3962710056 >> 172.21.15.45:0/3962710056 conn(0x7fced0078aa0 msgr2=0x7fced0103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:20.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.907+0000 7fced8800640 1 -- 172.21.15.45:0/3962710056 shutdown_connections 2024-03-28T17:49:20.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:20.907+0000 7fced8800640 1 -- 172.21.15.45:0/3962710056 wait complete. 2024-03-28T17:49:21.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:20 smithi071 ceph-mon[36597]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:49:21.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:20 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/2690843274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:49:21.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:20 smithi151 ceph-mon[36481]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:49:21.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:20 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/2690843274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:49:21.925 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T17:49:21.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T17:49:21.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T17:49:22.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.165+0000 7f76f3200640 1 -- 172.21.15.45:0/2866456410 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76ec103720 msgr2=0x7f76ec103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:22.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.165+0000 7f76f3200640 1 --2- 172.21.15.45:0/2866456410 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76ec103720 0x7f76ec103ba0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f76dc009930 tx=0x7f76dc02bc30 comp rx=0 tx=0).stop 2024-03-28T17:49:22.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.165+0000 7f76f3200640 1 -- 172.21.15.45:0/2866456410 shutdown_connections 2024-03-28T17:49:22.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.165+0000 7f76f3200640 1 --2- 172.21.15.45:0/2866456410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76ec1040e0 0x7f76ec1105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.165+0000 7f76f3200640 1 --2- 172.21.15.45:0/2866456410 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76ec103720 0x7f76ec103ba0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.165+0000 7f76f3200640 1 --2- 172.21.15.45:0/2866456410 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76ec109730 0x7f76ec109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.165+0000 7f76f3200640 1 -- 172.21.15.45:0/2866456410 >> 172.21.15.45:0/2866456410 conn(0x7f76ec0ff1b0 msgr2=0x7f76ec1015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:22.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.165+0000 7f76f3200640 1 -- 172.21.15.45:0/2866456410 shutdown_connections 2024-03-28T17:49:22.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.165+0000 7f76f3200640 1 -- 172.21.15.45:0/2866456410 wait complete. 2024-03-28T17:49:22.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.166+0000 7f76f3200640 1 Processor -- start 2024-03-28T17:49:22.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.166+0000 7f76f3200640 1 -- start start 2024-03-28T17:49:22.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.166+0000 7f76f3200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76ec103720 0x7f76ec19a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.166+0000 7f76f3200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76ec1040e0 0x7f76ec19a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.167+0000 7f76f3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76ec109730 0x7f76ec19ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.167+0000 7f76f3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec113b90 con 0x7f76ec109730 2024-03-28T17:49:22.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.167+0000 7f76f3200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec113a10 con 0x7f76ec1040e0 2024-03-28T17:49:22.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.167+0000 7f76f3200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec113d10 con 0x7f76ec103720 2024-03-28T17:49:22.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.167+0000 7f76f1400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76ec1040e0 0x7f76ec19a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.167+0000 7f76f1e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76ec103720 0x7f76ec19a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.167+0000 7f76f1e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76ec103720 0x7f76ec19a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:56672/0 (socket says 172.21.15.45:56672) 2024-03-28T17:49:22.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.167+0000 7f76f1e00640 1 -- 172.21.15.45:0/1937005420 learned_addr learned my addr 172.21.15.45:0/1937005420 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:22.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.167+0000 7f76f1400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76ec1040e0 0x7f76ec19a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40618/0 (socket says 172.21.15.45:40618) 2024-03-28T17:49:22.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.167+0000 7f76f2800640 1 --2- 172.21.15.45:0/1937005420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76ec109730 0x7f76ec19ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.167+0000 7f76f1400640 1 -- 172.21.15.45:0/1937005420 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76ec103720 msgr2=0x7f76ec19a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:22.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.167+0000 7f76f1400640 1 --2- 172.21.15.45:0/1937005420 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76ec103720 0x7f76ec19a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.167+0000 7f76f1400640 1 -- 172.21.15.45:0/1937005420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76ec109730 msgr2=0x7f76ec19ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:22.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.167+0000 7f76f1400640 1 --2- 172.21.15.45:0/1937005420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76ec109730 0x7f76ec19ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.167+0000 7f76f1400640 1 -- 172.21.15.45:0/1937005420 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ec19f340 con 0x7f76ec1040e0 2024-03-28T17:49:22.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.167+0000 7f76f2800640 1 --2- 172.21.15.45:0/1937005420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76ec109730 0x7f76ec19ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:49:22.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.167+0000 7f76f1e00640 1 --2- 172.21.15.45:0/1937005420 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76ec103720 0x7f76ec19a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:49:22.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.168+0000 7f76f1400640 1 --2- 172.21.15.45:0/1937005420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76ec1040e0 0x7f76ec19a890 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f76dc036040 tx=0x7f76dc005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:22.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.168+0000 7f76e6a00640 1 -- 172.21.15.45:0/1937005420 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76dc043020 con 0x7f76ec1040e0 2024-03-28T17:49:22.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.168+0000 7f76e6a00640 1 -- 172.21.15.45:0/1937005420 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76dc005000 con 0x7f76ec1040e0 2024-03-28T17:49:22.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.168+0000 7f76f3200640 1 -- 172.21.15.45:0/1937005420 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76ec19f5d0 con 0x7f76ec1040e0 2024-03-28T17:49:22.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.168+0000 7f76e6a00640 1 -- 172.21.15.45:0/1937005420 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76dc003af0 con 0x7f76ec1040e0 2024-03-28T17:49:22.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.168+0000 7f76f3200640 1 -- 172.21.15.45:0/1937005420 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76ec1a6ec0 con 0x7f76ec1040e0 2024-03-28T17:49:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.171+0000 7f76e6a00640 1 -- 172.21.15.45:0/1937005420 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 26) v1 ==== 137466+0+0 (secure 0 0 0) 0x7f76dc046020 con 0x7f76ec1040e0 2024-03-28T17:49:22.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.171+0000 7f76f7591740 1 Processor -- start 2024-03-28T17:49:22.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.172+0000 7f76f7591740 1 -- start start 2024-03-28T17:49:22.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.172+0000 7f76f7591740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc2d290 0x5653cfc2d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.172+0000 7f76f7591740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653cfc2dbb0 0x5653cfc36250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.172+0000 7f76f7591740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc36790 0x5653cfc38b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.172+0000 7f76f7591740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653cfb91eb0 con 0x5653cfc2dbb0 2024-03-28T17:49:22.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.172+0000 7f76f7591740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653cfb3aa60 con 0x5653cfc36790 2024-03-28T17:49:22.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.172+0000 7f76f7591740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653cfb8df50 con 0x5653cfc2d290 2024-03-28T17:49:22.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.172+0000 7f76f2800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc2d290 0x5653cfc2d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.172+0000 7f76f1400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc36790 0x5653cfc38b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.172+0000 7f76f2800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc2d290 0x5653cfc2d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:56680/0 (socket says 172.21.15.45:56680) 2024-03-28T17:49:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.172+0000 7f76f2800640 1 -- 172.21.15.45:0/2179242597 learned_addr learned my addr 172.21.15.45:0/2179242597 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.172+0000 7f76f1e00640 1 --2- 172.21.15.45:0/2179242597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653cfc2dbb0 0x5653cfc36250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.173+0000 7f76f2800640 1 -- 172.21.15.45:0/2179242597 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc36790 msgr2=0x5653cfc38b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.173+0000 7f76f2800640 1 --2- 172.21.15.45:0/2179242597 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc36790 0x5653cfc38b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.173+0000 7f76f2800640 1 -- 172.21.15.45:0/2179242597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653cfc2dbb0 msgr2=0x5653cfc36250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.173+0000 7f76f2800640 1 --2- 172.21.15.45:0/2179242597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653cfc2dbb0 0x5653cfc36250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.173+0000 7f76e6a00640 1 --2- 172.21.15.45:0/1937005420 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f76d40a2a00 0x7f76d40a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.173+0000 7f76f2800640 1 -- 172.21.15.45:0/2179242597 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5653cf96e7e0 con 0x5653cfc2d290 2024-03-28T17:49:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.173+0000 7f76e6a00640 1 -- 172.21.15.45:0/1937005420 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f76dc0f0ed0 con 0x7f76ec1040e0 2024-03-28T17:49:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.173+0000 7f76f2800640 1 --2- 172.21.15.45:0/2179242597 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc2d290 0x5653cfc2d670 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f76e8009fd0 tx=0x7f76e80108e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbcbf77a489c58cc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:22.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.173+0000 7f76f2800640 1 --2- 172.21.15.45:0/1937005420 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f76d40a2a00 0x7f76d40a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.173+0000 7f76db000640 1 -- 172.21.15.45:0/2179242597 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e8011940 con 0x5653cfc2d290 2024-03-28T17:49:22.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.173+0000 7f76db000640 1 -- 172.21.15.45:0/2179242597 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76e801c480 con 0x5653cfc2d290 2024-03-28T17:49:22.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.174+0000 7f76db000640 1 -- 172.21.15.45:0/2179242597 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e801a740 con 0x5653cfc2d290 2024-03-28T17:49:22.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.174+0000 7f76f7591740 1 -- 172.21.15.45:0/2179242597 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc2d290 msgr2=0x5653cfc2d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:22.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.174+0000 7f76f7591740 1 --2- 172.21.15.45:0/2179242597 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc2d290 0x5653cfc2d670 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f76e8009fd0 tx=0x7f76e80108e0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.174+0000 7f76f2800640 1 --2- 172.21.15.45:0/1937005420 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f76d40a2a00 0x7f76d40a4ec0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f76e801b000 tx=0x7f76e801a9c0 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:22.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.174+0000 7f76f7591740 1 -- 172.21.15.45:0/2179242597 shutdown_connections 2024-03-28T17:49:22.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.174+0000 7f76f7591740 1 --2- 172.21.15.45:0/2179242597 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc36790 0x5653cfc38b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.174+0000 7f76f7591740 1 --2- 172.21.15.45:0/2179242597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653cfc2dbb0 0x5653cfc36250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.174+0000 7f76f7591740 1 --2- 172.21.15.45:0/2179242597 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc2d290 0x5653cfc2d670 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.174+0000 7f76f7591740 1 -- 172.21.15.45:0/2179242597 >> 172.21.15.45:0/2179242597 conn(0x5653cfc279c0 msgr2=0x5653cfc27da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:22.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.174+0000 7f76f7591740 1 -- 172.21.15.45:0/2179242597 shutdown_connections 2024-03-28T17:49:22.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.174+0000 7f76f7591740 1 -- 172.21.15.45:0/2179242597 wait complete. 2024-03-28T17:49:22.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.175+0000 7f76f7591740 1 -- start start 2024-03-28T17:49:22.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.175+0000 7f76f7591740 1 Processor -- start 2024-03-28T17:49:22.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.176+0000 7f76f7591740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653cfc2d290 0x5653cfc360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.176+0000 7f76f7591740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc2dbb0 0x5653cfc33450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.176+0000 7f76f7591740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc36790 0x5653cfc33990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.176+0000 7f76f7591740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653cfc39580 con 0x5653cfc2d290 2024-03-28T17:49:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.176+0000 7f76f7591740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653cfb4deb0 con 0x5653cfc36790 2024-03-28T17:49:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.176+0000 7f76f7591740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653cfc39700 con 0x5653cfc2dbb0 2024-03-28T17:49:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.176+0000 7f76f1400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc2dbb0 0x5653cfc33450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.176+0000 7f76f1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653cfc2d290 0x5653cfc360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.176+0000 7f76f2800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc36790 0x5653cfc33990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.176+0000 7f76f7591740 10 client.0 _ll_register_callbacks cb 0x5653cfb42150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.176+0000 7f76f2800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc36790 0x5653cfc33990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40638/0 (socket says 172.21.15.45:40638) 2024-03-28T17:49:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.176+0000 7f76f1400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc2dbb0 0x5653cfc33450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:56688/0 (socket says 172.21.15.45:56688) 2024-03-28T17:49:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.176+0000 7f76f2800640 1 -- 172.21.15.45:0/768989623 learned_addr learned my addr 172.21.15.45:0/768989623 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.176+0000 7f76f1400640 1 -- 172.21.15.45:0/768989623 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc36790 msgr2=0x5653cfc33990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.176+0000 7f76f1400640 1 --2- 172.21.15.45:0/768989623 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc36790 0x5653cfc33990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.176+0000 7f76f1400640 1 -- 172.21.15.45:0/768989623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653cfc2d290 msgr2=0x5653cfc360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:22.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.176+0000 7f76f1400640 1 --2- 172.21.15.45:0/768989623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653cfc2d290 0x5653cfc360c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.176+0000 7f76f1400640 1 -- 172.21.15.45:0/768989623 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5653cf9cef70 con 0x5653cfc2dbb0 2024-03-28T17:49:22.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.177+0000 7f76f1400640 1 --2- 172.21.15.45:0/768989623 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc2dbb0 0x5653cfc33450 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f76dc0ee640 tx=0x7f76dc009620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2b14290fc50c91c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:22.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.177+0000 7f76cf400640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:49:22.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.177+0000 7f76cf400640 1 -- 172.21.15.45:0/768989623 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76dc0bce00 con 0x5653cfc2dbb0 2024-03-28T17:49:22.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.177+0000 7f76cf400640 1 -- 172.21.15.45:0/768989623 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f76dc0bd700 con 0x5653cfc2dbb0 2024-03-28T17:49:22.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.177+0000 7f76f7591740 1 -- 172.21.15.45:0/768989623 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f76e801c480 con 0x5653cfc2dbb0 2024-03-28T17:49:22.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.177+0000 7f76cf400640 1 -- 172.21.15.45:0/768989623 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76dc0bdba0 con 0x5653cfc2dbb0 2024-03-28T17:49:22.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.177+0000 7f76f7591740 10 client.25522 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:22.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.177+0000 7f76f7591740 1 -- 172.21.15.45:0/768989623 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5653cf866170 con 0x5653cfc2dbb0 2024-03-28T17:49:22.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.177+0000 7f76cf400640 1 -- 172.21.15.45:0/768989623 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76e801c480 con 0x5653cfc2dbb0 2024-03-28T17:49:22.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.178+0000 7f76cf400640 1 -- 172.21.15.45:0/768989623 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f76dc0d6190 con 0x5653cfc2dbb0 2024-03-28T17:49:22.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.178+0000 7f76f7591740 10 client.25522 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:22.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.178+0000 7f76f7591740 10 client.25522 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:49:22.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.178+0000 7f76f7591740 20 client.25522 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:22.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.178+0000 7f76f7591740 1 --2- 172.21.15.45:0/768989623 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5653cfc46760 0x5653cfc48bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.178+0000 7f76f7591740 4 client.25522 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:49:22.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.178+0000 7f76f7591740 1 -- 172.21.15.45:0/768989623 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5653cfb9f3f0 con 0x5653cfc46760 2024-03-28T17:49:22.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.178+0000 7f76f1e00640 1 --2- 172.21.15.45:0/768989623 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5653cfc46760 0x5653cfc48bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.178+0000 7f76f1e00640 1 --2- 172.21.15.45:0/768989623 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5653cfc46760 0x5653cfc48bb0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcfb374a2debea13 server_cookie=e3ad63e58a410c84 in_seq=0 out_seq=0 2024-03-28T17:49:22.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.178+0000 7f76cf400640 10 client.25522 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:22.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.179+0000 7f76cf400640 1 -- 172.21.15.45:0/768989623 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5653cfb9f3f0 con 0x5653cfc46760 2024-03-28T17:49:22.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.179+0000 7f76cf400640 10 client.25522 handle_command_reply: tid=0 2024-03-28T17:49:22.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.179+0000 7f76f7591740 1 client.25522 shutdown 2024-03-28T17:49:22.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.179+0000 7f76f7591740 1 -- 172.21.15.45:0/768989623 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc2dbb0 msgr2=0x5653cfc33450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:22.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.179+0000 7f76f7591740 1 --2- 172.21.15.45:0/768989623 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc2dbb0 0x5653cfc33450 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f76dc0ee640 tx=0x7f76dc009620 comp rx=0 tx=0).stop 2024-03-28T17:49:22.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.179+0000 7f76f7591740 1 -- 172.21.15.45:0/768989623 shutdown_connections 2024-03-28T17:49:22.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.179+0000 7f76f7591740 1 --2- 172.21.15.45:0/768989623 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5653cfc46760 0x5653cfc48bb0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.180+0000 7f76f7591740 1 --2- 172.21.15.45:0/768989623 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc36790 0x5653cfc33990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.180+0000 7f76f7591740 1 --2- 172.21.15.45:0/768989623 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc2dbb0 0x5653cfc33450 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.180+0000 7f76cf400640 0 client.25522 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:22.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.180+0000 7f76f7591740 1 --2- 172.21.15.45:0/768989623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653cfc2d290 0x5653cfc360c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.180+0000 7f76f7591740 1 -- 172.21.15.45:0/768989623 >> 172.21.15.45:0/768989623 conn(0x5653cfc279c0 msgr2=0x5653cfc29710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:22.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.180+0000 7f76f7591740 1 -- 172.21.15.45:0/768989623 shutdown_connections 2024-03-28T17:49:22.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.180+0000 7f76f7591740 1 -- 172.21.15.45:0/768989623 wait complete. 2024-03-28T17:49:22.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.180+0000 7f76f7591740 20 client.25522 trim_cache size 0 max 16384 2024-03-28T17:49:22.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.198+0000 7f76f7591740 1 Processor -- start 2024-03-28T17:49:22.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.198+0000 7f76f7591740 1 -- start start 2024-03-28T17:49:22.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.198+0000 7f76f7591740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc2d290 0x5653cfc2e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.198+0000 7f76f7591740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653cfc2dbb0 0x5653cfc35b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.198+0000 7f76f7591740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc36790 0x5653cfc36050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.198+0000 7f76f7591740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec1b17b0 con 0x5653cfc2dbb0 2024-03-28T17:49:22.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.198+0000 7f76f7591740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec1af2b0 con 0x5653cfc2d290 2024-03-28T17:49:22.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.198+0000 7f76f7591740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec1ad350 con 0x5653cfc36790 2024-03-28T17:49:22.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.198+0000 7f76f1e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc2d290 0x5653cfc2e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.198+0000 7f76f1e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc2d290 0x5653cfc2e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40650/0 (socket says 172.21.15.45:40650) 2024-03-28T17:49:22.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.198+0000 7f76f1e00640 1 -- 172.21.15.45:0/682728441 learned_addr learned my addr 172.21.15.45:0/682728441 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:22.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.198+0000 7f76f2800640 1 --2- 172.21.15.45:0/682728441 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc36790 0x5653cfc36050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.198+0000 7f76f1400640 1 --2- 172.21.15.45:0/682728441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653cfc2dbb0 0x5653cfc35b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.199+0000 7f76f1e00640 1 -- 172.21.15.45:0/682728441 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc36790 msgr2=0x5653cfc36050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:22.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.199+0000 7f76f1e00640 1 --2- 172.21.15.45:0/682728441 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc36790 0x5653cfc36050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.199+0000 7f76f1e00640 1 -- 172.21.15.45:0/682728441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653cfc2dbb0 msgr2=0x5653cfc35b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:22.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.199+0000 7f76f1e00640 1 --2- 172.21.15.45:0/682728441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653cfc2dbb0 0x5653cfc35b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.199+0000 7f76f1e00640 1 -- 172.21.15.45:0/682728441 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5653cfc319a0 con 0x5653cfc2d290 2024-03-28T17:49:22.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.199+0000 7f76f1400640 1 --2- 172.21.15.45:0/682728441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653cfc2dbb0 0x5653cfc35b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:49:22.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.199+0000 7f76f1e00640 1 --2- 172.21.15.45:0/682728441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc2d290 0x5653cfc2e7e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f76e0004700 tx=0x7f76e00126d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c9421320fd0dc41 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:22.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.199+0000 7f76da600640 1 -- 172.21.15.45:0/682728441 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e0012110 con 0x5653cfc2d290 2024-03-28T17:49:22.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.199+0000 7f76da600640 1 -- 172.21.15.45:0/682728441 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76e0022cb0 con 0x5653cfc2d290 2024-03-28T17:49:22.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.199+0000 7f76da600640 1 -- 172.21.15.45:0/682728441 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e001f650 con 0x5653cfc2d290 2024-03-28T17:49:22.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.200+0000 7f76f7591740 1 -- 172.21.15.45:0/682728441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc2d290 msgr2=0x5653cfc2e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:22.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.200+0000 7f76f7591740 1 --2- 172.21.15.45:0/682728441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc2d290 0x5653cfc2e7e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f76e0004700 tx=0x7f76e00126d0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.200+0000 7f76f7591740 1 -- 172.21.15.45:0/682728441 shutdown_connections 2024-03-28T17:49:22.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.200+0000 7f76f7591740 1 --2- 172.21.15.45:0/682728441 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc36790 0x5653cfc36050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.200+0000 7f76f7591740 1 --2- 172.21.15.45:0/682728441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653cfc2dbb0 0x5653cfc35b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.200+0000 7f76f7591740 1 --2- 172.21.15.45:0/682728441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc2d290 0x5653cfc2e7e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.200+0000 7f76f7591740 1 -- 172.21.15.45:0/682728441 >> 172.21.15.45:0/682728441 conn(0x5653cfc279c0 msgr2=0x5653cfc2e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:22.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.200+0000 7f76f7591740 1 -- 172.21.15.45:0/682728441 shutdown_connections 2024-03-28T17:49:22.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.200+0000 7f76f7591740 1 -- 172.21.15.45:0/682728441 wait complete. 2024-03-28T17:49:22.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.201+0000 7f76f7591740 1 -- start start 2024-03-28T17:49:22.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.201+0000 7f76f7591740 1 Processor -- start 2024-03-28T17:49:22.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.201+0000 7f76f7591740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653cfc2d290 0x5653cfc3acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.201+0000 7f76f7591740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc2dbb0 0x5653cfc3b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.201+0000 7f76f7591740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc36790 0x5653cfc3b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.201+0000 7f76f7591740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec1ad950 con 0x5653cfc2d290 2024-03-28T17:49:22.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.201+0000 7f76f7591740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec1aec50 con 0x5653cfc2dbb0 2024-03-28T17:49:22.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.201+0000 7f76f7591740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec1adf90 con 0x5653cfc36790 2024-03-28T17:49:22.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.201+0000 7f76f7591740 10 client.0 _ll_register_callbacks cb 0x7f76dc005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:22.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.202+0000 7f76f1400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc2dbb0 0x5653cfc3b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.202+0000 7f76f2800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc36790 0x5653cfc3b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.202+0000 7f76f2800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc36790 0x5653cfc3b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:56716/0 (socket says 172.21.15.45:56716) 2024-03-28T17:49:22.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.202+0000 7f76f1400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc2dbb0 0x5653cfc3b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40652/0 (socket says 172.21.15.45:40652) 2024-03-28T17:49:22.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.202+0000 7f76f2800640 1 -- 172.21.15.45:0/1789003846 learned_addr learned my addr 172.21.15.45:0/1789003846 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:22.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.202+0000 7f76f1e00640 1 --2- 172.21.15.45:0/1789003846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653cfc2d290 0x5653cfc3acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.202+0000 7f76f2800640 1 -- 172.21.15.45:0/1789003846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc2dbb0 msgr2=0x5653cfc3b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:22.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.202+0000 7f76f2800640 1 --2- 172.21.15.45:0/1789003846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc2dbb0 0x5653cfc3b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.202+0000 7f76f2800640 1 -- 172.21.15.45:0/1789003846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653cfc2d290 msgr2=0x5653cfc3acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:22.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.202+0000 7f76f2800640 1 --2- 172.21.15.45:0/1789003846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653cfc2d290 0x5653cfc3acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.202+0000 7f76f2800640 1 -- 172.21.15.45:0/1789003846 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5653cfca2050 con 0x5653cfc36790 2024-03-28T17:49:22.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.202+0000 7f76f1400640 1 --2- 172.21.15.45:0/1789003846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc2dbb0 0x5653cfc3b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:49:22.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.202+0000 7f76f2800640 1 --2- 172.21.15.45:0/1789003846 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc36790 0x5653cfc3b740 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f76e8006010 tx=0x7f76e8017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31646d722b09aeea server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:22.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.202+0000 7f76cf400640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:49:22.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.203+0000 7f76cf400640 1 -- 172.21.15.45:0/1789003846 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e8037020 con 0x5653cfc36790 2024-03-28T17:49:22.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.203+0000 7f76f7591740 1 -- 172.21.15.45:0/1789003846 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f76e0022cb0 con 0x5653cfc36790 2024-03-28T17:49:22.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.203+0000 7f76cf400640 1 -- 172.21.15.45:0/1789003846 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f76e801a590 con 0x5653cfc36790 2024-03-28T17:49:22.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.203+0000 7f76cf400640 1 -- 172.21.15.45:0/1789003846 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76e8042050 con 0x5653cfc36790 2024-03-28T17:49:22.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.203+0000 7f76f7591740 10 client.25528 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:22.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.203+0000 7f76f7591740 1 -- 172.21.15.45:0/1789003846 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5653cfca32f0 con 0x5653cfc36790 2024-03-28T17:49:22.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.203+0000 7f76cf400640 1 -- 172.21.15.45:0/1789003846 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76e0022cb0 con 0x5653cfc36790 2024-03-28T17:49:22.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.203+0000 7f76cf400640 1 -- 172.21.15.45:0/1789003846 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f76e801c480 con 0x5653cfc36790 2024-03-28T17:49:22.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.203+0000 7f76f7591740 10 client.25528 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:22.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.203+0000 7f76f7591740 10 client.25528 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:49:22.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.203+0000 7f76f7591740 20 client.25528 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:22.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.203+0000 7f76f7591740 1 --2- 172.21.15.45:0/1789003846 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5653cfc46760 0x5653cfca5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.203+0000 7f76f7591740 4 client.25528 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T17:49:22.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.204+0000 7f76f7591740 1 -- 172.21.15.45:0/1789003846 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5653cfb99740 con 0x5653cfc46760 2024-03-28T17:49:22.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.204+0000 7f76f1e00640 1 --2- 172.21.15.45:0/1789003846 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5653cfc46760 0x5653cfca5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.204+0000 7f76f1e00640 1 --2- 172.21.15.45:0/1789003846 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5653cfc46760 0x5653cfca5840 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c98eb8004b6c8f3c server_cookie=15c1463430cb80a5 in_seq=0 out_seq=0 2024-03-28T17:49:22.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.204+0000 7f76cf400640 10 client.25528 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:22.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.205+0000 7f76cf400640 1 -- 172.21.15.45:0/1789003846 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5653cfb99740 con 0x5653cfc46760 2024-03-28T17:49:22.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.205+0000 7f76cf400640 10 client.25528 handle_command_reply: tid=0 2024-03-28T17:49:22.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.205+0000 7f76f7591740 1 client.25528 shutdown 2024-03-28T17:49:22.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.205+0000 7f76f7591740 1 -- 172.21.15.45:0/1789003846 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc36790 msgr2=0x5653cfc3b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:22.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.205+0000 7f76f7591740 1 --2- 172.21.15.45:0/1789003846 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc36790 0x5653cfc3b740 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f76e8006010 tx=0x7f76e8017000 comp rx=0 tx=0).stop 2024-03-28T17:49:22.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.206+0000 7f76f7591740 1 -- 172.21.15.45:0/1789003846 shutdown_connections 2024-03-28T17:49:22.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.206+0000 7f76f7591740 1 --2- 172.21.15.45:0/1789003846 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5653cfc46760 0x5653cfca5840 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.206+0000 7f76f7591740 1 --2- 172.21.15.45:0/1789003846 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5653cfc36790 0x5653cfc3b740 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.206+0000 7f76f7591740 1 --2- 172.21.15.45:0/1789003846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5653cfc2dbb0 0x5653cfc3b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.206+0000 7f76f7591740 1 --2- 172.21.15.45:0/1789003846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653cfc2d290 0x5653cfc3acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.206+0000 7f76cf400640 0 client.25528 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:22.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.206+0000 7f76f7591740 1 -- 172.21.15.45:0/1789003846 >> 172.21.15.45:0/1789003846 conn(0x5653cfc279c0 msgr2=0x5653cfc26a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:22.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.206+0000 7f76f7591740 1 -- 172.21.15.45:0/1789003846 shutdown_connections 2024-03-28T17:49:22.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.206+0000 7f76f7591740 1 -- 172.21.15.45:0/1789003846 wait complete. 2024-03-28T17:49:22.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.206+0000 7f76f7591740 20 client.25528 trim_cache size 0 max 16384 2024-03-28T17:49:22.213 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:49:22.213 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-28T17:49:22.213 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "631fd4f2-373a-469d-a0ec-51d619a74719", 2024-03-28T17:49:22.213 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-28T17:49:22.213 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:49:22.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.206+0000 7f76f3200640 1 -- 172.21.15.45:0/1937005420 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f76d40a2a00 msgr2=0x7f76d40a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:22.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.206+0000 7f76f3200640 1 --2- 172.21.15.45:0/1937005420 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f76d40a2a00 0x7f76d40a4ec0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f76e801b000 tx=0x7f76e801a9c0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.206+0000 7f76f3200640 1 -- 172.21.15.45:0/1937005420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76ec1040e0 msgr2=0x7f76ec19a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:22.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.207+0000 7f76f3200640 1 --2- 172.21.15.45:0/1937005420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76ec1040e0 0x7f76ec19a890 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f76dc036040 tx=0x7f76dc005e80 comp rx=0 tx=0).stop 2024-03-28T17:49:22.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.207+0000 7f76f3200640 1 -- 172.21.15.45:0/1937005420 shutdown_connections 2024-03-28T17:49:22.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.207+0000 7f76f3200640 1 --2- 172.21.15.45:0/1937005420 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f76d40a2a00 0x7f76d40a4ec0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.207+0000 7f76f3200640 1 --2- 172.21.15.45:0/1937005420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76ec109730 0x7f76ec19ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.207+0000 7f76f3200640 1 --2- 172.21.15.45:0/1937005420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76ec1040e0 0x7f76ec19a890 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.207+0000 7f76f3200640 1 --2- 172.21.15.45:0/1937005420 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76ec103720 0x7f76ec19a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.207+0000 7f76f3200640 1 -- 172.21.15.45:0/1937005420 >> 172.21.15.45:0/1937005420 conn(0x7f76ec0ff1b0 msgr2=0x7f76ec0ffb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:22.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.207+0000 7f76f3200640 1 -- 172.21.15.45:0/1937005420 shutdown_connections 2024-03-28T17:49:22.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.207+0000 7f76f3200640 1 -- 172.21.15.45:0/1937005420 wait complete. 2024-03-28T17:49:22.227 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:49:22.227 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:49:22.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.451+0000 7fa462c00640 1 -- 172.21.15.45:0/4264787138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa45c10bae0 msgr2=0x7fa45c10df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:22.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.451+0000 7fa462c00640 1 --2- 172.21.15.45:0/4264787138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa45c10bae0 0x7fa45c10df80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa454009ca0 tx=0x7fa45402c180 comp rx=0 tx=0).stop 2024-03-28T17:49:22.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.451+0000 7fa462c00640 1 -- 172.21.15.45:0/4264787138 shutdown_connections 2024-03-28T17:49:22.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.451+0000 7fa462c00640 1 --2- 172.21.15.45:0/4264787138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa45c10bae0 0x7fa45c10df80 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.451+0000 7fa462c00640 1 --2- 172.21.15.45:0/4264787138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa45c102fe0 0x7fa45c10b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.451+0000 7fa462c00640 1 --2- 172.21.15.45:0/4264787138 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa45c1026c0 0x7fa45c102aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.451+0000 7fa462c00640 1 -- 172.21.15.45:0/4264787138 >> 172.21.15.45:0/4264787138 conn(0x7fa45c0fbec0 msgr2=0x7fa45c0fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:22.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.451+0000 7fa462c00640 1 -- 172.21.15.45:0/4264787138 shutdown_connections 2024-03-28T17:49:22.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.451+0000 7fa462c00640 1 -- 172.21.15.45:0/4264787138 wait complete. 2024-03-28T17:49:22.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.452+0000 7fa462c00640 1 Processor -- start 2024-03-28T17:49:22.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.453+0000 7fa462c00640 1 -- start start 2024-03-28T17:49:22.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.453+0000 7fa462c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa45c1026c0 0x7fa45c199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.453+0000 7fa462c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa45c102fe0 0x7fa45c19a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.453+0000 7fa462c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa45c10bae0 0x7fa45c19e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.453+0000 7fa462c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c1118a0 con 0x7fa45c1026c0 2024-03-28T17:49:22.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.453+0000 7fa462c00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c111720 con 0x7fa45c10bae0 2024-03-28T17:49:22.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.453+0000 7fa462c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c111a20 con 0x7fa45c102fe0 2024-03-28T17:49:22.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.454+0000 7fa461800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa45c1026c0 0x7fa45c199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.454+0000 7fa461800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa45c1026c0 0x7fa45c199fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46758/0 (socket says 172.21.15.45:46758) 2024-03-28T17:49:22.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.454+0000 7fa460e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa45c102fe0 0x7fa45c19a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.454+0000 7fa461800640 1 -- 172.21.15.45:0/1142767860 learned_addr learned my addr 172.21.15.45:0/1142767860 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:22.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.454+0000 7fa462200640 1 --2- 172.21.15.45:0/1142767860 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa45c10bae0 0x7fa45c19e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.454+0000 7fa462200640 1 -- 172.21.15.45:0/1142767860 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa45c102fe0 msgr2=0x7fa45c19a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:22.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.454+0000 7fa462200640 1 --2- 172.21.15.45:0/1142767860 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa45c102fe0 0x7fa45c19a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.454+0000 7fa462200640 1 -- 172.21.15.45:0/1142767860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa45c1026c0 msgr2=0x7fa45c199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:22.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.454+0000 7fa462200640 1 --2- 172.21.15.45:0/1142767860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa45c1026c0 0x7fa45c199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.454+0000 7fa462200640 1 -- 172.21.15.45:0/1142767860 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa45c19ef90 con 0x7fa45c10bae0 2024-03-28T17:49:22.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.454+0000 7fa462200640 1 --2- 172.21.15.45:0/1142767860 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa45c10bae0 0x7fa45c19e860 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa454009bf0 tx=0x7fa454035e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:22.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.455+0000 7fa45a000640 1 -- 172.21.15.45:0/1142767860 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa454039040 con 0x7fa45c10bae0 2024-03-28T17:49:22.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.455+0000 7fa45a000640 1 -- 172.21.15.45:0/1142767860 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa454044070 con 0x7fa45c10bae0 2024-03-28T17:49:22.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.455+0000 7fa45a000640 1 -- 172.21.15.45:0/1142767860 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa454032960 con 0x7fa45c10bae0 2024-03-28T17:49:22.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.455+0000 7fa462c00640 1 -- 172.21.15.45:0/1142767860 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa45c19f220 con 0x7fa45c10bae0 2024-03-28T17:49:22.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.455+0000 7fa462c00640 1 -- 172.21.15.45:0/1142767860 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa45c19f690 con 0x7fa45c10bae0 2024-03-28T17:49:22.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.458+0000 7fa45a000640 1 -- 172.21.15.45:0/1142767860 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 26) v1 ==== 137466+0+0 (secure 0 0 0) 0x7fa454032b00 con 0x7fa45c10bae0 2024-03-28T17:49:22.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.458+0000 7fa466fe6740 1 Processor -- start 2024-03-28T17:49:22.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.458+0000 7fa466fe6740 1 -- start start 2024-03-28T17:49:22.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.458+0000 7fa466fe6740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19aefdd0 0x55ff19aeeb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.458+0000 7fa466fe6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ff19aef0f0 0x55ff19aef570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.458+0000 7fa466fe6740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19af8460 0x55ff19afa850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.458+0000 7fa466fe6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff19a53250 con 0x55ff19aef0f0 2024-03-28T17:49:22.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.458+0000 7fa466fe6740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff199fba30 con 0x55ff19aefdd0 2024-03-28T17:49:22.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.458+0000 7fa466fe6740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff19a4f2f0 con 0x55ff19af8460 2024-03-28T17:49:22.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.459+0000 7fa460e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19aefdd0 0x55ff19aeeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.459+0000 7fa460e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19aefdd0 0x55ff19aeeb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40684/0 (socket says 172.21.15.45:40684) 2024-03-28T17:49:22.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.459+0000 7fa460e00640 1 -- 172.21.15.45:0/639591030 learned_addr learned my addr 172.21.15.45:0/639591030 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:22.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.459+0000 7fa461800640 1 --2- 172.21.15.45:0/639591030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ff19aef0f0 0x55ff19aef570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.459+0000 7fa460e00640 1 --2- 172.21.15.45:0/639591030 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19af8460 0x55ff19afa850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.459+0000 7fa460e00640 1 -- 172.21.15.45:0/639591030 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19af8460 msgr2=0x55ff19afa850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:22.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.459+0000 7fa460e00640 1 --2- 172.21.15.45:0/639591030 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19af8460 0x55ff19afa850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.459+0000 7fa460e00640 1 -- 172.21.15.45:0/639591030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ff19aef0f0 msgr2=0x55ff19aef570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:22.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.459+0000 7fa460e00640 1 --2- 172.21.15.45:0/639591030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ff19aef0f0 0x55ff19aef570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.459+0000 7fa460e00640 1 -- 172.21.15.45:0/639591030 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff199e6690 con 0x55ff19aefdd0 2024-03-28T17:49:22.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.459+0000 7fa460e00640 1 --2- 172.21.15.45:0/639591030 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19aefdd0 0x55ff19aeeb20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa448012730 tx=0x7fa44800b6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a6958cf6ae72d68 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:22.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.460+0000 7fa44ea00640 1 -- 172.21.15.45:0/639591030 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa448012cb0 con 0x55ff19aefdd0 2024-03-28T17:49:22.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.460+0000 7fa44ea00640 1 -- 172.21.15.45:0/639591030 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa448012e50 con 0x55ff19aefdd0 2024-03-28T17:49:22.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.460+0000 7fa44ea00640 1 -- 172.21.15.45:0/639591030 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44801eb10 con 0x55ff19aefdd0 2024-03-28T17:49:22.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.460+0000 7fa45a000640 1 --2- 172.21.15.45:0/1142767860 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fa4440a2a80 0x7fa4440a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.460+0000 7fa466fe6740 1 -- 172.21.15.45:0/639591030 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19aefdd0 msgr2=0x55ff19aeeb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:22.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.460+0000 7fa466fe6740 1 --2- 172.21.15.45:0/639591030 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19aefdd0 0x55ff19aeeb20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa448012730 tx=0x7fa44800b6f0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.460+0000 7fa466fe6740 1 -- 172.21.15.45:0/639591030 shutdown_connections 2024-03-28T17:49:22.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.460+0000 7fa466fe6740 1 --2- 172.21.15.45:0/639591030 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19af8460 0x55ff19afa850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.460+0000 7fa466fe6740 1 --2- 172.21.15.45:0/639591030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ff19aef0f0 0x55ff19aef570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.460+0000 7fa466fe6740 1 --2- 172.21.15.45:0/639591030 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19aefdd0 0x55ff19aeeb20 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.460+0000 7fa466fe6740 1 -- 172.21.15.45:0/639591030 >> 172.21.15.45:0/639591030 conn(0x55ff19ab6200 msgr2=0x55ff19ae9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:22.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.460+0000 7fa45a000640 1 -- 172.21.15.45:0/1142767860 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fa4540f1290 con 0x7fa45c10bae0 2024-03-28T17:49:22.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.460+0000 7fa461800640 1 --2- 172.21.15.45:0/1142767860 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fa4440a2a80 0x7fa4440a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.461+0000 7fa466fe6740 1 -- 172.21.15.45:0/639591030 shutdown_connections 2024-03-28T17:49:22.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.461+0000 7fa466fe6740 1 -- 172.21.15.45:0/639591030 wait complete. 2024-03-28T17:49:22.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.461+0000 7fa466fe6740 1 -- start start 2024-03-28T17:49:22.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.461+0000 7fa461800640 1 --2- 172.21.15.45:0/1142767860 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fa4440a2a80 0x7fa4440a4f40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x55ff19af0320 tx=0x7fa450005f90 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:22.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.461+0000 7fa466fe6740 1 Processor -- start 2024-03-28T17:49:22.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.462+0000 7fa466fe6740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19aef0f0 0x55ff19af7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.462+0000 7fa466fe6740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19aefdd0 0x55ff19af4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.462+0000 7fa466fe6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ff19af8460 0x55ff19af5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.462+0000 7fa466fe6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff19afb2d0 con 0x55ff19af8460 2024-03-28T17:49:22.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.462+0000 7fa466fe6740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff19a0ee80 con 0x55ff19aef0f0 2024-03-28T17:49:22.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.462+0000 7fa466fe6740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff19afb450 con 0x55ff19aefdd0 2024-03-28T17:49:22.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.462+0000 7fa461800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19aef0f0 0x55ff19af7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.462+0000 7fa461800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19aef0f0 0x55ff19af7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40690/0 (socket says 172.21.15.45:40690) 2024-03-28T17:49:22.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.462+0000 7fa461800640 1 -- 172.21.15.45:0/3106310728 learned_addr learned my addr 172.21.15.45:0/3106310728 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:22.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.462+0000 7fa460e00640 1 --2- 172.21.15.45:0/3106310728 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19aefdd0 0x55ff19af4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.462+0000 7fa466fe6740 10 client.0 _ll_register_callbacks cb 0x55ff19a21130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:22.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.462+0000 7fa462200640 1 --2- 172.21.15.45:0/3106310728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ff19af8460 0x55ff19af5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.463+0000 7fa460e00640 1 -- 172.21.15.45:0/3106310728 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19aef0f0 msgr2=0x55ff19af7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:22.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.463+0000 7fa460e00640 1 --2- 172.21.15.45:0/3106310728 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19aef0f0 0x55ff19af7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.463+0000 7fa460e00640 1 -- 172.21.15.45:0/3106310728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ff19af8460 msgr2=0x55ff19af5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:22.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.463+0000 7fa460e00640 1 --2- 172.21.15.45:0/3106310728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ff19af8460 0x55ff19af5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.463+0000 7fa460e00640 1 -- 172.21.15.45:0/3106310728 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ff198317d0 con 0x55ff19aefdd0 2024-03-28T17:49:22.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.463+0000 7fa461800640 1 --2- 172.21.15.45:0/3106310728 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19aef0f0 0x55ff19af7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:49:22.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.463+0000 7fa460e00640 1 --2- 172.21.15.45:0/3106310728 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19aefdd0 0x55ff19af4ef0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa448015040 tx=0x7fa44801e6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a723d234a4ee594f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:22.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.463+0000 7fa443400640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:49:22.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.463+0000 7fa443400640 1 -- 172.21.15.45:0/3106310728 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa448012cb0 con 0x55ff19aefdd0 2024-03-28T17:49:22.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.463+0000 7fa466fe6740 1 -- 172.21.15.45:0/3106310728 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa448012e50 con 0x55ff19aefdd0 2024-03-28T17:49:22.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.463+0000 7fa443400640 1 -- 172.21.15.45:0/3106310728 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fa448004450 con 0x55ff19aefdd0 2024-03-28T17:49:22.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.463+0000 7fa443400640 1 -- 172.21.15.45:0/3106310728 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa44801e8f0 con 0x55ff19aefdd0 2024-03-28T17:49:22.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.463+0000 7fa466fe6740 10 client.25540 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:22.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.463+0000 7fa466fe6740 1 -- 172.21.15.45:0/3106310728 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55ff197673f0 con 0x55ff19aefdd0 2024-03-28T17:49:22.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.464+0000 7fa443400640 1 -- 172.21.15.45:0/3106310728 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa448012e50 con 0x55ff19aefdd0 2024-03-28T17:49:22.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.464+0000 7fa443400640 1 -- 172.21.15.45:0/3106310728 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fa448010740 con 0x55ff19aefdd0 2024-03-28T17:49:22.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.464+0000 7fa466fe6740 10 client.25540 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:22.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.464+0000 7fa466fe6740 10 client.25540 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:49:22.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.464+0000 7fa466fe6740 20 client.25540 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:22.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.464+0000 7fa466fe6740 1 --2- 172.21.15.45:0/3106310728 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ff19b08240 0x55ff19b0a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.464+0000 7fa466fe6740 4 client.25540 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:49:22.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.464+0000 7fa466fe6740 1 -- 172.21.15.45:0/3106310728 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ff19a60790 con 0x55ff19b08240 2024-03-28T17:49:22.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.465+0000 7fa461800640 1 --2- 172.21.15.45:0/3106310728 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ff19b08240 0x55ff19b0a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.465+0000 7fa461800640 1 --2- 172.21.15.45:0/3106310728 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ff19b08240 0x55ff19b0a630 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f233a0066e1714 server_cookie=2c534b466f9986a5 in_seq=0 out_seq=0 2024-03-28T17:49:22.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.465+0000 7fa443400640 10 client.25540 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:22.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.466+0000 7fa443400640 1 -- 172.21.15.45:0/3106310728 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ff19a60790 con 0x55ff19b08240 2024-03-28T17:49:22.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.466+0000 7fa443400640 10 client.25540 handle_command_reply: tid=0 2024-03-28T17:49:22.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.466+0000 7fa466fe6740 1 client.25540 shutdown 2024-03-28T17:49:22.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.466+0000 7fa466fe6740 1 -- 172.21.15.45:0/3106310728 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19aefdd0 msgr2=0x55ff19af4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:22.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.466+0000 7fa466fe6740 1 --2- 172.21.15.45:0/3106310728 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19aefdd0 0x55ff19af4ef0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa448015040 tx=0x7fa44801e6b0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.466+0000 7fa466fe6740 1 -- 172.21.15.45:0/3106310728 shutdown_connections 2024-03-28T17:49:22.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.466+0000 7fa466fe6740 1 --2- 172.21.15.45:0/3106310728 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ff19b08240 0x55ff19b0a630 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.466+0000 7fa466fe6740 1 --2- 172.21.15.45:0/3106310728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ff19af8460 0x55ff19af5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.466+0000 7fa466fe6740 1 --2- 172.21.15.45:0/3106310728 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19aefdd0 0x55ff19af4ef0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.466+0000 7fa443400640 0 client.25540 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:22.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.466+0000 7fa466fe6740 1 --2- 172.21.15.45:0/3106310728 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19aef0f0 0x55ff19af7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.466+0000 7fa466fe6740 1 -- 172.21.15.45:0/3106310728 >> 172.21.15.45:0/3106310728 conn(0x55ff19ab6200 msgr2=0x55ff19ae8330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:22.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.466+0000 7fa466fe6740 1 -- 172.21.15.45:0/3106310728 shutdown_connections 2024-03-28T17:49:22.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.466+0000 7fa466fe6740 1 -- 172.21.15.45:0/3106310728 wait complete. 2024-03-28T17:49:22.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.466+0000 7fa466fe6740 20 client.25540 trim_cache size 0 max 16384 2024-03-28T17:49:22.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.484+0000 7fa466fe6740 1 Processor -- start 2024-03-28T17:49:22.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.484+0000 7fa466fe6740 1 -- start start 2024-03-28T17:49:22.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.484+0000 7fa466fe6740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19aef0f0 0x55ff19af7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.484+0000 7fa466fe6740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19aefdd0 0x55ff19af7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.484+0000 7fa466fe6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ff19af8460 0x55ff19b03bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.484+0000 7fa466fe6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c1b16e0 con 0x55ff19af8460 2024-03-28T17:49:22.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.484+0000 7fa466fe6740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c1af1e0 con 0x55ff19aefdd0 2024-03-28T17:49:22.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.484+0000 7fa466fe6740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c1ad260 con 0x55ff19aef0f0 2024-03-28T17:49:22.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.484+0000 7fa460e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19aefdd0 0x55ff19af7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.484+0000 7fa460e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19aefdd0 0x55ff19af7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40704/0 (socket says 172.21.15.45:40704) 2024-03-28T17:49:22.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.484+0000 7fa460e00640 1 -- 172.21.15.45:0/384594931 learned_addr learned my addr 172.21.15.45:0/384594931 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:22.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.484+0000 7fa461800640 1 --2- 172.21.15.45:0/384594931 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19aef0f0 0x55ff19af7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.484+0000 7fa462200640 1 --2- 172.21.15.45:0/384594931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ff19af8460 0x55ff19b03bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.485+0000 7fa460e00640 1 -- 172.21.15.45:0/384594931 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19aef0f0 msgr2=0x55ff19af7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:22.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.485+0000 7fa460e00640 1 --2- 172.21.15.45:0/384594931 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19aef0f0 0x55ff19af7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.485+0000 7fa460e00640 1 -- 172.21.15.45:0/384594931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ff19af8460 msgr2=0x55ff19b03bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:22.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.485+0000 7fa460e00640 1 --2- 172.21.15.45:0/384594931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ff19af8460 0x55ff19b03bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.485+0000 7fa460e00640 1 -- 172.21.15.45:0/384594931 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff197270f0 con 0x55ff19aefdd0 2024-03-28T17:49:22.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.485+0000 7fa462200640 1 --2- 172.21.15.45:0/384594931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ff19af8460 0x55ff19b03bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:49:22.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.485+0000 7fa461800640 1 --2- 172.21.15.45:0/384594931 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19aef0f0 0x55ff19af7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:49:22.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.485+0000 7fa460e00640 1 --2- 172.21.15.45:0/384594931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19aefdd0 0x55ff19af7b60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa448010b90 tx=0x7fa448005900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54ec2279e1e7c2f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:22.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.485+0000 7fa44e000640 1 -- 172.21.15.45:0/384594931 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44800b3b0 con 0x55ff19aefdd0 2024-03-28T17:49:22.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.485+0000 7fa44e000640 1 -- 172.21.15.45:0/384594931 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa448016040 con 0x55ff19aefdd0 2024-03-28T17:49:22.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.485+0000 7fa44e000640 1 -- 172.21.15.45:0/384594931 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4480044b0 con 0x55ff19aefdd0 2024-03-28T17:49:22.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.486+0000 7fa466fe6740 1 -- 172.21.15.45:0/384594931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19aefdd0 msgr2=0x55ff19af7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:22.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.486+0000 7fa466fe6740 1 --2- 172.21.15.45:0/384594931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19aefdd0 0x55ff19af7b60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa448010b90 tx=0x7fa448005900 comp rx=0 tx=0).stop 2024-03-28T17:49:22.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.486+0000 7fa466fe6740 1 -- 172.21.15.45:0/384594931 shutdown_connections 2024-03-28T17:49:22.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.486+0000 7fa466fe6740 1 --2- 172.21.15.45:0/384594931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ff19af8460 0x55ff19b03bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.486+0000 7fa466fe6740 1 --2- 172.21.15.45:0/384594931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19aefdd0 0x55ff19af7b60 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.486+0000 7fa466fe6740 1 --2- 172.21.15.45:0/384594931 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19aef0f0 0x55ff19af7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.486+0000 7fa466fe6740 1 -- 172.21.15.45:0/384594931 >> 172.21.15.45:0/384594931 conn(0x55ff19ab6200 msgr2=0x55ff19af37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:22.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.486+0000 7fa466fe6740 1 -- 172.21.15.45:0/384594931 shutdown_connections 2024-03-28T17:49:22.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.486+0000 7fa466fe6740 1 -- 172.21.15.45:0/384594931 wait complete. 2024-03-28T17:49:22.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.487+0000 7fa466fe6740 1 -- start start 2024-03-28T17:49:22.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.487+0000 7fa466fe6740 1 Processor -- start 2024-03-28T17:49:22.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.487+0000 7fa466fe6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ff19aef0f0 0x55ff19af1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.487+0000 7fa466fe6740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19aefdd0 0x55ff19af1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.487+0000 7fa466fe6740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19af8460 0x55ff19af2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.487+0000 7fa466fe6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c1ad860 con 0x55ff19aef0f0 2024-03-28T17:49:22.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.487+0000 7fa466fe6740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c1aeb80 con 0x55ff19af8460 2024-03-28T17:49:22.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.487+0000 7fa466fe6740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c1adec0 con 0x55ff19aefdd0 2024-03-28T17:49:22.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.487+0000 7fa466fe6740 10 client.0 _ll_register_callbacks cb 0x7fa450009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:22.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.488+0000 7fa460e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19aefdd0 0x55ff19af1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.488+0000 7fa461800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ff19aef0f0 0x55ff19af1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.488+0000 7fa462200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19af8460 0x55ff19af2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.488+0000 7fa462200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19af8460 0x55ff19af2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40710/0 (socket says 172.21.15.45:40710) 2024-03-28T17:49:22.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.488+0000 7fa462200640 1 -- 172.21.15.45:0/3228806582 learned_addr learned my addr 172.21.15.45:0/3228806582 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:22.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.488+0000 7fa461800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ff19aef0f0 0x55ff19af1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46798/0 (socket says 172.21.15.45:46798) 2024-03-28T17:49:22.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.488+0000 7fa460e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19aefdd0 0x55ff19af1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:56744/0 (socket says 172.21.15.45:56744) 2024-03-28T17:49:22.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.488+0000 7fa462200640 1 -- 172.21.15.45:0/3228806582 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19aefdd0 msgr2=0x55ff19af1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:22.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.488+0000 7fa462200640 1 --2- 172.21.15.45:0/3228806582 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19aefdd0 0x55ff19af1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.488+0000 7fa462200640 1 -- 172.21.15.45:0/3228806582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ff19aef0f0 msgr2=0x55ff19af1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:22.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.488+0000 7fa462200640 1 --2- 172.21.15.45:0/3228806582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ff19aef0f0 0x55ff19af1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.488+0000 7fa462200640 1 -- 172.21.15.45:0/3228806582 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ff19b62ae0 con 0x55ff19af8460 2024-03-28T17:49:22.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.488+0000 7fa460e00640 1 --2- 172.21.15.45:0/3228806582 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19aefdd0 0x55ff19af1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:49:22.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.488+0000 7fa462200640 1 --2- 172.21.15.45:0/3228806582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19af8460 0x55ff19af2200 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa45402ca30 tx=0x7fa4540bd630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ad0c9e997af85fd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:22.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.488+0000 7fa443400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:49:22.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.488+0000 7fa443400640 1 -- 172.21.15.45:0/3228806582 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4540bdf60 con 0x55ff19af8460 2024-03-28T17:49:22.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.488+0000 7fa443400640 1 -- 172.21.15.45:0/3228806582 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fa4540a80b0 con 0x55ff19af8460 2024-03-28T17:49:22.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.488+0000 7fa466fe6740 1 -- 172.21.15.45:0/3228806582 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa448016040 con 0x55ff19af8460 2024-03-28T17:49:22.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.489+0000 7fa443400640 1 -- 172.21.15.45:0/3228806582 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4540bd860 con 0x55ff19af8460 2024-03-28T17:49:22.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.489+0000 7fa466fe6740 10 client.25571 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:22.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.489+0000 7fa466fe6740 1 -- 172.21.15.45:0/3228806582 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55ff19b63dd0 con 0x55ff19af8460 2024-03-28T17:49:22.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.489+0000 7fa443400640 1 -- 172.21.15.45:0/3228806582 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa448016040 con 0x55ff19af8460 2024-03-28T17:49:22.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.489+0000 7fa443400640 1 -- 172.21.15.45:0/3228806582 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fa45403c030 con 0x55ff19af8460 2024-03-28T17:49:22.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.489+0000 7fa466fe6740 10 client.25571 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:22.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.489+0000 7fa466fe6740 10 client.25571 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:49:22.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.489+0000 7fa466fe6740 20 client.25571 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:22.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.489+0000 7fa466fe6740 1 --2- 172.21.15.45:0/3228806582 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ff19b644a0 0x55ff19b66890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:22.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.489+0000 7fa466fe6740 4 client.25571 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:49:22.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.489+0000 7fa466fe6740 1 -- 172.21.15.45:0/3228806582 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ff19a5aae0 con 0x55ff19b644a0 2024-03-28T17:49:22.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.489+0000 7fa461800640 1 --2- 172.21.15.45:0/3228806582 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ff19b644a0 0x55ff19b66890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:22.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.490+0000 7fa461800640 1 --2- 172.21.15.45:0/3228806582 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ff19b644a0 0x55ff19b66890 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9d318c71cb5b9e9 server_cookie=ac96d01b10193a6 in_seq=0 out_seq=0 2024-03-28T17:49:22.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.490+0000 7fa443400640 10 client.25571 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:22.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.490+0000 7fa443400640 1 -- 172.21.15.45:0/3228806582 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55ff19a5aae0 con 0x55ff19b644a0 2024-03-28T17:49:22.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.490+0000 7fa443400640 10 client.25571 handle_command_reply: tid=0 2024-03-28T17:49:22.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.490+0000 7fa466fe6740 1 client.25571 shutdown 2024-03-28T17:49:22.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.490+0000 7fa466fe6740 1 -- 172.21.15.45:0/3228806582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19af8460 msgr2=0x55ff19af2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:22.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.490+0000 7fa466fe6740 1 --2- 172.21.15.45:0/3228806582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19af8460 0x55ff19af2200 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa45402ca30 tx=0x7fa4540bd630 comp rx=0 tx=0).stop 2024-03-28T17:49:22.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.490+0000 7fa466fe6740 1 -- 172.21.15.45:0/3228806582 shutdown_connections 2024-03-28T17:49:22.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.490+0000 7fa466fe6740 1 --2- 172.21.15.45:0/3228806582 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ff19b644a0 0x55ff19b66890 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.490+0000 7fa466fe6740 1 --2- 172.21.15.45:0/3228806582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ff19af8460 0x55ff19af2200 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.490+0000 7fa466fe6740 1 --2- 172.21.15.45:0/3228806582 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ff19aefdd0 0x55ff19af1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.490+0000 7fa466fe6740 1 --2- 172.21.15.45:0/3228806582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ff19aef0f0 0x55ff19af1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.490+0000 7fa443400640 0 client.25571 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:22.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.490+0000 7fa466fe6740 1 -- 172.21.15.45:0/3228806582 >> 172.21.15.45:0/3228806582 conn(0x55ff19ab6200 msgr2=0x55ff19aeaf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:22.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.490+0000 7fa466fe6740 1 -- 172.21.15.45:0/3228806582 shutdown_connections 2024-03-28T17:49:22.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.490+0000 7fa466fe6740 1 -- 172.21.15.45:0/3228806582 wait complete. 2024-03-28T17:49:22.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.490+0000 7fa466fe6740 20 client.25571 trim_cache size 0 max 16384 2024-03-28T17:49:22.496 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:49:22.496 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T17:49:22.497 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T17:49:22.497 INFO:teuthology.orchestra.run.smithi045.stdout: "631fd4f2-373a-469d-a0ec-51d619a74719": { 2024-03-28T17:49:22.497 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T17:49:22.497 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "631fd4f2-373a-469d-a0ec-51d619a74719", 2024-03-28T17:49:22.497 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T17:49:22.497 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:49:22.497 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:49:22.497 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:49:22.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.491+0000 7fa462c00640 1 -- 172.21.15.45:0/1142767860 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fa4440a2a80 msgr2=0x7fa4440a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:22.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.491+0000 7fa462c00640 1 --2- 172.21.15.45:0/1142767860 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fa4440a2a80 0x7fa4440a4f40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x55ff19af0320 tx=0x7fa450005f90 comp rx=0 tx=0).stop 2024-03-28T17:49:22.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.491+0000 7fa462c00640 1 -- 172.21.15.45:0/1142767860 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa45c10bae0 msgr2=0x7fa45c19e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:22.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.491+0000 7fa462c00640 1 --2- 172.21.15.45:0/1142767860 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa45c10bae0 0x7fa45c19e860 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa454009bf0 tx=0x7fa454035e50 comp rx=0 tx=0).stop 2024-03-28T17:49:22.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.491+0000 7fa462c00640 1 -- 172.21.15.45:0/1142767860 shutdown_connections 2024-03-28T17:49:22.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.491+0000 7fa462c00640 1 --2- 172.21.15.45:0/1142767860 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fa4440a2a80 0x7fa4440a4f40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.491+0000 7fa462c00640 1 --2- 172.21.15.45:0/1142767860 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa45c10bae0 0x7fa45c19e860 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.491+0000 7fa462c00640 1 --2- 172.21.15.45:0/1142767860 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa45c102fe0 0x7fa45c19a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.491+0000 7fa462c00640 1 --2- 172.21.15.45:0/1142767860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa45c1026c0 0x7fa45c199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:22.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.491+0000 7fa462c00640 1 -- 172.21.15.45:0/1142767860 >> 172.21.15.45:0/1142767860 conn(0x7fa45c0fbec0 msgr2=0x7fa45c0fc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:22.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.491+0000 7fa462c00640 1 -- 172.21.15.45:0/1142767860 shutdown_connections 2024-03-28T17:49:22.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:22.491+0000 7fa462c00640 1 -- 172.21.15.45:0/1142767860 wait complete. 2024-03-28T17:49:22.509 INFO:tasks.cephfs.filesystem:scrub status for tag:631fd4f2-373a-469d-a0ec-51d619a74719 - {'path': '/', 'tag': '631fd4f2-373a-469d-a0ec-51d619a74719', 'options': 'recursive,force'} 2024-03-28T17:49:22.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:22 smithi045 ceph-mon[31773]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T17:49:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:22 smithi071 ceph-mon[36597]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T17:49:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:22 smithi151 ceph-mon[36481]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T17:49:23.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:23 smithi045 ceph-mon[31773]: scrub queued for path: / 2024-03-28T17:49:23.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:23 smithi045 ceph-mon[31773]: scrub summary: idle+waiting paths [/] 2024-03-28T17:49:23.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:23 smithi045 ceph-mon[31773]: scrub summary: active paths [/] 2024-03-28T17:49:23.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:23 smithi045 ceph-mon[31773]: scrub summary: idle+waiting paths [/] 2024-03-28T17:49:23.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:23 smithi151 ceph-mon[36481]: scrub queued for path: / 2024-03-28T17:49:23.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:23 smithi151 ceph-mon[36481]: scrub summary: idle+waiting paths [/] 2024-03-28T17:49:23.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:23 smithi151 ceph-mon[36481]: scrub summary: active paths [/] 2024-03-28T17:49:23.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:23 smithi151 ceph-mon[36481]: scrub summary: idle+waiting paths [/] 2024-03-28T17:49:24.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:23 smithi071 ceph-mon[36597]: scrub queued for path: / 2024-03-28T17:49:24.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:23 smithi071 ceph-mon[36597]: scrub summary: idle+waiting paths [/] 2024-03-28T17:49:24.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:23 smithi071 ceph-mon[36597]: scrub summary: active paths [/] 2024-03-28T17:49:24.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:23 smithi071 ceph-mon[36597]: scrub summary: idle+waiting paths [/] 2024-03-28T17:49:24.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:24 smithi045 ceph-mon[31773]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:49:24.987 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:24 smithi151 ceph-mon[36481]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:49:25.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:24 smithi071 ceph-mon[36597]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:49:26.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:26 smithi045 ceph-mon[31773]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:49:26.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:26 smithi045 ceph-mon[31773]: scrub summary: idle 2024-03-28T17:49:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:26 smithi071 ceph-mon[36597]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:49:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:26 smithi071 ceph-mon[36597]: scrub summary: idle 2024-03-28T17:49:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:26 smithi151 ceph-mon[36481]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:49:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:26 smithi151 ceph-mon[36481]: scrub summary: idle 2024-03-28T17:49:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:28 smithi045 ceph-mon[31773]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T17:49:29.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:28 smithi071 ceph-mon[36597]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T17:49:29.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:28 smithi151 ceph-mon[36481]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T17:49:30.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:30 smithi045 ceph-mon[31773]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T17:49:31.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:30 smithi071 ceph-mon[36597]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T17:49:31.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:30 smithi151 ceph-mon[36481]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T17:49:32.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:32 smithi045 ceph-mon[31773]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T17:49:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:32 smithi071 ceph-mon[36597]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T17:49:33.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:32 smithi151 ceph-mon[36481]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T17:49:34.874 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:34 smithi071 ceph-mon[36597]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T17:49:34.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:34 smithi045 ceph-mon[31773]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T17:49:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:34 smithi151 ceph-mon[36481]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T17:49:36.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:36 smithi045 ceph-mon[31773]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T17:49:37.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:36 smithi071 ceph-mon[36597]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T17:49:37.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:36 smithi151 ceph-mon[36481]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T17:49:38.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:38 smithi045 ceph-mon[31773]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T17:49:39.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:38 smithi071 ceph-mon[36597]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T17:49:39.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:38 smithi151 ceph-mon[36481]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T17:49:40.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:40 smithi045 ceph-mon[31773]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:49:41.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:40 smithi071 ceph-mon[36597]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:49:41.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:40 smithi151 ceph-mon[36481]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:49:42.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:42 smithi045 ceph-mon[31773]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-28T17:49:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:42 smithi071 ceph-mon[36597]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-28T17:49:43.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:42 smithi151 ceph-mon[36481]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-28T17:49:44.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:44 smithi045 ceph-mon[31773]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-28T17:49:45.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:44 smithi071 ceph-mon[36597]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-28T17:49:45.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:44 smithi151 ceph-mon[36481]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-28T17:49:46.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:46 smithi045 ceph-mon[31773]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-28T17:49:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:46 smithi071 ceph-mon[36597]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-28T17:49:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:46 smithi151 ceph-mon[36481]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-28T17:49:48.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:48 smithi045 ceph-mon[31773]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-28T17:49:49.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:48 smithi071 ceph-mon[36597]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-28T17:49:49.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:48 smithi151 ceph-mon[36481]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-28T17:49:50.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:50 smithi045 ceph-mon[31773]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-28T17:49:51.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:50 smithi071 ceph-mon[36597]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-28T17:49:51.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:50 smithi151 ceph-mon[36481]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-28T17:49:52.510 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:49:52.510 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:49:52.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.712+0000 7fdc59000640 1 -- 172.21.15.45:0/2105875419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc54101520 msgr2=0x7fdc541019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:52.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.712+0000 7fdc59000640 1 --2- 172.21.15.45:0/2105875419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc54101520 0x7fdc541019a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fdc40009930 tx=0x7fdc4002bc30 comp rx=0 tx=0).stop 2024-03-28T17:49:52.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.712+0000 7fdc59000640 1 -- 172.21.15.45:0/2105875419 shutdown_connections 2024-03-28T17:49:52.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.712+0000 7fdc59000640 1 --2- 172.21.15.45:0/2105875419 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc54101ee0 0x7fdc5410e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.712+0000 7fdc59000640 1 --2- 172.21.15.45:0/2105875419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc54101520 0x7fdc541019a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.712+0000 7fdc59000640 1 --2- 172.21.15.45:0/2105875419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc541075e0 0x7fdc541079c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.712+0000 7fdc59000640 1 -- 172.21.15.45:0/2105875419 >> 172.21.15.45:0/2105875419 conn(0x7fdc540fd010 msgr2=0x7fdc540ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:52.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.712+0000 7fdc59000640 1 -- 172.21.15.45:0/2105875419 shutdown_connections 2024-03-28T17:49:52.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.712+0000 7fdc59000640 1 -- 172.21.15.45:0/2105875419 wait complete. 2024-03-28T17:49:52.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.713+0000 7fdc59000640 1 Processor -- start 2024-03-28T17:49:52.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.714+0000 7fdc59000640 1 -- start start 2024-03-28T17:49:52.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.714+0000 7fdc59000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc54101520 0x7fdc54197d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:52.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.714+0000 7fdc59000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc54101ee0 0x7fdc541982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:52.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.714+0000 7fdc59000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc541075e0 0x7fdc5419c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:52.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.714+0000 7fdc59000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc54111910 con 0x7fdc54101520 2024-03-28T17:49:52.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.714+0000 7fdc59000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc54111790 con 0x7fdc541075e0 2024-03-28T17:49:52.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.714+0000 7fdc59000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc54111a90 con 0x7fdc54101ee0 2024-03-28T17:49:52.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.714+0000 7fdc53400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc54101520 0x7fdc54197d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:52.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.715+0000 7fdc53400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc54101520 0x7fdc54197d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46952/0 (socket says 172.21.15.45:46952) 2024-03-28T17:49:52.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.715+0000 7fdc53400640 1 -- 172.21.15.45:0/1927626607 learned_addr learned my addr 172.21.15.45:0/1927626607 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:52.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.715+0000 7fdc52a00640 1 --2- 172.21.15.45:0/1927626607 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc54101ee0 0x7fdc541982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:52.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.715+0000 7fdc53e00640 1 --2- 172.21.15.45:0/1927626607 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc541075e0 0x7fdc5419c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:52.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.715+0000 7fdc52a00640 1 -- 172.21.15.45:0/1927626607 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc541075e0 msgr2=0x7fdc5419c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:52.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.715+0000 7fdc52a00640 1 --2- 172.21.15.45:0/1927626607 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc541075e0 0x7fdc5419c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.715+0000 7fdc52a00640 1 -- 172.21.15.45:0/1927626607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc54101520 msgr2=0x7fdc54197d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:52.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.715+0000 7fdc52a00640 1 --2- 172.21.15.45:0/1927626607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc54101520 0x7fdc54197d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.715+0000 7fdc52a00640 1 -- 172.21.15.45:0/1927626607 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5419cd80 con 0x7fdc54101ee0 2024-03-28T17:49:52.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.715+0000 7fdc53e00640 1 --2- 172.21.15.45:0/1927626607 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc541075e0 0x7fdc5419c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:49:52.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.716+0000 7fdc52a00640 1 --2- 172.21.15.45:0/1927626607 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc54101ee0 0x7fdc541982d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdc40036040 tx=0x7fdc40004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:52.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.716+0000 7fdc4be00640 1 -- 172.21.15.45:0/1927626607 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc40043020 con 0x7fdc54101ee0 2024-03-28T17:49:52.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.716+0000 7fdc4be00640 1 -- 172.21.15.45:0/1927626607 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc400054e0 con 0x7fdc54101ee0 2024-03-28T17:49:52.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.716+0000 7fdc4be00640 1 -- 172.21.15.45:0/1927626607 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc40003b30 con 0x7fdc54101ee0 2024-03-28T17:49:52.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.716+0000 7fdc59000640 1 -- 172.21.15.45:0/1927626607 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc5419d010 con 0x7fdc54101ee0 2024-03-28T17:49:52.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.716+0000 7fdc59000640 1 -- 172.21.15.45:0/1927626607 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc540738d0 con 0x7fdc54101ee0 2024-03-28T17:49:52.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.721+0000 7fdc5d3d3740 1 Processor -- start 2024-03-28T17:49:52.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.721+0000 7fdc5d3d3740 1 -- start start 2024-03-28T17:49:52.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.722+0000 7fdc4be00640 1 -- 172.21.15.45:0/1927626607 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 26) v1 ==== 137466+0+0 (secure 0 0 0) 0x7fdc40046020 con 0x7fdc54101ee0 2024-03-28T17:49:52.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.722+0000 7fdc5d3d3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a71133edd0 0x55a71133db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:52.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.722+0000 7fdc5d3d3740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55a71133e0f0 0x55a71133e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:52.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.722+0000 7fdc5d3d3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a711347460 0x55a711349850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:52.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.722+0000 7fdc5d3d3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7112a2250 con 0x55a71133edd0 2024-03-28T17:49:52.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.722+0000 7fdc5d3d3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a71124aa30 con 0x55a711347460 2024-03-28T17:49:52.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.722+0000 7fdc5d3d3740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a71129e2f0 con 0x55a71133e0f0 2024-03-28T17:49:52.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.723+0000 7fdc52a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a711347460 0x55a711349850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:52.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.723+0000 7fdc53e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a71133edd0 0x55a71133db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:52.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.723+0000 7fdc52a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a711347460 0x55a711349850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:46942/0 (socket says 172.21.15.45:46942) 2024-03-28T17:49:52.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.723+0000 7fdc52a00640 1 -- 172.21.15.45:0/2539072965 learned_addr learned my addr 172.21.15.45:0/2539072965 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:52.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.723+0000 7fdc53400640 1 --2- 172.21.15.45:0/2539072965 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55a71133e0f0 0x55a71133e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:52.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.723+0000 7fdc52a00640 1 -- 172.21.15.45:0/2539072965 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55a71133e0f0 msgr2=0x55a71133e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:52.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.723+0000 7fdc52a00640 1 --2- 172.21.15.45:0/2539072965 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55a71133e0f0 0x55a71133e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.723+0000 7fdc52a00640 1 -- 172.21.15.45:0/2539072965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a71133edd0 msgr2=0x55a71133db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:52.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.723+0000 7fdc52a00640 1 --2- 172.21.15.45:0/2539072965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a71133edd0 0x55a71133db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.723+0000 7fdc52a00640 1 -- 172.21.15.45:0/2539072965 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a711235690 con 0x55a711347460 2024-03-28T17:49:52.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.724+0000 7fdc52a00640 1 --2- 172.21.15.45:0/2539072965 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a711347460 0x55a711349850 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdc400edfe0 tx=0x7fdc400f6a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58cfd71507254cc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:52.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.724+0000 7fdc48c00640 1 -- 172.21.15.45:0/2539072965 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc400ff070 con 0x55a711347460 2024-03-28T17:49:52.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.724+0000 7fdc48c00640 1 -- 172.21.15.45:0/2539072965 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc400f8e10 con 0x55a711347460 2024-03-28T17:49:52.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.724+0000 7fdc48c00640 1 -- 172.21.15.45:0/2539072965 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc400f20d0 con 0x55a711347460 2024-03-28T17:49:52.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.724+0000 7fdc4be00640 1 --2- 172.21.15.45:0/1927626607 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fdc380a2a80 0x7fdc380a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:52.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.725+0000 7fdc4be00640 1 -- 172.21.15.45:0/1927626607 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fdc400f08c0 con 0x7fdc54101ee0 2024-03-28T17:49:52.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.725+0000 7fdc53e00640 1 --2- 172.21.15.45:0/1927626607 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fdc380a2a80 0x7fdc380a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:52.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.725+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/2539072965 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a711347460 msgr2=0x55a711349850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:52.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.725+0000 7fdc5d3d3740 1 --2- 172.21.15.45:0/2539072965 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a711347460 0x55a711349850 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdc400edfe0 tx=0x7fdc400f6a70 comp rx=0 tx=0).stop 2024-03-28T17:49:52.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.726+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/2539072965 shutdown_connections 2024-03-28T17:49:52.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.726+0000 7fdc5d3d3740 1 --2- 172.21.15.45:0/2539072965 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a711347460 0x55a711349850 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.726+0000 7fdc5d3d3740 1 --2- 172.21.15.45:0/2539072965 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55a71133e0f0 0x55a71133e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.726+0000 7fdc5d3d3740 1 --2- 172.21.15.45:0/2539072965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a71133edd0 0x55a71133db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.726+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/2539072965 >> 172.21.15.45:0/2539072965 conn(0x55a711305200 msgr2=0x55a711338b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:52.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.726+0000 7fdc53e00640 1 --2- 172.21.15.45:0/1927626607 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fdc380a2a80 0x7fdc380a4f40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c009fd0 tx=0x7fdc4c009500 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:52.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.726+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/2539072965 shutdown_connections 2024-03-28T17:49:52.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.726+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/2539072965 wait complete. 2024-03-28T17:49:52.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.726+0000 7fdc5d3d3740 1 -- start start 2024-03-28T17:49:52.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.727+0000 7fdc5d3d3740 1 Processor -- start 2024-03-28T17:49:52.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.727+0000 7fdc5d3d3740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55a71133e0f0 0x55a711346b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:52.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.727+0000 7fdc5d3d3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a71133edd0 0x55a711343ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:52.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.727+0000 7fdc5d3d3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a711347460 0x55a711344430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:52.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.728+0000 7fdc5d3d3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a71134a090 con 0x55a71133edd0 2024-03-28T17:49:52.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.728+0000 7fdc5d3d3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a71125de80 con 0x55a711347460 2024-03-28T17:49:52.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.728+0000 7fdc5d3d3740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a71134a210 con 0x55a71133e0f0 2024-03-28T17:49:52.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.728+0000 7fdc5d3d3740 10 client.0 _ll_register_callbacks cb 0x55a711270130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:52.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.728+0000 7fdc53e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a711347460 0x55a711344430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:52.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.728+0000 7fdc53e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a711347460 0x55a711344430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:46958/0 (socket says 172.21.15.45:46958) 2024-03-28T17:49:52.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.728+0000 7fdc53e00640 1 -- 172.21.15.45:0/3234724708 learned_addr learned my addr 172.21.15.45:0/3234724708 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:52.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.728+0000 7fdc53400640 1 --2- 172.21.15.45:0/3234724708 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55a71133e0f0 0x55a711346b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:52.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.728+0000 7fdc52a00640 1 --2- 172.21.15.45:0/3234724708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a71133edd0 0x55a711343ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:52.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.728+0000 7fdc53e00640 1 -- 172.21.15.45:0/3234724708 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55a71133e0f0 msgr2=0x55a711346b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:52.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.728+0000 7fdc53e00640 1 --2- 172.21.15.45:0/3234724708 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55a71133e0f0 0x55a711346b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.728+0000 7fdc53e00640 1 -- 172.21.15.45:0/3234724708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a71133edd0 msgr2=0x55a711343ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:52.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.728+0000 7fdc53e00640 1 --2- 172.21.15.45:0/3234724708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a71133edd0 0x55a711343ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.728+0000 7fdc53e00640 1 -- 172.21.15.45:0/3234724708 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a7110807d0 con 0x55a711347460 2024-03-28T17:49:52.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.728+0000 7fdc52a00640 1 --2- 172.21.15.45:0/3234724708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a71133edd0 0x55a711343ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T17:49:52.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.729+0000 7fdc53e00640 1 --2- 172.21.15.45:0/3234724708 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a711347460 0x55a711344430 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c003e20 tx=0x7fdc4c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6de332c506dfd41 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:52.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.729+0000 7fdc3d600640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:49:52.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.729+0000 7fdc3d600640 1 -- 172.21.15.45:0/3234724708 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c004550 con 0x55a711347460 2024-03-28T17:49:52.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.729+0000 7fdc3d600640 1 -- 172.21.15.45:0/3234724708 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fdc4c018030 con 0x55a711347460 2024-03-28T17:49:52.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.729+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/3234724708 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdc400f8e10 con 0x55a711347460 2024-03-28T17:49:52.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.729+0000 7fdc3d600640 1 -- 172.21.15.45:0/3234724708 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc4c0046f0 con 0x55a711347460 2024-03-28T17:49:52.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.729+0000 7fdc5d3d3740 10 client.25583 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:52.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.729+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/3234724708 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a710fb63f0 con 0x55a711347460 2024-03-28T17:49:52.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.730+0000 7fdc3d600640 1 -- 172.21.15.45:0/3234724708 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc400f8e10 con 0x55a711347460 2024-03-28T17:49:52.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.730+0000 7fdc3d600640 1 -- 172.21.15.45:0/3234724708 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fdc4c028490 con 0x55a711347460 2024-03-28T17:49:52.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.730+0000 7fdc5d3d3740 10 client.25583 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:52.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.730+0000 7fdc5d3d3740 10 client.25583 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:49:52.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.730+0000 7fdc5d3d3740 20 client.25583 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:52.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.730+0000 7fdc5d3d3740 1 --2- 172.21.15.45:0/3234724708 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55a7113570e0 0x55a711359560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:52.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.730+0000 7fdc5d3d3740 4 client.25583 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:49:52.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.730+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/3234724708 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a7112af790 con 0x55a7113570e0 2024-03-28T17:49:52.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.731+0000 7fdc53400640 1 --2- 172.21.15.45:0/3234724708 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55a7113570e0 0x55a711359560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:52.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.731+0000 7fdc53400640 1 --2- 172.21.15.45:0/3234724708 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55a7113570e0 0x55a711359560 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=656e4c0ec2cf17fa server_cookie=2e8f4afcc8cbd731 in_seq=0 out_seq=0 2024-03-28T17:49:52.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.731+0000 7fdc3d600640 10 client.25583 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:52.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.732+0000 7fdc3d600640 1 -- 172.21.15.45:0/3234724708 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a7112af790 con 0x55a7113570e0 2024-03-28T17:49:52.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.732+0000 7fdc3d600640 10 client.25583 handle_command_reply: tid=0 2024-03-28T17:49:52.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.732+0000 7fdc5d3d3740 1 client.25583 shutdown 2024-03-28T17:49:52.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.732+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/3234724708 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a711347460 msgr2=0x55a711344430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:52.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.732+0000 7fdc5d3d3740 1 --2- 172.21.15.45:0/3234724708 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a711347460 0x55a711344430 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c003e20 tx=0x7fdc4c004ab0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.733+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/3234724708 shutdown_connections 2024-03-28T17:49:52.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.733+0000 7fdc5d3d3740 1 --2- 172.21.15.45:0/3234724708 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55a7113570e0 0x55a711359560 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.733+0000 7fdc5d3d3740 1 --2- 172.21.15.45:0/3234724708 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a711347460 0x55a711344430 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.733+0000 7fdc5d3d3740 1 --2- 172.21.15.45:0/3234724708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a71133edd0 0x55a711343ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.733+0000 7fdc5d3d3740 1 --2- 172.21.15.45:0/3234724708 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55a71133e0f0 0x55a711346b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.733+0000 7fdc3d600640 0 client.25583 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:52.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.733+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/3234724708 >> 172.21.15.45:0/3234724708 conn(0x55a711305200 msgr2=0x55a71133a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:52.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.733+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/3234724708 shutdown_connections 2024-03-28T17:49:52.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.733+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/3234724708 wait complete. 2024-03-28T17:49:52.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.733+0000 7fdc5d3d3740 20 client.25583 trim_cache size 0 max 16384 2024-03-28T17:49:52.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.767+0000 7fdc5d3d3740 1 Processor -- start 2024-03-28T17:49:52.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.767+0000 7fdc5d3d3740 1 -- start start 2024-03-28T17:49:52.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.768+0000 7fdc5d3d3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a71133e0f0 0x55a711346620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:52.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.768+0000 7fdc5d3d3740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55a71133edd0 0x55a711346b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:52.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.768+0000 7fdc5d3d3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a711347460 0x55a711352920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:52.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.768+0000 7fdc5d3d3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc541ad330 con 0x55a711347460 2024-03-28T17:49:52.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.768+0000 7fdc5d3d3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc541aae30 con 0x55a71133e0f0 2024-03-28T17:49:52.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.768+0000 7fdc5d3d3740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc541a8eb0 con 0x55a71133edd0 2024-03-28T17:49:52.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.768+0000 7fdc53400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a71133e0f0 0x55a711346620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:52.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.768+0000 7fdc53400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a71133e0f0 0x55a711346620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:46962/0 (socket says 172.21.15.45:46962) 2024-03-28T17:49:52.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.768+0000 7fdc53400640 1 -- 172.21.15.45:0/1957082430 learned_addr learned my addr 172.21.15.45:0/1957082430 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:52.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.768+0000 7fdc52a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55a71133edd0 0x55a711346b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:52.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.768+0000 7fdc53e00640 1 --2- 172.21.15.45:0/1957082430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a711347460 0x55a711352920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:52.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.768+0000 7fdc53400640 1 -- 172.21.15.45:0/1957082430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55a71133edd0 msgr2=0x55a711346b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:52.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.768+0000 7fdc53400640 1 --2- 172.21.15.45:0/1957082430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55a71133edd0 0x55a711346b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.768+0000 7fdc53400640 1 -- 172.21.15.45:0/1957082430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a711347460 msgr2=0x55a711352920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:52.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.768+0000 7fdc53400640 1 --2- 172.21.15.45:0/1957082430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a711347460 0x55a711352920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.768+0000 7fdc53400640 1 -- 172.21.15.45:0/1957082430 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a710f760f0 con 0x55a71133e0f0 2024-03-28T17:49:52.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.768+0000 7fdc53e00640 1 --2- 172.21.15.45:0/1957082430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a711347460 0x55a711352920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T17:49:52.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.769+0000 7fdc53400640 1 --2- 172.21.15.45:0/1957082430 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a71133e0f0 0x55a711346620 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fdc44014f10 tx=0x7fdc4400b4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a58d7670be828506 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:52.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.769+0000 7fdc3fe00640 1 -- 172.21.15.45:0/1957082430 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4400b6c0 con 0x55a71133e0f0 2024-03-28T17:49:52.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.769+0000 7fdc3fe00640 1 -- 172.21.15.45:0/1957082430 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc4402a030 con 0x55a71133e0f0 2024-03-28T17:49:52.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.769+0000 7fdc3fe00640 1 -- 172.21.15.45:0/1957082430 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc44003e10 con 0x55a71133e0f0 2024-03-28T17:49:52.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.770+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/1957082430 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a71133e0f0 msgr2=0x55a711346620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:52.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.770+0000 7fdc5d3d3740 1 --2- 172.21.15.45:0/1957082430 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a71133e0f0 0x55a711346620 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fdc44014f10 tx=0x7fdc4400b4a0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.770+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/1957082430 shutdown_connections 2024-03-28T17:49:52.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.770+0000 7fdc5d3d3740 1 --2- 172.21.15.45:0/1957082430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a711347460 0x55a711352920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.770+0000 7fdc5d3d3740 1 --2- 172.21.15.45:0/1957082430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55a71133edd0 0x55a711346b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.770+0000 7fdc5d3d3740 1 --2- 172.21.15.45:0/1957082430 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a71133e0f0 0x55a711346620 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.770+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/1957082430 >> 172.21.15.45:0/1957082430 conn(0x55a711305200 msgr2=0x55a7113427e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:52.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.770+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/1957082430 shutdown_connections 2024-03-28T17:49:52.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.771+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/1957082430 wait complete. 2024-03-28T17:49:52.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.771+0000 7fdc5d3d3740 1 -- start start 2024-03-28T17:49:52.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.771+0000 7fdc5d3d3740 1 Processor -- start 2024-03-28T17:49:52.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.772+0000 7fdc5d3d3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a71133e0f0 0x55a711341610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:52.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.772+0000 7fdc5d3d3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a71133edd0 0x55a711341b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:52.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.772+0000 7fdc5d3d3740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55a711347460 0x55a711347fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:52.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.772+0000 7fdc5d3d3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc541a94b0 con 0x55a71133e0f0 2024-03-28T17:49:52.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.772+0000 7fdc5d3d3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc541aa7d0 con 0x55a71133edd0 2024-03-28T17:49:52.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.772+0000 7fdc5d3d3740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc541a9b10 con 0x55a711347460 2024-03-28T17:49:52.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.772+0000 7fdc5d3d3740 10 client.0 _ll_register_callbacks cb 0x7fdc4c012070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:52.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.772+0000 7fdc53400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a71133e0f0 0x55a711341610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:52.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.772+0000 7fdc53e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55a711347460 0x55a711347fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:52.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.772+0000 7fdc53e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55a711347460 0x55a711347fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:33518/0 (socket says 172.21.15.45:33518) 2024-03-28T17:49:52.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.772+0000 7fdc53e00640 1 -- 172.21.15.45:0/749509905 learned_addr learned my addr 172.21.15.45:0/749509905 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:52.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.772+0000 7fdc53400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a71133e0f0 0x55a711341610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46994/0 (socket says 172.21.15.45:46994) 2024-03-28T17:49:52.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.772+0000 7fdc52a00640 1 --2- 172.21.15.45:0/749509905 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a71133edd0 0x55a711341b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:52.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.773+0000 7fdc53e00640 1 -- 172.21.15.45:0/749509905 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a71133edd0 msgr2=0x55a711341b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:52.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.773+0000 7fdc53e00640 1 --2- 172.21.15.45:0/749509905 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a71133edd0 0x55a711341b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.773+0000 7fdc53e00640 1 -- 172.21.15.45:0/749509905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a71133e0f0 msgr2=0x55a711341610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:52.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.773+0000 7fdc53e00640 1 --2- 172.21.15.45:0/749509905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a71133e0f0 0x55a711341610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.773+0000 7fdc53e00640 1 -- 172.21.15.45:0/749509905 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a7113486d0 con 0x55a711347460 2024-03-28T17:49:52.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.773+0000 7fdc52a00640 1 --2- 172.21.15.45:0/749509905 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a71133edd0 0x55a711341b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:49:52.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.773+0000 7fdc53e00640 1 --2- 172.21.15.45:0/749509905 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55a711347460 0x55a711347fc0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c012190 tx=0x7fdc4c012760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61654288f852c11a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:52.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.773+0000 7fdc3d600640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:49:52.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.773+0000 7fdc3d600640 1 -- 172.21.15.45:0/749509905 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c028420 con 0x55a711347460 2024-03-28T17:49:52.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.774+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/749509905 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdc4402a030 con 0x55a711347460 2024-03-28T17:49:52.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.774+0000 7fdc3d600640 1 -- 172.21.15.45:0/749509905 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fdc4c0285c0 con 0x55a711347460 2024-03-28T17:49:52.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.774+0000 7fdc3d600640 1 -- 172.21.15.45:0/749509905 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc4c004b20 con 0x55a711347460 2024-03-28T17:49:52.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.774+0000 7fdc5d3d3740 10 client.25564 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:52.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.774+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/749509905 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a7113b3370 con 0x55a711347460 2024-03-28T17:49:52.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.774+0000 7fdc3d600640 1 -- 172.21.15.45:0/749509905 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc4402a030 con 0x55a711347460 2024-03-28T17:49:52.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.774+0000 7fdc3d600640 1 -- 172.21.15.45:0/749509905 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fdc4c037030 con 0x55a711347460 2024-03-28T17:49:52.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.774+0000 7fdc5d3d3740 10 client.25564 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:52.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.774+0000 7fdc5d3d3740 10 client.25564 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:49:52.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.774+0000 7fdc5d3d3740 20 client.25564 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:52.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.774+0000 7fdc5d3d3740 1 --2- 172.21.15.45:0/749509905 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55a7113b3a50 0x55a7113b5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:52.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.774+0000 7fdc5d3d3740 4 client.25564 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:49:52.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.774+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/749509905 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a7112a9ae0 con 0x55a7113b3a50 2024-03-28T17:49:52.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.775+0000 7fdc53400640 1 --2- 172.21.15.45:0/749509905 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55a7113b3a50 0x55a7113b5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:52.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.775+0000 7fdc53400640 1 --2- 172.21.15.45:0/749509905 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55a7113b3a50 0x55a7113b5e70 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f65e8f7f9ac78834 server_cookie=bc9981439187f84e in_seq=0 out_seq=0 2024-03-28T17:49:52.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.775+0000 7fdc3d600640 10 client.25564 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:52.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.775+0000 7fdc3d600640 1 -- 172.21.15.45:0/749509905 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55a7112a9ae0 con 0x55a7113b3a50 2024-03-28T17:49:52.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.775+0000 7fdc3d600640 10 client.25564 handle_command_reply: tid=0 2024-03-28T17:49:52.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.775+0000 7fdc5d3d3740 1 client.25564 shutdown 2024-03-28T17:49:52.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.775+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/749509905 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55a711347460 msgr2=0x55a711347fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:52.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.775+0000 7fdc5d3d3740 1 --2- 172.21.15.45:0/749509905 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55a711347460 0x55a711347fc0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c012190 tx=0x7fdc4c012760 comp rx=0 tx=0).stop 2024-03-28T17:49:52.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.776+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/749509905 shutdown_connections 2024-03-28T17:49:52.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.776+0000 7fdc5d3d3740 1 --2- 172.21.15.45:0/749509905 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55a7113b3a50 0x55a7113b5e70 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.776+0000 7fdc5d3d3740 1 --2- 172.21.15.45:0/749509905 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55a711347460 0x55a711347fc0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.776+0000 7fdc5d3d3740 1 --2- 172.21.15.45:0/749509905 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a71133edd0 0x55a711341b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.776+0000 7fdc5d3d3740 1 --2- 172.21.15.45:0/749509905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a71133e0f0 0x55a711341610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.776+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/749509905 >> 172.21.15.45:0/749509905 conn(0x55a711305200 msgr2=0x55a711337330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:52.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.776+0000 7fdc3d600640 0 client.25564 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:52.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.776+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/749509905 shutdown_connections 2024-03-28T17:49:52.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.776+0000 7fdc5d3d3740 1 -- 172.21.15.45:0/749509905 wait complete. 2024-03-28T17:49:52.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.776+0000 7fdc5d3d3740 20 client.25564 trim_cache size 0 max 16384 2024-03-28T17:49:52.780 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:49:52.780 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-28T17:49:52.780 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-28T17:49:52.780 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:49:52.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.777+0000 7fdc59000640 1 -- 172.21.15.45:0/1927626607 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fdc380a2a80 msgr2=0x7fdc380a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:52.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.777+0000 7fdc59000640 1 --2- 172.21.15.45:0/1927626607 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fdc380a2a80 0x7fdc380a4f40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c009fd0 tx=0x7fdc4c009500 comp rx=0 tx=0).stop 2024-03-28T17:49:52.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.777+0000 7fdc59000640 1 -- 172.21.15.45:0/1927626607 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc54101ee0 msgr2=0x7fdc541982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:52.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.777+0000 7fdc59000640 1 --2- 172.21.15.45:0/1927626607 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc54101ee0 0x7fdc541982d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdc40036040 tx=0x7fdc40004360 comp rx=0 tx=0).stop 2024-03-28T17:49:52.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.777+0000 7fdc59000640 1 -- 172.21.15.45:0/1927626607 shutdown_connections 2024-03-28T17:49:52.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.777+0000 7fdc59000640 1 --2- 172.21.15.45:0/1927626607 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fdc380a2a80 0x7fdc380a4f40 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.777+0000 7fdc59000640 1 --2- 172.21.15.45:0/1927626607 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc541075e0 0x7fdc5419c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.777+0000 7fdc59000640 1 --2- 172.21.15.45:0/1927626607 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc54101ee0 0x7fdc541982d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.777+0000 7fdc59000640 1 --2- 172.21.15.45:0/1927626607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc54101520 0x7fdc54197d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.777+0000 7fdc59000640 1 -- 172.21.15.45:0/1927626607 >> 172.21.15.45:0/1927626607 conn(0x7fdc540fd010 msgr2=0x7fdc540fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:52.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.777+0000 7fdc59000640 1 -- 172.21.15.45:0/1927626607 shutdown_connections 2024-03-28T17:49:52.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.777+0000 7fdc59000640 1 -- 172.21.15.45:0/1927626607 wait complete. 2024-03-28T17:49:52.803 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T17:49:52.804 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:49:52.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:52 smithi045 ceph-mon[31773]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-28T17:49:52.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.997+0000 7f86eb200640 1 -- 172.21.15.45:0/2183576322 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86ec1096a0 msgr2=0x7f86ec109a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:52.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.997+0000 7f86eb200640 1 --2- 172.21.15.45:0/2183576322 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86ec1096a0 0x7f86ec109a80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f86dc009900 tx=0x7f86dc02bc50 comp rx=0 tx=0).stop 2024-03-28T17:49:52.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.998+0000 7f86eb200640 1 -- 172.21.15.45:0/2183576322 shutdown_connections 2024-03-28T17:49:52.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.998+0000 7f86eb200640 1 --2- 172.21.15.45:0/2183576322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86ec104050 0x7f86ec110560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.998+0000 7f86eb200640 1 --2- 172.21.15.45:0/2183576322 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86ec103690 0x7f86ec103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.998+0000 7f86eb200640 1 --2- 172.21.15.45:0/2183576322 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86ec1096a0 0x7f86ec109a80 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:52.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.998+0000 7f86eb200640 1 -- 172.21.15.45:0/2183576322 >> 172.21.15.45:0/2183576322 conn(0x7f86ec0ff140 msgr2=0x7f86ec101560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:52.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.998+0000 7f86eb200640 1 -- 172.21.15.45:0/2183576322 shutdown_connections 2024-03-28T17:49:52.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.998+0000 7f86eb200640 1 -- 172.21.15.45:0/2183576322 wait complete. 2024-03-28T17:49:53.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.999+0000 7f86eb200640 1 Processor -- start 2024-03-28T17:49:53.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.999+0000 7f86eb200640 1 -- start start 2024-03-28T17:49:53.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.999+0000 7f86eb200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86ec103690 0x7f86ec19a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.999+0000 7f86eb200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86ec104050 0x7f86ec19a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.999+0000 7f86eb200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86ec1096a0 0x7f86ec19e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.999+0000 7f86eb200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec113ac0 con 0x7f86ec1096a0 2024-03-28T17:49:53.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.999+0000 7f86eb200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec113940 con 0x7f86ec103690 2024-03-28T17:49:53.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:52.999+0000 7f86eb200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec113c40 con 0x7f86ec104050 2024-03-28T17:49:53.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.000+0000 7f86e9e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86ec103690 0x7f86ec19a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.000+0000 7f86e9400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86ec104050 0x7f86ec19a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.000+0000 7f86e9400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86ec104050 0x7f86ec19a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:33538/0 (socket says 172.21.15.45:33538) 2024-03-28T17:49:53.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.000+0000 7f86e9e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86ec103690 0x7f86ec19a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:46994/0 (socket says 172.21.15.45:46994) 2024-03-28T17:49:53.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.000+0000 7f86e9400640 1 -- 172.21.15.45:0/1741705831 learned_addr learned my addr 172.21.15.45:0/1741705831 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:53.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.000+0000 7f86ea800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86ec1096a0 0x7f86ec19e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.000+0000 7f86e9e00640 1 -- 172.21.15.45:0/1741705831 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86ec104050 msgr2=0x7f86ec19a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:53.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.000+0000 7f86e9e00640 1 --2- 172.21.15.45:0/1741705831 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86ec104050 0x7f86ec19a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.000+0000 7f86e9e00640 1 -- 172.21.15.45:0/1741705831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86ec1096a0 msgr2=0x7f86ec19e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:53.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.000+0000 7f86e9e00640 1 --2- 172.21.15.45:0/1741705831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86ec1096a0 0x7f86ec19e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.000+0000 7f86e9e00640 1 -- 172.21.15.45:0/1741705831 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ec19efc0 con 0x7f86ec103690 2024-03-28T17:49:53.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.001+0000 7f86e9e00640 1 --2- 172.21.15.45:0/1741705831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86ec103690 0x7f86ec19a020 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f86dc02c190 tx=0x7f86dc0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:53.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.001+0000 7f86e2a00640 1 -- 172.21.15.45:0/1741705831 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc041070 con 0x7f86ec103690 2024-03-28T17:49:53.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.001+0000 7f86e2a00640 1 -- 172.21.15.45:0/1741705831 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f86dc02cce0 con 0x7f86ec103690 2024-03-28T17:49:53.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.001+0000 7f86e2a00640 1 -- 172.21.15.45:0/1741705831 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc003c70 con 0x7f86ec103690 2024-03-28T17:49:53.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.001+0000 7f86eb200640 1 -- 172.21.15.45:0/1741705831 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86ec19f250 con 0x7f86ec103690 2024-03-28T17:49:53.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.001+0000 7f86eb200640 1 -- 172.21.15.45:0/1741705831 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86ec19f6c0 con 0x7f86ec103690 2024-03-28T17:49:53.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.003+0000 7f86eb200640 1 -- 172.21.15.45:0/1741705831 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86ec104dd0 con 0x7f86ec103690 2024-03-28T17:49:53.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.005+0000 7f86e2a00640 1 -- 172.21.15.45:0/1741705831 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 26) v1 ==== 137466+0+0 (secure 0 0 0) 0x7f86dc044070 con 0x7f86ec103690 2024-03-28T17:49:53.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.008+0000 7f86e2a00640 1 --2- 172.21.15.45:0/1741705831 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f86d00a2a80 0x7f86d00a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.008+0000 7f86e2a00640 1 -- 172.21.15.45:0/1741705831 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f86dc0f0aa0 con 0x7f86ec103690 2024-03-28T17:49:53.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.008+0000 7f86e9400640 1 --2- 172.21.15.45:0/1741705831 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f86d00a2a80 0x7f86d00a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.009+0000 7f86e9400640 1 --2- 172.21.15.45:0/1741705831 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f86d00a2a80 0x7f86d00a4f40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f86d80095f0 tx=0x7f86d8005f90 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:53.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.011+0000 7f86e2a00640 1 -- 172.21.15.45:0/1741705831 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f86ec19f6c0 con 0x7f86ec103690 2024-03-28T17:49:53.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:52 smithi071 ceph-mon[36597]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-28T17:49:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:52 smithi151 ceph-mon[36481]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-28T17:49:53.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.327+0000 7f86eb200640 1 -- 172.21.15.45:0/1741705831 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86ec10bc00 con 0x7f86ec103690 2024-03-28T17:49:53.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.328+0000 7f86e2a00640 1 -- 172.21.15.45:0/1741705831 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7f86dc0bd120 con 0x7f86ec103690 2024-03-28T17:49:53.329 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:49:53.329 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6839/2313500150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":2313500150},{"type":"v1","addr":"172.21.15.45:6839","nonce":2313500150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6841/2647972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":2647972571},{"type":"v1","addr":"172.21.15.45:6841","nonce":2647972571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T17:46:12.261135+0000","modified":"2024-03-28T17:48:38.762497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24584,"mds_1":24530,"mds_2":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6835/2876354410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":2876354410},{"type":"v1","addr":"172.21.15.151:6835","nonce":2876354410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24500":{"gid":24500,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/4234998385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":4234998385},{"type":"v1","addr":"172.21.15.71:6835","nonce":4234998385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6837/1238964845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":1238964845},{"type":"v1","addr":"172.21.15.151:6837","nonce":1238964845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.151:6839/3003564901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6838","nonce":3003564901},{"type":"v1","addr":"172.21.15.151:6839","nonce":3003564901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6837/2008752655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2008752655},{"type":"v1","addr":"172.21.15.71:6837","nonce":2008752655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.71:6839/2709938800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":2709938800},{"type":"v1","addr":"172.21.15.71:6839","nonce":2709938800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24584,"qdb_cluster":[24584,24529,24530]},"id":1}]} 2024-03-28T17:49:53.329 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 23 2024-03-28T17:49:53.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.331+0000 7f86eb200640 1 -- 172.21.15.45:0/1741705831 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f86d00a2a80 msgr2=0x7f86d00a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:53.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.331+0000 7f86eb200640 1 --2- 172.21.15.45:0/1741705831 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f86d00a2a80 0x7f86d00a4f40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f86d80095f0 tx=0x7f86d8005f90 comp rx=0 tx=0).stop 2024-03-28T17:49:53.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.331+0000 7f86eb200640 1 -- 172.21.15.45:0/1741705831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86ec103690 msgr2=0x7f86ec19a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:53.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.331+0000 7f86eb200640 1 --2- 172.21.15.45:0/1741705831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86ec103690 0x7f86ec19a020 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f86dc02c190 tx=0x7f86dc0042f0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.331+0000 7f86eb200640 1 -- 172.21.15.45:0/1741705831 shutdown_connections 2024-03-28T17:49:53.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.331+0000 7f86eb200640 1 --2- 172.21.15.45:0/1741705831 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f86d00a2a80 0x7f86d00a4f40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.331+0000 7f86eb200640 1 --2- 172.21.15.45:0/1741705831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86ec1096a0 0x7f86ec19e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.331+0000 7f86eb200640 1 --2- 172.21.15.45:0/1741705831 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86ec104050 0x7f86ec19a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.331+0000 7f86eb200640 1 --2- 172.21.15.45:0/1741705831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86ec103690 0x7f86ec19a020 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.331+0000 7f86eb200640 1 -- 172.21.15.45:0/1741705831 >> 172.21.15.45:0/1741705831 conn(0x7f86ec0ff140 msgr2=0x7f86ec0ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:53.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.331+0000 7f86eb200640 1 -- 172.21.15.45:0/1741705831 shutdown_connections 2024-03-28T17:49:53.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.331+0000 7f86eb200640 1 -- 172.21.15.45:0/1741705831 wait complete. 2024-03-28T17:49:53.350 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T17:49:53.350 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T17:49:53.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.592+0000 7fa0b3200640 1 -- 172.21.15.45:0/2447635675 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0b410ddb0 msgr2=0x7fa0b41101c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:53.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.592+0000 7fa0b3200640 1 --2- 172.21.15.45:0/2447635675 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0b410ddb0 0x7fa0b41101c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac009ca0 tx=0x7fa0ac02c180 comp rx=0 tx=0).stop 2024-03-28T17:49:53.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.592+0000 7fa0b3200640 1 -- 172.21.15.45:0/2447635675 shutdown_connections 2024-03-28T17:49:53.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.592+0000 7fa0b3200640 1 --2- 172.21.15.45:0/2447635675 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0b410ddb0 0x7fa0b41101c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.592+0000 7fa0b3200640 1 --2- 172.21.15.45:0/2447635675 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0b4074390 0x7fa0b410d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.592+0000 7fa0b3200640 1 --2- 172.21.15.45:0/2447635675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0b4073a70 0x7fa0b4073e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.592+0000 7fa0b3200640 1 -- 172.21.15.45:0/2447635675 >> 172.21.15.45:0/2447635675 conn(0x7fa0b40fd040 msgr2=0x7fa0b40ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:53.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.592+0000 7fa0b3200640 1 -- 172.21.15.45:0/2447635675 shutdown_connections 2024-03-28T17:49:53.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.592+0000 7fa0b3200640 1 -- 172.21.15.45:0/2447635675 wait complete. 2024-03-28T17:49:53.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.593+0000 7fa0b3200640 1 Processor -- start 2024-03-28T17:49:53.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.593+0000 7fa0b3200640 1 -- start start 2024-03-28T17:49:53.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.593+0000 7fa0b3200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0b4073a70 0x7fa0b4199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.593+0000 7fa0b3200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0b4074390 0x7fa0b419a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.594+0000 7fa0b3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0b410ddb0 0x7fa0b419e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.594+0000 7fa0b3200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b4113b10 con 0x7fa0b410ddb0 2024-03-28T17:49:53.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.594+0000 7fa0b3200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b4113990 con 0x7fa0b4074390 2024-03-28T17:49:53.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.594+0000 7fa0b3200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b4113c90 con 0x7fa0b4073a70 2024-03-28T17:49:53.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.594+0000 7fa0b1e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0b4073a70 0x7fa0b4199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.594+0000 7fa0b1400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0b4074390 0x7fa0b419a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.594+0000 7fa0b1e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0b4073a70 0x7fa0b4199f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:33562/0 (socket says 172.21.15.45:33562) 2024-03-28T17:49:53.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.594+0000 7fa0b1e00640 1 -- 172.21.15.45:0/1101173981 learned_addr learned my addr 172.21.15.45:0/1101173981 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:53.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.594+0000 7fa0b2800640 1 --2- 172.21.15.45:0/1101173981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0b410ddb0 0x7fa0b419e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.594+0000 7fa0b1e00640 1 -- 172.21.15.45:0/1101173981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0b4074390 msgr2=0x7fa0b419a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:53.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.594+0000 7fa0b1e00640 1 --2- 172.21.15.45:0/1101173981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0b4074390 0x7fa0b419a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.594+0000 7fa0b1e00640 1 -- 172.21.15.45:0/1101173981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0b410ddb0 msgr2=0x7fa0b419e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:53.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.594+0000 7fa0b1e00640 1 --2- 172.21.15.45:0/1101173981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0b410ddb0 0x7fa0b419e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.594+0000 7fa0b1e00640 1 -- 172.21.15.45:0/1101173981 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b419ef80 con 0x7fa0b4073a70 2024-03-28T17:49:53.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.594+0000 7fa0b1400640 1 --2- 172.21.15.45:0/1101173981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0b4074390 0x7fa0b419a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:49:53.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.595+0000 7fa0b1e00640 1 --2- 172.21.15.45:0/1101173981 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0b4073a70 0x7fa0b4199f90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa0a400e9f0 tx=0x7fa0a400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:53.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.595+0000 7fa0aaa00640 1 -- 172.21.15.45:0/1101173981 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a400cd80 con 0x7fa0b4073a70 2024-03-28T17:49:53.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.595+0000 7fa0aaa00640 1 -- 172.21.15.45:0/1101173981 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0a4010910 con 0x7fa0b4073a70 2024-03-28T17:49:53.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.595+0000 7fa0aaa00640 1 -- 172.21.15.45:0/1101173981 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a4018990 con 0x7fa0b4073a70 2024-03-28T17:49:53.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.595+0000 7fa0b3200640 1 -- 172.21.15.45:0/1101173981 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b419f270 con 0x7fa0b4073a70 2024-03-28T17:49:53.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.595+0000 7fa0b3200640 1 -- 172.21.15.45:0/1101173981 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b41a6b00 con 0x7fa0b4073a70 2024-03-28T17:49:53.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.599+0000 7fa0bb990740 1 Processor -- start 2024-03-28T17:49:53.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.599+0000 7fa0bb990740 1 -- start start 2024-03-28T17:49:53.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.600+0000 7fa0aaa00640 1 -- 172.21.15.45:0/1101173981 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 26) v1 ==== 137466+0+0 (secure 0 0 0) 0x7fa0a4010ac0 con 0x7fa0b4073a70 2024-03-28T17:49:53.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.600+0000 7fa0bb990740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ad2850 0x562264ad2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.600+0000 7fa0bb990740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562264ad3200 0x562264ae4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.600+0000 7fa0bb990740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ae4610 0x562264ae6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.600+0000 7fa0bb990740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562264a362c0 con 0x562264ad3200 2024-03-28T17:49:53.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.600+0000 7fa0bb990740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622649dec90 con 0x562264ad2850 2024-03-28T17:49:53.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.600+0000 7fa0bb990740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x562264a32360 con 0x562264ae4610 2024-03-28T17:49:53.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.600+0000 7fa0b2800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ad2850 0x562264ad2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.601+0000 7fa0b2800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ad2850 0x562264ad2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:47024/0 (socket says 172.21.15.45:47024) 2024-03-28T17:49:53.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.601+0000 7fa0b2800640 1 -- 172.21.15.45:0/2659461825 learned_addr learned my addr 172.21.15.45:0/2659461825 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:53.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.601+0000 7fa0b1400640 1 --2- 172.21.15.45:0/2659461825 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ae4610 0x562264ae6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.601+0000 7fa0b1e00640 1 --2- 172.21.15.45:0/2659461825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562264ad3200 0x562264ae4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.601+0000 7fa0b1400640 1 -- 172.21.15.45:0/2659461825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ad2850 msgr2=0x562264ad2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:53.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.601+0000 7fa0b1400640 1 --2- 172.21.15.45:0/2659461825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ad2850 0x562264ad2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.601+0000 7fa0b1400640 1 -- 172.21.15.45:0/2659461825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562264ad3200 msgr2=0x562264ae4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:53.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.601+0000 7fa0b1400640 1 --2- 172.21.15.45:0/2659461825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562264ad3200 0x562264ae4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.601+0000 7fa0b1400640 1 -- 172.21.15.45:0/2659461825 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562264a03080 con 0x562264ae4610 2024-03-28T17:49:53.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.601+0000 7fa0b1400640 1 --2- 172.21.15.45:0/2659461825 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ae4610 0x562264ae6a00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0009b80 tx=0x7fa0a00117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4f8a2174ad44c3d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:53.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.602+0000 7fa09f000640 1 -- 172.21.15.45:0/2659461825 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a0012450 con 0x562264ae4610 2024-03-28T17:49:53.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.602+0000 7fa09f000640 1 -- 172.21.15.45:0/2659461825 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0a0012ad0 con 0x562264ae4610 2024-03-28T17:49:53.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.602+0000 7fa09f000640 1 -- 172.21.15.45:0/2659461825 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a000fb10 con 0x562264ae4610 2024-03-28T17:49:53.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.602+0000 7fa0aaa00640 1 --2- 172.21.15.45:0/1101173981 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fa0980a2b00 0x7fa0980a4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.602+0000 7fa0aaa00640 1 -- 172.21.15.45:0/1101173981 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fa0a40cf380 con 0x7fa0b4073a70 2024-03-28T17:49:53.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.603+0000 7fa0b2800640 1 --2- 172.21.15.45:0/1101173981 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fa0980a2b00 0x7fa0980a4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.603+0000 7fa0bb990740 1 -- 172.21.15.45:0/2659461825 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ae4610 msgr2=0x562264ae6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:53.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.603+0000 7fa0bb990740 1 --2- 172.21.15.45:0/2659461825 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ae4610 0x562264ae6a00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0009b80 tx=0x7fa0a00117b0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.603+0000 7fa0bb990740 1 -- 172.21.15.45:0/2659461825 shutdown_connections 2024-03-28T17:49:53.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.603+0000 7fa0bb990740 1 --2- 172.21.15.45:0/2659461825 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ae4610 0x562264ae6a00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.603+0000 7fa0bb990740 1 --2- 172.21.15.45:0/2659461825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562264ad3200 0x562264ae4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.603+0000 7fa0bb990740 1 --2- 172.21.15.45:0/2659461825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ad2850 0x562264ad2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.604+0000 7fa0bb990740 1 -- 172.21.15.45:0/2659461825 >> 172.21.15.45:0/2659461825 conn(0x562264accec0 msgr2=0x562264acd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:53.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.604+0000 7fa0b2800640 1 --2- 172.21.15.45:0/1101173981 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fa0980a2b00 0x7fa0980a4fc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac009a60 tx=0x7fa0ac005c50 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:53.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.604+0000 7fa0bb990740 1 -- 172.21.15.45:0/2659461825 shutdown_connections 2024-03-28T17:49:53.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.604+0000 7fa0bb990740 1 -- 172.21.15.45:0/2659461825 wait complete. 2024-03-28T17:49:53.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.604+0000 7fa0bb990740 1 -- start start 2024-03-28T17:49:53.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.605+0000 7fa0bb990740 1 Processor -- start 2024-03-28T17:49:53.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.605+0000 7fa0bb990740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562264ad2850 0x562264ae3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.605+0000 7fa0bb990740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ad3200 0x562264ae1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.605+0000 7fa0bb990740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ae4610 0x562264ae19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.605+0000 7fa0bb990740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562264ae7520 con 0x562264ad2850 2024-03-28T17:49:53.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.605+0000 7fa0bb990740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622649f20d0 con 0x562264ae4610 2024-03-28T17:49:53.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.605+0000 7fa0bb990740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x562264ae76a0 con 0x562264ad3200 2024-03-28T17:49:53.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.606+0000 7fa0bb990740 10 client.0 _ll_register_callbacks cb 0x5622649f6e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:53.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.606+0000 7fa0b2800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ae4610 0x562264ae19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.606+0000 7fa0b2800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ae4610 0x562264ae19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:47026/0 (socket says 172.21.15.45:47026) 2024-03-28T17:49:53.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.606+0000 7fa0b1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562264ad2850 0x562264ae3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.606+0000 7fa0b2800640 1 -- 172.21.15.45:0/3257447675 learned_addr learned my addr 172.21.15.45:0/3257447675 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:53.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.606+0000 7fa0b1400640 1 --2- 172.21.15.45:0/3257447675 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ad3200 0x562264ae1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.606+0000 7fa0b1e00640 1 -- 172.21.15.45:0/3257447675 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ad3200 msgr2=0x562264ae1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:53.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.606+0000 7fa0b1e00640 1 --2- 172.21.15.45:0/3257447675 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ad3200 0x562264ae1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.606+0000 7fa0b1e00640 1 -- 172.21.15.45:0/3257447675 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ae4610 msgr2=0x562264ae19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:53.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.606+0000 7fa0b1e00640 1 --2- 172.21.15.45:0/3257447675 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ae4610 0x562264ae19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.606+0000 7fa0b1e00640 1 -- 172.21.15.45:0/3257447675 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56226474a3f0 con 0x562264ad2850 2024-03-28T17:49:53.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.606+0000 7fa0b1400640 1 --2- 172.21.15.45:0/3257447675 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ad3200 0x562264ae1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:49:53.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.607+0000 7fa0b1e00640 1 --2- 172.21.15.45:0/3257447675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562264ad2850 0x562264ae3f60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa0a40cc9e0 tx=0x7fa0a409b610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b048e76ec383751a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:53.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.607+0000 7fa093400640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-28T17:49:53.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.607+0000 7fa093400640 1 -- 172.21.15.45:0/3257447675 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a409bf40 con 0x562264ad2850 2024-03-28T17:49:53.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.607+0000 7fa093400640 1 -- 172.21.15.45:0/3257447675 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fa0a40ccc90 con 0x562264ad2850 2024-03-28T17:49:53.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.607+0000 7fa0bb990740 1 -- 172.21.15.45:0/3257447675 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa0a0012ad0 con 0x562264ad2850 2024-03-28T17:49:53.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.607+0000 7fa093400640 1 -- 172.21.15.45:0/3257447675 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0a40cce90 con 0x562264ad2850 2024-03-28T17:49:53.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.607+0000 7fa0bb990740 10 client.15453 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:53.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.607+0000 7fa0bb990740 1 -- 172.21.15.45:0/3257447675 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5622648147d0 con 0x562264ad2850 2024-03-28T17:49:53.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.608+0000 7fa093400640 1 -- 172.21.15.45:0/3257447675 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0a0012ad0 con 0x562264ad2850 2024-03-28T17:49:53.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.608+0000 7fa093400640 1 -- 172.21.15.45:0/3257447675 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fa0a409cb70 con 0x562264ad2850 2024-03-28T17:49:53.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.608+0000 7fa0bb990740 10 client.15453 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:53.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.608+0000 7fa0bb990740 10 client.15453 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-28T17:49:53.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.608+0000 7fa0bb990740 20 client.15453 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:53.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.608+0000 7fa0bb990740 1 --2- 172.21.15.45:0/3257447675 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x562264af4460 0x562264af6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.608+0000 7fa0bb990740 4 client.15453 mds_command: new command op to 24529 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:49:53.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.608+0000 7fa0bb990740 1 -- 172.21.15.45:0/3257447675 --> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562264a43800 con 0x562264af4460 2024-03-28T17:49:53.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.608+0000 7fa0b1400640 1 --2- 172.21.15.45:0/3257447675 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x562264af4460 0x562264af6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.609+0000 7fa0b1400640 1 --2- 172.21.15.45:0/3257447675 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x562264af4460 0x562264af6850 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d22340fb4874e33d server_cookie=1fa894cf6ae1949b in_seq=0 out_seq=0 2024-03-28T17:49:53.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.609+0000 7fa093400640 10 client.15453 ms_handle_connect on v2:172.21.15.151:6838/3003564901 2024-03-28T17:49:53.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.609+0000 7fa093400640 1 -- 172.21.15.45:0/3257447675 <== mds.2 v2:172.21.15.151:6838/3003564901 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562264a43800 con 0x562264af4460 2024-03-28T17:49:53.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.609+0000 7fa093400640 10 client.15453 handle_command_reply: tid=0 2024-03-28T17:49:53.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.610+0000 7fa0bb990740 1 client.15453 shutdown 2024-03-28T17:49:53.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.610+0000 7fa0bb990740 1 -- 172.21.15.45:0/3257447675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562264ad2850 msgr2=0x562264ae3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:53.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.610+0000 7fa0bb990740 1 --2- 172.21.15.45:0/3257447675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562264ad2850 0x562264ae3f60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa0a40cc9e0 tx=0x7fa0a409b610 comp rx=0 tx=0).stop 2024-03-28T17:49:53.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.610+0000 7fa0bb990740 1 -- 172.21.15.45:0/3257447675 shutdown_connections 2024-03-28T17:49:53.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.610+0000 7fa0bb990740 1 --2- 172.21.15.45:0/3257447675 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x562264af4460 0x562264af6850 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.610+0000 7fa0bb990740 1 --2- 172.21.15.45:0/3257447675 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ae4610 0x562264ae19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.610+0000 7fa0bb990740 1 --2- 172.21.15.45:0/3257447675 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ad3200 0x562264ae1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.610+0000 7fa0bb990740 1 --2- 172.21.15.45:0/3257447675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562264ad2850 0x562264ae3f60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.610+0000 7fa0bb990740 1 -- 172.21.15.45:0/3257447675 >> 172.21.15.45:0/3257447675 conn(0x562264accec0 msgr2=0x562264acecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:53.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.610+0000 7fa093400640 0 client.15453 ms_handle_reset on v2:172.21.15.151:6838/3003564901 2024-03-28T17:49:53.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.610+0000 7fa0bb990740 1 -- 172.21.15.45:0/3257447675 shutdown_connections 2024-03-28T17:49:53.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.610+0000 7fa0bb990740 1 -- 172.21.15.45:0/3257447675 wait complete. 2024-03-28T17:49:53.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.610+0000 7fa0bb990740 20 client.15453 trim_cache size 0 max 16384 2024-03-28T17:49:53.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.626+0000 7fa0bb990740 1 Processor -- start 2024-03-28T17:49:53.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.626+0000 7fa0bb990740 1 -- start start 2024-03-28T17:49:53.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.626+0000 7fa0bb990740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ad2850 0x562264acedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.626+0000 7fa0bb990740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562264ad3200 0x562264ae0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.626+0000 7fa0bb990740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ae4610 0x562264ae1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.626+0000 7fa0bb990740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b41ad460 con 0x562264ad3200 2024-03-28T17:49:53.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.626+0000 7fa0bb990740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b41aaf60 con 0x562264ad2850 2024-03-28T17:49:53.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.626+0000 7fa0bb990740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b41a8fe0 con 0x562264ae4610 2024-03-28T17:49:53.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.627+0000 7fa0b1e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ad2850 0x562264acedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.627+0000 7fa0b1e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ad2850 0x562264acedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:47028/0 (socket says 172.21.15.45:47028) 2024-03-28T17:49:53.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.627+0000 7fa0b1e00640 1 -- 172.21.15.45:0/1433997346 learned_addr learned my addr 172.21.15.45:0/1433997346 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:53.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.627+0000 7fa0b1400640 1 --2- 172.21.15.45:0/1433997346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562264ad3200 0x562264ae0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.627+0000 7fa0b2800640 1 --2- 172.21.15.45:0/1433997346 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ae4610 0x562264ae1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.627+0000 7fa0b1e00640 1 -- 172.21.15.45:0/1433997346 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ae4610 msgr2=0x562264ae1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:53.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.627+0000 7fa0b1e00640 1 --2- 172.21.15.45:0/1433997346 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ae4610 0x562264ae1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.627+0000 7fa0b1e00640 1 -- 172.21.15.45:0/1433997346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562264ad3200 msgr2=0x562264ae0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:53.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.627+0000 7fa0b1e00640 1 --2- 172.21.15.45:0/1433997346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562264ad3200 0x562264ae0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.627+0000 7fa0b1e00640 1 -- 172.21.15.45:0/1433997346 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562264ad1bb0 con 0x562264ad2850 2024-03-28T17:49:53.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.627+0000 7fa0b2800640 1 --2- 172.21.15.45:0/1433997346 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ae4610 0x562264ae1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:49:53.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.627+0000 7fa0b1e00640 1 --2- 172.21.15.45:0/1433997346 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ad2850 0x562264acedf0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4023040 tx=0x7fa0a4000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc041ac632e6c30d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:53.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.627+0000 7fa09e600640 1 -- 172.21.15.45:0/1433997346 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a409bf40 con 0x562264ad2850 2024-03-28T17:49:53.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.628+0000 7fa09e600640 1 -- 172.21.15.45:0/1433997346 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0a409f990 con 0x562264ad2850 2024-03-28T17:49:53.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.628+0000 7fa09e600640 1 -- 172.21.15.45:0/1433997346 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a4018b30 con 0x562264ad2850 2024-03-28T17:49:53.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.628+0000 7fa0bb990740 1 -- 172.21.15.45:0/1433997346 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ad2850 msgr2=0x562264acedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:53.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.628+0000 7fa0bb990740 1 --2- 172.21.15.45:0/1433997346 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ad2850 0x562264acedf0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4023040 tx=0x7fa0a4000bc0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.628+0000 7fa0bb990740 1 -- 172.21.15.45:0/1433997346 shutdown_connections 2024-03-28T17:49:53.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.628+0000 7fa0bb990740 1 --2- 172.21.15.45:0/1433997346 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ae4610 0x562264ae1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.628+0000 7fa0bb990740 1 --2- 172.21.15.45:0/1433997346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562264ad3200 0x562264ae0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.628+0000 7fa0bb990740 1 --2- 172.21.15.45:0/1433997346 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ad2850 0x562264acedf0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.628+0000 7fa0bb990740 1 -- 172.21.15.45:0/1433997346 >> 172.21.15.45:0/1433997346 conn(0x562264accec0 msgr2=0x562264adc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:53.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.628+0000 7fa0bb990740 1 -- 172.21.15.45:0/1433997346 shutdown_connections 2024-03-28T17:49:53.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.628+0000 7fa0bb990740 1 -- 172.21.15.45:0/1433997346 wait complete. 2024-03-28T17:49:53.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.629+0000 7fa0bb990740 1 -- start start 2024-03-28T17:49:53.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.629+0000 7fa0bb990740 1 Processor -- start 2024-03-28T17:49:53.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.629+0000 7fa0bb990740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ad2850 0x562264aec160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.629+0000 7fa0bb990740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ad3200 0x562264aec6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.629+0000 7fa0bb990740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562264ae4610 0x562264aea5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.630+0000 7fa0bb990740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b41a95e0 con 0x562264ae4610 2024-03-28T17:49:53.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.630+0000 7fa0bb990740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b41aa900 con 0x562264ad2850 2024-03-28T17:49:53.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.630+0000 7fa0bb990740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b41a9c40 con 0x562264ad3200 2024-03-28T17:49:53.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.630+0000 7fa0bb990740 10 client.0 _ll_register_callbacks cb 0x7fa0a40cc870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:53.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.630+0000 7fa0b1e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ad2850 0x562264aec160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.630+0000 7fa0b1400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ad3200 0x562264aec6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.630+0000 7fa0b1e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ad2850 0x562264aec160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:47032/0 (socket says 172.21.15.45:47032) 2024-03-28T17:49:53.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.630+0000 7fa0b1e00640 1 -- 172.21.15.45:0/900371380 learned_addr learned my addr 172.21.15.45:0/900371380 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:53.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.630+0000 7fa0b2800640 1 --2- 172.21.15.45:0/900371380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562264ae4610 0x562264aea5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.630+0000 7fa0b1e00640 1 -- 172.21.15.45:0/900371380 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ad3200 msgr2=0x562264aec6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:53.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.630+0000 7fa0b1e00640 1 --2- 172.21.15.45:0/900371380 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ad3200 0x562264aec6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.630+0000 7fa0b1e00640 1 -- 172.21.15.45:0/900371380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562264ae4610 msgr2=0x562264aea5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:53.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.630+0000 7fa0b1e00640 1 --2- 172.21.15.45:0/900371380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562264ae4610 0x562264aea5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.630+0000 7fa0b1e00640 1 -- 172.21.15.45:0/900371380 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562264aeadc0 con 0x562264ad2850 2024-03-28T17:49:53.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.630+0000 7fa0b1400640 1 --2- 172.21.15.45:0/900371380 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ad3200 0x562264aec6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:49:53.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.630+0000 7fa0b2800640 1 --2- 172.21.15.45:0/900371380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562264ae4610 0x562264aea5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:49:53.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.630+0000 7fa0b1e00640 1 --2- 172.21.15.45:0/900371380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ad2850 0x562264aec160 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa0a409cfd0 tx=0x7fa0a409b740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74f7e815ab6dd6f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:53.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.631+0000 7fa093400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:49:53.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.631+0000 7fa093400640 1 -- 172.21.15.45:0/900371380 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a40a3fe0 con 0x562264ad2850 2024-03-28T17:49:53.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.631+0000 7fa093400640 1 -- 172.21.15.45:0/900371380 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fa0a4018b30 con 0x562264ad2850 2024-03-28T17:49:53.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.631+0000 7fa0bb990740 1 -- 172.21.15.45:0/900371380 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa0a409f990 con 0x562264ad2850 2024-03-28T17:49:53.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.631+0000 7fa093400640 1 -- 172.21.15.45:0/900371380 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0a409b840 con 0x562264ad2850 2024-03-28T17:49:53.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.631+0000 7fa0bb990740 10 client.25613 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:53.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.631+0000 7fa0bb990740 1 -- 172.21.15.45:0/900371380 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562264b50800 con 0x562264ad2850 2024-03-28T17:49:53.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.631+0000 7fa093400640 1 -- 172.21.15.45:0/900371380 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0a409f990 con 0x562264ad2850 2024-03-28T17:49:53.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.632+0000 7fa093400640 1 -- 172.21.15.45:0/900371380 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fa0a40976b0 con 0x562264ad2850 2024-03-28T17:49:53.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.632+0000 7fa0bb990740 10 client.25613 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:53.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.632+0000 7fa0bb990740 10 client.25613 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-28T17:49:53.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.632+0000 7fa0bb990740 20 client.25613 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:53.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.632+0000 7fa0bb990740 1 --2- 172.21.15.45:0/900371380 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x562264b50cd0 0x562264b530c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.632+0000 7fa0bb990740 4 client.25613 mds_command: new command op to 24529 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T17:49:53.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.632+0000 7fa0bb990740 1 -- 172.21.15.45:0/900371380 --> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562264a3db50 con 0x562264b50cd0 2024-03-28T17:49:53.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.632+0000 7fa0b1400640 1 --2- 172.21.15.45:0/900371380 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x562264b50cd0 0x562264b530c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.633+0000 7fa0b1400640 1 --2- 172.21.15.45:0/900371380 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x562264b50cd0 0x562264b530c0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a56186ce362f1860 server_cookie=d27dc1aab277a32f in_seq=0 out_seq=0 2024-03-28T17:49:53.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.633+0000 7fa093400640 10 client.25613 ms_handle_connect on v2:172.21.15.151:6838/3003564901 2024-03-28T17:49:53.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.633+0000 7fa093400640 1 -- 172.21.15.45:0/900371380 <== mds.2 v2:172.21.15.151:6838/3003564901 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562264a3db50 con 0x562264b50cd0 2024-03-28T17:49:53.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.633+0000 7fa093400640 10 client.25613 handle_command_reply: tid=0 2024-03-28T17:49:53.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.633+0000 7fa0bb990740 1 client.25613 shutdown 2024-03-28T17:49:53.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.633+0000 7fa0bb990740 1 -- 172.21.15.45:0/900371380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ad2850 msgr2=0x562264aec160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:53.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.633+0000 7fa0bb990740 1 --2- 172.21.15.45:0/900371380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ad2850 0x562264aec160 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa0a409cfd0 tx=0x7fa0a409b740 comp rx=0 tx=0).stop 2024-03-28T17:49:53.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.633+0000 7fa0bb990740 1 -- 172.21.15.45:0/900371380 shutdown_connections 2024-03-28T17:49:53.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.633+0000 7fa0bb990740 1 --2- 172.21.15.45:0/900371380 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x562264b50cd0 0x562264b530c0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.633+0000 7fa0bb990740 1 --2- 172.21.15.45:0/900371380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562264ae4610 0x562264aea5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.633+0000 7fa0bb990740 1 --2- 172.21.15.45:0/900371380 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562264ad3200 0x562264aec6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.633+0000 7fa0bb990740 1 --2- 172.21.15.45:0/900371380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562264ad2850 0x562264aec160 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.633+0000 7fa093400640 0 client.25613 ms_handle_reset on v2:172.21.15.151:6838/3003564901 2024-03-28T17:49:53.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.633+0000 7fa0bb990740 1 -- 172.21.15.45:0/900371380 >> 172.21.15.45:0/900371380 conn(0x562264accec0 msgr2=0x562264afb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:53.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.633+0000 7fa0bb990740 1 -- 172.21.15.45:0/900371380 shutdown_connections 2024-03-28T17:49:53.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.633+0000 7fa0bb990740 1 -- 172.21.15.45:0/900371380 wait complete. 2024-03-28T17:49:53.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.633+0000 7fa0bb990740 20 client.25613 trim_cache size 0 max 16384 2024-03-28T17:49:53.638 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-28T17:49:53.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.634+0000 7fa0b3200640 1 -- 172.21.15.45:0/1101173981 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fa0980a2b00 msgr2=0x7fa0980a4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:53.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.634+0000 7fa0b3200640 1 --2- 172.21.15.45:0/1101173981 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fa0980a2b00 0x7fa0980a4fc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac009a60 tx=0x7fa0ac005c50 comp rx=0 tx=0).stop 2024-03-28T17:49:53.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.634+0000 7fa0b3200640 1 -- 172.21.15.45:0/1101173981 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0b4073a70 msgr2=0x7fa0b4199f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:53.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.634+0000 7fa0b3200640 1 --2- 172.21.15.45:0/1101173981 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0b4073a70 0x7fa0b4199f90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa0a400e9f0 tx=0x7fa0a400eeb0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.634+0000 7fa0b3200640 1 -- 172.21.15.45:0/1101173981 shutdown_connections 2024-03-28T17:49:53.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.634+0000 7fa0b3200640 1 --2- 172.21.15.45:0/1101173981 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fa0980a2b00 0x7fa0980a4fc0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.634+0000 7fa0b3200640 1 --2- 172.21.15.45:0/1101173981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0b410ddb0 0x7fa0b419e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.634+0000 7fa0b3200640 1 --2- 172.21.15.45:0/1101173981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0b4074390 0x7fa0b419a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.634+0000 7fa0b3200640 1 --2- 172.21.15.45:0/1101173981 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa0b4073a70 0x7fa0b4199f90 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.634+0000 7fa0b3200640 1 -- 172.21.15.45:0/1101173981 >> 172.21.15.45:0/1101173981 conn(0x7fa0b40fd040 msgr2=0x7fa0b40fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:53.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.634+0000 7fa0b3200640 1 -- 172.21.15.45:0/1101173981 shutdown_connections 2024-03-28T17:49:53.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.634+0000 7fa0b3200640 1 -- 172.21.15.45:0/1101173981 wait complete. 2024-03-28T17:49:53.651 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T17:49:53.651 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T17:49:53.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.845+0000 7fd3f8a00640 1 -- 172.21.15.45:0/1336134279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3f41075f0 msgr2=0x7fd3f41079d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:53.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.845+0000 7fd3f8a00640 1 --2- 172.21.15.45:0/1336134279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3f41075f0 0x7fd3f41079d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4009930 tx=0x7fd3e402bc20 comp rx=0 tx=0).stop 2024-03-28T17:49:53.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.845+0000 7fd3f8a00640 1 -- 172.21.15.45:0/1336134279 shutdown_connections 2024-03-28T17:49:53.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.845+0000 7fd3f8a00640 1 --2- 172.21.15.45:0/1336134279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3f4101ef0 0x7fd3f410e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.845+0000 7fd3f8a00640 1 --2- 172.21.15.45:0/1336134279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3f4101530 0x7fd3f41019b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.845+0000 7fd3f8a00640 1 --2- 172.21.15.45:0/1336134279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3f41075f0 0x7fd3f41079d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.845+0000 7fd3f8a00640 1 -- 172.21.15.45:0/1336134279 >> 172.21.15.45:0/1336134279 conn(0x7fd3f40fd040 msgr2=0x7fd3f40ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:53.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.845+0000 7fd3f8a00640 1 -- 172.21.15.45:0/1336134279 shutdown_connections 2024-03-28T17:49:53.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.845+0000 7fd3f8a00640 1 -- 172.21.15.45:0/1336134279 wait complete. 2024-03-28T17:49:53.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.846+0000 7fd3f8a00640 1 Processor -- start 2024-03-28T17:49:53.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.846+0000 7fd3f8a00640 1 -- start start 2024-03-28T17:49:53.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.847+0000 7fd3f8a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3f4101530 0x7fd3f419a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.847+0000 7fd3f8a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3f4101ef0 0x7fd3f419a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.847+0000 7fd3f8a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3f41075f0 0x7fd3f419e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.847+0000 7fd3f8a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f4111920 con 0x7fd3f4101ef0 2024-03-28T17:49:53.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.847+0000 7fd3f8a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f41117a0 con 0x7fd3f41075f0 2024-03-28T17:49:53.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.847+0000 7fd3f8a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f4111aa0 con 0x7fd3f4101530 2024-03-28T17:49:53.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.847+0000 7fd3f2800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3f4101530 0x7fd3f419a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.847+0000 7fd3f2800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3f4101530 0x7fd3f419a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:33614/0 (socket says 172.21.15.45:33614) 2024-03-28T17:49:53.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.847+0000 7fd3f2800640 1 -- 172.21.15.45:0/2293437105 learned_addr learned my addr 172.21.15.45:0/2293437105 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:53.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.847+0000 7fd3f1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3f4101ef0 0x7fd3f419a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.847+0000 7fd3f3200640 1 --2- 172.21.15.45:0/2293437105 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3f41075f0 0x7fd3f419e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.847+0000 7fd3f2800640 1 -- 172.21.15.45:0/2293437105 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3f41075f0 msgr2=0x7fd3f419e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:53.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.847+0000 7fd3f2800640 1 --2- 172.21.15.45:0/2293437105 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3f41075f0 0x7fd3f419e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.847+0000 7fd3f2800640 1 -- 172.21.15.45:0/2293437105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3f4101ef0 msgr2=0x7fd3f419a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:53.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.847+0000 7fd3f2800640 1 --2- 172.21.15.45:0/2293437105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3f4101ef0 0x7fd3f419a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.847+0000 7fd3f2800640 1 -- 172.21.15.45:0/2293437105 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f419f090 con 0x7fd3f4101530 2024-03-28T17:49:53.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.848+0000 7fd3f1e00640 1 --2- 172.21.15.45:0/2293437105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3f4101ef0 0x7fd3f419a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:49:53.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.848+0000 7fd3f3200640 1 --2- 172.21.15.45:0/2293437105 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3f41075f0 0x7fd3f419e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:49:53.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.848+0000 7fd3f2800640 1 --2- 172.21.15.45:0/2293437105 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3f4101530 0x7fd3f419a0a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4005ab0 tx=0x7fd3e40042e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:53.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.848+0000 7fd3eb400640 1 -- 172.21.15.45:0/2293437105 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e4043020 con 0x7fd3f4101530 2024-03-28T17:49:53.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.848+0000 7fd3f8a00640 1 -- 172.21.15.45:0/2293437105 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f419f320 con 0x7fd3f4101530 2024-03-28T17:49:53.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.848+0000 7fd3eb400640 1 -- 172.21.15.45:0/2293437105 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd3e40044d0 con 0x7fd3f4101530 2024-03-28T17:49:53.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.848+0000 7fd3eb400640 1 -- 172.21.15.45:0/2293437105 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e4003b90 con 0x7fd3f4101530 2024-03-28T17:49:53.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.848+0000 7fd3f8a00640 1 -- 172.21.15.45:0/2293437105 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f41a6c10 con 0x7fd3f4101530 2024-03-28T17:49:53.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.851+0000 7fd3eb400640 1 -- 172.21.15.45:0/2293437105 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 26) v1 ==== 137466+0+0 (secure 0 0 0) 0x7fd3e4046020 con 0x7fd3f4101530 2024-03-28T17:49:53.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.853+0000 7fd3fc187740 1 Processor -- start 2024-03-28T17:49:53.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.853+0000 7fd3fc187740 1 -- start start 2024-03-28T17:49:53.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.853+0000 7fd3eb400640 1 --2- 172.21.15.45:0/2293437105 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd3d80a2a00 0x7fd3d80a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.853+0000 7fd3fc187740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9cc8b0 0x5630ab9ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.853+0000 7fd3fc187740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9cd260 0x5630ab9d5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.853+0000 7fd3fc187740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9d5de0 0x5630ab9d8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.853+0000 7fd3fc187740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630ab9302c0 con 0x5630ab9cc8b0 2024-03-28T17:49:53.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.853+0000 7fd3eb400640 1 -- 172.21.15.45:0/2293437105 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fd3e40f0ff0 con 0x7fd3f4101530 2024-03-28T17:49:53.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.853+0000 7fd3fc187740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630ab8d8c90 con 0x5630ab9d5de0 2024-03-28T17:49:53.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.853+0000 7fd3fc187740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630ab92c360 con 0x5630ab9cd260 2024-03-28T17:49:53.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.854+0000 7fd3f3200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9d5de0 0x5630ab9d8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.854+0000 7fd3f3200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9d5de0 0x5630ab9d8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:47058/0 (socket says 172.21.15.45:47058) 2024-03-28T17:49:53.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.854+0000 7fd3f3200640 1 -- 172.21.15.45:0/920775124 learned_addr learned my addr 172.21.15.45:0/920775124 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:53.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.854+0000 7fd3f2800640 1 --2- 172.21.15.45:0/920775124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9cc8b0 0x5630ab9ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.854+0000 7fd3f1e00640 1 --2- 172.21.15.45:0/920775124 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9cd260 0x5630ab9d5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.854+0000 7fd3f3200640 1 --2- 172.21.15.45:0/2293437105 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd3d80a2a00 0x7fd3d80a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.854+0000 7fd3f3200640 1 -- 172.21.15.45:0/920775124 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9cd260 msgr2=0x5630ab9d5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:53.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.854+0000 7fd3f3200640 1 --2- 172.21.15.45:0/920775124 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9cd260 0x5630ab9d5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.854+0000 7fd3f3200640 1 -- 172.21.15.45:0/920775124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9cc8b0 msgr2=0x5630ab9ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:53.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.854+0000 7fd3f3200640 1 --2- 172.21.15.45:0/920775124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9cc8b0 0x5630ab9ccc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.854+0000 7fd3f3200640 1 -- 172.21.15.45:0/920775124 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630ab8fd080 con 0x5630ab9d5de0 2024-03-28T17:49:53.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.854+0000 7fd3f2800640 1 --2- 172.21.15.45:0/920775124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9cc8b0 0x5630ab9ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:49:53.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.855+0000 7fd3f3200640 1 --2- 172.21.15.45:0/920775124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9d5de0 0x5630ab9d8230 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec0129d0 tx=0x7fd3ec012e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e89cf2d38db412a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:53.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.855+0000 7fd3df400640 1 -- 172.21.15.45:0/920775124 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec012410 con 0x5630ab9d5de0 2024-03-28T17:49:53.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.855+0000 7fd3f3200640 1 --2- 172.21.15.45:0/2293437105 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd3d80a2a00 0x7fd3d80a4ec0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec01b000 tx=0x7fd3ec01d000 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:53.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.855+0000 7fd3df400640 1 -- 172.21.15.45:0/920775124 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd3ec00cda0 con 0x5630ab9d5de0 2024-03-28T17:49:53.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.855+0000 7fd3df400640 1 -- 172.21.15.45:0/920775124 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec01dd60 con 0x5630ab9d5de0 2024-03-28T17:49:53.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.856+0000 7fd3fc187740 1 -- 172.21.15.45:0/920775124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9d5de0 msgr2=0x5630ab9d8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:53.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.856+0000 7fd3fc187740 1 --2- 172.21.15.45:0/920775124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9d5de0 0x5630ab9d8230 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec0129d0 tx=0x7fd3ec012e90 comp rx=0 tx=0).stop 2024-03-28T17:49:53.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.856+0000 7fd3fc187740 1 -- 172.21.15.45:0/920775124 shutdown_connections 2024-03-28T17:49:53.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.856+0000 7fd3fc187740 1 --2- 172.21.15.45:0/920775124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9d5de0 0x5630ab9d8230 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.856+0000 7fd3fc187740 1 --2- 172.21.15.45:0/920775124 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9cd260 0x5630ab9d5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.856+0000 7fd3fc187740 1 --2- 172.21.15.45:0/920775124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9cc8b0 0x5630ab9ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.856+0000 7fd3fc187740 1 -- 172.21.15.45:0/920775124 >> 172.21.15.45:0/920775124 conn(0x5630ab9c6f80 msgr2=0x5630ab9c7360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:53.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.856+0000 7fd3fc187740 1 -- 172.21.15.45:0/920775124 shutdown_connections 2024-03-28T17:49:53.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.856+0000 7fd3fc187740 1 -- 172.21.15.45:0/920775124 wait complete. 2024-03-28T17:49:53.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.857+0000 7fd3fc187740 1 -- start start 2024-03-28T17:49:53.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.857+0000 7fd3fc187740 1 Processor -- start 2024-03-28T17:49:53.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.858+0000 7fd3fc187740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9cc8b0 0x5630ab9d57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.858+0000 7fd3fc187740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9cd260 0x5630ab9d2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.858+0000 7fd3fc187740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9d5de0 0x5630ab9d3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.858+0000 7fd3fc187740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630ab9d8cf0 con 0x5630ab9d5de0 2024-03-28T17:49:53.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.858+0000 7fd3fc187740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630ab8ec0d0 con 0x5630ab9cc8b0 2024-03-28T17:49:53.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.858+0000 7fd3fc187740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630ab9d8e70 con 0x5630ab9cd260 2024-03-28T17:49:53.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.858+0000 7fd3fc187740 10 client.0 _ll_register_callbacks cb 0x5630ab8f0e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:53.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.858+0000 7fd3f3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9d5de0 0x5630ab9d3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.858+0000 7fd3f1e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9cd260 0x5630ab9d2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.858+0000 7fd3f2800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9cc8b0 0x5630ab9d57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.858+0000 7fd3f3200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9d5de0 0x5630ab9d3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47102/0 (socket says 172.21.15.45:47102) 2024-03-28T17:49:53.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.858+0000 7fd3f1e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9cd260 0x5630ab9d2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:33634/0 (socket says 172.21.15.45:33634) 2024-03-28T17:49:53.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.858+0000 7fd3f2800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9cc8b0 0x5630ab9d57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:47074/0 (socket says 172.21.15.45:47074) 2024-03-28T17:49:53.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.858+0000 7fd3f3200640 1 -- 172.21.15.45:0/4235644588 learned_addr learned my addr 172.21.15.45:0/4235644588 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:53.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.859+0000 7fd3f2800640 1 -- 172.21.15.45:0/4235644588 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9cd260 msgr2=0x5630ab9d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:53.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.859+0000 7fd3f2800640 1 --2- 172.21.15.45:0/4235644588 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9cd260 0x5630ab9d2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.859+0000 7fd3f2800640 1 -- 172.21.15.45:0/4235644588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9d5de0 msgr2=0x5630ab9d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:53.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.859+0000 7fd3f2800640 1 --2- 172.21.15.45:0/4235644588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9d5de0 0x5630ab9d3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.859+0000 7fd3f2800640 1 -- 172.21.15.45:0/4235644588 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5630ab6443f0 con 0x5630ab9cc8b0 2024-03-28T17:49:53.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.859+0000 7fd3f2800640 1 --2- 172.21.15.45:0/4235644588 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9cc8b0 0x5630ab9d57f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd3e40ee780 tx=0x7fd3e40bd3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d2d25fd497e75d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:53.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.859+0000 7fd3d3e00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:49:53.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.859+0000 7fd3d3e00640 1 -- 172.21.15.45:0/4235644588 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e40bcf40 con 0x5630ab9cc8b0 2024-03-28T17:49:53.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.859+0000 7fd3d3e00640 1 -- 172.21.15.45:0/4235644588 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fd3e40b9c50 con 0x5630ab9cc8b0 2024-03-28T17:49:53.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.860+0000 7fd3fc187740 1 -- 172.21.15.45:0/4235644588 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd3ec00cda0 con 0x5630ab9cc8b0 2024-03-28T17:49:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.860+0000 7fd3d3e00640 1 -- 172.21.15.45:0/4235644588 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd3e40db050 con 0x5630ab9cc8b0 2024-03-28T17:49:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.860+0000 7fd3fc187740 10 client.25625 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.860+0000 7fd3fc187740 1 -- 172.21.15.45:0/4235644588 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5630ab70e7d0 con 0x5630ab9cc8b0 2024-03-28T17:49:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.860+0000 7fd3d3e00640 1 -- 172.21.15.45:0/4235644588 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd3ec00cda0 con 0x5630ab9cc8b0 2024-03-28T17:49:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.860+0000 7fd3d3e00640 1 -- 172.21.15.45:0/4235644588 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fd3e40ee920 con 0x5630ab9cc8b0 2024-03-28T17:49:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.860+0000 7fd3fc187740 10 client.25625 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.860+0000 7fd3fc187740 10 client.25625 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T17:49:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.860+0000 7fd3fc187740 20 client.25625 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.860+0000 7fd3fc187740 1 --2- 172.21.15.45:0/4235644588 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x5630ab9e5c60 0x5630ab9e8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.860+0000 7fd3fc187740 4 client.25625 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:49:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.860+0000 7fd3fc187740 1 -- 172.21.15.45:0/4235644588 --> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5630ab93d800 con 0x5630ab9e5c60 2024-03-28T17:49:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.861+0000 7fd3f1e00640 1 --2- 172.21.15.45:0/4235644588 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x5630ab9e5c60 0x5630ab9e8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.861+0000 7fd3f1e00640 1 --2- 172.21.15.45:0/4235644588 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x5630ab9e5c60 0x5630ab9e8080 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6411d8be0f138fd0 server_cookie=5f0da870f725ebc3 in_seq=0 out_seq=0 2024-03-28T17:49:53.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.861+0000 7fd3d3e00640 10 client.25625 ms_handle_connect on v2:172.21.15.71:6836/2008752655 2024-03-28T17:49:53.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.862+0000 7fd3d3e00640 1 -- 172.21.15.45:0/4235644588 <== mds.1 v2:172.21.15.71:6836/2008752655 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5630ab93d800 con 0x5630ab9e5c60 2024-03-28T17:49:53.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.862+0000 7fd3d3e00640 10 client.25625 handle_command_reply: tid=0 2024-03-28T17:49:53.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.862+0000 7fd3fc187740 1 client.25625 shutdown 2024-03-28T17:49:53.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.862+0000 7fd3fc187740 1 -- 172.21.15.45:0/4235644588 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9cc8b0 msgr2=0x5630ab9d57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:53.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.862+0000 7fd3fc187740 1 --2- 172.21.15.45:0/4235644588 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9cc8b0 0x5630ab9d57f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd3e40ee780 tx=0x7fd3e40bd3b0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.863+0000 7fd3fc187740 1 -- 172.21.15.45:0/4235644588 shutdown_connections 2024-03-28T17:49:53.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.863+0000 7fd3fc187740 1 --2- 172.21.15.45:0/4235644588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9d5de0 0x5630ab9d3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.863+0000 7fd3fc187740 1 --2- 172.21.15.45:0/4235644588 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x5630ab9e5c60 0x5630ab9e8080 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.863+0000 7fd3fc187740 1 --2- 172.21.15.45:0/4235644588 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9cd260 0x5630ab9d2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.863+0000 7fd3fc187740 1 --2- 172.21.15.45:0/4235644588 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9cc8b0 0x5630ab9d57f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.863+0000 7fd3d3e00640 0 client.25625 ms_handle_reset on v2:172.21.15.71:6836/2008752655 2024-03-28T17:49:53.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.863+0000 7fd3fc187740 1 -- 172.21.15.45:0/4235644588 >> 172.21.15.45:0/4235644588 conn(0x5630ab9c6f80 msgr2=0x5630ab9c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:53.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.863+0000 7fd3fc187740 1 -- 172.21.15.45:0/4235644588 shutdown_connections 2024-03-28T17:49:53.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.863+0000 7fd3fc187740 1 -- 172.21.15.45:0/4235644588 wait complete. 2024-03-28T17:49:53.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.863+0000 7fd3fc187740 20 client.25625 trim_cache size 0 max 16384 2024-03-28T17:49:53.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.888+0000 7fd3fc187740 1 Processor -- start 2024-03-28T17:49:53.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.888+0000 7fd3fc187740 1 -- start start 2024-03-28T17:49:53.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.888+0000 7fd3fc187740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9cc8b0 0x5630ab9c5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.888+0000 7fd3fc187740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9cd260 0x5630ab9d52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.888+0000 7fd3fc187740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9d5de0 0x5630ab9d57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.888+0000 7fd3fc187740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f41ad400 con 0x5630ab9cc8b0 2024-03-28T17:49:53.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.888+0000 7fd3fc187740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f41aaf00 con 0x5630ab9d5de0 2024-03-28T17:49:53.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.888+0000 7fd3fc187740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f41a8fe0 con 0x5630ab9cd260 2024-03-28T17:49:53.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.889+0000 7fd3f3200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9d5de0 0x5630ab9d57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.889+0000 7fd3f3200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9d5de0 0x5630ab9d57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:47086/0 (socket says 172.21.15.45:47086) 2024-03-28T17:49:53.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.889+0000 7fd3f3200640 1 -- 172.21.15.45:0/612696285 learned_addr learned my addr 172.21.15.45:0/612696285 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:53.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.889+0000 7fd3f1e00640 1 --2- 172.21.15.45:0/612696285 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9cd260 0x5630ab9d52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.889+0000 7fd3f2800640 1 --2- 172.21.15.45:0/612696285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9cc8b0 0x5630ab9c5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.889+0000 7fd3f3200640 1 -- 172.21.15.45:0/612696285 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9cd260 msgr2=0x5630ab9d52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:53.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.889+0000 7fd3f3200640 1 --2- 172.21.15.45:0/612696285 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9cd260 0x5630ab9d52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.889+0000 7fd3f3200640 1 -- 172.21.15.45:0/612696285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9cc8b0 msgr2=0x5630ab9c5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.889+0000 7fd3f3200640 1 --2- 172.21.15.45:0/612696285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9cc8b0 0x5630ab9c5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.889+0000 7fd3f3200640 1 -- 172.21.15.45:0/612696285 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630ab9d10d0 con 0x5630ab9d5de0 2024-03-28T17:49:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.889+0000 7fd3f1e00640 1 --2- 172.21.15.45:0/612696285 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9cd260 0x5630ab9d52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:49:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.889+0000 7fd3f2800640 1 --2- 172.21.15.45:0/612696285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9cc8b0 0x5630ab9c5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:49:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.890+0000 7fd3f3200640 1 --2- 172.21.15.45:0/612696285 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9d5de0 0x5630ab9d57f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec013fd0 tx=0x7fd3ec015050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4411a81db622fd72 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.890+0000 7fd3dea00640 1 -- 172.21.15.45:0/612696285 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec0123f0 con 0x5630ab9d5de0 2024-03-28T17:49:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.890+0000 7fd3dea00640 1 -- 172.21.15.45:0/612696285 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd3ec012590 con 0x5630ab9d5de0 2024-03-28T17:49:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.890+0000 7fd3dea00640 1 -- 172.21.15.45:0/612696285 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec004c90 con 0x5630ab9d5de0 2024-03-28T17:49:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.891+0000 7fd3fc187740 1 -- 172.21.15.45:0/612696285 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9d5de0 msgr2=0x5630ab9d57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.891+0000 7fd3fc187740 1 --2- 172.21.15.45:0/612696285 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9d5de0 0x5630ab9d57f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec013fd0 tx=0x7fd3ec015050 comp rx=0 tx=0).stop 2024-03-28T17:49:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.891+0000 7fd3fc187740 1 -- 172.21.15.45:0/612696285 shutdown_connections 2024-03-28T17:49:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.891+0000 7fd3fc187740 1 --2- 172.21.15.45:0/612696285 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9d5de0 0x5630ab9d57f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.891+0000 7fd3fc187740 1 --2- 172.21.15.45:0/612696285 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9cd260 0x5630ab9d52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.891+0000 7fd3fc187740 1 --2- 172.21.15.45:0/612696285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9cc8b0 0x5630ab9c5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.891+0000 7fd3fc187740 1 -- 172.21.15.45:0/612696285 >> 172.21.15.45:0/612696285 conn(0x5630ab9c6f80 msgr2=0x5630ab9c58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:53.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.891+0000 7fd3fc187740 1 -- 172.21.15.45:0/612696285 shutdown_connections 2024-03-28T17:49:53.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.891+0000 7fd3fc187740 1 -- 172.21.15.45:0/612696285 wait complete. 2024-03-28T17:49:53.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.891+0000 7fd3fc187740 1 -- start start 2024-03-28T17:49:53.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.892+0000 7fd3fc187740 1 Processor -- start 2024-03-28T17:49:53.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.892+0000 7fd3fc187740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9cc8b0 0x5630ab9dd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.892+0000 7fd3fc187740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9cd260 0x5630ab9dddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.892+0000 7fd3fc187740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9d5de0 0x5630ab9dbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.892+0000 7fd3fc187740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f41a95e0 con 0x5630ab9cd260 2024-03-28T17:49:53.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.892+0000 7fd3fc187740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f41aa8a0 con 0x5630ab9cc8b0 2024-03-28T17:49:53.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.892+0000 7fd3fc187740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f41a9be0 con 0x5630ab9d5de0 2024-03-28T17:49:53.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.892+0000 7fd3fc187740 10 client.0 _ll_register_callbacks cb 0x7fd3e40b9880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:53.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.892+0000 7fd3f3200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9d5de0 0x5630ab9dbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.892+0000 7fd3f1e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9cd260 0x5630ab9dddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.893+0000 7fd3f3200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9d5de0 0x5630ab9dbcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:33656/0 (socket says 172.21.15.45:33656) 2024-03-28T17:49:53.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.893+0000 7fd3f3200640 1 -- 172.21.15.45:0/599301886 learned_addr learned my addr 172.21.15.45:0/599301886 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:53.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.893+0000 7fd3f2800640 1 --2- 172.21.15.45:0/599301886 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9cc8b0 0x5630ab9dd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.893+0000 7fd3f3200640 1 -- 172.21.15.45:0/599301886 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9cc8b0 msgr2=0x5630ab9dd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:53.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.893+0000 7fd3f3200640 1 --2- 172.21.15.45:0/599301886 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9cc8b0 0x5630ab9dd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.893+0000 7fd3f3200640 1 -- 172.21.15.45:0/599301886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9cd260 msgr2=0x5630ab9dddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:53.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.893+0000 7fd3f3200640 1 --2- 172.21.15.45:0/599301886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9cd260 0x5630ab9dddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.893+0000 7fd3f3200640 1 -- 172.21.15.45:0/599301886 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5630ab9dc4b0 con 0x5630ab9d5de0 2024-03-28T17:49:53.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.893+0000 7fd3f1e00640 1 --2- 172.21.15.45:0/599301886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9cd260 0x5630ab9dddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:49:53.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.893+0000 7fd3f3200640 1 --2- 172.21.15.45:0/599301886 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9d5de0 0x5630ab9dbcd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec01f040 tx=0x7fd3ec009c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a89aad89e3868207 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:53.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.893+0000 7fd3d3e00640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:49:53.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.894+0000 7fd3d3e00640 1 -- 172.21.15.45:0/599301886 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec01d610 con 0x5630ab9d5de0 2024-03-28T17:49:53.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.894+0000 7fd3d3e00640 1 -- 172.21.15.45:0/599301886 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fd3ec02c040 con 0x5630ab9d5de0 2024-03-28T17:49:53.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.894+0000 7fd3fc187740 1 -- 172.21.15.45:0/599301886 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd3ec012590 con 0x5630ab9d5de0 2024-03-28T17:49:53.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.894+0000 7fd3d3e00640 1 -- 172.21.15.45:0/599301886 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd3ec004b10 con 0x5630ab9d5de0 2024-03-28T17:49:53.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.894+0000 7fd3fc187740 10 client.25606 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:53.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.894+0000 7fd3fc187740 1 -- 172.21.15.45:0/599301886 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5630aba42040 con 0x5630ab9d5de0 2024-03-28T17:49:53.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.894+0000 7fd3d3e00640 1 -- 172.21.15.45:0/599301886 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd3ec012590 con 0x5630ab9d5de0 2024-03-28T17:49:53.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.894+0000 7fd3d3e00640 1 -- 172.21.15.45:0/599301886 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fd3ec044470 con 0x5630ab9d5de0 2024-03-28T17:49:53.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.894+0000 7fd3fc187740 10 client.25606 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:53.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.894+0000 7fd3fc187740 10 client.25606 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T17:49:53.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.894+0000 7fd3fc187740 20 client.25606 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:53.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.895+0000 7fd3fc187740 1 --2- 172.21.15.45:0/599301886 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x5630aba42470 0x5630aba44860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:53.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.895+0000 7fd3fc187740 4 client.25606 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T17:49:53.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.895+0000 7fd3fc187740 1 -- 172.21.15.45:0/599301886 --> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5630ab937b50 con 0x5630aba42470 2024-03-28T17:49:53.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.895+0000 7fd3f2800640 1 --2- 172.21.15.45:0/599301886 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x5630aba42470 0x5630aba44860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:53.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.895+0000 7fd3d3e00640 10 client.25606 ms_handle_connect on v2:172.21.15.71:6836/2008752655 2024-03-28T17:49:53.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.895+0000 7fd3f2800640 1 --2- 172.21.15.45:0/599301886 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x5630aba42470 0x5630aba44860 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=44fc8ecfae16c3d7 server_cookie=91c8b1af56f28c50 in_seq=0 out_seq=0 2024-03-28T17:49:53.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.896+0000 7fd3d3e00640 1 -- 172.21.15.45:0/599301886 <== mds.1 v2:172.21.15.71:6836/2008752655 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5630ab937b50 con 0x5630aba42470 2024-03-28T17:49:53.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.896+0000 7fd3d3e00640 10 client.25606 handle_command_reply: tid=0 2024-03-28T17:49:53.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.896+0000 7fd3fc187740 1 client.25606 shutdown 2024-03-28T17:49:53.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.896+0000 7fd3fc187740 1 -- 172.21.15.45:0/599301886 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9d5de0 msgr2=0x5630ab9dbcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:53.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.896+0000 7fd3fc187740 1 --2- 172.21.15.45:0/599301886 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9d5de0 0x5630ab9dbcd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec01f040 tx=0x7fd3ec009c60 comp rx=0 tx=0).stop 2024-03-28T17:49:53.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.896+0000 7fd3fc187740 1 -- 172.21.15.45:0/599301886 shutdown_connections 2024-03-28T17:49:53.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.896+0000 7fd3fc187740 1 --2- 172.21.15.45:0/599301886 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x5630aba42470 0x5630aba44860 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.896+0000 7fd3fc187740 1 --2- 172.21.15.45:0/599301886 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5630ab9d5de0 0x5630ab9dbcd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.896+0000 7fd3fc187740 1 --2- 172.21.15.45:0/599301886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630ab9cd260 0x5630ab9dddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.896+0000 7fd3fc187740 1 --2- 172.21.15.45:0/599301886 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630ab9cc8b0 0x5630ab9dd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.896+0000 7fd3fc187740 1 -- 172.21.15.45:0/599301886 >> 172.21.15.45:0/599301886 conn(0x5630ab9c6f80 msgr2=0x5630ab9c6160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:53.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.896+0000 7fd3d3e00640 0 client.25606 ms_handle_reset on v2:172.21.15.71:6836/2008752655 2024-03-28T17:49:53.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.896+0000 7fd3fc187740 1 -- 172.21.15.45:0/599301886 shutdown_connections 2024-03-28T17:49:53.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.896+0000 7fd3fc187740 1 -- 172.21.15.45:0/599301886 wait complete. 2024-03-28T17:49:53.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.896+0000 7fd3fc187740 20 client.25606 trim_cache size 0 max 16384 2024-03-28T17:49:53.900 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-28T17:49:53.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.897+0000 7fd3f8a00640 1 -- 172.21.15.45:0/2293437105 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd3d80a2a00 msgr2=0x7fd3d80a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:53.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.897+0000 7fd3f8a00640 1 --2- 172.21.15.45:0/2293437105 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd3d80a2a00 0x7fd3d80a4ec0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec01b000 tx=0x7fd3ec01d000 comp rx=0 tx=0).stop 2024-03-28T17:49:53.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.897+0000 7fd3f8a00640 1 -- 172.21.15.45:0/2293437105 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3f4101530 msgr2=0x7fd3f419a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:53.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.897+0000 7fd3f8a00640 1 --2- 172.21.15.45:0/2293437105 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3f4101530 0x7fd3f419a0a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4005ab0 tx=0x7fd3e40042e0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.897+0000 7fd3f8a00640 1 -- 172.21.15.45:0/2293437105 shutdown_connections 2024-03-28T17:49:53.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.897+0000 7fd3f8a00640 1 --2- 172.21.15.45:0/2293437105 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd3d80a2a00 0x7fd3d80a4ec0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.897+0000 7fd3f8a00640 1 --2- 172.21.15.45:0/2293437105 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3f41075f0 0x7fd3f419e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.897+0000 7fd3f8a00640 1 --2- 172.21.15.45:0/2293437105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3f4101ef0 0x7fd3f419a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.897+0000 7fd3f8a00640 1 --2- 172.21.15.45:0/2293437105 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3f4101530 0x7fd3f419a0a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:53.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.897+0000 7fd3f8a00640 1 -- 172.21.15.45:0/2293437105 >> 172.21.15.45:0/2293437105 conn(0x7fd3f40fd040 msgr2=0x7fd3f40fe500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:53.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.898+0000 7fd3f8a00640 1 -- 172.21.15.45:0/2293437105 shutdown_connections 2024-03-28T17:49:53.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:53.898+0000 7fd3f8a00640 1 -- 172.21.15.45:0/2293437105 wait complete. 2024-03-28T17:49:53.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:53 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/1741705831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:49:53.921 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T17:49:53.921 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T17:49:53.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:53 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/1741705831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:49:54.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:53 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/1741705831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:49:54.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.124+0000 7fbab5200640 1 -- 172.21.15.45:0/2262912085 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbab0102730 msgr2=0x7fbab0102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:54.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.124+0000 7fbab5200640 1 --2- 172.21.15.45:0/2262912085 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbab0102730 0x7fbab0102b10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbaa0009900 tx=0x7fbaa002bcd0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.124+0000 7fbab5200640 1 -- 172.21.15.45:0/2262912085 shutdown_connections 2024-03-28T17:49:54.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.124+0000 7fbab5200640 1 --2- 172.21.15.45:0/2262912085 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbab010bb00 0x7fbab010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.124+0000 7fbab5200640 1 --2- 172.21.15.45:0/2262912085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbab0103050 0x7fbab010b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.124+0000 7fbab5200640 1 --2- 172.21.15.45:0/2262912085 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbab0102730 0x7fbab0102b10 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.124+0000 7fbab5200640 1 -- 172.21.15.45:0/2262912085 >> 172.21.15.45:0/2262912085 conn(0x7fbab00fbef0 msgr2=0x7fbab00fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:54.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.124+0000 7fbab5200640 1 -- 172.21.15.45:0/2262912085 shutdown_connections 2024-03-28T17:49:54.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.124+0000 7fbab5200640 1 -- 172.21.15.45:0/2262912085 wait complete. 2024-03-28T17:49:54.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.125+0000 7fbab5200640 1 Processor -- start 2024-03-28T17:49:54.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.125+0000 7fbab5200640 1 -- start start 2024-03-28T17:49:54.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.126+0000 7fbab5200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbab0102730 0x7fbab0197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:54.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.126+0000 7fbab5200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbab0103050 0x7fbab0198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:54.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.126+0000 7fbab5200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbab010bb00 0x7fbab019c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:54.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.126+0000 7fbab5200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab0111900 con 0x7fbab0102730 2024-03-28T17:49:54.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.126+0000 7fbab5200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab0111780 con 0x7fbab010bb00 2024-03-28T17:49:54.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.126+0000 7fbab5200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab0111a80 con 0x7fbab0103050 2024-03-28T17:49:54.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.126+0000 7fbaaf400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbab0102730 0x7fbab0197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:54.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.126+0000 7fbaaea00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbab0103050 0x7fbab0198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:54.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.126+0000 7fbaaf400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbab0102730 0x7fbab0197dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47132/0 (socket says 172.21.15.45:47132) 2024-03-28T17:49:54.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.126+0000 7fbaaea00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbab0103050 0x7fbab0198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:33666/0 (socket says 172.21.15.45:33666) 2024-03-28T17:49:54.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.126+0000 7fbaaf400640 1 -- 172.21.15.45:0/3761059027 learned_addr learned my addr 172.21.15.45:0/3761059027 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:54.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.126+0000 7fbaafe00640 1 --2- 172.21.15.45:0/3761059027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbab010bb00 0x7fbab019c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:54.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.127+0000 7fbaaea00640 1 -- 172.21.15.45:0/3761059027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbab010bb00 msgr2=0x7fbab019c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:54.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.127+0000 7fbaaea00640 1 --2- 172.21.15.45:0/3761059027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbab010bb00 0x7fbab019c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.127+0000 7fbaaea00640 1 -- 172.21.15.45:0/3761059027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbab0102730 msgr2=0x7fbab0197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:54.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.127+0000 7fbaaea00640 1 --2- 172.21.15.45:0/3761059027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbab0102730 0x7fbab0197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.127+0000 7fbaaea00640 1 -- 172.21.15.45:0/3761059027 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab019cdb0 con 0x7fbab0103050 2024-03-28T17:49:54.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.127+0000 7fbaaea00640 1 --2- 172.21.15.45:0/3761059027 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbab0103050 0x7fbab0198300 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fba9c00d6d0 tx=0x7fba9c00db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:54.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.127+0000 7fbaa7e00640 1 -- 172.21.15.45:0/3761059027 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9c017070 con 0x7fbab0103050 2024-03-28T17:49:54.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.127+0000 7fbaa7e00640 1 -- 172.21.15.45:0/3761059027 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba9c010460 con 0x7fbab0103050 2024-03-28T17:49:54.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.127+0000 7fbab5200640 1 -- 172.21.15.45:0/3761059027 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab019d0a0 con 0x7fbab0103050 2024-03-28T17:49:54.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.127+0000 7fbaa7e00640 1 -- 172.21.15.45:0/3761059027 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9c00f600 con 0x7fbab0103050 2024-03-28T17:49:54.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.128+0000 7fbab5200640 1 -- 172.21.15.45:0/3761059027 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab01a4930 con 0x7fbab0103050 2024-03-28T17:49:54.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.131+0000 7fbab945c740 1 Processor -- start 2024-03-28T17:49:54.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.131+0000 7fbab945c740 1 -- start start 2024-03-28T17:49:54.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.131+0000 7fbab945c740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7a7790 0x558a8c7a7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:54.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.131+0000 7fbab945c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a8c7a8140 0x558a8c7b0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:54.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.131+0000 7fbab945c740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558a8c7b1240 0x558a8c7b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:54.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.131+0000 7fbab945c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a8c70b2c0 con 0x558a8c7a8140 2024-03-28T17:49:54.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.131+0000 7fbab945c740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a8c6b3c90 con 0x558a8c7a7790 2024-03-28T17:49:54.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.131+0000 7fbab945c740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a8c707360 con 0x558a8c7b1240 2024-03-28T17:49:54.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.132+0000 7fbaafe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7a7790 0x558a8c7a7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:54.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.132+0000 7fbaafe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7a7790 0x558a8c7a7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:47116/0 (socket says 172.21.15.45:47116) 2024-03-28T17:49:54.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.132+0000 7fbaafe00640 1 -- 172.21.15.45:0/1079108131 learned_addr learned my addr 172.21.15.45:0/1079108131 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:54.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.132+0000 7fbaaf400640 1 --2- 172.21.15.45:0/1079108131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a8c7a8140 0x558a8c7b0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:54.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.132+0000 7fbaafe00640 1 -- 172.21.15.45:0/1079108131 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558a8c7b1240 msgr2=0x558a8c7b16a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T17:49:54.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.132+0000 7fbaafe00640 1 --2- 172.21.15.45:0/1079108131 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558a8c7b1240 0x558a8c7b16a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.132+0000 7fbaafe00640 1 -- 172.21.15.45:0/1079108131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a8c7a8140 msgr2=0x558a8c7b0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:54.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.132+0000 7fbaafe00640 1 --2- 172.21.15.45:0/1079108131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a8c7a8140 0x558a8c7b0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.132+0000 7fbaafe00640 1 -- 172.21.15.45:0/1079108131 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558a8c6d8080 con 0x558a8c7a7790 2024-03-28T17:49:54.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.132+0000 7fbaafe00640 1 --2- 172.21.15.45:0/1079108131 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7a7790 0x558a8c7a7b70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8009f50 tx=0x7fbaa80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e51ad4aef49ac4ba server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:54.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.133+0000 7fbaa7e00640 1 -- 172.21.15.45:0/3761059027 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 26) v1 ==== 137466+0+0 (secure 0 0 0) 0x7fba9c01a070 con 0x7fbab0103050 2024-03-28T17:49:54.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.133+0000 7fbaa4c00640 1 -- 172.21.15.45:0/1079108131 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8004b20 con 0x558a8c7a7790 2024-03-28T17:49:54.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.133+0000 7fbaa4c00640 1 -- 172.21.15.45:0/1079108131 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbaa8004cc0 con 0x558a8c7a7790 2024-03-28T17:49:54.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.133+0000 7fbaa4c00640 1 -- 172.21.15.45:0/1079108131 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa80054c0 con 0x558a8c7a7790 2024-03-28T17:49:54.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.133+0000 7fbab945c740 1 -- 172.21.15.45:0/1079108131 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7a7790 msgr2=0x558a8c7a7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:54.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.133+0000 7fbab945c740 1 --2- 172.21.15.45:0/1079108131 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7a7790 0x558a8c7a7b70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8009f50 tx=0x7fbaa80118d0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.133+0000 7fbab945c740 1 -- 172.21.15.45:0/1079108131 shutdown_connections 2024-03-28T17:49:54.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.133+0000 7fbab945c740 1 --2- 172.21.15.45:0/1079108131 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558a8c7b1240 0x558a8c7b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.133+0000 7fbab945c740 1 --2- 172.21.15.45:0/1079108131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a8c7a8140 0x558a8c7b0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.133+0000 7fbab945c740 1 --2- 172.21.15.45:0/1079108131 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7a7790 0x558a8c7a7b70 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.133+0000 7fbab945c740 1 -- 172.21.15.45:0/1079108131 >> 172.21.15.45:0/1079108131 conn(0x558a8c7a1f80 msgr2=0x558a8c7a2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:54.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.135+0000 7fbab945c740 1 -- 172.21.15.45:0/1079108131 shutdown_connections 2024-03-28T17:49:54.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.135+0000 7fbab945c740 1 -- 172.21.15.45:0/1079108131 wait complete. 2024-03-28T17:49:54.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.135+0000 7fbaa7e00640 1 --2- 172.21.15.45:0/3761059027 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fba940a2990 0x7fba940a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:54.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.135+0000 7fbab945c740 1 -- start start 2024-03-28T17:49:54.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.135+0000 7fbaa7e00640 1 -- 172.21.15.45:0/3761059027 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fba9c0cea60 con 0x7fbab0103050 2024-03-28T17:49:54.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.135+0000 7fbab945c740 1 Processor -- start 2024-03-28T17:49:54.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.136+0000 7fbaaf400640 1 --2- 172.21.15.45:0/3761059027 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fba940a2990 0x7fba940a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:54.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.136+0000 7fbab945c740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558a8c7a7790 0x558a8c7b76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:54.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.136+0000 7fbab945c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a8c7a8140 0x558a8c7b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:54.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.136+0000 7fbab945c740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7b1240 0x558a8c7b5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:54.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.136+0000 7fbab945c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a8c7b8180 con 0x558a8c7a8140 2024-03-28T17:49:54.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.136+0000 7fbab945c740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a8c6c70d0 con 0x558a8c7b1240 2024-03-28T17:49:54.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.136+0000 7fbab945c740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a8c7b8300 con 0x558a8c7a7790 2024-03-28T17:49:54.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.136+0000 7fbab945c740 10 client.0 _ll_register_callbacks cb 0x558a8c6cbe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:54.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.136+0000 7fbaaf400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558a8c7a7790 0x558a8c7b76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:54.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.136+0000 7fbaaf400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558a8c7a7790 0x558a8c7b76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:33670/0 (socket says 172.21.15.45:33670) 2024-03-28T17:49:54.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.136+0000 7fbaaf400640 1 -- 172.21.15.45:0/921400655 learned_addr learned my addr 172.21.15.45:0/921400655 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:54.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.137+0000 7fbaafe00640 1 --2- 172.21.15.45:0/921400655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7b1240 0x558a8c7b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:54.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.137+0000 7fbaaea00640 1 --2- 172.21.15.45:0/921400655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a8c7a8140 0x558a8c7b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:54.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.137+0000 7fbaafe00640 1 -- 172.21.15.45:0/921400655 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558a8c7a7790 msgr2=0x558a8c7b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:54.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.137+0000 7fbaafe00640 1 --2- 172.21.15.45:0/921400655 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558a8c7a7790 0x558a8c7b76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.137+0000 7fbaafe00640 1 -- 172.21.15.45:0/921400655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a8c7a8140 msgr2=0x558a8c7b4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:54.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.137+0000 7fbaafe00640 1 --2- 172.21.15.45:0/921400655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a8c7a8140 0x558a8c7b4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.137+0000 7fbaafe00640 1 -- 172.21.15.45:0/921400655 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558a8c41f3f0 con 0x558a8c7b1240 2024-03-28T17:49:54.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.137+0000 7fbaaf400640 1 --2- 172.21.15.45:0/3761059027 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fba940a2990 0x7fba940a4e50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x558a8c7a8580 tx=0x7fbaa002e040 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:54.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.137+0000 7fbaaea00640 1 --2- 172.21.15.45:0/921400655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a8c7a8140 0x558a8c7b4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:49:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.137+0000 7fbaafe00640 1 --2- 172.21.15.45:0/921400655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7b1240 0x558a8c7b5090 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8012fd0 tx=0x7fbaa800fca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8617de5ff9b3bcd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.138+0000 7fba99600640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:49:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.138+0000 7fba99600640 1 -- 172.21.15.45:0/921400655 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8028020 con 0x558a8c7b1240 2024-03-28T17:49:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.138+0000 7fbab945c740 1 -- 172.21.15.45:0/921400655 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbaa8004cc0 con 0x558a8c7b1240 2024-03-28T17:49:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.138+0000 7fba99600640 1 -- 172.21.15.45:0/921400655 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fbaa800f750 con 0x558a8c7b1240 2024-03-28T17:49:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.138+0000 7fba99600640 1 -- 172.21.15.45:0/921400655 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbaa8009be0 con 0x558a8c7b1240 2024-03-28T17:49:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.138+0000 7fbab945c740 10 client.25643 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.138+0000 7fbab945c740 1 -- 172.21.15.45:0/921400655 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558a8c4e97d0 con 0x558a8c7b1240 2024-03-28T17:49:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.138+0000 7fba99600640 1 -- 172.21.15.45:0/921400655 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbaa8004cc0 con 0x558a8c7b1240 2024-03-28T17:49:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.138+0000 7fba99600640 1 -- 172.21.15.45:0/921400655 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fbaa8004040 con 0x558a8c7b1240 2024-03-28T17:49:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.139+0000 7fbab945c740 10 client.25643 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.139+0000 7fbab945c740 10 client.25643 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:49:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.139+0000 7fbab945c740 20 client.25643 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.139+0000 7fbab945c740 1 --2- 172.21.15.45:0/921400655 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x558a8c7c4f80 0x558a8c7c7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:54.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.139+0000 7fbab945c740 4 client.25643 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:49:54.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.139+0000 7fbab945c740 1 -- 172.21.15.45:0/921400655 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558a8c718800 con 0x558a8c7c4f80 2024-03-28T17:49:54.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.139+0000 7fbaaf400640 1 --2- 172.21.15.45:0/921400655 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x558a8c7c4f80 0x558a8c7c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:54.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.139+0000 7fbaaf400640 1 --2- 172.21.15.45:0/921400655 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x558a8c7c4f80 0x558a8c7c7370 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dd862193ac18344 server_cookie=c4a0ec5f2f7c4aa4 in_seq=0 out_seq=0 2024-03-28T17:49:54.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.139+0000 7fba99600640 10 client.25643 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:54.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.140+0000 7fba99600640 1 -- 172.21.15.45:0/921400655 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558a8c718800 con 0x558a8c7c4f80 2024-03-28T17:49:54.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.140+0000 7fba99600640 10 client.25643 handle_command_reply: tid=0 2024-03-28T17:49:54.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.140+0000 7fbab945c740 1 client.25643 shutdown 2024-03-28T17:49:54.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.140+0000 7fbab945c740 1 -- 172.21.15.45:0/921400655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7b1240 msgr2=0x558a8c7b5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:54.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.140+0000 7fbab945c740 1 --2- 172.21.15.45:0/921400655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7b1240 0x558a8c7b5090 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8012fd0 tx=0x7fbaa800fca0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.141+0000 7fbab945c740 1 -- 172.21.15.45:0/921400655 shutdown_connections 2024-03-28T17:49:54.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.141+0000 7fbab945c740 1 --2- 172.21.15.45:0/921400655 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x558a8c7c4f80 0x558a8c7c7370 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.141+0000 7fbab945c740 1 --2- 172.21.15.45:0/921400655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7b1240 0x558a8c7b5090 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.141+0000 7fbab945c740 1 --2- 172.21.15.45:0/921400655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a8c7a8140 0x558a8c7b4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.141+0000 7fbab945c740 1 --2- 172.21.15.45:0/921400655 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558a8c7a7790 0x558a8c7b76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.141+0000 7fba99600640 0 client.25643 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:54.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.141+0000 7fbab945c740 1 -- 172.21.15.45:0/921400655 >> 172.21.15.45:0/921400655 conn(0x558a8c7a1f80 msgr2=0x558a8c7a3cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:54.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.141+0000 7fbab945c740 1 -- 172.21.15.45:0/921400655 shutdown_connections 2024-03-28T17:49:54.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.141+0000 7fbab945c740 1 -- 172.21.15.45:0/921400655 wait complete. 2024-03-28T17:49:54.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.141+0000 7fbab945c740 20 client.25643 trim_cache size 0 max 16384 2024-03-28T17:49:54.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.154+0000 7fbab945c740 1 Processor -- start 2024-03-28T17:49:54.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.154+0000 7fbab945c740 1 -- start start 2024-03-28T17:49:54.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.154+0000 7fbab945c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a8c7a7790 0x558a8c7a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:54.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.154+0000 7fbab945c740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7a8140 0x558a8c7b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:54.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.154+0000 7fbab945c740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558a8c7b1240 0x558a8c7b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:54.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.154+0000 7fbab945c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab01ad400 con 0x558a8c7a7790 2024-03-28T17:49:54.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.154+0000 7fbab945c740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab01aaf00 con 0x558a8c7a8140 2024-03-28T17:49:54.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.154+0000 7fbab945c740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab01a8f80 con 0x558a8c7b1240 2024-03-28T17:49:54.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.155+0000 7fbaaea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7a8140 0x558a8c7b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:54.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.155+0000 7fbaaf400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a8c7a7790 0x558a8c7a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:54.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.155+0000 7fbaaea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7a8140 0x558a8c7b2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:47142/0 (socket says 172.21.15.45:47142) 2024-03-28T17:49:54.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.155+0000 7fbaaea00640 1 -- 172.21.15.45:0/554818984 learned_addr learned my addr 172.21.15.45:0/554818984 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:54.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.155+0000 7fbaafe00640 1 --2- 172.21.15.45:0/554818984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558a8c7b1240 0x558a8c7b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:54.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.155+0000 7fbaafe00640 1 -- 172.21.15.45:0/554818984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7a8140 msgr2=0x558a8c7b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:54.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.155+0000 7fbaafe00640 1 --2- 172.21.15.45:0/554818984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7a8140 0x558a8c7b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.155+0000 7fbaafe00640 1 -- 172.21.15.45:0/554818984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a8c7a7790 msgr2=0x558a8c7a0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:54.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.155+0000 7fbaafe00640 1 --2- 172.21.15.45:0/554818984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a8c7a7790 0x558a8c7a0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.155+0000 7fbaafe00640 1 -- 172.21.15.45:0/554818984 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558a8c7a73d0 con 0x558a8c7b1240 2024-03-28T17:49:54.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.156+0000 7fbaafe00640 1 --2- 172.21.15.45:0/554818984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558a8c7b1240 0x558a8c7b71b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8012fd0 tx=0x7fbaa8003f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbb7dfabfa9f0d2c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:54.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.156+0000 7fba9be00640 1 -- 172.21.15.45:0/554818984 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8004d10 con 0x558a8c7b1240 2024-03-28T17:49:54.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.156+0000 7fba9be00640 1 -- 172.21.15.45:0/554818984 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbaa8009ad0 con 0x558a8c7b1240 2024-03-28T17:49:54.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.156+0000 7fba9be00640 1 -- 172.21.15.45:0/554818984 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8024470 con 0x558a8c7b1240 2024-03-28T17:49:54.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.156+0000 7fbab945c740 1 -- 172.21.15.45:0/554818984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558a8c7b1240 msgr2=0x558a8c7b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:54.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.156+0000 7fbab945c740 1 --2- 172.21.15.45:0/554818984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558a8c7b1240 0x558a8c7b71b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8012fd0 tx=0x7fbaa8003f60 comp rx=0 tx=0).stop 2024-03-28T17:49:54.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.157+0000 7fbab945c740 1 -- 172.21.15.45:0/554818984 shutdown_connections 2024-03-28T17:49:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.157+0000 7fbab945c740 1 --2- 172.21.15.45:0/554818984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558a8c7b1240 0x558a8c7b71b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.157+0000 7fbab945c740 1 --2- 172.21.15.45:0/554818984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7a8140 0x558a8c7b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.157+0000 7fbab945c740 1 --2- 172.21.15.45:0/554818984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a8c7a7790 0x558a8c7a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.157+0000 7fbab945c740 1 -- 172.21.15.45:0/554818984 >> 172.21.15.45:0/554818984 conn(0x558a8c7a1f80 msgr2=0x558a8c7a08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.157+0000 7fbab945c740 1 -- 172.21.15.45:0/554818984 shutdown_connections 2024-03-28T17:49:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.157+0000 7fbab945c740 1 -- 172.21.15.45:0/554818984 wait complete. 2024-03-28T17:49:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.157+0000 7fbab945c740 1 -- start start 2024-03-28T17:49:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.157+0000 7fbab945c740 1 Processor -- start 2024-03-28T17:49:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.158+0000 7fbab945c740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558a8c7a7790 0x558a8c7bcd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.158+0000 7fbab945c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a8c7a8140 0x558a8c7bd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.158+0000 7fbab945c740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7b1240 0x558a8c7b9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.158+0000 7fbab945c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab01a9580 con 0x558a8c7a8140 2024-03-28T17:49:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.158+0000 7fbab945c740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab01aa8a0 con 0x558a8c7b1240 2024-03-28T17:49:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.158+0000 7fbab945c740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab01a9be0 con 0x558a8c7a7790 2024-03-28T17:49:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.158+0000 7fbab945c740 10 client.0 _ll_register_callbacks cb 0x7fbaa0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:54.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.158+0000 7fbaaea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a8c7a8140 0x558a8c7bd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:54.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.158+0000 7fbaafe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7b1240 0x558a8c7b9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:54.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.158+0000 7fbaaea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a8c7a8140 0x558a8c7bd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47174/0 (socket says 172.21.15.45:47174) 2024-03-28T17:49:54.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.158+0000 7fbaaea00640 1 -- 172.21.15.45:0/831536538 learned_addr learned my addr 172.21.15.45:0/831536538 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:54.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.159+0000 7fbaaf400640 1 --2- 172.21.15.45:0/831536538 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558a8c7a7790 0x558a8c7bcd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:54.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.159+0000 7fbaaea00640 1 -- 172.21.15.45:0/831536538 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558a8c7a7790 msgr2=0x558a8c7bcd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:54.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.159+0000 7fbaaea00640 1 --2- 172.21.15.45:0/831536538 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558a8c7a7790 0x558a8c7bcd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.159+0000 7fbaaea00640 1 -- 172.21.15.45:0/831536538 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7b1240 msgr2=0x558a8c7b9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:54.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.159+0000 7fbaaea00640 1 --2- 172.21.15.45:0/831536538 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7b1240 0x558a8c7b9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.159+0000 7fbaaea00640 1 -- 172.21.15.45:0/831536538 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558a8c7b9930 con 0x558a8c7a8140 2024-03-28T17:49:54.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.159+0000 7fbaaea00640 1 --2- 172.21.15.45:0/831536538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a8c7a8140 0x558a8c7bd260 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fba9c09aef0 tx=0x7fba9c09b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=346fe8c8ec78de30 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:54.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.159+0000 7fba99600640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-28T17:49:54.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.159+0000 7fba99600640 1 -- 172.21.15.45:0/831536538 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9c0a3700 con 0x558a8c7a8140 2024-03-28T17:49:54.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.159+0000 7fba99600640 1 -- 172.21.15.45:0/831536538 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fba9c004230 con 0x558a8c7a8140 2024-03-28T17:49:54.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.159+0000 7fbab945c740 1 -- 172.21.15.45:0/831536538 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbaa8009ad0 con 0x558a8c7a8140 2024-03-28T17:49:54.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.160+0000 7fba99600640 1 -- 172.21.15.45:0/831536538 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba9c004430 con 0x558a8c7a8140 2024-03-28T17:49:54.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.160+0000 7fbab945c740 10 client.15495 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:54.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.160+0000 7fbab945c740 1 -- 172.21.15.45:0/831536538 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558a8c7a8860 con 0x558a8c7a8140 2024-03-28T17:49:54.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.160+0000 7fba99600640 1 -- 172.21.15.45:0/831536538 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbaa8009ad0 con 0x558a8c7a8140 2024-03-28T17:49:54.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.160+0000 7fba99600640 1 -- 172.21.15.45:0/831536538 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fba9c09a3b0 con 0x558a8c7a8140 2024-03-28T17:49:54.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.160+0000 7fbab945c740 10 client.15495 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:54.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.160+0000 7fbab945c740 10 client.15495 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:49:54.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.160+0000 7fbab945c740 20 client.15495 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:54.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.160+0000 7fbab945c740 1 --2- 172.21.15.45:0/831536538 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x558a8c7a8f70 0x558a8c7ab390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:54.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.160+0000 7fbab945c740 4 client.15495 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T17:49:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.160+0000 7fbab945c740 1 -- 172.21.15.45:0/831536538 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558a8c712b50 con 0x558a8c7a8f70 2024-03-28T17:49:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.161+0000 7fbaaf400640 1 --2- 172.21.15.45:0/831536538 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x558a8c7a8f70 0x558a8c7ab390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.161+0000 7fba99600640 10 client.15495 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.161+0000 7fbaaf400640 1 --2- 172.21.15.45:0/831536538 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x558a8c7a8f70 0x558a8c7ab390 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e52d94c2927638f server_cookie=1b4a69a6f7935ceb in_seq=0 out_seq=0 2024-03-28T17:49:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.161+0000 7fba99600640 1 -- 172.21.15.45:0/831536538 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558a8c712b50 con 0x558a8c7a8f70 2024-03-28T17:49:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.161+0000 7fba99600640 10 client.15495 handle_command_reply: tid=0 2024-03-28T17:49:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.161+0000 7fbab945c740 1 client.15495 shutdown 2024-03-28T17:49:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.162+0000 7fbab945c740 1 -- 172.21.15.45:0/831536538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a8c7a8140 msgr2=0x558a8c7bd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.162+0000 7fbab945c740 1 --2- 172.21.15.45:0/831536538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a8c7a8140 0x558a8c7bd260 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fba9c09aef0 tx=0x7fba9c09b790 comp rx=0 tx=0).stop 2024-03-28T17:49:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.162+0000 7fbab945c740 1 -- 172.21.15.45:0/831536538 shutdown_connections 2024-03-28T17:49:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.162+0000 7fbab945c740 1 --2- 172.21.15.45:0/831536538 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x558a8c7a8f70 0x558a8c7ab390 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.162+0000 7fbab945c740 1 --2- 172.21.15.45:0/831536538 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a8c7b1240 0x558a8c7b9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.162+0000 7fbab945c740 1 --2- 172.21.15.45:0/831536538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a8c7a8140 0x558a8c7bd260 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.162+0000 7fbab945c740 1 --2- 172.21.15.45:0/831536538 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558a8c7a7790 0x558a8c7bcd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.162+0000 7fbab945c740 1 -- 172.21.15.45:0/831536538 >> 172.21.15.45:0/831536538 conn(0x558a8c7a1f80 msgr2=0x558a8c7a1160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:54.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.162+0000 7fba99600640 0 client.15495 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:54.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.162+0000 7fbab945c740 1 -- 172.21.15.45:0/831536538 shutdown_connections 2024-03-28T17:49:54.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.162+0000 7fbab945c740 1 -- 172.21.15.45:0/831536538 wait complete. 2024-03-28T17:49:54.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.162+0000 7fbab945c740 20 client.15495 trim_cache size 0 max 16384 2024-03-28T17:49:54.165 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-28T17:49:54.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.163+0000 7fbab5200640 1 -- 172.21.15.45:0/3761059027 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fba940a2990 msgr2=0x7fba940a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:54.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.163+0000 7fbab5200640 1 --2- 172.21.15.45:0/3761059027 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fba940a2990 0x7fba940a4e50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x558a8c7a8580 tx=0x7fbaa002e040 comp rx=0 tx=0).stop 2024-03-28T17:49:54.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.163+0000 7fbab5200640 1 -- 172.21.15.45:0/3761059027 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbab0103050 msgr2=0x7fbab0198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:54.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.163+0000 7fbab5200640 1 --2- 172.21.15.45:0/3761059027 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbab0103050 0x7fbab0198300 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fba9c00d6d0 tx=0x7fba9c00db90 comp rx=0 tx=0).stop 2024-03-28T17:49:54.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.163+0000 7fbab5200640 1 -- 172.21.15.45:0/3761059027 shutdown_connections 2024-03-28T17:49:54.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.163+0000 7fbab5200640 1 --2- 172.21.15.45:0/3761059027 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fba940a2990 0x7fba940a4e50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.163+0000 7fbab5200640 1 --2- 172.21.15.45:0/3761059027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbab010bb00 0x7fbab019c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.163+0000 7fbab5200640 1 --2- 172.21.15.45:0/3761059027 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbab0103050 0x7fbab0198300 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.163+0000 7fbab5200640 1 --2- 172.21.15.45:0/3761059027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbab0102730 0x7fbab0197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:54.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.163+0000 7fbab5200640 1 -- 172.21.15.45:0/3761059027 >> 172.21.15.45:0/3761059027 conn(0x7fbab00fbef0 msgr2=0x7fbab00fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:54.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.163+0000 7fbab5200640 1 -- 172.21.15.45:0/3761059027 shutdown_connections 2024-03-28T17:49:54.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:54.163+0000 7fbab5200640 1 -- 172.21.15.45:0/3761059027 wait complete. 2024-03-28T17:49:54.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:54 smithi045 ceph-mon[31773]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:49:55.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:54 smithi071 ceph-mon[36597]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:49:55.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:54 smithi151 ceph-mon[36481]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:49:55.180 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T17:49:55.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T17:49:55.180 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T17:49:55.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.383+0000 7fdb42200640 1 -- 172.21.15.45:0/2257775013 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb3c103840 msgr2=0x7fdb3c103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:55.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.383+0000 7fdb42200640 1 --2- 172.21.15.45:0/2257775013 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb3c103840 0x7fdb3c103cc0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fdb28009900 tx=0x7fdb2802bc30 comp rx=0 tx=0).stop 2024-03-28T17:49:55.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.383+0000 7fdb42200640 1 -- 172.21.15.45:0/2257775013 shutdown_connections 2024-03-28T17:49:55.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.383+0000 7fdb42200640 1 --2- 172.21.15.45:0/2257775013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb3c104200 0x7fdb3c110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.383+0000 7fdb42200640 1 --2- 172.21.15.45:0/2257775013 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb3c103840 0x7fdb3c103cc0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.383+0000 7fdb42200640 1 --2- 172.21.15.45:0/2257775013 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdb3c109850 0x7fdb3c109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.383+0000 7fdb42200640 1 -- 172.21.15.45:0/2257775013 >> 172.21.15.45:0/2257775013 conn(0x7fdb3c0ff350 msgr2=0x7fdb3c101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:55.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.383+0000 7fdb42200640 1 -- 172.21.15.45:0/2257775013 shutdown_connections 2024-03-28T17:49:55.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.383+0000 7fdb42200640 1 -- 172.21.15.45:0/2257775013 wait complete. 2024-03-28T17:49:55.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.384+0000 7fdb42200640 1 Processor -- start 2024-03-28T17:49:55.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.384+0000 7fdb42200640 1 -- start start 2024-03-28T17:49:55.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.384+0000 7fdb42200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb3c103840 0x7fdb3c19a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.385+0000 7fdb42200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb3c104200 0x7fdb3c19a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.385+0000 7fdb42200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdb3c109850 0x7fdb3c19ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.385+0000 7fdb42200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c113cb0 con 0x7fdb3c104200 2024-03-28T17:49:55.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.385+0000 7fdb42200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c113b30 con 0x7fdb3c103840 2024-03-28T17:49:55.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.385+0000 7fdb42200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c113e30 con 0x7fdb3c109850 2024-03-28T17:49:55.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.385+0000 7fdb3b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb3c103840 0x7fdb3c19a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.385+0000 7fdb3be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdb3c109850 0x7fdb3c19ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.385+0000 7fdb3b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb3c103840 0x7fdb3c19a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:47180/0 (socket says 172.21.15.45:47180) 2024-03-28T17:49:55.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.385+0000 7fdb3b400640 1 -- 172.21.15.45:0/3978263022 learned_addr learned my addr 172.21.15.45:0/3978263022 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:55.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.385+0000 7fdb3aa00640 1 --2- 172.21.15.45:0/3978263022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb3c104200 0x7fdb3c19a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.385+0000 7fdb3b400640 1 -- 172.21.15.45:0/3978263022 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdb3c109850 msgr2=0x7fdb3c19ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:55.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.385+0000 7fdb3b400640 1 --2- 172.21.15.45:0/3978263022 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdb3c109850 0x7fdb3c19ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.385+0000 7fdb3b400640 1 -- 172.21.15.45:0/3978263022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb3c104200 msgr2=0x7fdb3c19a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:55.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.385+0000 7fdb3b400640 1 --2- 172.21.15.45:0/3978263022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb3c104200 0x7fdb3c19a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.385+0000 7fdb3b400640 1 -- 172.21.15.45:0/3978263022 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3c19f180 con 0x7fdb3c103840 2024-03-28T17:49:55.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.385+0000 7fdb3aa00640 1 --2- 172.21.15.45:0/3978263022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb3c104200 0x7fdb3c19a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:49:55.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.386+0000 7fdb3b400640 1 --2- 172.21.15.45:0/3978263022 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb3c103840 0x7fdb3c19a190 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c00e9f0 tx=0x7fdb2c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:55.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.386+0000 7fdb33e00640 1 -- 172.21.15.45:0/3978263022 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c00cd80 con 0x7fdb3c103840 2024-03-28T17:49:55.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.386+0000 7fdb33e00640 1 -- 172.21.15.45:0/3978263022 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb2c005500 con 0x7fdb3c103840 2024-03-28T17:49:55.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.386+0000 7fdb42200640 1 -- 172.21.15.45:0/3978263022 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb3c19f470 con 0x7fdb3c103840 2024-03-28T17:49:55.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.386+0000 7fdb33e00640 1 -- 172.21.15.45:0/3978263022 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c010690 con 0x7fdb3c103840 2024-03-28T17:49:55.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.386+0000 7fdb42200640 1 -- 172.21.15.45:0/3978263022 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb3c1a6d00 con 0x7fdb3c103840 2024-03-28T17:49:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.390+0000 7fdb45016740 1 Processor -- start 2024-03-28T17:49:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.390+0000 7fdb33e00640 1 -- 172.21.15.45:0/3978263022 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 26) v1 ==== 137466+0+0 (secure 0 0 0) 0x7fdb2c0046c0 con 0x7fdb3c103840 2024-03-28T17:49:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.391+0000 7fdb45016740 1 -- start start 2024-03-28T17:49:55.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.391+0000 7fdb45016740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ee92134080 0x55ee92134460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.391+0000 7fdb45016740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee921349a0 0x55ee9213d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.391+0000 7fdb45016740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee9213d580 0x55ee9213f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.391+0000 7fdb45016740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee92098eb0 con 0x55ee921349a0 2024-03-28T17:49:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.391+0000 7fdb45016740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee92041a60 con 0x55ee92134080 2024-03-28T17:49:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.391+0000 7fdb45016740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee92094f50 con 0x55ee9213d580 2024-03-28T17:49:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.391+0000 7fdb3be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ee92134080 0x55ee92134460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.392+0000 7fdb3be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ee92134080 0x55ee92134460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:47196/0 (socket says 172.21.15.45:47196) 2024-03-28T17:49:55.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.392+0000 7fdb3be00640 1 -- 172.21.15.45:0/2567345795 learned_addr learned my addr 172.21.15.45:0/2567345795 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:55.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.392+0000 7fdb3b400640 1 --2- 172.21.15.45:0/2567345795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee921349a0 0x55ee9213d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.392+0000 7fdb3be00640 1 -- 172.21.15.45:0/2567345795 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee9213d580 msgr2=0x55ee9213f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:55.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.392+0000 7fdb3aa00640 1 --2- 172.21.15.45:0/2567345795 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee9213d580 0x55ee9213f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.392+0000 7fdb3be00640 1 --2- 172.21.15.45:0/2567345795 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee9213d580 0x55ee9213f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.392+0000 7fdb3be00640 1 -- 172.21.15.45:0/2567345795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee921349a0 msgr2=0x55ee9213d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:55.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.392+0000 7fdb3be00640 1 --2- 172.21.15.45:0/2567345795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee921349a0 0x55ee9213d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.392+0000 7fdb3be00640 1 -- 172.21.15.45:0/2567345795 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee91e757e0 con 0x55ee92134080 2024-03-28T17:49:55.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.392+0000 7fdb3aa00640 1 --2- 172.21.15.45:0/2567345795 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee9213d580 0x55ee9213f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T17:49:55.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.392+0000 7fdb33e00640 1 --2- 172.21.15.45:0/3978263022 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fdb200a2a80 0x7fdb200a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.392+0000 7fdb33e00640 1 -- 172.21.15.45:0/3978263022 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fdb2c0cf380 con 0x7fdb3c103840 2024-03-28T17:49:55.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.392+0000 7fdb3be00640 1 --2- 172.21.15.45:0/2567345795 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ee92134080 0x55ee92134460 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fdb34009fd0 tx=0x7fdb340118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=760bab859826a66b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:55.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.393+0000 7fdb27e00640 1 -- 172.21.15.45:0/2567345795 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb34004d20 con 0x55ee92134080 2024-03-28T17:49:55.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.393+0000 7fdb27e00640 1 -- 172.21.15.45:0/2567345795 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb34004050 con 0x55ee92134080 2024-03-28T17:49:55.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.393+0000 7fdb27e00640 1 -- 172.21.15.45:0/2567345795 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb34005260 con 0x55ee92134080 2024-03-28T17:49:55.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.393+0000 7fdb3b400640 1 --2- 172.21.15.45:0/3978263022 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fdb200a2a80 0x7fdb200a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.393+0000 7fdb45016740 1 -- 172.21.15.45:0/2567345795 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ee92134080 msgr2=0x55ee92134460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:55.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.393+0000 7fdb45016740 1 --2- 172.21.15.45:0/2567345795 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ee92134080 0x55ee92134460 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fdb34009fd0 tx=0x7fdb340118d0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.394+0000 7fdb3b400640 1 --2- 172.21.15.45:0/3978263022 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fdb200a2a80 0x7fdb200a4f40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c0cca80 tx=0x7fdb2c004a40 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:55.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.394+0000 7fdb45016740 1 -- 172.21.15.45:0/2567345795 shutdown_connections 2024-03-28T17:49:55.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.394+0000 7fdb45016740 1 --2- 172.21.15.45:0/2567345795 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee9213d580 0x55ee9213f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.394+0000 7fdb45016740 1 --2- 172.21.15.45:0/2567345795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee921349a0 0x55ee9213d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.394+0000 7fdb45016740 1 --2- 172.21.15.45:0/2567345795 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ee92134080 0x55ee92134460 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.394+0000 7fdb45016740 1 -- 172.21.15.45:0/2567345795 >> 172.21.15.45:0/2567345795 conn(0x55ee9212e7b0 msgr2=0x55ee9212eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:55.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.394+0000 7fdb45016740 1 -- 172.21.15.45:0/2567345795 shutdown_connections 2024-03-28T17:49:55.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.394+0000 7fdb45016740 1 -- 172.21.15.45:0/2567345795 wait complete. 2024-03-28T17:49:55.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.394+0000 7fdb45016740 1 -- start start 2024-03-28T17:49:55.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.394+0000 7fdb45016740 1 Processor -- start 2024-03-28T17:49:55.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.395+0000 7fdb45016740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee92134080 0x55ee9213cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.395+0000 7fdb45016740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee921349a0 0x55ee9213a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.395+0000 7fdb45016740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ee9213d580 0x55ee9213a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.395+0000 7fdb45016740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee92140370 con 0x55ee92134080 2024-03-28T17:49:55.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.395+0000 7fdb45016740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee92054eb0 con 0x55ee9213d580 2024-03-28T17:49:55.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.395+0000 7fdb45016740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee921404f0 con 0x55ee921349a0 2024-03-28T17:49:55.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.395+0000 7fdb45016740 10 client.0 _ll_register_callbacks cb 0x55ee92049150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:55.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.395+0000 7fdb3be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee92134080 0x55ee9213cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.396+0000 7fdb3be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee92134080 0x55ee9213cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47226/0 (socket says 172.21.15.45:47226) 2024-03-28T17:49:55.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.396+0000 7fdb3be00640 1 -- 172.21.15.45:0/3193943650 learned_addr learned my addr 172.21.15.45:0/3193943650 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:55.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.396+0000 7fdb3be00640 1 --2- 172.21.15.45:0/3193943650 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ee9213d580 0x55ee9213a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.396+0000 7fdb3aa00640 1 --2- 172.21.15.45:0/3193943650 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee921349a0 0x55ee9213a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.396+0000 7fdb3aa00640 1 -- 172.21.15.45:0/3193943650 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ee9213d580 msgr2=0x55ee9213a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:55.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.396+0000 7fdb3aa00640 1 --2- 172.21.15.45:0/3193943650 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ee9213d580 0x55ee9213a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.396+0000 7fdb3aa00640 1 -- 172.21.15.45:0/3193943650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee92134080 msgr2=0x55ee9213cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.396+0000 7fdb3aa00640 1 --2- 172.21.15.45:0/3193943650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee92134080 0x55ee9213cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.396+0000 7fdb3aa00640 1 -- 172.21.15.45:0/3193943650 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ee91ed5f70 con 0x55ee921349a0 2024-03-28T17:49:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.396+0000 7fdb3aa00640 1 --2- 172.21.15.45:0/3193943650 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee921349a0 0x55ee9213a290 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdb2802c170 tx=0x7fdb28004520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46ded11e816c5d9c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.396+0000 7fdb24c00640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:49:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.397+0000 7fdb24c00640 1 -- 172.21.15.45:0/3193943650 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb28042070 con 0x55ee921349a0 2024-03-28T17:49:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.397+0000 7fdb45016740 1 -- 172.21.15.45:0/3193943650 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdb34004050 con 0x55ee921349a0 2024-03-28T17:49:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.397+0000 7fdb24c00640 1 -- 172.21.15.45:0/3193943650 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fdb2802ccc0 con 0x55ee921349a0 2024-03-28T17:49:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.397+0000 7fdb24c00640 1 -- 172.21.15.45:0/3193943650 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb2804d050 con 0x55ee921349a0 2024-03-28T17:49:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.397+0000 7fdb45016740 10 client.25630 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.397+0000 7fdb45016740 1 -- 172.21.15.45:0/3193943650 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55ee91d6d170 con 0x55ee921349a0 2024-03-28T17:49:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.397+0000 7fdb24c00640 1 -- 172.21.15.45:0/3193943650 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb34004050 con 0x55ee921349a0 2024-03-28T17:49:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.398+0000 7fdb24c00640 1 -- 172.21.15.45:0/3193943650 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fdb28004b40 con 0x55ee921349a0 2024-03-28T17:49:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.398+0000 7fdb45016740 10 client.25630 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.398+0000 7fdb45016740 10 client.25630 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:49:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.398+0000 7fdb45016740 20 client.25630 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.398+0000 7fdb45016740 1 --2- 172.21.15.45:0/3193943650 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ee9214d530 0x55ee9214f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.398+0000 7fdb45016740 4 client.25630 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:49:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.398+0000 7fdb45016740 1 -- 172.21.15.45:0/3193943650 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ee920a63f0 con 0x55ee9214d530 2024-03-28T17:49:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.398+0000 7fdb3be00640 1 --2- 172.21.15.45:0/3193943650 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ee9214d530 0x55ee9214f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.398+0000 7fdb3be00640 1 --2- 172.21.15.45:0/3193943650 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ee9214d530 0x55ee9214f920 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62f0ddffdb3b9816 server_cookie=8c6b91722287bc88 in_seq=0 out_seq=0 2024-03-28T17:49:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.398+0000 7fdb24c00640 10 client.25630 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.399+0000 7fdb24c00640 1 -- 172.21.15.45:0/3193943650 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ee920a63f0 con 0x55ee9214d530 2024-03-28T17:49:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.399+0000 7fdb24c00640 10 client.25630 handle_command_reply: tid=0 2024-03-28T17:49:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.399+0000 7fdb45016740 1 client.25630 shutdown 2024-03-28T17:49:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.399+0000 7fdb45016740 1 -- 172.21.15.45:0/3193943650 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee921349a0 msgr2=0x55ee9213a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.399+0000 7fdb45016740 1 --2- 172.21.15.45:0/3193943650 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee921349a0 0x55ee9213a290 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdb2802c170 tx=0x7fdb28004520 comp rx=0 tx=0).stop 2024-03-28T17:49:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.399+0000 7fdb45016740 1 -- 172.21.15.45:0/3193943650 shutdown_connections 2024-03-28T17:49:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.399+0000 7fdb45016740 1 --2- 172.21.15.45:0/3193943650 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ee9214d530 0x55ee9214f920 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.399+0000 7fdb45016740 1 --2- 172.21.15.45:0/3193943650 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ee9213d580 0x55ee9213a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.399+0000 7fdb45016740 1 --2- 172.21.15.45:0/3193943650 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee921349a0 0x55ee9213a290 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.399+0000 7fdb45016740 1 --2- 172.21.15.45:0/3193943650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee92134080 0x55ee9213cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.400+0000 7fdb45016740 1 -- 172.21.15.45:0/3193943650 >> 172.21.15.45:0/3193943650 conn(0x55ee9212e7b0 msgr2=0x55ee92130500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.400+0000 7fdb24c00640 0 client.25630 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:55.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.400+0000 7fdb45016740 1 -- 172.21.15.45:0/3193943650 shutdown_connections 2024-03-28T17:49:55.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.400+0000 7fdb45016740 1 -- 172.21.15.45:0/3193943650 wait complete. 2024-03-28T17:49:55.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.400+0000 7fdb45016740 20 client.25630 trim_cache size 0 max 16384 2024-03-28T17:49:55.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.420+0000 7fdb45016740 1 Processor -- start 2024-03-28T17:49:55.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.420+0000 7fdb45016740 1 -- start start 2024-03-28T17:49:55.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.421+0000 7fdb45016740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee92134080 0x55ee9213c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.421+0000 7fdb45016740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee921349a0 0x55ee9213cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.421+0000 7fdb45016740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ee9213d580 0x55ee92148c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.421+0000 7fdb45016740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c1ad590 con 0x55ee92134080 2024-03-28T17:49:55.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.421+0000 7fdb45016740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c1ab090 con 0x55ee9213d580 2024-03-28T17:49:55.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.421+0000 7fdb45016740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c1a9130 con 0x55ee921349a0 2024-03-28T17:49:55.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.421+0000 7fdb3be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee92134080 0x55ee9213c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.421+0000 7fdb3aa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee921349a0 0x55ee9213cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.421+0000 7fdb3be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee92134080 0x55ee9213c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47228/0 (socket says 172.21.15.45:47228) 2024-03-28T17:49:55.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.421+0000 7fdb3be00640 1 -- 172.21.15.45:0/2475095008 learned_addr learned my addr 172.21.15.45:0/2475095008 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:55.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.421+0000 7fdb3aa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee921349a0 0x55ee9213cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:33730/0 (socket says 172.21.15.45:33730) 2024-03-28T17:49:55.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.421+0000 7fdb3be00640 1 --2- 172.21.15.45:0/2475095008 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ee9213d580 0x55ee92148c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.422+0000 7fdb3aa00640 1 -- 172.21.15.45:0/2475095008 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ee9213d580 msgr2=0x55ee92148c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:55.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.422+0000 7fdb3aa00640 1 --2- 172.21.15.45:0/2475095008 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ee9213d580 0x55ee92148c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.422+0000 7fdb3aa00640 1 -- 172.21.15.45:0/2475095008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee92134080 msgr2=0x55ee9213c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:55.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.422+0000 7fdb3aa00640 1 --2- 172.21.15.45:0/2475095008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee92134080 0x55ee9213c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.422+0000 7fdb3aa00640 1 -- 172.21.15.45:0/2475095008 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee92129c20 con 0x55ee921349a0 2024-03-28T17:49:55.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.422+0000 7fdb3aa00640 1 --2- 172.21.15.45:0/2475095008 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee921349a0 0x55ee9213cf00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdb28006010 tx=0x7fdb28005bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=607097b47e2ea132 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:55.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.422+0000 7fdb27400640 1 -- 172.21.15.45:0/2475095008 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2804a070 con 0x55ee921349a0 2024-03-28T17:49:55.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.422+0000 7fdb27400640 1 -- 172.21.15.45:0/2475095008 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb28009780 con 0x55ee921349a0 2024-03-28T17:49:55.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.422+0000 7fdb27400640 1 -- 172.21.15.45:0/2475095008 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb28042030 con 0x55ee921349a0 2024-03-28T17:49:55.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.423+0000 7fdb45016740 1 -- 172.21.15.45:0/2475095008 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee921349a0 msgr2=0x55ee9213cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:55.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.423+0000 7fdb45016740 1 --2- 172.21.15.45:0/2475095008 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee921349a0 0x55ee9213cf00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdb28006010 tx=0x7fdb28005bc0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.423+0000 7fdb45016740 1 -- 172.21.15.45:0/2475095008 shutdown_connections 2024-03-28T17:49:55.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.423+0000 7fdb45016740 1 --2- 172.21.15.45:0/2475095008 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ee9213d580 0x55ee92148c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.423+0000 7fdb45016740 1 --2- 172.21.15.45:0/2475095008 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee921349a0 0x55ee9213cf00 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.423+0000 7fdb45016740 1 --2- 172.21.15.45:0/2475095008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee92134080 0x55ee9213c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.423+0000 7fdb45016740 1 -- 172.21.15.45:0/2475095008 >> 172.21.15.45:0/2475095008 conn(0x55ee9212e7b0 msgr2=0x55ee92130400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:55.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.423+0000 7fdb45016740 1 -- 172.21.15.45:0/2475095008 shutdown_connections 2024-03-28T17:49:55.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.423+0000 7fdb45016740 1 -- 172.21.15.45:0/2475095008 wait complete. 2024-03-28T17:49:55.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.424+0000 7fdb45016740 1 -- start start 2024-03-28T17:49:55.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.424+0000 7fdb45016740 1 Processor -- start 2024-03-28T17:49:55.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.425+0000 7fdb45016740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee92134080 0x55ee92145300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.425+0000 7fdb45016740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ee921349a0 0x55ee92137440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.425+0000 7fdb45016740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9213d580 0x55ee92137980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.425+0000 7fdb45016740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c1a9730 con 0x55ee9213d580 2024-03-28T17:49:55.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.425+0000 7fdb45016740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c1aaa30 con 0x55ee921349a0 2024-03-28T17:49:55.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.425+0000 7fdb45016740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c1a9d70 con 0x55ee92134080 2024-03-28T17:49:55.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.425+0000 7fdb45016740 10 client.0 _ll_register_callbacks cb 0x7fdb340093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:55.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.425+0000 7fdb3be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee92134080 0x55ee92145300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.425+0000 7fdb3be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee92134080 0x55ee92145300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:33734/0 (socket says 172.21.15.45:33734) 2024-03-28T17:49:55.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.425+0000 7fdb3be00640 1 -- 172.21.15.45:0/3685817189 learned_addr learned my addr 172.21.15.45:0/3685817189 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:55.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.425+0000 7fdb3be00640 1 --2- 172.21.15.45:0/3685817189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9213d580 0x55ee92137980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.425+0000 7fdb3aa00640 1 --2- 172.21.15.45:0/3685817189 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ee921349a0 0x55ee92137440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.425+0000 7fdb3be00640 1 -- 172.21.15.45:0/3685817189 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ee921349a0 msgr2=0x55ee92137440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:55.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.425+0000 7fdb3be00640 1 --2- 172.21.15.45:0/3685817189 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ee921349a0 0x55ee92137440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.425+0000 7fdb3be00640 1 -- 172.21.15.45:0/3685817189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9213d580 msgr2=0x55ee92137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:55.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.425+0000 7fdb3be00640 1 --2- 172.21.15.45:0/3685817189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9213d580 0x55ee92137980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.425+0000 7fdb3be00640 1 -- 172.21.15.45:0/3685817189 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ee92137f70 con 0x55ee92134080 2024-03-28T17:49:55.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.426+0000 7fdb3be00640 1 --2- 172.21.15.45:0/3685817189 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee92134080 0x55ee92145300 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdb34011df0 tx=0x7fdb34005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a34d31cdb4d9c63 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:55.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.426+0000 7fdb24c00640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:49:55.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.426+0000 7fdb24c00640 1 -- 172.21.15.45:0/3685817189 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb34004990 con 0x55ee92134080 2024-03-28T17:49:55.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.426+0000 7fdb45016740 1 -- 172.21.15.45:0/3685817189 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdb28009780 con 0x55ee92134080 2024-03-28T17:49:55.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.426+0000 7fdb24c00640 1 -- 172.21.15.45:0/3685817189 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fdb34004b30 con 0x55ee92134080 2024-03-28T17:49:55.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.426+0000 7fdb24c00640 1 -- 172.21.15.45:0/3685817189 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb34004d30 con 0x55ee92134080 2024-03-28T17:49:55.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.426+0000 7fdb45016740 10 client.25636 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:55.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.426+0000 7fdb45016740 1 -- 172.21.15.45:0/3685817189 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55ee921a9c00 con 0x55ee92134080 2024-03-28T17:49:55.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.427+0000 7fdb24c00640 1 -- 172.21.15.45:0/3685817189 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb28009780 con 0x55ee92134080 2024-03-28T17:49:55.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.427+0000 7fdb24c00640 1 -- 172.21.15.45:0/3685817189 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7fdb340051b0 con 0x55ee92134080 2024-03-28T17:49:55.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.427+0000 7fdb45016740 10 client.25636 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:55.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.427+0000 7fdb45016740 10 client.25636 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:49:55.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.427+0000 7fdb45016740 20 client.25636 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:55.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.427+0000 7fdb45016740 1 --2- 172.21.15.45:0/3685817189 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ee921aa0f0 0x55ee921ac4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.427+0000 7fdb45016740 4 client.25636 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T17:49:55.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.427+0000 7fdb45016740 1 -- 172.21.15.45:0/3685817189 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55ee920a0740 con 0x55ee921aa0f0 2024-03-28T17:49:55.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.427+0000 7fdb3aa00640 1 --2- 172.21.15.45:0/3685817189 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ee921aa0f0 0x55ee921ac4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.428+0000 7fdb3aa00640 1 --2- 172.21.15.45:0/3685817189 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ee921aa0f0 0x55ee921ac4e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d28cbadae04f7cc4 server_cookie=4201eb188fdcc8e3 in_seq=0 out_seq=0 2024-03-28T17:49:55.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.428+0000 7fdb24c00640 10 client.25636 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:55.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.429+0000 7fdb24c00640 1 -- 172.21.15.45:0/3685817189 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55ee920a0740 con 0x55ee921aa0f0 2024-03-28T17:49:55.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.429+0000 7fdb24c00640 10 client.25636 handle_command_reply: tid=0 2024-03-28T17:49:55.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.429+0000 7fdb45016740 1 client.25636 shutdown 2024-03-28T17:49:55.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.429+0000 7fdb45016740 1 -- 172.21.15.45:0/3685817189 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee92134080 msgr2=0x55ee92145300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:55.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.429+0000 7fdb45016740 1 --2- 172.21.15.45:0/3685817189 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee92134080 0x55ee92145300 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdb34011df0 tx=0x7fdb34005fd0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.429+0000 7fdb45016740 1 -- 172.21.15.45:0/3685817189 shutdown_connections 2024-03-28T17:49:55.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.429+0000 7fdb45016740 1 --2- 172.21.15.45:0/3685817189 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ee921aa0f0 0x55ee921ac4e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.429+0000 7fdb45016740 1 --2- 172.21.15.45:0/3685817189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9213d580 0x55ee92137980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.429+0000 7fdb45016740 1 --2- 172.21.15.45:0/3685817189 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ee921349a0 0x55ee92137440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.429+0000 7fdb45016740 1 --2- 172.21.15.45:0/3685817189 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ee92134080 0x55ee92145300 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.429+0000 7fdb45016740 1 -- 172.21.15.45:0/3685817189 >> 172.21.15.45:0/3685817189 conn(0x55ee9212e7b0 msgr2=0x55ee9212cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:55.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.429+0000 7fdb24c00640 0 client.25636 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:55.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.430+0000 7fdb45016740 1 -- 172.21.15.45:0/3685817189 shutdown_connections 2024-03-28T17:49:55.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.430+0000 7fdb45016740 1 -- 172.21.15.45:0/3685817189 wait complete. 2024-03-28T17:49:55.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.430+0000 7fdb45016740 20 client.25636 trim_cache size 0 max 16384 2024-03-28T17:49:55.432 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:49:55.432 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-28T17:49:55.432 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "95b86461-a16a-4fda-a328-61a4ada868c6", 2024-03-28T17:49:55.432 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-28T17:49:55.432 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:49:55.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.431+0000 7fdb42200640 1 -- 172.21.15.45:0/3978263022 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fdb200a2a80 msgr2=0x7fdb200a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:55.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.431+0000 7fdb42200640 1 --2- 172.21.15.45:0/3978263022 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fdb200a2a80 0x7fdb200a4f40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c0cca80 tx=0x7fdb2c004a40 comp rx=0 tx=0).stop 2024-03-28T17:49:55.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.431+0000 7fdb42200640 1 -- 172.21.15.45:0/3978263022 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb3c103840 msgr2=0x7fdb3c19a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:55.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.431+0000 7fdb42200640 1 --2- 172.21.15.45:0/3978263022 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb3c103840 0x7fdb3c19a190 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c00e9f0 tx=0x7fdb2c00edf0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.431+0000 7fdb42200640 1 -- 172.21.15.45:0/3978263022 shutdown_connections 2024-03-28T17:49:55.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.431+0000 7fdb42200640 1 --2- 172.21.15.45:0/3978263022 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fdb200a2a80 0x7fdb200a4f40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.431+0000 7fdb42200640 1 --2- 172.21.15.45:0/3978263022 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdb3c109850 0x7fdb3c19ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.431+0000 7fdb42200640 1 --2- 172.21.15.45:0/3978263022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb3c104200 0x7fdb3c19a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.431+0000 7fdb42200640 1 --2- 172.21.15.45:0/3978263022 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb3c103840 0x7fdb3c19a190 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.431+0000 7fdb42200640 1 -- 172.21.15.45:0/3978263022 >> 172.21.15.45:0/3978263022 conn(0x7fdb3c0ff350 msgr2=0x7fdb3c0ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:55.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.431+0000 7fdb42200640 1 -- 172.21.15.45:0/3978263022 shutdown_connections 2024-03-28T17:49:55.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.431+0000 7fdb42200640 1 -- 172.21.15.45:0/3978263022 wait complete. 2024-03-28T17:49:55.453 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:49:55.454 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:49:55.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.637+0000 7f3cfbe00640 1 -- 172.21.15.45:0/4145194772 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3cfc102ff0 msgr2=0x7f3cfc10b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:55.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.637+0000 7f3cfbe00640 1 --2- 172.21.15.45:0/4145194772 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3cfc102ff0 0x7f3cfc10b4d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8009900 tx=0x7f3ce802bcd0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.637+0000 7f3cfbe00640 1 -- 172.21.15.45:0/4145194772 shutdown_connections 2024-03-28T17:49:55.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.637+0000 7f3cfbe00640 1 --2- 172.21.15.45:0/4145194772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cfc10baf0 0x7f3cfc10df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.637+0000 7f3cfbe00640 1 --2- 172.21.15.45:0/4145194772 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3cfc102ff0 0x7f3cfc10b4d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.637+0000 7f3cfbe00640 1 --2- 172.21.15.45:0/4145194772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cfc1026d0 0x7f3cfc102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.637+0000 7f3cfbe00640 1 -- 172.21.15.45:0/4145194772 >> 172.21.15.45:0/4145194772 conn(0x7f3cfc0fbef0 msgr2=0x7f3cfc0fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:55.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.637+0000 7f3cfbe00640 1 -- 172.21.15.45:0/4145194772 shutdown_connections 2024-03-28T17:49:55.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.637+0000 7f3cfbe00640 1 -- 172.21.15.45:0/4145194772 wait complete. 2024-03-28T17:49:55.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.638+0000 7f3cfbe00640 1 Processor -- start 2024-03-28T17:49:55.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.639+0000 7f3cfbe00640 1 -- start start 2024-03-28T17:49:55.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.639+0000 7f3cfbe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3cfc1026d0 0x7f3cfc100320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.639+0000 7f3cfbe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cfc102ff0 0x7f3cfc100860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.639+0000 7f3cfbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cfc10baf0 0x7f3cfc100da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.639+0000 7f3cfbe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc111900 con 0x7f3cfc10baf0 2024-03-28T17:49:55.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.639+0000 7f3cfbe00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc111780 con 0x7f3cfc102ff0 2024-03-28T17:49:55.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.639+0000 7f3cfbe00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc111a80 con 0x7f3cfc1026d0 2024-03-28T17:49:55.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.640+0000 7f3cfaa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3cfc1026d0 0x7f3cfc100320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.640+0000 7f3cfaa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3cfc1026d0 0x7f3cfc100320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:33750/0 (socket says 172.21.15.45:33750) 2024-03-28T17:49:55.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.640+0000 7f3cfaa00640 1 -- 172.21.15.45:0/4228373929 learned_addr learned my addr 172.21.15.45:0/4228373929 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:55.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.640+0000 7f3cfa000640 1 --2- 172.21.15.45:0/4228373929 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cfc102ff0 0x7f3cfc100860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.640+0000 7f3cfb400640 1 --2- 172.21.15.45:0/4228373929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cfc10baf0 0x7f3cfc100da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.640+0000 7f3cfaa00640 1 -- 172.21.15.45:0/4228373929 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cfc102ff0 msgr2=0x7f3cfc100860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:55.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.640+0000 7f3cfaa00640 1 --2- 172.21.15.45:0/4228373929 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cfc102ff0 0x7f3cfc100860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.640+0000 7f3cfaa00640 1 -- 172.21.15.45:0/4228373929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cfc10baf0 msgr2=0x7f3cfc100da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:55.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.640+0000 7f3cfaa00640 1 --2- 172.21.15.45:0/4228373929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cfc10baf0 0x7f3cfc100da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.640+0000 7f3cfaa00640 1 -- 172.21.15.45:0/4228373929 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc1a62e0 con 0x7f3cfc1026d0 2024-03-28T17:49:55.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.640+0000 7f3cfaa00640 1 --2- 172.21.15.45:0/4228373929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3cfc1026d0 0x7f3cfc100320 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3cec00e9f0 tx=0x7f3cec00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:55.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.641+0000 7f3d00a00640 1 -- 172.21.15.45:0/4228373929 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec00cd80 con 0x7f3cfc1026d0 2024-03-28T17:49:55.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.641+0000 7f3d00a00640 1 -- 172.21.15.45:0/4228373929 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3cec010910 con 0x7f3cfc1026d0 2024-03-28T17:49:55.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.641+0000 7f3d00a00640 1 -- 172.21.15.45:0/4228373929 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec018980 con 0x7f3cfc1026d0 2024-03-28T17:49:55.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.641+0000 7f3cfbe00640 1 -- 172.21.15.45:0/4228373929 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cfc1a65d0 con 0x7f3cfc1026d0 2024-03-28T17:49:55.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.641+0000 7f3cfbe00640 1 -- 172.21.15.45:0/4228373929 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cfc1a6aa0 con 0x7f3cfc1026d0 2024-03-28T17:49:55.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.645+0000 7f3d04a07740 1 Processor -- start 2024-03-28T17:49:55.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.645+0000 7f3d00a00640 1 -- 172.21.15.45:0/4228373929 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 26) v1 ==== 137466+0+0 (secure 0 0 0) 0x7f3cec010ac0 con 0x7f3cfc1026d0 2024-03-28T17:49:55.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.645+0000 7f3d04a07740 1 -- start start 2024-03-28T17:49:55.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.646+0000 7f3d04a07740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba9033dd0 0x563ba9032b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.646+0000 7f3d04a07740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ba90330f0 0x563ba9033570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.646+0000 7f3d04a07740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba9044c70 0x563ba9047060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.646+0000 7f3d04a07740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ba8f97250 con 0x563ba90330f0 2024-03-28T17:49:55.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.646+0000 7f3d04a07740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ba8f3fa30 con 0x563ba9044c70 2024-03-28T17:49:55.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.646+0000 7f3d04a07740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ba8f932f0 con 0x563ba9033dd0 2024-03-28T17:49:55.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.646+0000 7f3cfb400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba9033dd0 0x563ba9032b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.646+0000 7f3cfaa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ba90330f0 0x563ba9033570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.646+0000 7f3cfb400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba9033dd0 0x563ba9032b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:33758/0 (socket says 172.21.15.45:33758) 2024-03-28T17:49:55.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.646+0000 7f3cfb400640 1 -- 172.21.15.45:0/2900759815 learned_addr learned my addr 172.21.15.45:0/2900759815 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:55.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.646+0000 7f3cfa000640 1 --2- 172.21.15.45:0/2900759815 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba9044c70 0x563ba9047060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.647+0000 7f3cfb400640 1 -- 172.21.15.45:0/2900759815 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba9044c70 msgr2=0x563ba9047060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:55.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.647+0000 7f3cfb400640 1 --2- 172.21.15.45:0/2900759815 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba9044c70 0x563ba9047060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.647+0000 7f3cfb400640 1 -- 172.21.15.45:0/2900759815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ba90330f0 msgr2=0x563ba9033570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:55.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.647+0000 7f3cfb400640 1 --2- 172.21.15.45:0/2900759815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ba90330f0 0x563ba9033570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.647+0000 7f3cfb400640 1 -- 172.21.15.45:0/2900759815 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ba8f2a690 con 0x563ba9033dd0 2024-03-28T17:49:55.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.647+0000 7f3cfaa00640 1 --2- 172.21.15.45:0/2900759815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ba90330f0 0x563ba9033570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:49:55.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.647+0000 7f3d00a00640 1 --2- 172.21.15.45:0/4228373929 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f3ce00a2a80 0x7f3ce00a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.647+0000 7f3cfb400640 1 --2- 172.21.15.45:0/2900759815 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba9033dd0 0x563ba9032b20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4009f50 tx=0x7f3cf40118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77d7ab86011ee9e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:55.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.647+0000 7f3d00a00640 1 -- 172.21.15.45:0/4228373929 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f3cec0cf280 con 0x7f3cfc1026d0 2024-03-28T17:49:55.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.647+0000 7f3cfaa00640 1 --2- 172.21.15.45:0/4228373929 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f3ce00a2a80 0x7f3ce00a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.648+0000 7f3ce7e00640 1 -- 172.21.15.45:0/2900759815 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf4004b20 con 0x563ba9033dd0 2024-03-28T17:49:55.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.648+0000 7f3ce7e00640 1 -- 172.21.15.45:0/2900759815 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3cf4004cc0 con 0x563ba9033dd0 2024-03-28T17:49:55.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.648+0000 7f3ce7e00640 1 -- 172.21.15.45:0/2900759815 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf4005020 con 0x563ba9033dd0 2024-03-28T17:49:55.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.648+0000 7f3d04a07740 1 -- 172.21.15.45:0/2900759815 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba9033dd0 msgr2=0x563ba9032b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:55.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.648+0000 7f3cfaa00640 1 --2- 172.21.15.45:0/4228373929 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f3ce00a2a80 0x7f3ce00a4f40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3cec0cc980 tx=0x7f3cec0d2cc0 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:55.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.648+0000 7f3d04a07740 1 --2- 172.21.15.45:0/2900759815 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba9033dd0 0x563ba9032b20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4009f50 tx=0x7f3cf40118d0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.649+0000 7f3d04a07740 1 -- 172.21.15.45:0/2900759815 shutdown_connections 2024-03-28T17:49:55.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.649+0000 7f3d04a07740 1 --2- 172.21.15.45:0/2900759815 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba9044c70 0x563ba9047060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.649+0000 7f3d04a07740 1 --2- 172.21.15.45:0/2900759815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ba90330f0 0x563ba9033570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.649+0000 7f3d04a07740 1 --2- 172.21.15.45:0/2900759815 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba9033dd0 0x563ba9032b20 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.649+0000 7f3d04a07740 1 -- 172.21.15.45:0/2900759815 >> 172.21.15.45:0/2900759815 conn(0x563ba8ffa200 msgr2=0x563ba902db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:55.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.649+0000 7f3d04a07740 1 -- 172.21.15.45:0/2900759815 shutdown_connections 2024-03-28T17:49:55.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.649+0000 7f3d04a07740 1 -- 172.21.15.45:0/2900759815 wait complete. 2024-03-28T17:49:55.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.649+0000 7f3d04a07740 1 -- start start 2024-03-28T17:49:55.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.650+0000 7f3d04a07740 1 Processor -- start 2024-03-28T17:49:55.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.650+0000 7f3d04a07740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba90330f0 0x563ba9044390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.650+0000 7f3d04a07740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba9033dd0 0x563ba9041720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.650+0000 7f3d04a07740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ba9044c70 0x563ba9041c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.650+0000 7f3d04a07740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ba9047ae0 con 0x563ba9044c70 2024-03-28T17:49:55.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.650+0000 7f3d04a07740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ba8f52e80 con 0x563ba9033dd0 2024-03-28T17:49:55.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.650+0000 7f3d04a07740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ba9047c60 con 0x563ba90330f0 2024-03-28T17:49:55.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.650+0000 7f3d04a07740 10 client.0 _ll_register_callbacks cb 0x563ba8f65130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:55.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.651+0000 7f3cfa000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba9033dd0 0x563ba9041720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.651+0000 7f3cfb400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba90330f0 0x563ba9044390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.651+0000 7f3cfa000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba9033dd0 0x563ba9041720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:47244/0 (socket says 172.21.15.45:47244) 2024-03-28T17:49:55.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.651+0000 7f3cfa000640 1 -- 172.21.15.45:0/2913413782 learned_addr learned my addr 172.21.15.45:0/2913413782 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:55.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.651+0000 7f3cfb400640 1 --2- 172.21.15.45:0/2913413782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ba9044c70 0x563ba9041c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.651+0000 7f3cfa000640 1 -- 172.21.15.45:0/2913413782 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba90330f0 msgr2=0x563ba9044390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:55.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.651+0000 7f3cfa000640 1 --2- 172.21.15.45:0/2913413782 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba90330f0 0x563ba9044390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.651+0000 7f3cfa000640 1 -- 172.21.15.45:0/2913413782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ba9044c70 msgr2=0x563ba9041c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:55.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.651+0000 7f3cfa000640 1 --2- 172.21.15.45:0/2913413782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ba9044c70 0x563ba9041c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.651+0000 7f3cfa000640 1 -- 172.21.15.45:0/2913413782 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563ba8d757d0 con 0x563ba9033dd0 2024-03-28T17:49:55.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.651+0000 7f3cfb400640 1 --2- 172.21.15.45:0/2913413782 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba90330f0 0x563ba9044390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:49:55.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.651+0000 7f3cfa000640 1 --2- 172.21.15.45:0/2913413782 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba9033dd0 0x563ba9041720 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8009230 tx=0x7f3ce8005e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14917d85e54a317c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:55.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.651+0000 7f3ce4c00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:49:55.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.652+0000 7f3ce4c00640 1 -- 172.21.15.45:0/2913413782 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce8043070 con 0x563ba9033dd0 2024-03-28T17:49:55.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.652+0000 7f3d04a07740 1 -- 172.21.15.45:0/2913413782 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3cf4004cc0 con 0x563ba9033dd0 2024-03-28T17:49:55.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.652+0000 7f3ce4c00640 1 -- 172.21.15.45:0/2913413782 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f3ce802cb30 con 0x563ba9033dd0 2024-03-28T17:49:55.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.652+0000 7f3ce4c00640 1 -- 172.21.15.45:0/2913413782 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ce802cd30 con 0x563ba9033dd0 2024-03-28T17:49:55.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.652+0000 7f3d04a07740 10 client.25679 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:55.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.652+0000 7f3d04a07740 1 -- 172.21.15.45:0/2913413782 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563ba8cab3f0 con 0x563ba9033dd0 2024-03-28T17:49:55.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.652+0000 7f3ce4c00640 1 -- 172.21.15.45:0/2913413782 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3cf4004cc0 con 0x563ba9033dd0 2024-03-28T17:49:55.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.652+0000 7f3ce4c00640 1 -- 172.21.15.45:0/2913413782 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f3ce80035f0 con 0x563ba9033dd0 2024-03-28T17:49:55.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.652+0000 7f3d04a07740 10 client.25679 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:55.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.653+0000 7f3d04a07740 10 client.25679 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:49:55.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.653+0000 7f3d04a07740 20 client.25679 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:55.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.653+0000 7f3d04a07740 1 --2- 172.21.15.45:0/2913413782 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x563ba9054b50 0x563ba9056f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.653+0000 7f3d04a07740 4 client.25679 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:49:55.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.653+0000 7f3d04a07740 1 -- 172.21.15.45:0/2913413782 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563ba8fa4790 con 0x563ba9054b50 2024-03-28T17:49:55.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.653+0000 7f3cfb400640 1 --2- 172.21.15.45:0/2913413782 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x563ba9054b50 0x563ba9056f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.653+0000 7f3cfb400640 1 --2- 172.21.15.45:0/2913413782 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x563ba9054b50 0x563ba9056f40 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6315580c90736e6 server_cookie=2f67bfa9cb121911 in_seq=0 out_seq=0 2024-03-28T17:49:55.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.653+0000 7f3ce4c00640 10 client.25679 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:55.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.654+0000 7f3ce4c00640 1 -- 172.21.15.45:0/2913413782 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563ba8fa4790 con 0x563ba9054b50 2024-03-28T17:49:55.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.654+0000 7f3ce4c00640 10 client.25679 handle_command_reply: tid=0 2024-03-28T17:49:55.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.654+0000 7f3d04a07740 1 client.25679 shutdown 2024-03-28T17:49:55.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.655+0000 7f3d04a07740 1 -- 172.21.15.45:0/2913413782 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba9033dd0 msgr2=0x563ba9041720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:55.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.655+0000 7f3d04a07740 1 --2- 172.21.15.45:0/2913413782 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba9033dd0 0x563ba9041720 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8009230 tx=0x7f3ce8005e70 comp rx=0 tx=0).stop 2024-03-28T17:49:55.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.655+0000 7f3d04a07740 1 -- 172.21.15.45:0/2913413782 shutdown_connections 2024-03-28T17:49:55.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.655+0000 7f3d04a07740 1 --2- 172.21.15.45:0/2913413782 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x563ba9054b50 0x563ba9056f40 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.655+0000 7f3d04a07740 1 --2- 172.21.15.45:0/2913413782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ba9044c70 0x563ba9041c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.655+0000 7f3d04a07740 1 --2- 172.21.15.45:0/2913413782 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba9033dd0 0x563ba9041720 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.655+0000 7f3d04a07740 1 --2- 172.21.15.45:0/2913413782 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba90330f0 0x563ba9044390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.655+0000 7f3ce4c00640 0 client.25679 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:55.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.655+0000 7f3d04a07740 1 -- 172.21.15.45:0/2913413782 >> 172.21.15.45:0/2913413782 conn(0x563ba8ffa200 msgr2=0x563ba902f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:55.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.655+0000 7f3d04a07740 1 -- 172.21.15.45:0/2913413782 shutdown_connections 2024-03-28T17:49:55.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.655+0000 7f3d04a07740 1 -- 172.21.15.45:0/2913413782 wait complete. 2024-03-28T17:49:55.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.655+0000 7f3d04a07740 20 client.25679 trim_cache size 0 max 16384 2024-03-28T17:49:55.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.680+0000 7f3d04a07740 1 Processor -- start 2024-03-28T17:49:55.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.680+0000 7f3d04a07740 1 -- start start 2024-03-28T17:49:55.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.681+0000 7f3d04a07740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba90330f0 0x563ba9043e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.681+0000 7f3d04a07740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ba9033dd0 0x563ba9044390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.681+0000 7f3d04a07740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba9044c70 0x563ba9046060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.681+0000 7f3d04a07740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc1ad320 con 0x563ba9033dd0 2024-03-28T17:49:55.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.681+0000 7f3d04a07740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc1aae20 con 0x563ba90330f0 2024-03-28T17:49:55.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.681+0000 7f3d04a07740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc1a8ec0 con 0x563ba9044c70 2024-03-28T17:49:55.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.681+0000 7f3cfb400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba90330f0 0x563ba9043e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.681+0000 7f3cfa000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ba9033dd0 0x563ba9044390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.681+0000 7f3cfb400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba90330f0 0x563ba9043e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:47246/0 (socket says 172.21.15.45:47246) 2024-03-28T17:49:55.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.681+0000 7f3cfb400640 1 -- 172.21.15.45:0/362375701 learned_addr learned my addr 172.21.15.45:0/362375701 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:55.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.681+0000 7f3cfb400640 1 --2- 172.21.15.45:0/362375701 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba9044c70 0x563ba9046060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.682+0000 7f3cfb400640 1 -- 172.21.15.45:0/362375701 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba9044c70 msgr2=0x563ba9046060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:55.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.682+0000 7f3cfb400640 1 --2- 172.21.15.45:0/362375701 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba9044c70 0x563ba9046060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.682+0000 7f3cfb400640 1 -- 172.21.15.45:0/362375701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ba9033dd0 msgr2=0x563ba9044390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:55.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.682+0000 7f3cfb400640 1 --2- 172.21.15.45:0/362375701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ba9033dd0 0x563ba9044390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.682+0000 7f3cfb400640 1 -- 172.21.15.45:0/362375701 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ba8c6b0f0 con 0x563ba90330f0 2024-03-28T17:49:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.682+0000 7f3cfb400640 1 --2- 172.21.15.45:0/362375701 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba90330f0 0x563ba9043e50 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4009270 tx=0x7f3cf4009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c2d782f69a56ac9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.682+0000 7f3ce7400640 1 -- 172.21.15.45:0/362375701 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf402d2a0 con 0x563ba90330f0 2024-03-28T17:49:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.682+0000 7f3ce7400640 1 -- 172.21.15.45:0/362375701 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3cf402dd90 con 0x563ba90330f0 2024-03-28T17:49:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.682+0000 7f3ce7400640 1 -- 172.21.15.45:0/362375701 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf4003b10 con 0x563ba90330f0 2024-03-28T17:49:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.683+0000 7f3d04a07740 1 -- 172.21.15.45:0/362375701 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba90330f0 msgr2=0x563ba9043e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.683+0000 7f3d04a07740 1 --2- 172.21.15.45:0/362375701 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba90330f0 0x563ba9043e50 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4009270 tx=0x7f3cf4009500 comp rx=0 tx=0).stop 2024-03-28T17:49:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.683+0000 7f3d04a07740 1 -- 172.21.15.45:0/362375701 shutdown_connections 2024-03-28T17:49:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.683+0000 7f3d04a07740 1 --2- 172.21.15.45:0/362375701 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba9044c70 0x563ba9046060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.683+0000 7f3d04a07740 1 --2- 172.21.15.45:0/362375701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ba9033dd0 0x563ba9044390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.683+0000 7f3d04a07740 1 --2- 172.21.15.45:0/362375701 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba90330f0 0x563ba9043e50 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.683+0000 7f3d04a07740 1 -- 172.21.15.45:0/362375701 >> 172.21.15.45:0/362375701 conn(0x563ba8ffa200 msgr2=0x563ba9040fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.683+0000 7f3d04a07740 1 -- 172.21.15.45:0/362375701 shutdown_connections 2024-03-28T17:49:55.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.683+0000 7f3d04a07740 1 -- 172.21.15.45:0/362375701 wait complete. 2024-03-28T17:49:55.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.684+0000 7f3d04a07740 1 -- start start 2024-03-28T17:49:55.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.684+0000 7f3d04a07740 1 Processor -- start 2024-03-28T17:49:55.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.685+0000 7f3d04a07740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba90330f0 0x563ba903ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.685+0000 7f3d04a07740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba9033dd0 0x563ba903f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.685+0000 7f3d04a07740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ba9044c70 0x563ba904fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.685+0000 7f3d04a07740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc1a94c0 con 0x563ba9044c70 2024-03-28T17:49:55.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.685+0000 7f3d04a07740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc1aa7c0 con 0x563ba9033dd0 2024-03-28T17:49:55.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.685+0000 7f3d04a07740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc1a9b00 con 0x563ba90330f0 2024-03-28T17:49:55.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.685+0000 7f3d04a07740 10 client.0 _ll_register_callbacks cb 0x7f3cf40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:49:55.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.685+0000 7f3cfb400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba90330f0 0x563ba903ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.685+0000 7f3cfa000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba9033dd0 0x563ba903f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.685+0000 7f3cfa000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba9033dd0 0x563ba903f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:47256/0 (socket says 172.21.15.45:47256) 2024-03-28T17:49:55.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.685+0000 7f3cfa000640 1 -- 172.21.15.45:0/3193065176 learned_addr learned my addr 172.21.15.45:0/3193065176 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:49:55.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.685+0000 7f3cfb400640 1 --2- 172.21.15.45:0/3193065176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ba9044c70 0x563ba904fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.685+0000 7f3cfa000640 1 -- 172.21.15.45:0/3193065176 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba90330f0 msgr2=0x563ba903ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:55.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.685+0000 7f3cfa000640 1 --2- 172.21.15.45:0/3193065176 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba90330f0 0x563ba903ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.685+0000 7f3cfa000640 1 -- 172.21.15.45:0/3193065176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ba9044c70 msgr2=0x563ba904fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:49:55.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.685+0000 7f3cfa000640 1 --2- 172.21.15.45:0/3193065176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ba9044c70 0x563ba904fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.685+0000 7f3cfa000640 1 -- 172.21.15.45:0/3193065176 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563ba9050440 con 0x563ba9033dd0 2024-03-28T17:49:55.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.686+0000 7f3cfb400640 1 --2- 172.21.15.45:0/3193065176 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba90330f0 0x563ba903ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:49:55.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.686+0000 7f3cfa000640 1 --2- 172.21.15.45:0/3193065176 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba9033dd0 0x563ba903f2d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3ce802c2e0 tx=0x7f3ce8000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6580b2b032da7c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:49:55.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.686+0000 7f3ce4c00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:49:55.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.686+0000 7f3ce4c00640 1 -- 172.21.15.45:0/3193065176 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce8043040 con 0x563ba9033dd0 2024-03-28T17:49:55.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.686+0000 7f3ce4c00640 1 -- 172.21.15.45:0/3193065176 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f3ce80046c0 con 0x563ba9033dd0 2024-03-28T17:49:55.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.686+0000 7f3d04a07740 1 -- 172.21.15.45:0/3193065176 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3cf402dd90 con 0x563ba9033dd0 2024-03-28T17:49:55.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.686+0000 7f3ce4c00640 1 -- 172.21.15.45:0/3193065176 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ce8005850 con 0x563ba9033dd0 2024-03-28T17:49:55.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.686+0000 7f3d04a07740 10 client.25685 fetch_fsmap learned FSMap version 23 2024-03-28T17:49:55.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.686+0000 7f3d04a07740 1 -- 172.21.15.45:0/3193065176 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563ba90b0ee0 con 0x563ba9033dd0 2024-03-28T17:49:55.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.687+0000 7f3ce4c00640 1 -- 172.21.15.45:0/3193065176 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3cf402dd90 con 0x563ba9033dd0 2024-03-28T17:49:55.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.687+0000 7f3ce4c00640 1 -- 172.21.15.45:0/3193065176 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f3ce8062470 con 0x563ba9033dd0 2024-03-28T17:49:55.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.687+0000 7f3d04a07740 10 client.25685 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:49:55.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.687+0000 7f3d04a07740 10 client.25685 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:49:55.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.687+0000 7f3d04a07740 20 client.25685 populate_metadata read hostname 'smithi045' 2024-03-28T17:49:55.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.687+0000 7f3d04a07740 1 --2- 172.21.15.45:0/3193065176 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x563ba90b15e0 0x563ba90b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:49:55.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.687+0000 7f3d04a07740 4 client.25685 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:49:55.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.687+0000 7f3d04a07740 1 -- 172.21.15.45:0/3193065176 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563ba8f9eae0 con 0x563ba90b15e0 2024-03-28T17:49:55.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.687+0000 7f3cfb400640 1 --2- 172.21.15.45:0/3193065176 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x563ba90b15e0 0x563ba90b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:49:55.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.687+0000 7f3cfb400640 1 --2- 172.21.15.45:0/3193065176 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x563ba90b15e0 0x563ba90b3a00 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ab5a09aa9b269c server_cookie=6ac732084c304415 in_seq=0 out_seq=0 2024-03-28T17:49:55.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.687+0000 7f3ce4c00640 10 client.25685 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:55.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.688+0000 7f3ce4c00640 1 -- 172.21.15.45:0/3193065176 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x563ba8f9eae0 con 0x563ba90b15e0 2024-03-28T17:49:55.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.688+0000 7f3ce4c00640 10 client.25685 handle_command_reply: tid=0 2024-03-28T17:49:55.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.688+0000 7f3d04a07740 1 client.25685 shutdown 2024-03-28T17:49:55.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.688+0000 7f3d04a07740 1 -- 172.21.15.45:0/3193065176 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba9033dd0 msgr2=0x563ba903f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:55.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.688+0000 7f3d04a07740 1 --2- 172.21.15.45:0/3193065176 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba9033dd0 0x563ba903f2d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3ce802c2e0 tx=0x7f3ce8000bc0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.688+0000 7f3d04a07740 1 -- 172.21.15.45:0/3193065176 shutdown_connections 2024-03-28T17:49:55.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.688+0000 7f3d04a07740 1 --2- 172.21.15.45:0/3193065176 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x563ba90b15e0 0x563ba90b3a00 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.688+0000 7f3d04a07740 1 --2- 172.21.15.45:0/3193065176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ba9044c70 0x563ba904fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.688+0000 7f3d04a07740 1 --2- 172.21.15.45:0/3193065176 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ba9033dd0 0x563ba903f2d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.688+0000 7f3ce4c00640 0 client.25685 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:49:55.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.688+0000 7f3d04a07740 1 --2- 172.21.15.45:0/3193065176 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563ba90330f0 0x563ba903ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.688+0000 7f3d04a07740 1 -- 172.21.15.45:0/3193065176 >> 172.21.15.45:0/3193065176 conn(0x563ba8ffa200 msgr2=0x563ba902cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:55.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.688+0000 7f3d04a07740 1 -- 172.21.15.45:0/3193065176 shutdown_connections 2024-03-28T17:49:55.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.688+0000 7f3d04a07740 1 -- 172.21.15.45:0/3193065176 wait complete. 2024-03-28T17:49:55.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.688+0000 7f3d04a07740 20 client.25685 trim_cache size 0 max 16384 2024-03-28T17:49:55.692 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:49:55.692 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T17:49:55.692 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T17:49:55.692 INFO:teuthology.orchestra.run.smithi045.stdout: "95b86461-a16a-4fda-a328-61a4ada868c6": { 2024-03-28T17:49:55.692 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T17:49:55.692 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "95b86461-a16a-4fda-a328-61a4ada868c6", 2024-03-28T17:49:55.692 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T17:49:55.692 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:49:55.692 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:49:55.693 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:49:55.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.689+0000 7f3cfbe00640 1 -- 172.21.15.45:0/4228373929 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f3ce00a2a80 msgr2=0x7f3ce00a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:55.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.689+0000 7f3cfbe00640 1 --2- 172.21.15.45:0/4228373929 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f3ce00a2a80 0x7f3ce00a4f40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3cec0cc980 tx=0x7f3cec0d2cc0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.689+0000 7f3cfbe00640 1 -- 172.21.15.45:0/4228373929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3cfc1026d0 msgr2=0x7f3cfc100320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:49:55.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.689+0000 7f3cfbe00640 1 --2- 172.21.15.45:0/4228373929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3cfc1026d0 0x7f3cfc100320 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3cec00e9f0 tx=0x7f3cec00eeb0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.689+0000 7f3cfbe00640 1 -- 172.21.15.45:0/4228373929 shutdown_connections 2024-03-28T17:49:55.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.689+0000 7f3cfbe00640 1 --2- 172.21.15.45:0/4228373929 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f3ce00a2a80 0x7f3ce00a4f40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.689+0000 7f3cfbe00640 1 --2- 172.21.15.45:0/4228373929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cfc10baf0 0x7f3cfc100da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.689+0000 7f3cfbe00640 1 --2- 172.21.15.45:0/4228373929 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cfc102ff0 0x7f3cfc100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.690+0000 7f3cfbe00640 1 --2- 172.21.15.45:0/4228373929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3cfc1026d0 0x7f3cfc100320 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:49:55.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.690+0000 7f3cfbe00640 1 -- 172.21.15.45:0/4228373929 >> 172.21.15.45:0/4228373929 conn(0x7f3cfc0fbef0 msgr2=0x7f3cfc0fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:49:55.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.690+0000 7f3cfbe00640 1 -- 172.21.15.45:0/4228373929 shutdown_connections 2024-03-28T17:49:55.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:49:55.690+0000 7f3cfbe00640 1 -- 172.21.15.45:0/4228373929 wait complete. 2024-03-28T17:49:55.712 INFO:tasks.cephfs.filesystem:scrub status for tag:95b86461-a16a-4fda-a328-61a4ada868c6 - {'path': '/', 'tag': '95b86461-a16a-4fda-a328-61a4ada868c6', 'options': 'recursive,force'} 2024-03-28T17:49:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:56 smithi071 ceph-mon[36597]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:49:57.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:56 smithi071 ceph-mon[36597]: scrub queued for path: / 2024-03-28T17:49:57.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:56 smithi071 ceph-mon[36597]: scrub summary: idle+waiting paths [/] 2024-03-28T17:49:57.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:56 smithi071 ceph-mon[36597]: scrub summary: active paths [/] 2024-03-28T17:49:57.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:56 smithi071 ceph-mon[36597]: scrub summary: idle+waiting paths [/] 2024-03-28T17:49:57.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:56 smithi071 ceph-mon[36597]: scrub summary: idle 2024-03-28T17:49:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:56 smithi151 ceph-mon[36481]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:49:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:56 smithi151 ceph-mon[36481]: scrub queued for path: / 2024-03-28T17:49:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:56 smithi151 ceph-mon[36481]: scrub summary: idle+waiting paths [/] 2024-03-28T17:49:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:56 smithi151 ceph-mon[36481]: scrub summary: active paths [/] 2024-03-28T17:49:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:56 smithi151 ceph-mon[36481]: scrub summary: idle+waiting paths [/] 2024-03-28T17:49:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:56 smithi151 ceph-mon[36481]: scrub summary: idle 2024-03-28T17:49:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:56 smithi045 ceph-mon[31773]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T17:49:57.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:56 smithi045 ceph-mon[31773]: scrub queued for path: / 2024-03-28T17:49:57.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:56 smithi045 ceph-mon[31773]: scrub summary: idle+waiting paths [/] 2024-03-28T17:49:57.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:56 smithi045 ceph-mon[31773]: scrub summary: active paths [/] 2024-03-28T17:49:57.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:56 smithi045 ceph-mon[31773]: scrub summary: idle+waiting paths [/] 2024-03-28T17:49:57.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:56 smithi045 ceph-mon[31773]: scrub summary: idle 2024-03-28T17:49:59.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:49:58 smithi071 ceph-mon[36597]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T17:49:59.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:49:58 smithi151 ceph-mon[36481]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T17:49:59.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:49:58 smithi045 ceph-mon[31773]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T17:50:00.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:00 smithi045 ceph-mon[31773]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:50:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:00 smithi045 ceph-mon[31773]: overall HEALTH_OK 2024-03-28T17:50:00.916 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:50:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:50:00.564+0000 7f52d7a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:50:00.916 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:50:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:50:00.564+0000 7f52d7a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:50:00.916 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:50:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:50:00.564+0000 7f52d7a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:50:00.916 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:50:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:50:00.564+0000 7f52d7a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:50:00.916 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 17:50:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T17:50:00.564+0000 7f52d7a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T17:50:01.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:00 smithi071 ceph-mon[36597]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:50:01.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:00 smithi071 ceph-mon[36597]: overall HEALTH_OK 2024-03-28T17:50:01.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:00 smithi151 ceph-mon[36481]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:50:01.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:00 smithi151 ceph-mon[36481]: overall HEALTH_OK 2024-03-28T17:50:03.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:02 smithi071 ceph-mon[36597]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T17:50:03.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:02 smithi071 ceph-mon[36597]: mgrmap e27: x(active, since 62s), standbys: z, y 2024-03-28T17:50:03.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:02 smithi151 ceph-mon[36481]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T17:50:03.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:02 smithi151 ceph-mon[36481]: mgrmap e27: x(active, since 62s), standbys: z, y 2024-03-28T17:50:03.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:02 smithi045 ceph-mon[31773]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T17:50:03.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:02 smithi045 ceph-mon[31773]: mgrmap e27: x(active, since 62s), standbys: z, y 2024-03-28T17:50:05.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:04 smithi071 ceph-mon[36597]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T17:50:05.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:04 smithi151 ceph-mon[36481]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T17:50:05.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:04 smithi045 ceph-mon[31773]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T17:50:07.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:06 smithi071 ceph-mon[36597]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:50:07.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:06 smithi151 ceph-mon[36481]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:50:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:06 smithi045 ceph-mon[31773]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:50:09.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:08 smithi071 ceph-mon[36597]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:50:09.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:08 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:50:09.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:08 smithi151 ceph-mon[36481]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:50:09.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:08 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:50:09.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:08 smithi045 ceph-mon[31773]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:50:09.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:08 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:50:10.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:09 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:50:10.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:09 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:50:10.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:09 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:50:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:09 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:50:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:09 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:50:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:09 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:50:10.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:09 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:50:10.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:09 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:50:10.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:09 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:50:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:10 smithi071 ceph-mon[36597]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-28T17:50:11.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:10 smithi151 ceph-mon[36481]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-28T17:50:11.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:10 smithi045 ceph-mon[31773]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-28T17:50:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:12 smithi071 ceph-mon[36597]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T17:50:13.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:12 smithi151 ceph-mon[36481]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T17:50:13.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:12 smithi045 ceph-mon[31773]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T17:50:14.968 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:14 smithi045 ceph-mon[31773]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-28T17:50:15.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:14 smithi071 ceph-mon[36597]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-28T17:50:15.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:14 smithi151 ceph-mon[36481]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-28T17:50:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:16 smithi071 ceph-mon[36597]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:50:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:16 smithi151 ceph-mon[36481]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:50:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:16 smithi045 ceph-mon[31773]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:50:19.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:18 smithi071 ceph-mon[36597]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T17:50:19.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:18 smithi151 ceph-mon[36481]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T17:50:19.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:18 smithi045 ceph-mon[31773]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T17:50:21.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:20 smithi071 ceph-mon[36597]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T17:50:21.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:20 smithi151 ceph-mon[36481]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T17:50:21.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:20 smithi045 ceph-mon[31773]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T17:50:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:22 smithi071 ceph-mon[36597]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-28T17:50:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:22 smithi151 ceph-mon[36481]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-28T17:50:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:22 smithi045 ceph-mon[31773]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-28T17:50:25.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:24 smithi071 ceph-mon[36597]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:50:25.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:24 smithi151 ceph-mon[36481]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:50:25.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:24 smithi045 ceph-mon[31773]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:50:25.612 INFO:tasks.workunit.client.0.smithi045.stderr:Updating files: 59% (7298/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 98% (12128/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-28T17:50:25.713 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:50:25.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:50:25.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.938+0000 7f3c04c00640 1 -- 172.21.15.45:0/3900797520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c00100e40 msgr2=0x7f3c0010db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:25.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.938+0000 7f3c04c00640 1 --2- 172.21.15.45:0/3900797520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c00100e40 0x7f3c0010db00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3be8009900 tx=0x7f3be802bc40 comp rx=0 tx=0).stop 2024-03-28T17:50:25.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.938+0000 7f3c04c00640 1 -- 172.21.15.45:0/3900797520 shutdown_connections 2024-03-28T17:50:25.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.938+0000 7f3c04c00640 1 --2- 172.21.15.45:0/3900797520 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c0010e040 0x7f3c00110430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.938+0000 7f3c04c00640 1 --2- 172.21.15.45:0/3900797520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c00100e40 0x7f3c0010db00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.938+0000 7f3c04c00640 1 --2- 172.21.15.45:0/3900797520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c00100490 0x7f3c00100870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.938+0000 7f3c04c00640 1 -- 172.21.15.45:0/3900797520 >> 172.21.15.45:0/3900797520 conn(0x7f3c00077700 msgr2=0x7f3c00077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:25.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.938+0000 7f3c04c00640 1 -- 172.21.15.45:0/3900797520 shutdown_connections 2024-03-28T17:50:25.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.939+0000 7f3c04c00640 1 -- 172.21.15.45:0/3900797520 wait complete. 2024-03-28T17:50:25.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.939+0000 7f3c04c00640 1 Processor -- start 2024-03-28T17:50:25.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.940+0000 7f3c04c00640 1 -- start start 2024-03-28T17:50:25.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.940+0000 7f3c04c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c00100490 0x7f3c0019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:25.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.940+0000 7f3c04c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c00100e40 0x7f3c0019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:25.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.940+0000 7f3c04c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0010e040 0x7f3c0019ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:25.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.940+0000 7f3c04c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c00113d30 con 0x7f3c0010e040 2024-03-28T17:50:25.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.940+0000 7f3c04c00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c00113bb0 con 0x7f3c00100490 2024-03-28T17:50:25.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.940+0000 7f3c04c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c00113eb0 con 0x7f3c00100e40 2024-03-28T17:50:25.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.940+0000 7f3bfde00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c00100490 0x7f3c0019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:25.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.940+0000 7f3bfde00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c00100490 0x7f3c0019a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:33212/0 (socket says 172.21.15.45:33212) 2024-03-28T17:50:25.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.940+0000 7f3bfde00640 1 -- 172.21.15.45:0/3495994541 learned_addr learned my addr 172.21.15.45:0/3495994541 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:25.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.940+0000 7f3bf5400640 1 --2- 172.21.15.45:0/3495994541 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c00100e40 0x7f3c0019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:25.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.940+0000 7f3bfe800640 1 --2- 172.21.15.45:0/3495994541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0010e040 0x7f3c0019ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:25.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.941+0000 7f3bfde00640 1 -- 172.21.15.45:0/3495994541 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c00100e40 msgr2=0x7f3c0019a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:25.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.941+0000 7f3bfde00640 1 --2- 172.21.15.45:0/3495994541 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c00100e40 0x7f3c0019a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.941+0000 7f3bfde00640 1 -- 172.21.15.45:0/3495994541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0010e040 msgr2=0x7f3c0019ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:25.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.941+0000 7f3bfde00640 1 --2- 172.21.15.45:0/3495994541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0010e040 0x7f3c0019ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.941+0000 7f3bfde00640 1 -- 172.21.15.45:0/3495994541 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0019f1c0 con 0x7f3c00100490 2024-03-28T17:50:25.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.941+0000 7f3bfe800640 1 --2- 172.21.15.45:0/3495994541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0010e040 0x7f3c0019ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:50:25.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.941+0000 7f3bfde00640 1 --2- 172.21.15.45:0/3495994541 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c00100490 0x7f3c0019a1d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000eab0 tx=0x7f3bf000ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:25.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.941+0000 7f3bf7400640 1 -- 172.21.15.45:0/3495994541 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf000ce40 con 0x7f3c00100490 2024-03-28T17:50:25.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.941+0000 7f3bf7400640 1 -- 172.21.15.45:0/3495994541 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bf0011950 con 0x7f3c00100490 2024-03-28T17:50:25.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.941+0000 7f3c04c00640 1 -- 172.21.15.45:0/3495994541 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c0019f4b0 con 0x7f3c00100490 2024-03-28T17:50:25.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.941+0000 7f3bf7400640 1 -- 172.21.15.45:0/3495994541 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0010ae0 con 0x7f3c00100490 2024-03-28T17:50:25.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.941+0000 7f3c04c00640 1 -- 172.21.15.45:0/3495994541 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c001a6d40 con 0x7f3c00100490 2024-03-28T17:50:25.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.944+0000 7f3bf7400640 1 -- 172.21.15.45:0/3495994541 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 27) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f3bf0011b00 con 0x7f3c00100490 2024-03-28T17:50:25.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.944+0000 7f3c07a14740 1 Processor -- start 2024-03-28T17:50:25.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.945+0000 7f3c07a14740 1 -- start start 2024-03-28T17:50:25.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.945+0000 7f3c07a14740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56156dc7fc00 0x56156dc7e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:25.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.945+0000 7f3c07a14740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc7ef20 0x56156dc7f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:25.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.945+0000 7f3c07a14740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56156dc88290 0x56156dc8a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:25.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.945+0000 7f3c07a14740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56156dbe3250 con 0x56156dc7fc00 2024-03-28T17:50:25.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.945+0000 7f3c07a14740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x56156db8ba30 con 0x56156dc7ef20 2024-03-28T17:50:25.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.945+0000 7f3c07a14740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x56156dbdf2f0 con 0x56156dc88290 2024-03-28T17:50:25.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.945+0000 7f3bfde00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc7ef20 0x56156dc7f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:25.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.945+0000 7f3bfe800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56156dc7fc00 0x56156dc7e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:25.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.945+0000 7f3bf5400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56156dc88290 0x56156dc8a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:25.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.945+0000 7f3bfde00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc7ef20 0x56156dc7f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:33224/0 (socket says 172.21.15.45:33224) 2024-03-28T17:50:25.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.945+0000 7f3bfde00640 1 -- 172.21.15.45:0/1418251120 learned_addr learned my addr 172.21.15.45:0/1418251120 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:25.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.946+0000 7f3bfde00640 1 -- 172.21.15.45:0/1418251120 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56156dc88290 msgr2=0x56156dc8a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:25.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.946+0000 7f3bfde00640 1 --2- 172.21.15.45:0/1418251120 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56156dc88290 0x56156dc8a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.946+0000 7f3bfde00640 1 -- 172.21.15.45:0/1418251120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56156dc7fc00 msgr2=0x56156dc7e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:25.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.946+0000 7f3bfde00640 1 --2- 172.21.15.45:0/1418251120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56156dc7fc00 0x56156dc7e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.946+0000 7f3bfde00640 1 -- 172.21.15.45:0/1418251120 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56156db76690 con 0x56156dc7ef20 2024-03-28T17:50:25.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.946+0000 7f3bf5400640 1 --2- 172.21.15.45:0/1418251120 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56156dc88290 0x56156dc8a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:50:25.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.946+0000 7f3bf7400640 1 --2- 172.21.15.45:0/3495994541 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f3be40a2a90 0x7f3be40a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:25.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.946+0000 7f3bfde00640 1 --2- 172.21.15.45:0/1418251120 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc7ef20 0x56156dc7f3a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00ccd80 tx=0x7f3bf00d5cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6b372796376b057 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:25.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.946+0000 7f3bf7400640 1 -- 172.21.15.45:0/3495994541 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f3bf00cf840 con 0x7f3c00100490 2024-03-28T17:50:25.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.946+0000 7f3bfe800640 1 --2- 172.21.15.45:0/3495994541 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f3be40a2a90 0x7f3be40a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:25.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.946+0000 7f3bef000640 1 -- 172.21.15.45:0/1418251120 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0097ec0 con 0x56156dc7ef20 2024-03-28T17:50:25.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.946+0000 7f3bef000640 1 -- 172.21.15.45:0/1418251120 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bf009c320 con 0x56156dc7ef20 2024-03-28T17:50:25.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.946+0000 7f3bef000640 1 -- 172.21.15.45:0/1418251120 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf009b580 con 0x56156dc7ef20 2024-03-28T17:50:25.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.947+0000 7f3bfe800640 1 --2- 172.21.15.45:0/3495994541 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f3be40a2a90 0x7f3be40a4f50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3c0006ad50 tx=0x7f3bf800b4d0 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:25.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.947+0000 7f3c07a14740 1 -- 172.21.15.45:0/1418251120 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc7ef20 msgr2=0x56156dc7f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:25.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.947+0000 7f3c07a14740 1 --2- 172.21.15.45:0/1418251120 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc7ef20 0x56156dc7f3a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00ccd80 tx=0x7f3bf00d5cd0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.947+0000 7f3c07a14740 1 -- 172.21.15.45:0/1418251120 shutdown_connections 2024-03-28T17:50:25.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.947+0000 7f3c07a14740 1 --2- 172.21.15.45:0/1418251120 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56156dc88290 0x56156dc8a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.947+0000 7f3c07a14740 1 --2- 172.21.15.45:0/1418251120 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc7ef20 0x56156dc7f3a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.947+0000 7f3c07a14740 1 --2- 172.21.15.45:0/1418251120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56156dc7fc00 0x56156dc7e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.947+0000 7f3c07a14740 1 -- 172.21.15.45:0/1418251120 >> 172.21.15.45:0/1418251120 conn(0x56156dc46200 msgr2=0x56156dc79970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:25.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.947+0000 7f3c07a14740 1 -- 172.21.15.45:0/1418251120 shutdown_connections 2024-03-28T17:50:25.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.947+0000 7f3c07a14740 1 -- 172.21.15.45:0/1418251120 wait complete. 2024-03-28T17:50:25.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.948+0000 7f3c07a14740 1 -- start start 2024-03-28T17:50:25.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.948+0000 7f3c07a14740 1 Processor -- start 2024-03-28T17:50:25.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.948+0000 7f3c07a14740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc7ef20 0x56156dc87940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:25.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.948+0000 7f3c07a14740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56156dc7fc00 0x56156dc84cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:25.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.948+0000 7f3c07a14740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56156dc88290 0x56156dc85210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:25.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.948+0000 7f3c07a14740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56156dc8b100 con 0x56156dc7fc00 2024-03-28T17:50:25.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.948+0000 7f3c07a14740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x56156db9ee80 con 0x56156dc7ef20 2024-03-28T17:50:25.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.948+0000 7f3c07a14740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x56156dc8b280 con 0x56156dc88290 2024-03-28T17:50:25.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.949+0000 7f3c07a14740 10 client.0 _ll_register_callbacks cb 0x56156dbb1130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:50:25.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.949+0000 7f3bfde00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc7ef20 0x56156dc87940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:25.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.949+0000 7f3bfde00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc7ef20 0x56156dc87940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:33232/0 (socket says 172.21.15.45:33232) 2024-03-28T17:50:25.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.949+0000 7f3bfe800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56156dc88290 0x56156dc85210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:25.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.949+0000 7f3bfde00640 1 -- 172.21.15.45:0/4214461302 learned_addr learned my addr 172.21.15.45:0/4214461302 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:25.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.949+0000 7f3bf5400640 1 --2- 172.21.15.45:0/4214461302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56156dc7fc00 0x56156dc84cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:25.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.949+0000 7f3bfde00640 1 -- 172.21.15.45:0/4214461302 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56156dc88290 msgr2=0x56156dc85210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:25.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.949+0000 7f3bfde00640 1 --2- 172.21.15.45:0/4214461302 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56156dc88290 0x56156dc85210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.949+0000 7f3bfde00640 1 -- 172.21.15.45:0/4214461302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56156dc7fc00 msgr2=0x56156dc84cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:25.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.949+0000 7f3bfde00640 1 --2- 172.21.15.45:0/4214461302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56156dc7fc00 0x56156dc84cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.949+0000 7f3bfde00640 1 -- 172.21.15.45:0/4214461302 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56156d9c17d0 con 0x56156dc7ef20 2024-03-28T17:50:25.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.949+0000 7f3bfde00640 1 --2- 172.21.15.45:0/4214461302 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc7ef20 0x56156dc87940 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3bf009b920 tx=0x7f3bf00cfac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3faaec40de4e5265 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:25.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.949+0000 7f3be3400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:50:25.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.950+0000 7f3be3400640 1 -- 172.21.15.45:0/4214461302 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf009b760 con 0x56156dc7ef20 2024-03-28T17:50:25.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.950+0000 7f3be3400640 1 -- 172.21.15.45:0/4214461302 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f3bf009be80 con 0x56156dc7ef20 2024-03-28T17:50:25.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.950+0000 7f3c07a14740 1 -- 172.21.15.45:0/4214461302 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3bf009c320 con 0x56156dc7ef20 2024-03-28T17:50:25.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.950+0000 7f3be3400640 1 -- 172.21.15.45:0/4214461302 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3bf009b9f0 con 0x56156dc7ef20 2024-03-28T17:50:25.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.950+0000 7f3c07a14740 10 client.25703 fetch_fsmap learned FSMap version 23 2024-03-28T17:50:25.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.950+0000 7f3c07a14740 1 -- 172.21.15.45:0/4214461302 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56156d8f73f0 con 0x56156dc7ef20 2024-03-28T17:50:25.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.950+0000 7f3be3400640 1 -- 172.21.15.45:0/4214461302 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bf009c320 con 0x56156dc7ef20 2024-03-28T17:50:25.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.950+0000 7f3be3400640 1 -- 172.21.15.45:0/4214461302 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f3bf00b4ce0 con 0x56156dc7ef20 2024-03-28T17:50:25.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.950+0000 7f3c07a14740 10 client.25703 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:50:25.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.950+0000 7f3c07a14740 10 client.25703 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:50:25.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.950+0000 7f3c07a14740 20 client.25703 populate_metadata read hostname 'smithi045' 2024-03-28T17:50:25.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.950+0000 7f3c07a14740 1 --2- 172.21.15.45:0/4214461302 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56156dc98060 0x56156dc9a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:25.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.950+0000 7f3c07a14740 4 client.25703 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:50:25.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.950+0000 7f3c07a14740 1 -- 172.21.15.45:0/4214461302 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56156dbf0790 con 0x56156dc98060 2024-03-28T17:50:25.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.951+0000 7f3bf5400640 1 --2- 172.21.15.45:0/4214461302 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56156dc98060 0x56156dc9a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:25.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.951+0000 7f3bf5400640 1 --2- 172.21.15.45:0/4214461302 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56156dc98060 0x56156dc9a450 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9a75aa0cdc467c2 server_cookie=3baf6215ffe45ce1 in_seq=0 out_seq=0 2024-03-28T17:50:25.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.951+0000 7f3be3400640 10 client.25703 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:50:25.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.952+0000 7f3be3400640 1 -- 172.21.15.45:0/4214461302 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56156dbf0790 con 0x56156dc98060 2024-03-28T17:50:25.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.952+0000 7f3be3400640 10 client.25703 handle_command_reply: tid=0 2024-03-28T17:50:25.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.952+0000 7f3c07a14740 1 client.25703 shutdown 2024-03-28T17:50:25.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.952+0000 7f3c07a14740 1 -- 172.21.15.45:0/4214461302 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc7ef20 msgr2=0x56156dc87940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:25.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.952+0000 7f3c07a14740 1 --2- 172.21.15.45:0/4214461302 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc7ef20 0x56156dc87940 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3bf009b920 tx=0x7f3bf00cfac0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.952+0000 7f3c07a14740 1 -- 172.21.15.45:0/4214461302 shutdown_connections 2024-03-28T17:50:25.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.952+0000 7f3c07a14740 1 --2- 172.21.15.45:0/4214461302 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56156dc98060 0x56156dc9a450 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.952+0000 7f3c07a14740 1 --2- 172.21.15.45:0/4214461302 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56156dc88290 0x56156dc85210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.952+0000 7f3c07a14740 1 --2- 172.21.15.45:0/4214461302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56156dc7fc00 0x56156dc84cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.952+0000 7f3c07a14740 1 --2- 172.21.15.45:0/4214461302 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc7ef20 0x56156dc87940 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.952+0000 7f3c07a14740 1 -- 172.21.15.45:0/4214461302 >> 172.21.15.45:0/4214461302 conn(0x56156dc46200 msgr2=0x56156dc7b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:25.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.952+0000 7f3be3400640 0 client.25703 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:50:25.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.952+0000 7f3c07a14740 1 -- 172.21.15.45:0/4214461302 shutdown_connections 2024-03-28T17:50:25.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.952+0000 7f3c07a14740 1 -- 172.21.15.45:0/4214461302 wait complete. 2024-03-28T17:50:25.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.952+0000 7f3c07a14740 20 client.25703 trim_cache size 0 max 16384 2024-03-28T17:50:25.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.970+0000 7f3c07a14740 1 Processor -- start 2024-03-28T17:50:25.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.970+0000 7f3c07a14740 1 -- start start 2024-03-28T17:50:25.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.970+0000 7f3c07a14740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56156dc7ef20 0x56156dc94580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:25.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.970+0000 7f3c07a14740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56156dc7fc00 0x56156dc89680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:25.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.970+0000 7f3c07a14740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc88290 0x56156dc89bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:25.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.970+0000 7f3c07a14740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c001b19c0 con 0x56156dc7fc00 2024-03-28T17:50:25.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.970+0000 7f3c07a14740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c001af4c0 con 0x56156dc88290 2024-03-28T17:50:25.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.970+0000 7f3c07a14740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c001ad540 con 0x56156dc7ef20 2024-03-28T17:50:25.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.970+0000 7f3bfde00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56156dc7ef20 0x56156dc94580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:25.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.970+0000 7f3bf5400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56156dc7fc00 0x56156dc89680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:25.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.970+0000 7f3bfe800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc88290 0x56156dc89bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:25.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.970+0000 7f3bfde00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56156dc7ef20 0x56156dc94580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:46780/0 (socket says 172.21.15.45:46780) 2024-03-28T17:50:25.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.970+0000 7f3bfde00640 1 -- 172.21.15.45:0/4195727662 learned_addr learned my addr 172.21.15.45:0/4195727662 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:25.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.971+0000 7f3bfde00640 1 -- 172.21.15.45:0/4195727662 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc88290 msgr2=0x56156dc89bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:25.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.971+0000 7f3bfde00640 1 --2- 172.21.15.45:0/4195727662 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc88290 0x56156dc89bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.971+0000 7f3bfde00640 1 -- 172.21.15.45:0/4195727662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56156dc7fc00 msgr2=0x56156dc89680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:25.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.971+0000 7f3bfde00640 1 --2- 172.21.15.45:0/4195727662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56156dc7fc00 0x56156dc89680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.971+0000 7f3bfde00640 1 -- 172.21.15.45:0/4195727662 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56156d8b70f0 con 0x56156dc7ef20 2024-03-28T17:50:25.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.971+0000 7f3bfde00640 1 --2- 172.21.15.45:0/4195727662 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56156dc7ef20 0x56156dc94580 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00985a0 tx=0x7f3bf00d5cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a35940f7e4324b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:25.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.971+0000 7f3bee600640 1 -- 172.21.15.45:0/4195727662 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf009b760 con 0x56156dc7ef20 2024-03-28T17:50:25.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.971+0000 7f3bee600640 1 -- 172.21.15.45:0/4195727662 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bf00ccd20 con 0x56156dc7ef20 2024-03-28T17:50:25.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.971+0000 7f3bee600640 1 -- 172.21.15.45:0/4195727662 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf009b580 con 0x56156dc7ef20 2024-03-28T17:50:25.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.972+0000 7f3c07a14740 1 -- 172.21.15.45:0/4195727662 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56156dc7ef20 msgr2=0x56156dc94580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:25.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.972+0000 7f3c07a14740 1 --2- 172.21.15.45:0/4195727662 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56156dc7ef20 0x56156dc94580 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00985a0 tx=0x7f3bf00d5cd0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.972+0000 7f3c07a14740 1 -- 172.21.15.45:0/4195727662 shutdown_connections 2024-03-28T17:50:25.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.972+0000 7f3c07a14740 1 --2- 172.21.15.45:0/4195727662 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc88290 0x56156dc89bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.972+0000 7f3c07a14740 1 --2- 172.21.15.45:0/4195727662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56156dc7fc00 0x56156dc89680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.972+0000 7f3c07a14740 1 --2- 172.21.15.45:0/4195727662 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56156dc7ef20 0x56156dc94580 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.972+0000 7f3c07a14740 1 -- 172.21.15.45:0/4195727662 >> 172.21.15.45:0/4195727662 conn(0x56156dc46200 msgr2=0x56156dc794a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:25.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.972+0000 7f3c07a14740 1 -- 172.21.15.45:0/4195727662 shutdown_connections 2024-03-28T17:50:25.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.972+0000 7f3c07a14740 1 -- 172.21.15.45:0/4195727662 wait complete. 2024-03-28T17:50:25.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.972+0000 7f3c07a14740 1 -- start start 2024-03-28T17:50:25.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.973+0000 7f3c07a14740 1 Processor -- start 2024-03-28T17:50:25.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.973+0000 7f3c07a14740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc7ef20 0x56156dc98f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:25.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.973+0000 7f3c07a14740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56156dc7fc00 0x56156dc99450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:25.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.973+0000 7f3c07a14740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56156dc88290 0x56156dcf2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:25.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.973+0000 7f3c07a14740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c001adb40 con 0x56156dc7fc00 2024-03-28T17:50:25.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.973+0000 7f3c07a14740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c001aee60 con 0x56156dc7ef20 2024-03-28T17:50:25.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.973+0000 7f3c07a14740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c001ae1a0 con 0x56156dc88290 2024-03-28T17:50:25.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.973+0000 7f3c07a14740 10 client.0 _ll_register_callbacks cb 0x7f3bf009cf50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:50:25.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.973+0000 7f3bfde00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc7ef20 0x56156dc98f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:25.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.973+0000 7f3bfde00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc7ef20 0x56156dc98f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:33264/0 (socket says 172.21.15.45:33264) 2024-03-28T17:50:25.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.973+0000 7f3bfde00640 1 -- 172.21.15.45:0/3481731358 learned_addr learned my addr 172.21.15.45:0/3481731358 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:25.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.973+0000 7f3bfe800640 1 --2- 172.21.15.45:0/3481731358 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56156dc88290 0x56156dcf2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:25.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.973+0000 7f3bf5400640 1 --2- 172.21.15.45:0/3481731358 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56156dc7fc00 0x56156dc99450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:25.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.974+0000 7f3bfde00640 1 -- 172.21.15.45:0/3481731358 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56156dc88290 msgr2=0x56156dcf2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:25.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.974+0000 7f3bfde00640 1 --2- 172.21.15.45:0/3481731358 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56156dc88290 0x56156dcf2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.974+0000 7f3bfde00640 1 -- 172.21.15.45:0/3481731358 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56156dc7fc00 msgr2=0x56156dc99450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:25.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.974+0000 7f3bfde00640 1 --2- 172.21.15.45:0/3481731358 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56156dc7fc00 0x56156dc99450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.974+0000 7f3bfde00640 1 -- 172.21.15.45:0/3481731358 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56156dcf2ac0 con 0x56156dc7ef20 2024-03-28T17:50:25.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.974+0000 7f3bf5400640 1 --2- 172.21.15.45:0/3481731358 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56156dc7fc00 0x56156dc99450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:50:25.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.974+0000 7f3bfde00640 1 --2- 172.21.15.45:0/3481731358 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc7ef20 0x56156dc98f10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0099ed0 tx=0x7f3bf009c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4829d75145480108 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:25.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.974+0000 7f3be3400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:50:25.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.974+0000 7f3be3400640 1 -- 172.21.15.45:0/3481731358 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf007f060 con 0x56156dc7ef20 2024-03-28T17:50:25.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.974+0000 7f3be3400640 1 -- 172.21.15.45:0/3481731358 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f3bf00a3f30 con 0x56156dc7ef20 2024-03-28T17:50:25.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.974+0000 7f3c07a14740 1 -- 172.21.15.45:0/3481731358 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3bf00ccd20 con 0x56156dc7ef20 2024-03-28T17:50:25.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.974+0000 7f3be3400640 1 -- 172.21.15.45:0/3481731358 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3bf0097ec0 con 0x56156dc7ef20 2024-03-28T17:50:25.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.974+0000 7f3c07a14740 10 client.25709 fetch_fsmap learned FSMap version 23 2024-03-28T17:50:25.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.974+0000 7f3c07a14740 1 -- 172.21.15.45:0/3481731358 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56156dcf2ac0 con 0x56156dc7ef20 2024-03-28T17:50:25.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.975+0000 7f3be3400640 1 -- 172.21.15.45:0/3481731358 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bf00ccd20 con 0x56156dc7ef20 2024-03-28T17:50:25.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.975+0000 7f3be3400640 1 -- 172.21.15.45:0/3481731358 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f3bf00b4ce0 con 0x56156dc7ef20 2024-03-28T17:50:25.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.975+0000 7f3c07a14740 10 client.25709 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:50:25.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.975+0000 7f3c07a14740 10 client.25709 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:50:25.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.975+0000 7f3c07a14740 20 client.25709 populate_metadata read hostname 'smithi045' 2024-03-28T17:50:25.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.975+0000 7f3c07a14740 1 --2- 172.21.15.45:0/3481731358 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56156dcf3f90 0x56156dcf6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:25.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.975+0000 7f3c07a14740 4 client.25709 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:50:25.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.975+0000 7f3c07a14740 1 -- 172.21.15.45:0/3481731358 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56156dbeaae0 con 0x56156dcf3f90 2024-03-28T17:50:25.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.975+0000 7f3bf5400640 1 --2- 172.21.15.45:0/3481731358 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56156dcf3f90 0x56156dcf6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:25.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.976+0000 7f3be3400640 10 client.25709 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:50:25.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.976+0000 7f3bf5400640 1 --2- 172.21.15.45:0/3481731358 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56156dcf3f90 0x56156dcf6380 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a81957778f1b7e2 server_cookie=991bc862f994fe36 in_seq=0 out_seq=0 2024-03-28T17:50:25.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.976+0000 7f3be3400640 1 -- 172.21.15.45:0/3481731358 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56156dbeaae0 con 0x56156dcf3f90 2024-03-28T17:50:25.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.976+0000 7f3be3400640 10 client.25709 handle_command_reply: tid=0 2024-03-28T17:50:25.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.976+0000 7f3c07a14740 1 client.25709 shutdown 2024-03-28T17:50:25.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.976+0000 7f3c07a14740 1 -- 172.21.15.45:0/3481731358 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc7ef20 msgr2=0x56156dc98f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:25.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.976+0000 7f3c07a14740 1 --2- 172.21.15.45:0/3481731358 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc7ef20 0x56156dc98f10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0099ed0 tx=0x7f3bf009c320 comp rx=0 tx=0).stop 2024-03-28T17:50:25.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.976+0000 7f3c07a14740 1 -- 172.21.15.45:0/3481731358 shutdown_connections 2024-03-28T17:50:25.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.976+0000 7f3c07a14740 1 --2- 172.21.15.45:0/3481731358 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56156dcf3f90 0x56156dcf6380 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.976+0000 7f3c07a14740 1 --2- 172.21.15.45:0/3481731358 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56156dc88290 0x56156dcf2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.976+0000 7f3c07a14740 1 --2- 172.21.15.45:0/3481731358 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56156dc7fc00 0x56156dc99450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.976+0000 7f3be3400640 0 client.25709 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:50:25.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.976+0000 7f3c07a14740 1 --2- 172.21.15.45:0/3481731358 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56156dc7ef20 0x56156dc98f10 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.976+0000 7f3c07a14740 1 -- 172.21.15.45:0/3481731358 >> 172.21.15.45:0/3481731358 conn(0x56156dc46200 msgr2=0x56156dc834e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:25.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.976+0000 7f3c07a14740 1 -- 172.21.15.45:0/3481731358 shutdown_connections 2024-03-28T17:50:25.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.976+0000 7f3c07a14740 1 -- 172.21.15.45:0/3481731358 wait complete. 2024-03-28T17:50:25.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.976+0000 7f3c07a14740 20 client.25709 trim_cache size 0 max 16384 2024-03-28T17:50:25.982 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:50:25.982 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-28T17:50:25.982 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-28T17:50:25.982 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:50:25.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.977+0000 7f3c04c00640 1 -- 172.21.15.45:0/3495994541 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f3be40a2a90 msgr2=0x7f3be40a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:25.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.977+0000 7f3c04c00640 1 --2- 172.21.15.45:0/3495994541 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f3be40a2a90 0x7f3be40a4f50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3c0006ad50 tx=0x7f3bf800b4d0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.977+0000 7f3c04c00640 1 -- 172.21.15.45:0/3495994541 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c00100490 msgr2=0x7f3c0019a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:25.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.977+0000 7f3c04c00640 1 --2- 172.21.15.45:0/3495994541 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c00100490 0x7f3c0019a1d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000eab0 tx=0x7f3bf000ef70 comp rx=0 tx=0).stop 2024-03-28T17:50:25.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.977+0000 7f3c04c00640 1 -- 172.21.15.45:0/3495994541 shutdown_connections 2024-03-28T17:50:25.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.977+0000 7f3c04c00640 1 --2- 172.21.15.45:0/3495994541 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f3be40a2a90 0x7f3be40a4f50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.977+0000 7f3c04c00640 1 --2- 172.21.15.45:0/3495994541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0010e040 0x7f3c0019ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.977+0000 7f3c04c00640 1 --2- 172.21.15.45:0/3495994541 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c00100e40 0x7f3c0019a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.977+0000 7f3c04c00640 1 --2- 172.21.15.45:0/3495994541 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c00100490 0x7f3c0019a1d0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:25.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.977+0000 7f3c04c00640 1 -- 172.21.15.45:0/3495994541 >> 172.21.15.45:0/3495994541 conn(0x7f3c00077700 msgr2=0x7f3c0010f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:25.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.977+0000 7f3c04c00640 1 -- 172.21.15.45:0/3495994541 shutdown_connections 2024-03-28T17:50:25.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:25.977+0000 7f3c04c00640 1 -- 172.21.15.45:0/3495994541 wait complete. 2024-03-28T17:50:25.994 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T17:50:25.994 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:50:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.197+0000 7fbd9ba00640 1 -- 172.21.15.45:0/2777211267 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd941028d0 msgr2=0x7fbd94102cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.197+0000 7fbd9ba00640 1 --2- 172.21.15.45:0/2777211267 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd941028d0 0x7fbd94102cb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fbd88009900 tx=0x7fbd8802bcd0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.197+0000 7fbd9ba00640 1 -- 172.21.15.45:0/2777211267 shutdown_connections 2024-03-28T17:50:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.197+0000 7fbd9ba00640 1 --2- 172.21.15.45:0/2777211267 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd9410bca0 0x7fbd9410e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.197+0000 7fbd9ba00640 1 --2- 172.21.15.45:0/2777211267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd941031f0 0x7fbd9410b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.197+0000 7fbd9ba00640 1 --2- 172.21.15.45:0/2777211267 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd941028d0 0x7fbd94102cb0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.197+0000 7fbd9ba00640 1 -- 172.21.15.45:0/2777211267 >> 172.21.15.45:0/2777211267 conn(0x7fbd94076920 msgr2=0x7fbd94076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.197+0000 7fbd9ba00640 1 -- 172.21.15.45:0/2777211267 shutdown_connections 2024-03-28T17:50:26.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.197+0000 7fbd9ba00640 1 -- 172.21.15.45:0/2777211267 wait complete. 2024-03-28T17:50:26.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.198+0000 7fbd9ba00640 1 Processor -- start 2024-03-28T17:50:26.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.198+0000 7fbd9ba00640 1 -- start start 2024-03-28T17:50:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.198+0000 7fbd9ba00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd941028d0 0x7fbd9419a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.198+0000 7fbd9ba00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd941031f0 0x7fbd9419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.198+0000 7fbd9ba00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd9410bca0 0x7fbd9419e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.198+0000 7fbd9ba00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd94111a80 con 0x7fbd941031f0 2024-03-28T17:50:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.198+0000 7fbd9ba00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd94111900 con 0x7fbd941028d0 2024-03-28T17:50:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.198+0000 7fbd9ba00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd94111c00 con 0x7fbd9410bca0 2024-03-28T17:50:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.199+0000 7fbd99000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd941028d0 0x7fbd9419a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.199+0000 7fbd99000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd941028d0 0x7fbd9419a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:33280/0 (socket says 172.21.15.45:33280) 2024-03-28T17:50:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.199+0000 7fbd99a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd9410bca0 0x7fbd9419e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.199+0000 7fbd99000640 1 -- 172.21.15.45:0/3493390979 learned_addr learned my addr 172.21.15.45:0/3493390979 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.199+0000 7fbd93e00640 1 --2- 172.21.15.45:0/3493390979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd941031f0 0x7fbd9419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.199+0000 7fbd99000640 1 -- 172.21.15.45:0/3493390979 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd9410bca0 msgr2=0x7fbd9419e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.199+0000 7fbd99000640 1 --2- 172.21.15.45:0/3493390979 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd9410bca0 0x7fbd9419e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.199+0000 7fbd99000640 1 -- 172.21.15.45:0/3493390979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd941031f0 msgr2=0x7fbd9419a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:26.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.199+0000 7fbd99000640 1 --2- 172.21.15.45:0/3493390979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd941031f0 0x7fbd9419a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.199+0000 7fbd99000640 1 -- 172.21.15.45:0/3493390979 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9419f110 con 0x7fbd941028d0 2024-03-28T17:50:26.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.199+0000 7fbd99000640 1 --2- 172.21.15.45:0/3493390979 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd941028d0 0x7fbd9419a120 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbd88000c00 tx=0x7fbd880043c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:26.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.200+0000 7fbd91600640 1 -- 172.21.15.45:0/3493390979 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd88042020 con 0x7fbd941028d0 2024-03-28T17:50:26.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.200+0000 7fbd91600640 1 -- 172.21.15.45:0/3493390979 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd880054e0 con 0x7fbd941028d0 2024-03-28T17:50:26.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.200+0000 7fbd91600640 1 -- 172.21.15.45:0/3493390979 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd88003af0 con 0x7fbd941028d0 2024-03-28T17:50:26.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.200+0000 7fbd9ba00640 1 -- 172.21.15.45:0/3493390979 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd9419f3a0 con 0x7fbd941028d0 2024-03-28T17:50:26.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.200+0000 7fbd9ba00640 1 -- 172.21.15.45:0/3493390979 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd941a6c90 con 0x7fbd941028d0 2024-03-28T17:50:26.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.201+0000 7fbd9ba00640 1 -- 172.21.15.45:0/3493390979 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd9406a4f0 con 0x7fbd941028d0 2024-03-28T17:50:26.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.203+0000 7fbd91600640 1 -- 172.21.15.45:0/3493390979 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 27) v1 ==== 137523+0+0 (secure 0 0 0) 0x7fbd88045020 con 0x7fbd941028d0 2024-03-28T17:50:26.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.204+0000 7fbd91600640 1 --2- 172.21.15.45:0/3493390979 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbd7c0a2a90 0x7fbd7c0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.204+0000 7fbd91600640 1 -- 172.21.15.45:0/3493390979 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fbd880f0d40 con 0x7fbd941028d0 2024-03-28T17:50:26.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.204+0000 7fbd93e00640 1 --2- 172.21.15.45:0/3493390979 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbd7c0a2a90 0x7fbd7c0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.205+0000 7fbd93e00640 1 --2- 172.21.15.45:0/3493390979 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbd7c0a2a90 0x7fbd7c0a4f50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbd940ffb40 tx=0x7fbd8000b590 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:26.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.208+0000 7fbd91600640 1 -- 172.21.15.45:0/3493390979 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbd8804c020 con 0x7fbd941028d0 2024-03-28T17:50:26.299 INFO:tasks.workunit.client.0.smithi045.stderr:Note: switching to '3be9a5c9bc793e11e2800a8c0c696e8b46742033'. 2024-03-28T17:50:26.299 INFO:tasks.workunit.client.0.smithi045.stderr: 2024-03-28T17:50:26.299 INFO:tasks.workunit.client.0.smithi045.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-28T17:50:26.300 INFO:tasks.workunit.client.0.smithi045.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-28T17:50:26.300 INFO:tasks.workunit.client.0.smithi045.stderr:state without impacting any branches by switching back to a branch. 2024-03-28T17:50:26.300 INFO:tasks.workunit.client.0.smithi045.stderr: 2024-03-28T17:50:26.300 INFO:tasks.workunit.client.0.smithi045.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-28T17:50:26.300 INFO:tasks.workunit.client.0.smithi045.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-28T17:50:26.300 INFO:tasks.workunit.client.0.smithi045.stderr: 2024-03-28T17:50:26.300 INFO:tasks.workunit.client.0.smithi045.stderr: git switch -c 2024-03-28T17:50:26.300 INFO:tasks.workunit.client.0.smithi045.stderr: 2024-03-28T17:50:26.300 INFO:tasks.workunit.client.0.smithi045.stderr:Or undo this operation with: 2024-03-28T17:50:26.300 INFO:tasks.workunit.client.0.smithi045.stderr: 2024-03-28T17:50:26.300 INFO:tasks.workunit.client.0.smithi045.stderr: git switch - 2024-03-28T17:50:26.300 INFO:tasks.workunit.client.0.smithi045.stderr: 2024-03-28T17:50:26.300 INFO:tasks.workunit.client.0.smithi045.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-28T17:50:26.300 INFO:tasks.workunit.client.0.smithi045.stderr: 2024-03-28T17:50:26.301 INFO:tasks.workunit.client.0.smithi045.stderr:HEAD is now at 3be9a5c9bc7 dummy commit #1 2024-03-28T17:50:26.309 DEBUG:teuthology.orchestra.run.smithi045:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-28T17:50:26.366 INFO:tasks.workunit.client.0.smithi045.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-28T17:50:26.372 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T17:50:26.373 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-28T17:50:26.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.456+0000 7fbd9ba00640 1 -- 172.21.15.45:0/3493390979 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd940688e0 con 0x7fbd941028d0 2024-03-28T17:50:26.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.457+0000 7fbd91600640 1 -- 172.21.15.45:0/3493390979 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11393 (secure 0 0 0) 0x7fbd880bda00 con 0x7fbd941028d0 2024-03-28T17:50:26.457 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:50:26.457 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6839/2313500150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":2313500150},{"type":"v1","addr":"172.21.15.45:6839","nonce":2313500150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6841/2647972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":2647972571},{"type":"v1","addr":"172.21.15.45:6841","nonce":2647972571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T17:46:12.261135+0000","modified":"2024-03-28T17:48:38.762497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24584,"mds_1":24530,"mds_2":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6835/2876354410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":2876354410},{"type":"v1","addr":"172.21.15.151:6835","nonce":2876354410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24500":{"gid":24500,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/4234998385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":4234998385},{"type":"v1","addr":"172.21.15.71:6835","nonce":4234998385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6837/1238964845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":1238964845},{"type":"v1","addr":"172.21.15.151:6837","nonce":1238964845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.151:6839/3003564901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6838","nonce":3003564901},{"type":"v1","addr":"172.21.15.151:6839","nonce":3003564901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6837/2008752655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2008752655},{"type":"v1","addr":"172.21.15.71:6837","nonce":2008752655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.71:6839/2709938800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":2709938800},{"type":"v1","addr":"172.21.15.71:6839","nonce":2709938800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24584,"qdb_cluster":[24584,24529,24530]},"id":1}]} 2024-03-28T17:50:26.458 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 23 2024-03-28T17:50:26.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.459+0000 7fbd9ba00640 1 -- 172.21.15.45:0/3493390979 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbd7c0a2a90 msgr2=0x7fbd7c0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:26.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.459+0000 7fbd9ba00640 1 --2- 172.21.15.45:0/3493390979 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbd7c0a2a90 0x7fbd7c0a4f50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbd940ffb40 tx=0x7fbd8000b590 comp rx=0 tx=0).stop 2024-03-28T17:50:26.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.459+0000 7fbd9ba00640 1 -- 172.21.15.45:0/3493390979 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd941028d0 msgr2=0x7fbd9419a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:26.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.460+0000 7fbd9ba00640 1 --2- 172.21.15.45:0/3493390979 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd941028d0 0x7fbd9419a120 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbd88000c00 tx=0x7fbd880043c0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.460+0000 7fbd9ba00640 1 -- 172.21.15.45:0/3493390979 shutdown_connections 2024-03-28T17:50:26.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.460+0000 7fbd9ba00640 1 --2- 172.21.15.45:0/3493390979 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbd7c0a2a90 0x7fbd7c0a4f50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.460+0000 7fbd9ba00640 1 --2- 172.21.15.45:0/3493390979 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd9410bca0 0x7fbd9419e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.460+0000 7fbd9ba00640 1 --2- 172.21.15.45:0/3493390979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd941031f0 0x7fbd9419a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.460+0000 7fbd9ba00640 1 --2- 172.21.15.45:0/3493390979 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd941028d0 0x7fbd9419a120 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.460+0000 7fbd9ba00640 1 -- 172.21.15.45:0/3493390979 >> 172.21.15.45:0/3493390979 conn(0x7fbd94076920 msgr2=0x7fbd94101620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:26.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.460+0000 7fbd9ba00640 1 -- 172.21.15.45:0/3493390979 shutdown_connections 2024-03-28T17:50:26.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.460+0000 7fbd9ba00640 1 -- 172.21.15.45:0/3493390979 wait complete. 2024-03-28T17:50:26.477 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T17:50:26.477 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T17:50:26.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.687+0000 7f40e5c00640 1 -- 172.21.15.45:0/2562436340 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40e0109730 msgr2=0x7f40e0109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:26.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.687+0000 7f40e5c00640 1 --2- 172.21.15.45:0/2562436340 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40e0109730 0x7f40e0109b10 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f40d0009930 tx=0x7f40d002bc30 comp rx=0 tx=0).stop 2024-03-28T17:50:26.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.687+0000 7f40e5c00640 1 -- 172.21.15.45:0/2562436340 shutdown_connections 2024-03-28T17:50:26.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.687+0000 7f40e5c00640 1 --2- 172.21.15.45:0/2562436340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40e01040e0 0x7f40e01105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.687+0000 7f40e5c00640 1 --2- 172.21.15.45:0/2562436340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f40e0103720 0x7f40e0103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.687+0000 7f40e5c00640 1 --2- 172.21.15.45:0/2562436340 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40e0109730 0x7f40e0109b10 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.687+0000 7f40e5c00640 1 -- 172.21.15.45:0/2562436340 >> 172.21.15.45:0/2562436340 conn(0x7f40e00ff1b0 msgr2=0x7f40e01015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:26.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.687+0000 7f40e5c00640 1 -- 172.21.15.45:0/2562436340 shutdown_connections 2024-03-28T17:50:26.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.687+0000 7f40e5c00640 1 -- 172.21.15.45:0/2562436340 wait complete. 2024-03-28T17:50:26.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.688+0000 7f40e5c00640 1 Processor -- start 2024-03-28T17:50:26.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.688+0000 7f40e5c00640 1 -- start start 2024-03-28T17:50:26.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.689+0000 7f40e5c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40e0103720 0x7f40e019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.689+0000 7f40e5c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f40e01040e0 0x7f40e019a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.689+0000 7f40e5c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40e019eb40 0x7f40e019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.689+0000 7f40e5c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e0113b50 con 0x7f40e0103720 2024-03-28T17:50:26.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.689+0000 7f40e5c00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e01139d0 con 0x7f40e01040e0 2024-03-28T17:50:26.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.689+0000 7f40e5c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e0113cd0 con 0x7f40e019eb40 2024-03-28T17:50:26.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.689+0000 7f40dfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40e0103720 0x7f40e019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.689+0000 7f40df400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f40e01040e0 0x7f40e019a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.689+0000 7f40dfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40e0103720 0x7f40e019a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42218/0 (socket says 172.21.15.45:42218) 2024-03-28T17:50:26.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.689+0000 7f40dfe00640 1 -- 172.21.15.45:0/2052740704 learned_addr learned my addr 172.21.15.45:0/2052740704 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:26.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.689+0000 7f40e5200640 1 --2- 172.21.15.45:0/2052740704 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40e019eb40 0x7f40e019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.689+0000 7f40df400640 1 -- 172.21.15.45:0/2052740704 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40e019eb40 msgr2=0x7f40e019eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:26.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.689+0000 7f40df400640 1 --2- 172.21.15.45:0/2052740704 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40e019eb40 0x7f40e019eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.689+0000 7f40df400640 1 -- 172.21.15.45:0/2052740704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40e0103720 msgr2=0x7f40e019a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:26.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.689+0000 7f40df400640 1 --2- 172.21.15.45:0/2052740704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40e0103720 0x7f40e019a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.689+0000 7f40df400640 1 -- 172.21.15.45:0/2052740704 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e019f630 con 0x7f40e01040e0 2024-03-28T17:50:26.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.690+0000 7f40df400640 1 --2- 172.21.15.45:0/2052740704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f40e01040e0 0x7f40e019a7c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f40c8005fd0 tx=0x7f40c800dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:26.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.690+0000 7f40dcc00640 1 -- 172.21.15.45:0/2052740704 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c8009c30 con 0x7f40e01040e0 2024-03-28T17:50:26.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.690+0000 7f40dcc00640 1 -- 172.21.15.45:0/2052740704 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40c8009dd0 con 0x7f40e01040e0 2024-03-28T17:50:26.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.690+0000 7f40dcc00640 1 -- 172.21.15.45:0/2052740704 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c8010460 con 0x7f40e01040e0 2024-03-28T17:50:26.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.690+0000 7f40e5c00640 1 -- 172.21.15.45:0/2052740704 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40e019f920 con 0x7f40e01040e0 2024-03-28T17:50:26.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.690+0000 7f40e5c00640 1 -- 172.21.15.45:0/2052740704 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e01a70c0 con 0x7f40e01040e0 2024-03-28T17:50:26.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.693+0000 7f40dcc00640 1 -- 172.21.15.45:0/2052740704 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 27) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f40c8003ef0 con 0x7f40e01040e0 2024-03-28T17:50:26.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.693+0000 7f40e9fd0740 1 Processor -- start 2024-03-28T17:50:26.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.694+0000 7f40e9fd0740 1 -- start start 2024-03-28T17:50:26.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.694+0000 7f40e9fd0740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d748b0 0x563bc9d74c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.694+0000 7f40e9fd0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563bc9d75260 0x563bc9d7d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.694+0000 7f40e9fd0740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d7dde0 0x563bc9d80230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.694+0000 7f40e9fd0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bc9cd82c0 con 0x563bc9d75260 2024-03-28T17:50:26.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.694+0000 7f40e9fd0740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bc9c80c90 con 0x563bc9d7dde0 2024-03-28T17:50:26.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.694+0000 7f40e9fd0740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bc9cd4360 con 0x563bc9d748b0 2024-03-28T17:50:26.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.694+0000 7f40e5200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d748b0 0x563bc9d74c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.694+0000 7f40dfe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563bc9d75260 0x563bc9d7d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.694+0000 7f40e5200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d748b0 0x563bc9d74c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:46864/0 (socket says 172.21.15.45:46864) 2024-03-28T17:50:26.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.694+0000 7f40e5200640 1 -- 172.21.15.45:0/2282166368 learned_addr learned my addr 172.21.15.45:0/2282166368 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:26.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.694+0000 7f40df400640 1 --2- 172.21.15.45:0/2282166368 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d7dde0 0x563bc9d80230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.695+0000 7f40e5200640 1 -- 172.21.15.45:0/2282166368 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d7dde0 msgr2=0x563bc9d80230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:26.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.695+0000 7f40e5200640 1 --2- 172.21.15.45:0/2282166368 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d7dde0 0x563bc9d80230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.695+0000 7f40e5200640 1 -- 172.21.15.45:0/2282166368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563bc9d75260 msgr2=0x563bc9d7d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:26.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.695+0000 7f40e5200640 1 --2- 172.21.15.45:0/2282166368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563bc9d75260 0x563bc9d7d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.695+0000 7f40e5200640 1 -- 172.21.15.45:0/2282166368 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563bc9ca5080 con 0x563bc9d748b0 2024-03-28T17:50:26.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.695+0000 7f40df400640 1 --2- 172.21.15.45:0/2282166368 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d7dde0 0x563bc9d80230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:50:26.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.695+0000 7f40dcc00640 1 --2- 172.21.15.45:0/2052740704 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f40c40a2b10 0x7f40c40a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.695+0000 7f40e5200640 1 --2- 172.21.15.45:0/2282166368 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d748b0 0x563bc9d74c90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f40d8009f70 tx=0x7f40d8010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb7f56355bef197b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:26.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.695+0000 7f40dcc00640 1 -- 172.21.15.45:0/2052740704 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f40c80cebb0 con 0x7f40e01040e0 2024-03-28T17:50:26.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.695+0000 7f40d5600640 1 -- 172.21.15.45:0/2282166368 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d8016070 con 0x563bc9d748b0 2024-03-28T17:50:26.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.695+0000 7f40d5600640 1 -- 172.21.15.45:0/2282166368 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40d8004d20 con 0x563bc9d748b0 2024-03-28T17:50:26.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.695+0000 7f40dfe00640 1 --2- 172.21.15.45:0/2052740704 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f40c40a2b10 0x7f40c40a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.695+0000 7f40d5600640 1 -- 172.21.15.45:0/2282166368 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d8012950 con 0x563bc9d748b0 2024-03-28T17:50:26.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.696+0000 7f40e9fd0740 1 -- 172.21.15.45:0/2282166368 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d748b0 msgr2=0x563bc9d74c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:26.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.696+0000 7f40e9fd0740 1 --2- 172.21.15.45:0/2282166368 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d748b0 0x563bc9d74c90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f40d8009f70 tx=0x7f40d8010800 comp rx=0 tx=0).stop 2024-03-28T17:50:26.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.696+0000 7f40e9fd0740 1 -- 172.21.15.45:0/2282166368 shutdown_connections 2024-03-28T17:50:26.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.696+0000 7f40dfe00640 1 --2- 172.21.15.45:0/2052740704 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f40c40a2b10 0x7f40c40a4fd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f40d0006010 tx=0x7f40d002df90 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:26.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.696+0000 7f40e9fd0740 1 --2- 172.21.15.45:0/2282166368 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d7dde0 0x563bc9d80230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.696+0000 7f40e9fd0740 1 --2- 172.21.15.45:0/2282166368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563bc9d75260 0x563bc9d7d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.696+0000 7f40e9fd0740 1 --2- 172.21.15.45:0/2282166368 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d748b0 0x563bc9d74c90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.696+0000 7f40e9fd0740 1 -- 172.21.15.45:0/2282166368 >> 172.21.15.45:0/2282166368 conn(0x563bc9d6ef80 msgr2=0x563bc9d6f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:26.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.696+0000 7f40e9fd0740 1 -- 172.21.15.45:0/2282166368 shutdown_connections 2024-03-28T17:50:26.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.696+0000 7f40e9fd0740 1 -- 172.21.15.45:0/2282166368 wait complete. 2024-03-28T17:50:26.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.696+0000 7f40e9fd0740 1 -- start start 2024-03-28T17:50:26.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.697+0000 7f40e9fd0740 1 Processor -- start 2024-03-28T17:50:26.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.697+0000 7f40e9fd0740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d748b0 0x563bc9d7d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.697+0000 7f40e9fd0740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d75260 0x563bc9d7aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.697+0000 7f40e9fd0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563bc9d7dde0 0x563bc9d7b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.697+0000 7f40e9fd0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bc9d80d50 con 0x563bc9d7dde0 2024-03-28T17:50:26.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.697+0000 7f40e9fd0740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bc9c940d0 con 0x563bc9d748b0 2024-03-28T17:50:26.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.697+0000 7f40e9fd0740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bc9d80ed0 con 0x563bc9d75260 2024-03-28T17:50:26.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.697+0000 7f40e9fd0740 10 client.0 _ll_register_callbacks cb 0x563bc9c98e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:50:26.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.697+0000 7f40dfe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d748b0 0x563bc9d7d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.697+0000 7f40dfe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d748b0 0x563bc9d7d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:33310/0 (socket says 172.21.15.45:33310) 2024-03-28T17:50:26.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.697+0000 7f40df400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d75260 0x563bc9d7aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.697+0000 7f40dfe00640 1 -- 172.21.15.45:0/1038630956 learned_addr learned my addr 172.21.15.45:0/1038630956 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:26.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.697+0000 7f40e5200640 1 --2- 172.21.15.45:0/1038630956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563bc9d7dde0 0x563bc9d7b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.698+0000 7f40df400640 1 -- 172.21.15.45:0/1038630956 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d748b0 msgr2=0x563bc9d7d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:26.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.698+0000 7f40df400640 1 --2- 172.21.15.45:0/1038630956 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d748b0 0x563bc9d7d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.698+0000 7f40df400640 1 -- 172.21.15.45:0/1038630956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563bc9d7dde0 msgr2=0x563bc9d7b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:26.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.698+0000 7f40df400640 1 --2- 172.21.15.45:0/1038630956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563bc9d7dde0 0x563bc9d7b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.698+0000 7f40df400640 1 -- 172.21.15.45:0/1038630956 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563bc99ec3f0 con 0x563bc9d75260 2024-03-28T17:50:26.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.698+0000 7f40df400640 1 --2- 172.21.15.45:0/1038630956 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d75260 0x563bc9d7aab0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f40c80cc150 tx=0x7f40c809ac40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=965e523dbb5be5b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:26.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.698+0000 7f40ce000640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:50:26.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.698+0000 7f40ce000640 1 -- 172.21.15.45:0/1038630956 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c8020050 con 0x563bc9d75260 2024-03-28T17:50:26.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.698+0000 7f40ce000640 1 -- 172.21.15.45:0/1038630956 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f40c80a32a0 con 0x563bc9d75260 2024-03-28T17:50:26.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.698+0000 7f40e9fd0740 1 -- 172.21.15.45:0/1038630956 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f40d8004d20 con 0x563bc9d75260 2024-03-28T17:50:26.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.699+0000 7f40ce000640 1 -- 172.21.15.45:0/1038630956 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40c80d3040 con 0x563bc9d75260 2024-03-28T17:50:26.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.699+0000 7f40e9fd0740 10 client.25696 fetch_fsmap learned FSMap version 23 2024-03-28T17:50:26.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.699+0000 7f40e9fd0740 1 -- 172.21.15.45:0/1038630956 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563bc9ab67d0 con 0x563bc9d75260 2024-03-28T17:50:26.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.699+0000 7f40ce000640 1 -- 172.21.15.45:0/1038630956 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40d8004d20 con 0x563bc9d75260 2024-03-28T17:50:26.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.699+0000 7f40ce000640 1 -- 172.21.15.45:0/1038630956 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f40c80cc430 con 0x563bc9d75260 2024-03-28T17:50:26.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.699+0000 7f40e9fd0740 10 client.25696 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:50:26.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.699+0000 7f40e9fd0740 10 client.25696 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-28T17:50:26.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.699+0000 7f40e9fd0740 20 client.25696 populate_metadata read hostname 'smithi045' 2024-03-28T17:50:26.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.699+0000 7f40e9fd0740 1 --2- 172.21.15.45:0/1038630956 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x563bc9d8dc80 0x563bc9d90070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.699+0000 7f40e9fd0740 4 client.25696 mds_command: new command op to 24529 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:50:26.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.699+0000 7f40e9fd0740 1 -- 172.21.15.45:0/1038630956 --> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563bc9ce5800 con 0x563bc9d8dc80 2024-03-28T17:50:26.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.700+0000 7f40dfe00640 1 --2- 172.21.15.45:0/1038630956 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x563bc9d8dc80 0x563bc9d90070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.700+0000 7f40dfe00640 1 --2- 172.21.15.45:0/1038630956 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x563bc9d8dc80 0x563bc9d90070 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8ed00326b9bffaa9 server_cookie=1b92ae4e8798ec20 in_seq=0 out_seq=0 2024-03-28T17:50:26.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.700+0000 7f40ce000640 10 client.25696 ms_handle_connect on v2:172.21.15.151:6838/3003564901 2024-03-28T17:50:26.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.701+0000 7f40ce000640 1 -- 172.21.15.45:0/1038630956 <== mds.2 v2:172.21.15.151:6838/3003564901 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563bc9ce5800 con 0x563bc9d8dc80 2024-03-28T17:50:26.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.701+0000 7f40ce000640 10 client.25696 handle_command_reply: tid=0 2024-03-28T17:50:26.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.701+0000 7f40e9fd0740 1 client.25696 shutdown 2024-03-28T17:50:26.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.701+0000 7f40e9fd0740 1 -- 172.21.15.45:0/1038630956 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d75260 msgr2=0x563bc9d7aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:26.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.701+0000 7f40e9fd0740 1 --2- 172.21.15.45:0/1038630956 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d75260 0x563bc9d7aab0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f40c80cc150 tx=0x7f40c809ac40 comp rx=0 tx=0).stop 2024-03-28T17:50:26.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.701+0000 7f40e9fd0740 1 -- 172.21.15.45:0/1038630956 shutdown_connections 2024-03-28T17:50:26.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.701+0000 7f40e9fd0740 1 --2- 172.21.15.45:0/1038630956 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x563bc9d8dc80 0x563bc9d90070 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.701+0000 7f40e9fd0740 1 --2- 172.21.15.45:0/1038630956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563bc9d7dde0 0x563bc9d7b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.701+0000 7f40e9fd0740 1 --2- 172.21.15.45:0/1038630956 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d75260 0x563bc9d7aab0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.701+0000 7f40e9fd0740 1 --2- 172.21.15.45:0/1038630956 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d748b0 0x563bc9d7d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.701+0000 7f40ce000640 0 client.25696 ms_handle_reset on v2:172.21.15.151:6838/3003564901 2024-03-28T17:50:26.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.701+0000 7f40e9fd0740 1 -- 172.21.15.45:0/1038630956 >> 172.21.15.45:0/1038630956 conn(0x563bc9d6ef80 msgr2=0x563bc9d70cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:26.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.701+0000 7f40e9fd0740 1 -- 172.21.15.45:0/1038630956 shutdown_connections 2024-03-28T17:50:26.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.701+0000 7f40e9fd0740 1 -- 172.21.15.45:0/1038630956 wait complete. 2024-03-28T17:50:26.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.701+0000 7f40e9fd0740 20 client.25696 trim_cache size 0 max 16384 2024-03-28T17:50:26.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.717+0000 7f40e9fd0740 1 Processor -- start 2024-03-28T17:50:26.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.718+0000 7f40e9fd0740 1 -- start start 2024-03-28T17:50:26.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.718+0000 7f40e9fd0740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d748b0 0x563bc9d7d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.718+0000 7f40e9fd0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563bc9d75260 0x563bc9d7d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.718+0000 7f40e9fd0740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d7dde0 0x563bc9d77d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.718+0000 7f40e9fd0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e01b1e40 con 0x563bc9d75260 2024-03-28T17:50:26.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.718+0000 7f40e9fd0740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e01af940 con 0x563bc9d748b0 2024-03-28T17:50:26.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.718+0000 7f40e9fd0740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e01ad9c0 con 0x563bc9d7dde0 2024-03-28T17:50:26.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.718+0000 7f40dfe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d748b0 0x563bc9d7d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.718+0000 7f40dfe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d748b0 0x563bc9d7d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:33322/0 (socket says 172.21.15.45:33322) 2024-03-28T17:50:26.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.718+0000 7f40dfe00640 1 -- 172.21.15.45:0/3423222431 learned_addr learned my addr 172.21.15.45:0/3423222431 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:26.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.718+0000 7f40e5200640 1 --2- 172.21.15.45:0/3423222431 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d7dde0 0x563bc9d77d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.718+0000 7f40df400640 1 --2- 172.21.15.45:0/3423222431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563bc9d75260 0x563bc9d7d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.718+0000 7f40dfe00640 1 -- 172.21.15.45:0/3423222431 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d7dde0 msgr2=0x563bc9d77d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:26.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.718+0000 7f40dfe00640 1 --2- 172.21.15.45:0/3423222431 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d7dde0 0x563bc9d77d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.718+0000 7f40dfe00640 1 -- 172.21.15.45:0/3423222431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563bc9d75260 msgr2=0x563bc9d7d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:26.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.718+0000 7f40dfe00640 1 --2- 172.21.15.45:0/3423222431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563bc9d75260 0x563bc9d7d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.718+0000 7f40dfe00640 1 -- 172.21.15.45:0/3423222431 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563bc9d6a8b0 con 0x563bc9d748b0 2024-03-28T17:50:26.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.718+0000 7f40df400640 1 --2- 172.21.15.45:0/3423222431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563bc9d75260 0x563bc9d7d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T17:50:26.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.719+0000 7f40dfe00640 1 --2- 172.21.15.45:0/3423222431 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d748b0 0x563bc9d7d1e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f40d0000c00 tx=0x7f40d0042000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f75635969c648c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:26.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.719+0000 7f40d4c00640 1 -- 172.21.15.45:0/3423222431 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d0046070 con 0x563bc9d748b0 2024-03-28T17:50:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.719+0000 7f40d4c00640 1 -- 172.21.15.45:0/3423222431 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40d0058050 con 0x563bc9d748b0 2024-03-28T17:50:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.719+0000 7f40d4c00640 1 -- 172.21.15.45:0/3423222431 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d0004f80 con 0x563bc9d748b0 2024-03-28T17:50:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.720+0000 7f40e9fd0740 1 -- 172.21.15.45:0/3423222431 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d748b0 msgr2=0x563bc9d7d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.720+0000 7f40e9fd0740 1 --2- 172.21.15.45:0/3423222431 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d748b0 0x563bc9d7d1e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f40d0000c00 tx=0x7f40d0042000 comp rx=0 tx=0).stop 2024-03-28T17:50:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.720+0000 7f40e9fd0740 1 -- 172.21.15.45:0/3423222431 shutdown_connections 2024-03-28T17:50:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.720+0000 7f40e9fd0740 1 --2- 172.21.15.45:0/3423222431 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d7dde0 0x563bc9d77d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.720+0000 7f40e9fd0740 1 --2- 172.21.15.45:0/3423222431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563bc9d75260 0x563bc9d7d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.720+0000 7f40e9fd0740 1 --2- 172.21.15.45:0/3423222431 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d748b0 0x563bc9d7d1e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.720+0000 7f40e9fd0740 1 -- 172.21.15.45:0/3423222431 >> 172.21.15.45:0/3423222431 conn(0x563bc9d6ef80 msgr2=0x563bc9d70c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.720+0000 7f40e9fd0740 1 -- 172.21.15.45:0/3423222431 shutdown_connections 2024-03-28T17:50:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.720+0000 7f40e9fd0740 1 -- 172.21.15.45:0/3423222431 wait complete. 2024-03-28T17:50:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.720+0000 7f40e9fd0740 1 -- start start 2024-03-28T17:50:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.720+0000 7f40e9fd0740 1 Processor -- start 2024-03-28T17:50:26.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.721+0000 7f40e9fd0740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d748b0 0x563bc9d84b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.721+0000 7f40e9fd0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563bc9d75260 0x563bc9d850c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.721+0000 7f40e9fd0740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d7dde0 0x563bc9de94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.721+0000 7f40e9fd0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e01adfc0 con 0x563bc9d75260 2024-03-28T17:50:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.721+0000 7f40e9fd0740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e01af2e0 con 0x563bc9d7dde0 2024-03-28T17:50:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.721+0000 7f40e9fd0740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e01ae620 con 0x563bc9d748b0 2024-03-28T17:50:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.721+0000 7f40e9fd0740 10 client.0 _ll_register_callbacks cb 0x7f40c80973b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:50:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.721+0000 7f40e5200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d7dde0 0x563bc9de94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.721+0000 7f40dfe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d748b0 0x563bc9d84b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.721+0000 7f40df400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563bc9d75260 0x563bc9d850c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.721+0000 7f40dfe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d748b0 0x563bc9d84b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:46896/0 (socket says 172.21.15.45:46896) 2024-03-28T17:50:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.721+0000 7f40dfe00640 1 -- 172.21.15.45:0/3932761429 learned_addr learned my addr 172.21.15.45:0/3932761429 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.721+0000 7f40e5200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d7dde0 0x563bc9de94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:33334/0 (socket says 172.21.15.45:33334) 2024-03-28T17:50:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.722+0000 7f40dfe00640 1 -- 172.21.15.45:0/3932761429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d7dde0 msgr2=0x563bc9de94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.722+0000 7f40dfe00640 1 --2- 172.21.15.45:0/3932761429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d7dde0 0x563bc9de94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.722+0000 7f40dfe00640 1 -- 172.21.15.45:0/3932761429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563bc9d75260 msgr2=0x563bc9d850c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:26.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.722+0000 7f40dfe00640 1 --2- 172.21.15.45:0/3932761429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563bc9d75260 0x563bc9d850c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.722+0000 7f40dfe00640 1 -- 172.21.15.45:0/3932761429 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563bc9de99f0 con 0x563bc9d748b0 2024-03-28T17:50:26.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.722+0000 7f40e5200640 1 --2- 172.21.15.45:0/3932761429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d7dde0 0x563bc9de94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:50:26.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.722+0000 7f40dfe00640 1 --2- 172.21.15.45:0/3932761429 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d748b0 0x563bc9d84b80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f40d00092a0 tx=0x7f40d0058360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5254c3652c7e853e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:26.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.722+0000 7f40ce000640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:50:26.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.722+0000 7f40ce000640 1 -- 172.21.15.45:0/3932761429 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d002ca80 con 0x563bc9d748b0 2024-03-28T17:50:26.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.722+0000 7f40ce000640 1 -- 172.21.15.45:0/3932761429 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f40d002cc20 con 0x563bc9d748b0 2024-03-28T17:50:26.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.722+0000 7f40e9fd0740 1 -- 172.21.15.45:0/3932761429 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f40d0058050 con 0x563bc9d748b0 2024-03-28T17:50:26.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.726+0000 7f40ce000640 1 -- 172.21.15.45:0/3932761429 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40d0004880 con 0x563bc9d748b0 2024-03-28T17:50:26.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.727+0000 7f40e9fd0740 10 client.25702 fetch_fsmap learned FSMap version 23 2024-03-28T17:50:26.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.727+0000 7f40e9fd0740 1 -- 172.21.15.45:0/3932761429 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563bc9deabd0 con 0x563bc9d748b0 2024-03-28T17:50:26.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.727+0000 7f40ce000640 1 -- 172.21.15.45:0/3932761429 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40d0058050 con 0x563bc9d748b0 2024-03-28T17:50:26.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.727+0000 7f40ce000640 1 -- 172.21.15.45:0/3932761429 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f40d0058a80 con 0x563bc9d748b0 2024-03-28T17:50:26.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.727+0000 7f40e9fd0740 10 client.25702 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:50:26.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.727+0000 7f40e9fd0740 10 client.25702 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-28T17:50:26.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.727+0000 7f40e9fd0740 20 client.25702 populate_metadata read hostname 'smithi045' 2024-03-28T17:50:26.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.727+0000 7f40e9fd0740 1 --2- 172.21.15.45:0/3932761429 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x563bc9deb240 0x563bc9ded630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.727+0000 7f40e9fd0740 4 client.25702 mds_command: new command op to 24529 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T17:50:26.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.727+0000 7f40e9fd0740 1 -- 172.21.15.45:0/3932761429 --> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563bc9cdfb50 con 0x563bc9deb240 2024-03-28T17:50:26.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.727+0000 7f40df400640 1 --2- 172.21.15.45:0/3932761429 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x563bc9deb240 0x563bc9ded630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.728+0000 7f40df400640 1 --2- 172.21.15.45:0/3932761429 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x563bc9deb240 0x563bc9ded630 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=be25da38559ece11 server_cookie=bbc1689dbe73e574 in_seq=0 out_seq=0 2024-03-28T17:50:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.728+0000 7f40ce000640 10 client.25702 ms_handle_connect on v2:172.21.15.151:6838/3003564901 2024-03-28T17:50:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.728+0000 7f40ce000640 1 -- 172.21.15.45:0/3932761429 <== mds.2 v2:172.21.15.151:6838/3003564901 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563bc9cdfb50 con 0x563bc9deb240 2024-03-28T17:50:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.728+0000 7f40ce000640 10 client.25702 handle_command_reply: tid=0 2024-03-28T17:50:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.728+0000 7f40e9fd0740 1 client.25702 shutdown 2024-03-28T17:50:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.728+0000 7f40e9fd0740 1 -- 172.21.15.45:0/3932761429 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d748b0 msgr2=0x563bc9d84b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.728+0000 7f40e9fd0740 1 --2- 172.21.15.45:0/3932761429 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d748b0 0x563bc9d84b80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f40d00092a0 tx=0x7f40d0058360 comp rx=0 tx=0).stop 2024-03-28T17:50:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.728+0000 7f40e9fd0740 1 -- 172.21.15.45:0/3932761429 shutdown_connections 2024-03-28T17:50:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.728+0000 7f40e9fd0740 1 --2- 172.21.15.45:0/3932761429 >> [v2:172.21.15.151:6838/3003564901,v1:172.21.15.151:6839/3003564901] conn(0x563bc9deb240 0x563bc9ded630 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.728+0000 7f40e9fd0740 1 --2- 172.21.15.45:0/3932761429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563bc9d7dde0 0x563bc9de94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.728+0000 7f40e9fd0740 1 --2- 172.21.15.45:0/3932761429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563bc9d75260 0x563bc9d850c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.728+0000 7f40e9fd0740 1 --2- 172.21.15.45:0/3932761429 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563bc9d748b0 0x563bc9d84b80 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.728+0000 7f40ce000640 0 client.25702 ms_handle_reset on v2:172.21.15.151:6838/3003564901 2024-03-28T17:50:26.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.728+0000 7f40e9fd0740 1 -- 172.21.15.45:0/3932761429 >> 172.21.15.45:0/3932761429 conn(0x563bc9d6ef80 msgr2=0x563bc9d75640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:26.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.729+0000 7f40e9fd0740 1 -- 172.21.15.45:0/3932761429 shutdown_connections 2024-03-28T17:50:26.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.729+0000 7f40e9fd0740 1 -- 172.21.15.45:0/3932761429 wait complete. 2024-03-28T17:50:26.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.729+0000 7f40e9fd0740 20 client.25702 trim_cache size 0 max 16384 2024-03-28T17:50:26.731 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-28T17:50:26.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.729+0000 7f40e5c00640 1 -- 172.21.15.45:0/2052740704 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f40c40a2b10 msgr2=0x7f40c40a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:26.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.729+0000 7f40e5c00640 1 --2- 172.21.15.45:0/2052740704 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f40c40a2b10 0x7f40c40a4fd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f40d0006010 tx=0x7f40d002df90 comp rx=0 tx=0).stop 2024-03-28T17:50:26.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.729+0000 7f40e5c00640 1 -- 172.21.15.45:0/2052740704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f40e01040e0 msgr2=0x7f40e019a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:26.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.729+0000 7f40e5c00640 1 --2- 172.21.15.45:0/2052740704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f40e01040e0 0x7f40e019a7c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f40c8005fd0 tx=0x7f40c800dde0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.730+0000 7f40e5c00640 1 -- 172.21.15.45:0/2052740704 shutdown_connections 2024-03-28T17:50:26.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.730+0000 7f40e5c00640 1 --2- 172.21.15.45:0/2052740704 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f40c40a2b10 0x7f40c40a4fd0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.730+0000 7f40e5c00640 1 --2- 172.21.15.45:0/2052740704 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40e019eb40 0x7f40e019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.730+0000 7f40e5c00640 1 --2- 172.21.15.45:0/2052740704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f40e01040e0 0x7f40e019a7c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.730+0000 7f40e5c00640 1 --2- 172.21.15.45:0/2052740704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40e0103720 0x7f40e019a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.730+0000 7f40e5c00640 1 -- 172.21.15.45:0/2052740704 >> 172.21.15.45:0/2052740704 conn(0x7f40e00ff1b0 msgr2=0x7f40e00ffad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:26.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.730+0000 7f40e5c00640 1 -- 172.21.15.45:0/2052740704 shutdown_connections 2024-03-28T17:50:26.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.730+0000 7f40e5c00640 1 -- 172.21.15.45:0/2052740704 wait complete. 2024-03-28T17:50:26.744 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:26 smithi045 ceph-mon[31773]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-28T17:50:26.744 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:26 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3493390979' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:50:26.747 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T17:50:26.747 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T17:50:26.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.957+0000 7f7e42000640 1 -- 172.21.15.45:0/616641316 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e3c10baf0 msgr2=0x7f7e3c10df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.957+0000 7f7e42000640 1 --2- 172.21.15.45:0/616641316 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e3c10baf0 0x7f7e3c10df90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7e34009ca0 tx=0x7f7e3402c180 comp rx=0 tx=0).stop 2024-03-28T17:50:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.957+0000 7f7e42000640 1 -- 172.21.15.45:0/616641316 shutdown_connections 2024-03-28T17:50:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.957+0000 7f7e42000640 1 --2- 172.21.15.45:0/616641316 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e3c10baf0 0x7f7e3c10df90 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.957+0000 7f7e42000640 1 --2- 172.21.15.45:0/616641316 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e3c102ff0 0x7f7e3c10b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.957+0000 7f7e42000640 1 --2- 172.21.15.45:0/616641316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e3c1026d0 0x7f7e3c102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.957+0000 7f7e42000640 1 -- 172.21.15.45:0/616641316 >> 172.21.15.45:0/616641316 conn(0x7f7e3c0fbef0 msgr2=0x7f7e3c0fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.957+0000 7f7e42000640 1 -- 172.21.15.45:0/616641316 shutdown_connections 2024-03-28T17:50:26.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.957+0000 7f7e42000640 1 -- 172.21.15.45:0/616641316 wait complete. 2024-03-28T17:50:26.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.958+0000 7f7e42000640 1 Processor -- start 2024-03-28T17:50:26.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.959+0000 7f7e42000640 1 -- start start 2024-03-28T17:50:26.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.959+0000 7f7e42000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e3c1026d0 0x7f7e3c100320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.959+0000 7f7e42000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e3c102ff0 0x7f7e3c100860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.959+0000 7f7e42000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e3c10baf0 0x7f7e3c100da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.959+0000 7f7e42000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c1118d0 con 0x7f7e3c102ff0 2024-03-28T17:50:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.959+0000 7f7e42000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c111750 con 0x7f7e3c10baf0 2024-03-28T17:50:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.959+0000 7f7e42000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c111a50 con 0x7f7e3c1026d0 2024-03-28T17:50:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.959+0000 7f7e40c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e3c1026d0 0x7f7e3c100320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.959+0000 7f7e41600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e3c10baf0 0x7f7e3c100da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.959+0000 7f7e40c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e3c1026d0 0x7f7e3c100320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:46902/0 (socket says 172.21.15.45:46902) 2024-03-28T17:50:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.959+0000 7f7e40c00640 1 -- 172.21.15.45:0/1452718544 learned_addr learned my addr 172.21.15.45:0/1452718544 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.960+0000 7f7e3be00640 1 --2- 172.21.15.45:0/1452718544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e3c102ff0 0x7f7e3c100860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.960+0000 7f7e41600640 1 -- 172.21.15.45:0/1452718544 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e3c1026d0 msgr2=0x7f7e3c100320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.960+0000 7f7e41600640 1 --2- 172.21.15.45:0/1452718544 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e3c1026d0 0x7f7e3c100320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.960+0000 7f7e41600640 1 -- 172.21.15.45:0/1452718544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e3c102ff0 msgr2=0x7f7e3c100860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.960+0000 7f7e41600640 1 --2- 172.21.15.45:0/1452718544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e3c102ff0 0x7f7e3c100860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.960+0000 7f7e41600640 1 -- 172.21.15.45:0/1452718544 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3c1a6250 con 0x7f7e3c10baf0 2024-03-28T17:50:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.960+0000 7f7e40c00640 1 --2- 172.21.15.45:0/1452718544 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e3c1026d0 0x7f7e3c100320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:50:26.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.960+0000 7f7e3be00640 1 --2- 172.21.15.45:0/1452718544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e3c102ff0 0x7f7e3c100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T17:50:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.960+0000 7f7e41600640 1 --2- 172.21.15.45:0/1452718544 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e3c10baf0 0x7f7e3c100da0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7e34009bf0 tx=0x7f7e34035d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.961+0000 7f7e39600640 1 -- 172.21.15.45:0/1452718544 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e34039040 con 0x7f7e3c10baf0 2024-03-28T17:50:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.961+0000 7f7e39600640 1 -- 172.21.15.45:0/1452718544 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e34044070 con 0x7f7e3c10baf0 2024-03-28T17:50:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.961+0000 7f7e39600640 1 -- 172.21.15.45:0/1452718544 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e340328d0 con 0x7f7e3c10baf0 2024-03-28T17:50:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.961+0000 7f7e42000640 1 -- 172.21.15.45:0/1452718544 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e3c1a64e0 con 0x7f7e3c10baf0 2024-03-28T17:50:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.961+0000 7f7e42000640 1 -- 172.21.15.45:0/1452718544 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e3c1a6950 con 0x7f7e3c10baf0 2024-03-28T17:50:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.964+0000 7f7e39600640 1 -- 172.21.15.45:0/1452718544 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 27) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f7e34032a70 con 0x7f7e3c10baf0 2024-03-28T17:50:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.965+0000 7f7e463b5740 1 Processor -- start 2024-03-28T17:50:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.965+0000 7f7e463b5740 1 -- start start 2024-03-28T17:50:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.965+0000 7f7e463b5740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ed8b0 0x555e440edc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.965+0000 7f7e463b5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e440ee260 0x555e440f6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.965+0000 7f7e463b5740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555e440f6de0 0x555e440f9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.965+0000 7f7e463b5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e440512c0 con 0x555e440ee260 2024-03-28T17:50:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.965+0000 7f7e463b5740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e43ff9c90 con 0x555e440f6de0 2024-03-28T17:50:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.965+0000 7f7e463b5740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4404d360 con 0x555e440ed8b0 2024-03-28T17:50:26.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.966+0000 7f7e3be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ed8b0 0x555e440edc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.966+0000 7f7e3be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ed8b0 0x555e440edc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:46918/0 (socket says 172.21.15.45:46918) 2024-03-28T17:50:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.966+0000 7f7e3be00640 1 -- 172.21.15.45:0/171132972 learned_addr learned my addr 172.21.15.45:0/171132972 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.966+0000 7f7e40c00640 1 --2- 172.21.15.45:0/171132972 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e440ee260 0x555e440f6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.966+0000 7f7e3be00640 1 --2- 172.21.15.45:0/171132972 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555e440f6de0 0x555e440f9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.966+0000 7f7e39600640 1 --2- 172.21.15.45:0/1452718544 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f7e240a2b10 0x7f7e240a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.966+0000 7f7e3be00640 1 -- 172.21.15.45:0/171132972 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555e440f6de0 msgr2=0x555e440f9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.966+0000 7f7e3be00640 1 --2- 172.21.15.45:0/171132972 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555e440f6de0 0x555e440f9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.966+0000 7f7e3be00640 1 -- 172.21.15.45:0/171132972 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e440ee260 msgr2=0x555e440f6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.966+0000 7f7e3be00640 1 --2- 172.21.15.45:0/171132972 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e440ee260 0x555e440f6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.966+0000 7f7e3be00640 1 -- 172.21.15.45:0/171132972 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e4401e080 con 0x555e440ed8b0 2024-03-28T17:50:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.966+0000 7f7e41600640 1 --2- 172.21.15.45:0/1452718544 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f7e240a2b10 0x7f7e240a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.966+0000 7f7e39600640 1 -- 172.21.15.45:0/1452718544 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f7e340f13f0 con 0x7f7e3c10baf0 2024-03-28T17:50:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.966+0000 7f7e3be00640 1 --2- 172.21.15.45:0/171132972 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ed8b0 0x555e440edc90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7e28009460 tx=0x7f7e280096b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14c0e19c3334a3ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:26.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.967+0000 7f7e2dc00640 1 -- 172.21.15.45:0/171132972 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e28014110 con 0x555e440ed8b0 2024-03-28T17:50:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.967+0000 7f7e2dc00640 1 -- 172.21.15.45:0/171132972 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e280142b0 con 0x555e440ed8b0 2024-03-28T17:50:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.967+0000 7f7e2dc00640 1 -- 172.21.15.45:0/171132972 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e280035f0 con 0x555e440ed8b0 2024-03-28T17:50:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.967+0000 7f7e463b5740 1 -- 172.21.15.45:0/171132972 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ed8b0 msgr2=0x555e440edc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.967+0000 7f7e41600640 1 --2- 172.21.15.45:0/1452718544 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f7e240a2b10 0x7f7e240a4fd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7e340eeb00 tx=0x7f7e3402c180 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.967+0000 7f7e463b5740 1 --2- 172.21.15.45:0/171132972 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ed8b0 0x555e440edc90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7e28009460 tx=0x7f7e280096b0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.967+0000 7f7e463b5740 1 -- 172.21.15.45:0/171132972 shutdown_connections 2024-03-28T17:50:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.967+0000 7f7e463b5740 1 --2- 172.21.15.45:0/171132972 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555e440f6de0 0x555e440f9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.967+0000 7f7e463b5740 1 --2- 172.21.15.45:0/171132972 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e440ee260 0x555e440f6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.967+0000 7f7e463b5740 1 --2- 172.21.15.45:0/171132972 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ed8b0 0x555e440edc90 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.967+0000 7f7e463b5740 1 -- 172.21.15.45:0/171132972 >> 172.21.15.45:0/171132972 conn(0x555e440e7f80 msgr2=0x555e440e8360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.968+0000 7f7e463b5740 1 -- 172.21.15.45:0/171132972 shutdown_connections 2024-03-28T17:50:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.968+0000 7f7e463b5740 1 -- 172.21.15.45:0/171132972 wait complete. 2024-03-28T17:50:26.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.968+0000 7f7e463b5740 1 -- start start 2024-03-28T17:50:26.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.968+0000 7f7e463b5740 1 Processor -- start 2024-03-28T17:50:26.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.969+0000 7f7e463b5740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ed8b0 0x555e440f6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.969+0000 7f7e463b5740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555e440ee260 0x555e440f3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.969+0000 7f7e463b5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e440f6de0 0x555e440f40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.969+0000 7f7e463b5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e440f9d50 con 0x555e440f6de0 2024-03-28T17:50:26.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.969+0000 7f7e463b5740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4400d0d0 con 0x555e440ee260 2024-03-28T17:50:26.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.969+0000 7f7e463b5740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e440f9ed0 con 0x555e440ed8b0 2024-03-28T17:50:26.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.969+0000 7f7e463b5740 10 client.0 _ll_register_callbacks cb 0x555e44011e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:50:26.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.969+0000 7f7e40c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555e440ee260 0x555e440f3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.969+0000 7f7e3be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ed8b0 0x555e440f6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.969+0000 7f7e40c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555e440ee260 0x555e440f3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:33366/0 (socket says 172.21.15.45:33366) 2024-03-28T17:50:26.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.969+0000 7f7e40c00640 1 -- 172.21.15.45:0/3486191488 learned_addr learned my addr 172.21.15.45:0/3486191488 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:26.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.969+0000 7f7e3be00640 1 --2- 172.21.15.45:0/3486191488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e440f6de0 0x555e440f40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.969+0000 7f7e40c00640 1 -- 172.21.15.45:0/3486191488 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ed8b0 msgr2=0x555e440f6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:26.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.969+0000 7f7e40c00640 1 --2- 172.21.15.45:0/3486191488 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ed8b0 0x555e440f6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.969+0000 7f7e40c00640 1 -- 172.21.15.45:0/3486191488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e440f6de0 msgr2=0x555e440f40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:26.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.969+0000 7f7e40c00640 1 --2- 172.21.15.45:0/3486191488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e440f6de0 0x555e440f40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.969+0000 7f7e40c00640 1 -- 172.21.15.45:0/3486191488 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555e43d653f0 con 0x555e440ee260 2024-03-28T17:50:26.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.969+0000 7f7e3be00640 1 --2- 172.21.15.45:0/3486191488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e440f6de0 0x555e440f40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:50:26.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.970+0000 7f7e40c00640 1 --2- 172.21.15.45:0/3486191488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555e440ee260 0x555e440f3ad0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7e30012d20 tx=0x7f7e3000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=324533dd718c0d64 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:26.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.970+0000 7f7e22000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:50:26.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.970+0000 7f7e22000640 1 -- 172.21.15.45:0/3486191488 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e30004050 con 0x555e440ee260 2024-03-28T17:50:26.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.970+0000 7f7e22000640 1 -- 172.21.15.45:0/3486191488 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f7e300041f0 con 0x555e440ee260 2024-03-28T17:50:26.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.970+0000 7f7e463b5740 1 -- 172.21.15.45:0/3486191488 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555e4405e800 con 0x555e440ee260 2024-03-28T17:50:26.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.970+0000 7f7e22000640 1 -- 172.21.15.45:0/3486191488 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e300043f0 con 0x555e440ee260 2024-03-28T17:50:26.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.970+0000 7f7e463b5740 10 client.25745 fetch_fsmap learned FSMap version 23 2024-03-28T17:50:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.970+0000 7f7e463b5740 1 -- 172.21.15.45:0/3486191488 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555e43e2f7d0 con 0x555e440ee260 2024-03-28T17:50:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.970+0000 7f7e22000640 1 -- 172.21.15.45:0/3486191488 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x555e4405e800 con 0x555e440ee260 2024-03-28T17:50:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.971+0000 7f7e22000640 1 -- 172.21.15.45:0/3486191488 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7e300104c0 con 0x555e440ee260 2024-03-28T17:50:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.971+0000 7f7e463b5740 10 client.25745 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:50:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.971+0000 7f7e463b5740 10 client.25745 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T17:50:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.971+0000 7f7e463b5740 20 client.25745 populate_metadata read hostname 'smithi045' 2024-03-28T17:50:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.971+0000 7f7e463b5740 1 --2- 172.21.15.45:0/3486191488 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x555e44106d00 0x555e44109120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.971+0000 7f7e463b5740 4 client.25745 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:50:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.971+0000 7f7e463b5740 1 -- 172.21.15.45:0/3486191488 --> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555e44058b50 con 0x555e44106d00 2024-03-28T17:50:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.971+0000 7f7e3be00640 1 --2- 172.21.15.45:0/3486191488 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x555e44106d00 0x555e44109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.972+0000 7f7e3be00640 1 --2- 172.21.15.45:0/3486191488 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x555e44106d00 0x555e44109120 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7109428269b7bd6a server_cookie=3f9f9bcb10dcefe3 in_seq=0 out_seq=0 2024-03-28T17:50:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.972+0000 7f7e22000640 10 client.25745 ms_handle_connect on v2:172.21.15.71:6836/2008752655 2024-03-28T17:50:26.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.972+0000 7f7e22000640 1 -- 172.21.15.45:0/3486191488 <== mds.1 v2:172.21.15.71:6836/2008752655 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555e44058b50 con 0x555e44106d00 2024-03-28T17:50:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.972+0000 7f7e22000640 10 client.25745 handle_command_reply: tid=0 2024-03-28T17:50:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.972+0000 7f7e463b5740 1 client.25745 shutdown 2024-03-28T17:50:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.972+0000 7f7e463b5740 1 -- 172.21.15.45:0/3486191488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555e440ee260 msgr2=0x555e440f3ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.972+0000 7f7e463b5740 1 --2- 172.21.15.45:0/3486191488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555e440ee260 0x555e440f3ad0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7e30012d20 tx=0x7f7e3000f790 comp rx=0 tx=0).stop 2024-03-28T17:50:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.973+0000 7f7e463b5740 1 -- 172.21.15.45:0/3486191488 shutdown_connections 2024-03-28T17:50:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.973+0000 7f7e463b5740 1 --2- 172.21.15.45:0/3486191488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e440f6de0 0x555e440f40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.973+0000 7f7e463b5740 1 --2- 172.21.15.45:0/3486191488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555e440ee260 0x555e440f3ad0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.973+0000 7f7e463b5740 1 --2- 172.21.15.45:0/3486191488 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x555e44106d00 0x555e44109120 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.973+0000 7f7e463b5740 1 --2- 172.21.15.45:0/3486191488 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ed8b0 0x555e440f6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.973+0000 7f7e22000640 0 client.25745 ms_handle_reset on v2:172.21.15.71:6836/2008752655 2024-03-28T17:50:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.973+0000 7f7e463b5740 1 -- 172.21.15.45:0/3486191488 >> 172.21.15.45:0/3486191488 conn(0x555e440e7f80 msgr2=0x555e440e9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.973+0000 7f7e463b5740 1 -- 172.21.15.45:0/3486191488 shutdown_connections 2024-03-28T17:50:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.973+0000 7f7e463b5740 1 -- 172.21.15.45:0/3486191488 wait complete. 2024-03-28T17:50:26.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.973+0000 7f7e463b5740 20 client.25745 trim_cache size 0 max 16384 2024-03-28T17:50:26.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.987+0000 7f7e463b5740 1 Processor -- start 2024-03-28T17:50:26.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.987+0000 7f7e463b5740 1 -- start start 2024-03-28T17:50:26.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.987+0000 7f7e463b5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e440ed8b0 0x555e440e3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.988+0000 7f7e463b5740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ee260 0x555e440f6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.988+0000 7f7e463b5740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555e440f6de0 0x555e440f6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.988+0000 7f7e463b5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c1b1740 con 0x555e440ed8b0 2024-03-28T17:50:26.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.988+0000 7f7e463b5740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c1af240 con 0x555e440f6de0 2024-03-28T17:50:26.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.988+0000 7f7e463b5740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c1ad2c0 con 0x555e440ee260 2024-03-28T17:50:26.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.988+0000 7f7e3be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e440ed8b0 0x555e440e3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.988+0000 7f7e40c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ee260 0x555e440f6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.988+0000 7f7e3be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e440ed8b0 0x555e440e3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42304/0 (socket says 172.21.15.45:42304) 2024-03-28T17:50:26.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.988+0000 7f7e40c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ee260 0x555e440f6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:46940/0 (socket says 172.21.15.45:46940) 2024-03-28T17:50:26.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.988+0000 7f7e3be00640 1 -- 172.21.15.45:0/980451807 learned_addr learned my addr 172.21.15.45:0/980451807 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:26.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.988+0000 7f7e3be00640 1 --2- 172.21.15.45:0/980451807 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555e440f6de0 0x555e440f6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.988+0000 7f7e40c00640 1 -- 172.21.15.45:0/980451807 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555e440f6de0 msgr2=0x555e440f6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:26.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.988+0000 7f7e40c00640 1 --2- 172.21.15.45:0/980451807 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555e440f6de0 0x555e440f6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.988+0000 7f7e40c00640 1 -- 172.21.15.45:0/980451807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e440ed8b0 msgr2=0x555e440e3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:26.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.988+0000 7f7e40c00640 1 --2- 172.21.15.45:0/980451807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e440ed8b0 0x555e440e3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.988+0000 7f7e40c00640 1 -- 172.21.15.45:0/980451807 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e440f4450 con 0x555e440ee260 2024-03-28T17:50:26.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.988+0000 7f7e3be00640 1 --2- 172.21.15.45:0/980451807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e440ed8b0 0x555e440e3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:50:26.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.989+0000 7f7e40c00640 1 --2- 172.21.15.45:0/980451807 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ee260 0x555e440f6200 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7e30010830 tx=0x7f7e3000fee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c74d352e1eaea9c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:26.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.989+0000 7f7e2d200640 1 -- 172.21.15.45:0/980451807 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e30029040 con 0x555e440ee260 2024-03-28T17:50:26.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.989+0000 7f7e2d200640 1 -- 172.21.15.45:0/980451807 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e300127f0 con 0x555e440ee260 2024-03-28T17:50:26.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.989+0000 7f7e2d200640 1 -- 172.21.15.45:0/980451807 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e300055e0 con 0x555e440ee260 2024-03-28T17:50:26.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.990+0000 7f7e463b5740 1 -- 172.21.15.45:0/980451807 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ee260 msgr2=0x555e440f6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:26.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.990+0000 7f7e463b5740 1 --2- 172.21.15.45:0/980451807 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ee260 0x555e440f6200 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7e30010830 tx=0x7f7e3000fee0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.990+0000 7f7e463b5740 1 -- 172.21.15.45:0/980451807 shutdown_connections 2024-03-28T17:50:26.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.990+0000 7f7e463b5740 1 --2- 172.21.15.45:0/980451807 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555e440f6de0 0x555e440f6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.990+0000 7f7e463b5740 1 --2- 172.21.15.45:0/980451807 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ee260 0x555e440f6200 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.990+0000 7f7e463b5740 1 --2- 172.21.15.45:0/980451807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e440ed8b0 0x555e440e3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.990+0000 7f7e463b5740 1 -- 172.21.15.45:0/980451807 >> 172.21.15.45:0/980451807 conn(0x555e440e7f80 msgr2=0x555e440e7160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:26.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.990+0000 7f7e463b5740 1 -- 172.21.15.45:0/980451807 shutdown_connections 2024-03-28T17:50:26.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.990+0000 7f7e463b5740 1 -- 172.21.15.45:0/980451807 wait complete. 2024-03-28T17:50:26.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.990+0000 7f7e463b5740 1 -- start start 2024-03-28T17:50:26.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.990+0000 7f7e463b5740 1 Processor -- start 2024-03-28T17:50:26.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.991+0000 7f7e463b5740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ed8b0 0x555e440fcca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.991+0000 7f7e463b5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e440ee260 0x555e440fd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.991+0000 7f7e463b5740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555e440f6de0 0x555e44161600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.991+0000 7f7e463b5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c1ad8c0 con 0x555e440ee260 2024-03-28T17:50:26.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.991+0000 7f7e463b5740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c1aebe0 con 0x555e440f6de0 2024-03-28T17:50:26.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.991+0000 7f7e463b5740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c1adf20 con 0x555e440ed8b0 2024-03-28T17:50:26.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.991+0000 7f7e463b5740 10 client.0 _ll_register_callbacks cb 0x7f7e30009cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:50:26.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.991+0000 7f7e3be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ed8b0 0x555e440fcca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.991+0000 7f7e3be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ed8b0 0x555e440fcca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:46956/0 (socket says 172.21.15.45:46956) 2024-03-28T17:50:26.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.991+0000 7f7e3be00640 1 -- 172.21.15.45:0/2080663219 learned_addr learned my addr 172.21.15.45:0/2080663219 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:26.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.991+0000 7f7e40c00640 1 --2- 172.21.15.45:0/2080663219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e440ee260 0x555e440fd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.991+0000 7f7e3be00640 1 --2- 172.21.15.45:0/2080663219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555e440f6de0 0x555e44161600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.992+0000 7f7e3be00640 1 -- 172.21.15.45:0/2080663219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555e440f6de0 msgr2=0x555e44161600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.992+0000 7f7e3be00640 1 --2- 172.21.15.45:0/2080663219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555e440f6de0 0x555e44161600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.992+0000 7f7e3be00640 1 -- 172.21.15.45:0/2080663219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e440ee260 msgr2=0x555e440fd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.992+0000 7f7e3be00640 1 --2- 172.21.15.45:0/2080663219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e440ee260 0x555e440fd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.992+0000 7f7e3be00640 1 -- 172.21.15.45:0/2080663219 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555e44161b90 con 0x555e440ed8b0 2024-03-28T17:50:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.992+0000 7f7e3be00640 1 --2- 172.21.15.45:0/2080663219 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ed8b0 0x555e440fcca0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7e28009fd0 tx=0x7f7e28016040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71980b4da3bda9aa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.992+0000 7f7e22000640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:50:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.992+0000 7f7e22000640 1 -- 172.21.15.45:0/2080663219 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2800fc40 con 0x555e440ed8b0 2024-03-28T17:50:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.992+0000 7f7e22000640 1 -- 172.21.15.45:0/2080663219 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f7e2802e4b0 con 0x555e440ed8b0 2024-03-28T17:50:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.992+0000 7f7e463b5740 1 -- 172.21.15.45:0/2080663219 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7e300127f0 con 0x555e440ed8b0 2024-03-28T17:50:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.992+0000 7f7e22000640 1 -- 172.21.15.45:0/2080663219 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e2800fde0 con 0x555e440ed8b0 2024-03-28T17:50:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.992+0000 7f7e463b5740 10 client.25720 fetch_fsmap learned FSMap version 23 2024-03-28T17:50:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.992+0000 7f7e463b5740 1 -- 172.21.15.45:0/2080663219 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555e44162e90 con 0x555e440ed8b0 2024-03-28T17:50:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.993+0000 7f7e22000640 1 -- 172.21.15.45:0/2080663219 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e300127f0 con 0x555e440ed8b0 2024-03-28T17:50:26.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.993+0000 7f7e22000640 1 -- 172.21.15.45:0/2080663219 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f7e2802e910 con 0x555e440ed8b0 2024-03-28T17:50:26.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.993+0000 7f7e463b5740 10 client.25720 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:50:26.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.993+0000 7f7e463b5740 10 client.25720 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T17:50:26.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.993+0000 7f7e463b5740 20 client.25720 populate_metadata read hostname 'smithi045' 2024-03-28T17:50:26.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.993+0000 7f7e463b5740 1 --2- 172.21.15.45:0/2080663219 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x555e441632a0 0x555e441656c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:26.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.993+0000 7f7e463b5740 4 client.25720 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T17:50:26.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.993+0000 7f7e463b5740 1 -- 172.21.15.45:0/2080663219 --> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555e43fd6500 con 0x555e441632a0 2024-03-28T17:50:26.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.993+0000 7f7e40c00640 1 --2- 172.21.15.45:0/2080663219 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x555e441632a0 0x555e441656c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:26.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.994+0000 7f7e40c00640 1 --2- 172.21.15.45:0/2080663219 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x555e441632a0 0x555e441656c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4004d8597c1d5cfd server_cookie=aa3af747e1980e9e in_seq=0 out_seq=0 2024-03-28T17:50:26.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.994+0000 7f7e22000640 10 client.25720 ms_handle_connect on v2:172.21.15.71:6836/2008752655 2024-03-28T17:50:26.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.994+0000 7f7e22000640 1 -- 172.21.15.45:0/2080663219 <== mds.1 v2:172.21.15.71:6836/2008752655 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555e43fd6500 con 0x555e441632a0 2024-03-28T17:50:26.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.994+0000 7f7e22000640 10 client.25720 handle_command_reply: tid=0 2024-03-28T17:50:26.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.994+0000 7f7e463b5740 1 client.25720 shutdown 2024-03-28T17:50:26.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.994+0000 7f7e463b5740 1 -- 172.21.15.45:0/2080663219 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ed8b0 msgr2=0x555e440fcca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:26.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.994+0000 7f7e463b5740 1 --2- 172.21.15.45:0/2080663219 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ed8b0 0x555e440fcca0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7e28009fd0 tx=0x7f7e28016040 comp rx=0 tx=0).stop 2024-03-28T17:50:26.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.994+0000 7f7e463b5740 1 -- 172.21.15.45:0/2080663219 shutdown_connections 2024-03-28T17:50:26.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.994+0000 7f7e463b5740 1 --2- 172.21.15.45:0/2080663219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555e440f6de0 0x555e44161600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.994+0000 7f7e463b5740 1 --2- 172.21.15.45:0/2080663219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e440ee260 0x555e440fd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.994+0000 7f7e463b5740 1 --2- 172.21.15.45:0/2080663219 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x555e441632a0 0x555e441656c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.994+0000 7f7e463b5740 1 --2- 172.21.15.45:0/2080663219 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555e440ed8b0 0x555e440fcca0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:26.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.994+0000 7f7e22000640 0 client.25720 ms_handle_reset on v2:172.21.15.71:6836/2008752655 2024-03-28T17:50:26.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.994+0000 7f7e463b5740 1 -- 172.21.15.45:0/2080663219 >> 172.21.15.45:0/2080663219 conn(0x555e440e7f80 msgr2=0x555e440e7160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:26.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.994+0000 7f7e463b5740 1 -- 172.21.15.45:0/2080663219 shutdown_connections 2024-03-28T17:50:26.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.994+0000 7f7e463b5740 1 -- 172.21.15.45:0/2080663219 wait complete. 2024-03-28T17:50:26.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.994+0000 7f7e463b5740 20 client.25720 trim_cache size 0 max 16384 2024-03-28T17:50:27.000 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-28T17:50:27.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.995+0000 7f7e42000640 1 -- 172.21.15.45:0/1452718544 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f7e240a2b10 msgr2=0x7f7e240a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:27.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.995+0000 7f7e42000640 1 --2- 172.21.15.45:0/1452718544 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f7e240a2b10 0x7f7e240a4fd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7e340eeb00 tx=0x7f7e3402c180 comp rx=0 tx=0).stop 2024-03-28T17:50:27.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.995+0000 7f7e42000640 1 -- 172.21.15.45:0/1452718544 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e3c10baf0 msgr2=0x7f7e3c100da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.995+0000 7f7e42000640 1 --2- 172.21.15.45:0/1452718544 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e3c10baf0 0x7f7e3c100da0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7e34009bf0 tx=0x7f7e34035d80 comp rx=0 tx=0).stop 2024-03-28T17:50:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.995+0000 7f7e42000640 1 -- 172.21.15.45:0/1452718544 shutdown_connections 2024-03-28T17:50:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.995+0000 7f7e42000640 1 --2- 172.21.15.45:0/1452718544 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f7e240a2b10 0x7f7e240a4fd0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.995+0000 7f7e42000640 1 --2- 172.21.15.45:0/1452718544 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e3c10baf0 0x7f7e3c100da0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.995+0000 7f7e42000640 1 --2- 172.21.15.45:0/1452718544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e3c102ff0 0x7f7e3c100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.995+0000 7f7e42000640 1 --2- 172.21.15.45:0/1452718544 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e3c1026d0 0x7f7e3c100320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.995+0000 7f7e42000640 1 -- 172.21.15.45:0/1452718544 >> 172.21.15.45:0/1452718544 conn(0x7f7e3c0fbef0 msgr2=0x7f7e3c0fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.995+0000 7f7e42000640 1 -- 172.21.15.45:0/1452718544 shutdown_connections 2024-03-28T17:50:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:26.995+0000 7f7e42000640 1 -- 172.21.15.45:0/1452718544 wait complete. 2024-03-28T17:50:27.012 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T17:50:27.012 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T17:50:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:26 smithi071 ceph-mon[36597]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-28T17:50:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:26 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3493390979' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:50:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:26 smithi151 ceph-mon[36481]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-28T17:50:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:26 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3493390979' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:50:27.216 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-28T17:50:27.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.218+0000 7ff815000640 1 -- 172.21.15.45:0/3389605111 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff810103840 msgr2=0x7ff810103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:27.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.218+0000 7ff815000640 1 --2- 172.21.15.45:0/3389605111 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff810103840 0x7ff810103cc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc009900 tx=0x7ff7fc02bc50 comp rx=0 tx=0).stop 2024-03-28T17:50:27.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.218+0000 7ff815000640 1 -- 172.21.15.45:0/3389605111 shutdown_connections 2024-03-28T17:50:27.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.218+0000 7ff815000640 1 --2- 172.21.15.45:0/3389605111 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff810104200 0x7ff810110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.218+0000 7ff815000640 1 --2- 172.21.15.45:0/3389605111 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff810103840 0x7ff810103cc0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.218+0000 7ff815000640 1 --2- 172.21.15.45:0/3389605111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff810109850 0x7ff810109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.218+0000 7ff815000640 1 -- 172.21.15.45:0/3389605111 >> 172.21.15.45:0/3389605111 conn(0x7ff8100ff350 msgr2=0x7ff810101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:27.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.219+0000 7ff815000640 1 -- 172.21.15.45:0/3389605111 shutdown_connections 2024-03-28T17:50:27.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.219+0000 7ff815000640 1 -- 172.21.15.45:0/3389605111 wait complete. 2024-03-28T17:50:27.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.220+0000 7ff815000640 1 Processor -- start 2024-03-28T17:50:27.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.220+0000 7ff815000640 1 -- start start 2024-03-28T17:50:27.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.220+0000 7ff815000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff810103840 0x7ff81019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:27.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.220+0000 7ff815000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff810104200 0x7ff81019a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:27.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.220+0000 7ff815000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff810109850 0x7ff81019ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:27.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.220+0000 7ff815000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff810113cb0 con 0x7ff810109850 2024-03-28T17:50:27.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.220+0000 7ff815000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff810113b30 con 0x7ff810103840 2024-03-28T17:50:27.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.220+0000 7ff815000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff810113e30 con 0x7ff810104200 2024-03-28T17:50:27.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.221+0000 7ff80de00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff810103840 0x7ff81019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:27.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.221+0000 7ff80de00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff810103840 0x7ff81019a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:33404/0 (socket says 172.21.15.45:33404) 2024-03-28T17:50:27.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.221+0000 7ff80de00640 1 -- 172.21.15.45:0/2074992741 learned_addr learned my addr 172.21.15.45:0/2074992741 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:27.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.221+0000 7ff80d400640 1 --2- 172.21.15.45:0/2074992741 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff810104200 0x7ff81019a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:27.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.221+0000 7ff80e800640 1 --2- 172.21.15.45:0/2074992741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff810109850 0x7ff81019ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:27.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.221+0000 7ff80de00640 1 -- 172.21.15.45:0/2074992741 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff810104200 msgr2=0x7ff81019a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:27.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.221+0000 7ff80de00640 1 --2- 172.21.15.45:0/2074992741 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff810104200 0x7ff81019a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.221+0000 7ff80de00640 1 -- 172.21.15.45:0/2074992741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff810109850 msgr2=0x7ff81019ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:27.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.221+0000 7ff80de00640 1 --2- 172.21.15.45:0/2074992741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff810109850 0x7ff81019ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.221+0000 7ff80de00640 1 -- 172.21.15.45:0/2074992741 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff81019f150 con 0x7ff810103840 2024-03-28T17:50:27.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.221+0000 7ff80e800640 1 --2- 172.21.15.45:0/2074992741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff810109850 0x7ff81019ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:50:27.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.221+0000 7ff80de00640 1 --2- 172.21.15.45:0/2074992741 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff810103840 0x7ff81019a1b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff80000e9f0 tx=0x7ff80000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:27.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.222+0000 7ff806a00640 1 -- 172.21.15.45:0/2074992741 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80000cd80 con 0x7ff810103840 2024-03-28T17:50:27.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.222+0000 7ff806a00640 1 -- 172.21.15.45:0/2074992741 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff800010910 con 0x7ff810103840 2024-03-28T17:50:27.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.222+0000 7ff806a00640 1 -- 172.21.15.45:0/2074992741 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff800018990 con 0x7ff810103840 2024-03-28T17:50:27.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.222+0000 7ff815000640 1 -- 172.21.15.45:0/2074992741 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff81019f440 con 0x7ff810103840 2024-03-28T17:50:27.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.222+0000 7ff815000640 1 -- 172.21.15.45:0/2074992741 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8101a6cd0 con 0x7ff810103840 2024-03-28T17:50:27.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.225+0000 7ff806a00640 1 -- 172.21.15.45:0/2074992741 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 27) v1 ==== 137523+0+0 (secure 0 0 0) 0x7ff800010ac0 con 0x7ff810103840 2024-03-28T17:50:27.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.225+0000 7ff817c7e740 1 Processor -- start 2024-03-28T17:50:27.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.225+0000 7ff817c7e740 1 -- start start 2024-03-28T17:50:27.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.226+0000 7ff817c7e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c5ad77a670 0x55c5ad77aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:27.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.226+0000 7ff817c7e740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77b020 0x55c5ad78be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:27.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.226+0000 7ff817c7e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad78c410 0x55c5ad78e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:27.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.226+0000 7ff817c7e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5ad6de2c0 con 0x55c5ad78c410 2024-03-28T17:50:27.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.226+0000 7ff817c7e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5ad686c90 con 0x55c5ad77a670 2024-03-28T17:50:27.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.226+0000 7ff817c7e740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5ad6da360 con 0x55c5ad77b020 2024-03-28T17:50:27.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.226+0000 7ff80e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c5ad77a670 0x55c5ad77aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:27.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.226+0000 7ff80e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c5ad77a670 0x55c5ad77aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:33412/0 (socket says 172.21.15.45:33412) 2024-03-28T17:50:27.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.226+0000 7ff80de00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77b020 0x55c5ad78be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:27.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.226+0000 7ff80e800640 1 -- 172.21.15.45:0/1058567323 learned_addr learned my addr 172.21.15.45:0/1058567323 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:27.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.226+0000 7ff80d400640 1 --2- 172.21.15.45:0/1058567323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad78c410 0x55c5ad78e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:27.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.226+0000 7ff80e800640 1 -- 172.21.15.45:0/1058567323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77b020 msgr2=0x55c5ad78be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:27.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.226+0000 7ff80e800640 1 --2- 172.21.15.45:0/1058567323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77b020 0x55c5ad78be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.226+0000 7ff80e800640 1 -- 172.21.15.45:0/1058567323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad78c410 msgr2=0x55c5ad78e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:27.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.226+0000 7ff80e800640 1 --2- 172.21.15.45:0/1058567323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad78c410 0x55c5ad78e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.226+0000 7ff80e800640 1 -- 172.21.15.45:0/1058567323 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5ad6ab080 con 0x55c5ad77a670 2024-03-28T17:50:27.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.227+0000 7ff80d400640 1 --2- 172.21.15.45:0/1058567323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad78c410 0x55c5ad78e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:50:27.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.227+0000 7ff806a00640 1 --2- 172.21.15.45:0/2074992741 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7ff7f40a2b10 0x7ff7f40a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:27.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.227+0000 7ff806a00640 1 -- 172.21.15.45:0/2074992741 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7ff8000cf410 con 0x7ff810103840 2024-03-28T17:50:27.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.227+0000 7ff80e800640 1 --2- 172.21.15.45:0/1058567323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c5ad77a670 0x55c5ad77aa50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff808009fd0 tx=0x7ff8080108e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8700ecdaff7ade54 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:27.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.227+0000 7ff80de00640 1 --2- 172.21.15.45:0/2074992741 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7ff7f40a2b10 0x7ff7f40a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:27.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.227+0000 7ff7fb400640 1 -- 172.21.15.45:0/1058567323 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff808011760 con 0x55c5ad77a670 2024-03-28T17:50:27.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.227+0000 7ff7fb400640 1 -- 172.21.15.45:0/1058567323 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff808011de0 con 0x55c5ad77a670 2024-03-28T17:50:27.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.227+0000 7ff7fb400640 1 -- 172.21.15.45:0/1058567323 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80800e3c0 con 0x55c5ad77a670 2024-03-28T17:50:27.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.228+0000 7ff817c7e740 1 -- 172.21.15.45:0/1058567323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c5ad77a670 msgr2=0x55c5ad77aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:27.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.228+0000 7ff817c7e740 1 --2- 172.21.15.45:0/1058567323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c5ad77a670 0x55c5ad77aa50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff808009fd0 tx=0x7ff8080108e0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.228+0000 7ff80de00640 1 --2- 172.21.15.45:0/2074992741 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7ff7f40a2b10 0x7ff7f40a4fd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff8000cca10 tx=0x7ff8000d2ea0 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:27.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.228+0000 7ff817c7e740 1 -- 172.21.15.45:0/1058567323 shutdown_connections 2024-03-28T17:50:27.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.228+0000 7ff817c7e740 1 --2- 172.21.15.45:0/1058567323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad78c410 0x55c5ad78e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.228+0000 7ff817c7e740 1 --2- 172.21.15.45:0/1058567323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77b020 0x55c5ad78be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.228+0000 7ff817c7e740 1 --2- 172.21.15.45:0/1058567323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c5ad77a670 0x55c5ad77aa50 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.228+0000 7ff817c7e740 1 -- 172.21.15.45:0/1058567323 >> 172.21.15.45:0/1058567323 conn(0x55c5ad774da0 msgr2=0x55c5ad775180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:27.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.228+0000 7ff817c7e740 1 -- 172.21.15.45:0/1058567323 shutdown_connections 2024-03-28T17:50:27.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.228+0000 7ff817c7e740 1 -- 172.21.15.45:0/1058567323 wait complete. 2024-03-28T17:50:27.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.229+0000 7ff817c7e740 1 -- start start 2024-03-28T17:50:27.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.229+0000 7ff817c7e740 1 Processor -- start 2024-03-28T17:50:27.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.229+0000 7ff817c7e740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77a670 0x55c5ad78bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:27.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.229+0000 7ff817c7e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad77b020 0x55c5ad7890c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:27.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.229+0000 7ff817c7e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c5ad78c410 0x55c5ad789660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:27.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.229+0000 7ff817c7e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5ad78f340 con 0x55c5ad77b020 2024-03-28T17:50:27.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.229+0000 7ff817c7e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5ad69a0d0 con 0x55c5ad78c410 2024-03-28T17:50:27.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.229+0000 7ff817c7e740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5ad78f4c0 con 0x55c5ad77a670 2024-03-28T17:50:27.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.230+0000 7ff817c7e740 10 client.0 _ll_register_callbacks cb 0x55c5ad69ee80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:50:27.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.230+0000 7ff80d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad77b020 0x55c5ad7890c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:27.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.230+0000 7ff80d400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad77b020 0x55c5ad7890c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42346/0 (socket says 172.21.15.45:42346) 2024-03-28T17:50:27.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.230+0000 7ff80e800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77a670 0x55c5ad78bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:27.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.230+0000 7ff80d400640 1 -- 172.21.15.45:0/2872645121 learned_addr learned my addr 172.21.15.45:0/2872645121 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:27.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.230+0000 7ff80e800640 1 --2- 172.21.15.45:0/2872645121 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c5ad78c410 0x55c5ad789660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:27.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.230+0000 7ff80d400640 1 -- 172.21.15.45:0/2872645121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77a670 msgr2=0x55c5ad78bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:27.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.230+0000 7ff80d400640 1 --2- 172.21.15.45:0/2872645121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77a670 0x55c5ad78bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.230+0000 7ff80d400640 1 -- 172.21.15.45:0/2872645121 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c5ad78c410 msgr2=0x55c5ad789660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:27.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.230+0000 7ff80d400640 1 --2- 172.21.15.45:0/2872645121 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c5ad78c410 0x55c5ad789660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.230+0000 7ff80d400640 1 -- 172.21.15.45:0/2872645121 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c5ad3f23f0 con 0x55c5ad77b020 2024-03-28T17:50:27.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.230+0000 7ff80e800640 1 --2- 172.21.15.45:0/2872645121 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c5ad78c410 0x55c5ad789660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:50:27.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.230+0000 7ff80d400640 1 --2- 172.21.15.45:0/2872645121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad77b020 0x55c5ad7890c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc02cfd0 tx=0x7ff7fc005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f32c8b501640b91 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:27.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.230+0000 7ff7efe00640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-28T17:50:27.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.231+0000 7ff7efe00640 1 -- 172.21.15.45:0/2872645121 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc02cad0 con 0x55c5ad77b020 2024-03-28T17:50:27.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.231+0000 7ff817c7e740 1 -- 172.21.15.45:0/2872645121 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff808011de0 con 0x55c5ad77b020 2024-03-28T17:50:27.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.231+0000 7ff7efe00640 1 -- 172.21.15.45:0/2872645121 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7ff7fc02cc70 con 0x55c5ad77b020 2024-03-28T17:50:27.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.231+0000 7ff7efe00640 1 -- 172.21.15.45:0/2872645121 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff7fc0046c0 con 0x55c5ad77b020 2024-03-28T17:50:27.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.231+0000 7ff817c7e740 10 client.15603 fetch_fsmap learned FSMap version 23 2024-03-28T17:50:27.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.231+0000 7ff817c7e740 1 -- 172.21.15.45:0/2872645121 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c5ad4bc7d0 con 0x55c5ad77b020 2024-03-28T17:50:27.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.232+0000 7ff7efe00640 1 -- 172.21.15.45:0/2872645121 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff808011de0 con 0x55c5ad77b020 2024-03-28T17:50:27.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.232+0000 7ff7efe00640 1 -- 172.21.15.45:0/2872645121 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ff7fc0035f0 con 0x55c5ad77b020 2024-03-28T17:50:27.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.232+0000 7ff817c7e740 10 client.15603 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:50:27.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.232+0000 7ff817c7e740 10 client.15603 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:50:27.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.232+0000 7ff817c7e740 20 client.15603 populate_metadata read hostname 'smithi045' 2024-03-28T17:50:27.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.232+0000 7ff817c7e740 1 --2- 172.21.15.45:0/2872645121 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55c5ad79c230 0x55c5ad79e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:27.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.232+0000 7ff817c7e740 4 client.15603 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:50:27.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.232+0000 7ff817c7e740 1 -- 172.21.15.45:0/2872645121 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c5ad6eb800 con 0x55c5ad79c230 2024-03-28T17:50:27.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.232+0000 7ff80e800640 1 --2- 172.21.15.45:0/2872645121 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55c5ad79c230 0x55c5ad79e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:27.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.233+0000 7ff80e800640 1 --2- 172.21.15.45:0/2872645121 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55c5ad79c230 0x55c5ad79e620 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c334f56a06e7875d server_cookie=d827c7a2d6ab8c54 in_seq=0 out_seq=0 2024-03-28T17:50:27.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.233+0000 7ff7efe00640 10 client.15603 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:50:27.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.233+0000 7ff7efe00640 1 -- 172.21.15.45:0/2872645121 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c5ad6eb800 con 0x55c5ad79c230 2024-03-28T17:50:27.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.233+0000 7ff7efe00640 10 client.15603 handle_command_reply: tid=0 2024-03-28T17:50:27.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.233+0000 7ff817c7e740 1 client.15603 shutdown 2024-03-28T17:50:27.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.234+0000 7ff817c7e740 1 -- 172.21.15.45:0/2872645121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad77b020 msgr2=0x55c5ad7890c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:27.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.234+0000 7ff817c7e740 1 --2- 172.21.15.45:0/2872645121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad77b020 0x55c5ad7890c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc02cfd0 tx=0x7ff7fc005e50 comp rx=0 tx=0).stop 2024-03-28T17:50:27.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.234+0000 7ff817c7e740 1 -- 172.21.15.45:0/2872645121 shutdown_connections 2024-03-28T17:50:27.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.234+0000 7ff817c7e740 1 --2- 172.21.15.45:0/2872645121 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55c5ad79c230 0x55c5ad79e620 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.234+0000 7ff817c7e740 1 --2- 172.21.15.45:0/2872645121 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c5ad78c410 0x55c5ad789660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.234+0000 7ff817c7e740 1 --2- 172.21.15.45:0/2872645121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad77b020 0x55c5ad7890c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.234+0000 7ff817c7e740 1 --2- 172.21.15.45:0/2872645121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77a670 0x55c5ad78bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.234+0000 7ff7efe00640 0 client.15603 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:50:27.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.234+0000 7ff817c7e740 1 -- 172.21.15.45:0/2872645121 >> 172.21.15.45:0/2872645121 conn(0x55c5ad774da0 msgr2=0x55c5ad776af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:27.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.234+0000 7ff817c7e740 1 -- 172.21.15.45:0/2872645121 shutdown_connections 2024-03-28T17:50:27.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.234+0000 7ff817c7e740 1 -- 172.21.15.45:0/2872645121 wait complete. 2024-03-28T17:50:27.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.234+0000 7ff817c7e740 20 client.15603 trim_cache size 0 max 16384 2024-03-28T17:50:27.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.248+0000 7ff817c7e740 1 Processor -- start 2024-03-28T17:50:27.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.248+0000 7ff817c7e740 1 -- start start 2024-03-28T17:50:27.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.248+0000 7ff817c7e740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77a670 0x55c5ad78b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:27.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.248+0000 7ff817c7e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad77b020 0x55c5ad78bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:27.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.248+0000 7ff817c7e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c5ad78c410 0x55c5ad786340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:27.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.248+0000 7ff817c7e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8101ad5e0 con 0x55c5ad77b020 2024-03-28T17:50:27.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.248+0000 7ff817c7e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8101ab0e0 con 0x55c5ad78c410 2024-03-28T17:50:27.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.248+0000 7ff817c7e740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8101a9180 con 0x55c5ad77a670 2024-03-28T17:50:27.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.248+0000 7ff80e800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77a670 0x55c5ad78b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:27.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.248+0000 7ff80e800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77a670 0x55c5ad78b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:46992/0 (socket says 172.21.15.45:46992) 2024-03-28T17:50:27.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.248+0000 7ff80e800640 1 -- 172.21.15.45:0/1870706034 learned_addr learned my addr 172.21.15.45:0/1870706034 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:27.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.248+0000 7ff80e800640 1 --2- 172.21.15.45:0/1870706034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c5ad78c410 0x55c5ad786340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:27.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.248+0000 7ff80d400640 1 --2- 172.21.15.45:0/1870706034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad77b020 0x55c5ad78bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:27.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.249+0000 7ff80e800640 1 -- 172.21.15.45:0/1870706034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c5ad78c410 msgr2=0x55c5ad786340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:27.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.249+0000 7ff80e800640 1 --2- 172.21.15.45:0/1870706034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c5ad78c410 0x55c5ad786340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.249+0000 7ff80e800640 1 -- 172.21.15.45:0/1870706034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad77b020 msgr2=0x55c5ad78bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:27.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.249+0000 7ff80e800640 1 --2- 172.21.15.45:0/1870706034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad77b020 0x55c5ad78bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.249+0000 7ff80e800640 1 -- 172.21.15.45:0/1870706034 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5ad79e990 con 0x55c5ad77a670 2024-03-28T17:50:27.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.249+0000 7ff80e800640 1 --2- 172.21.15.45:0/1870706034 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77a670 0x55c5ad78b7f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff808009fa0 tx=0x7ff80801d040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a925daa543f18c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:27.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.249+0000 7ff7faa00640 1 -- 172.21.15.45:0/1870706034 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80802e0b0 con 0x55c5ad77a670 2024-03-28T17:50:27.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.249+0000 7ff7faa00640 1 -- 172.21.15.45:0/1870706034 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff80802e250 con 0x55c5ad77a670 2024-03-28T17:50:27.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.249+0000 7ff7faa00640 1 -- 172.21.15.45:0/1870706034 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80802ea90 con 0x55c5ad77a670 2024-03-28T17:50:27.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.250+0000 7ff817c7e740 1 -- 172.21.15.45:0/1870706034 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77a670 msgr2=0x55c5ad78b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:27.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.250+0000 7ff817c7e740 1 --2- 172.21.15.45:0/1870706034 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77a670 0x55c5ad78b7f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff808009fa0 tx=0x7ff80801d040 comp rx=0 tx=0).stop 2024-03-28T17:50:27.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.250+0000 7ff817c7e740 1 -- 172.21.15.45:0/1870706034 shutdown_connections 2024-03-28T17:50:27.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.250+0000 7ff817c7e740 1 --2- 172.21.15.45:0/1870706034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c5ad78c410 0x55c5ad786340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.250+0000 7ff817c7e740 1 --2- 172.21.15.45:0/1870706034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad77b020 0x55c5ad78bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.250+0000 7ff817c7e740 1 --2- 172.21.15.45:0/1870706034 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77a670 0x55c5ad78b7f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.250+0000 7ff817c7e740 1 -- 172.21.15.45:0/1870706034 >> 172.21.15.45:0/1870706034 conn(0x55c5ad774da0 msgr2=0x55c5ad784670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:27.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.250+0000 7ff817c7e740 1 -- 172.21.15.45:0/1870706034 shutdown_connections 2024-03-28T17:50:27.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.250+0000 7ff817c7e740 1 -- 172.21.15.45:0/1870706034 wait complete. 2024-03-28T17:50:27.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.250+0000 7ff817c7e740 1 -- start start 2024-03-28T17:50:27.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.251+0000 7ff817c7e740 1 Processor -- start 2024-03-28T17:50:27.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.251+0000 7ff817c7e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c5ad77a670 0x55c5ad791d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:27.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.251+0000 7ff817c7e740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77b020 0x55c5ad792260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:27.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.251+0000 7ff817c7e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad78c410 0x55c5ad7927a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:27.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.251+0000 7ff817c7e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8101a9780 con 0x55c5ad78c410 2024-03-28T17:50:27.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.251+0000 7ff817c7e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8101aaa80 con 0x55c5ad77a670 2024-03-28T17:50:27.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.251+0000 7ff817c7e740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8101a9dc0 con 0x55c5ad77b020 2024-03-28T17:50:27.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.251+0000 7ff817c7e740 10 client.0 _ll_register_callbacks cb 0x7ff8080093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:50:27.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.251+0000 7ff80d400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77b020 0x55c5ad792260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:27.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.251+0000 7ff80e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c5ad77a670 0x55c5ad791d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:27.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.251+0000 7ff80d400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77b020 0x55c5ad792260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:46998/0 (socket says 172.21.15.45:46998) 2024-03-28T17:50:27.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.251+0000 7ff80d400640 1 -- 172.21.15.45:0/3323987047 learned_addr learned my addr 172.21.15.45:0/3323987047 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:27.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.251+0000 7ff80e800640 1 --2- 172.21.15.45:0/3323987047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad78c410 0x55c5ad7927a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:27.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.252+0000 7ff80d400640 1 -- 172.21.15.45:0/3323987047 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c5ad77a670 msgr2=0x55c5ad791d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:27.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.252+0000 7ff80d400640 1 --2- 172.21.15.45:0/3323987047 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c5ad77a670 0x55c5ad791d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.252+0000 7ff80d400640 1 -- 172.21.15.45:0/3323987047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad78c410 msgr2=0x55c5ad7927a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:27.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.252+0000 7ff80d400640 1 --2- 172.21.15.45:0/3323987047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad78c410 0x55c5ad7927a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.252+0000 7ff80d400640 1 -- 172.21.15.45:0/3323987047 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c5ad7f6bd0 con 0x55c5ad77b020 2024-03-28T17:50:27.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.252+0000 7ff80e800640 1 --2- 172.21.15.45:0/3323987047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad78c410 0x55c5ad7927a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:50:27.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.252+0000 7ff80d400640 1 --2- 172.21.15.45:0/3323987047 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77b020 0x55c5ad792260 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc02e010 tx=0x7ff7fc0038a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abbad9f3c18d583a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:27.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.252+0000 7ff7efe00640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:50:27.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.252+0000 7ff7efe00640 1 -- 172.21.15.45:0/3323987047 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc05d060 con 0x55c5ad77b020 2024-03-28T17:50:27.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.252+0000 7ff7efe00640 1 -- 172.21.15.45:0/3323987047 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7ff7fc004a60 con 0x55c5ad77b020 2024-03-28T17:50:27.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.252+0000 7ff817c7e740 1 -- 172.21.15.45:0/3323987047 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff80802e250 con 0x55c5ad77b020 2024-03-28T17:50:27.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.253+0000 7ff7efe00640 1 -- 172.21.15.45:0/3323987047 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff7fc003d40 con 0x55c5ad77b020 2024-03-28T17:50:27.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.253+0000 7ff817c7e740 10 client.25738 fetch_fsmap learned FSMap version 23 2024-03-28T17:50:27.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.253+0000 7ff817c7e740 1 -- 172.21.15.45:0/3323987047 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c5ad7f7eb0 con 0x55c5ad77b020 2024-03-28T17:50:27.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.253+0000 7ff7efe00640 1 -- 172.21.15.45:0/3323987047 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff80802e250 con 0x55c5ad77b020 2024-03-28T17:50:27.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.253+0000 7ff7efe00640 1 -- 172.21.15.45:0/3323987047 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7ff7fc0052c0 con 0x55c5ad77b020 2024-03-28T17:50:27.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.253+0000 7ff817c7e740 10 client.25738 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:50:27.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.253+0000 7ff817c7e740 10 client.25738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:50:27.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.253+0000 7ff817c7e740 20 client.25738 populate_metadata read hostname 'smithi045' 2024-03-28T17:50:27.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.253+0000 7ff817c7e740 1 --2- 172.21.15.45:0/3323987047 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55c5ad7f8550 0x55c5ad7fa970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:27.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.253+0000 7ff817c7e740 4 client.25738 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T17:50:27.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.253+0000 7ff817c7e740 1 -- 172.21.15.45:0/3323987047 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c5ad6e5b50 con 0x55c5ad7f8550 2024-03-28T17:50:27.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.253+0000 7ff80e800640 1 --2- 172.21.15.45:0/3323987047 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55c5ad7f8550 0x55c5ad7fa970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:27.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.254+0000 7ff80e800640 1 --2- 172.21.15.45:0/3323987047 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55c5ad7f8550 0x55c5ad7fa970 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f7ab102220c36a3 server_cookie=10491fea207849ff in_seq=0 out_seq=0 2024-03-28T17:50:27.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.254+0000 7ff7efe00640 10 client.25738 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:50:27.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.254+0000 7ff7efe00640 1 -- 172.21.15.45:0/3323987047 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c5ad6e5b50 con 0x55c5ad7f8550 2024-03-28T17:50:27.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.254+0000 7ff7efe00640 10 client.25738 handle_command_reply: tid=0 2024-03-28T17:50:27.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.254+0000 7ff817c7e740 1 client.25738 shutdown 2024-03-28T17:50:27.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.254+0000 7ff817c7e740 1 -- 172.21.15.45:0/3323987047 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77b020 msgr2=0x55c5ad792260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:27.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.254+0000 7ff817c7e740 1 --2- 172.21.15.45:0/3323987047 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77b020 0x55c5ad792260 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc02e010 tx=0x7ff7fc0038a0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.254+0000 7ff817c7e740 1 -- 172.21.15.45:0/3323987047 shutdown_connections 2024-03-28T17:50:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.254+0000 7ff817c7e740 1 --2- 172.21.15.45:0/3323987047 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55c5ad7f8550 0x55c5ad7fa970 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.254+0000 7ff817c7e740 1 --2- 172.21.15.45:0/3323987047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5ad78c410 0x55c5ad7927a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.254+0000 7ff817c7e740 1 --2- 172.21.15.45:0/3323987047 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55c5ad77b020 0x55c5ad792260 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.254+0000 7ff817c7e740 1 --2- 172.21.15.45:0/3323987047 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c5ad77a670 0x55c5ad791d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.254+0000 7ff7efe00640 0 client.25738 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:50:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.254+0000 7ff817c7e740 1 -- 172.21.15.45:0/3323987047 >> 172.21.15.45:0/3323987047 conn(0x55c5ad774da0 msgr2=0x55c5ad773620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.254+0000 7ff817c7e740 1 -- 172.21.15.45:0/3323987047 shutdown_connections 2024-03-28T17:50:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.254+0000 7ff817c7e740 1 -- 172.21.15.45:0/3323987047 wait complete. 2024-03-28T17:50:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.255+0000 7ff817c7e740 20 client.25738 trim_cache size 0 max 16384 2024-03-28T17:50:27.259 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-28T17:50:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.255+0000 7ff815000640 1 -- 172.21.15.45:0/2074992741 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7ff7f40a2b10 msgr2=0x7ff7f40a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.255+0000 7ff815000640 1 --2- 172.21.15.45:0/2074992741 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7ff7f40a2b10 0x7ff7f40a4fd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff8000cca10 tx=0x7ff8000d2ea0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.255+0000 7ff815000640 1 -- 172.21.15.45:0/2074992741 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff810103840 msgr2=0x7ff81019a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.255+0000 7ff815000640 1 --2- 172.21.15.45:0/2074992741 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff810103840 0x7ff81019a1b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff80000e9f0 tx=0x7ff80000eeb0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.255+0000 7ff815000640 1 -- 172.21.15.45:0/2074992741 shutdown_connections 2024-03-28T17:50:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.255+0000 7ff815000640 1 --2- 172.21.15.45:0/2074992741 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7ff7f40a2b10 0x7ff7f40a4fd0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.255+0000 7ff815000640 1 --2- 172.21.15.45:0/2074992741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff810109850 0x7ff81019ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.255+0000 7ff815000640 1 --2- 172.21.15.45:0/2074992741 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff810104200 0x7ff81019a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.255+0000 7ff815000640 1 --2- 172.21.15.45:0/2074992741 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff810103840 0x7ff81019a1b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:27.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.255+0000 7ff815000640 1 -- 172.21.15.45:0/2074992741 >> 172.21.15.45:0/2074992741 conn(0x7ff8100ff350 msgr2=0x7ff8100ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:27.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.255+0000 7ff815000640 1 -- 172.21.15.45:0/2074992741 shutdown_connections 2024-03-28T17:50:27.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:27.255+0000 7ff815000640 1 -- 172.21.15.45:0/2074992741 wait complete. 2024-03-28T17:50:27.392 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-28T17:50:27.484 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T17:50:27.490 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T17:50:27.490 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-28T17:50:27.578 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T17:50:27.587 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-28T17:50:27.587 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-28T17:50:27.638 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-03-28T17:50:27.639 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-03-28T17:50:27.639 DEBUG:teuthology.orchestra.run.smithi045:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-03-28T17:50:27.711 INFO:tasks.workunit.client.0.smithi045.stderr:+ dbench 1 2024-03-28T17:50:27.722 INFO:tasks.workunit.client.0.smithi045.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-28T17:50:27.723 INFO:tasks.workunit.client.0.smithi045.stdout: 2024-03-28T17:50:27.723 INFO:tasks.workunit.client.0.smithi045.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-03-28T17:50:27.731 INFO:tasks.workunit.client.0.smithi045.stdout:0 of 1 processes prepared for launch 0 sec 2024-03-28T17:50:27.832 INFO:tasks.workunit.client.0.smithi045.stdout:1 of 1 processes prepared for launch 0 sec 2024-03-28T17:50:27.832 INFO:tasks.workunit.client.0.smithi045.stdout:releasing clients 2024-03-28T17:50:28.275 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T17:50:28.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T17:50:28.275 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T17:50:28.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.522+0000 7f5c09400640 1 -- 172.21.15.45:0/2916192723 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c04101520 msgr2=0x7f5c041019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:28.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.522+0000 7f5c09400640 1 --2- 172.21.15.45:0/2916192723 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c04101520 0x7f5c041019a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0009930 tx=0x7f5bf002bc20 comp rx=0 tx=0).stop 2024-03-28T17:50:28.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.522+0000 7f5c09400640 1 -- 172.21.15.45:0/2916192723 shutdown_connections 2024-03-28T17:50:28.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.522+0000 7f5c09400640 1 --2- 172.21.15.45:0/2916192723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c04101ee0 0x7f5c0410e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.522+0000 7f5c09400640 1 --2- 172.21.15.45:0/2916192723 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c04101520 0x7f5c041019a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.522+0000 7f5c09400640 1 --2- 172.21.15.45:0/2916192723 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c041075e0 0x7f5c041079c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.522+0000 7f5c09400640 1 -- 172.21.15.45:0/2916192723 >> 172.21.15.45:0/2916192723 conn(0x7f5c040fd010 msgr2=0x7f5c040ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:28.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.522+0000 7f5c09400640 1 -- 172.21.15.45:0/2916192723 shutdown_connections 2024-03-28T17:50:28.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.522+0000 7f5c09400640 1 -- 172.21.15.45:0/2916192723 wait complete. 2024-03-28T17:50:28.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.523+0000 7f5c09400640 1 Processor -- start 2024-03-28T17:50:28.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.523+0000 7f5c09400640 1 -- start start 2024-03-28T17:50:28.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.524+0000 7f5c09400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c04101520 0x7f5c04199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.524+0000 7f5c09400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c04101ee0 0x7f5c0419a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.524+0000 7f5c09400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c041075e0 0x7f5c0419e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.524+0000 7f5c09400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c041118f0 con 0x7f5c04101ee0 2024-03-28T17:50:28.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.524+0000 7f5c09400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c04111770 con 0x7f5c04101520 2024-03-28T17:50:28.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.524+0000 7f5c09400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c04111a70 con 0x7f5c041075e0 2024-03-28T17:50:28.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.524+0000 7f5c03e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c04101520 0x7f5c04199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.524+0000 7f5c08a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c041075e0 0x7f5c0419e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.524+0000 7f5c03e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c04101520 0x7f5c04199fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:33458/0 (socket says 172.21.15.45:33458) 2024-03-28T17:50:28.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.524+0000 7f5c03e00640 1 -- 172.21.15.45:0/48973368 learned_addr learned my addr 172.21.15.45:0/48973368 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:28.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.524+0000 7f5c03400640 1 --2- 172.21.15.45:0/48973368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c04101ee0 0x7f5c0419a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.524+0000 7f5c03e00640 1 -- 172.21.15.45:0/48973368 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c041075e0 msgr2=0x7f5c0419e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:28.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.524+0000 7f5c03e00640 1 --2- 172.21.15.45:0/48973368 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c041075e0 0x7f5c0419e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.524+0000 7f5c03e00640 1 -- 172.21.15.45:0/48973368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c04101ee0 msgr2=0x7f5c0419a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:28.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.524+0000 7f5c03e00640 1 --2- 172.21.15.45:0/48973368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c04101ee0 0x7f5c0419a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.524+0000 7f5c03e00640 1 -- 172.21.15.45:0/48973368 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c0419efa0 con 0x7f5c04101520 2024-03-28T17:50:28.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.524+0000 7f5c08a00640 1 --2- 172.21.15.45:0/48973368 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c041075e0 0x7f5c0419e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:50:28.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.525+0000 7f5c03e00640 1 --2- 172.21.15.45:0/48973368 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c04101520 0x7f5c04199fb0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400e9f0 tx=0x7f5bf400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:28.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.525+0000 7f5c00c00640 1 -- 172.21.15.45:0/48973368 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf400cd80 con 0x7f5c04101520 2024-03-28T17:50:28.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.525+0000 7f5c00c00640 1 -- 172.21.15.45:0/48973368 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5bf4010910 con 0x7f5c04101520 2024-03-28T17:50:28.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.525+0000 7f5c00c00640 1 -- 172.21.15.45:0/48973368 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4018990 con 0x7f5c04101520 2024-03-28T17:50:28.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.525+0000 7f5c09400640 1 -- 172.21.15.45:0/48973368 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c0419f290 con 0x7f5c04101520 2024-03-28T17:50:28.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.525+0000 7f5c09400640 1 -- 172.21.15.45:0/48973368 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c041a6b20 con 0x7f5c04101520 2024-03-28T17:50:28.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.530+0000 7f5c0d322740 1 Processor -- start 2024-03-28T17:50:28.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.530+0000 7f5c0d322740 1 -- start start 2024-03-28T17:50:28.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.531+0000 7f5c00c00640 1 -- 172.21.15.45:0/48973368 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 27) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f5bf4010ac0 con 0x7f5c04101520 2024-03-28T17:50:28.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.531+0000 7f5c0d322740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a126e290 0x5617a126e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.531+0000 7f5c0d322740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5617a126ebb0 0x5617a1277250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.531+0000 7f5c0d322740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a1277790 0x5617a1279b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.531+0000 7f5c0d322740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617a11d2eb0 con 0x5617a1277790 2024-03-28T17:50:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.531+0000 7f5c0d322740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617a117ba60 con 0x5617a126e290 2024-03-28T17:50:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.531+0000 7f5c0d322740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617a11cef50 con 0x5617a126ebb0 2024-03-28T17:50:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.531+0000 7f5c08a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a126e290 0x5617a126e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.531+0000 7f5c03e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5617a126ebb0 0x5617a1277250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.531+0000 7f5c08a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a126e290 0x5617a126e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:33462/0 (socket says 172.21.15.45:33462) 2024-03-28T17:50:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.531+0000 7f5c08a00640 1 -- 172.21.15.45:0/1171759085 learned_addr learned my addr 172.21.15.45:0/1171759085 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.531+0000 7f5c03400640 1 --2- 172.21.15.45:0/1171759085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a1277790 0x5617a1279b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.531+0000 7f5c08a00640 1 -- 172.21.15.45:0/1171759085 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5617a126ebb0 msgr2=0x5617a1277250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.531+0000 7f5c08a00640 1 --2- 172.21.15.45:0/1171759085 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5617a126ebb0 0x5617a1277250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.531+0000 7f5c08a00640 1 -- 172.21.15.45:0/1171759085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a1277790 msgr2=0x5617a1279b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.532+0000 7f5c08a00640 1 --2- 172.21.15.45:0/1171759085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a1277790 0x5617a1279b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.532+0000 7f5c08a00640 1 -- 172.21.15.45:0/1171759085 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617a0faf7e0 con 0x5617a126e290 2024-03-28T17:50:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.532+0000 7f5c08a00640 1 --2- 172.21.15.45:0/1171759085 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a126e290 0x5617a126e670 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc0117e0 tx=0x7f5bfc011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bf5c93bdb088c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.532+0000 7f5bf9200640 1 -- 172.21.15.45:0/1171759085 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc011310 con 0x5617a126e290 2024-03-28T17:50:28.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.532+0000 7f5bf9200640 1 -- 172.21.15.45:0/1171759085 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5bfc003e50 con 0x5617a126e290 2024-03-28T17:50:28.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.532+0000 7f5bf9200640 1 -- 172.21.15.45:0/1171759085 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc004620 con 0x5617a126e290 2024-03-28T17:50:28.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.532+0000 7f5c00c00640 1 --2- 172.21.15.45:0/48973368 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f5be80a2a90 0x7f5be80a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.533+0000 7f5c00c00640 1 -- 172.21.15.45:0/48973368 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f5bf40cf490 con 0x7f5c04101520 2024-03-28T17:50:28.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.533+0000 7f5c03e00640 1 --2- 172.21.15.45:0/48973368 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f5be80a2a90 0x7f5be80a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.533+0000 7f5c0d322740 1 -- 172.21.15.45:0/1171759085 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a126e290 msgr2=0x5617a126e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:28.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.533+0000 7f5c0d322740 1 --2- 172.21.15.45:0/1171759085 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a126e290 0x5617a126e670 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc0117e0 tx=0x7f5bfc011ca0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.533+0000 7f5c0d322740 1 -- 172.21.15.45:0/1171759085 shutdown_connections 2024-03-28T17:50:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.533+0000 7f5c0d322740 1 --2- 172.21.15.45:0/1171759085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a1277790 0x5617a1279b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.533+0000 7f5c0d322740 1 --2- 172.21.15.45:0/1171759085 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5617a126ebb0 0x5617a1277250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.533+0000 7f5c0d322740 1 --2- 172.21.15.45:0/1171759085 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a126e290 0x5617a126e670 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.533+0000 7f5c0d322740 1 -- 172.21.15.45:0/1171759085 >> 172.21.15.45:0/1171759085 conn(0x5617a12689c0 msgr2=0x5617a1268da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.533+0000 7f5c0d322740 1 -- 172.21.15.45:0/1171759085 shutdown_connections 2024-03-28T17:50:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.533+0000 7f5c0d322740 1 -- 172.21.15.45:0/1171759085 wait complete. 2024-03-28T17:50:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.534+0000 7f5c03e00640 1 --2- 172.21.15.45:0/48973368 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f5be80a2a90 0x7f5be80a4f50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5bf40cca90 tx=0x7f5bf40d2f20 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.534+0000 7f5c0d322740 1 -- start start 2024-03-28T17:50:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.534+0000 7f5c0d322740 1 Processor -- start 2024-03-28T17:50:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.535+0000 7f5c0d322740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a126e290 0x5617a1277160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.535+0000 7f5c0d322740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5617a126ebb0 0x5617a12744f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.535+0000 7f5c0d322740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a1277790 0x5617a1274b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.535+0000 7f5c0d322740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617a127a580 con 0x5617a1277790 2024-03-28T17:50:28.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.535+0000 7f5c0d322740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617a118eeb0 con 0x5617a126e290 2024-03-28T17:50:28.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.535+0000 7f5c0d322740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617a127a700 con 0x5617a126ebb0 2024-03-28T17:50:28.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.535+0000 7f5c0d322740 10 client.0 _ll_register_callbacks cb 0x5617a1183150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:50:28.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.535+0000 7f5c08a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a126e290 0x5617a1277160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.535+0000 7f5c08a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a126e290 0x5617a1277160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:33470/0 (socket says 172.21.15.45:33470) 2024-03-28T17:50:28.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.535+0000 7f5c03400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5617a126ebb0 0x5617a12744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.535+0000 7f5c08a00640 1 -- 172.21.15.45:0/907849557 learned_addr learned my addr 172.21.15.45:0/907849557 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:28.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.535+0000 7f5c08a00640 1 --2- 172.21.15.45:0/907849557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a1277790 0x5617a1274b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.535+0000 7f5c03400640 1 -- 172.21.15.45:0/907849557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a126e290 msgr2=0x5617a1277160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:28.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.535+0000 7f5c03400640 1 --2- 172.21.15.45:0/907849557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a126e290 0x5617a1277160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.535+0000 7f5c03400640 1 -- 172.21.15.45:0/907849557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a1277790 msgr2=0x5617a1274b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:28.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.535+0000 7f5c03400640 1 --2- 172.21.15.45:0/907849557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a1277790 0x5617a1274b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.535+0000 7f5c03400640 1 -- 172.21.15.45:0/907849557 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5617a100ff70 con 0x5617a126ebb0 2024-03-28T17:50:28.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.536+0000 7f5c03400640 1 --2- 172.21.15.45:0/907849557 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5617a126ebb0 0x5617a12744f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0036040 tx=0x7f5bf0004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7756cbe1e96d57fc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:28.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.536+0000 7f5bed600640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:50:28.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.536+0000 7f5bed600640 1 -- 172.21.15.45:0/907849557 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf002ca80 con 0x5617a126ebb0 2024-03-28T17:50:28.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.536+0000 7f5bed600640 1 -- 172.21.15.45:0/907849557 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f5bf002cc20 con 0x5617a126ebb0 2024-03-28T17:50:28.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.536+0000 7f5c0d322740 1 -- 172.21.15.45:0/907849557 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5bfc003e50 con 0x5617a126ebb0 2024-03-28T17:50:28.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.536+0000 7f5bed600640 1 -- 172.21.15.45:0/907849557 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5bf00046c0 con 0x5617a126ebb0 2024-03-28T17:50:28.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.536+0000 7f5c0d322740 10 client.25750 fetch_fsmap learned FSMap version 23 2024-03-28T17:50:28.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.536+0000 7f5c0d322740 1 -- 172.21.15.45:0/907849557 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5617a0ea7170 con 0x5617a126ebb0 2024-03-28T17:50:28.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.537+0000 7f5bed600640 1 -- 172.21.15.45:0/907849557 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5bfc003e50 con 0x5617a126ebb0 2024-03-28T17:50:28.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.537+0000 7f5bed600640 1 -- 172.21.15.45:0/907849557 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f5bf0051050 con 0x5617a126ebb0 2024-03-28T17:50:28.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.537+0000 7f5c0d322740 10 client.25750 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:50:28.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.537+0000 7f5c0d322740 10 client.25750 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:50:28.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.537+0000 7f5c0d322740 20 client.25750 populate_metadata read hostname 'smithi045' 2024-03-28T17:50:28.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.537+0000 7f5c0d322740 1 --2- 172.21.15.45:0/907849557 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5617a1287840 0x5617a1289c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.537+0000 7f5c0d322740 4 client.25750 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:50:28.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.537+0000 7f5c0d322740 1 -- 172.21.15.45:0/907849557 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5617a11e03f0 con 0x5617a1287840 2024-03-28T17:50:28.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.537+0000 7f5c08a00640 1 --2- 172.21.15.45:0/907849557 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5617a1287840 0x5617a1289c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.538+0000 7f5c08a00640 1 --2- 172.21.15.45:0/907849557 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5617a1287840 0x5617a1289c30 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=750bd1cfe1c9d66e server_cookie=798282dd68b2f169 in_seq=0 out_seq=0 2024-03-28T17:50:28.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.538+0000 7f5bed600640 10 client.25750 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:50:28.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.539+0000 7f5bed600640 1 -- 172.21.15.45:0/907849557 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5617a11e03f0 con 0x5617a1287840 2024-03-28T17:50:28.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.539+0000 7f5bed600640 10 client.25750 handle_command_reply: tid=0 2024-03-28T17:50:28.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.539+0000 7f5c0d322740 1 client.25750 shutdown 2024-03-28T17:50:28.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.540+0000 7f5c0d322740 1 -- 172.21.15.45:0/907849557 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5617a126ebb0 msgr2=0x5617a12744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:28.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.540+0000 7f5c0d322740 1 --2- 172.21.15.45:0/907849557 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5617a126ebb0 0x5617a12744f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0036040 tx=0x7f5bf0004610 comp rx=0 tx=0).stop 2024-03-28T17:50:28.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.540+0000 7f5c0d322740 1 -- 172.21.15.45:0/907849557 shutdown_connections 2024-03-28T17:50:28.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.540+0000 7f5c0d322740 1 --2- 172.21.15.45:0/907849557 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5617a1287840 0x5617a1289c30 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.540+0000 7f5c0d322740 1 --2- 172.21.15.45:0/907849557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a1277790 0x5617a1274b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.540+0000 7f5c0d322740 1 --2- 172.21.15.45:0/907849557 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5617a126ebb0 0x5617a12744f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.540+0000 7f5c0d322740 1 --2- 172.21.15.45:0/907849557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a126e290 0x5617a1277160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.540+0000 7f5bed600640 0 client.25750 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:50:28.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.540+0000 7f5c0d322740 1 -- 172.21.15.45:0/907849557 >> 172.21.15.45:0/907849557 conn(0x5617a12689c0 msgr2=0x5617a126a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:28.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.540+0000 7f5c0d322740 1 -- 172.21.15.45:0/907849557 shutdown_connections 2024-03-28T17:50:28.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.540+0000 7f5c0d322740 1 -- 172.21.15.45:0/907849557 wait complete. 2024-03-28T17:50:28.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.540+0000 7f5c0d322740 20 client.25750 trim_cache size 0 max 16384 2024-03-28T17:50:28.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.559+0000 7f5c0d322740 1 Processor -- start 2024-03-28T17:50:28.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.559+0000 7f5c0d322740 1 -- start start 2024-03-28T17:50:28.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.560+0000 7f5c0d322740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a126e290 0x5617a1276c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.560+0000 7f5c0d322740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a126ebb0 0x5617a1277160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.560+0000 7f5c0d322740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5617a1277790 0x5617a1282e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.560+0000 7f5c0d322740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c041b17c0 con 0x5617a126ebb0 2024-03-28T17:50:28.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.560+0000 7f5c0d322740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c041af2c0 con 0x5617a126e290 2024-03-28T17:50:28.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.560+0000 7f5c0d322740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c041ad340 con 0x5617a1277790 2024-03-28T17:50:28.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.560+0000 7f5c08a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a126e290 0x5617a1276c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.560+0000 7f5c03400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a126ebb0 0x5617a1277160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.560+0000 7f5c08a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a126e290 0x5617a1276c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:33482/0 (socket says 172.21.15.45:33482) 2024-03-28T17:50:28.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.560+0000 7f5c08a00640 1 -- 172.21.15.45:0/3243543293 learned_addr learned my addr 172.21.15.45:0/3243543293 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:28.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.560+0000 7f5c08a00640 1 --2- 172.21.15.45:0/3243543293 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5617a1277790 0x5617a1282e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.560+0000 7f5c08a00640 1 -- 172.21.15.45:0/3243543293 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5617a1277790 msgr2=0x5617a1282e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:28.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.560+0000 7f5c08a00640 1 --2- 172.21.15.45:0/3243543293 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5617a1277790 0x5617a1282e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.560+0000 7f5c08a00640 1 -- 172.21.15.45:0/3243543293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a126ebb0 msgr2=0x5617a1277160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:28.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.560+0000 7f5c08a00640 1 --2- 172.21.15.45:0/3243543293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a126ebb0 0x5617a1277160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.560+0000 7f5c08a00640 1 -- 172.21.15.45:0/3243543293 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617a1267d30 con 0x5617a126e290 2024-03-28T17:50:28.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.560+0000 7f5c03400640 1 --2- 172.21.15.45:0/3243543293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a126ebb0 0x5617a1277160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:50:28.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.561+0000 7f5c08a00640 1 --2- 172.21.15.45:0/3243543293 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a126e290 0x5617a1276c20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc004100 tx=0x7f5bfc01c040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4313c0dd7f6863ee server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:28.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.561+0000 7f5befe00640 1 -- 172.21.15.45:0/3243543293 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc003e30 con 0x5617a126e290 2024-03-28T17:50:28.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.561+0000 7f5befe00640 1 -- 172.21.15.45:0/3243543293 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5bfc00fe80 con 0x5617a126e290 2024-03-28T17:50:28.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.561+0000 7f5befe00640 1 -- 172.21.15.45:0/3243543293 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc005860 con 0x5617a126e290 2024-03-28T17:50:28.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.562+0000 7f5c0d322740 1 -- 172.21.15.45:0/3243543293 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a126e290 msgr2=0x5617a1276c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:28.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.562+0000 7f5c0d322740 1 --2- 172.21.15.45:0/3243543293 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a126e290 0x5617a1276c20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc004100 tx=0x7f5bfc01c040 comp rx=0 tx=0).stop 2024-03-28T17:50:28.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.562+0000 7f5c0d322740 1 -- 172.21.15.45:0/3243543293 shutdown_connections 2024-03-28T17:50:28.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.562+0000 7f5c0d322740 1 --2- 172.21.15.45:0/3243543293 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5617a1277790 0x5617a1282e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.562+0000 7f5c0d322740 1 --2- 172.21.15.45:0/3243543293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a126ebb0 0x5617a1277160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.562+0000 7f5c0d322740 1 --2- 172.21.15.45:0/3243543293 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a126e290 0x5617a1276c20 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.562+0000 7f5c0d322740 1 -- 172.21.15.45:0/3243543293 >> 172.21.15.45:0/3243543293 conn(0x5617a12689c0 msgr2=0x5617a126a480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:28.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.562+0000 7f5c0d322740 1 -- 172.21.15.45:0/3243543293 shutdown_connections 2024-03-28T17:50:28.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.562+0000 7f5c0d322740 1 -- 172.21.15.45:0/3243543293 wait complete. 2024-03-28T17:50:28.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.562+0000 7f5c0d322740 1 -- start start 2024-03-28T17:50:28.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.563+0000 7f5c0d322740 1 Processor -- start 2024-03-28T17:50:28.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.563+0000 7f5c0d322740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5617a126e290 0x5617a127bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.563+0000 7f5c0d322740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a126ebb0 0x5617a127c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.563+0000 7f5c0d322740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a1277790 0x5617a127c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.563+0000 7f5c0d322740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c041ad940 con 0x5617a126ebb0 2024-03-28T17:50:28.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.563+0000 7f5c0d322740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c041aec60 con 0x5617a1277790 2024-03-28T17:50:28.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.563+0000 7f5c0d322740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c041adfa0 con 0x5617a126e290 2024-03-28T17:50:28.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.563+0000 7f5c0d322740 10 client.0 _ll_register_callbacks cb 0x7f5bf4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:50:28.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.563+0000 7f5c08a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5617a126e290 0x5617a127bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.564+0000 7f5c08a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5617a126e290 0x5617a127bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:47050/0 (socket says 172.21.15.45:47050) 2024-03-28T17:50:28.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.564+0000 7f5c08a00640 1 -- 172.21.15.45:0/1870713610 learned_addr learned my addr 172.21.15.45:0/1870713610 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:28.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.564+0000 7f5c03400640 1 --2- 172.21.15.45:0/1870713610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a126ebb0 0x5617a127c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.564+0000 7f5c08a00640 1 --2- 172.21.15.45:0/1870713610 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a1277790 0x5617a127c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.564+0000 7f5c08a00640 1 -- 172.21.15.45:0/1870713610 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a1277790 msgr2=0x5617a127c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:28.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.564+0000 7f5c08a00640 1 --2- 172.21.15.45:0/1870713610 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a1277790 0x5617a127c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.564+0000 7f5c08a00640 1 -- 172.21.15.45:0/1870713610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a126ebb0 msgr2=0x5617a127c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:28.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.564+0000 7f5c08a00640 1 --2- 172.21.15.45:0/1870713610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a126ebb0 0x5617a127c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.564+0000 7f5c08a00640 1 -- 172.21.15.45:0/1870713610 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5617a1289fa0 con 0x5617a126e290 2024-03-28T17:50:28.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.564+0000 7f5c03400640 1 --2- 172.21.15.45:0/1870713610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a126ebb0 0x5617a127c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:50:28.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.564+0000 7f5c08a00640 1 --2- 172.21.15.45:0/1870713610 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5617a126e290 0x5617a127bd00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc005540 tx=0x7f5bfc0096c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54bb8d9477ac0cf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:28.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.564+0000 7f5bed600640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:50:28.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.564+0000 7f5bed600640 1 -- 172.21.15.45:0/1870713610 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc005760 con 0x5617a126e290 2024-03-28T17:50:28.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.564+0000 7f5c0d322740 1 -- 172.21.15.45:0/1870713610 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5617a11da740 con 0x5617a126e290 2024-03-28T17:50:28.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.565+0000 7f5bed600640 1 -- 172.21.15.45:0/1870713610 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f5bfc030080 con 0x5617a126e290 2024-03-28T17:50:28.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.565+0000 7f5bed600640 1 -- 172.21.15.45:0/1870713610 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5bfc045050 con 0x5617a126e290 2024-03-28T17:50:28.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.565+0000 7f5c0d322740 10 client.25756 fetch_fsmap learned FSMap version 23 2024-03-28T17:50:28.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.565+0000 7f5c0d322740 1 -- 172.21.15.45:0/1870713610 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5617a12ec980 con 0x5617a126e290 2024-03-28T17:50:28.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.565+0000 7f5bed600640 1 -- 172.21.15.45:0/1870713610 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5617a11da740 con 0x5617a126e290 2024-03-28T17:50:28.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.565+0000 7f5bed600640 1 -- 172.21.15.45:0/1870713610 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f5bfc03d040 con 0x5617a126e290 2024-03-28T17:50:28.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.565+0000 7f5c0d322740 10 client.25756 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:50:28.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.565+0000 7f5c0d322740 10 client.25756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:50:28.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.565+0000 7f5c0d322740 20 client.25756 populate_metadata read hostname 'smithi045' 2024-03-28T17:50:28.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.565+0000 7f5c0d322740 1 --2- 172.21.15.45:0/1870713610 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5617a12eccf0 0x5617a12ef110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.565+0000 7f5c0d322740 4 client.25756 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T17:50:28.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.565+0000 7f5c0d322740 1 -- 172.21.15.45:0/1870713610 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5617a1158a40 con 0x5617a12eccf0 2024-03-28T17:50:28.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.565+0000 7f5c03400640 1 --2- 172.21.15.45:0/1870713610 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5617a12eccf0 0x5617a12ef110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.566+0000 7f5c03400640 1 --2- 172.21.15.45:0/1870713610 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5617a12eccf0 0x5617a12ef110 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c91cdc1c66fd9485 server_cookie=d97faeae354f3988 in_seq=0 out_seq=0 2024-03-28T17:50:28.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.566+0000 7f5bed600640 10 client.25756 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:50:28.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.568+0000 7f5bed600640 1 -- 172.21.15.45:0/1870713610 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5617a1158a40 con 0x5617a12eccf0 2024-03-28T17:50:28.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.568+0000 7f5bed600640 10 client.25756 handle_command_reply: tid=0 2024-03-28T17:50:28.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.568+0000 7f5c0d322740 1 client.25756 shutdown 2024-03-28T17:50:28.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.568+0000 7f5c0d322740 1 -- 172.21.15.45:0/1870713610 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5617a126e290 msgr2=0x5617a127bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.568+0000 7f5c0d322740 1 --2- 172.21.15.45:0/1870713610 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5617a126e290 0x5617a127bd00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc005540 tx=0x7f5bfc0096c0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.568+0000 7f5c0d322740 1 -- 172.21.15.45:0/1870713610 shutdown_connections 2024-03-28T17:50:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.568+0000 7f5c0d322740 1 --2- 172.21.15.45:0/1870713610 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5617a12eccf0 0x5617a12ef110 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.568+0000 7f5c0d322740 1 --2- 172.21.15.45:0/1870713610 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5617a1277790 0x5617a127c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.568+0000 7f5c0d322740 1 --2- 172.21.15.45:0/1870713610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a126ebb0 0x5617a127c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.568+0000 7f5c0d322740 1 --2- 172.21.15.45:0/1870713610 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5617a126e290 0x5617a127bd00 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.568+0000 7f5bed600640 0 client.25756 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:50:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.568+0000 7f5c0d322740 1 -- 172.21.15.45:0/1870713610 >> 172.21.15.45:0/1870713610 conn(0x5617a12689c0 msgr2=0x5617a128e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.569+0000 7f5c0d322740 1 -- 172.21.15.45:0/1870713610 shutdown_connections 2024-03-28T17:50:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.569+0000 7f5c0d322740 1 -- 172.21.15.45:0/1870713610 wait complete. 2024-03-28T17:50:28.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.569+0000 7f5c0d322740 20 client.25756 trim_cache size 0 max 16384 2024-03-28T17:50:28.572 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:50:28.572 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-28T17:50:28.572 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "ec5935e7-ea12-45ef-8178-4b3bae324da7", 2024-03-28T17:50:28.572 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-28T17:50:28.572 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:50:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.569+0000 7f5c09400640 1 -- 172.21.15.45:0/48973368 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f5be80a2a90 msgr2=0x7f5be80a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.569+0000 7f5c09400640 1 --2- 172.21.15.45:0/48973368 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f5be80a2a90 0x7f5be80a4f50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5bf40cca90 tx=0x7f5bf40d2f20 comp rx=0 tx=0).stop 2024-03-28T17:50:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.569+0000 7f5c09400640 1 -- 172.21.15.45:0/48973368 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c04101520 msgr2=0x7f5c04199fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.569+0000 7f5c09400640 1 --2- 172.21.15.45:0/48973368 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c04101520 0x7f5c04199fb0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400e9f0 tx=0x7f5bf400eeb0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.570+0000 7f5c09400640 1 -- 172.21.15.45:0/48973368 shutdown_connections 2024-03-28T17:50:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.570+0000 7f5c09400640 1 --2- 172.21.15.45:0/48973368 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f5be80a2a90 0x7f5be80a4f50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.570+0000 7f5c09400640 1 --2- 172.21.15.45:0/48973368 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c041075e0 0x7f5c0419e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.570+0000 7f5c09400640 1 --2- 172.21.15.45:0/48973368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c04101ee0 0x7f5c0419a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.570+0000 7f5c09400640 1 --2- 172.21.15.45:0/48973368 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c04101520 0x7f5c04199fb0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.570+0000 7f5c09400640 1 -- 172.21.15.45:0/48973368 >> 172.21.15.45:0/48973368 conn(0x7f5c040fd010 msgr2=0x7f5c040741e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.570+0000 7f5c09400640 1 -- 172.21.15.45:0/48973368 shutdown_connections 2024-03-28T17:50:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.570+0000 7f5c09400640 1 -- 172.21.15.45:0/48973368 wait complete. 2024-03-28T17:50:28.589 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:50:28.590 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:50:28.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.809+0000 7f553d600640 1 -- 172.21.15.45:0/1669376298 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55381064a0 msgr2=0x7f5538106880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:28.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.809+0000 7f553d600640 1 --2- 172.21.15.45:0/1669376298 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55381064a0 0x7f5538106880 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5528009900 tx=0x7f552802bc30 comp rx=0 tx=0).stop 2024-03-28T17:50:28.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.809+0000 7f553d600640 1 -- 172.21.15.45:0/1669376298 shutdown_connections 2024-03-28T17:50:28.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.809+0000 7f553d600640 1 --2- 172.21.15.45:0/1669376298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5538100dc0 0x7f553810d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.809+0000 7f553d600640 1 --2- 172.21.15.45:0/1669376298 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5538100400 0x7f5538100880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.809+0000 7f553d600640 1 --2- 172.21.15.45:0/1669376298 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55381064a0 0x7f5538106880 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.809+0000 7f553d600640 1 -- 172.21.15.45:0/1669376298 >> 172.21.15.45:0/1669376298 conn(0x7f55380fbef0 msgr2=0x7f55380fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:28.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.809+0000 7f553d600640 1 -- 172.21.15.45:0/1669376298 shutdown_connections 2024-03-28T17:50:28.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.809+0000 7f553d600640 1 -- 172.21.15.45:0/1669376298 wait complete. 2024-03-28T17:50:28.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.810+0000 7f553d600640 1 Processor -- start 2024-03-28T17:50:28.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.811+0000 7f553d600640 1 -- start start 2024-03-28T17:50:28.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.811+0000 7f553d600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5538100400 0x7f553819a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.811+0000 7f553d600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5538100dc0 0x7f553819a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.811+0000 7f553d600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55381064a0 0x7f553819e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.811+0000 7f553d600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55381107a0 con 0x7f5538100dc0 2024-03-28T17:50:28.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.811+0000 7f553d600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5538110620 con 0x7f55381064a0 2024-03-28T17:50:28.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.811+0000 7f553d600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5538110920 con 0x7f5538100400 2024-03-28T17:50:28.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.811+0000 7f5537400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5538100dc0 0x7f553819a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.811+0000 7f553cc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55381064a0 0x7f553819e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.811+0000 7f5537e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5538100400 0x7f553819a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.811+0000 7f553cc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55381064a0 0x7f553819e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:33524/0 (socket says 172.21.15.45:33524) 2024-03-28T17:50:28.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.811+0000 7f5537e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5538100400 0x7f553819a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:47064/0 (socket says 172.21.15.45:47064) 2024-03-28T17:50:28.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.811+0000 7f553cc00640 1 -- 172.21.15.45:0/904691226 learned_addr learned my addr 172.21.15.45:0/904691226 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:28.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.812+0000 7f553cc00640 1 -- 172.21.15.45:0/904691226 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5538100400 msgr2=0x7f553819a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:28.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.812+0000 7f553cc00640 1 --2- 172.21.15.45:0/904691226 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5538100400 0x7f553819a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.812+0000 7f553cc00640 1 -- 172.21.15.45:0/904691226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5538100dc0 msgr2=0x7f553819a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:28.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.812+0000 7f553cc00640 1 --2- 172.21.15.45:0/904691226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5538100dc0 0x7f553819a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.812+0000 7f553cc00640 1 -- 172.21.15.45:0/904691226 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553819f080 con 0x7f55381064a0 2024-03-28T17:50:28.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.812+0000 7f5537e00640 1 --2- 172.21.15.45:0/904691226 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5538100400 0x7f553819a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:50:28.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.812+0000 7f553cc00640 1 --2- 172.21.15.45:0/904691226 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55381064a0 0x7f553819e950 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f553000eb10 tx=0x7f553000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:28.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.812+0000 7f5534c00640 1 -- 172.21.15.45:0/904691226 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5530004b20 con 0x7f55381064a0 2024-03-28T17:50:28.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.812+0000 7f553d600640 1 -- 172.21.15.45:0/904691226 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f553819f370 con 0x7f55381064a0 2024-03-28T17:50:28.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.812+0000 7f5534c00640 1 -- 172.21.15.45:0/904691226 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5530004cc0 con 0x7f55381064a0 2024-03-28T17:50:28.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.812+0000 7f5534c00640 1 -- 172.21.15.45:0/904691226 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5530010760 con 0x7f55381064a0 2024-03-28T17:50:28.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.813+0000 7f553d600640 1 -- 172.21.15.45:0/904691226 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55381a6c00 con 0x7f55381064a0 2024-03-28T17:50:28.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.816+0000 7f5534c00640 1 -- 172.21.15.45:0/904691226 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 27) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f5530005020 con 0x7f55381064a0 2024-03-28T17:50:28.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.816+0000 7f5541964740 1 Processor -- start 2024-03-28T17:50:28.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.816+0000 7f5541964740 1 -- start start 2024-03-28T17:50:28.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.817+0000 7f5541964740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60ddd0 0x55bbcf60cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.817+0000 7f5541964740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60d0f0 0x55bbcf60d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.817+0000 7f5541964740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bbcf616460 0x55bbcf618850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.817+0000 7f5541964740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbcf571250 con 0x55bbcf60d0f0 2024-03-28T17:50:28.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.817+0000 7f5541964740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbcf519a30 con 0x55bbcf616460 2024-03-28T17:50:28.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.817+0000 7f5541964740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbcf56d2f0 con 0x55bbcf60ddd0 2024-03-28T17:50:28.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.817+0000 7f5537e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60d0f0 0x55bbcf60d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.817+0000 7f5537400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60ddd0 0x55bbcf60cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.817+0000 7f5537e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60d0f0 0x55bbcf60d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42446/0 (socket says 172.21.15.45:42446) 2024-03-28T17:50:28.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.817+0000 7f5537400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60ddd0 0x55bbcf60cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:47068/0 (socket says 172.21.15.45:47068) 2024-03-28T17:50:28.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.817+0000 7f5537e00640 1 -- 172.21.15.45:0/4216805258 learned_addr learned my addr 172.21.15.45:0/4216805258 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:28.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.817+0000 7f5534c00640 1 --2- 172.21.15.45:0/904691226 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f551c0a2b10 0x7f551c0a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.817+0000 7f5537400640 1 --2- 172.21.15.45:0/4216805258 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bbcf616460 0x55bbcf618850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.817+0000 7f5534c00640 1 -- 172.21.15.45:0/904691226 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f55300cf600 con 0x7f55381064a0 2024-03-28T17:50:28.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.818+0000 7f5537400640 1 -- 172.21.15.45:0/4216805258 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bbcf616460 msgr2=0x55bbcf618850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:28.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.818+0000 7f5537400640 1 --2- 172.21.15.45:0/4216805258 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bbcf616460 0x55bbcf618850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.818+0000 7f5537400640 1 -- 172.21.15.45:0/4216805258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60d0f0 msgr2=0x55bbcf60d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:28.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.818+0000 7f5537400640 1 --2- 172.21.15.45:0/4216805258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60d0f0 0x55bbcf60d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.818+0000 7f5537400640 1 -- 172.21.15.45:0/4216805258 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bbcf504690 con 0x55bbcf60ddd0 2024-03-28T17:50:28.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.818+0000 7f5537e00640 1 --2- 172.21.15.45:0/4216805258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60d0f0 0x55bbcf60d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:50:28.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.818+0000 7f553cc00640 1 --2- 172.21.15.45:0/904691226 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f551c0a2b10 0x7f551c0a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.818+0000 7f5537400640 1 --2- 172.21.15.45:0/4216805258 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60ddd0 0x55bbcf60cb20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5538069400 tx=0x7f5524004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a447626e8403b524 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:28.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.818+0000 7f552d200640 1 -- 172.21.15.45:0/4216805258 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5524012ca0 con 0x55bbcf60ddd0 2024-03-28T17:50:28.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.818+0000 7f552d200640 1 -- 172.21.15.45:0/4216805258 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5524012e40 con 0x55bbcf60ddd0 2024-03-28T17:50:28.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.818+0000 7f552d200640 1 -- 172.21.15.45:0/4216805258 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55240049c0 con 0x55bbcf60ddd0 2024-03-28T17:50:28.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.819+0000 7f5541964740 1 -- 172.21.15.45:0/4216805258 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60ddd0 msgr2=0x55bbcf60cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:28.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.819+0000 7f5541964740 1 --2- 172.21.15.45:0/4216805258 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60ddd0 0x55bbcf60cb20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5538069400 tx=0x7f5524004730 comp rx=0 tx=0).stop 2024-03-28T17:50:28.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.819+0000 7f553cc00640 1 --2- 172.21.15.45:0/904691226 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f551c0a2b10 0x7f551c0a4fd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f55300ccc00 tx=0x7f5530005b40 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:28.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.819+0000 7f5541964740 1 -- 172.21.15.45:0/4216805258 shutdown_connections 2024-03-28T17:50:28.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.819+0000 7f5541964740 1 --2- 172.21.15.45:0/4216805258 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bbcf616460 0x55bbcf618850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.819+0000 7f5541964740 1 --2- 172.21.15.45:0/4216805258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60d0f0 0x55bbcf60d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.819+0000 7f5541964740 1 --2- 172.21.15.45:0/4216805258 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60ddd0 0x55bbcf60cb20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.819+0000 7f5541964740 1 -- 172.21.15.45:0/4216805258 >> 172.21.15.45:0/4216805258 conn(0x55bbcf5d4200 msgr2=0x55bbcf607b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:28.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.819+0000 7f5541964740 1 -- 172.21.15.45:0/4216805258 shutdown_connections 2024-03-28T17:50:28.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.819+0000 7f5541964740 1 -- 172.21.15.45:0/4216805258 wait complete. 2024-03-28T17:50:28.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.820+0000 7f5541964740 1 -- start start 2024-03-28T17:50:28.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.820+0000 7f5541964740 1 Processor -- start 2024-03-28T17:50:28.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.820+0000 7f5541964740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60d0f0 0x55bbcf615b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.820+0000 7f5541964740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60ddd0 0x55bbcf612ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.821+0000 7f5541964740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bbcf616460 0x55bbcf6133e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.821+0000 7f5541964740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbcf6192d0 con 0x55bbcf60ddd0 2024-03-28T17:50:28.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.821+0000 7f5541964740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbcf52ce80 con 0x55bbcf616460 2024-03-28T17:50:28.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.821+0000 7f5541964740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbcf619450 con 0x55bbcf60d0f0 2024-03-28T17:50:28.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.821+0000 7f5541964740 10 client.0 _ll_register_callbacks cb 0x55bbcf53f130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:50:28.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.821+0000 7f5537400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60d0f0 0x55bbcf615b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.821+0000 7f5537400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60d0f0 0x55bbcf615b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:47072/0 (socket says 172.21.15.45:47072) 2024-03-28T17:50:28.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.821+0000 7f5537400640 1 -- 172.21.15.45:0/581367601 learned_addr learned my addr 172.21.15.45:0/581367601 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:28.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.821+0000 7f5537400640 1 --2- 172.21.15.45:0/581367601 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bbcf616460 0x55bbcf6133e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.821+0000 7f5537e00640 1 --2- 172.21.15.45:0/581367601 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60ddd0 0x55bbcf612ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.821+0000 7f5537400640 1 -- 172.21.15.45:0/581367601 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bbcf616460 msgr2=0x55bbcf6133e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:28.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.821+0000 7f5537400640 1 --2- 172.21.15.45:0/581367601 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bbcf616460 0x55bbcf6133e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.821+0000 7f5537400640 1 -- 172.21.15.45:0/581367601 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60ddd0 msgr2=0x55bbcf612ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:28.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.821+0000 7f5537400640 1 --2- 172.21.15.45:0/581367601 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60ddd0 0x55bbcf612ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.821+0000 7f5537400640 1 -- 172.21.15.45:0/581367601 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bbcf34f7d0 con 0x55bbcf60d0f0 2024-03-28T17:50:28.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.821+0000 7f5537e00640 1 --2- 172.21.15.45:0/581367601 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60ddd0 0x55bbcf612ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:50:28.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.822+0000 7f5537400640 1 --2- 172.21.15.45:0/581367601 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60d0f0 0x55bbcf615b10 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f552400b470 tx=0x7f5524014040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc2dc3c2706a7fa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:28.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.822+0000 7f5521600640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:50:28.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.822+0000 7f5521600640 1 -- 172.21.15.45:0/581367601 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5524004420 con 0x55bbcf60d0f0 2024-03-28T17:50:28.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.822+0000 7f5521600640 1 -- 172.21.15.45:0/581367601 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f5524010740 con 0x55bbcf60d0f0 2024-03-28T17:50:28.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.822+0000 7f5541964740 1 -- 172.21.15.45:0/581367601 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bbcf57e790 con 0x55bbcf60d0f0 2024-03-28T17:50:28.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.822+0000 7f5521600640 1 -- 172.21.15.45:0/581367601 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5524033050 con 0x55bbcf60d0f0 2024-03-28T17:50:28.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.822+0000 7f5541964740 10 client.25768 fetch_fsmap learned FSMap version 23 2024-03-28T17:50:28.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.822+0000 7f5541964740 1 -- 172.21.15.45:0/581367601 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55bbcf2853f0 con 0x55bbcf60d0f0 2024-03-28T17:50:28.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.822+0000 7f5521600640 1 -- 172.21.15.45:0/581367601 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55bbcf57e790 con 0x55bbcf60d0f0 2024-03-28T17:50:28.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.823+0000 7f5521600640 1 -- 172.21.15.45:0/581367601 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f5524003bf0 con 0x55bbcf60d0f0 2024-03-28T17:50:28.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.823+0000 7f5541964740 10 client.25768 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:50:28.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.823+0000 7f5541964740 10 client.25768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:50:28.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.823+0000 7f5541964740 20 client.25768 populate_metadata read hostname 'smithi045' 2024-03-28T17:50:28.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.823+0000 7f5541964740 1 --2- 172.21.15.45:0/581367601 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55bbcf626200 0x55bbcf6285f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.823+0000 7f5541964740 4 client.25768 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:50:28.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.823+0000 7f5541964740 1 -- 172.21.15.45:0/581367601 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bbcf578ae0 con 0x55bbcf626200 2024-03-28T17:50:28.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.823+0000 7f5537e00640 1 --2- 172.21.15.45:0/581367601 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55bbcf626200 0x55bbcf6285f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.824+0000 7f5537e00640 1 --2- 172.21.15.45:0/581367601 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55bbcf626200 0x55bbcf6285f0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a277e2f84fe8dc6 server_cookie=7d440d6c088bedf4 in_seq=0 out_seq=0 2024-03-28T17:50:28.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.824+0000 7f5521600640 10 client.25768 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:50:28.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.826+0000 7f5521600640 1 -- 172.21.15.45:0/581367601 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bbcf578ae0 con 0x55bbcf626200 2024-03-28T17:50:28.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.826+0000 7f5521600640 10 client.25768 handle_command_reply: tid=0 2024-03-28T17:50:28.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.826+0000 7f5541964740 1 client.25768 shutdown 2024-03-28T17:50:28.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.826+0000 7f5541964740 1 -- 172.21.15.45:0/581367601 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60d0f0 msgr2=0x55bbcf615b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:28.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.826+0000 7f5541964740 1 --2- 172.21.15.45:0/581367601 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60d0f0 0x55bbcf615b10 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f552400b470 tx=0x7f5524014040 comp rx=0 tx=0).stop 2024-03-28T17:50:28.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.826+0000 7f5541964740 1 -- 172.21.15.45:0/581367601 shutdown_connections 2024-03-28T17:50:28.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.826+0000 7f5541964740 1 --2- 172.21.15.45:0/581367601 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55bbcf626200 0x55bbcf6285f0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.826+0000 7f5541964740 1 --2- 172.21.15.45:0/581367601 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bbcf616460 0x55bbcf6133e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.826+0000 7f5541964740 1 --2- 172.21.15.45:0/581367601 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60ddd0 0x55bbcf612ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.826+0000 7f5521600640 0 client.25768 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:50:28.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.826+0000 7f5541964740 1 --2- 172.21.15.45:0/581367601 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60d0f0 0x55bbcf615b10 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.826+0000 7f5541964740 1 -- 172.21.15.45:0/581367601 >> 172.21.15.45:0/581367601 conn(0x55bbcf5d4200 msgr2=0x55bbcf609890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:28.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.827+0000 7f5541964740 1 -- 172.21.15.45:0/581367601 shutdown_connections 2024-03-28T17:50:28.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.827+0000 7f5541964740 1 -- 172.21.15.45:0/581367601 wait complete. 2024-03-28T17:50:28.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.827+0000 7f5541964740 20 client.25768 trim_cache size 0 max 16384 2024-03-28T17:50:28.832 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1699 26.23 MB/sec warmup 1 sec latency 10.560 ms 2024-03-28T17:50:28.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.845+0000 7f5541964740 1 Processor -- start 2024-03-28T17:50:28.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.845+0000 7f5541964740 1 -- start start 2024-03-28T17:50:28.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.845+0000 7f5541964740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60d0f0 0x55bbcf612650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.845+0000 7f5541964740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60ddd0 0x55bbcf609630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.845+0000 7f5541964740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bbcf616460 0x55bbcf617850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.845+0000 7f5541964740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55381b1800 con 0x55bbcf60d0f0 2024-03-28T17:50:28.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.845+0000 7f5541964740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55381af300 con 0x55bbcf616460 2024-03-28T17:50:28.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.845+0000 7f5541964740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55381ad380 con 0x55bbcf60ddd0 2024-03-28T17:50:28.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.846+0000 7f5537400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60d0f0 0x55bbcf612650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.846+0000 7f5537400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60d0f0 0x55bbcf612650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42468/0 (socket says 172.21.15.45:42468) 2024-03-28T17:50:28.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.846+0000 7f5537e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60ddd0 0x55bbcf609630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.846+0000 7f5537400640 1 -- 172.21.15.45:0/1957013230 learned_addr learned my addr 172.21.15.45:0/1957013230 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.846+0000 7f5537400640 1 --2- 172.21.15.45:0/1957013230 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bbcf616460 0x55bbcf617850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.846+0000 7f5537e00640 1 -- 172.21.15.45:0/1957013230 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bbcf616460 msgr2=0x55bbcf617850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.846+0000 7f5537e00640 1 --2- 172.21.15.45:0/1957013230 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bbcf616460 0x55bbcf617850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.846+0000 7f5537e00640 1 -- 172.21.15.45:0/1957013230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60d0f0 msgr2=0x55bbcf612650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.846+0000 7f5537e00640 1 --2- 172.21.15.45:0/1957013230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60d0f0 0x55bbcf612650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.846+0000 7f5537e00640 1 -- 172.21.15.45:0/1957013230 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bbcf2450f0 con 0x55bbcf60ddd0 2024-03-28T17:50:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.846+0000 7f5537400640 1 --2- 172.21.15.45:0/1957013230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60d0f0 0x55bbcf612650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:50:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.846+0000 7f5537e00640 1 --2- 172.21.15.45:0/1957013230 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60ddd0 0x55bbcf609630 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5528009230 tx=0x7f552802e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9846149260cb5196 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.847+0000 7f5523e00640 1 -- 172.21.15.45:0/1957013230 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5528004040 con 0x55bbcf60ddd0 2024-03-28T17:50:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.847+0000 7f5523e00640 1 -- 172.21.15.45:0/1957013230 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f552804c030 con 0x55bbcf60ddd0 2024-03-28T17:50:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.847+0000 7f5523e00640 1 -- 172.21.15.45:0/1957013230 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5528005320 con 0x55bbcf60ddd0 2024-03-28T17:50:28.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.847+0000 7f5541964740 1 -- 172.21.15.45:0/1957013230 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60ddd0 msgr2=0x55bbcf609630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:28.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.847+0000 7f5541964740 1 --2- 172.21.15.45:0/1957013230 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60ddd0 0x55bbcf609630 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5528009230 tx=0x7f552802e040 comp rx=0 tx=0).stop 2024-03-28T17:50:28.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.848+0000 7f5541964740 1 -- 172.21.15.45:0/1957013230 shutdown_connections 2024-03-28T17:50:28.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.848+0000 7f5541964740 1 --2- 172.21.15.45:0/1957013230 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bbcf616460 0x55bbcf617850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.848+0000 7f5541964740 1 --2- 172.21.15.45:0/1957013230 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60ddd0 0x55bbcf609630 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.848+0000 7f5541964740 1 --2- 172.21.15.45:0/1957013230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60d0f0 0x55bbcf612650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.848+0000 7f5541964740 1 -- 172.21.15.45:0/1957013230 >> 172.21.15.45:0/1957013230 conn(0x55bbcf5d4200 msgr2=0x55bbcf608f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:28.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.848+0000 7f5541964740 1 -- 172.21.15.45:0/1957013230 shutdown_connections 2024-03-28T17:50:28.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.848+0000 7f5541964740 1 -- 172.21.15.45:0/1957013230 wait complete. 2024-03-28T17:50:28.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.848+0000 7f5541964740 1 -- start start 2024-03-28T17:50:28.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.848+0000 7f5541964740 1 Processor -- start 2024-03-28T17:50:28.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.849+0000 7f5541964740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60d0f0 0x55bbcf61e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.849+0000 7f5541964740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60ddd0 0x55bbcf619790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.849+0000 7f5541964740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bbcf616460 0x55bbcf619cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.849+0000 7f5541964740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55381ad980 con 0x55bbcf60d0f0 2024-03-28T17:50:28.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.849+0000 7f5541964740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55381aeca0 con 0x55bbcf616460 2024-03-28T17:50:28.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.849+0000 7f5541964740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55381adfe0 con 0x55bbcf60ddd0 2024-03-28T17:50:28.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.849+0000 7f5541964740 10 client.0 _ll_register_callbacks cb 0x7f5528009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:50:28.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.849+0000 7f5537e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60ddd0 0x55bbcf619790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.849+0000 7f5537400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60d0f0 0x55bbcf61e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.849+0000 7f5537e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60ddd0 0x55bbcf619790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:47084/0 (socket says 172.21.15.45:47084) 2024-03-28T17:50:28.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.849+0000 7f5537e00640 1 -- 172.21.15.45:0/1466477984 learned_addr learned my addr 172.21.15.45:0/1466477984 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:28.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.849+0000 7f5537400640 1 --2- 172.21.15.45:0/1466477984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bbcf616460 0x55bbcf619cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.849+0000 7f5537e00640 1 -- 172.21.15.45:0/1466477984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bbcf616460 msgr2=0x55bbcf619cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:28.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.849+0000 7f5537e00640 1 --2- 172.21.15.45:0/1466477984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bbcf616460 0x55bbcf619cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.850+0000 7f5537e00640 1 -- 172.21.15.45:0/1466477984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60d0f0 msgr2=0x55bbcf61e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:28.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.850+0000 7f5537e00640 1 --2- 172.21.15.45:0/1466477984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60d0f0 0x55bbcf61e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.850+0000 7f5537e00640 1 -- 172.21.15.45:0/1466477984 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bbcf61a480 con 0x55bbcf60ddd0 2024-03-28T17:50:28.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.850+0000 7f5537e00640 1 --2- 172.21.15.45:0/1466477984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60ddd0 0x55bbcf619790 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5528036040 tx=0x7f5528000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29234d3df4bbbb6b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:28.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.850+0000 7f5521600640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:50:28.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.850+0000 7f5521600640 1 -- 172.21.15.45:0/1466477984 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552802cab0 con 0x55bbcf60ddd0 2024-03-28T17:50:28.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.850+0000 7f5541964740 1 -- 172.21.15.45:0/1466477984 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f552804c030 con 0x55bbcf60ddd0 2024-03-28T17:50:28.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.850+0000 7f5521600640 1 -- 172.21.15.45:0/1466477984 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f552802cc50 con 0x55bbcf60ddd0 2024-03-28T17:50:28.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.850+0000 7f5521600640 1 -- 172.21.15.45:0/1466477984 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5528004f20 con 0x55bbcf60ddd0 2024-03-28T17:50:28.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.851+0000 7f5541964740 10 client.25774 fetch_fsmap learned FSMap version 23 2024-03-28T17:50:28.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.851+0000 7f5541964740 1 -- 172.21.15.45:0/1466477984 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55bbcf681860 con 0x55bbcf60ddd0 2024-03-28T17:50:28.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.851+0000 7f5521600640 1 -- 172.21.15.45:0/1466477984 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f552804c030 con 0x55bbcf60ddd0 2024-03-28T17:50:28.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.851+0000 7f5521600640 1 -- 172.21.15.45:0/1466477984 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 23) v1 ==== 7918+0+0 (secure 0 0 0) 0x7f5528048a80 con 0x55bbcf60ddd0 2024-03-28T17:50:28.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.851+0000 7f5541964740 10 client.25774 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T17:50:28.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.851+0000 7f5541964740 10 client.25774 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:50:28.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.851+0000 7f5541964740 20 client.25774 populate_metadata read hostname 'smithi045' 2024-03-28T17:50:28.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.851+0000 7f5541964740 1 --2- 172.21.15.45:0/1466477984 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55bbcf681f00 0x55bbcf684320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:28.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.851+0000 7f5541964740 4 client.25774 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:50:28.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.851+0000 7f5541964740 1 -- 172.21.15.45:0/1466477984 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55bbcf613770 con 0x55bbcf681f00 2024-03-28T17:50:28.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.851+0000 7f5537400640 1 --2- 172.21.15.45:0/1466477984 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55bbcf681f00 0x55bbcf684320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:28.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.852+0000 7f5537400640 1 --2- 172.21.15.45:0/1466477984 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55bbcf681f00 0x55bbcf684320 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=241b94180cc1bf46 server_cookie=1059eb30bd581ff1 in_seq=0 out_seq=0 2024-03-28T17:50:28.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.852+0000 7f5521600640 10 client.25774 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:50:28.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.852+0000 7f5521600640 1 -- 172.21.15.45:0/1466477984 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55bbcf613770 con 0x55bbcf681f00 2024-03-28T17:50:28.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.852+0000 7f5521600640 10 client.25774 handle_command_reply: tid=0 2024-03-28T17:50:28.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.852+0000 7f5541964740 1 client.25774 shutdown 2024-03-28T17:50:28.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.852+0000 7f5541964740 1 -- 172.21.15.45:0/1466477984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60ddd0 msgr2=0x55bbcf619790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:28.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.852+0000 7f5541964740 1 --2- 172.21.15.45:0/1466477984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60ddd0 0x55bbcf619790 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5528036040 tx=0x7f5528000bc0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.852+0000 7f5541964740 1 -- 172.21.15.45:0/1466477984 shutdown_connections 2024-03-28T17:50:28.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.852+0000 7f5541964740 1 --2- 172.21.15.45:0/1466477984 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55bbcf681f00 0x55bbcf684320 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.852+0000 7f5541964740 1 --2- 172.21.15.45:0/1466477984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bbcf616460 0x55bbcf619cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.852+0000 7f5541964740 1 --2- 172.21.15.45:0/1466477984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bbcf60ddd0 0x55bbcf619790 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.852+0000 7f5541964740 1 --2- 172.21.15.45:0/1466477984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bbcf60d0f0 0x55bbcf61e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.852+0000 7f5521600640 0 client.25774 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:50:28.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.852+0000 7f5541964740 1 -- 172.21.15.45:0/1466477984 >> 172.21.15.45:0/1466477984 conn(0x55bbcf5d4200 msgr2=0x55bbcf609790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:28.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.853+0000 7f5541964740 1 -- 172.21.15.45:0/1466477984 shutdown_connections 2024-03-28T17:50:28.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.853+0000 7f5541964740 1 -- 172.21.15.45:0/1466477984 wait complete. 2024-03-28T17:50:28.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.853+0000 7f5541964740 20 client.25774 trim_cache size 0 max 16384 2024-03-28T17:50:28.858 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:50:28.858 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T17:50:28.858 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T17:50:28.858 INFO:teuthology.orchestra.run.smithi045.stdout: "ec5935e7-ea12-45ef-8178-4b3bae324da7": { 2024-03-28T17:50:28.858 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T17:50:28.858 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "ec5935e7-ea12-45ef-8178-4b3bae324da7", 2024-03-28T17:50:28.859 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T17:50:28.859 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:50:28.859 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:50:28.859 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:50:28.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.853+0000 7f553d600640 1 -- 172.21.15.45:0/904691226 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f551c0a2b10 msgr2=0x7f551c0a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:28.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.853+0000 7f553d600640 1 --2- 172.21.15.45:0/904691226 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f551c0a2b10 0x7f551c0a4fd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f55300ccc00 tx=0x7f5530005b40 comp rx=0 tx=0).stop 2024-03-28T17:50:28.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.853+0000 7f553d600640 1 -- 172.21.15.45:0/904691226 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55381064a0 msgr2=0x7f553819e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:28.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.853+0000 7f553d600640 1 --2- 172.21.15.45:0/904691226 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55381064a0 0x7f553819e950 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f553000eb10 tx=0x7f553000c2d0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.853+0000 7f553d600640 1 -- 172.21.15.45:0/904691226 shutdown_connections 2024-03-28T17:50:28.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.853+0000 7f553d600640 1 --2- 172.21.15.45:0/904691226 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f551c0a2b10 0x7f551c0a4fd0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.853+0000 7f553d600640 1 --2- 172.21.15.45:0/904691226 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55381064a0 0x7f553819e950 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.853+0000 7f553d600640 1 --2- 172.21.15.45:0/904691226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5538100dc0 0x7f553819a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.853+0000 7f553d600640 1 --2- 172.21.15.45:0/904691226 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5538100400 0x7f553819a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:28.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.853+0000 7f553d600640 1 -- 172.21.15.45:0/904691226 >> 172.21.15.45:0/904691226 conn(0x7f55380fbef0 msgr2=0x7f553810be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:28.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.854+0000 7f553d600640 1 -- 172.21.15.45:0/904691226 shutdown_connections 2024-03-28T17:50:28.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:28.854+0000 7f553d600640 1 -- 172.21.15.45:0/904691226 wait complete. 2024-03-28T17:50:28.871 INFO:tasks.cephfs.filesystem:scrub status for tag:ec5935e7-ea12-45ef-8178-4b3bae324da7 - {'path': '/', 'tag': 'ec5935e7-ea12-45ef-8178-4b3bae324da7', 'options': 'recursive,force'} 2024-03-28T17:50:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:28 smithi045 ceph-mon[31773]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T17:50:29.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:28 smithi071 ceph-mon[36597]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T17:50:29.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:28 smithi151 ceph-mon[36481]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T17:50:29.832 INFO:tasks.workunit.client.0.smithi045.stdout: 1 3875 19.88 MB/sec warmup 2 sec latency 14.199 ms 2024-03-28T17:50:30.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:29 smithi071 ceph-mon[36597]: scrub queued for path: / 2024-03-28T17:50:30.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:29 smithi071 ceph-mon[36597]: scrub summary: idle+waiting paths [/] 2024-03-28T17:50:30.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:29 smithi071 ceph-mon[36597]: scrub summary: active paths [/] 2024-03-28T17:50:30.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:29 smithi071 ceph-mon[36597]: scrub summary: idle+waiting paths [/] 2024-03-28T17:50:30.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:29 smithi151 ceph-mon[36481]: scrub queued for path: / 2024-03-28T17:50:30.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:29 smithi151 ceph-mon[36481]: scrub summary: idle+waiting paths [/] 2024-03-28T17:50:30.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:29 smithi151 ceph-mon[36481]: scrub summary: active paths [/] 2024-03-28T17:50:30.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:29 smithi151 ceph-mon[36481]: scrub summary: idle+waiting paths [/] 2024-03-28T17:50:30.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:29 smithi045 ceph-mon[31773]: scrub queued for path: / 2024-03-28T17:50:30.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:29 smithi045 ceph-mon[31773]: scrub summary: idle+waiting paths [/] 2024-03-28T17:50:30.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:29 smithi045 ceph-mon[31773]: scrub summary: active paths [/] 2024-03-28T17:50:30.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:29 smithi045 ceph-mon[31773]: scrub summary: idle+waiting paths [/] 2024-03-28T17:50:30.832 INFO:tasks.workunit.client.0.smithi045.stdout: 1 5453 15.17 MB/sec warmup 3 sec latency 10.802 ms 2024-03-28T17:50:31.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:30 smithi071 ceph-mon[36597]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:50:31.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:30 smithi151 ceph-mon[36481]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:50:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:30 smithi045 ceph-mon[31773]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:50:31.833 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7388 14.72 MB/sec warmup 4 sec latency 15.904 ms 2024-03-28T17:50:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:31 smithi071 ceph-mon[36597]: scrub summary: idle 2024-03-28T17:50:32.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:31 smithi151 ceph-mon[36481]: scrub summary: idle 2024-03-28T17:50:32.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:31 smithi045 ceph-mon[31773]: scrub summary: idle 2024-03-28T17:50:32.556 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:50:32 smithi151 podman[70807]: 2024-03-28 17:50:32.18800703 +0000 UTC m=+0.019711992 container died 7e607e6e801e76097ed1f76696f1a7cf6aa739cefd6f33385cba163549785a52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-28T17:50:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:32 smithi151 ceph-mon[36481]: pgmap v49: 97 pgs: 97 active+clean; 3.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 192 KiB/s wr, 36 op/s 2024-03-28T17:50:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:32 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:50:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:32 smithi151 ceph-mon[36481]: mgrmap e28: x(active, since 92s), standbys: z, y 2024-03-28T17:50:32.832 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 11.84 MB/sec warmup 5 sec latency 729.538 ms 2024-03-28T17:50:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:32 smithi071 ceph-mon[36597]: pgmap v49: 97 pgs: 97 active+clean; 3.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 192 KiB/s wr, 36 op/s 2024-03-28T17:50:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:32 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:50:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:32 smithi071 ceph-mon[36597]: mgrmap e28: x(active, since 92s), standbys: z, y 2024-03-28T17:50:33.138 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:32 smithi045 ceph-mon[31773]: pgmap v49: 97 pgs: 97 active+clean; 3.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 192 KiB/s wr, 36 op/s 2024-03-28T17:50:33.138 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:32 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:50:33.138 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:32 smithi045 ceph-mon[31773]: mgrmap e28: x(active, since 92s), standbys: z, y 2024-03-28T17:50:33.323 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:50:33 smithi151 podman[70807]: 2024-03-28 17:50:33.017476151 +0000 UTC m=+0.849181106 container remove 7e607e6e801e76097ed1f76696f1a7cf6aa739cefd6f33385cba163549785a52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0) 2024-03-28T17:50:33.324 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:50:33 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:50:33.324 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:50:33 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.i.service: Failed with result 'exit-code'. 2024-03-28T17:50:33.324 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:50:33 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.i.service: Consumed 1min 15.640s CPU time. 2024-03-28T17:50:33.833 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 9.87 MB/sec warmup 6 sec latency 1729.647 ms 2024-03-28T17:50:34.833 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 8.46 MB/sec warmup 7 sec latency 2729.781 ms 2024-03-28T17:50:35.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:34 smithi071 ceph-mon[36597]: pgmap v50: 97 pgs: 97 active+clean; 3.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 192 KiB/s wr, 33 op/s 2024-03-28T17:50:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:34 smithi151 ceph-mon[36481]: pgmap v50: 97 pgs: 97 active+clean; 3.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 192 KiB/s wr, 33 op/s 2024-03-28T17:50:35.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:34 smithi045 ceph-mon[31773]: pgmap v50: 97 pgs: 97 active+clean; 3.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 192 KiB/s wr, 33 op/s 2024-03-28T17:50:35.833 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 7.40 MB/sec warmup 8 sec latency 3729.913 ms 2024-03-28T17:50:36.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:35 smithi151 ceph-mon[36481]: pgmap v51: 97 pgs: 97 active+clean; 4.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 374 KiB/s wr, 67 op/s 2024-03-28T17:50:36.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:35 smithi045 ceph-mon[31773]: pgmap v51: 97 pgs: 97 active+clean; 4.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 374 KiB/s wr, 67 op/s 2024-03-28T17:50:36.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:35 smithi071 ceph-mon[36597]: pgmap v51: 97 pgs: 97 active+clean; 4.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 374 KiB/s wr, 67 op/s 2024-03-28T17:50:36.833 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 6.58 MB/sec warmup 9 sec latency 4730.073 ms 2024-03-28T17:50:37.833 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 5.92 MB/sec warmup 10 sec latency 5730.194 ms 2024-03-28T17:50:38.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:37 smithi071 ceph-mon[36597]: pgmap v52: 97 pgs: 97 active+clean; 5.0 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 389 KiB/s wr, 66 op/s 2024-03-28T17:50:38.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:37 smithi151 ceph-mon[36481]: pgmap v52: 97 pgs: 97 active+clean; 5.0 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 389 KiB/s wr, 66 op/s 2024-03-28T17:50:38.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:37 smithi045 ceph-mon[31773]: pgmap v52: 97 pgs: 97 active+clean; 5.0 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 389 KiB/s wr, 66 op/s 2024-03-28T17:50:38.834 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 5.38 MB/sec warmup 11 sec latency 6730.294 ms 2024-03-28T17:50:39.833 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 4.93 MB/sec warmup 12 sec latency 7730.385 ms 2024-03-28T17:50:40.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:39 smithi071 ceph-mon[36597]: pgmap v53: 97 pgs: 97 active+clean; 5.0 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 389 KiB/s wr, 66 op/s 2024-03-28T17:50:40.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:39 smithi151 ceph-mon[36481]: pgmap v53: 97 pgs: 97 active+clean; 5.0 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 389 KiB/s wr, 66 op/s 2024-03-28T17:50:40.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:39 smithi045 ceph-mon[31773]: pgmap v53: 97 pgs: 97 active+clean; 5.0 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 389 KiB/s wr, 66 op/s 2024-03-28T17:50:40.833 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 4.55 MB/sec warmup 13 sec latency 8730.514 ms 2024-03-28T17:50:41.834 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 4.23 MB/sec warmup 14 sec latency 9730.662 ms 2024-03-28T17:50:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:41 smithi071 ceph-mon[36597]: pgmap v54: 97 pgs: 97 active+clean; 18 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 1.5 MiB/s wr, 72 op/s 2024-03-28T17:50:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:41 smithi151 ceph-mon[36481]: pgmap v54: 97 pgs: 97 active+clean; 18 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 1.5 MiB/s wr, 72 op/s 2024-03-28T17:50:42.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:41 smithi045 ceph-mon[31773]: pgmap v54: 97 pgs: 97 active+clean; 18 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 1.5 MiB/s wr, 72 op/s 2024-03-28T17:50:42.834 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 3.95 MB/sec warmup 15 sec latency 10730.823 ms 2024-03-28T17:50:43.574 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:50:43 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.i.service: Scheduled restart job, restart counter is at 1. 2024-03-28T17:50:43.575 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:50:43 smithi151 systemd[1]: Stopped Ceph mds.i for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:50:43.575 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:50:43 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.i.service: Consumed 1min 15.640s CPU time. 2024-03-28T17:50:43.575 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:50:43 smithi151 systemd[1]: Starting Ceph mds.i for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:50:43.828 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:50:43 smithi151 podman[70902]: 2024-03-28 17:50:43.478118191 +0000 UTC m=+0.021974427 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:50:43.829 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:50:43 smithi151 podman[70902]: 2024-03-28 17:50:43.592981296 +0000 UTC m=+0.136837598 container create 267fa701c8e13123b6c2a3a9fa798643d5d80813beb088a077af7c6a89152df4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS) 2024-03-28T17:50:43.834 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 3.70 MB/sec warmup 16 sec latency 11730.913 ms 2024-03-28T17:50:44.191 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:44 smithi151 ceph-mon[36481]: pgmap v55: 97 pgs: 97 active+clean; 18 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 1.3 MiB/s wr, 41 op/s 2024-03-28T17:50:44.192 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:44 smithi151 ceph-mon[36481]: daemon mds.i restarted 2024-03-28T17:50:44.192 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:44 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:50:44.192 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:44 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T17:50:44.192 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:44 smithi151 ceph-mon[36481]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T17:50:44.192 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:44 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] up:boot 2024-03-28T17:50:44.192 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:44 smithi151 ceph-mon[36481]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-03-28T17:50:44.192 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:44 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:50:44.192 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:44 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:50:44.192 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:50:43 smithi151 podman[70902]: 2024-03-28 17:50:43.829291024 +0000 UTC m=+0.373147268 container init 267fa701c8e13123b6c2a3a9fa798643d5d80813beb088a077af7c6a89152df4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-28T17:50:44.192 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:50:43 smithi151 podman[70902]: 2024-03-28 17:50:43.832100947 +0000 UTC m=+0.375957194 container start 267fa701c8e13123b6c2a3a9fa798643d5d80813beb088a077af7c6a89152df4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-28T17:50:44.193 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:50:43 smithi151 ceph-mds[70920]: -- 172.21.15.151:0/3168819182 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56060c747a40 con 0x56060c71ac00 2024-03-28T17:50:44.193 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:50:43 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: starting mds.i at 2024-03-28T17:50:44.193 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:50:43 smithi151 bash[70902]: 267fa701c8e13123b6c2a3a9fa798643d5d80813beb088a077af7c6a89152df4 2024-03-28T17:50:44.193 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:50:43 smithi151 systemd[1]: Started Ceph mds.i for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:50:44.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:44 smithi071 ceph-mon[36597]: pgmap v55: 97 pgs: 97 active+clean; 18 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 1.3 MiB/s wr, 41 op/s 2024-03-28T17:50:44.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:44 smithi071 ceph-mon[36597]: daemon mds.i restarted 2024-03-28T17:50:44.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:44 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:50:44.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:44 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T17:50:44.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:44 smithi071 ceph-mon[36597]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T17:50:44.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:44 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] up:boot 2024-03-28T17:50:44.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:44 smithi071 ceph-mon[36597]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-03-28T17:50:44.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:44 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:50:44.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:44 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:50:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:44 smithi045 ceph-mon[31773]: pgmap v55: 97 pgs: 97 active+clean; 18 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 1.3 MiB/s wr, 41 op/s 2024-03-28T17:50:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:44 smithi045 ceph-mon[31773]: daemon mds.i restarted 2024-03-28T17:50:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:44 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:50:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:44 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T17:50:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:44 smithi045 ceph-mon[31773]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T17:50:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:44 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] up:boot 2024-03-28T17:50:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:44 smithi045 ceph-mon[31773]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-03-28T17:50:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:44 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:50:44.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:44 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:50:44.834 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 3.48 MB/sec warmup 17 sec latency 12731.047 ms 2024-03-28T17:50:45.834 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 3.29 MB/sec warmup 18 sec latency 13731.201 ms 2024-03-28T17:50:46.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:46 smithi071 ceph-mon[36597]: pgmap v57: 97 pgs: 97 active+clean; 26 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 2.5 MiB/s wr, 31 op/s 2024-03-28T17:50:46.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:46 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/2876354410,v1:172.21.15.151:6835/2876354410] up:resolve 2024-03-28T17:50:46.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:46 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:50:46.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:46 smithi151 ceph-mon[36481]: pgmap v57: 97 pgs: 97 active+clean; 26 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 2.5 MiB/s wr, 31 op/s 2024-03-28T17:50:46.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:46 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/2876354410,v1:172.21.15.151:6835/2876354410] up:resolve 2024-03-28T17:50:46.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:46 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:50:46.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:46 smithi045 ceph-mon[31773]: pgmap v57: 97 pgs: 97 active+clean; 26 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 2.5 MiB/s wr, 31 op/s 2024-03-28T17:50:46.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:46 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/2876354410,v1:172.21.15.151:6835/2876354410] up:resolve 2024-03-28T17:50:46.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:46 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:50:46.834 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 3.12 MB/sec warmup 19 sec latency 14731.362 ms 2024-03-28T17:50:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:47 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.000999988 2024-03-28T17:50:47.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:47 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/2876354410,v1:172.21.15.151:6835/2876354410] up:reconnect 2024-03-28T17:50:47.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:47 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:50:47.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:47 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.0229997 2024-03-28T17:50:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:47 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.000999988 2024-03-28T17:50:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:47 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/2876354410,v1:172.21.15.151:6835/2876354410] up:reconnect 2024-03-28T17:50:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:47 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:50:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:47 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.0229997 2024-03-28T17:50:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:47 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.000999988 2024-03-28T17:50:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:47 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/2876354410,v1:172.21.15.151:6835/2876354410] up:reconnect 2024-03-28T17:50:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:47 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:50:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:47 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.0229997 2024-03-28T17:50:47.835 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 2.96 MB/sec warmup 20 sec latency 15731.467 ms 2024-03-28T17:50:48.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:48 smithi071 ceph-mon[36597]: pgmap v58: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 2.6 MiB/s wr, 32 op/s 2024-03-28T17:50:48.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:48 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/2876354410,v1:172.21.15.151:6835/2876354410] up:rejoin 2024-03-28T17:50:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:48 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:50:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:48 smithi071 ceph-mon[36597]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-28T17:50:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-mon[36481]: pgmap v58: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 2.6 MiB/s wr, 32 op/s 2024-03-28T17:50:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/2876354410,v1:172.21.15.151:6835/2876354410] up:rejoin 2024-03-28T17:50:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:50:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-mon[36481]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-28T17:50:48.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fae84278700 time 2024-03-28T17:50:48.079445+0000 2024-03-28T17:50:48.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:50:48.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:50:48.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fae8e8a24a1] 2024-03-28T17:50:48.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fae8e8a2667] 2024-03-28T17:50:48.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d75927223] 2024-03-28T17:50:48.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d757a15d6] 2024-03-28T17:50:48.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d757a1da1] 2024-03-28T17:50:48.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d757a42d5] 2024-03-28T17:50:48.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d757f4233] 2024-03-28T17:50:48.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d757f6197] 2024-03-28T17:50:48.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d756f511b] 2024-03-28T17:50:48.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d756f56bc] 2024-03-28T17:50:48.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d756df1ff] 2024-03-28T17:50:48.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fae8eb34488] 2024-03-28T17:50:48.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 13: (DispatchQueue::entry()+0x51f) [0x7fae8eb3160f] 2024-03-28T17:50:48.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fae8ec04e81] 2024-03-28T17:50:48.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fae8d61f1ca] 2024-03-28T17:50:48.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 16: clone() 2024-03-28T17:50:48.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 2024-03-28T17:50:48.080+0000 7fae84278700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fae84278700 time 2024-03-28T17:50:48.079445+0000 2024-03-28T17:50:48.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:50:48.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 2024-03-28T17:50:48.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:50:48.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fae8e8a24a1] 2024-03-28T17:50:48.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fae8e8a2667] 2024-03-28T17:50:48.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d75927223] 2024-03-28T17:50:48.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d757a15d6] 2024-03-28T17:50:48.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d757a1da1] 2024-03-28T17:50:48.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d757a42d5] 2024-03-28T17:50:48.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d757f4233] 2024-03-28T17:50:48.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d757f6197] 2024-03-28T17:50:48.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d756f511b] 2024-03-28T17:50:48.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d756f56bc] 2024-03-28T17:50:48.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d756df1ff] 2024-03-28T17:50:48.327 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fae8eb34488] 2024-03-28T17:50:48.327 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 13: (DispatchQueue::entry()+0x51f) [0x7fae8eb3160f] 2024-03-28T17:50:48.327 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fae8ec04e81] 2024-03-28T17:50:48.327 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fae8d61f1ca] 2024-03-28T17:50:48.327 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 16: clone() 2024-03-28T17:50:48.327 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 2024-03-28T17:50:48.327 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: *** Caught signal (Aborted) ** 2024-03-28T17:50:48.327 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: in thread 7fae84278700 thread_name:ms_dispatch 2024-03-28T17:50:48.327 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:50:48.327 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fae8d629d20] 2024-03-28T17:50:48.328 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 2: gsignal() 2024-03-28T17:50:48.328 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 3: abort() 2024-03-28T17:50:48.328 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fae8e8a24fb] 2024-03-28T17:50:48.328 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fae8e8a2667] 2024-03-28T17:50:48.328 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d75927223] 2024-03-28T17:50:48.328 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d757a15d6] 2024-03-28T17:50:48.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d757a1da1] 2024-03-28T17:50:48.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d757a42d5] 2024-03-28T17:50:48.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d757f4233] 2024-03-28T17:50:48.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d757f6197] 2024-03-28T17:50:48.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d756f511b] 2024-03-28T17:50:48.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d756f56bc] 2024-03-28T17:50:48.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d756df1ff] 2024-03-28T17:50:48.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fae8eb34488] 2024-03-28T17:50:48.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 16: (DispatchQueue::entry()+0x51f) [0x7fae8eb3160f] 2024-03-28T17:50:48.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fae8ec04e81] 2024-03-28T17:50:48.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fae8d61f1ca] 2024-03-28T17:50:48.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 19: clone() 2024-03-28T17:50:48.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 2024-03-28T17:50:48.081+0000 7fae84278700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:50:48.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: in thread 7fae84278700 thread_name:ms_dispatch 2024-03-28T17:50:48.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 2024-03-28T17:50:48.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:50:48.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fae8d629d20] 2024-03-28T17:50:48.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 2: gsignal() 2024-03-28T17:50:48.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 3: abort() 2024-03-28T17:50:48.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fae8e8a24fb] 2024-03-28T17:50:48.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fae8e8a2667] 2024-03-28T17:50:48.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d75927223] 2024-03-28T17:50:48.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d757a15d6] 2024-03-28T17:50:48.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d757a1da1] 2024-03-28T17:50:48.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d757a42d5] 2024-03-28T17:50:48.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d757f4233] 2024-03-28T17:50:48.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d757f6197] 2024-03-28T17:50:48.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d756f511b] 2024-03-28T17:50:48.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d756f56bc] 2024-03-28T17:50:48.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d756df1ff] 2024-03-28T17:50:48.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fae8eb34488] 2024-03-28T17:50:48.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 16: (DispatchQueue::entry()+0x51f) [0x7fae8eb3160f] 2024-03-28T17:50:48.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fae8ec04e81] 2024-03-28T17:50:48.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fae8d61f1ca] 2024-03-28T17:50:48.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 19: clone() 2024-03-28T17:50:48.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:50:48.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 2024-03-28T17:50:48.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: -1> 2024-03-28T17:50:48.080+0000 7fae84278700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fae84278700 time 2024-03-28T17:50:48.079445+0000 2024-03-28T17:50:48.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:50:48.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 2024-03-28T17:50:48.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:50:48.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fae8e8a24a1] 2024-03-28T17:50:48.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fae8e8a2667] 2024-03-28T17:50:48.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d75927223] 2024-03-28T17:50:48.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d757a15d6] 2024-03-28T17:50:48.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d757a1da1] 2024-03-28T17:50:48.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d757a42d5] 2024-03-28T17:50:48.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d757f4233] 2024-03-28T17:50:48.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d757f6197] 2024-03-28T17:50:48.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d756f511b] 2024-03-28T17:50:48.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d756f56bc] 2024-03-28T17:50:48.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d756df1ff] 2024-03-28T17:50:48.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fae8eb34488] 2024-03-28T17:50:48.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 13: (DispatchQueue::entry()+0x51f) [0x7fae8eb3160f] 2024-03-28T17:50:48.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fae8ec04e81] 2024-03-28T17:50:48.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fae8d61f1ca] 2024-03-28T17:50:48.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 16: clone() 2024-03-28T17:50:48.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 2024-03-28T17:50:48.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 0> 2024-03-28T17:50:48.081+0000 7fae84278700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:50:48.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: in thread 7fae84278700 thread_name:ms_dispatch 2024-03-28T17:50:48.333 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 2024-03-28T17:50:48.333 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:50:48.333 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fae8d629d20] 2024-03-28T17:50:48.333 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 2: gsignal() 2024-03-28T17:50:48.333 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 3: abort() 2024-03-28T17:50:48.333 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fae8e8a24fb] 2024-03-28T17:50:48.333 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fae8e8a2667] 2024-03-28T17:50:48.333 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d75927223] 2024-03-28T17:50:48.333 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d757a15d6] 2024-03-28T17:50:48.333 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d757a1da1] 2024-03-28T17:50:48.333 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d757a42d5] 2024-03-28T17:50:48.333 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d757f4233] 2024-03-28T17:50:48.333 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d757f6197] 2024-03-28T17:50:48.333 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d756f511b] 2024-03-28T17:50:48.333 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d756f56bc] 2024-03-28T17:50:48.334 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d756df1ff] 2024-03-28T17:50:48.334 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fae8eb34488] 2024-03-28T17:50:48.334 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 16: (DispatchQueue::entry()+0x51f) [0x7fae8eb3160f] 2024-03-28T17:50:48.334 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fae8ec04e81] 2024-03-28T17:50:48.334 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fae8d61f1ca] 2024-03-28T17:50:48.334 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 19: clone() 2024-03-28T17:50:48.334 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:50:48.334 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 2024-03-28T17:50:48.334 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: -1> 2024-03-28T17:50:48.080+0000 7fae84278700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fae84278700 time 2024-03-28T17:50:48.079445+0000 2024-03-28T17:50:48.334 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:50:48.334 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 2024-03-28T17:50:48.334 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:50:48.334 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fae8e8a24a1] 2024-03-28T17:50:48.334 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fae8e8a2667] 2024-03-28T17:50:48.335 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d75927223] 2024-03-28T17:50:48.335 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d757a15d6] 2024-03-28T17:50:48.335 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d757a1da1] 2024-03-28T17:50:48.335 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d757a42d5] 2024-03-28T17:50:48.335 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d757f4233] 2024-03-28T17:50:48.335 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d757f6197] 2024-03-28T17:50:48.335 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d756f511b] 2024-03-28T17:50:48.335 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d756f56bc] 2024-03-28T17:50:48.335 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d756df1ff] 2024-03-28T17:50:48.335 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fae8eb34488] 2024-03-28T17:50:48.335 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 13: (DispatchQueue::entry()+0x51f) [0x7fae8eb3160f] 2024-03-28T17:50:48.335 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fae8ec04e81] 2024-03-28T17:50:48.335 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fae8d61f1ca] 2024-03-28T17:50:48.335 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 16: clone() 2024-03-28T17:50:48.336 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 2024-03-28T17:50:48.336 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 0> 2024-03-28T17:50:48.081+0000 7fae84278700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:50:48.336 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: in thread 7fae84278700 thread_name:ms_dispatch 2024-03-28T17:50:48.336 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 2024-03-28T17:50:48.336 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:50:48.336 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fae8d629d20] 2024-03-28T17:50:48.336 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 2: gsignal() 2024-03-28T17:50:48.336 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 3: abort() 2024-03-28T17:50:48.336 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fae8e8a24fb] 2024-03-28T17:50:48.336 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fae8e8a2667] 2024-03-28T17:50:48.336 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d75927223] 2024-03-28T17:50:48.336 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d757a15d6] 2024-03-28T17:50:48.336 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d757a1da1] 2024-03-28T17:50:48.336 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d757a42d5] 2024-03-28T17:50:48.337 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d757f4233] 2024-03-28T17:50:48.337 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d757f6197] 2024-03-28T17:50:48.337 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d756f511b] 2024-03-28T17:50:48.337 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d756f56bc] 2024-03-28T17:50:48.337 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d756df1ff] 2024-03-28T17:50:48.337 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fae8eb34488] 2024-03-28T17:50:48.337 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 16: (DispatchQueue::entry()+0x51f) [0x7fae8eb3160f] 2024-03-28T17:50:48.337 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fae8ec04e81] 2024-03-28T17:50:48.337 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fae8d61f1ca] 2024-03-28T17:50:48.337 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 19: clone() 2024-03-28T17:50:48.337 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:50:48.337 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[66588]: 2024-03-28T17:50:48.337 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 podman[70969]: 2024-03-28 17:50:48.14047041 +0000 UTC m=+0.020414736 container died 124475ae99a5b3a5483c66b5ef3411709281e637887b1d0c7a2beaf85bc51f9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T17:50:48.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:48 smithi045 ceph-mon[31773]: pgmap v58: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 2.6 MiB/s wr, 32 op/s 2024-03-28T17:50:48.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:48 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/2876354410,v1:172.21.15.151:6835/2876354410] up:rejoin 2024-03-28T17:50:48.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:48 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:50:48.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:48 smithi045 ceph-mon[31773]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-28T17:50:48.835 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 2.82 MB/sec warmup 21 sec latency 16731.564 ms 2024-03-28T17:50:49.203 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:49 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:50:49.203 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:49 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:50:49.203 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:49 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/2876354410,v1:172.21.15.151:6835/2876354410] up:active 2024-03-28T17:50:49.203 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:49 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T17:50:49.203 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:49 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:50:49.203 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 podman[70969]: 2024-03-28 17:50:48.952224978 +0000 UTC m=+0.832169310 container remove 124475ae99a5b3a5483c66b5ef3411709281e637887b1d0c7a2beaf85bc51f9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T17:50:49.203 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:48 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:50:49.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:49 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:50:49.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:49 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:50:49.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:49 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/2876354410,v1:172.21.15.151:6835/2876354410] up:active 2024-03-28T17:50:49.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:49 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T17:50:49.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:49 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:50:49.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:49 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:50:49.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:49 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:50:49.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:49 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/2876354410,v1:172.21.15.151:6835/2876354410] up:active 2024-03-28T17:50:49.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:49 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T17:50:49.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:49 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:50:49.573 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:49 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-28T17:50:49.835 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 2.69 MB/sec warmup 22 sec latency 17731.704 ms 2024-03-28T17:50:50.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:50 smithi045 ceph-mon[31773]: pgmap v59: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 2.6 MiB/s wr, 32 op/s 2024-03-28T17:50:50.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:50 smithi151 ceph-mon[36481]: pgmap v59: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 2.6 MiB/s wr, 32 op/s 2024-03-28T17:50:50.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:50 smithi071 ceph-mon[36597]: pgmap v59: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 2.6 MiB/s wr, 32 op/s 2024-03-28T17:50:50.835 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 2.57 MB/sec warmup 23 sec latency 18731.868 ms 2024-03-28T17:50:51.835 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 2.47 MB/sec warmup 24 sec latency 19731.969 ms 2024-03-28T17:50:52.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:52 smithi045 ceph-mon[31773]: pgmap v60: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 1.3 MiB/s wr, 34 op/s 2024-03-28T17:50:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:52 smithi151 ceph-mon[36481]: pgmap v60: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 1.3 MiB/s wr, 34 op/s 2024-03-28T17:50:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:52 smithi071 ceph-mon[36597]: pgmap v60: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 1.3 MiB/s wr, 34 op/s 2024-03-28T17:50:52.835 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 2.37 MB/sec warmup 25 sec latency 20732.070 ms 2024-03-28T17:50:53.835 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 2.28 MB/sec warmup 26 sec latency 21732.202 ms 2024-03-28T17:50:54.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:54 smithi151 ceph-mon[36481]: pgmap v61: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 1.3 MiB/s wr, 34 op/s 2024-03-28T17:50:54.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:54 smithi045 ceph-mon[31773]: pgmap v61: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 1.3 MiB/s wr, 34 op/s 2024-03-28T17:50:54.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:54 smithi071 ceph-mon[36597]: pgmap v61: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 1.3 MiB/s wr, 34 op/s 2024-03-28T17:50:54.835 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 2.19 MB/sec warmup 27 sec latency 22732.309 ms 2024-03-28T17:50:55.836 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 2.11 MB/sec warmup 28 sec latency 23732.465 ms 2024-03-28T17:50:56.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:56 smithi045 ceph-mon[31773]: pgmap v62: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 1.2 MiB/s wr, 34 op/s 2024-03-28T17:50:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:56 smithi071 ceph-mon[36597]: pgmap v62: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 1.2 MiB/s wr, 34 op/s 2024-03-28T17:50:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:56 smithi151 ceph-mon[36481]: pgmap v62: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 1.2 MiB/s wr, 34 op/s 2024-03-28T17:50:56.836 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 2.04 MB/sec warmup 29 sec latency 24732.613 ms 2024-03-28T17:50:57.836 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.97 MB/sec warmup 30 sec latency 25732.800 ms 2024-03-28T17:50:58.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:50:58 smithi045 ceph-mon[31773]: pgmap v63: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 105 KiB/s wr, 13 op/s 2024-03-28T17:50:58.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:50:58 smithi071 ceph-mon[36597]: pgmap v63: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 105 KiB/s wr, 13 op/s 2024-03-28T17:50:58.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:50:58 smithi151 ceph-mon[36481]: pgmap v63: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 105 KiB/s wr, 13 op/s 2024-03-28T17:50:58.836 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.91 MB/sec warmup 31 sec latency 26732.958 ms 2024-03-28T17:50:58.872 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:50:58.872 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:50:59.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.116+0000 7f2b85a00640 1 -- 172.21.15.45:0/2324852993 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b801026d0 msgr2=0x7f2b80102ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:59.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.116+0000 7f2b85a00640 1 --2- 172.21.15.45:0/2324852993 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b801026d0 0x7f2b80102ab0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2b70009ca0 tx=0x7f2b7002c190 comp rx=0 tx=0).stop 2024-03-28T17:50:59.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.116+0000 7f2b85a00640 1 -- 172.21.15.45:0/2324852993 shutdown_connections 2024-03-28T17:50:59.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.116+0000 7f2b85a00640 1 --2- 172.21.15.45:0/2324852993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b8010baf0 0x7f2b8010df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.116+0000 7f2b85a00640 1 --2- 172.21.15.45:0/2324852993 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b80102ff0 0x7f2b8010b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.116+0000 7f2b85a00640 1 --2- 172.21.15.45:0/2324852993 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b801026d0 0x7f2b80102ab0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.116+0000 7f2b85a00640 1 -- 172.21.15.45:0/2324852993 >> 172.21.15.45:0/2324852993 conn(0x7f2b800fbef0 msgr2=0x7f2b800fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:59.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.116+0000 7f2b85a00640 1 -- 172.21.15.45:0/2324852993 shutdown_connections 2024-03-28T17:50:59.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.116+0000 7f2b85a00640 1 -- 172.21.15.45:0/2324852993 wait complete. 2024-03-28T17:50:59.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.118+0000 7f2b85a00640 1 Processor -- start 2024-03-28T17:50:59.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.118+0000 7f2b85a00640 1 -- start start 2024-03-28T17:50:59.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.118+0000 7f2b85a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b801026d0 0x7f2b80199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.118+0000 7f2b85a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b80102ff0 0x7f2b8019a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.118+0000 7f2b85a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b8010baf0 0x7f2b8019e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.118+0000 7f2b85a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b801118d0 con 0x7f2b801026d0 2024-03-28T17:50:59.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.118+0000 7f2b85a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b80111750 con 0x7f2b8010baf0 2024-03-28T17:50:59.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.118+0000 7f2b85a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b80111a50 con 0x7f2b80102ff0 2024-03-28T17:50:59.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.118+0000 7f2b7fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b801026d0 0x7f2b80199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.119+0000 7f2b7fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b801026d0 0x7f2b80199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36352/0 (socket says 172.21.15.45:36352) 2024-03-28T17:50:59.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.119+0000 7f2b7fe00640 1 -- 172.21.15.45:0/1872007627 learned_addr learned my addr 172.21.15.45:0/1872007627 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:59.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.119+0000 7f2b85000640 1 --2- 172.21.15.45:0/1872007627 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b8010baf0 0x7f2b8019e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.119+0000 7f2b7f400640 1 --2- 172.21.15.45:0/1872007627 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b80102ff0 0x7f2b8019a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.119+0000 7f2b7fe00640 1 -- 172.21.15.45:0/1872007627 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b80102ff0 msgr2=0x7f2b8019a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:59.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.119+0000 7f2b7fe00640 1 --2- 172.21.15.45:0/1872007627 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b80102ff0 0x7f2b8019a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.119+0000 7f2b7fe00640 1 -- 172.21.15.45:0/1872007627 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b8010baf0 msgr2=0x7f2b8019e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:59.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.119+0000 7f2b7fe00640 1 --2- 172.21.15.45:0/1872007627 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b8010baf0 0x7f2b8019e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.119+0000 7f2b7fe00640 1 -- 172.21.15.45:0/1872007627 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8019efe0 con 0x7f2b801026d0 2024-03-28T17:50:59.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.119+0000 7f2b85000640 1 --2- 172.21.15.45:0/1872007627 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b8010baf0 0x7f2b8019e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:50:59.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.119+0000 7f2b7fe00640 1 --2- 172.21.15.45:0/1872007627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b801026d0 0x7f2b80199ff0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2b70005e30 tx=0x7f2b70035d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:59.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.120+0000 7f2b7cc00640 1 -- 172.21.15.45:0/1872007627 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b70039040 con 0x7f2b801026d0 2024-03-28T17:50:59.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.120+0000 7f2b85a00640 1 -- 172.21.15.45:0/1872007627 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b8019f270 con 0x7f2b801026d0 2024-03-28T17:50:59.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.120+0000 7f2b7cc00640 1 -- 172.21.15.45:0/1872007627 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2b70044070 con 0x7f2b801026d0 2024-03-28T17:50:59.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.120+0000 7f2b85a00640 1 -- 172.21.15.45:0/1872007627 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b801a6b60 con 0x7f2b801026d0 2024-03-28T17:50:59.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.120+0000 7f2b7cc00640 1 -- 172.21.15.45:0/1872007627 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b70032870 con 0x7f2b801026d0 2024-03-28T17:50:59.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.123+0000 7f2b89e1c740 1 Processor -- start 2024-03-28T17:50:59.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.123+0000 7f2b7cc00640 1 -- 172.21.15.45:0/1872007627 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f2b70032a10 con 0x7f2b801026d0 2024-03-28T17:50:59.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.124+0000 7f2b89e1c740 1 -- start start 2024-03-28T17:50:59.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.124+0000 7f2b89e1c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563a36631dd0 0x563a36630b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.124+0000 7f2b89e1c740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a366310f0 0x563a36631570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.124+0000 7f2b89e1c740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563a3663a460 0x563a3663c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.124+0000 7f2b89e1c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a36595250 con 0x563a36631dd0 2024-03-28T17:50:59.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.124+0000 7f2b89e1c740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a3653da30 con 0x563a366310f0 2024-03-28T17:50:59.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.124+0000 7f2b89e1c740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a365912f0 con 0x563a3663a460 2024-03-28T17:50:59.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.124+0000 7f2b7fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a366310f0 0x563a36631570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.124+0000 7f2b85000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563a36631dd0 0x563a36630b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.124+0000 7f2b7fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a366310f0 0x563a36631570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39712/0 (socket says 172.21.15.45:39712) 2024-03-28T17:50:59.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.124+0000 7f2b7fe00640 1 -- 172.21.15.45:0/3334145032 learned_addr learned my addr 172.21.15.45:0/3334145032 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:59.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.124+0000 7f2b7f400640 1 --2- 172.21.15.45:0/3334145032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563a3663a460 0x563a3663c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.125+0000 7f2b7fe00640 1 -- 172.21.15.45:0/3334145032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563a3663a460 msgr2=0x563a3663c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:59.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.125+0000 7f2b7fe00640 1 --2- 172.21.15.45:0/3334145032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563a3663a460 0x563a3663c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.125+0000 7f2b7fe00640 1 -- 172.21.15.45:0/3334145032 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563a36631dd0 msgr2=0x563a36630b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:59.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.125+0000 7f2b7fe00640 1 --2- 172.21.15.45:0/3334145032 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563a36631dd0 0x563a36630b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.125+0000 7f2b7fe00640 1 -- 172.21.15.45:0/3334145032 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a36528690 con 0x563a366310f0 2024-03-28T17:50:59.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.125+0000 7f2b7f400640 1 --2- 172.21.15.45:0/3334145032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563a3663a460 0x563a3663c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:50:59.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.125+0000 7f2b7fe00640 1 --2- 172.21.15.45:0/3334145032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a366310f0 0x563a36631570 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f2b700ee9e0 tx=0x7f2b700f93f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b0e14a306a3cab5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:59.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.125+0000 7f2b75600640 1 -- 172.21.15.45:0/3334145032 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b700f34e0 con 0x563a366310f0 2024-03-28T17:50:59.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.125+0000 7f2b75600640 1 -- 172.21.15.45:0/3334145032 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2b700fe030 con 0x563a366310f0 2024-03-28T17:50:59.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.125+0000 7f2b75600640 1 -- 172.21.15.45:0/3334145032 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b701023d0 con 0x563a366310f0 2024-03-28T17:50:59.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.125+0000 7f2b7cc00640 1 --2- 172.21.15.45:0/1872007627 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f2b640a2970 0x7f2b640a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.126+0000 7f2b7cc00640 1 -- 172.21.15.45:0/1872007627 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8598+0+0 (secure 0 0 0) 0x7f2b700f12f0 con 0x7f2b801026d0 2024-03-28T17:50:59.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.126+0000 7f2b85000640 1 --2- 172.21.15.45:0/1872007627 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f2b640a2970 0x7f2b640a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.126+0000 7f2b89e1c740 1 -- 172.21.15.45:0/3334145032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a366310f0 msgr2=0x563a36631570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:59.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.126+0000 7f2b89e1c740 1 --2- 172.21.15.45:0/3334145032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a366310f0 0x563a36631570 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f2b700ee9e0 tx=0x7f2b700f93f0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.126+0000 7f2b89e1c740 1 -- 172.21.15.45:0/3334145032 shutdown_connections 2024-03-28T17:50:59.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.126+0000 7f2b89e1c740 1 --2- 172.21.15.45:0/3334145032 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563a3663a460 0x563a3663c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.126+0000 7f2b89e1c740 1 --2- 172.21.15.45:0/3334145032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a366310f0 0x563a36631570 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.126+0000 7f2b89e1c740 1 --2- 172.21.15.45:0/3334145032 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563a36631dd0 0x563a36630b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.126+0000 7f2b89e1c740 1 -- 172.21.15.45:0/3334145032 >> 172.21.15.45:0/3334145032 conn(0x563a365f8200 msgr2=0x563a3662bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:59.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.126+0000 7f2b89e1c740 1 -- 172.21.15.45:0/3334145032 shutdown_connections 2024-03-28T17:50:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.126+0000 7f2b85000640 1 --2- 172.21.15.45:0/1872007627 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f2b640a2970 0x7f2b640a4e30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2b78005fd0 tx=0x7f2b78005cc0 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.126+0000 7f2b89e1c740 1 -- 172.21.15.45:0/3334145032 wait complete. 2024-03-28T17:50:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.127+0000 7f2b89e1c740 1 -- start start 2024-03-28T17:50:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.127+0000 7f2b89e1c740 1 Processor -- start 2024-03-28T17:50:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.128+0000 7f2b89e1c740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a366310f0 0x563a36639b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.128+0000 7f2b89e1c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563a36631dd0 0x563a36636f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.128+0000 7f2b89e1c740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563a3663a460 0x563a36637450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.128+0000 7f2b89e1c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a3663d090 con 0x563a36631dd0 2024-03-28T17:50:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.128+0000 7f2b89e1c740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a36550e80 con 0x563a366310f0 2024-03-28T17:50:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.128+0000 7f2b89e1c740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a3663d210 con 0x563a3663a460 2024-03-28T17:50:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.128+0000 7f2b89e1c740 10 client.0 _ll_register_callbacks cb 0x563a36563130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:50:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.128+0000 7f2b7fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a366310f0 0x563a36639b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.128+0000 7f2b7fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a366310f0 0x563a36639b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39722/0 (socket says 172.21.15.45:39722) 2024-03-28T17:50:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.128+0000 7f2b7fe00640 1 -- 172.21.15.45:0/3898218710 learned_addr learned my addr 172.21.15.45:0/3898218710 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.128+0000 7f2b7f400640 1 --2- 172.21.15.45:0/3898218710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563a36631dd0 0x563a36636f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.128+0000 7f2b85000640 1 --2- 172.21.15.45:0/3898218710 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563a3663a460 0x563a36637450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.128+0000 7f2b7fe00640 1 -- 172.21.15.45:0/3898218710 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563a3663a460 msgr2=0x563a36637450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.128+0000 7f2b7fe00640 1 --2- 172.21.15.45:0/3898218710 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563a3663a460 0x563a36637450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.128+0000 7f2b7fe00640 1 -- 172.21.15.45:0/3898218710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563a36631dd0 msgr2=0x563a36636f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.128+0000 7f2b7fe00640 1 --2- 172.21.15.45:0/3898218710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563a36631dd0 0x563a36636f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.128+0000 7f2b7fe00640 1 -- 172.21.15.45:0/3898218710 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563a363737d0 con 0x563a366310f0 2024-03-28T17:50:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.128+0000 7f2b7f400640 1 --2- 172.21.15.45:0/3898218710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563a36631dd0 0x563a36636f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:50:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.128+0000 7f2b85000640 1 --2- 172.21.15.45:0/3898218710 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563a3663a460 0x563a36637450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:50:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.129+0000 7f2b7fe00640 1 --2- 172.21.15.45:0/3898218710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a366310f0 0x563a36639b80 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2b700bd940 tx=0x7f2b700f18e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74704d44a00222fb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.129+0000 7f2b6a000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:50:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.129+0000 7f2b6a000640 1 -- 172.21.15.45:0/3898218710 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b700b9c50 con 0x563a366310f0 2024-03-28T17:50:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.129+0000 7f2b6a000640 1 -- 172.21.15.45:0/3898218710 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8598+0+0 (secure 0 0 0) 0x7f2b700ba020 con 0x563a366310f0 2024-03-28T17:50:59.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.129+0000 7f2b89e1c740 1 -- 172.21.15.45:0/3898218710 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2b700fe030 con 0x563a366310f0 2024-03-28T17:50:59.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.129+0000 7f2b6a000640 1 -- 172.21.15.45:0/3898218710 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b700bd310 con 0x563a366310f0 2024-03-28T17:50:59.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.129+0000 7f2b89e1c740 10 client.25823 fetch_fsmap learned FSMap version 31 2024-03-28T17:50:59.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.129+0000 7f2b89e1c740 1 -- 172.21.15.45:0/3898218710 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x563a362a93f0 con 0x563a366310f0 2024-03-28T17:50:59.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.129+0000 7f2b6a000640 1 -- 172.21.15.45:0/3898218710 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2b700fe030 con 0x563a366310f0 2024-03-28T17:50:59.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.129+0000 7f2b6a000640 1 -- 172.21.15.45:0/3898218710 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 31) v1 ==== 7922+0+0 (secure 0 0 0) 0x7f2b700bece0 con 0x563a366310f0 2024-03-28T17:50:59.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.129+0000 7f2b89e1c740 10 client.25823 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T17:50:59.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.129+0000 7f2b89e1c740 10 client.25823 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:50:59.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.129+0000 7f2b89e1c740 20 client.25823 populate_metadata read hostname 'smithi045' 2024-03-28T17:50:59.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.129+0000 7f2b89e1c740 1 --2- 172.21.15.45:0/3898218710 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x563a36649f30 0x563a3664c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.130+0000 7f2b89e1c740 4 client.25823 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:50:59.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.130+0000 7f2b89e1c740 1 -- 172.21.15.45:0/3898218710 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563a365a2790 con 0x563a36649f30 2024-03-28T17:50:59.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.130+0000 7f2b7f400640 1 --2- 172.21.15.45:0/3898218710 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x563a36649f30 0x563a3664c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.130+0000 7f2b7f400640 1 --2- 172.21.15.45:0/3898218710 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x563a36649f30 0x563a3664c320 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46c38c66056016ab server_cookie=35c831d37d5e5c1a in_seq=0 out_seq=0 2024-03-28T17:50:59.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.130+0000 7f2b6a000640 10 client.25823 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:50:59.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.131+0000 7f2b6a000640 1 -- 172.21.15.45:0/3898218710 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563a365a2790 con 0x563a36649f30 2024-03-28T17:50:59.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.131+0000 7f2b6a000640 10 client.25823 handle_command_reply: tid=0 2024-03-28T17:50:59.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.131+0000 7f2b89e1c740 1 client.25823 shutdown 2024-03-28T17:50:59.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.131+0000 7f2b89e1c740 1 -- 172.21.15.45:0/3898218710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a366310f0 msgr2=0x563a36639b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:59.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.131+0000 7f2b89e1c740 1 --2- 172.21.15.45:0/3898218710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a366310f0 0x563a36639b80 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2b700bd940 tx=0x7f2b700f18e0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.131+0000 7f2b89e1c740 1 -- 172.21.15.45:0/3898218710 shutdown_connections 2024-03-28T17:50:59.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.131+0000 7f2b89e1c740 1 --2- 172.21.15.45:0/3898218710 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x563a36649f30 0x563a3664c320 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.131+0000 7f2b89e1c740 1 --2- 172.21.15.45:0/3898218710 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563a3663a460 0x563a36637450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.131+0000 7f2b89e1c740 1 --2- 172.21.15.45:0/3898218710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563a36631dd0 0x563a36636f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.131+0000 7f2b89e1c740 1 --2- 172.21.15.45:0/3898218710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a366310f0 0x563a36639b80 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.131+0000 7f2b6a000640 0 client.25823 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:50:59.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.131+0000 7f2b89e1c740 1 -- 172.21.15.45:0/3898218710 >> 172.21.15.45:0/3898218710 conn(0x563a365f8200 msgr2=0x563a3662d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:59.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.132+0000 7f2b89e1c740 1 -- 172.21.15.45:0/3898218710 shutdown_connections 2024-03-28T17:50:59.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.132+0000 7f2b89e1c740 1 -- 172.21.15.45:0/3898218710 wait complete. 2024-03-28T17:50:59.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.132+0000 7f2b89e1c740 20 client.25823 trim_cache size 0 max 16384 2024-03-28T17:50:59.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.149+0000 7f2b89e1c740 1 Processor -- start 2024-03-28T17:50:59.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.149+0000 7f2b89e1c740 1 -- start start 2024-03-28T17:50:59.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.150+0000 7f2b89e1c740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563a366310f0 0x563a36639640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.150+0000 7f2b89e1c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563a36631dd0 0x563a36639b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.150+0000 7f2b89e1c740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a3663a460 0x563a3663b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.150+0000 7f2b89e1c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b801b18d0 con 0x563a36631dd0 2024-03-28T17:50:59.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.150+0000 7f2b89e1c740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b801af3a0 con 0x563a3663a460 2024-03-28T17:50:59.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.150+0000 7f2b89e1c740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b801ad440 con 0x563a366310f0 2024-03-28T17:50:59.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.150+0000 7f2b85000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a3663a460 0x563a3663b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.150+0000 7f2b7fe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563a366310f0 0x563a36639640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.150+0000 7f2b85000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a3663a460 0x563a3663b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39736/0 (socket says 172.21.15.45:39736) 2024-03-28T17:50:59.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.150+0000 7f2b85000640 1 -- 172.21.15.45:0/2195108314 learned_addr learned my addr 172.21.15.45:0/2195108314 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:59.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.150+0000 7f2b7f400640 1 --2- 172.21.15.45:0/2195108314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563a36631dd0 0x563a36639b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.150+0000 7f2b85000640 1 -- 172.21.15.45:0/2195108314 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563a366310f0 msgr2=0x563a36639640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:59.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.150+0000 7f2b85000640 1 --2- 172.21.15.45:0/2195108314 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563a366310f0 0x563a36639640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.150+0000 7f2b85000640 1 -- 172.21.15.45:0/2195108314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563a36631dd0 msgr2=0x563a36639b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:59.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.150+0000 7f2b85000640 1 --2- 172.21.15.45:0/2195108314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563a36631dd0 0x563a36639b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.150+0000 7f2b85000640 1 -- 172.21.15.45:0/2195108314 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a362690f0 con 0x563a3663a460 2024-03-28T17:50:59.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.151+0000 7f2b85000640 1 --2- 172.21.15.45:0/2195108314 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a3663a460 0x563a3663b850 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2b780108e0 tx=0x7f2b78016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3843b1f8e7a499fd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:59.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.151+0000 7f2b74c00640 1 -- 172.21.15.45:0/2195108314 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7802d070 con 0x563a3663a460 2024-03-28T17:50:59.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.151+0000 7f2b74c00640 1 -- 172.21.15.45:0/2195108314 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2b78028410 con 0x563a3663a460 2024-03-28T17:50:59.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.151+0000 7f2b74c00640 1 -- 172.21.15.45:0/2195108314 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7801c400 con 0x563a3663a460 2024-03-28T17:50:59.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.152+0000 7f2b89e1c740 1 -- 172.21.15.45:0/2195108314 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a3663a460 msgr2=0x563a3663b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:59.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.152+0000 7f2b89e1c740 1 --2- 172.21.15.45:0/2195108314 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a3663a460 0x563a3663b850 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2b780108e0 tx=0x7f2b78016000 comp rx=0 tx=0).stop 2024-03-28T17:50:59.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.152+0000 7f2b89e1c740 1 -- 172.21.15.45:0/2195108314 shutdown_connections 2024-03-28T17:50:59.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.152+0000 7f2b89e1c740 1 --2- 172.21.15.45:0/2195108314 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a3663a460 0x563a3663b850 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.152+0000 7f2b89e1c740 1 --2- 172.21.15.45:0/2195108314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563a36631dd0 0x563a36639b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.152+0000 7f2b89e1c740 1 --2- 172.21.15.45:0/2195108314 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563a366310f0 0x563a36639640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.152+0000 7f2b89e1c740 1 -- 172.21.15.45:0/2195108314 >> 172.21.15.45:0/2195108314 conn(0x563a365f8200 msgr2=0x563a3662b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:59.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.152+0000 7f2b89e1c740 1 -- 172.21.15.45:0/2195108314 shutdown_connections 2024-03-28T17:50:59.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.152+0000 7f2b89e1c740 1 -- 172.21.15.45:0/2195108314 wait complete. 2024-03-28T17:50:59.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.152+0000 7f2b89e1c740 1 -- start start 2024-03-28T17:50:59.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.152+0000 7f2b89e1c740 1 Processor -- start 2024-03-28T17:50:59.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.153+0000 7f2b89e1c740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a366310f0 0x563a36634350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.153+0000 7f2b89e1c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563a36631dd0 0x563a36634890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.153+0000 7f2b89e1c740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563a3663a460 0x563a36638b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.153+0000 7f2b89e1c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b801ada40 con 0x563a36631dd0 2024-03-28T17:50:59.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.153+0000 7f2b89e1c740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b801aed40 con 0x563a366310f0 2024-03-28T17:50:59.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.153+0000 7f2b89e1c740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b801ae080 con 0x563a3663a460 2024-03-28T17:50:59.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.153+0000 7f2b89e1c740 10 client.0 _ll_register_callbacks cb 0x7f2b7800ba30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:50:59.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.153+0000 7f2b7f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563a36631dd0 0x563a36634890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.153+0000 7f2b7fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a366310f0 0x563a36634350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.153+0000 7f2b7f400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563a36631dd0 0x563a36634890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36372/0 (socket says 172.21.15.45:36372) 2024-03-28T17:50:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.153+0000 7f2b7fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a366310f0 0x563a36634350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39740/0 (socket says 172.21.15.45:39740) 2024-03-28T17:50:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.153+0000 7f2b7f400640 1 -- 172.21.15.45:0/3406231630 learned_addr learned my addr 172.21.15.45:0/3406231630 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.153+0000 7f2b85000640 1 --2- 172.21.15.45:0/3406231630 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563a3663a460 0x563a36638b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.154+0000 7f2b7fe00640 1 -- 172.21.15.45:0/3406231630 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563a3663a460 msgr2=0x563a36638b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.154+0000 7f2b7fe00640 1 --2- 172.21.15.45:0/3406231630 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563a3663a460 0x563a36638b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.154+0000 7f2b7fe00640 1 -- 172.21.15.45:0/3406231630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563a36631dd0 msgr2=0x563a36634890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.154+0000 7f2b7fe00640 1 --2- 172.21.15.45:0/3406231630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563a36631dd0 0x563a36634890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.154+0000 7f2b7fe00640 1 -- 172.21.15.45:0/3406231630 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563a36639180 con 0x563a366310f0 2024-03-28T17:50:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.154+0000 7f2b7f400640 1 --2- 172.21.15.45:0/3406231630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563a36631dd0 0x563a36634890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:50:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.154+0000 7f2b7fe00640 1 --2- 172.21.15.45:0/3406231630 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a366310f0 0x563a36634350 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2b7004d000 tx=0x7f2b700f93f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b11eeb18054313a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.154+0000 7f2b6a000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:50:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.154+0000 7f2b6a000640 1 -- 172.21.15.45:0/3406231630 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b700bd4f0 con 0x563a366310f0 2024-03-28T17:50:59.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.154+0000 7f2b6a000640 1 -- 172.21.15.45:0/3406231630 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8598+0+0 (secure 0 0 0) 0x7f2b700c5ec0 con 0x563a366310f0 2024-03-28T17:50:59.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.154+0000 7f2b89e1c740 1 -- 172.21.15.45:0/3406231630 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2b78028410 con 0x563a366310f0 2024-03-28T17:50:59.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.154+0000 7f2b6a000640 1 -- 172.21.15.45:0/3406231630 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b700bd780 con 0x563a366310f0 2024-03-28T17:50:59.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.154+0000 7f2b89e1c740 10 client.25829 fetch_fsmap learned FSMap version 31 2024-03-28T17:50:59.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.154+0000 7f2b89e1c740 1 -- 172.21.15.45:0/3406231630 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x563a36639180 con 0x563a366310f0 2024-03-28T17:50:59.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.154+0000 7f2b6a000640 1 -- 172.21.15.45:0/3406231630 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2b78028410 con 0x563a366310f0 2024-03-28T17:50:59.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.155+0000 7f2b6a000640 1 -- 172.21.15.45:0/3406231630 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 31) v1 ==== 7922+0+0 (secure 0 0 0) 0x7f2b700bece0 con 0x563a366310f0 2024-03-28T17:50:59.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.155+0000 7f2b89e1c740 10 client.25829 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T17:50:59.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.155+0000 7f2b89e1c740 10 client.25829 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:50:59.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.155+0000 7f2b89e1c740 20 client.25829 populate_metadata read hostname 'smithi045' 2024-03-28T17:50:59.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.155+0000 7f2b89e1c740 1 --2- 172.21.15.45:0/3406231630 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x563a366a6210 0x563a366a8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.155+0000 7f2b89e1c740 4 client.25829 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:50:59.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.155+0000 7f2b89e1c740 1 -- 172.21.15.45:0/3406231630 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563a3659cae0 con 0x563a366a6210 2024-03-28T17:50:59.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.155+0000 7f2b7f400640 1 --2- 172.21.15.45:0/3406231630 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x563a366a6210 0x563a366a8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.156+0000 7f2b7f400640 1 --2- 172.21.15.45:0/3406231630 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x563a366a6210 0x563a366a8630 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8844d2109b658197 server_cookie=c39bf399d8bf5001 in_seq=0 out_seq=0 2024-03-28T17:50:59.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.156+0000 7f2b6a000640 10 client.25829 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:50:59.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.156+0000 7f2b6a000640 1 -- 172.21.15.45:0/3406231630 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x563a3659cae0 con 0x563a366a6210 2024-03-28T17:50:59.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.156+0000 7f2b6a000640 10 client.25829 handle_command_reply: tid=0 2024-03-28T17:50:59.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.156+0000 7f2b89e1c740 1 client.25829 shutdown 2024-03-28T17:50:59.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.156+0000 7f2b89e1c740 1 -- 172.21.15.45:0/3406231630 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a366310f0 msgr2=0x563a36634350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:59.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.156+0000 7f2b89e1c740 1 --2- 172.21.15.45:0/3406231630 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a366310f0 0x563a36634350 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2b7004d000 tx=0x7f2b700f93f0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.156+0000 7f2b89e1c740 1 -- 172.21.15.45:0/3406231630 shutdown_connections 2024-03-28T17:50:59.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.156+0000 7f2b89e1c740 1 --2- 172.21.15.45:0/3406231630 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x563a366a6210 0x563a366a8630 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.156+0000 7f2b89e1c740 1 --2- 172.21.15.45:0/3406231630 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x563a3663a460 0x563a36638b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.156+0000 7f2b89e1c740 1 --2- 172.21.15.45:0/3406231630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563a36631dd0 0x563a36634890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.156+0000 7f2b6a000640 0 client.25829 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:50:59.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.156+0000 7f2b89e1c740 1 --2- 172.21.15.45:0/3406231630 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563a366310f0 0x563a36634350 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.156+0000 7f2b89e1c740 1 -- 172.21.15.45:0/3406231630 >> 172.21.15.45:0/3406231630 conn(0x563a365f8200 msgr2=0x563a36635720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:59.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.156+0000 7f2b89e1c740 1 -- 172.21.15.45:0/3406231630 shutdown_connections 2024-03-28T17:50:59.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.156+0000 7f2b89e1c740 1 -- 172.21.15.45:0/3406231630 wait complete. 2024-03-28T17:50:59.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.156+0000 7f2b89e1c740 20 client.25829 trim_cache size 0 max 16384 2024-03-28T17:50:59.161 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:50:59.161 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-28T17:50:59.161 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-28T17:50:59.161 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:50:59.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.157+0000 7f2b85a00640 1 -- 172.21.15.45:0/1872007627 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f2b640a2970 msgr2=0x7f2b640a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:59.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.157+0000 7f2b85a00640 1 --2- 172.21.15.45:0/1872007627 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f2b640a2970 0x7f2b640a4e30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2b78005fd0 tx=0x7f2b78005cc0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.157+0000 7f2b85a00640 1 -- 172.21.15.45:0/1872007627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b801026d0 msgr2=0x7f2b80199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:59.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.157+0000 7f2b85a00640 1 --2- 172.21.15.45:0/1872007627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b801026d0 0x7f2b80199ff0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2b70005e30 tx=0x7f2b70035d70 comp rx=0 tx=0).stop 2024-03-28T17:50:59.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.157+0000 7f2b85a00640 1 -- 172.21.15.45:0/1872007627 shutdown_connections 2024-03-28T17:50:59.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.157+0000 7f2b85a00640 1 --2- 172.21.15.45:0/1872007627 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f2b640a2970 0x7f2b640a4e30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.157+0000 7f2b85a00640 1 --2- 172.21.15.45:0/1872007627 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b8010baf0 0x7f2b8019e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.157+0000 7f2b85a00640 1 --2- 172.21.15.45:0/1872007627 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b80102ff0 0x7f2b8019a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.157+0000 7f2b85a00640 1 --2- 172.21.15.45:0/1872007627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b801026d0 0x7f2b80199ff0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.157+0000 7f2b85a00640 1 -- 172.21.15.45:0/1872007627 >> 172.21.15.45:0/1872007627 conn(0x7f2b800fbef0 msgr2=0x7f2b800fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:59.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.157+0000 7f2b85a00640 1 -- 172.21.15.45:0/1872007627 shutdown_connections 2024-03-28T17:50:59.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.157+0000 7f2b85a00640 1 -- 172.21.15.45:0/1872007627 wait complete. 2024-03-28T17:50:59.175 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T17:50:59.175 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T17:50:59.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.394+0000 7fbb5d600640 1 -- 172.21.15.45:0/3781707921 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb58075bd0 msgr2=0x7fbb58075fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:59.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.394+0000 7fbb5d600640 1 --2- 172.21.15.45:0/3781707921 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb58075bd0 0x7fbb58075fb0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fbb48009930 tx=0x7fbb4802bc30 comp rx=0 tx=0).stop 2024-03-28T17:50:59.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.394+0000 7fbb5d600640 1 -- 172.21.15.45:0/3781707921 shutdown_connections 2024-03-28T17:50:59.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.394+0000 7fbb5d600640 1 --2- 172.21.15.45:0/3781707921 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb58112320 0x7fbb58114710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.394+0000 7fbb5d600640 1 --2- 172.21.15.45:0/3781707921 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb580764f0 0x7fbb58111de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.394+0000 7fbb5d600640 1 --2- 172.21.15.45:0/3781707921 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb58075bd0 0x7fbb58075fb0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.394+0000 7fbb5d600640 1 -- 172.21.15.45:0/3781707921 >> 172.21.15.45:0/3781707921 conn(0x7fbb580ff130 msgr2=0x7fbb58101550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:59.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.394+0000 7fbb5d600640 1 -- 172.21.15.45:0/3781707921 shutdown_connections 2024-03-28T17:50:59.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.394+0000 7fbb5d600640 1 -- 172.21.15.45:0/3781707921 wait complete. 2024-03-28T17:50:59.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.395+0000 7fbb5d600640 1 Processor -- start 2024-03-28T17:50:59.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.396+0000 7fbb5d600640 1 -- start start 2024-03-28T17:50:59.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.396+0000 7fbb5d600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb58075bd0 0x7fbb5819e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.396+0000 7fbb5d600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb580764f0 0x7fbb5819ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.396+0000 7fbb5d600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb58112320 0x7fbb581a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.396+0000 7fbb5d600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb58117ed0 con 0x7fbb58112320 2024-03-28T17:50:59.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.396+0000 7fbb5d600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb58117d50 con 0x7fbb58075bd0 2024-03-28T17:50:59.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.396+0000 7fbb5d600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb58118050 con 0x7fbb580764f0 2024-03-28T17:50:59.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.396+0000 7fbb57e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb58075bd0 0x7fbb5819e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.396+0000 7fbb57e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb58075bd0 0x7fbb5819e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39752/0 (socket says 172.21.15.45:39752) 2024-03-28T17:50:59.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.396+0000 7fbb57e00640 1 -- 172.21.15.45:0/3829083011 learned_addr learned my addr 172.21.15.45:0/3829083011 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:59.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.397+0000 7fbb5cc00640 1 --2- 172.21.15.45:0/3829083011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb58112320 0x7fbb581a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.397+0000 7fbb4fe00640 1 --2- 172.21.15.45:0/3829083011 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb580764f0 0x7fbb5819ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.397+0000 7fbb57e00640 1 -- 172.21.15.45:0/3829083011 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb580764f0 msgr2=0x7fbb5819ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:59.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.397+0000 7fbb57e00640 1 --2- 172.21.15.45:0/3829083011 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb580764f0 0x7fbb5819ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.397+0000 7fbb57e00640 1 -- 172.21.15.45:0/3829083011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb58112320 msgr2=0x7fbb581a2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:59.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.397+0000 7fbb57e00640 1 --2- 172.21.15.45:0/3829083011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb58112320 0x7fbb581a2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.397+0000 7fbb57e00640 1 -- 172.21.15.45:0/3829083011 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb581a34f0 con 0x7fbb58075bd0 2024-03-28T17:50:59.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.397+0000 7fbb57e00640 1 --2- 172.21.15.45:0/3829083011 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb58075bd0 0x7fbb5819e500 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbb48009900 tx=0x7fbb48004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:59.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.398+0000 7fbb55600640 1 -- 172.21.15.45:0/3829083011 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb48043020 con 0x7fbb58075bd0 2024-03-28T17:50:59.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.398+0000 7fbb55600640 1 -- 172.21.15.45:0/3829083011 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb480054e0 con 0x7fbb58075bd0 2024-03-28T17:50:59.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.398+0000 7fbb5d600640 1 -- 172.21.15.45:0/3829083011 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb581a3780 con 0x7fbb58075bd0 2024-03-28T17:50:59.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.398+0000 7fbb55600640 1 -- 172.21.15.45:0/3829083011 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb48003b80 con 0x7fbb58075bd0 2024-03-28T17:50:59.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.398+0000 7fbb5d600640 1 -- 172.21.15.45:0/3829083011 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb581ab070 con 0x7fbb58075bd0 2024-03-28T17:50:59.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.399+0000 7fbb5d600640 1 -- 172.21.15.45:0/3829083011 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb58068800 con 0x7fbb58075bd0 2024-03-28T17:50:59.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.401+0000 7fbb55600640 1 -- 172.21.15.45:0/3829083011 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fbb48046020 con 0x7fbb58075bd0 2024-03-28T17:50:59.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.403+0000 7fbb55600640 1 --2- 172.21.15.45:0/3829083011 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbb3c0a29f0 0x7fbb3c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.403+0000 7fbb55600640 1 -- 172.21.15.45:0/3829083011 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8598+0+0 (secure 0 0 0) 0x7fbb480f1270 con 0x7fbb58075bd0 2024-03-28T17:50:59.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.403+0000 7fbb4fe00640 1 --2- 172.21.15.45:0/3829083011 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbb3c0a29f0 0x7fbb3c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.404+0000 7fbb4fe00640 1 --2- 172.21.15.45:0/3829083011 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbb3c0a29f0 0x7fbb3c0a4eb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbb58101600 tx=0x7fbb40005f90 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:59.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.405+0000 7fbb55600640 1 -- 172.21.15.45:0/3829083011 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbb4804d020 con 0x7fbb58075bd0 2024-03-28T17:50:59.594 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:59 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 1. 2024-03-28T17:50:59.594 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:59 smithi151 systemd[1]: Stopped Ceph mds.f for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:50:59.594 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:59 smithi151 systemd[1]: Starting Ceph mds.f for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:50:59.594 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:59 smithi151 podman[71068]: 2024-03-28 17:50:59.459625513 +0000 UTC m=+0.018160789 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:50:59.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.650+0000 7fbb5d600640 1 -- 172.21.15.45:0/3829083011 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb5819f850 con 0x7fbb58075bd0 2024-03-28T17:50:59.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.651+0000 7fbb55600640 1 -- 172.21.15.45:0/3829083011 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+11396 (secure 0 0 0) 0x7fbb5819f850 con 0x7fbb58075bd0 2024-03-28T17:50:59.652 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-28T17:50:59.652 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14388,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/333952790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":333952790},{"type":"v1","addr":"172.21.15.45:6835","nonce":333952790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6839/2313500150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":2313500150},{"type":"v1","addr":"172.21.15.45:6839","nonce":2313500150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6841/2647972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":2647972571},{"type":"v1","addr":"172.21.15.45:6841","nonce":2647972571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6833/1898596562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1898596562},{"type":"v1","addr":"172.21.15.151:6833","nonce":1898596562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3145454285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3145454285},{"type":"v1","addr":"172.21.15.71:6833","nonce":3145454285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/1262971887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1262971887},{"type":"v1","addr":"172.21.15.45:6837","nonce":1262971887}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T17:46:12.261135+0000","modified":"2024-03-28T17:50:48.054708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24584,"mds_1":24530,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"f","rank":2,"incarnation":26,"state":"up:active","state_seq":79,"addr":"172.21.15.151:6835/2876354410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":2876354410},{"type":"v1","addr":"172.21.15.151:6835","nonce":2876354410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24500":{"gid":24500,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/4234998385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":4234998385},{"type":"v1","addr":"172.21.15.71:6835","nonce":4234998385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6837/1238964845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6836","nonce":1238964845},{"type":"v1","addr":"172.21.15.151:6837","nonce":1238964845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6837/2008752655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2008752655},{"type":"v1","addr":"172.21.15.71:6837","nonce":2008752655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.71:6839/2709938800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":2709938800},{"type":"v1","addr":"172.21.15.71:6839","nonce":2709938800}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_25780":{"gid":25780,"name":"i","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.151:6839/1272148386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6838","nonce":1272148386},{"type":"v1","addr":"172.21.15.151:6839","nonce":1272148386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24584,"qdb_cluster":[24499,24530,24584]},"id":1}]} 2024-03-28T17:50:59.652 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 31 2024-03-28T17:50:59.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.654+0000 7fbb5d600640 1 -- 172.21.15.45:0/3829083011 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbb3c0a29f0 msgr2=0x7fbb3c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:59.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.654+0000 7fbb5d600640 1 --2- 172.21.15.45:0/3829083011 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbb3c0a29f0 0x7fbb3c0a4eb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbb58101600 tx=0x7fbb40005f90 comp rx=0 tx=0).stop 2024-03-28T17:50:59.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.654+0000 7fbb5d600640 1 -- 172.21.15.45:0/3829083011 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb58075bd0 msgr2=0x7fbb5819e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:59.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.654+0000 7fbb5d600640 1 --2- 172.21.15.45:0/3829083011 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb58075bd0 0x7fbb5819e500 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbb48009900 tx=0x7fbb48004360 comp rx=0 tx=0).stop 2024-03-28T17:50:59.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.654+0000 7fbb5d600640 1 -- 172.21.15.45:0/3829083011 shutdown_connections 2024-03-28T17:50:59.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.654+0000 7fbb5d600640 1 --2- 172.21.15.45:0/3829083011 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbb3c0a29f0 0x7fbb3c0a4eb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.654+0000 7fbb5d600640 1 --2- 172.21.15.45:0/3829083011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb58112320 0x7fbb581a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.654+0000 7fbb5d600640 1 --2- 172.21.15.45:0/3829083011 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb580764f0 0x7fbb5819ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.654+0000 7fbb5d600640 1 --2- 172.21.15.45:0/3829083011 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb58075bd0 0x7fbb5819e500 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.654+0000 7fbb5d600640 1 -- 172.21.15.45:0/3829083011 >> 172.21.15.45:0/3829083011 conn(0x7fbb580ff130 msgr2=0x7fbb58113ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:59.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.654+0000 7fbb5d600640 1 -- 172.21.15.45:0/3829083011 shutdown_connections 2024-03-28T17:50:59.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.654+0000 7fbb5d600640 1 -- 172.21.15.45:0/3829083011 wait complete. 2024-03-28T17:50:59.674 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T17:50:59.675 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T17:50:59.836 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.85 MB/sec warmup 32 sec latency 27733.043 ms 2024-03-28T17:50:59.846 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:59 smithi151 podman[71068]: 2024-03-28 17:50:59.594612709 +0000 UTC m=+0.153147958 container create 0a51be78584ea4109b0507707af5ec218cfc1f46173b487a100a46a706e7992d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T17:50:59.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.909+0000 7f8501c00640 1 -- 172.21.15.45:0/1689634553 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84fc1064a0 msgr2=0x7f84fc106880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:59.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.909+0000 7f8501c00640 1 --2- 172.21.15.45:0/1689634553 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84fc1064a0 0x7f84fc106880 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f84ec009900 tx=0x7f84ec02bc30 comp rx=0 tx=0).stop 2024-03-28T17:50:59.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.909+0000 7f8501c00640 1 -- 172.21.15.45:0/1689634553 shutdown_connections 2024-03-28T17:50:59.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.909+0000 7f8501c00640 1 --2- 172.21.15.45:0/1689634553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84fc100dc0 0x7f84fc10d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.909+0000 7f8501c00640 1 --2- 172.21.15.45:0/1689634553 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84fc100400 0x7f84fc100880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.909+0000 7f8501c00640 1 --2- 172.21.15.45:0/1689634553 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84fc1064a0 0x7f84fc106880 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.909+0000 7f8501c00640 1 -- 172.21.15.45:0/1689634553 >> 172.21.15.45:0/1689634553 conn(0x7f84fc0fbed0 msgr2=0x7f84fc0fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:59.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.909+0000 7f8501c00640 1 -- 172.21.15.45:0/1689634553 shutdown_connections 2024-03-28T17:50:59.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.909+0000 7f8501c00640 1 -- 172.21.15.45:0/1689634553 wait complete. 2024-03-28T17:50:59.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.910+0000 7f8501c00640 1 Processor -- start 2024-03-28T17:50:59.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.910+0000 7f8501c00640 1 -- start start 2024-03-28T17:50:59.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.911+0000 7f8501c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84fc100400 0x7f84fc10dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.911+0000 7f8501c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84fc100dc0 0x7f84fc10e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.911+0000 7f8501c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84fc1064a0 0x7f84fc10e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.911+0000 7f8501c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc111970 con 0x7f84fc1064a0 2024-03-28T17:50:59.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.911+0000 7f8501c00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc1117f0 con 0x7f84fc100dc0 2024-03-28T17:50:59.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.911+0000 7f8501c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc111af0 con 0x7f84fc100400 2024-03-28T17:50:59.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.911+0000 7f8501200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84fc1064a0 0x7f84fc10e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.911+0000 7f84fbe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84fc100400 0x7f84fc10dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.911+0000 7f84f3400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84fc100dc0 0x7f84fc10e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.911+0000 7f8501200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84fc1064a0 0x7f84fc10e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36394/0 (socket says 172.21.15.45:36394) 2024-03-28T17:50:59.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.911+0000 7f8501200640 1 -- 172.21.15.45:0/3317024674 learned_addr learned my addr 172.21.15.45:0/3317024674 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:59.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.911+0000 7f84f3400640 1 -- 172.21.15.45:0/3317024674 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84fc100400 msgr2=0x7f84fc10dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:59.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.911+0000 7f84f3400640 1 --2- 172.21.15.45:0/3317024674 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84fc100400 0x7f84fc10dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.911+0000 7f84f3400640 1 -- 172.21.15.45:0/3317024674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84fc1064a0 msgr2=0x7f84fc10e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:59.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.911+0000 7f84f3400640 1 --2- 172.21.15.45:0/3317024674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84fc1064a0 0x7f84fc10e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.912+0000 7f84f3400640 1 -- 172.21.15.45:0/3317024674 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84fc1a62f0 con 0x7f84fc100dc0 2024-03-28T17:50:59.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.912+0000 7f8501200640 1 --2- 172.21.15.45:0/3317024674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84fc1064a0 0x7f84fc10e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:50:59.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.912+0000 7f84fbe00640 1 --2- 172.21.15.45:0/3317024674 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84fc100400 0x7f84fc10dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:50:59.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.912+0000 7f84f3400640 1 --2- 172.21.15.45:0/3317024674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84fc100dc0 0x7f84fc10e2a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f84e400e9f0 tx=0x7f84e400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:59.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.912+0000 7f84f9600640 1 -- 172.21.15.45:0/3317024674 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e400ccc0 con 0x7f84fc100dc0 2024-03-28T17:50:59.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.912+0000 7f84f9600640 1 -- 172.21.15.45:0/3317024674 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84e4005020 con 0x7f84fc100dc0 2024-03-28T17:50:59.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.912+0000 7f84f9600640 1 -- 172.21.15.45:0/3317024674 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e40049e0 con 0x7f84fc100dc0 2024-03-28T17:50:59.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.912+0000 7f8501c00640 1 -- 172.21.15.45:0/3317024674 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84fc1a65e0 con 0x7f84fc100dc0 2024-03-28T17:50:59.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.912+0000 7f8501c00640 1 -- 172.21.15.45:0/3317024674 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84fc1a6ae0 con 0x7f84fc100dc0 2024-03-28T17:50:59.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.917+0000 7f84f9600640 1 -- 172.21.15.45:0/3317024674 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f84e4005560 con 0x7f84fc100dc0 2024-03-28T17:50:59.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.917+0000 7f8505fef740 1 Processor -- start 2024-03-28T17:50:59.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.917+0000 7f8505fef740 1 -- start start 2024-03-28T17:50:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.918+0000 7f8505fef740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5be8b0 0x55b8dc5bec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.918+0000 7f8505fef740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b8dc5bf260 0x55b8dc5c7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.918+0000 7f8505fef740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5c7de0 0x55b8dc5ca230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.918+0000 7f8505fef740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8dc5222c0 con 0x55b8dc5bf260 2024-03-28T17:50:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.918+0000 7f8505fef740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8dc4cac90 con 0x55b8dc5c7de0 2024-03-28T17:50:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.918+0000 7f8505fef740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8dc51e360 con 0x55b8dc5be8b0 2024-03-28T17:50:59.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.918+0000 7f8501200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5be8b0 0x55b8dc5bec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.918+0000 7f8501200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5be8b0 0x55b8dc5bec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:39988/0 (socket says 172.21.15.45:39988) 2024-03-28T17:50:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.918+0000 7f8501200640 1 -- 172.21.15.45:0/3074142472 learned_addr learned my addr 172.21.15.45:0/3074142472 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.918+0000 7f84fbe00640 1 --2- 172.21.15.45:0/3074142472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b8dc5bf260 0x55b8dc5c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.918+0000 7f84f3400640 1 --2- 172.21.15.45:0/3074142472 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5c7de0 0x55b8dc5ca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.918+0000 7f8501200640 1 -- 172.21.15.45:0/3074142472 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5c7de0 msgr2=0x55b8dc5ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.918+0000 7f8501200640 1 --2- 172.21.15.45:0/3074142472 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5c7de0 0x55b8dc5ca230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.918+0000 7f8501200640 1 -- 172.21.15.45:0/3074142472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b8dc5bf260 msgr2=0x55b8dc5c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.918+0000 7f8501200640 1 --2- 172.21.15.45:0/3074142472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b8dc5bf260 0x55b8dc5c7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.918+0000 7f8501200640 1 -- 172.21.15.45:0/3074142472 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b8dc4ef080 con 0x55b8dc5be8b0 2024-03-28T17:50:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.919+0000 7f8501200640 1 --2- 172.21.15.45:0/3074142472 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5be8b0 0x55b8dc5bec90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f84f4011780 tx=0x7f84f4011c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a23e7654dcbeb6f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.919+0000 7f84f9600640 1 --2- 172.21.15.45:0/3317024674 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f84e00a29f0 0x7f84e00a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.919+0000 7f84f1600640 1 -- 172.21.15.45:0/3074142472 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f400fb40 con 0x55b8dc5be8b0 2024-03-28T17:50:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.919+0000 7f84f1600640 1 -- 172.21.15.45:0/3074142472 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84f4011e90 con 0x55b8dc5be8b0 2024-03-28T17:50:59.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.919+0000 7f84f1600640 1 -- 172.21.15.45:0/3074142472 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f401a650 con 0x55b8dc5be8b0 2024-03-28T17:50:59.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.919+0000 7f84f9600640 1 -- 172.21.15.45:0/3317024674 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8684+0+0 (secure 0 0 0) 0x7f84e40cee60 con 0x7f84fc100dc0 2024-03-28T17:50:59.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.919+0000 7f84fbe00640 1 --2- 172.21.15.45:0/3317024674 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f84e00a29f0 0x7f84e00a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.920+0000 7f8505fef740 1 -- 172.21.15.45:0/3074142472 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5be8b0 msgr2=0x55b8dc5bec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:59.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.920+0000 7f8505fef740 1 --2- 172.21.15.45:0/3074142472 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5be8b0 0x55b8dc5bec90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f84f4011780 tx=0x7f84f4011c40 comp rx=0 tx=0).stop 2024-03-28T17:50:59.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.920+0000 7f8505fef740 1 -- 172.21.15.45:0/3074142472 shutdown_connections 2024-03-28T17:50:59.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.920+0000 7f84fbe00640 1 --2- 172.21.15.45:0/3317024674 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f84e00a29f0 0x7f84e00a4eb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f84ec02efd0 tx=0x7f84ec000bc0 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:59.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.920+0000 7f8505fef740 1 --2- 172.21.15.45:0/3074142472 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5c7de0 0x55b8dc5ca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.920+0000 7f8505fef740 1 --2- 172.21.15.45:0/3074142472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b8dc5bf260 0x55b8dc5c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.920+0000 7f8505fef740 1 --2- 172.21.15.45:0/3074142472 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5be8b0 0x55b8dc5bec90 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.920+0000 7f8505fef740 1 -- 172.21.15.45:0/3074142472 >> 172.21.15.45:0/3074142472 conn(0x55b8dc5b8f80 msgr2=0x55b8dc5b9360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:59.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.920+0000 7f8505fef740 1 -- 172.21.15.45:0/3074142472 shutdown_connections 2024-03-28T17:50:59.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.920+0000 7f8505fef740 1 -- 172.21.15.45:0/3074142472 wait complete. 2024-03-28T17:50:59.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.921+0000 7f8505fef740 1 -- start start 2024-03-28T17:50:59.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.921+0000 7f8505fef740 1 Processor -- start 2024-03-28T17:50:59.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.921+0000 7f8505fef740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5be8b0 0x55b8dc5c76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.921+0000 7f8505fef740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b8dc5bf260 0x55b8dc5c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.921+0000 7f8505fef740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5c7de0 0x55b8dc5c4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.921+0000 7f8505fef740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8dc5cad50 con 0x55b8dc5bf260 2024-03-28T17:50:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.921+0000 7f8505fef740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8dc4de0d0 con 0x55b8dc5c7de0 2024-03-28T17:50:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.921+0000 7f8505fef740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8dc5caed0 con 0x55b8dc5be8b0 2024-03-28T17:50:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.921+0000 7f8505fef740 10 client.0 _ll_register_callbacks cb 0x55b8dc4e2e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:50:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.921+0000 7f84fbe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5be8b0 0x55b8dc5c76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.922+0000 7f8501200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5c7de0 0x55b8dc5c4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.922+0000 7f8501200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5c7de0 0x55b8dc5c4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39792/0 (socket says 172.21.15.45:39792) 2024-03-28T17:50:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.922+0000 7f84fbe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5be8b0 0x55b8dc5c76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:40000/0 (socket says 172.21.15.45:40000) 2024-03-28T17:50:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.922+0000 7f8501200640 1 -- 172.21.15.45:0/3779819151 learned_addr learned my addr 172.21.15.45:0/3779819151 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.922+0000 7f84f3400640 1 --2- 172.21.15.45:0/3779819151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b8dc5bf260 0x55b8dc5c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.922+0000 7f8501200640 1 -- 172.21.15.45:0/3779819151 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5be8b0 msgr2=0x55b8dc5c76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.922+0000 7f8501200640 1 --2- 172.21.15.45:0/3779819151 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5be8b0 0x55b8dc5c76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.922+0000 7f8501200640 1 -- 172.21.15.45:0/3779819151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b8dc5bf260 msgr2=0x55b8dc5c4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:59.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.922+0000 7f8501200640 1 --2- 172.21.15.45:0/3779819151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b8dc5bf260 0x55b8dc5c4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.922+0000 7f8501200640 1 -- 172.21.15.45:0/3779819151 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b8dc2363f0 con 0x55b8dc5c7de0 2024-03-28T17:50:59.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.922+0000 7f8501200640 1 --2- 172.21.15.45:0/3779819151 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5c7de0 0x55b8dc5c4fb0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f84f4004ca0 tx=0x7f84f4004cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=affaa0321e6ea627 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:59.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.922+0000 7f84ea000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:50:59.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.923+0000 7f84ea000640 1 -- 172.21.15.45:0/3779819151 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f4028020 con 0x55b8dc5c7de0 2024-03-28T17:50:59.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.923+0000 7f84ea000640 1 -- 172.21.15.45:0/3779819151 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8684+0+0 (secure 0 0 0) 0x7f84f400fd50 con 0x55b8dc5c7de0 2024-03-28T17:50:59.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.923+0000 7f8505fef740 1 -- 172.21.15.45:0/3779819151 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f84f4011e90 con 0x55b8dc5c7de0 2024-03-28T17:50:59.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.923+0000 7f84ea000640 1 -- 172.21.15.45:0/3779819151 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84f4033050 con 0x55b8dc5c7de0 2024-03-28T17:50:59.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.923+0000 7f8505fef740 10 client.25853 fetch_fsmap learned FSMap version 33 2024-03-28T17:50:59.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.923+0000 7f8505fef740 1 -- 172.21.15.45:0/3779819151 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55b8dc3007d0 con 0x55b8dc5c7de0 2024-03-28T17:50:59.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.923+0000 7f84ea000640 1 -- 172.21.15.45:0/3779819151 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84f4011e90 con 0x55b8dc5c7de0 2024-03-28T17:50:59.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.923+0000 7f84ea000640 1 -- 172.21.15.45:0/3779819151 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 33) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f84f400f040 con 0x55b8dc5c7de0 2024-03-28T17:50:59.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.923+0000 7f8505fef740 10 client.25853 fetch_fsmap finished waiting for FSMap version 33 2024-03-28T17:50:59.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.923+0000 7f8505fef740 10 client.25853 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-28T17:50:59.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.923+0000 7f8505fef740 20 client.25853 populate_metadata read hostname 'smithi045' 2024-03-28T17:50:59.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.923+0000 7f8505fef740 1 --2- 172.21.15.45:0/3779819151 >> [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] conn(0x55b8dc5d7bc0 0x55b8dc5d9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.923+0000 7f8505fef740 4 client.25853 mds_command: new command op to 25780 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:50:59.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.923+0000 7f8505fef740 1 -- 172.21.15.45:0/3779819151 --> [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b8dc52f800 con 0x55b8dc5d7bc0 2024-03-28T17:50:59.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.924+0000 7f84fbe00640 1 --2- 172.21.15.45:0/3779819151 >> [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] conn(0x55b8dc5d7bc0 0x55b8dc5d9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.924+0000 7f84fbe00640 1 --2- 172.21.15.45:0/3779819151 >> [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] conn(0x55b8dc5d7bc0 0x55b8dc5d9fb0 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1238991d3ab6af5f server_cookie=a10ca7be0781694f in_seq=0 out_seq=0 2024-03-28T17:50:59.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.924+0000 7f84ea000640 10 client.25853 ms_handle_connect on v2:172.21.15.151:6838/1272148386 2024-03-28T17:50:59.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.925+0000 7f84ea000640 1 -- 172.21.15.45:0/3779819151 <== mds.2 v2:172.21.15.151:6838/1272148386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b8dc52f800 con 0x55b8dc5d7bc0 2024-03-28T17:50:59.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.925+0000 7f84ea000640 10 client.25853 handle_command_reply: tid=0 2024-03-28T17:50:59.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.925+0000 7f8505fef740 1 client.25853 shutdown 2024-03-28T17:50:59.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.926+0000 7f8505fef740 1 -- 172.21.15.45:0/3779819151 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5c7de0 msgr2=0x55b8dc5c4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:59.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.926+0000 7f8505fef740 1 --2- 172.21.15.45:0/3779819151 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5c7de0 0x55b8dc5c4fb0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f84f4004ca0 tx=0x7f84f4004cd0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.926+0000 7f8505fef740 1 -- 172.21.15.45:0/3779819151 shutdown_connections 2024-03-28T17:50:59.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.926+0000 7f8505fef740 1 --2- 172.21.15.45:0/3779819151 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5c7de0 0x55b8dc5c4fb0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.926+0000 7f8505fef740 1 --2- 172.21.15.45:0/3779819151 >> [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] conn(0x55b8dc5d7bc0 0x55b8dc5d9fb0 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.926+0000 7f8505fef740 1 --2- 172.21.15.45:0/3779819151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b8dc5bf260 0x55b8dc5c4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.926+0000 7f8505fef740 1 --2- 172.21.15.45:0/3779819151 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5be8b0 0x55b8dc5c76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.926+0000 7f8505fef740 1 -- 172.21.15.45:0/3779819151 >> 172.21.15.45:0/3779819151 conn(0x55b8dc5b8f80 msgr2=0x55b8dc5bacd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:59.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.926+0000 7f84ea000640 0 client.25853 ms_handle_reset on v2:172.21.15.151:6838/1272148386 2024-03-28T17:50:59.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.926+0000 7f8505fef740 1 -- 172.21.15.45:0/3779819151 shutdown_connections 2024-03-28T17:50:59.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.926+0000 7f8505fef740 1 -- 172.21.15.45:0/3779819151 wait complete. 2024-03-28T17:50:59.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.926+0000 7f8505fef740 20 client.25853 trim_cache size 0 max 16384 2024-03-28T17:50:59.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.940+0000 7f8505fef740 1 Processor -- start 2024-03-28T17:50:59.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.940+0000 7f8505fef740 1 -- start start 2024-03-28T17:50:59.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.941+0000 7f8505fef740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5be8b0 0x55b8dc5b7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.941+0000 7f8505fef740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b8dc5bf260 0x55b8dc5c7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.941+0000 7f8505fef740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5c7de0 0x55b8dc5c76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.941+0000 7f8505fef740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc1afa70 con 0x55b8dc5bf260 2024-03-28T17:50:59.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.941+0000 7f8505fef740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc1ad4b0 con 0x55b8dc5be8b0 2024-03-28T17:50:59.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.941+0000 7f8505fef740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc1af410 con 0x55b8dc5c7de0 2024-03-28T17:50:59.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.941+0000 7f84fbe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5be8b0 0x55b8dc5b7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.941+0000 7f84fbe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5be8b0 0x55b8dc5b7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39802/0 (socket says 172.21.15.45:39802) 2024-03-28T17:50:59.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.941+0000 7f84fbe00640 1 -- 172.21.15.45:0/2927123483 learned_addr learned my addr 172.21.15.45:0/2927123483 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:59.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.941+0000 7f84f3400640 1 --2- 172.21.15.45:0/2927123483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b8dc5bf260 0x55b8dc5c7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.941+0000 7f8501200640 1 --2- 172.21.15.45:0/2927123483 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5c7de0 0x55b8dc5c76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.941+0000 7f84fbe00640 1 -- 172.21.15.45:0/2927123483 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5c7de0 msgr2=0x55b8dc5c76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:59.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.941+0000 7f84fbe00640 1 --2- 172.21.15.45:0/2927123483 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5c7de0 0x55b8dc5c76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.941+0000 7f84fbe00640 1 -- 172.21.15.45:0/2927123483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b8dc5bf260 msgr2=0x55b8dc5c7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:59.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.941+0000 7f84fbe00640 1 --2- 172.21.15.45:0/2927123483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b8dc5bf260 0x55b8dc5c7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.941+0000 7f84fbe00640 1 -- 172.21.15.45:0/2927123483 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b8dc5c5320 con 0x55b8dc5be8b0 2024-03-28T17:50:59.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.941+0000 7f8501200640 1 --2- 172.21.15.45:0/2927123483 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5c7de0 0x55b8dc5c76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:50:59.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.941+0000 7f84f3400640 1 --2- 172.21.15.45:0/2927123483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b8dc5bf260 0x55b8dc5c7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:50:59.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.942+0000 7f84fbe00640 1 --2- 172.21.15.45:0/2927123483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5be8b0 0x55b8dc5b7fe0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f84ec009230 tx=0x7f84ec02e070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=114c86ba29107329 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:59.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.942+0000 7f84f0c00640 1 -- 172.21.15.45:0/2927123483 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec05a070 con 0x55b8dc5be8b0 2024-03-28T17:50:59.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.942+0000 7f84f0c00640 1 -- 172.21.15.45:0/2927123483 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84ec005460 con 0x55b8dc5be8b0 2024-03-28T17:50:59.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.942+0000 7f84f0c00640 1 -- 172.21.15.45:0/2927123483 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec055a00 con 0x55b8dc5be8b0 2024-03-28T17:50:59.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.943+0000 7f8505fef740 1 -- 172.21.15.45:0/2927123483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5be8b0 msgr2=0x55b8dc5b7fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:59.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.943+0000 7f8505fef740 1 --2- 172.21.15.45:0/2927123483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5be8b0 0x55b8dc5b7fe0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f84ec009230 tx=0x7f84ec02e070 comp rx=0 tx=0).stop 2024-03-28T17:50:59.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.943+0000 7f8505fef740 1 -- 172.21.15.45:0/2927123483 shutdown_connections 2024-03-28T17:50:59.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.943+0000 7f8505fef740 1 --2- 172.21.15.45:0/2927123483 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5c7de0 0x55b8dc5c76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.943+0000 7f8505fef740 1 --2- 172.21.15.45:0/2927123483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b8dc5bf260 0x55b8dc5c7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.943+0000 7f8505fef740 1 --2- 172.21.15.45:0/2927123483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5be8b0 0x55b8dc5b7fe0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.943+0000 7f8505fef740 1 -- 172.21.15.45:0/2927123483 >> 172.21.15.45:0/2927123483 conn(0x55b8dc5b8f80 msgr2=0x55b8dc5b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:59.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.943+0000 7f8505fef740 1 -- 172.21.15.45:0/2927123483 shutdown_connections 2024-03-28T17:50:59.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.943+0000 7f8505fef740 1 -- 172.21.15.45:0/2927123483 wait complete. 2024-03-28T17:50:59.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.943+0000 7f8505fef740 1 -- start start 2024-03-28T17:50:59.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.943+0000 7f8505fef740 1 Processor -- start 2024-03-28T17:50:59.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.944+0000 7f8505fef740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5be8b0 0x55b8dc5da250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.944+0000 7f8505fef740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b8dc5bf260 0x55b8dc5da790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.944+0000 7f8505fef740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5c7de0 0x55b8dc5dacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.944+0000 7f8505fef740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc1aedb0 con 0x55b8dc5bf260 2024-03-28T17:50:59.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.944+0000 7f8505fef740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc1adab0 con 0x55b8dc5c7de0 2024-03-28T17:50:59.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.944+0000 7f8505fef740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc1ae0f0 con 0x55b8dc5be8b0 2024-03-28T17:50:59.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.944+0000 7f8505fef740 10 client.0 _ll_register_callbacks cb 0x7f84ec009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:50:59.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.944+0000 7f84fbe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5be8b0 0x55b8dc5da250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.944+0000 7f8501200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5c7de0 0x55b8dc5dacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.944+0000 7f84f3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b8dc5bf260 0x55b8dc5da790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.944+0000 7f84fbe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5be8b0 0x55b8dc5da250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:40028/0 (socket says 172.21.15.45:40028) 2024-03-28T17:50:59.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.944+0000 7f84fbe00640 1 -- 172.21.15.45:0/1311442740 learned_addr learned my addr 172.21.15.45:0/1311442740 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:50:59.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.944+0000 7f8501200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5c7de0 0x55b8dc5dacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39812/0 (socket says 172.21.15.45:39812) 2024-03-28T17:50:59.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.945+0000 7f8501200640 1 -- 172.21.15.45:0/1311442740 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5be8b0 msgr2=0x55b8dc5da250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:59.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.945+0000 7f8501200640 1 --2- 172.21.15.45:0/1311442740 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5be8b0 0x55b8dc5da250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.945+0000 7f8501200640 1 -- 172.21.15.45:0/1311442740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b8dc5bf260 msgr2=0x55b8dc5da790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:50:59.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.945+0000 7f8501200640 1 --2- 172.21.15.45:0/1311442740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b8dc5bf260 0x55b8dc5da790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.945+0000 7f8501200640 1 -- 172.21.15.45:0/1311442740 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b8dc6322f0 con 0x55b8dc5c7de0 2024-03-28T17:50:59.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.945+0000 7f84fbe00640 1 --2- 172.21.15.45:0/1311442740 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5be8b0 0x55b8dc5da250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:50:59.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.945+0000 7f8501200640 1 --2- 172.21.15.45:0/1311442740 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5c7de0 0x55b8dc5dacd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f84f4000f80 tx=0x7f84f40114d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b80765bffb377375 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:50:59.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.945+0000 7f84ea000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:50:59.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.945+0000 7f84ea000640 1 -- 172.21.15.45:0/1311442740 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f4028070 con 0x55b8dc5c7de0 2024-03-28T17:50:59.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.945+0000 7f8505fef740 1 -- 172.21.15.45:0/1311442740 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f84ec005460 con 0x55b8dc5c7de0 2024-03-28T17:50:59.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.945+0000 7f84ea000640 1 -- 172.21.15.45:0/1311442740 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8684+0+0 (secure 0 0 0) 0x7f84f400fd60 con 0x55b8dc5c7de0 2024-03-28T17:50:59.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.945+0000 7f84ea000640 1 -- 172.21.15.45:0/1311442740 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84f4045050 con 0x55b8dc5c7de0 2024-03-28T17:50:59.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.945+0000 7f8505fef740 10 client.25859 fetch_fsmap learned FSMap version 33 2024-03-28T17:50:59.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.945+0000 7f8505fef740 1 -- 172.21.15.45:0/1311442740 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55b8dc6335d0 con 0x55b8dc5c7de0 2024-03-28T17:50:59.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.946+0000 7f84ea000640 1 -- 172.21.15.45:0/1311442740 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84ec005460 con 0x55b8dc5c7de0 2024-03-28T17:50:59.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.946+0000 7f84ea000640 1 -- 172.21.15.45:0/1311442740 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 33) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f84f4024460 con 0x55b8dc5c7de0 2024-03-28T17:50:59.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.946+0000 7f8505fef740 10 client.25859 fetch_fsmap finished waiting for FSMap version 33 2024-03-28T17:50:59.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.946+0000 7f8505fef740 10 client.25859 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-28T17:50:59.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.946+0000 7f8505fef740 20 client.25859 populate_metadata read hostname 'smithi045' 2024-03-28T17:50:59.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.946+0000 7f8505fef740 1 --2- 172.21.15.45:0/1311442740 >> [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] conn(0x55b8dc633bd0 0x55b8dc635fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:50:59.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.946+0000 7f8505fef740 4 client.25859 mds_command: new command op to 25780 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T17:50:59.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.946+0000 7f8505fef740 1 -- 172.21.15.45:0/1311442740 --> [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b8dc529b50 con 0x55b8dc633bd0 2024-03-28T17:50:59.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.946+0000 7f84fbe00640 1 --2- 172.21.15.45:0/1311442740 >> [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] conn(0x55b8dc633bd0 0x55b8dc635fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:50:59.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.947+0000 7f84fbe00640 1 --2- 172.21.15.45:0/1311442740 >> [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] conn(0x55b8dc633bd0 0x55b8dc635fc0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=80acbd5016d0f8a8 server_cookie=e9cf22cf14b3becd in_seq=0 out_seq=0 2024-03-28T17:50:59.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.947+0000 7f84ea000640 10 client.25859 ms_handle_connect on v2:172.21.15.151:6838/1272148386 2024-03-28T17:50:59.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.947+0000 7f84ea000640 1 -- 172.21.15.45:0/1311442740 <== mds.2 v2:172.21.15.151:6838/1272148386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b8dc529b50 con 0x55b8dc633bd0 2024-03-28T17:50:59.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.947+0000 7f84ea000640 10 client.25859 handle_command_reply: tid=0 2024-03-28T17:50:59.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.947+0000 7f8505fef740 1 client.25859 shutdown 2024-03-28T17:50:59.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.947+0000 7f8505fef740 1 -- 172.21.15.45:0/1311442740 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5c7de0 msgr2=0x55b8dc5dacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:59.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.947+0000 7f8505fef740 1 --2- 172.21.15.45:0/1311442740 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5c7de0 0x55b8dc5dacd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f84f4000f80 tx=0x7f84f40114d0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.947+0000 7f8505fef740 1 -- 172.21.15.45:0/1311442740 shutdown_connections 2024-03-28T17:50:59.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.947+0000 7f8505fef740 1 --2- 172.21.15.45:0/1311442740 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b8dc5c7de0 0x55b8dc5dacd0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.947+0000 7f8505fef740 1 --2- 172.21.15.45:0/1311442740 >> [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] conn(0x55b8dc633bd0 0x55b8dc635fc0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.947+0000 7f8505fef740 1 --2- 172.21.15.45:0/1311442740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b8dc5bf260 0x55b8dc5da790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.947+0000 7f8505fef740 1 --2- 172.21.15.45:0/1311442740 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b8dc5be8b0 0x55b8dc5da250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.947+0000 7f84ea000640 0 client.25859 ms_handle_reset on v2:172.21.15.151:6838/1272148386 2024-03-28T17:50:59.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.947+0000 7f8505fef740 1 -- 172.21.15.45:0/1311442740 >> 172.21.15.45:0/1311442740 conn(0x55b8dc5b8f80 msgr2=0x55b8dc5b8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:59.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.947+0000 7f8505fef740 1 -- 172.21.15.45:0/1311442740 shutdown_connections 2024-03-28T17:50:59.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.947+0000 7f8505fef740 1 -- 172.21.15.45:0/1311442740 wait complete. 2024-03-28T17:50:59.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.947+0000 7f8505fef740 20 client.25859 trim_cache size 0 max 16384 2024-03-28T17:50:59.954 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-28T17:50:59.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.948+0000 7f8501c00640 1 -- 172.21.15.45:0/3317024674 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f84e00a29f0 msgr2=0x7f84e00a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:59.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.948+0000 7f8501c00640 1 --2- 172.21.15.45:0/3317024674 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f84e00a29f0 0x7f84e00a4eb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f84ec02efd0 tx=0x7f84ec000bc0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.948+0000 7f8501c00640 1 -- 172.21.15.45:0/3317024674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84fc100dc0 msgr2=0x7f84fc10e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:50:59.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.948+0000 7f8501c00640 1 --2- 172.21.15.45:0/3317024674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84fc100dc0 0x7f84fc10e2a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f84e400e9f0 tx=0x7f84e400eeb0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.948+0000 7f8501c00640 1 -- 172.21.15.45:0/3317024674 shutdown_connections 2024-03-28T17:50:59.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.948+0000 7f8501c00640 1 --2- 172.21.15.45:0/3317024674 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f84e00a29f0 0x7f84e00a4eb0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.948+0000 7f8501c00640 1 --2- 172.21.15.45:0/3317024674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84fc1064a0 0x7f84fc10e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.948+0000 7f8501c00640 1 --2- 172.21.15.45:0/3317024674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84fc100dc0 0x7f84fc10e2a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.948+0000 7f8501c00640 1 --2- 172.21.15.45:0/3317024674 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84fc100400 0x7f84fc10dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:50:59.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.948+0000 7f8501c00640 1 -- 172.21.15.45:0/3317024674 >> 172.21.15.45:0/3317024674 conn(0x7f84fc0fbed0 msgr2=0x7f84fc0fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:50:59.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.948+0000 7f8501c00640 1 -- 172.21.15.45:0/3317024674 shutdown_connections 2024-03-28T17:50:59.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:50:59.948+0000 7f8501c00640 1 -- 172.21.15.45:0/3317024674 wait complete. 2024-03-28T17:50:59.966 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T17:50:59.966 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T17:51:00.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.180+0000 7f7a68400640 1 -- 172.21.15.45:0/3219644489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a60107640 msgr2=0x7f7a60107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:00.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.180+0000 7f7a68400640 1 --2- 172.21.15.45:0/3219644489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a60107640 0x7f7a60107a20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7a58009860 tx=0x7f7a5802bb70 comp rx=0 tx=0).stop 2024-03-28T17:51:00.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.180+0000 7f7a68400640 1 -- 172.21.15.45:0/3219644489 shutdown_connections 2024-03-28T17:51:00.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.180+0000 7f7a68400640 1 --2- 172.21.15.45:0/3219644489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a60101ff0 0x7f7a6010e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.180+0000 7f7a68400640 1 --2- 172.21.15.45:0/3219644489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a60101630 0x7f7a60101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.180+0000 7f7a68400640 1 --2- 172.21.15.45:0/3219644489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a60107640 0x7f7a60107a20 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.180+0000 7f7a68400640 1 -- 172.21.15.45:0/3219644489 >> 172.21.15.45:0/3219644489 conn(0x7f7a60078aa0 msgr2=0x7f7a60078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:00.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.180+0000 7f7a68400640 1 -- 172.21.15.45:0/3219644489 shutdown_connections 2024-03-28T17:51:00.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.180+0000 7f7a68400640 1 -- 172.21.15.45:0/3219644489 wait complete. 2024-03-28T17:51:00.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.181+0000 7f7a68400640 1 Processor -- start 2024-03-28T17:51:00.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.181+0000 7f7a68400640 1 -- start start 2024-03-28T17:51:00.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.182+0000 7f7a68400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a60101630 0x7f7a600739a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.182+0000 7f7a68400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a60101ff0 0x7f7a60073ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.182+0000 7f7a68400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a60107640 0x7f7a60074420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.182+0000 7f7a68400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a60111a10 con 0x7f7a60101ff0 2024-03-28T17:51:00.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.182+0000 7f7a68400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a60111890 con 0x7f7a60101630 2024-03-28T17:51:00.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.182+0000 7f7a68400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a60111b90 con 0x7f7a60107640 2024-03-28T17:51:00.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.182+0000 7f7a65a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a60101630 0x7f7a600739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.182+0000 7f7a65000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a60101ff0 0x7f7a60073ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.182+0000 7f7a66400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a60107640 0x7f7a60074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.182+0000 7f7a65a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a60101630 0x7f7a600739a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39832/0 (socket says 172.21.15.45:39832) 2024-03-28T17:51:00.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.182+0000 7f7a65a00640 1 -- 172.21.15.45:0/2523445467 learned_addr learned my addr 172.21.15.45:0/2523445467 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:00.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.182+0000 7f7a66400640 1 -- 172.21.15.45:0/2523445467 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a60101630 msgr2=0x7f7a600739a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:00.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.182+0000 7f7a66400640 1 --2- 172.21.15.45:0/2523445467 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a60101630 0x7f7a600739a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.182+0000 7f7a66400640 1 -- 172.21.15.45:0/2523445467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a60101ff0 msgr2=0x7f7a60073ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:00.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.182+0000 7f7a66400640 1 --2- 172.21.15.45:0/2523445467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a60101ff0 0x7f7a60073ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.182+0000 7f7a66400640 1 -- 172.21.15.45:0/2523445467 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a601a6410 con 0x7f7a60107640 2024-03-28T17:51:00.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.182+0000 7f7a65a00640 1 --2- 172.21.15.45:0/2523445467 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a60101630 0x7f7a600739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:51:00.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.183+0000 7f7a66400640 1 --2- 172.21.15.45:0/2523445467 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a60107640 0x7f7a60074420 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7a5000e970 tx=0x7f7a5000ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:00.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.183+0000 7f7a5e000640 1 -- 172.21.15.45:0/2523445467 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5000ccc0 con 0x7f7a60107640 2024-03-28T17:51:00.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.183+0000 7f7a68400640 1 -- 172.21.15.45:0/2523445467 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a601a6700 con 0x7f7a60107640 2024-03-28T17:51:00.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.183+0000 7f7a5e000640 1 -- 172.21.15.45:0/2523445467 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a50005020 con 0x7f7a60107640 2024-03-28T17:51:00.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.183+0000 7f7a5e000640 1 -- 172.21.15.45:0/2523445467 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a50004a00 con 0x7f7a60107640 2024-03-28T17:51:00.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.183+0000 7f7a68400640 1 -- 172.21.15.45:0/2523445467 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a601a6c00 con 0x7f7a60107640 2024-03-28T17:51:00.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.186+0000 7f7a5e000640 1 -- 172.21.15.45:0/2523445467 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f7a500051d0 con 0x7f7a60107640 2024-03-28T17:51:00.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.187+0000 7f7a6b1c2740 1 Processor -- start 2024-03-28T17:51:00.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.187+0000 7f7a6b1c2740 1 -- start start 2024-03-28T17:51:00.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.187+0000 7f7a6b1c2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f91b5286d0 0x55f91b528ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.187+0000 7f7a6b1c2740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b529080 0x55f91b531630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.187+0000 7f7a6b1c2740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f91b531c00 0x55f91b534050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.187+0000 7f7a6b1c2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f91b48c2c0 con 0x55f91b5286d0 2024-03-28T17:51:00.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.187+0000 7f7a6b1c2740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f91b434c90 con 0x55f91b529080 2024-03-28T17:51:00.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.187+0000 7f7a6b1c2740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f91b488360 con 0x55f91b531c00 2024-03-28T17:51:00.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.188+0000 7f7a65a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b529080 0x55f91b531630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.188+0000 7f7a65a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b529080 0x55f91b531630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39842/0 (socket says 172.21.15.45:39842) 2024-03-28T17:51:00.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.188+0000 7f7a65a00640 1 -- 172.21.15.45:0/1772264377 learned_addr learned my addr 172.21.15.45:0/1772264377 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:00.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.188+0000 7f7a65000640 1 --2- 172.21.15.45:0/1772264377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f91b5286d0 0x55f91b528ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.188+0000 7f7a65a00640 1 -- 172.21.15.45:0/1772264377 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f91b531c00 msgr2=0x55f91b534050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:00.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.188+0000 7f7a65a00640 1 --2- 172.21.15.45:0/1772264377 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f91b531c00 0x55f91b534050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.188+0000 7f7a65a00640 1 -- 172.21.15.45:0/1772264377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f91b5286d0 msgr2=0x55f91b528ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:00.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.188+0000 7f7a65a00640 1 --2- 172.21.15.45:0/1772264377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f91b5286d0 0x55f91b528ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.188+0000 7f7a65a00640 1 -- 172.21.15.45:0/1772264377 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f91b459080 con 0x55f91b529080 2024-03-28T17:51:00.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.188+0000 7f7a65000640 1 --2- 172.21.15.45:0/1772264377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f91b5286d0 0x55f91b528ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:51:00.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.188+0000 7f7a5e000640 1 --2- 172.21.15.45:0/2523445467 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f7a440a2970 0x7f7a440a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.188+0000 7f7a65a00640 1 --2- 172.21.15.45:0/1772264377 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b529080 0x55f91b531630 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7a58035fb0 tx=0x7f7a580056d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67243fa45d0d0372 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:00.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.189+0000 7f7a5e000640 1 -- 172.21.15.45:0/2523445467 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8684+0+0 (secure 0 0 0) 0x7f7a500ceeb0 con 0x7f7a60107640 2024-03-28T17:51:00.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.189+0000 7f7a56a00640 1 -- 172.21.15.45:0/1772264377 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a58044020 con 0x55f91b529080 2024-03-28T17:51:00.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.189+0000 7f7a56a00640 1 -- 172.21.15.45:0/1772264377 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a580323c0 con 0x55f91b529080 2024-03-28T17:51:00.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.189+0000 7f7a56a00640 1 -- 172.21.15.45:0/1772264377 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a58039040 con 0x55f91b529080 2024-03-28T17:51:00.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.189+0000 7f7a65000640 1 --2- 172.21.15.45:0/2523445467 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f7a440a2970 0x7f7a440a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.189+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/1772264377 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b529080 msgr2=0x55f91b531630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:00.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.189+0000 7f7a6b1c2740 1 --2- 172.21.15.45:0/1772264377 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b529080 0x55f91b531630 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7a58035fb0 tx=0x7f7a580056d0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.189+0000 7f7a65000640 1 --2- 172.21.15.45:0/2523445467 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f7a440a2970 0x7f7a440a4e30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7a60075430 tx=0x7f7a4c0095c0 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:00.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.189+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/1772264377 shutdown_connections 2024-03-28T17:51:00.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.189+0000 7f7a6b1c2740 1 --2- 172.21.15.45:0/1772264377 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f91b531c00 0x55f91b534050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.189+0000 7f7a6b1c2740 1 --2- 172.21.15.45:0/1772264377 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b529080 0x55f91b531630 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.189+0000 7f7a6b1c2740 1 --2- 172.21.15.45:0/1772264377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f91b5286d0 0x55f91b528ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.189+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/1772264377 >> 172.21.15.45:0/1772264377 conn(0x55f91b522da0 msgr2=0x55f91b523180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:00.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.190+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/1772264377 shutdown_connections 2024-03-28T17:51:00.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.190+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/1772264377 wait complete. 2024-03-28T17:51:00.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.190+0000 7f7a6b1c2740 1 -- start start 2024-03-28T17:51:00.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.190+0000 7f7a6b1c2740 1 Processor -- start 2024-03-28T17:51:00.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.191+0000 7f7a6b1c2740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b5286d0 0x55f91b5314f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.191+0000 7f7a6b1c2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f91b529080 0x55f91b52e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.191+0000 7f7a6b1c2740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f91b531c00 0x55f91b52ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.191+0000 7f7a6b1c2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f91b534b10 con 0x55f91b529080 2024-03-28T17:51:00.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.191+0000 7f7a6b1c2740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f91b4480d0 con 0x55f91b5286d0 2024-03-28T17:51:00.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.191+0000 7f7a6b1c2740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f91b534c90 con 0x55f91b531c00 2024-03-28T17:51:00.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.191+0000 7f7a6b1c2740 10 client.0 _ll_register_callbacks cb 0x55f91b44ce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:51:00.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.191+0000 7f7a65a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b5286d0 0x55f91b5314f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.191+0000 7f7a65a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b5286d0 0x55f91b5314f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39844/0 (socket says 172.21.15.45:39844) 2024-03-28T17:51:00.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.191+0000 7f7a65a00640 1 -- 172.21.15.45:0/1760589185 learned_addr learned my addr 172.21.15.45:0/1760589185 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:00.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.191+0000 7f7a65000640 1 --2- 172.21.15.45:0/1760589185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f91b529080 0x55f91b52e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.191+0000 7f7a66400640 1 --2- 172.21.15.45:0/1760589185 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f91b531c00 0x55f91b52ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.191+0000 7f7a65a00640 1 -- 172.21.15.45:0/1760589185 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f91b531c00 msgr2=0x55f91b52ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:00.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.191+0000 7f7a65a00640 1 --2- 172.21.15.45:0/1760589185 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f91b531c00 0x55f91b52ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.191+0000 7f7a65a00640 1 -- 172.21.15.45:0/1760589185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f91b529080 msgr2=0x55f91b52e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:00.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.191+0000 7f7a65a00640 1 --2- 172.21.15.45:0/1760589185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f91b529080 0x55f91b52e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.191+0000 7f7a65a00640 1 -- 172.21.15.45:0/1760589185 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f91b1a03f0 con 0x55f91b5286d0 2024-03-28T17:51:00.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.191+0000 7f7a66400640 1 --2- 172.21.15.45:0/1760589185 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f91b531c00 0x55f91b52ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:51:00.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.192+0000 7f7a65a00640 1 --2- 172.21.15.45:0/1760589185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b5286d0 0x55f91b5314f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7a580057c0 tx=0x7f7a58005890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1296b5e07ea6604 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:00.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.192+0000 7f7a4b400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:51:00.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.192+0000 7f7a4b400640 1 -- 172.21.15.45:0/1760589185 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a58050070 con 0x55f91b5286d0 2024-03-28T17:51:00.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.192+0000 7f7a4b400640 1 -- 172.21.15.45:0/1760589185 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8684+0+0 (secure 0 0 0) 0x7f7a58004450 con 0x55f91b5286d0 2024-03-28T17:51:00.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.192+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/1760589185 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7a580323c0 con 0x55f91b5286d0 2024-03-28T17:51:00.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.192+0000 7f7a4b400640 1 -- 172.21.15.45:0/1760589185 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a5805c050 con 0x55f91b5286d0 2024-03-28T17:51:00.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.192+0000 7f7a6b1c2740 10 client.25871 fetch_fsmap learned FSMap version 33 2024-03-28T17:51:00.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.192+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/1760589185 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f91b26a7d0 con 0x55f91b5286d0 2024-03-28T17:51:00.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.193+0000 7f7a4b400640 1 -- 172.21.15.45:0/1760589185 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a580323c0 con 0x55f91b5286d0 2024-03-28T17:51:00.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.193+0000 7f7a4b400640 1 -- 172.21.15.45:0/1760589185 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 33) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f7a58032a90 con 0x55f91b5286d0 2024-03-28T17:51:00.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.193+0000 7f7a6b1c2740 10 client.25871 fetch_fsmap finished waiting for FSMap version 33 2024-03-28T17:51:00.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.193+0000 7f7a6b1c2740 10 client.25871 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T17:51:00.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.193+0000 7f7a6b1c2740 20 client.25871 populate_metadata read hostname 'smithi045' 2024-03-28T17:51:00.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.193+0000 7f7a6b1c2740 1 --2- 172.21.15.45:0/1760589185 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x55f91b541910 0x55f91b543d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.193+0000 7f7a6b1c2740 4 client.25871 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:51:00.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.193+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/1760589185 --> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f91b499800 con 0x55f91b541910 2024-03-28T17:51:00.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.193+0000 7f7a65000640 1 --2- 172.21.15.45:0/1760589185 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x55f91b541910 0x55f91b543d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.194+0000 7f7a65000640 1 --2- 172.21.15.45:0/1760589185 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x55f91b541910 0x55f91b543d00 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ba6f3efb5165b3a0 server_cookie=60a37308931e4554 in_seq=0 out_seq=0 2024-03-28T17:51:00.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.194+0000 7f7a4b400640 10 client.25871 ms_handle_connect on v2:172.21.15.71:6836/2008752655 2024-03-28T17:51:00.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.194+0000 7f7a4b400640 1 -- 172.21.15.45:0/1760589185 <== mds.1 v2:172.21.15.71:6836/2008752655 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f91b499800 con 0x55f91b541910 2024-03-28T17:51:00.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.194+0000 7f7a4b400640 10 client.25871 handle_command_reply: tid=0 2024-03-28T17:51:00.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.195+0000 7f7a6b1c2740 1 client.25871 shutdown 2024-03-28T17:51:00.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.195+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/1760589185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b5286d0 msgr2=0x55f91b5314f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:00.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.195+0000 7f7a6b1c2740 1 --2- 172.21.15.45:0/1760589185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b5286d0 0x55f91b5314f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7a580057c0 tx=0x7f7a58005890 comp rx=0 tx=0).stop 2024-03-28T17:51:00.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.195+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/1760589185 shutdown_connections 2024-03-28T17:51:00.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.195+0000 7f7a6b1c2740 1 --2- 172.21.15.45:0/1760589185 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x55f91b541910 0x55f91b543d00 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.195+0000 7f7a6b1c2740 1 --2- 172.21.15.45:0/1760589185 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f91b531c00 0x55f91b52ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.195+0000 7f7a4b400640 0 client.25871 ms_handle_reset on v2:172.21.15.71:6836/2008752655 2024-03-28T17:51:00.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.195+0000 7f7a6b1c2740 1 --2- 172.21.15.45:0/1760589185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f91b529080 0x55f91b52e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.195+0000 7f7a6b1c2740 1 --2- 172.21.15.45:0/1760589185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b5286d0 0x55f91b5314f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.195+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/1760589185 >> 172.21.15.45:0/1760589185 conn(0x55f91b522da0 msgr2=0x55f91b524af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:00.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.195+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/1760589185 shutdown_connections 2024-03-28T17:51:00.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.195+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/1760589185 wait complete. 2024-03-28T17:51:00.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.195+0000 7f7a6b1c2740 20 client.25871 trim_cache size 0 max 16384 2024-03-28T17:51:00.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.209+0000 7f7a6b1c2740 1 Processor -- start 2024-03-28T17:51:00.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.209+0000 7f7a6b1c2740 1 -- start start 2024-03-28T17:51:00.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.209+0000 7f7a6b1c2740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f91b5286d0 0x55f91b52e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.209+0000 7f7a6b1c2740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b529080 0x55f91b531020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.209+0000 7f7a6b1c2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f91b531c00 0x55f91b52bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.209+0000 7f7a6b1c2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a601afc20 con 0x55f91b531c00 2024-03-28T17:51:00.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.209+0000 7f7a6b1c2740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a601ad660 con 0x55f91b529080 2024-03-28T17:51:00.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.209+0000 7f7a6b1c2740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a601af5c0 con 0x55f91b5286d0 2024-03-28T17:51:00.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.210+0000 7f7a65000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b529080 0x55f91b531020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.210+0000 7f7a65000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b529080 0x55f91b531020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39846/0 (socket says 172.21.15.45:39846) 2024-03-28T17:51:00.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.210+0000 7f7a65000640 1 -- 172.21.15.45:0/1828333279 learned_addr learned my addr 172.21.15.45:0/1828333279 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:00.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.210+0000 7f7a66400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f91b531c00 0x55f91b52bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.210+0000 7f7a65a00640 1 --2- 172.21.15.45:0/1828333279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f91b5286d0 0x55f91b52e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.210+0000 7f7a65000640 1 -- 172.21.15.45:0/1828333279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f91b5286d0 msgr2=0x55f91b52e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:00.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.210+0000 7f7a65000640 1 --2- 172.21.15.45:0/1828333279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f91b5286d0 0x55f91b52e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.210+0000 7f7a65000640 1 -- 172.21.15.45:0/1828333279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f91b531c00 msgr2=0x55f91b52bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:00.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.210+0000 7f7a65000640 1 --2- 172.21.15.45:0/1828333279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f91b531c00 0x55f91b52bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.210+0000 7f7a65000640 1 -- 172.21.15.45:0/1828333279 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f91b52f250 con 0x55f91b529080 2024-03-28T17:51:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.210+0000 7f7a65000640 1 --2- 172.21.15.45:0/1828333279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b529080 0x55f91b531020 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x55f91b534da0 tx=0x7f7a4c019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=375e1c4229aaeed1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.210+0000 7f7a56000640 1 -- 172.21.15.45:0/1828333279 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c02f070 con 0x55f91b529080 2024-03-28T17:51:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.210+0000 7f7a56000640 1 -- 172.21.15.45:0/1828333279 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a4c0292f0 con 0x55f91b529080 2024-03-28T17:51:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.210+0000 7f7a56000640 1 -- 172.21.15.45:0/1828333279 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c00da90 con 0x55f91b529080 2024-03-28T17:51:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.211+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/1828333279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b529080 msgr2=0x55f91b531020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.211+0000 7f7a6b1c2740 1 --2- 172.21.15.45:0/1828333279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b529080 0x55f91b531020 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x55f91b534da0 tx=0x7f7a4c019000 comp rx=0 tx=0).stop 2024-03-28T17:51:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.211+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/1828333279 shutdown_connections 2024-03-28T17:51:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.211+0000 7f7a6b1c2740 1 --2- 172.21.15.45:0/1828333279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f91b531c00 0x55f91b52bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.211+0000 7f7a6b1c2740 1 --2- 172.21.15.45:0/1828333279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b529080 0x55f91b531020 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.211+0000 7f7a6b1c2740 1 --2- 172.21.15.45:0/1828333279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f91b5286d0 0x55f91b52e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.211+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/1828333279 >> 172.21.15.45:0/1828333279 conn(0x55f91b522da0 msgr2=0x55f91b529d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:00.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.211+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/1828333279 shutdown_connections 2024-03-28T17:51:00.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.211+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/1828333279 wait complete. 2024-03-28T17:51:00.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.212+0000 7f7a6b1c2740 1 -- start start 2024-03-28T17:51:00.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.212+0000 7f7a6b1c2740 1 Processor -- start 2024-03-28T17:51:00.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.212+0000 7f7a6b1c2740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f91b5286d0 0x55f91b546850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.212+0000 7f7a6b1c2740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b529080 0x55f91b544bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.212+0000 7f7a6b1c2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f91b531c00 0x55f91b545130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.212+0000 7f7a6b1c2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a601aef60 con 0x55f91b531c00 2024-03-28T17:51:00.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.212+0000 7f7a6b1c2740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a601adc60 con 0x55f91b529080 2024-03-28T17:51:00.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.212+0000 7f7a6b1c2740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a601ae2a0 con 0x55f91b5286d0 2024-03-28T17:51:00.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.212+0000 7f7a6b1c2740 10 client.0 _ll_register_callbacks cb 0x7f7a4c009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:51:00.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.213+0000 7f7a65000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b529080 0x55f91b544bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.213+0000 7f7a65000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b529080 0x55f91b544bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39858/0 (socket says 172.21.15.45:39858) 2024-03-28T17:51:00.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.213+0000 7f7a65000640 1 -- 172.21.15.45:0/2344795767 learned_addr learned my addr 172.21.15.45:0/2344795767 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:00.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.213+0000 7f7a66400640 1 --2- 172.21.15.45:0/2344795767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f91b531c00 0x55f91b545130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.213+0000 7f7a65a00640 1 --2- 172.21.15.45:0/2344795767 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f91b5286d0 0x55f91b546850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.213+0000 7f7a65000640 1 -- 172.21.15.45:0/2344795767 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f91b5286d0 msgr2=0x55f91b546850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:00.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.213+0000 7f7a65000640 1 --2- 172.21.15.45:0/2344795767 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f91b5286d0 0x55f91b546850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.213+0000 7f7a65000640 1 -- 172.21.15.45:0/2344795767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f91b531c00 msgr2=0x55f91b545130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:00.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.213+0000 7f7a65000640 1 --2- 172.21.15.45:0/2344795767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f91b531c00 0x55f91b545130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.213+0000 7f7a65000640 1 -- 172.21.15.45:0/2344795767 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f91b545670 con 0x55f91b529080 2024-03-28T17:51:00.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.213+0000 7f7a65000640 1 --2- 172.21.15.45:0/2344795767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b529080 0x55f91b544bf0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c003d20 tx=0x7f7a4c00da40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d4c1cf32ad2c1a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:00.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.213+0000 7f7a4b400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:51:00.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.213+0000 7f7a4b400640 1 -- 172.21.15.45:0/2344795767 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c02f040 con 0x55f91b529080 2024-03-28T17:51:00.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.214+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/2344795767 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f91b493b50 con 0x55f91b529080 2024-03-28T17:51:00.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.214+0000 7f7a4b400640 1 -- 172.21.15.45:0/2344795767 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8684+0+0 (secure 0 0 0) 0x7f7a4c00dc30 con 0x55f91b529080 2024-03-28T17:51:00.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.214+0000 7f7a4b400640 1 -- 172.21.15.45:0/2344795767 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a4c00de50 con 0x55f91b529080 2024-03-28T17:51:00.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.214+0000 7f7a6b1c2740 10 client.25877 fetch_fsmap learned FSMap version 33 2024-03-28T17:51:00.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.214+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/2344795767 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f91b59dbc0 con 0x55f91b529080 2024-03-28T17:51:00.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.214+0000 7f7a4b400640 1 -- 172.21.15.45:0/2344795767 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f91b493b50 con 0x55f91b529080 2024-03-28T17:51:00.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.214+0000 7f7a4b400640 1 -- 172.21.15.45:0/2344795767 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 33) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f7a4c032070 con 0x55f91b529080 2024-03-28T17:51:00.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.214+0000 7f7a6b1c2740 10 client.25877 fetch_fsmap finished waiting for FSMap version 33 2024-03-28T17:51:00.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.214+0000 7f7a6b1c2740 10 client.25877 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T17:51:00.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.214+0000 7f7a6b1c2740 20 client.25877 populate_metadata read hostname 'smithi045' 2024-03-28T17:51:00.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.214+0000 7f7a6b1c2740 1 --2- 172.21.15.45:0/2344795767 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x55f91b59e230 0x55f91b5a0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.215+0000 7f7a6b1c2740 4 client.25877 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T17:51:00.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.215+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/2344795767 --> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f91b411500 con 0x55f91b59e230 2024-03-28T17:51:00.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.215+0000 7f7a65a00640 1 --2- 172.21.15.45:0/2344795767 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x55f91b59e230 0x55f91b5a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.215+0000 7f7a65a00640 1 --2- 172.21.15.45:0/2344795767 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x55f91b59e230 0x55f91b5a0650 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d30afc6820599707 server_cookie=e1ba2ac283b59d66 in_seq=0 out_seq=0 2024-03-28T17:51:00.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.215+0000 7f7a4b400640 10 client.25877 ms_handle_connect on v2:172.21.15.71:6836/2008752655 2024-03-28T17:51:00.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.215+0000 7f7a4b400640 1 -- 172.21.15.45:0/2344795767 <== mds.1 v2:172.21.15.71:6836/2008752655 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f91b411500 con 0x55f91b59e230 2024-03-28T17:51:00.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.215+0000 7f7a4b400640 10 client.25877 handle_command_reply: tid=0 2024-03-28T17:51:00.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.216+0000 7f7a6b1c2740 1 client.25877 shutdown 2024-03-28T17:51:00.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.216+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/2344795767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b529080 msgr2=0x55f91b544bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:00.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.216+0000 7f7a6b1c2740 1 --2- 172.21.15.45:0/2344795767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b529080 0x55f91b544bf0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c003d20 tx=0x7f7a4c00da40 comp rx=0 tx=0).stop 2024-03-28T17:51:00.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.216+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/2344795767 shutdown_connections 2024-03-28T17:51:00.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.216+0000 7f7a6b1c2740 1 --2- 172.21.15.45:0/2344795767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f91b531c00 0x55f91b545130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.216+0000 7f7a6b1c2740 1 --2- 172.21.15.45:0/2344795767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f91b529080 0x55f91b544bf0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.216+0000 7f7a6b1c2740 1 --2- 172.21.15.45:0/2344795767 >> [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] conn(0x55f91b59e230 0x55f91b5a0650 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.216+0000 7f7a6b1c2740 1 --2- 172.21.15.45:0/2344795767 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f91b5286d0 0x55f91b546850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.216+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/2344795767 >> 172.21.15.45:0/2344795767 conn(0x55f91b522da0 msgr2=0x55f91b523da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:00.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.216+0000 7f7a4b400640 0 client.25877 ms_handle_reset on v2:172.21.15.71:6836/2008752655 2024-03-28T17:51:00.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.216+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/2344795767 shutdown_connections 2024-03-28T17:51:00.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.216+0000 7f7a6b1c2740 1 -- 172.21.15.45:0/2344795767 wait complete. 2024-03-28T17:51:00.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.216+0000 7f7a6b1c2740 20 client.25877 trim_cache size 0 max 16384 2024-03-28T17:51:00.221 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-28T17:51:00.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.217+0000 7f7a68400640 1 -- 172.21.15.45:0/2523445467 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f7a440a2970 msgr2=0x7f7a440a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:00.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.217+0000 7f7a68400640 1 --2- 172.21.15.45:0/2523445467 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f7a440a2970 0x7f7a440a4e30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7a60075430 tx=0x7f7a4c0095c0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.217+0000 7f7a68400640 1 -- 172.21.15.45:0/2523445467 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a60107640 msgr2=0x7f7a60074420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:00.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.217+0000 7f7a68400640 1 --2- 172.21.15.45:0/2523445467 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a60107640 0x7f7a60074420 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7a5000e970 tx=0x7f7a5000ee30 comp rx=0 tx=0).stop 2024-03-28T17:51:00.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.217+0000 7f7a68400640 1 -- 172.21.15.45:0/2523445467 shutdown_connections 2024-03-28T17:51:00.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.217+0000 7f7a68400640 1 --2- 172.21.15.45:0/2523445467 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f7a440a2970 0x7f7a440a4e30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.217+0000 7f7a68400640 1 --2- 172.21.15.45:0/2523445467 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a60107640 0x7f7a60074420 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.217+0000 7f7a68400640 1 --2- 172.21.15.45:0/2523445467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a60101ff0 0x7f7a60073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.217+0000 7f7a68400640 1 --2- 172.21.15.45:0/2523445467 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a60101630 0x7f7a600739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.217+0000 7f7a68400640 1 -- 172.21.15.45:0/2523445467 >> 172.21.15.45:0/2523445467 conn(0x7f7a60078aa0 msgr2=0x7f7a60103f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:00.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.217+0000 7f7a68400640 1 -- 172.21.15.45:0/2523445467 shutdown_connections 2024-03-28T17:51:00.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.217+0000 7f7a68400640 1 -- 172.21.15.45:0/2523445467 wait complete. 2024-03-28T17:51:00.232 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:00 smithi045 ceph-mon[31773]: pgmap v64: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-28T17:51:00.233 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:00 smithi045 ceph-mon[31773]: from='client.? 172.21.15.45:0/3829083011' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:51:00.233 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:00 smithi045 ceph-mon[31773]: daemon mds.f restarted 2024-03-28T17:51:00.233 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:00 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:51:00.233 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:00 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T17:51:00.233 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:00 smithi045 ceph-mon[31773]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T17:51:00.233 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:00 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/1934350429,v1:172.21.15.151:6835/1934350429] up:boot 2024-03-28T17:51:00.233 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:00 smithi045 ceph-mon[31773]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-03-28T17:51:00.233 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:00 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:51:00.233 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:00 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:00.235 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T17:51:00.235 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T17:51:00.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:00 smithi151 ceph-mon[36481]: pgmap v64: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-28T17:51:00.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:00 smithi151 ceph-mon[36481]: from='client.? 172.21.15.45:0/3829083011' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:51:00.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:00 smithi151 ceph-mon[36481]: daemon mds.f restarted 2024-03-28T17:51:00.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:00 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:51:00.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:00 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T17:51:00.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:00 smithi151 ceph-mon[36481]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T17:51:00.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:00 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/1934350429,v1:172.21.15.151:6835/1934350429] up:boot 2024-03-28T17:51:00.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:00 smithi151 ceph-mon[36481]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-03-28T17:51:00.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:00 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:51:00.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:00 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:00.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:59 smithi151 podman[71068]: 2024-03-28 17:50:59.847319678 +0000 UTC m=+0.405854949 container init 0a51be78584ea4109b0507707af5ec218cfc1f46173b487a100a46a706e7992d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-28T17:51:00.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:59 smithi151 podman[71068]: 2024-03-28 17:50:59.849935286 +0000 UTC m=+0.408470546 container start 0a51be78584ea4109b0507707af5ec218cfc1f46173b487a100a46a706e7992d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T17:51:00.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:59 smithi151 ceph-mds[71086]: -- 172.21.15.151:0/693177233 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e9b6253a40 con 0x55e9b6227000 2024-03-28T17:51:00.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:59 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: starting mds.f at 2024-03-28T17:51:00.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:59 smithi151 bash[71068]: 0a51be78584ea4109b0507707af5ec218cfc1f46173b487a100a46a706e7992d 2024-03-28T17:51:00.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:50:59 smithi151 systemd[1]: Started Ceph mds.f for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:51:00.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.449+0000 7f37e4a00640 1 -- 172.21.15.45:0/1862168945 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37e0105b00 msgr2=0x7f37e0105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:00.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.449+0000 7f37e4a00640 1 --2- 172.21.15.45:0/1862168945 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37e0105b00 0x7f37e0105fb0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f37d8009cd0 tx=0x7f37d802c180 comp rx=0 tx=0).stop 2024-03-28T17:51:00.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.449+0000 7f37e4a00640 1 -- 172.21.15.45:0/1862168945 shutdown_connections 2024-03-28T17:51:00.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.449+0000 7f37e4a00640 1 --2- 172.21.15.45:0/1862168945 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37e0105b00 0x7f37e0105fb0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.449+0000 7f37e4a00640 1 --2- 172.21.15.45:0/1862168945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37e0101020 0x7f37e01054e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.449+0000 7f37e4a00640 1 --2- 172.21.15.45:0/1862168945 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37e0100700 0x7f37e0100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.449+0000 7f37e4a00640 1 -- 172.21.15.45:0/1862168945 >> 172.21.15.45:0/1862168945 conn(0x7f37e0077700 msgr2=0x7f37e0077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:00.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.449+0000 7f37e4a00640 1 -- 172.21.15.45:0/1862168945 shutdown_connections 2024-03-28T17:51:00.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.449+0000 7f37e4a00640 1 -- 172.21.15.45:0/1862168945 wait complete. 2024-03-28T17:51:00.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.450+0000 7f37e4a00640 1 Processor -- start 2024-03-28T17:51:00.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.450+0000 7f37e4a00640 1 -- start start 2024-03-28T17:51:00.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.451+0000 7f37e4a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37e0100700 0x7f37e019e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.451+0000 7f37e4a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37e0101020 0x7f37e019ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.451+0000 7f37e4a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37e0105b00 0x7f37e01a2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.451+0000 7f37e4a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e0118080 con 0x7f37e0105b00 2024-03-28T17:51:00.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.451+0000 7f37e4a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e0117f00 con 0x7f37e0101020 2024-03-28T17:51:00.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.451+0000 7f37e4a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e0118200 con 0x7f37e0100700 2024-03-28T17:51:00.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.451+0000 7f37dd400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37e0101020 0x7f37e019ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.451+0000 7f37dd400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37e0101020 0x7f37e019ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39876/0 (socket says 172.21.15.45:39876) 2024-03-28T17:51:00.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.451+0000 7f37dd400640 1 -- 172.21.15.45:0/2436702842 learned_addr learned my addr 172.21.15.45:0/2436702842 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:00.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.451+0000 7f37dde00640 1 --2- 172.21.15.45:0/2436702842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37e0100700 0x7f37e019e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.451+0000 7f37de800640 1 --2- 172.21.15.45:0/2436702842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37e0105b00 0x7f37e01a2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.451+0000 7f37dd400640 1 -- 172.21.15.45:0/2436702842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37e0100700 msgr2=0x7f37e019e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:00.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.451+0000 7f37dd400640 1 --2- 172.21.15.45:0/2436702842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37e0100700 0x7f37e019e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.451+0000 7f37dd400640 1 -- 172.21.15.45:0/2436702842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37e0105b00 msgr2=0x7f37e01a2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:00.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.451+0000 7f37dd400640 1 --2- 172.21.15.45:0/2436702842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37e0105b00 0x7f37e01a2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.451+0000 7f37dd400640 1 -- 172.21.15.45:0/2436702842 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e01a3530 con 0x7f37e0101020 2024-03-28T17:51:00.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.451+0000 7f37de800640 1 --2- 172.21.15.45:0/2436702842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37e0105b00 0x7f37e01a2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:51:00.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.452+0000 7f37dd400640 1 --2- 172.21.15.45:0/2436702842 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37e0101020 0x7f37e019ea80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f37cc00e9f0 tx=0x7f37cc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:00.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.452+0000 7f37d6a00640 1 -- 172.21.15.45:0/2436702842 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc00cd80 con 0x7f37e0101020 2024-03-28T17:51:00.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.452+0000 7f37d6a00640 1 -- 172.21.15.45:0/2436702842 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f37cc010910 con 0x7f37e0101020 2024-03-28T17:51:00.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.452+0000 7f37d6a00640 1 -- 172.21.15.45:0/2436702842 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc018990 con 0x7f37e0101020 2024-03-28T17:51:00.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.452+0000 7f37e4a00640 1 -- 172.21.15.45:0/2436702842 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37e01a3820 con 0x7f37e0101020 2024-03-28T17:51:00.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.452+0000 7f37e4a00640 1 -- 172.21.15.45:0/2436702842 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37e01ab0b0 con 0x7f37e0101020 2024-03-28T17:51:00.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.455+0000 7f37e77c8740 1 Processor -- start 2024-03-28T17:51:00.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.455+0000 7f37e77c8740 1 -- start start 2024-03-28T17:51:00.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.456+0000 7f37e77c8740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0d36d0 0x5572dc0d3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.456+0000 7f37d6a00640 1 -- 172.21.15.45:0/2436702842 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f37cc010ac0 con 0x7f37e0101020 2024-03-28T17:51:00.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.456+0000 7f37e77c8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572dc0d4080 0x5572dc0dc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.456+0000 7f37e77c8740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5572dc0dcc00 0x5572dc0df050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.456+0000 7f37e77c8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572dc0372c0 con 0x5572dc0d4080 2024-03-28T17:51:00.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.456+0000 7f37e77c8740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572dbfdfc90 con 0x5572dc0d36d0 2024-03-28T17:51:00.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.456+0000 7f37e77c8740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572dc033360 con 0x5572dc0dcc00 2024-03-28T17:51:00.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.456+0000 7f37de800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0d36d0 0x5572dc0d3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.456+0000 7f37de800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0d36d0 0x5572dc0d3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39886/0 (socket says 172.21.15.45:39886) 2024-03-28T17:51:00.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.456+0000 7f37de800640 1 -- 172.21.15.45:0/876682184 learned_addr learned my addr 172.21.15.45:0/876682184 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:00.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.456+0000 7f37dde00640 1 --2- 172.21.15.45:0/876682184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572dc0d4080 0x5572dc0dc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.456+0000 7f37dd400640 1 --2- 172.21.15.45:0/876682184 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5572dc0dcc00 0x5572dc0df050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.456+0000 7f37de800640 1 -- 172.21.15.45:0/876682184 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5572dc0dcc00 msgr2=0x5572dc0df050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:00.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.456+0000 7f37de800640 1 --2- 172.21.15.45:0/876682184 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5572dc0dcc00 0x5572dc0df050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.456+0000 7f37de800640 1 -- 172.21.15.45:0/876682184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572dc0d4080 msgr2=0x5572dc0dc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:00.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.456+0000 7f37de800640 1 --2- 172.21.15.45:0/876682184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572dc0d4080 0x5572dc0dc630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.457+0000 7f37de800640 1 -- 172.21.15.45:0/876682184 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572dc004080 con 0x5572dc0d36d0 2024-03-28T17:51:00.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.457+0000 7f37de800640 1 --2- 172.21.15.45:0/876682184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0d36d0 0x5572dc0d3ab0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f37d8009ca0 tx=0x7f37d8009880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfc77c2a043675b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:00.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.457+0000 7f37cb000640 1 -- 172.21.15.45:0/876682184 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d8044020 con 0x5572dc0d36d0 2024-03-28T17:51:00.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.457+0000 7f37cb000640 1 -- 172.21.15.45:0/876682184 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f37d8003e50 con 0x5572dc0d36d0 2024-03-28T17:51:00.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.457+0000 7f37cb000640 1 -- 172.21.15.45:0/876682184 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d8039040 con 0x5572dc0d36d0 2024-03-28T17:51:00.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.458+0000 7f37d6a00640 1 --2- 172.21.15.45:0/2436702842 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f37c40a2970 0x7f37c40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.458+0000 7f37e77c8740 1 -- 172.21.15.45:0/876682184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0d36d0 msgr2=0x5572dc0d3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:00.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.458+0000 7f37e77c8740 1 --2- 172.21.15.45:0/876682184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0d36d0 0x5572dc0d3ab0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f37d8009ca0 tx=0x7f37d8009880 comp rx=0 tx=0).stop 2024-03-28T17:51:00.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.458+0000 7f37d6a00640 1 -- 172.21.15.45:0/2436702842 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8684+0+0 (secure 0 0 0) 0x7f37cc0cf1b0 con 0x7f37e0101020 2024-03-28T17:51:00.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.458+0000 7f37dde00640 1 --2- 172.21.15.45:0/2436702842 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f37c40a2970 0x7f37c40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.458+0000 7f37e77c8740 1 -- 172.21.15.45:0/876682184 shutdown_connections 2024-03-28T17:51:00.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.458+0000 7f37e77c8740 1 --2- 172.21.15.45:0/876682184 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5572dc0dcc00 0x5572dc0df050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.458+0000 7f37e77c8740 1 --2- 172.21.15.45:0/876682184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572dc0d4080 0x5572dc0dc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.458+0000 7f37e77c8740 1 --2- 172.21.15.45:0/876682184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0d36d0 0x5572dc0d3ab0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.458+0000 7f37e77c8740 1 -- 172.21.15.45:0/876682184 >> 172.21.15.45:0/876682184 conn(0x5572dc0cdda0 msgr2=0x5572dc0ce180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:00.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.458+0000 7f37e77c8740 1 -- 172.21.15.45:0/876682184 shutdown_connections 2024-03-28T17:51:00.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.458+0000 7f37e77c8740 1 -- 172.21.15.45:0/876682184 wait complete. 2024-03-28T17:51:00.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.459+0000 7f37e77c8740 1 -- start start 2024-03-28T17:51:00.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.459+0000 7f37dde00640 1 --2- 172.21.15.45:0/2436702842 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f37c40a2970 0x7f37c40a4e30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x5572dc0cee10 tx=0x7f37d000b390 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:00.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.459+0000 7f37e77c8740 1 Processor -- start 2024-03-28T17:51:00.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.459+0000 7f37e77c8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572dc0d36d0 0x5572dc0dc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.459+0000 7f37e77c8740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5572dc0d4080 0x5572dc0d9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.459+0000 7f37e77c8740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0dcc00 0x5572dc0d9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.459+0000 7f37e77c8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572dc0dfb10 con 0x5572dc0d36d0 2024-03-28T17:51:00.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.459+0000 7f37e77c8740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572dbff30d0 con 0x5572dc0dcc00 2024-03-28T17:51:00.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.459+0000 7f37e77c8740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572dc0dfc90 con 0x5572dc0d4080 2024-03-28T17:51:00.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.459+0000 7f37e77c8740 10 client.0 _ll_register_callbacks cb 0x5572dbff7e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:51:00.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.460+0000 7f37de800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0dcc00 0x5572dc0d9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.460+0000 7f37dd400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5572dc0d4080 0x5572dc0d9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.460+0000 7f37de800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0dcc00 0x5572dc0d9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39890/0 (socket says 172.21.15.45:39890) 2024-03-28T17:51:00.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.460+0000 7f37de800640 1 -- 172.21.15.45:0/3420414802 learned_addr learned my addr 172.21.15.45:0/3420414802 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:00.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.460+0000 7f37dde00640 1 --2- 172.21.15.45:0/3420414802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572dc0d36d0 0x5572dc0dc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.460+0000 7f37de800640 1 -- 172.21.15.45:0/3420414802 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5572dc0d4080 msgr2=0x5572dc0d9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:00.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.460+0000 7f37de800640 1 --2- 172.21.15.45:0/3420414802 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5572dc0d4080 0x5572dc0d9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.460+0000 7f37de800640 1 -- 172.21.15.45:0/3420414802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572dc0d36d0 msgr2=0x5572dc0dc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:00.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.460+0000 7f37de800640 1 --2- 172.21.15.45:0/3420414802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572dc0d36d0 0x5572dc0dc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.460+0000 7f37de800640 1 -- 172.21.15.45:0/3420414802 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5572dbd4b3f0 con 0x5572dc0dcc00 2024-03-28T17:51:00.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.460+0000 7f37de800640 1 --2- 172.21.15.45:0/3420414802 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0dcc00 0x5572dc0d9e10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f37d8004af0 tx=0x7f37d8005c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cad85a2958b6930 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:00.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.460+0000 7f37bf400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:51:00.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.461+0000 7f37bf400640 1 -- 172.21.15.45:0/3420414802 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d804f070 con 0x5572dc0dcc00 2024-03-28T17:51:00.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.461+0000 7f37e77c8740 1 -- 172.21.15.45:0/3420414802 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f37d8003e50 con 0x5572dc0dcc00 2024-03-28T17:51:00.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.461+0000 7f37bf400640 1 -- 172.21.15.45:0/3420414802 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8684+0+0 (secure 0 0 0) 0x7f37d80326c0 con 0x5572dc0dcc00 2024-03-28T17:51:00.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.461+0000 7f37bf400640 1 -- 172.21.15.45:0/3420414802 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f37d805b050 con 0x5572dc0dcc00 2024-03-28T17:51:00.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.461+0000 7f37e77c8740 10 client.25889 fetch_fsmap learned FSMap version 33 2024-03-28T17:51:00.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.461+0000 7f37e77c8740 1 -- 172.21.15.45:0/3420414802 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5572dbe157d0 con 0x5572dc0dcc00 2024-03-28T17:51:00.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.461+0000 7f37bf400640 1 -- 172.21.15.45:0/3420414802 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f37d8003e50 con 0x5572dc0dcc00 2024-03-28T17:51:00.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.461+0000 7f37bf400640 1 -- 172.21.15.45:0/3420414802 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 33) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f37d8049450 con 0x5572dc0dcc00 2024-03-28T17:51:00.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.461+0000 7f37e77c8740 10 client.25889 fetch_fsmap finished waiting for FSMap version 33 2024-03-28T17:51:00.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.461+0000 7f37e77c8740 10 client.25889 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:51:00.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.461+0000 7f37e77c8740 20 client.25889 populate_metadata read hostname 'smithi045' 2024-03-28T17:51:00.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.461+0000 7f37e77c8740 1 --2- 172.21.15.45:0/3420414802 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5572dc0ec890 0x5572dc0eec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.461+0000 7f37e77c8740 4 client.25889 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:51:00.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.461+0000 7f37e77c8740 1 -- 172.21.15.45:0/3420414802 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5572dc044800 con 0x5572dc0ec890 2024-03-28T17:51:00.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.462+0000 7f37dde00640 1 --2- 172.21.15.45:0/3420414802 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5572dc0ec890 0x5572dc0eec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.462+0000 7f37dde00640 1 --2- 172.21.15.45:0/3420414802 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5572dc0ec890 0x5572dc0eec80 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32cb0ba315a67d84 server_cookie=80fb2729cda5de57 in_seq=0 out_seq=0 2024-03-28T17:51:00.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.462+0000 7f37bf400640 10 client.25889 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:51:00.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.463+0000 7f37bf400640 1 -- 172.21.15.45:0/3420414802 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5572dc044800 con 0x5572dc0ec890 2024-03-28T17:51:00.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.463+0000 7f37bf400640 10 client.25889 handle_command_reply: tid=0 2024-03-28T17:51:00.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.463+0000 7f37e77c8740 1 client.25889 shutdown 2024-03-28T17:51:00.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.463+0000 7f37e77c8740 1 -- 172.21.15.45:0/3420414802 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0dcc00 msgr2=0x5572dc0d9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:00.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.463+0000 7f37e77c8740 1 --2- 172.21.15.45:0/3420414802 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0dcc00 0x5572dc0d9e10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f37d8004af0 tx=0x7f37d8005c80 comp rx=0 tx=0).stop 2024-03-28T17:51:00.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.463+0000 7f37e77c8740 1 -- 172.21.15.45:0/3420414802 shutdown_connections 2024-03-28T17:51:00.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.463+0000 7f37e77c8740 1 --2- 172.21.15.45:0/3420414802 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5572dc0ec890 0x5572dc0eec80 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.463+0000 7f37e77c8740 1 --2- 172.21.15.45:0/3420414802 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0dcc00 0x5572dc0d9e10 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.463+0000 7f37e77c8740 1 --2- 172.21.15.45:0/3420414802 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5572dc0d4080 0x5572dc0d9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.463+0000 7f37e77c8740 1 --2- 172.21.15.45:0/3420414802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572dc0d36d0 0x5572dc0dc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.463+0000 7f37e77c8740 1 -- 172.21.15.45:0/3420414802 >> 172.21.15.45:0/3420414802 conn(0x5572dc0cdda0 msgr2=0x5572dc0cfaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:00.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.463+0000 7f37bf400640 0 client.25889 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:51:00.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.463+0000 7f37e77c8740 1 -- 172.21.15.45:0/3420414802 shutdown_connections 2024-03-28T17:51:00.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.464+0000 7f37e77c8740 1 -- 172.21.15.45:0/3420414802 wait complete. 2024-03-28T17:51:00.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.464+0000 7f37e77c8740 20 client.25889 trim_cache size 0 max 16384 2024-03-28T17:51:00.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.477+0000 7f37e77c8740 1 Processor -- start 2024-03-28T17:51:00.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.477+0000 7f37e77c8740 1 -- start start 2024-03-28T17:51:00.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.477+0000 7f37e77c8740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5572dc0d36d0 0x5572dc0cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.477+0000 7f37e77c8740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0d4080 0x5572dc0dbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.478+0000 7f37e77c8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572dc0dcc00 0x5572dc0dc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.478+0000 7f37e77c8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e01b40d0 con 0x5572dc0dcc00 2024-03-28T17:51:00.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.478+0000 7f37e77c8740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e01b1b10 con 0x5572dc0d4080 2024-03-28T17:51:00.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.478+0000 7f37e77c8740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e01b3a70 con 0x5572dc0d36d0 2024-03-28T17:51:00.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.478+0000 7f37dd400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0d4080 0x5572dc0dbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.478+0000 7f37dd400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0d4080 0x5572dc0dbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39906/0 (socket says 172.21.15.45:39906) 2024-03-28T17:51:00.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.478+0000 7f37dd400640 1 -- 172.21.15.45:0/513366314 learned_addr learned my addr 172.21.15.45:0/513366314 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:00.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.478+0000 7f37de800640 1 --2- 172.21.15.45:0/513366314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572dc0dcc00 0x5572dc0dc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.478+0000 7f37dde00640 1 --2- 172.21.15.45:0/513366314 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5572dc0d36d0 0x5572dc0cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.478+0000 7f37dd400640 1 -- 172.21.15.45:0/513366314 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5572dc0d36d0 msgr2=0x5572dc0cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:00.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.478+0000 7f37dd400640 1 --2- 172.21.15.45:0/513366314 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5572dc0d36d0 0x5572dc0cf8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.478+0000 7f37dd400640 1 -- 172.21.15.45:0/513366314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572dc0dcc00 msgr2=0x5572dc0dc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:00.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.478+0000 7f37dd400640 1 --2- 172.21.15.45:0/513366314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572dc0dcc00 0x5572dc0dc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.478+0000 7f37dd400640 1 -- 172.21.15.45:0/513366314 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572dc0d7de0 con 0x5572dc0d4080 2024-03-28T17:51:00.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.478+0000 7f37dd400640 1 --2- 172.21.15.45:0/513366314 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0d4080 0x5572dc0dbfc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f37cc0981e0 tx=0x7f37cc0d2d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58842266b3ce082b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:00.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.479+0000 7f37ca600640 1 -- 172.21.15.45:0/513366314 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc09b5e0 con 0x5572dc0d4080 2024-03-28T17:51:00.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.479+0000 7f37ca600640 1 -- 172.21.15.45:0/513366314 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f37cc09bf10 con 0x5572dc0d4080 2024-03-28T17:51:00.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.479+0000 7f37ca600640 1 -- 172.21.15.45:0/513366314 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc09b170 con 0x5572dc0d4080 2024-03-28T17:51:00.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.479+0000 7f37e77c8740 1 -- 172.21.15.45:0/513366314 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0d4080 msgr2=0x5572dc0dbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:00.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.479+0000 7f37e77c8740 1 --2- 172.21.15.45:0/513366314 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0d4080 0x5572dc0dbfc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f37cc0981e0 tx=0x7f37cc0d2d90 comp rx=0 tx=0).stop 2024-03-28T17:51:00.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.479+0000 7f37e77c8740 1 -- 172.21.15.45:0/513366314 shutdown_connections 2024-03-28T17:51:00.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.479+0000 7f37e77c8740 1 --2- 172.21.15.45:0/513366314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572dc0dcc00 0x5572dc0dc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.479+0000 7f37e77c8740 1 --2- 172.21.15.45:0/513366314 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0d4080 0x5572dc0dbfc0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.479+0000 7f37e77c8740 1 --2- 172.21.15.45:0/513366314 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5572dc0d36d0 0x5572dc0cf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.479+0000 7f37e77c8740 1 -- 172.21.15.45:0/513366314 >> 172.21.15.45:0/513366314 conn(0x5572dc0cdda0 msgr2=0x5572dc0cf1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:00.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.479+0000 7f37e77c8740 1 -- 172.21.15.45:0/513366314 shutdown_connections 2024-03-28T17:51:00.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.479+0000 7f37e77c8740 1 -- 172.21.15.45:0/513366314 wait complete. 2024-03-28T17:51:00.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.480+0000 7f37e77c8740 1 -- start start 2024-03-28T17:51:00.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.480+0000 7f37e77c8740 1 Processor -- start 2024-03-28T17:51:00.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.480+0000 7f37e77c8740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5572dc0d36d0 0x5572dc0ef7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.480+0000 7f37e77c8740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0d4080 0x5572dc0efd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.481+0000 7f37e77c8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572dc0dcc00 0x5572dc146dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.481+0000 7f37e77c8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e01b3410 con 0x5572dc0dcc00 2024-03-28T17:51:00.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.481+0000 7f37e77c8740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e01b2110 con 0x5572dc0d4080 2024-03-28T17:51:00.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.481+0000 7f37e77c8740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e01b2750 con 0x5572dc0d36d0 2024-03-28T17:51:00.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.481+0000 7f37e77c8740 10 client.0 _ll_register_callbacks cb 0x7f37d000b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:51:00.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.481+0000 7f37dd400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0d4080 0x5572dc0efd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.481+0000 7f37dde00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5572dc0d36d0 0x5572dc0ef7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.481+0000 7f37dd400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0d4080 0x5572dc0efd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39916/0 (socket says 172.21.15.45:39916) 2024-03-28T17:51:00.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.481+0000 7f37dd400640 1 -- 172.21.15.45:0/2777982058 learned_addr learned my addr 172.21.15.45:0/2777982058 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:00.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.481+0000 7f37de800640 1 --2- 172.21.15.45:0/2777982058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572dc0dcc00 0x5572dc146dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.481+0000 7f37dd400640 1 -- 172.21.15.45:0/2777982058 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5572dc0d36d0 msgr2=0x5572dc0ef7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:00.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.481+0000 7f37dd400640 1 --2- 172.21.15.45:0/2777982058 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5572dc0d36d0 0x5572dc0ef7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.481+0000 7f37dd400640 1 -- 172.21.15.45:0/2777982058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572dc0dcc00 msgr2=0x5572dc146dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:00.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.481+0000 7f37dd400640 1 --2- 172.21.15.45:0/2777982058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572dc0dcc00 0x5572dc146dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.481+0000 7f37dd400640 1 -- 172.21.15.45:0/2777982058 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5572dc147300 con 0x5572dc0d4080 2024-03-28T17:51:00.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.481+0000 7f37de800640 1 --2- 172.21.15.45:0/2777982058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572dc0dcc00 0x5572dc146dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:51:00.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.481+0000 7f37dd400640 1 --2- 172.21.15.45:0/2777982058 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0d4080 0x5572dc0efd10 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f37cc005f60 tx=0x7f37cc005c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a5ff1099814dc22 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:00.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.481+0000 7f37bf400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:51:00.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.482+0000 7f37bf400640 1 -- 172.21.15.45:0/2777982058 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc09b350 con 0x5572dc0d4080 2024-03-28T17:51:00.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.482+0000 7f37bf400640 1 -- 172.21.15.45:0/2777982058 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8684+0+0 (secure 0 0 0) 0x7f37cc09cb40 con 0x5572dc0d4080 2024-03-28T17:51:00.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.482+0000 7f37e77c8740 1 -- 172.21.15.45:0/2777982058 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f37cc09bf10 con 0x5572dc0d4080 2024-03-28T17:51:00.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.482+0000 7f37bf400640 1 -- 172.21.15.45:0/2777982058 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f37cc09b5e0 con 0x5572dc0d4080 2024-03-28T17:51:00.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.482+0000 7f37e77c8740 10 client.25895 fetch_fsmap learned FSMap version 33 2024-03-28T17:51:00.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.482+0000 7f37e77c8740 1 -- 172.21.15.45:0/2777982058 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5572dc1485d0 con 0x5572dc0d4080 2024-03-28T17:51:00.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.482+0000 7f37bf400640 1 -- 172.21.15.45:0/2777982058 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f37cc09bf10 con 0x5572dc0d4080 2024-03-28T17:51:00.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.482+0000 7f37bf400640 1 -- 172.21.15.45:0/2777982058 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 33) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f37cc0a7050 con 0x5572dc0d4080 2024-03-28T17:51:00.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.482+0000 7f37e77c8740 10 client.25895 fetch_fsmap finished waiting for FSMap version 33 2024-03-28T17:51:00.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.482+0000 7f37e77c8740 10 client.25895 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:51:00.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.482+0000 7f37e77c8740 20 client.25895 populate_metadata read hostname 'smithi045' 2024-03-28T17:51:00.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.482+0000 7f37e77c8740 1 --2- 172.21.15.45:0/2777982058 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5572dc148b50 0x5572dc14af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:00.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.482+0000 7f37e77c8740 4 client.25895 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T17:51:00.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.482+0000 7f37e77c8740 1 -- 172.21.15.45:0/2777982058 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5572dc03eb50 con 0x5572dc148b50 2024-03-28T17:51:00.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.483+0000 7f37dde00640 1 --2- 172.21.15.45:0/2777982058 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5572dc148b50 0x5572dc14af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:00.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.483+0000 7f37dde00640 1 --2- 172.21.15.45:0/2777982058 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5572dc148b50 0x5572dc14af70 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5781e2fec2b57b12 server_cookie=40df95b72a3f350a in_seq=0 out_seq=0 2024-03-28T17:51:00.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.483+0000 7f37bf400640 10 client.25895 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:51:00.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.483+0000 7f37bf400640 1 -- 172.21.15.45:0/2777982058 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5572dc03eb50 con 0x5572dc148b50 2024-03-28T17:51:00.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.483+0000 7f37bf400640 10 client.25895 handle_command_reply: tid=0 2024-03-28T17:51:00.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.483+0000 7f37e77c8740 1 client.25895 shutdown 2024-03-28T17:51:00.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.483+0000 7f37e77c8740 1 -- 172.21.15.45:0/2777982058 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0d4080 msgr2=0x5572dc0efd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:00.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.483+0000 7f37e77c8740 1 --2- 172.21.15.45:0/2777982058 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0d4080 0x5572dc0efd10 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f37cc005f60 tx=0x7f37cc005c70 comp rx=0 tx=0).stop 2024-03-28T17:51:00.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.483+0000 7f37e77c8740 1 -- 172.21.15.45:0/2777982058 shutdown_connections 2024-03-28T17:51:00.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.483+0000 7f37e77c8740 1 --2- 172.21.15.45:0/2777982058 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5572dc148b50 0x5572dc14af70 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.483+0000 7f37e77c8740 1 --2- 172.21.15.45:0/2777982058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572dc0dcc00 0x5572dc146dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.483+0000 7f37e77c8740 1 --2- 172.21.15.45:0/2777982058 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5572dc0d4080 0x5572dc0efd10 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.483+0000 7f37e77c8740 1 --2- 172.21.15.45:0/2777982058 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5572dc0d36d0 0x5572dc0ef7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.483+0000 7f37bf400640 0 client.25895 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:51:00.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.483+0000 7f37e77c8740 1 -- 172.21.15.45:0/2777982058 >> 172.21.15.45:0/2777982058 conn(0x5572dc0cdda0 msgr2=0x5572dc0d4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:00.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.484+0000 7f37e77c8740 1 -- 172.21.15.45:0/2777982058 shutdown_connections 2024-03-28T17:51:00.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.484+0000 7f37e77c8740 1 -- 172.21.15.45:0/2777982058 wait complete. 2024-03-28T17:51:00.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.484+0000 7f37e77c8740 20 client.25895 trim_cache size 0 max 16384 2024-03-28T17:51:00.488 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-28T17:51:00.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.484+0000 7f37e4a00640 1 -- 172.21.15.45:0/2436702842 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f37c40a2970 msgr2=0x7f37c40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:00.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.484+0000 7f37e4a00640 1 --2- 172.21.15.45:0/2436702842 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f37c40a2970 0x7f37c40a4e30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x5572dc0cee10 tx=0x7f37d000b390 comp rx=0 tx=0).stop 2024-03-28T17:51:00.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.484+0000 7f37e4a00640 1 -- 172.21.15.45:0/2436702842 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37e0101020 msgr2=0x7f37e019ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:00.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.484+0000 7f37e4a00640 1 --2- 172.21.15.45:0/2436702842 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37e0101020 0x7f37e019ea80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f37cc00e9f0 tx=0x7f37cc00eeb0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.484+0000 7f37e4a00640 1 -- 172.21.15.45:0/2436702842 shutdown_connections 2024-03-28T17:51:00.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.484+0000 7f37e4a00640 1 --2- 172.21.15.45:0/2436702842 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f37c40a2970 0x7f37c40a4e30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.484+0000 7f37e4a00640 1 --2- 172.21.15.45:0/2436702842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37e0105b00 0x7f37e01a2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.484+0000 7f37e4a00640 1 --2- 172.21.15.45:0/2436702842 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37e0101020 0x7f37e019ea80 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.484+0000 7f37e4a00640 1 --2- 172.21.15.45:0/2436702842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37e0100700 0x7f37e019e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:00.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.484+0000 7f37e4a00640 1 -- 172.21.15.45:0/2436702842 >> 172.21.15.45:0/2436702842 conn(0x7f37e0077700 msgr2=0x7f37e01034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:00.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.485+0000 7f37e4a00640 1 -- 172.21.15.45:0/2436702842 shutdown_connections 2024-03-28T17:51:00.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:00.485+0000 7f37e4a00640 1 -- 172.21.15.45:0/2436702842 wait complete. 2024-03-28T17:51:00.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:00 smithi071 ceph-mon[36597]: pgmap v64: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-28T17:51:00.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:00 smithi071 ceph-mon[36597]: from='client.? 172.21.15.45:0/3829083011' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T17:51:00.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:00 smithi071 ceph-mon[36597]: daemon mds.f restarted 2024-03-28T17:51:00.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:00 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:51:00.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:00 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T17:51:00.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:00 smithi071 ceph-mon[36597]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T17:51:00.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:00 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/1934350429,v1:172.21.15.151:6835/1934350429] up:boot 2024-03-28T17:51:00.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:00 smithi071 ceph-mon[36597]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-03-28T17:51:00.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:00 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:51:00.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:00 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:00.836 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.79 MB/sec warmup 33 sec latency 28733.135 ms 2024-03-28T17:51:01.504 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T17:51:01.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T17:51:01.504 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T17:51:01.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.745+0000 7f8c4be00640 1 -- 172.21.15.45:0/1139334869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c4c1122a0 msgr2=0x7f8c4c114690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:01.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.745+0000 7f8c4be00640 1 --2- 172.21.15.45:0/1139334869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c4c1122a0 0x7f8c4c114690 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8c44009cd0 tx=0x7f8c4402c170 comp rx=0 tx=0).stop 2024-03-28T17:51:01.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.745+0000 7f8c4be00640 1 -- 172.21.15.45:0/1139334869 shutdown_connections 2024-03-28T17:51:01.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.745+0000 7f8c4be00640 1 --2- 172.21.15.45:0/1139334869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c4c1122a0 0x7f8c4c114690 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.745+0000 7f8c4be00640 1 --2- 172.21.15.45:0/1139334869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c4c100730 0x7f8c4c111d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.745+0000 7f8c4be00640 1 --2- 172.21.15.45:0/1139334869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c4c0ffe10 0x7f8c4c1001f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.745+0000 7f8c4be00640 1 -- 172.21.15.45:0/1139334869 >> 172.21.15.45:0/1139334869 conn(0x7f8c4c0756e0 msgr2=0x7f8c4c075af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:01.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.745+0000 7f8c4be00640 1 -- 172.21.15.45:0/1139334869 shutdown_connections 2024-03-28T17:51:01.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.745+0000 7f8c4be00640 1 -- 172.21.15.45:0/1139334869 wait complete. 2024-03-28T17:51:01.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.746+0000 7f8c4be00640 1 Processor -- start 2024-03-28T17:51:01.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.746+0000 7f8c4be00640 1 -- start start 2024-03-28T17:51:01.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.746+0000 7f8c4be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c4c0ffe10 0x7f8c4c19e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:01.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.746+0000 7f8c4be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c4c100730 0x7f8c4c19e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:01.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.746+0000 7f8c4be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c4c1122a0 0x7f8c4c1a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:01.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.747+0000 7f8c4be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c117ef0 con 0x7f8c4c100730 2024-03-28T17:51:01.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.747+0000 7f8c4be00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c117d70 con 0x7f8c4c1122a0 2024-03-28T17:51:01.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.747+0000 7f8c4be00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c118070 con 0x7f8c4c0ffe10 2024-03-28T17:51:01.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.747+0000 7f8c4b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c4c1122a0 0x7f8c4c1a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:01.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.747+0000 7f8c4a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c4c100730 0x7f8c4c19e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:01.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.747+0000 7f8c4b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c4c1122a0 0x7f8c4c1a2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39938/0 (socket says 172.21.15.45:39938) 2024-03-28T17:51:01.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.747+0000 7f8c4b400640 1 -- 172.21.15.45:0/2827835128 learned_addr learned my addr 172.21.15.45:0/2827835128 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:01.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.747+0000 7f8c4aa00640 1 --2- 172.21.15.45:0/2827835128 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c4c0ffe10 0x7f8c4c19e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:01.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.747+0000 7f8c4b400640 1 -- 172.21.15.45:0/2827835128 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c4c0ffe10 msgr2=0x7f8c4c19e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:01.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.747+0000 7f8c4b400640 1 --2- 172.21.15.45:0/2827835128 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c4c0ffe10 0x7f8c4c19e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.747+0000 7f8c4b400640 1 -- 172.21.15.45:0/2827835128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c4c100730 msgr2=0x7f8c4c19e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:01.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.747+0000 7f8c4b400640 1 --2- 172.21.15.45:0/2827835128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c4c100730 0x7f8c4c19e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.747+0000 7f8c4b400640 1 -- 172.21.15.45:0/2827835128 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c4c1a3420 con 0x7f8c4c1122a0 2024-03-28T17:51:01.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.747+0000 7f8c4b400640 1 --2- 172.21.15.45:0/2827835128 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c4c1122a0 0x7f8c4c1a2cf0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8c44005e30 tx=0x7f8c44035d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:01.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.748+0000 7f8c50a00640 1 -- 172.21.15.45:0/2827835128 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c44039040 con 0x7f8c4c1122a0 2024-03-28T17:51:01.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.748+0000 7f8c50a00640 1 -- 172.21.15.45:0/2827835128 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8c44044070 con 0x7f8c4c1122a0 2024-03-28T17:51:01.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.748+0000 7f8c50a00640 1 -- 172.21.15.45:0/2827835128 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c44032860 con 0x7f8c4c1122a0 2024-03-28T17:51:01.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.748+0000 7f8c4be00640 1 -- 172.21.15.45:0/2827835128 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c4c1a36b0 con 0x7f8c4c1122a0 2024-03-28T17:51:01.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.748+0000 7f8c4be00640 1 -- 172.21.15.45:0/2827835128 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c4c1aafa0 con 0x7f8c4c1122a0 2024-03-28T17:51:01.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.751+0000 7f8c50a00640 1 -- 172.21.15.45:0/2827835128 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f8c44003b70 con 0x7f8c4c1122a0 2024-03-28T17:51:01.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.751+0000 7f8c54b76740 1 Processor -- start 2024-03-28T17:51:01.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.751+0000 7f8c54b76740 1 -- start start 2024-03-28T17:51:01.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.752+0000 7f8c54b76740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2290 0x55927c1a2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:01.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.752+0000 7f8c54b76740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55927c1a2bb0 0x55927c1ab250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:01.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.752+0000 7f8c54b76740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55927c1ab790 0x55927c1adb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:01.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.752+0000 7f8c54b76740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55927c106eb0 con 0x55927c1a2bb0 2024-03-28T17:51:01.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.752+0000 7f8c54b76740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55927c0afa60 con 0x55927c1a2290 2024-03-28T17:51:01.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.752+0000 7f8c54b76740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55927c102f50 con 0x55927c1ab790 2024-03-28T17:51:01.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.752+0000 7f8c4a000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2290 0x55927c1a2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:01.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.752+0000 7f8c4a000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2290 0x55927c1a2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39944/0 (socket says 172.21.15.45:39944) 2024-03-28T17:51:01.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.752+0000 7f8c4a000640 1 -- 172.21.15.45:0/2754539477 learned_addr learned my addr 172.21.15.45:0/2754539477 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:01.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.752+0000 7f8c4aa00640 1 --2- 172.21.15.45:0/2754539477 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55927c1a2bb0 0x55927c1ab250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:01.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.752+0000 7f8c4a000640 1 --2- 172.21.15.45:0/2754539477 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55927c1ab790 0x55927c1adb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:01.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.752+0000 7f8c4a000640 1 -- 172.21.15.45:0/2754539477 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55927c1ab790 msgr2=0x55927c1adb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:01.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.752+0000 7f8c4a000640 1 --2- 172.21.15.45:0/2754539477 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55927c1ab790 0x55927c1adb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.752+0000 7f8c4a000640 1 -- 172.21.15.45:0/2754539477 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55927c1a2bb0 msgr2=0x55927c1ab250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:01.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.752+0000 7f8c4a000640 1 --2- 172.21.15.45:0/2754539477 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55927c1a2bb0 0x55927c1ab250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.752+0000 7f8c4a000640 1 -- 172.21.15.45:0/2754539477 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55927bee37e0 con 0x55927c1a2290 2024-03-28T17:51:01.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.753+0000 7f8c50a00640 1 --2- 172.21.15.45:0/2827835128 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8c300a2970 0x7f8c300a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:01.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.753+0000 7f8c4a000640 1 --2- 172.21.15.45:0/2754539477 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2290 0x55927c1a2670 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8c38009fa0 tx=0x7f8c38005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62a5088af715469c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:01.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.753+0000 7f8c50a00640 1 -- 172.21.15.45:0/2827835128 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8684+0+0 (secure 0 0 0) 0x7f8c440f11d0 con 0x7f8c4c1122a0 2024-03-28T17:51:01.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.753+0000 7f8c4aa00640 1 --2- 172.21.15.45:0/2827835128 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8c300a2970 0x7f8c300a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:01.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.753+0000 7f8c37e00640 1 -- 172.21.15.45:0/2754539477 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c38004050 con 0x55927c1a2290 2024-03-28T17:51:01.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.753+0000 7f8c37e00640 1 -- 172.21.15.45:0/2754539477 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8c380041f0 con 0x55927c1a2290 2024-03-28T17:51:01.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.753+0000 7f8c37e00640 1 -- 172.21.15.45:0/2754539477 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3800f760 con 0x55927c1a2290 2024-03-28T17:51:01.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.754+0000 7f8c54b76740 1 -- 172.21.15.45:0/2754539477 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2290 msgr2=0x55927c1a2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:01.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.754+0000 7f8c4aa00640 1 --2- 172.21.15.45:0/2827835128 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8c300a2970 0x7f8c300a4e30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55927c1a1d60 tx=0x7f8c3c005f90 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:01.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.754+0000 7f8c54b76740 1 --2- 172.21.15.45:0/2754539477 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2290 0x55927c1a2670 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8c38009fa0 tx=0x7f8c38005850 comp rx=0 tx=0).stop 2024-03-28T17:51:01.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.754+0000 7f8c54b76740 1 -- 172.21.15.45:0/2754539477 shutdown_connections 2024-03-28T17:51:01.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.754+0000 7f8c54b76740 1 --2- 172.21.15.45:0/2754539477 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55927c1ab790 0x55927c1adb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.754+0000 7f8c54b76740 1 --2- 172.21.15.45:0/2754539477 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55927c1a2bb0 0x55927c1ab250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.754+0000 7f8c54b76740 1 --2- 172.21.15.45:0/2754539477 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2290 0x55927c1a2670 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.754+0000 7f8c54b76740 1 -- 172.21.15.45:0/2754539477 >> 172.21.15.45:0/2754539477 conn(0x55927c19c9c0 msgr2=0x55927c19cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:01.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.754+0000 7f8c54b76740 1 -- 172.21.15.45:0/2754539477 shutdown_connections 2024-03-28T17:51:01.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.754+0000 7f8c54b76740 1 -- 172.21.15.45:0/2754539477 wait complete. 2024-03-28T17:51:01.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.754+0000 7f8c54b76740 1 -- start start 2024-03-28T17:51:01.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.755+0000 7f8c54b76740 1 Processor -- start 2024-03-28T17:51:01.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.755+0000 7f8c54b76740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2290 0x55927c1ab150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:01.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.755+0000 7f8c54b76740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55927c1a2bb0 0x55927c1a84e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:01.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.755+0000 7f8c54b76740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55927c1ab790 0x55927c1a8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:01.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.755+0000 7f8c54b76740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55927c1ae580 con 0x55927c1ab790 2024-03-28T17:51:01.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.755+0000 7f8c54b76740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55927c0c2eb0 con 0x55927c1a2290 2024-03-28T17:51:01.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.755+0000 7f8c54b76740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55927c1ae700 con 0x55927c1a2bb0 2024-03-28T17:51:01.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.755+0000 7f8c54b76740 10 client.0 _ll_register_callbacks cb 0x55927c0b7150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:51:01.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.755+0000 7f8c4aa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2290 0x55927c1ab150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:01.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.755+0000 7f8c4aa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2290 0x55927c1ab150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39952/0 (socket says 172.21.15.45:39952) 2024-03-28T17:51:01.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.755+0000 7f8c4aa00640 1 -- 172.21.15.45:0/3090333796 learned_addr learned my addr 172.21.15.45:0/3090333796 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:01.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.755+0000 7f8c4b400640 1 --2- 172.21.15.45:0/3090333796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55927c1ab790 0x55927c1a8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:01.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.756+0000 7f8c4a000640 1 --2- 172.21.15.45:0/3090333796 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55927c1a2bb0 0x55927c1a84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:01.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.756+0000 7f8c4aa00640 1 -- 172.21.15.45:0/3090333796 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55927c1a2bb0 msgr2=0x55927c1a84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:01.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.756+0000 7f8c4aa00640 1 --2- 172.21.15.45:0/3090333796 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55927c1a2bb0 0x55927c1a84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.756+0000 7f8c4aa00640 1 -- 172.21.15.45:0/3090333796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55927c1ab790 msgr2=0x55927c1a8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:01.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.756+0000 7f8c4aa00640 1 --2- 172.21.15.45:0/3090333796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55927c1ab790 0x55927c1a8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.756+0000 7f8c4aa00640 1 -- 172.21.15.45:0/3090333796 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55927bf43f70 con 0x55927c1a2290 2024-03-28T17:51:01.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.756+0000 7f8c4aa00640 1 --2- 172.21.15.45:0/3090333796 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2290 0x55927c1ab150 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c01ea00 tx=0x7f8c3c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f78b3a79cd4b190 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:01.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.756+0000 7f8c34c00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:51:01.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.756+0000 7f8c34c00640 1 -- 172.21.15.45:0/3090333796 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c02e060 con 0x55927c1a2290 2024-03-28T17:51:01.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.756+0000 7f8c34c00640 1 -- 172.21.15.45:0/3090333796 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8684+0+0 (secure 0 0 0) 0x7f8c3c018030 con 0x55927c1a2290 2024-03-28T17:51:01.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.756+0000 7f8c54b76740 1 -- 172.21.15.45:0/3090333796 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8c380041f0 con 0x55927c1a2290 2024-03-28T17:51:01.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.757+0000 7f8c34c00640 1 -- 172.21.15.45:0/3090333796 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c3c035050 con 0x55927c1a2290 2024-03-28T17:51:01.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.757+0000 7f8c54b76740 10 client.25907 fetch_fsmap learned FSMap version 34 2024-03-28T17:51:01.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.757+0000 7f8c54b76740 1 -- 172.21.15.45:0/3090333796 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55927bddb170 con 0x55927c1a2290 2024-03-28T17:51:01.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.757+0000 7f8c34c00640 1 -- 172.21.15.45:0/3090333796 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8c380041f0 con 0x55927c1a2290 2024-03-28T17:51:01.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.757+0000 7f8c34c00640 1 -- 172.21.15.45:0/3090333796 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 34) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f8c3c01c5f0 con 0x55927c1a2290 2024-03-28T17:51:01.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.757+0000 7f8c54b76740 10 client.25907 fetch_fsmap finished waiting for FSMap version 34 2024-03-28T17:51:01.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.757+0000 7f8c54b76740 10 client.25907 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:51:01.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.757+0000 7f8c54b76740 20 client.25907 populate_metadata read hostname 'smithi045' 2024-03-28T17:51:01.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.757+0000 7f8c54b76740 1 --2- 172.21.15.45:0/3090333796 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55927c1bb8d0 0x55927c1bdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:01.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.757+0000 7f8c54b76740 4 client.25907 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:51:01.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.757+0000 7f8c54b76740 1 -- 172.21.15.45:0/3090333796 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55927c1143f0 con 0x55927c1bb8d0 2024-03-28T17:51:01.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.758+0000 7f8c4a000640 1 --2- 172.21.15.45:0/3090333796 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55927c1bb8d0 0x55927c1bdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:01.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.758+0000 7f8c4a000640 1 --2- 172.21.15.45:0/3090333796 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55927c1bb8d0 0x55927c1bdd00 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=375325820a3b19a0 server_cookie=3aa15c3784d5cdfb in_seq=0 out_seq=0 2024-03-28T17:51:01.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.758+0000 7f8c34c00640 10 client.25907 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:51:01.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.759+0000 7f8c34c00640 1 -- 172.21.15.45:0/3090333796 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55927c1143f0 con 0x55927c1bb8d0 2024-03-28T17:51:01.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.759+0000 7f8c34c00640 10 client.25907 handle_command_reply: tid=0 2024-03-28T17:51:01.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.759+0000 7f8c54b76740 1 client.25907 shutdown 2024-03-28T17:51:01.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.759+0000 7f8c54b76740 1 -- 172.21.15.45:0/3090333796 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2290 msgr2=0x55927c1ab150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:01.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.759+0000 7f8c54b76740 1 --2- 172.21.15.45:0/3090333796 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2290 0x55927c1ab150 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c01ea00 tx=0x7f8c3c004730 comp rx=0 tx=0).stop 2024-03-28T17:51:01.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.759+0000 7f8c54b76740 1 -- 172.21.15.45:0/3090333796 shutdown_connections 2024-03-28T17:51:01.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.759+0000 7f8c54b76740 1 --2- 172.21.15.45:0/3090333796 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55927c1bb8d0 0x55927c1bdd00 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.759+0000 7f8c54b76740 1 --2- 172.21.15.45:0/3090333796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55927c1ab790 0x55927c1a8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.759+0000 7f8c54b76740 1 --2- 172.21.15.45:0/3090333796 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55927c1a2bb0 0x55927c1a84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.759+0000 7f8c54b76740 1 --2- 172.21.15.45:0/3090333796 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2290 0x55927c1ab150 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.759+0000 7f8c34c00640 0 client.25907 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:51:01.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.759+0000 7f8c54b76740 1 -- 172.21.15.45:0/3090333796 >> 172.21.15.45:0/3090333796 conn(0x55927c19c9c0 msgr2=0x55927c19e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:01.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.759+0000 7f8c54b76740 1 -- 172.21.15.45:0/3090333796 shutdown_connections 2024-03-28T17:51:01.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.760+0000 7f8c54b76740 1 -- 172.21.15.45:0/3090333796 wait complete. 2024-03-28T17:51:01.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.760+0000 7f8c54b76740 20 client.25907 trim_cache size 0 max 16384 2024-03-28T17:51:01.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.778+0000 7f8c54b76740 1 Processor -- start 2024-03-28T17:51:01.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.778+0000 7f8c54b76740 1 -- start start 2024-03-28T17:51:01.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.778+0000 7f8c54b76740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55927c1a2290 0x55927c1aac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:01.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.778+0000 7f8c54b76740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2bb0 0x55927c1ab150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:01.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.778+0000 7f8c54b76740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55927c1ab790 0x55927c19b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:01.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.778+0000 7f8c54b76740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c1b3e50 con 0x55927c1ab790 2024-03-28T17:51:01.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.778+0000 7f8c54b76740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c1b1890 con 0x55927c1a2bb0 2024-03-28T17:51:01.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.778+0000 7f8c54b76740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c1b37f0 con 0x55927c1a2290 2024-03-28T17:51:01.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.778+0000 7f8c4a000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2bb0 0x55927c1ab150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:01.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.778+0000 7f8c4a000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2bb0 0x55927c1ab150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39966/0 (socket says 172.21.15.45:39966) 2024-03-28T17:51:01.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.778+0000 7f8c4a000640 1 -- 172.21.15.45:0/1131927752 learned_addr learned my addr 172.21.15.45:0/1131927752 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:01.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.779+0000 7f8c4aa00640 1 --2- 172.21.15.45:0/1131927752 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55927c1a2290 0x55927c1aac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:01.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.779+0000 7f8c4b400640 1 --2- 172.21.15.45:0/1131927752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55927c1ab790 0x55927c19b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:01.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.779+0000 7f8c4a000640 1 -- 172.21.15.45:0/1131927752 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55927c1a2290 msgr2=0x55927c1aac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:01.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.779+0000 7f8c4a000640 1 --2- 172.21.15.45:0/1131927752 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55927c1a2290 0x55927c1aac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.779+0000 7f8c4a000640 1 -- 172.21.15.45:0/1131927752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55927c1ab790 msgr2=0x55927c19b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:01.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.779+0000 7f8c4a000640 1 --2- 172.21.15.45:0/1131927752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55927c1ab790 0x55927c19b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.779+0000 7f8c4a000640 1 -- 172.21.15.45:0/1131927752 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55927c1a6a30 con 0x55927c1a2bb0 2024-03-28T17:51:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.779+0000 7f8c4a000640 1 --2- 172.21.15.45:0/1131927752 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2bb0 0x55927c1ab150 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8c38009fd0 tx=0x7f8c38003df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=868eaf3c68ecf28e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.779+0000 7f8c37400640 1 -- 172.21.15.45:0/1131927752 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c38030070 con 0x55927c1a2bb0 2024-03-28T17:51:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.779+0000 7f8c37400640 1 -- 172.21.15.45:0/1131927752 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8c38003e60 con 0x55927c1a2bb0 2024-03-28T17:51:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.780+0000 7f8c37400640 1 -- 172.21.15.45:0/1131927752 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3802a610 con 0x55927c1a2bb0 2024-03-28T17:51:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.780+0000 7f8c54b76740 1 -- 172.21.15.45:0/1131927752 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2bb0 msgr2=0x55927c1ab150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.780+0000 7f8c54b76740 1 --2- 172.21.15.45:0/1131927752 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2bb0 0x55927c1ab150 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8c38009fd0 tx=0x7f8c38003df0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.780+0000 7f8c54b76740 1 -- 172.21.15.45:0/1131927752 shutdown_connections 2024-03-28T17:51:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.780+0000 7f8c54b76740 1 --2- 172.21.15.45:0/1131927752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55927c1ab790 0x55927c19b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.780+0000 7f8c54b76740 1 --2- 172.21.15.45:0/1131927752 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2bb0 0x55927c1ab150 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.780+0000 7f8c54b76740 1 --2- 172.21.15.45:0/1131927752 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55927c1a2290 0x55927c1aac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.780+0000 7f8c54b76740 1 -- 172.21.15.45:0/1131927752 >> 172.21.15.45:0/1131927752 conn(0x55927c19c9c0 msgr2=0x55927c19b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.780+0000 7f8c54b76740 1 -- 172.21.15.45:0/1131927752 shutdown_connections 2024-03-28T17:51:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.780+0000 7f8c54b76740 1 -- 172.21.15.45:0/1131927752 wait complete. 2024-03-28T17:51:01.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.781+0000 7f8c54b76740 1 -- start start 2024-03-28T17:51:01.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.781+0000 7f8c54b76740 1 Processor -- start 2024-03-28T17:51:01.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.781+0000 7f8c54b76740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55927c1a2290 0x55927c1b36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:01.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.781+0000 7f8c54b76740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2bb0 0x55927c1afa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:01.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.782+0000 7f8c54b76740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55927c1ab790 0x55927c1affa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:01.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.782+0000 7f8c54b76740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c1b3190 con 0x55927c1ab790 2024-03-28T17:51:01.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.782+0000 7f8c54b76740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c1b1e90 con 0x55927c1a2bb0 2024-03-28T17:51:01.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.782+0000 7f8c54b76740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c1b24d0 con 0x55927c1a2290 2024-03-28T17:51:01.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.782+0000 7f8c54b76740 10 client.0 _ll_register_callbacks cb 0x7f8c3c009da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:51:01.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.782+0000 7f8c4b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55927c1ab790 0x55927c1affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:01.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.782+0000 7f8c4a000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2bb0 0x55927c1afa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:01.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.782+0000 7f8c4aa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55927c1a2290 0x55927c1b36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:01.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.782+0000 7f8c4aa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55927c1a2290 0x55927c1b36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:40182/0 (socket says 172.21.15.45:40182) 2024-03-28T17:51:01.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.782+0000 7f8c4a000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2bb0 0x55927c1afa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39974/0 (socket says 172.21.15.45:39974) 2024-03-28T17:51:01.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.782+0000 7f8c4aa00640 1 -- 172.21.15.45:0/4189765871 learned_addr learned my addr 172.21.15.45:0/4189765871 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:01.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.782+0000 7f8c4a000640 1 -- 172.21.15.45:0/4189765871 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55927c1a2290 msgr2=0x55927c1b36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:01.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.782+0000 7f8c4a000640 1 --2- 172.21.15.45:0/4189765871 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55927c1a2290 0x55927c1b36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.782+0000 7f8c4a000640 1 -- 172.21.15.45:0/4189765871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55927c1ab790 msgr2=0x55927c1affa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:01.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.782+0000 7f8c4a000640 1 --2- 172.21.15.45:0/4189765871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55927c1ab790 0x55927c1affa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.782+0000 7f8c4a000640 1 -- 172.21.15.45:0/4189765871 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55927c1b07b0 con 0x55927c1a2bb0 2024-03-28T17:51:01.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.782+0000 7f8c4aa00640 1 --2- 172.21.15.45:0/4189765871 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55927c1a2290 0x55927c1b36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:51:01.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.782+0000 7f8c4b400640 1 --2- 172.21.15.45:0/4189765871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55927c1ab790 0x55927c1affa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:51:01.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.783+0000 7f8c4a000640 1 --2- 172.21.15.45:0/4189765871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2bb0 0x55927c1afa60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x55927c1b8050 tx=0x7f8c38013510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9abd4fcd9ac77e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:01.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.783+0000 7f8c34c00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:51:01.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.783+0000 7f8c34c00640 1 -- 172.21.15.45:0/4189765871 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c38030070 con 0x55927c1a2bb0 2024-03-28T17:51:01.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.783+0000 7f8c34c00640 1 -- 172.21.15.45:0/4189765871 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8684+0+0 (secure 0 0 0) 0x7f8c38003a20 con 0x55927c1a2bb0 2024-03-28T17:51:01.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.783+0000 7f8c54b76740 1 -- 172.21.15.45:0/4189765871 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8c38003e60 con 0x55927c1a2bb0 2024-03-28T17:51:01.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.783+0000 7f8c34c00640 1 -- 172.21.15.45:0/4189765871 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c380035f0 con 0x55927c1a2bb0 2024-03-28T17:51:01.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.783+0000 7f8c54b76740 10 client.25913 fetch_fsmap learned FSMap version 34 2024-03-28T17:51:01.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.783+0000 7f8c54b76740 1 -- 172.21.15.45:0/4189765871 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55927c2181f0 con 0x55927c1a2bb0 2024-03-28T17:51:01.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.784+0000 7f8c34c00640 1 -- 172.21.15.45:0/4189765871 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8c38003e60 con 0x55927c1a2bb0 2024-03-28T17:51:01.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.784+0000 7f8c34c00640 1 -- 172.21.15.45:0/4189765871 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 34) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f8c3801e070 con 0x55927c1a2bb0 2024-03-28T17:51:01.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.784+0000 7f8c54b76740 10 client.25913 fetch_fsmap finished waiting for FSMap version 34 2024-03-28T17:51:01.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.784+0000 7f8c54b76740 10 client.25913 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:51:01.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.784+0000 7f8c54b76740 20 client.25913 populate_metadata read hostname 'smithi045' 2024-03-28T17:51:01.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.784+0000 7f8c54b76740 1 --2- 172.21.15.45:0/4189765871 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55927c218610 0x55927c21aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:01.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.784+0000 7f8c54b76740 4 client.25913 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T17:51:01.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.784+0000 7f8c54b76740 1 -- 172.21.15.45:0/4189765871 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55927c10e740 con 0x55927c218610 2024-03-28T17:51:01.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.784+0000 7f8c4aa00640 1 --2- 172.21.15.45:0/4189765871 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55927c218610 0x55927c21aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:01.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.785+0000 7f8c4aa00640 1 --2- 172.21.15.45:0/4189765871 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55927c218610 0x55927c21aa30 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e619eb83c70aba6 server_cookie=e56dc8e5ff8d81f5 in_seq=0 out_seq=0 2024-03-28T17:51:01.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.785+0000 7f8c34c00640 10 client.25913 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:51:01.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.786+0000 7f8c34c00640 1 -- 172.21.15.45:0/4189765871 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55927c10e740 con 0x55927c218610 2024-03-28T17:51:01.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.786+0000 7f8c34c00640 10 client.25913 handle_command_reply: tid=0 2024-03-28T17:51:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.786+0000 7f8c54b76740 1 client.25913 shutdown 2024-03-28T17:51:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.786+0000 7f8c54b76740 1 -- 172.21.15.45:0/4189765871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2bb0 msgr2=0x55927c1afa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.786+0000 7f8c54b76740 1 --2- 172.21.15.45:0/4189765871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2bb0 0x55927c1afa60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x55927c1b8050 tx=0x7f8c38013510 comp rx=0 tx=0).stop 2024-03-28T17:51:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.786+0000 7f8c54b76740 1 -- 172.21.15.45:0/4189765871 shutdown_connections 2024-03-28T17:51:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.786+0000 7f8c54b76740 1 --2- 172.21.15.45:0/4189765871 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55927c218610 0x55927c21aa30 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.786+0000 7f8c54b76740 1 --2- 172.21.15.45:0/4189765871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55927c1ab790 0x55927c1affa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.786+0000 7f8c54b76740 1 --2- 172.21.15.45:0/4189765871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55927c1a2bb0 0x55927c1afa60 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.786+0000 7f8c54b76740 1 --2- 172.21.15.45:0/4189765871 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55927c1a2290 0x55927c1b36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.786+0000 7f8c34c00640 0 client.25913 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:51:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.786+0000 7f8c54b76740 1 -- 172.21.15.45:0/4189765871 >> 172.21.15.45:0/4189765871 conn(0x55927c19c9c0 msgr2=0x55927c19ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.786+0000 7f8c54b76740 1 -- 172.21.15.45:0/4189765871 shutdown_connections 2024-03-28T17:51:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.786+0000 7f8c54b76740 1 -- 172.21.15.45:0/4189765871 wait complete. 2024-03-28T17:51:01.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.786+0000 7f8c54b76740 20 client.25913 trim_cache size 0 max 16384 2024-03-28T17:51:01.793 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:51:01.793 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-28T17:51:01.793 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T17:51:01.793 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-28T17:51:01.793 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:51:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.787+0000 7f8c4be00640 1 -- 172.21.15.45:0/2827835128 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8c300a2970 msgr2=0x7f8c300a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.787+0000 7f8c4be00640 1 --2- 172.21.15.45:0/2827835128 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8c300a2970 0x7f8c300a4e30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55927c1a1d60 tx=0x7f8c3c005f90 comp rx=0 tx=0).stop 2024-03-28T17:51:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.787+0000 7f8c4be00640 1 -- 172.21.15.45:0/2827835128 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c4c1122a0 msgr2=0x7f8c4c1a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.787+0000 7f8c4be00640 1 --2- 172.21.15.45:0/2827835128 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c4c1122a0 0x7f8c4c1a2cf0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8c44005e30 tx=0x7f8c44035d80 comp rx=0 tx=0).stop 2024-03-28T17:51:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.787+0000 7f8c4be00640 1 -- 172.21.15.45:0/2827835128 shutdown_connections 2024-03-28T17:51:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.787+0000 7f8c4be00640 1 --2- 172.21.15.45:0/2827835128 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8c300a2970 0x7f8c300a4e30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.787+0000 7f8c4be00640 1 --2- 172.21.15.45:0/2827835128 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c4c1122a0 0x7f8c4c1a2cf0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.787+0000 7f8c4be00640 1 --2- 172.21.15.45:0/2827835128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c4c100730 0x7f8c4c19e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.787+0000 7f8c4be00640 1 --2- 172.21.15.45:0/2827835128 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8c4c0ffe10 0x7f8c4c19e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.787+0000 7f8c4be00640 1 -- 172.21.15.45:0/2827835128 >> 172.21.15.45:0/2827835128 conn(0x7f8c4c0756e0 msgr2=0x7f8c4c10fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:01.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.787+0000 7f8c4be00640 1 -- 172.21.15.45:0/2827835128 shutdown_connections 2024-03-28T17:51:01.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:01.787+0000 7f8c4be00640 1 -- 172.21.15.45:0/2827835128 wait complete. 2024-03-28T17:51:01.809 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:51:01.809 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:51:01.836 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.74 MB/sec warmup 34 sec latency 29733.219 ms 2024-03-28T17:51:02.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.036+0000 7f2dc0e00640 1 -- 172.21.15.45:0/3912315121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dbc1002d0 msgr2=0x7f2dbc1006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:02.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.036+0000 7f2dc0e00640 1 --2- 172.21.15.45:0/3912315121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dbc1002d0 0x7f2dbc1006b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2da4009860 tx=0x7f2da402bb70 comp rx=0 tx=0).stop 2024-03-28T17:51:02.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.037+0000 7f2dc0e00640 1 -- 172.21.15.45:0/3912315121 shutdown_connections 2024-03-28T17:51:02.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.037+0000 7f2dc0e00640 1 --2- 172.21.15.45:0/3912315121 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2dbc10de80 0x7f2dbc110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.037+0000 7f2dc0e00640 1 --2- 172.21.15.45:0/3912315121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2dbc100c80 0x7f2dbc10d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.037+0000 7f2dc0e00640 1 --2- 172.21.15.45:0/3912315121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dbc1002d0 0x7f2dbc1006b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.037+0000 7f2dc0e00640 1 -- 172.21.15.45:0/3912315121 >> 172.21.15.45:0/3912315121 conn(0x7f2dbc0fbef0 msgr2=0x7f2dbc0fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:02.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.037+0000 7f2dc0e00640 1 -- 172.21.15.45:0/3912315121 shutdown_connections 2024-03-28T17:51:02.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.037+0000 7f2dc0e00640 1 -- 172.21.15.45:0/3912315121 wait complete. 2024-03-28T17:51:02.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.038+0000 7f2dc0e00640 1 Processor -- start 2024-03-28T17:51:02.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.038+0000 7f2dc0e00640 1 -- start start 2024-03-28T17:51:02.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.039+0000 7f2dc0e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2dbc1002d0 0x7f2dbc19a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:02.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.039+0000 7f2dc0e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dbc100c80 0x7f2dbc19a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:02.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.039+0000 7f2dc0e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2dbc10de80 0x7f2dbc19e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:02.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.039+0000 7f2dc0e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dbc113b20 con 0x7f2dbc100c80 2024-03-28T17:51:02.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.039+0000 7f2dc0e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dbc1139a0 con 0x7f2dbc1002d0 2024-03-28T17:51:02.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.039+0000 7f2dc0e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dbc113ca0 con 0x7f2dbc10de80 2024-03-28T17:51:02.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.039+0000 7f2dbb400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2dbc1002d0 0x7f2dbc19a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:02.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.039+0000 7f2dbaa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dbc100c80 0x7f2dbc19a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:02.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.039+0000 7f2dbb400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2dbc1002d0 0x7f2dbc19a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40006/0 (socket says 172.21.15.45:40006) 2024-03-28T17:51:02.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.039+0000 7f2dbb400640 1 -- 172.21.15.45:0/568845753 learned_addr learned my addr 172.21.15.45:0/568845753 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:02.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.039+0000 7f2dbbe00640 1 --2- 172.21.15.45:0/568845753 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2dbc10de80 0x7f2dbc19e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:02.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.040+0000 7f2dbb400640 1 -- 172.21.15.45:0/568845753 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2dbc10de80 msgr2=0x7f2dbc19e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:02.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.040+0000 7f2dbb400640 1 --2- 172.21.15.45:0/568845753 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2dbc10de80 0x7f2dbc19e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.040+0000 7f2dbb400640 1 -- 172.21.15.45:0/568845753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dbc100c80 msgr2=0x7f2dbc19a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:02.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.040+0000 7f2dbb400640 1 --2- 172.21.15.45:0/568845753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dbc100c80 0x7f2dbc19a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.040+0000 7f2dbb400640 1 -- 172.21.15.45:0/568845753 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dbc19f050 con 0x7f2dbc1002d0 2024-03-28T17:51:02.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.040+0000 7f2dbb400640 1 --2- 172.21.15.45:0/568845753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2dbc1002d0 0x7f2dbc19a060 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2da4005e00 tx=0x7f2da40045d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:02.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.040+0000 7f2db3e00640 1 -- 172.21.15.45:0/568845753 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da4041020 con 0x7f2dbc1002d0 2024-03-28T17:51:02.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.040+0000 7f2db3e00640 1 -- 172.21.15.45:0/568845753 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2da402c9f0 con 0x7f2dbc1002d0 2024-03-28T17:51:02.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.040+0000 7f2db3e00640 1 -- 172.21.15.45:0/568845753 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da4039040 con 0x7f2dbc1002d0 2024-03-28T17:51:02.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.040+0000 7f2dc0e00640 1 -- 172.21.15.45:0/568845753 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dbc19f2e0 con 0x7f2dbc1002d0 2024-03-28T17:51:02.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.040+0000 7f2dc0e00640 1 -- 172.21.15.45:0/568845753 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dbc1a6bd0 con 0x7f2dbc1002d0 2024-03-28T17:51:02.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.045+0000 7f2dc519a740 1 Processor -- start 2024-03-28T17:51:02.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.045+0000 7f2dc519a740 1 -- start start 2024-03-28T17:51:02.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.045+0000 7f2db3e00640 1 -- 172.21.15.45:0/568845753 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f2da4044020 con 0x7f2dbc1002d0 2024-03-28T17:51:02.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.045+0000 7f2dc519a740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559b2fbf4dd0 0x559b2fbf3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:02.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.045+0000 7f2dc519a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fbf40f0 0x559b2fbf4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:02.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.045+0000 7f2dc519a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b2fc05c70 0x559b2fc08060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:02.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.045+0000 7f2dc519a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b2fb58250 con 0x559b2fc05c70 2024-03-28T17:51:02.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.045+0000 7f2dc519a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b2fb00a30 con 0x559b2fbf40f0 2024-03-28T17:51:02.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.045+0000 7f2dc519a740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b2fb542f0 con 0x559b2fbf4dd0 2024-03-28T17:51:02.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.046+0000 7f2dbb400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fbf40f0 0x559b2fbf4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:02.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.046+0000 7f2dbb400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fbf40f0 0x559b2fbf4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40018/0 (socket says 172.21.15.45:40018) 2024-03-28T17:51:02.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.046+0000 7f2dbb400640 1 -- 172.21.15.45:0/267731007 learned_addr learned my addr 172.21.15.45:0/267731007 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:02.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.046+0000 7f2dbaa00640 1 --2- 172.21.15.45:0/267731007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b2fc05c70 0x559b2fc08060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:02.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.046+0000 7f2dbbe00640 1 --2- 172.21.15.45:0/267731007 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559b2fbf4dd0 0x559b2fbf3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:02.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.046+0000 7f2dbb400640 1 -- 172.21.15.45:0/267731007 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559b2fbf4dd0 msgr2=0x559b2fbf3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:02.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.046+0000 7f2dbb400640 1 --2- 172.21.15.45:0/267731007 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559b2fbf4dd0 0x559b2fbf3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.046+0000 7f2dbb400640 1 -- 172.21.15.45:0/267731007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b2fc05c70 msgr2=0x559b2fc08060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:02.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.046+0000 7f2dbb400640 1 --2- 172.21.15.45:0/267731007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b2fc05c70 0x559b2fc08060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.046+0000 7f2dbb400640 1 -- 172.21.15.45:0/267731007 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b2faeb690 con 0x559b2fbf40f0 2024-03-28T17:51:02.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.046+0000 7f2dbbe00640 1 --2- 172.21.15.45:0/267731007 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559b2fbf4dd0 0x559b2fbf3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:51:02.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.046+0000 7f2dbb400640 1 --2- 172.21.15.45:0/267731007 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fbf40f0 0x559b2fbf4570 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2da40edca0 tx=0x7f2da40f6680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62d430ed26611920 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:02.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.047+0000 7f2db0c00640 1 -- 172.21.15.45:0/267731007 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da40f0e30 con 0x559b2fbf40f0 2024-03-28T17:51:02.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.047+0000 7f2db0c00640 1 -- 172.21.15.45:0/267731007 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2da40fd030 con 0x559b2fbf40f0 2024-03-28T17:51:02.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.047+0000 7f2db0c00640 1 -- 172.21.15.45:0/267731007 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da4102980 con 0x559b2fbf40f0 2024-03-28T17:51:02.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.047+0000 7f2db3e00640 1 --2- 172.21.15.45:0/568845753 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f2da00a29f0 0x7f2da00a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:02.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.047+0000 7f2db3e00640 1 -- 172.21.15.45:0/568845753 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8684+0+0 (secure 0 0 0) 0x7f2da40f0570 con 0x7f2dbc1002d0 2024-03-28T17:51:02.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.047+0000 7f2dbbe00640 1 --2- 172.21.15.45:0/568845753 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f2da00a29f0 0x7f2da00a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:02.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.047+0000 7f2dc519a740 1 -- 172.21.15.45:0/267731007 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fbf40f0 msgr2=0x559b2fbf4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:02.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.047+0000 7f2dc519a740 1 --2- 172.21.15.45:0/267731007 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fbf40f0 0x559b2fbf4570 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2da40edca0 tx=0x7f2da40f6680 comp rx=0 tx=0).stop 2024-03-28T17:51:02.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.047+0000 7f2dc519a740 1 -- 172.21.15.45:0/267731007 shutdown_connections 2024-03-28T17:51:02.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.047+0000 7f2dc519a740 1 --2- 172.21.15.45:0/267731007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b2fc05c70 0x559b2fc08060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.047+0000 7f2dc519a740 1 --2- 172.21.15.45:0/267731007 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fbf40f0 0x559b2fbf4570 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.047+0000 7f2dc519a740 1 --2- 172.21.15.45:0/267731007 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559b2fbf4dd0 0x559b2fbf3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.047+0000 7f2dc519a740 1 -- 172.21.15.45:0/267731007 >> 172.21.15.45:0/267731007 conn(0x559b2fbbb200 msgr2=0x559b2fbeeb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:02.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.048+0000 7f2dc519a740 1 -- 172.21.15.45:0/267731007 shutdown_connections 2024-03-28T17:51:02.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.048+0000 7f2dc519a740 1 -- 172.21.15.45:0/267731007 wait complete. 2024-03-28T17:51:02.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.048+0000 7f2dbbe00640 1 --2- 172.21.15.45:0/568845753 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f2da00a29f0 0x7f2da00a4eb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc0698e0 tx=0x7f2db4009260 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:02.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.048+0000 7f2dc519a740 1 -- start start 2024-03-28T17:51:02.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.048+0000 7f2dc519a740 1 Processor -- start 2024-03-28T17:51:02.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.049+0000 7f2dc519a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fbf40f0 0x559b2fc05300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:02.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.049+0000 7f2dc519a740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559b2fbf4dd0 0x559b2fc02690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:02.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.049+0000 7f2dc519a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b2fc05c70 0x559b2fc02bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:02.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.049+0000 7f2dc519a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b2fc088a0 con 0x559b2fc05c70 2024-03-28T17:51:02.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.049+0000 7f2dc519a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b2fb13e80 con 0x559b2fbf40f0 2024-03-28T17:51:02.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.049+0000 7f2dc519a740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b2fc08a20 con 0x559b2fbf4dd0 2024-03-28T17:51:02.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.049+0000 7f2dc519a740 10 client.0 _ll_register_callbacks cb 0x559b2fb26130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:51:02.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.049+0000 7f2dbb400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fbf40f0 0x559b2fc05300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:02.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.049+0000 7f2dbb400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fbf40f0 0x559b2fc05300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40026/0 (socket says 172.21.15.45:40026) 2024-03-28T17:51:02.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.049+0000 7f2dbb400640 1 -- 172.21.15.45:0/584553135 learned_addr learned my addr 172.21.15.45:0/584553135 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:02.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.049+0000 7f2dbaa00640 1 --2- 172.21.15.45:0/584553135 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559b2fbf4dd0 0x559b2fc02690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:02.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.049+0000 7f2dbbe00640 1 --2- 172.21.15.45:0/584553135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b2fc05c70 0x559b2fc02bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:02.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.050+0000 7f2dbb400640 1 -- 172.21.15.45:0/584553135 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559b2fbf4dd0 msgr2=0x559b2fc02690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:02.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.050+0000 7f2dbb400640 1 --2- 172.21.15.45:0/584553135 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559b2fbf4dd0 0x559b2fc02690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.050+0000 7f2dbb400640 1 -- 172.21.15.45:0/584553135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b2fc05c70 msgr2=0x559b2fc02bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:02.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.050+0000 7f2dbb400640 1 --2- 172.21.15.45:0/584553135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b2fc05c70 0x559b2fc02bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.050+0000 7f2dbb400640 1 -- 172.21.15.45:0/584553135 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559b2f9367d0 con 0x559b2fbf40f0 2024-03-28T17:51:02.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.050+0000 7f2dbbe00640 1 --2- 172.21.15.45:0/584553135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b2fc05c70 0x559b2fc02bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:51:02.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.050+0000 7f2dbb400640 1 --2- 172.21.15.45:0/584553135 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fbf40f0 0x559b2fc05300 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2da40b90a0 tx=0x7f2da41023e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54b2e3ac06383132 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:02.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.050+0000 7f2da9600640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:51:02.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.050+0000 7f2da9600640 1 -- 172.21.15.45:0/584553135 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da40b8ee0 con 0x559b2fbf40f0 2024-03-28T17:51:02.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.050+0000 7f2da9600640 1 -- 172.21.15.45:0/584553135 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8684+0+0 (secure 0 0 0) 0x7f2da40baf40 con 0x559b2fbf40f0 2024-03-28T17:51:02.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.050+0000 7f2dc519a740 1 -- 172.21.15.45:0/584553135 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2da40fd030 con 0x559b2fbf40f0 2024-03-28T17:51:02.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.050+0000 7f2da9600640 1 -- 172.21.15.45:0/584553135 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2da40bc5a0 con 0x559b2fbf40f0 2024-03-28T17:51:02.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.050+0000 7f2dc519a740 10 client.25925 fetch_fsmap learned FSMap version 34 2024-03-28T17:51:02.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.050+0000 7f2dc519a740 1 -- 172.21.15.45:0/584553135 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x559b2f86c3f0 con 0x559b2fbf40f0 2024-03-28T17:51:02.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.051+0000 7f2da9600640 1 -- 172.21.15.45:0/584553135 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2da40fd030 con 0x559b2fbf40f0 2024-03-28T17:51:02.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.051+0000 7f2da9600640 1 -- 172.21.15.45:0/584553135 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 34) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2da403c030 con 0x559b2fbf40f0 2024-03-28T17:51:02.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.051+0000 7f2dc519a740 10 client.25925 fetch_fsmap finished waiting for FSMap version 34 2024-03-28T17:51:02.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.051+0000 7f2dc519a740 10 client.25925 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:51:02.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.051+0000 7f2dc519a740 20 client.25925 populate_metadata read hostname 'smithi045' 2024-03-28T17:51:02.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.051+0000 7f2dc519a740 1 --2- 172.21.15.45:0/584553135 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x559b2fc15700 0x559b2fc17af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:02.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.051+0000 7f2dc519a740 4 client.25925 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:51:02.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.051+0000 7f2dc519a740 1 -- 172.21.15.45:0/584553135 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559b2fb65790 con 0x559b2fc15700 2024-03-28T17:51:02.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.051+0000 7f2dbaa00640 1 --2- 172.21.15.45:0/584553135 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x559b2fc15700 0x559b2fc17af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:02.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.052+0000 7f2dbaa00640 1 --2- 172.21.15.45:0/584553135 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x559b2fc15700 0x559b2fc17af0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a53c5289a1c6c84 server_cookie=dac199104d4d2126 in_seq=0 out_seq=0 2024-03-28T17:51:02.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.052+0000 7f2da9600640 10 client.25925 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:51:02.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.052+0000 7f2da9600640 1 -- 172.21.15.45:0/584553135 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559b2fb65790 con 0x559b2fc15700 2024-03-28T17:51:02.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.052+0000 7f2da9600640 10 client.25925 handle_command_reply: tid=0 2024-03-28T17:51:02.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.052+0000 7f2dc519a740 1 client.25925 shutdown 2024-03-28T17:51:02.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.053+0000 7f2dc519a740 1 -- 172.21.15.45:0/584553135 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fbf40f0 msgr2=0x559b2fc05300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:02.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.053+0000 7f2dc519a740 1 --2- 172.21.15.45:0/584553135 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fbf40f0 0x559b2fc05300 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2da40b90a0 tx=0x7f2da41023e0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.053+0000 7f2dc519a740 1 -- 172.21.15.45:0/584553135 shutdown_connections 2024-03-28T17:51:02.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.053+0000 7f2dc519a740 1 --2- 172.21.15.45:0/584553135 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x559b2fc15700 0x559b2fc17af0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.053+0000 7f2dc519a740 1 --2- 172.21.15.45:0/584553135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b2fc05c70 0x559b2fc02bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.053+0000 7f2dc519a740 1 --2- 172.21.15.45:0/584553135 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559b2fbf4dd0 0x559b2fc02690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.053+0000 7f2dc519a740 1 --2- 172.21.15.45:0/584553135 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fbf40f0 0x559b2fc05300 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.053+0000 7f2da9600640 0 client.25925 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:51:02.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.053+0000 7f2dc519a740 1 -- 172.21.15.45:0/584553135 >> 172.21.15.45:0/584553135 conn(0x559b2fbbb200 msgr2=0x559b2fbf0890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:02.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.053+0000 7f2dc519a740 1 -- 172.21.15.45:0/584553135 shutdown_connections 2024-03-28T17:51:02.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.053+0000 7f2dc519a740 1 -- 172.21.15.45:0/584553135 wait complete. 2024-03-28T17:51:02.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.053+0000 7f2dc519a740 20 client.25925 trim_cache size 0 max 16384 2024-03-28T17:51:02.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.071+0000 7f2dc519a740 1 Processor -- start 2024-03-28T17:51:02.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.071+0000 7f2dc519a740 1 -- start start 2024-03-28T17:51:02.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.071+0000 7f2dc519a740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559b2fbf40f0 0x559b2fc04dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:02.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.071+0000 7f2dc519a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b2fbf4dd0 0x559b2fc05300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:02.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.071+0000 7f2dc519a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fc05c70 0x559b2fc07060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:02.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.071+0000 7f2dc519a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dbc1ab5f0 con 0x559b2fbf4dd0 2024-03-28T17:51:02.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.071+0000 7f2dc519a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dbc1a9030 con 0x559b2fc05c70 2024-03-28T17:51:02.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.071+0000 7f2dc519a740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dbc1aaf90 con 0x559b2fbf40f0 2024-03-28T17:51:02.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.072+0000 7f2dbbe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fc05c70 0x559b2fc07060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:02.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.072+0000 7f2dbbe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fc05c70 0x559b2fc07060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40034/0 (socket says 172.21.15.45:40034) 2024-03-28T17:51:02.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.072+0000 7f2dbbe00640 1 -- 172.21.15.45:0/287258471 learned_addr learned my addr 172.21.15.45:0/287258471 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:02.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.072+0000 7f2dbb400640 1 --2- 172.21.15.45:0/287258471 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559b2fbf40f0 0x559b2fc04dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:02.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.072+0000 7f2dbaa00640 1 --2- 172.21.15.45:0/287258471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b2fbf4dd0 0x559b2fc05300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:02.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.072+0000 7f2dbbe00640 1 -- 172.21.15.45:0/287258471 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559b2fbf40f0 msgr2=0x559b2fc04dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:02.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.072+0000 7f2dbbe00640 1 --2- 172.21.15.45:0/287258471 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559b2fbf40f0 0x559b2fc04dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.072+0000 7f2dbbe00640 1 -- 172.21.15.45:0/287258471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b2fbf4dd0 msgr2=0x559b2fc05300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:02.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.072+0000 7f2dbbe00640 1 --2- 172.21.15.45:0/287258471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b2fbf4dd0 0x559b2fc05300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.072+0000 7f2dbbe00640 1 -- 172.21.15.45:0/287258471 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b2f82c0f0 con 0x559b2fc05c70 2024-03-28T17:51:02.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.072+0000 7f2dbbe00640 1 --2- 172.21.15.45:0/287258471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fc05c70 0x559b2fc07060 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f2db40104f0 tx=0x7f2db4015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5232f6d2d70aace8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:02.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.073+0000 7f2dabe00640 1 -- 172.21.15.45:0/287258471 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db4029670 con 0x559b2fc05c70 2024-03-28T17:51:02.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.073+0000 7f2dabe00640 1 -- 172.21.15.45:0/287258471 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2db4024070 con 0x559b2fc05c70 2024-03-28T17:51:02.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.073+0000 7f2dabe00640 1 -- 172.21.15.45:0/287258471 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db401f840 con 0x559b2fc05c70 2024-03-28T17:51:02.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.073+0000 7f2dc519a740 1 -- 172.21.15.45:0/287258471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fc05c70 msgr2=0x559b2fc07060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:02.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.073+0000 7f2dc519a740 1 --2- 172.21.15.45:0/287258471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fc05c70 0x559b2fc07060 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f2db40104f0 tx=0x7f2db4015000 comp rx=0 tx=0).stop 2024-03-28T17:51:02.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.074+0000 7f2dc519a740 1 -- 172.21.15.45:0/287258471 shutdown_connections 2024-03-28T17:51:02.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.074+0000 7f2dc519a740 1 --2- 172.21.15.45:0/287258471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fc05c70 0x559b2fc07060 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.074+0000 7f2dc519a740 1 --2- 172.21.15.45:0/287258471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b2fbf4dd0 0x559b2fc05300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.074+0000 7f2dc519a740 1 --2- 172.21.15.45:0/287258471 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559b2fbf40f0 0x559b2fc04dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.074+0000 7f2dc519a740 1 -- 172.21.15.45:0/287258471 >> 172.21.15.45:0/287258471 conn(0x559b2fbbb200 msgr2=0x559b2fbee670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:02.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.074+0000 7f2dc519a740 1 -- 172.21.15.45:0/287258471 shutdown_connections 2024-03-28T17:51:02.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.074+0000 7f2dc519a740 1 -- 172.21.15.45:0/287258471 wait complete. 2024-03-28T17:51:02.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.074+0000 7f2dc519a740 1 -- start start 2024-03-28T17:51:02.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.074+0000 7f2dc519a740 1 Processor -- start 2024-03-28T17:51:02.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.075+0000 7f2dc519a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b2fbf40f0 0x559b2fc0d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:02.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.075+0000 7f2dc519a740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559b2fbf4dd0 0x559b2fc0d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:02.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.075+0000 7f2dc519a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fc05c70 0x559b2fbff500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:02.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.075+0000 7f2dc519a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dbc1aa930 con 0x559b2fbf40f0 2024-03-28T17:51:02.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.075+0000 7f2dc519a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dbc1a9630 con 0x559b2fc05c70 2024-03-28T17:51:02.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.075+0000 7f2dc519a740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dbc1a9c70 con 0x559b2fbf4dd0 2024-03-28T17:51:02.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.075+0000 7f2dc519a740 10 client.0 _ll_register_callbacks cb 0x7f2dac009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:51:02.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.075+0000 7f2dbb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b2fbf40f0 0x559b2fc0d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:02.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.075+0000 7f2dbaa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559b2fbf4dd0 0x559b2fc0d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:02.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.075+0000 7f2dbbe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fc05c70 0x559b2fbff500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:02.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.075+0000 7f2dbaa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559b2fbf4dd0 0x559b2fc0d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:40238/0 (socket says 172.21.15.45:40238) 2024-03-28T17:51:02.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.075+0000 7f2dbaa00640 1 -- 172.21.15.45:0/3672923803 learned_addr learned my addr 172.21.15.45:0/3672923803 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:02.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.075+0000 7f2dbb400640 1 -- 172.21.15.45:0/3672923803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559b2fbf4dd0 msgr2=0x559b2fc0d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:02.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.075+0000 7f2dbb400640 1 --2- 172.21.15.45:0/3672923803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559b2fbf4dd0 0x559b2fc0d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.075+0000 7f2dbb400640 1 -- 172.21.15.45:0/3672923803 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fc05c70 msgr2=0x559b2fbff500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:02.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.075+0000 7f2dbb400640 1 --2- 172.21.15.45:0/3672923803 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fc05c70 0x559b2fbff500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.075+0000 7f2dbb400640 1 -- 172.21.15.45:0/3672923803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559b2fbffd00 con 0x559b2fbf40f0 2024-03-28T17:51:02.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.075+0000 7f2dbbe00640 1 --2- 172.21.15.45:0/3672923803 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fc05c70 0x559b2fbff500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:51:02.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.076+0000 7f2dbb400640 1 --2- 172.21.15.45:0/3672923803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b2fbf40f0 0x559b2fc0d3f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2da40ee220 tx=0x7f2da40f62d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce51a45eeab98495 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:02.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.076+0000 7f2da9600640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-28T17:51:02.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.076+0000 7f2da9600640 1 -- 172.21.15.45:0/3672923803 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da40bc780 con 0x559b2fbf40f0 2024-03-28T17:51:02.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.076+0000 7f2da9600640 1 -- 172.21.15.45:0/3672923803 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8684+0+0 (secure 0 0 0) 0x7f2da4034640 con 0x559b2fbf40f0 2024-03-28T17:51:02.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.076+0000 7f2dc519a740 1 -- 172.21.15.45:0/3672923803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2db4024070 con 0x559b2fbf40f0 2024-03-28T17:51:02.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.076+0000 7f2da9600640 1 -- 172.21.15.45:0/3672923803 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2da40bca10 con 0x559b2fbf40f0 2024-03-28T17:51:02.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.076+0000 7f2dc519a740 10 client.15759 fetch_fsmap learned FSMap version 34 2024-03-28T17:51:02.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.076+0000 7f2dc519a740 1 -- 172.21.15.45:0/3672923803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x559b2fbffd00 con 0x559b2fbf40f0 2024-03-28T17:51:02.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.076+0000 7f2da9600640 1 -- 172.21.15.45:0/3672923803 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2db4024070 con 0x559b2fbf40f0 2024-03-28T17:51:02.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.077+0000 7f2da9600640 1 -- 172.21.15.45:0/3672923803 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 34) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2da40edde0 con 0x559b2fbf40f0 2024-03-28T17:51:02.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.077+0000 7f2dc519a740 10 client.15759 fetch_fsmap finished waiting for FSMap version 34 2024-03-28T17:51:02.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.077+0000 7f2dc519a740 10 client.15759 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:51:02.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.077+0000 7f2dc519a740 20 client.15759 populate_metadata read hostname 'smithi045' 2024-03-28T17:51:02.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.077+0000 7f2dc519a740 1 --2- 172.21.15.45:0/3672923803 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x559b2fc71b20 0x559b2fc73f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:02.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.077+0000 7f2dc519a740 4 client.15759 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:51:02.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.077+0000 7f2dc519a740 1 -- 172.21.15.45:0/3672923803 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559b2fb5fae0 con 0x559b2fc71b20 2024-03-28T17:51:02.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.077+0000 7f2dbaa00640 1 --2- 172.21.15.45:0/3672923803 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x559b2fc71b20 0x559b2fc73f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:02.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.077+0000 7f2dbaa00640 1 --2- 172.21.15.45:0/3672923803 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x559b2fc71b20 0x559b2fc73f40 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=172aca0497bf0c37 server_cookie=bef929fb3484a648 in_seq=0 out_seq=0 2024-03-28T17:51:02.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.077+0000 7f2da9600640 10 client.15759 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:51:02.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.078+0000 7f2da9600640 1 -- 172.21.15.45:0/3672923803 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x559b2fb5fae0 con 0x559b2fc71b20 2024-03-28T17:51:02.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.078+0000 7f2da9600640 10 client.15759 handle_command_reply: tid=0 2024-03-28T17:51:02.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.078+0000 7f2dc519a740 1 client.15759 shutdown 2024-03-28T17:51:02.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.078+0000 7f2dc519a740 1 -- 172.21.15.45:0/3672923803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b2fbf40f0 msgr2=0x559b2fc0d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:02.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.078+0000 7f2dc519a740 1 --2- 172.21.15.45:0/3672923803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b2fbf40f0 0x559b2fc0d3f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2da40ee220 tx=0x7f2da40f62d0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.078+0000 7f2dc519a740 1 -- 172.21.15.45:0/3672923803 shutdown_connections 2024-03-28T17:51:02.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.078+0000 7f2dc519a740 1 --2- 172.21.15.45:0/3672923803 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x559b2fc71b20 0x559b2fc73f40 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.078+0000 7f2dc519a740 1 --2- 172.21.15.45:0/3672923803 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2fc05c70 0x559b2fbff500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.078+0000 7f2dc519a740 1 --2- 172.21.15.45:0/3672923803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559b2fbf4dd0 0x559b2fc0d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.078+0000 7f2dc519a740 1 --2- 172.21.15.45:0/3672923803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559b2fbf40f0 0x559b2fc0d3f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.078+0000 7f2dc519a740 1 -- 172.21.15.45:0/3672923803 >> 172.21.15.45:0/3672923803 conn(0x559b2fbbb200 msgr2=0x559b2fbed330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:02.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.078+0000 7f2da9600640 0 client.15759 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:51:02.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.078+0000 7f2dc519a740 1 -- 172.21.15.45:0/3672923803 shutdown_connections 2024-03-28T17:51:02.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.078+0000 7f2dc519a740 1 -- 172.21.15.45:0/3672923803 wait complete. 2024-03-28T17:51:02.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.078+0000 7f2dc519a740 20 client.15759 trim_cache size 0 max 16384 2024-03-28T17:51:02.083 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:51:02.083 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "scrub active (3 inodes in the stack)", 2024-03-28T17:51:02.083 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T17:51:02.083 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T17:51:02.083 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T17:51:02.083 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T17:51:02.083 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T17:51:02.083 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:51:02.083 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:51:02.083 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:51:02.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.079+0000 7f2dc0e00640 1 -- 172.21.15.45:0/568845753 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f2da00a29f0 msgr2=0x7f2da00a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:02.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.079+0000 7f2dc0e00640 1 --2- 172.21.15.45:0/568845753 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f2da00a29f0 0x7f2da00a4eb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc0698e0 tx=0x7f2db4009260 comp rx=0 tx=0).stop 2024-03-28T17:51:02.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.079+0000 7f2dc0e00640 1 -- 172.21.15.45:0/568845753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2dbc1002d0 msgr2=0x7f2dbc19a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:02.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.079+0000 7f2dc0e00640 1 --2- 172.21.15.45:0/568845753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2dbc1002d0 0x7f2dbc19a060 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2da4005e00 tx=0x7f2da40045d0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.079+0000 7f2dc0e00640 1 -- 172.21.15.45:0/568845753 shutdown_connections 2024-03-28T17:51:02.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.079+0000 7f2dc0e00640 1 --2- 172.21.15.45:0/568845753 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f2da00a29f0 0x7f2da00a4eb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.079+0000 7f2dc0e00640 1 --2- 172.21.15.45:0/568845753 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2dbc10de80 0x7f2dbc19e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.079+0000 7f2dc0e00640 1 --2- 172.21.15.45:0/568845753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dbc100c80 0x7f2dbc19a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.079+0000 7f2dc0e00640 1 --2- 172.21.15.45:0/568845753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2dbc1002d0 0x7f2dbc19a060 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:02.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.079+0000 7f2dc0e00640 1 -- 172.21.15.45:0/568845753 >> 172.21.15.45:0/568845753 conn(0x7f2dbc0fbef0 msgr2=0x7f2dbc0fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:02.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.079+0000 7f2dc0e00640 1 -- 172.21.15.45:0/568845753 shutdown_connections 2024-03-28T17:51:02.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:02.079+0000 7f2dc0e00640 1 -- 172.21.15.45:0/568845753 wait complete. 2024-03-28T17:51:02.097 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T17:51:02.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:02 smithi045 ceph-mon[31773]: pgmap v66: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-28T17:51:02.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:02 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] up:resolve 2024-03-28T17:51:02.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:02 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:02.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:02 smithi045 ceph-mon[31773]: scrub queued for path: / 2024-03-28T17:51:02.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:02 smithi045 ceph-mon[31773]: scrub summary: idle+waiting paths [/] 2024-03-28T17:51:02.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:02 smithi045 ceph-mon[31773]: scrub summary: active paths [/] 2024-03-28T17:51:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:02 smithi071 ceph-mon[36597]: pgmap v66: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-28T17:51:02.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:02 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] up:resolve 2024-03-28T17:51:02.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:02 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:02.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:02 smithi071 ceph-mon[36597]: scrub queued for path: / 2024-03-28T17:51:02.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:02 smithi071 ceph-mon[36597]: scrub summary: idle+waiting paths [/] 2024-03-28T17:51:02.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:02 smithi071 ceph-mon[36597]: scrub summary: active paths [/] 2024-03-28T17:51:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:02 smithi151 ceph-mon[36481]: pgmap v66: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-28T17:51:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:02 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] up:resolve 2024-03-28T17:51:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:02 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:02 smithi151 ceph-mon[36481]: scrub queued for path: / 2024-03-28T17:51:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:02 smithi151 ceph-mon[36481]: scrub summary: idle+waiting paths [/] 2024-03-28T17:51:02.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:02 smithi151 ceph-mon[36481]: scrub summary: active paths [/] 2024-03-28T17:51:02.836 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.69 MB/sec warmup 35 sec latency 30733.366 ms 2024-03-28T17:51:03.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:03 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.000999988 2024-03-28T17:51:03.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:03 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] up:reconnect 2024-03-28T17:51:03.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:03 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:03.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:03 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999988 2024-03-28T17:51:03.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:03 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.000999988 2024-03-28T17:51:03.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:03 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] up:reconnect 2024-03-28T17:51:03.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:03 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:03.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:03 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999988 2024-03-28T17:51:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:03 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.000999988 2024-03-28T17:51:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:03 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] up:reconnect 2024-03-28T17:51:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:03 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:03 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999988 2024-03-28T17:51:03.836 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.64 MB/sec warmup 36 sec latency 31733.514 ms 2024-03-28T17:51:04.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:04 smithi045 ceph-mon[31773]: pgmap v67: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-28T17:51:04.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:04 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] up:rejoin 2024-03-28T17:51:04.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:04 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:04.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:04 smithi045 ceph-mon[31773]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-28T17:51:04.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:04 smithi071 ceph-mon[36597]: pgmap v67: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-28T17:51:04.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:04 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] up:rejoin 2024-03-28T17:51:04.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:04 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:04.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:04 smithi071 ceph-mon[36597]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-28T17:51:04.525 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-mon[36481]: pgmap v67: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-28T17:51:04.526 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] up:rejoin 2024-03-28T17:51:04.526 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:04.526 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-mon[36481]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-28T17:51:04.526 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fbf4aa47700 time 2024-03-28T17:51:04.148525+0000 2024-03-28T17:51:04.526 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:51:04.526 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:04.526 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fbf550714a1] 2024-03-28T17:51:04.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fbf55071667] 2024-03-28T17:51:04.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56060a0c4223] 2024-03-28T17:51:04.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560609f3e5d6] 2024-03-28T17:51:04.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560609f3eda1] 2024-03-28T17:51:04.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560609f412d5] 2024-03-28T17:51:04.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560609f91233] 2024-03-28T17:51:04.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560609f93197] 2024-03-28T17:51:04.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560609e9211b] 2024-03-28T17:51:04.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560609e926bc] 2024-03-28T17:51:04.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560609e7c1ff] 2024-03-28T17:51:04.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fbf55303488] 2024-03-28T17:51:04.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 13: (DispatchQueue::entry()+0x51f) [0x7fbf5530060f] 2024-03-28T17:51:04.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbf553d3e81] 2024-03-28T17:51:04.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fbf53dee1ca] 2024-03-28T17:51:04.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 16: clone() 2024-03-28T17:51:04.528 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 2024-03-28T17:51:04.149+0000 7fbf4aa47700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fbf4aa47700 time 2024-03-28T17:51:04.148525+0000 2024-03-28T17:51:04.528 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:51:04.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 2024-03-28T17:51:04.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:04.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fbf550714a1] 2024-03-28T17:51:04.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fbf55071667] 2024-03-28T17:51:04.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56060a0c4223] 2024-03-28T17:51:04.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560609f3e5d6] 2024-03-28T17:51:04.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560609f3eda1] 2024-03-28T17:51:04.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560609f412d5] 2024-03-28T17:51:04.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560609f91233] 2024-03-28T17:51:04.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560609f93197] 2024-03-28T17:51:04.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560609e9211b] 2024-03-28T17:51:04.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560609e926bc] 2024-03-28T17:51:04.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560609e7c1ff] 2024-03-28T17:51:04.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fbf55303488] 2024-03-28T17:51:04.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 13: (DispatchQueue::entry()+0x51f) [0x7fbf5530060f] 2024-03-28T17:51:04.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbf553d3e81] 2024-03-28T17:51:04.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fbf53dee1ca] 2024-03-28T17:51:04.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 16: clone() 2024-03-28T17:51:04.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 2024-03-28T17:51:04.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: *** Caught signal (Aborted) ** 2024-03-28T17:51:04.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: in thread 7fbf4aa47700 thread_name:ms_dispatch 2024-03-28T17:51:04.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:04.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbf53df8d20] 2024-03-28T17:51:04.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 2: gsignal() 2024-03-28T17:51:04.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 3: abort() 2024-03-28T17:51:04.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fbf550714fb] 2024-03-28T17:51:04.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fbf55071667] 2024-03-28T17:51:04.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56060a0c4223] 2024-03-28T17:51:04.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560609f3e5d6] 2024-03-28T17:51:04.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560609f3eda1] 2024-03-28T17:51:04.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560609f412d5] 2024-03-28T17:51:04.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560609f91233] 2024-03-28T17:51:04.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560609f93197] 2024-03-28T17:51:04.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560609e9211b] 2024-03-28T17:51:04.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560609e926bc] 2024-03-28T17:51:04.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560609e7c1ff] 2024-03-28T17:51:04.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fbf55303488] 2024-03-28T17:51:04.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 16: (DispatchQueue::entry()+0x51f) [0x7fbf5530060f] 2024-03-28T17:51:04.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbf553d3e81] 2024-03-28T17:51:04.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fbf53dee1ca] 2024-03-28T17:51:04.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 19: clone() 2024-03-28T17:51:04.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 2024-03-28T17:51:04.150+0000 7fbf4aa47700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:51:04.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: in thread 7fbf4aa47700 thread_name:ms_dispatch 2024-03-28T17:51:04.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 2024-03-28T17:51:04.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:04.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbf53df8d20] 2024-03-28T17:51:04.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 2: gsignal() 2024-03-28T17:51:04.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 3: abort() 2024-03-28T17:51:04.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fbf550714fb] 2024-03-28T17:51:04.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fbf55071667] 2024-03-28T17:51:04.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56060a0c4223] 2024-03-28T17:51:04.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560609f3e5d6] 2024-03-28T17:51:04.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560609f3eda1] 2024-03-28T17:51:04.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560609f412d5] 2024-03-28T17:51:04.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560609f91233] 2024-03-28T17:51:04.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560609f93197] 2024-03-28T17:51:04.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560609e9211b] 2024-03-28T17:51:04.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560609e926bc] 2024-03-28T17:51:04.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560609e7c1ff] 2024-03-28T17:51:04.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fbf55303488] 2024-03-28T17:51:04.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 16: (DispatchQueue::entry()+0x51f) [0x7fbf5530060f] 2024-03-28T17:51:04.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbf553d3e81] 2024-03-28T17:51:04.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fbf53dee1ca] 2024-03-28T17:51:04.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 19: clone() 2024-03-28T17:51:04.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:51:04.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 2024-03-28T17:51:04.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: -1> 2024-03-28T17:51:04.149+0000 7fbf4aa47700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fbf4aa47700 time 2024-03-28T17:51:04.148525+0000 2024-03-28T17:51:04.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:51:04.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 2024-03-28T17:51:04.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:04.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fbf550714a1] 2024-03-28T17:51:04.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fbf55071667] 2024-03-28T17:51:04.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56060a0c4223] 2024-03-28T17:51:04.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560609f3e5d6] 2024-03-28T17:51:04.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560609f3eda1] 2024-03-28T17:51:04.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560609f412d5] 2024-03-28T17:51:04.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560609f91233] 2024-03-28T17:51:04.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560609f93197] 2024-03-28T17:51:04.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560609e9211b] 2024-03-28T17:51:04.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560609e926bc] 2024-03-28T17:51:04.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560609e7c1ff] 2024-03-28T17:51:04.535 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fbf55303488] 2024-03-28T17:51:04.535 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 13: (DispatchQueue::entry()+0x51f) [0x7fbf5530060f] 2024-03-28T17:51:04.535 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbf553d3e81] 2024-03-28T17:51:04.535 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fbf53dee1ca] 2024-03-28T17:51:04.535 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 16: clone() 2024-03-28T17:51:04.535 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 2024-03-28T17:51:04.535 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 0> 2024-03-28T17:51:04.150+0000 7fbf4aa47700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:51:04.535 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: in thread 7fbf4aa47700 thread_name:ms_dispatch 2024-03-28T17:51:04.535 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 2024-03-28T17:51:04.535 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:04.535 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbf53df8d20] 2024-03-28T17:51:04.535 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 2: gsignal() 2024-03-28T17:51:04.535 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 3: abort() 2024-03-28T17:51:04.535 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fbf550714fb] 2024-03-28T17:51:04.535 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fbf55071667] 2024-03-28T17:51:04.536 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56060a0c4223] 2024-03-28T17:51:04.536 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560609f3e5d6] 2024-03-28T17:51:04.536 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560609f3eda1] 2024-03-28T17:51:04.536 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560609f412d5] 2024-03-28T17:51:04.536 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560609f91233] 2024-03-28T17:51:04.536 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560609f93197] 2024-03-28T17:51:04.537 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560609e9211b] 2024-03-28T17:51:04.537 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560609e926bc] 2024-03-28T17:51:04.537 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560609e7c1ff] 2024-03-28T17:51:04.537 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fbf55303488] 2024-03-28T17:51:04.537 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 16: (DispatchQueue::entry()+0x51f) [0x7fbf5530060f] 2024-03-28T17:51:04.537 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbf553d3e81] 2024-03-28T17:51:04.537 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fbf53dee1ca] 2024-03-28T17:51:04.537 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 19: clone() 2024-03-28T17:51:04.537 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:51:04.537 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 2024-03-28T17:51:04.537 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: -1> 2024-03-28T17:51:04.149+0000 7fbf4aa47700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fbf4aa47700 time 2024-03-28T17:51:04.148525+0000 2024-03-28T17:51:04.537 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:51:04.537 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 2024-03-28T17:51:04.537 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:04.538 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fbf550714a1] 2024-03-28T17:51:04.538 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fbf55071667] 2024-03-28T17:51:04.538 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56060a0c4223] 2024-03-28T17:51:04.538 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560609f3e5d6] 2024-03-28T17:51:04.538 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560609f3eda1] 2024-03-28T17:51:04.538 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560609f412d5] 2024-03-28T17:51:04.538 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560609f91233] 2024-03-28T17:51:04.538 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560609f93197] 2024-03-28T17:51:04.538 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560609e9211b] 2024-03-28T17:51:04.538 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560609e926bc] 2024-03-28T17:51:04.538 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560609e7c1ff] 2024-03-28T17:51:04.538 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fbf55303488] 2024-03-28T17:51:04.538 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 13: (DispatchQueue::entry()+0x51f) [0x7fbf5530060f] 2024-03-28T17:51:04.539 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbf553d3e81] 2024-03-28T17:51:04.539 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fbf53dee1ca] 2024-03-28T17:51:04.539 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 16: clone() 2024-03-28T17:51:04.539 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 2024-03-28T17:51:04.539 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 0> 2024-03-28T17:51:04.150+0000 7fbf4aa47700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:51:04.539 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: in thread 7fbf4aa47700 thread_name:ms_dispatch 2024-03-28T17:51:04.539 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 2024-03-28T17:51:04.539 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:04.539 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbf53df8d20] 2024-03-28T17:51:04.539 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 2: gsignal() 2024-03-28T17:51:04.539 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 3: abort() 2024-03-28T17:51:04.539 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fbf550714fb] 2024-03-28T17:51:04.539 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fbf55071667] 2024-03-28T17:51:04.540 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56060a0c4223] 2024-03-28T17:51:04.540 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560609f3e5d6] 2024-03-28T17:51:04.540 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560609f3eda1] 2024-03-28T17:51:04.540 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560609f412d5] 2024-03-28T17:51:04.540 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560609f91233] 2024-03-28T17:51:04.540 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560609f93197] 2024-03-28T17:51:04.540 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560609e9211b] 2024-03-28T17:51:04.540 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560609e926bc] 2024-03-28T17:51:04.540 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560609e7c1ff] 2024-03-28T17:51:04.540 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fbf55303488] 2024-03-28T17:51:04.540 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 16: (DispatchQueue::entry()+0x51f) [0x7fbf5530060f] 2024-03-28T17:51:04.540 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbf553d3e81] 2024-03-28T17:51:04.540 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fbf53dee1ca] 2024-03-28T17:51:04.540 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 19: clone() 2024-03-28T17:51:04.541 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:51:04.541 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[70916]: 2024-03-28T17:51:04.541 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 podman[71121]: 2024-03-28 17:51:04.201295939 +0000 UTC m=+0.020082058 container died 267fa701c8e13123b6c2a3a9fa798643d5d80813beb088a077af7c6a89152df4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-28T17:51:04.837 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.60 MB/sec warmup 37 sec latency 32733.609 ms 2024-03-28T17:51:05.323 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 podman[71121]: 2024-03-28 17:51:04.961662446 +0000 UTC m=+0.780448576 container remove 267fa701c8e13123b6c2a3a9fa798643d5d80813beb088a077af7c6a89152df4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-28T17:51:05.323 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:04 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:51:05.323 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:05 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.i.service: Failed with result 'exit-code'. 2024-03-28T17:51:05.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:05 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:51:05.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:05 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:51:05.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:05 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] up:active 2024-03-28T17:51:05.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:05 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:05.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:05 smithi151 ceph-mon[36481]: scrub summary: idle+waiting paths [/] 2024-03-28T17:51:05.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:05 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:51:05.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:05 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:51:05.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:05 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:51:05.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:05 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] up:active 2024-03-28T17:51:05.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:05 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:05.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:05 smithi045 ceph-mon[31773]: scrub summary: idle+waiting paths [/] 2024-03-28T17:51:05.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:05 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:51:05.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:05 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:51:05.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:05 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:51:05.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:05 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6838/1272148386,v1:172.21.15.151:6839/1272148386] up:active 2024-03-28T17:51:05.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:05 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:05.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:05 smithi071 ceph-mon[36597]: scrub summary: idle+waiting paths [/] 2024-03-28T17:51:05.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:05 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:51:05.837 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.56 MB/sec warmup 38 sec latency 33733.784 ms 2024-03-28T17:51:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:06 smithi045 ceph-mon[31773]: pgmap v68: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 30 op/s 2024-03-28T17:51:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:06 smithi071 ceph-mon[36597]: pgmap v68: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 30 op/s 2024-03-28T17:51:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:06 smithi151 ceph-mon[36481]: pgmap v68: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 30 op/s 2024-03-28T17:51:06.837 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.52 MB/sec warmup 39 sec latency 34733.921 ms 2024-03-28T17:51:07.837 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.48 MB/sec warmup 40 sec latency 35734.023 ms 2024-03-28T17:51:08.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:08 smithi045 ceph-mon[31773]: pgmap v69: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 30 op/s 2024-03-28T17:51:08.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:08 smithi071 ceph-mon[36597]: pgmap v69: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 30 op/s 2024-03-28T17:51:08.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:08 smithi151 ceph-mon[36481]: pgmap v69: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 30 op/s 2024-03-28T17:51:08.837 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.44 MB/sec warmup 41 sec latency 36734.139 ms 2024-03-28T17:51:09.837 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.41 MB/sec warmup 42 sec latency 37734.242 ms 2024-03-28T17:51:10.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:10 smithi045 ceph-mon[31773]: pgmap v70: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 30 op/s 2024-03-28T17:51:10.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:10 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:51:10.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:10 smithi071 ceph-mon[36597]: pgmap v70: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 30 op/s 2024-03-28T17:51:10.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:10 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:51:10.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:10 smithi151 ceph-mon[36481]: pgmap v70: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 30 op/s 2024-03-28T17:51:10.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:10 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:51:10.837 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.38 MB/sec warmup 43 sec latency 38734.359 ms 2024-03-28T17:51:11.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:11 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:51:11.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:11 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:51:11.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:11 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:51:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:11 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:51:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:11 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:51:11.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:11 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:51:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:11 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:51:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:11 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:51:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:11 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:51:11.837 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.35 MB/sec warmup 44 sec latency 39734.493 ms 2024-03-28T17:51:12.838 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.32 MB/sec warmup 45 sec latency 40734.651 ms 2024-03-28T17:51:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:12 smithi071 ceph-mon[36597]: pgmap v71: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 30 op/s 2024-03-28T17:51:13.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:12 smithi151 ceph-mon[36481]: pgmap v71: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 30 op/s 2024-03-28T17:51:13.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:12 smithi045 ceph-mon[31773]: pgmap v71: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 30 op/s 2024-03-28T17:51:13.838 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.29 MB/sec warmup 46 sec latency 41734.829 ms 2024-03-28T17:51:14.838 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.26 MB/sec warmup 47 sec latency 42734.995 ms 2024-03-28T17:51:14.969 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:14 smithi045 ceph-mon[31773]: pgmap v72: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-03-28T17:51:15.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:14 smithi071 ceph-mon[36597]: pgmap v72: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-03-28T17:51:15.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:14 smithi151 ceph-mon[36481]: pgmap v72: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-03-28T17:51:15.578 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:15 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.i.service: Scheduled restart job, restart counter is at 2. 2024-03-28T17:51:15.578 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:15 smithi151 systemd[1]: Stopped Ceph mds.i for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:51:15.579 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:15 smithi151 systemd[1]: Starting Ceph mds.i for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:51:15.579 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:15 smithi151 podman[71353]: 2024-03-28 17:51:15.464102135 +0000 UTC m=+0.018236627 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:51:15.831 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:15 smithi151 podman[71353]: 2024-03-28 17:51:15.579084971 +0000 UTC m=+0.133219444 container create d9cc3c2eb7c0046b41dc8d65b12364f29bca93fed4be29ce2ac8e018a7bb22cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T17:51:15.838 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.23 MB/sec warmup 48 sec latency 43735.153 ms 2024-03-28T17:51:16.218 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:15 smithi151 podman[71353]: 2024-03-28 17:51:15.831551639 +0000 UTC m=+0.385686122 container init d9cc3c2eb7c0046b41dc8d65b12364f29bca93fed4be29ce2ac8e018a7bb22cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T17:51:16.218 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:15 smithi151 podman[71353]: 2024-03-28 17:51:15.834300077 +0000 UTC m=+0.388434560 container start d9cc3c2eb7c0046b41dc8d65b12364f29bca93fed4be29ce2ac8e018a7bb22cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T17:51:16.218 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:15 smithi151 ceph-mds[71372]: -- 172.21.15.151:0/3180374122 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563684bdda40 con 0x563684bb0c00 2024-03-28T17:51:16.218 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: starting mds.i at 2024-03-28T17:51:16.218 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:15 smithi151 bash[71353]: d9cc3c2eb7c0046b41dc8d65b12364f29bca93fed4be29ce2ac8e018a7bb22cb 2024-03-28T17:51:16.219 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:15 smithi151 systemd[1]: Started Ceph mds.i for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:51:16.838 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.21 MB/sec warmup 49 sec latency 44735.291 ms 2024-03-28T17:51:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:16 smithi071 ceph-mon[36597]: pgmap v73: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-03-28T17:51:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:16 smithi071 ceph-mon[36597]: daemon mds.i restarted 2024-03-28T17:51:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:16 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:51:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:16 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T17:51:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:16 smithi071 ceph-mon[36597]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T17:51:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:16 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6838/4005741328,v1:172.21.15.151:6839/4005741328] up:boot 2024-03-28T17:51:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:16 smithi071 ceph-mon[36597]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-03-28T17:51:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:16 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:51:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:16 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:16 smithi151 ceph-mon[36481]: pgmap v73: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-03-28T17:51:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:16 smithi151 ceph-mon[36481]: daemon mds.i restarted 2024-03-28T17:51:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:16 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:51:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:16 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T17:51:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:16 smithi151 ceph-mon[36481]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T17:51:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:16 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6838/4005741328,v1:172.21.15.151:6839/4005741328] up:boot 2024-03-28T17:51:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:16 smithi151 ceph-mon[36481]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-03-28T17:51:17.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:16 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:51:17.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:16 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:16 smithi045 ceph-mon[31773]: pgmap v73: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-03-28T17:51:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:16 smithi045 ceph-mon[31773]: daemon mds.i restarted 2024-03-28T17:51:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:16 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:51:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:16 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T17:51:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:16 smithi045 ceph-mon[31773]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T17:51:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:16 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6838/4005741328,v1:172.21.15.151:6839/4005741328] up:boot 2024-03-28T17:51:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:16 smithi045 ceph-mon[31773]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-03-28T17:51:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:16 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:51:17.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:16 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:17.838 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.18 MB/sec warmup 50 sec latency 45735.408 ms 2024-03-28T17:51:18.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:17 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/1934350429,v1:172.21.15.151:6835/1934350429] up:resolve 2024-03-28T17:51:18.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:17 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:18.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:17 smithi045 ceph-mon[31773]: pgmap v75: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:51:18.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:17 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/1934350429,v1:172.21.15.151:6835/1934350429] up:resolve 2024-03-28T17:51:18.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:17 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:18.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:17 smithi071 ceph-mon[36597]: pgmap v75: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:51:18.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:17 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/1934350429,v1:172.21.15.151:6835/1934350429] up:resolve 2024-03-28T17:51:18.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:17 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:18.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:17 smithi151 ceph-mon[36481]: pgmap v75: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:51:18.838 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.16 MB/sec warmup 51 sec latency 46735.557 ms 2024-03-28T17:51:19.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:18 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299996 2024-03-28T17:51:19.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:18 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/1934350429,v1:172.21.15.151:6835/1934350429] up:reconnect 2024-03-28T17:51:19.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:18 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:19.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:18 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:51:19.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:18 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299996 2024-03-28T17:51:19.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:18 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/1934350429,v1:172.21.15.151:6835/1934350429] up:reconnect 2024-03-28T17:51:19.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:18 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:19.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:18 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:51:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:18 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299996 2024-03-28T17:51:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:18 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/1934350429,v1:172.21.15.151:6835/1934350429] up:reconnect 2024-03-28T17:51:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:18 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:18 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:51:19.839 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.14 MB/sec warmup 52 sec latency 47735.729 ms 2024-03-28T17:51:20.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:19 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/1934350429,v1:172.21.15.151:6835/1934350429] up:rejoin 2024-03-28T17:51:20.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:19 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:20.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:19 smithi071 ceph-mon[36597]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-28T17:51:20.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:19 smithi071 ceph-mon[36597]: pgmap v76: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:51:20.318 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/1934350429,v1:172.21.15.151:6835/1934350429] up:rejoin 2024-03-28T17:51:20.318 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:20.318 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-mon[36481]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-28T17:51:20.319 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-mon[36481]: pgmap v76: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:51:20.319 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f77795d7700 time 2024-03-28T17:51:19.941957+0000 2024-03-28T17:51:20.319 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:51:20.319 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:20.320 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7783c014a1] 2024-03-28T17:51:20.320 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7783c01667] 2024-03-28T17:51:20.320 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e9b45dc223] 2024-03-28T17:51:20.320 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e9b44565d6] 2024-03-28T17:51:20.320 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e9b4456da1] 2024-03-28T17:51:20.320 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e9b44592d5] 2024-03-28T17:51:20.320 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e9b44a9233] 2024-03-28T17:51:20.320 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e9b44ab197] 2024-03-28T17:51:20.320 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e9b43aa11b] 2024-03-28T17:51:20.320 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e9b43aa6bc] 2024-03-28T17:51:20.320 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e9b43941ff] 2024-03-28T17:51:20.320 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7783e93488] 2024-03-28T17:51:20.320 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 13: (DispatchQueue::entry()+0x51f) [0x7f7783e9060f] 2024-03-28T17:51:20.320 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7783f63e81] 2024-03-28T17:51:20.321 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f778297e1ca] 2024-03-28T17:51:20.321 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 16: clone() 2024-03-28T17:51:20.321 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 2024-03-28T17:51:19.942+0000 7f77795d7700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f77795d7700 time 2024-03-28T17:51:19.941957+0000 2024-03-28T17:51:20.321 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:51:20.321 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 2024-03-28T17:51:20.321 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:20.321 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7783c014a1] 2024-03-28T17:51:20.321 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7783c01667] 2024-03-28T17:51:20.321 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e9b45dc223] 2024-03-28T17:51:20.321 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e9b44565d6] 2024-03-28T17:51:20.321 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e9b4456da1] 2024-03-28T17:51:20.321 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e9b44592d5] 2024-03-28T17:51:20.321 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e9b44a9233] 2024-03-28T17:51:20.321 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e9b44ab197] 2024-03-28T17:51:20.322 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e9b43aa11b] 2024-03-28T17:51:20.322 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e9b43aa6bc] 2024-03-28T17:51:20.322 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e9b43941ff] 2024-03-28T17:51:20.322 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7783e93488] 2024-03-28T17:51:20.322 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 13: (DispatchQueue::entry()+0x51f) [0x7f7783e9060f] 2024-03-28T17:51:20.322 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7783f63e81] 2024-03-28T17:51:20.322 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f778297e1ca] 2024-03-28T17:51:20.322 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 16: clone() 2024-03-28T17:51:20.322 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 2024-03-28T17:51:20.322 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: *** Caught signal (Aborted) ** 2024-03-28T17:51:20.322 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: in thread 7f77795d7700 thread_name:ms_dispatch 2024-03-28T17:51:20.322 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:20.322 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7782988d20] 2024-03-28T17:51:20.323 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 2: gsignal() 2024-03-28T17:51:20.323 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 3: abort() 2024-03-28T17:51:20.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7783c014fb] 2024-03-28T17:51:20.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7783c01667] 2024-03-28T17:51:20.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e9b45dc223] 2024-03-28T17:51:20.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e9b44565d6] 2024-03-28T17:51:20.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e9b4456da1] 2024-03-28T17:51:20.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e9b44592d5] 2024-03-28T17:51:20.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e9b44a9233] 2024-03-28T17:51:20.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e9b44ab197] 2024-03-28T17:51:20.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e9b43aa11b] 2024-03-28T17:51:20.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e9b43aa6bc] 2024-03-28T17:51:20.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e9b43941ff] 2024-03-28T17:51:20.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7783e93488] 2024-03-28T17:51:20.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 16: (DispatchQueue::entry()+0x51f) [0x7f7783e9060f] 2024-03-28T17:51:20.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7783f63e81] 2024-03-28T17:51:20.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f778297e1ca] 2024-03-28T17:51:20.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 19: clone() 2024-03-28T17:51:20.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 2024-03-28T17:51:19.944+0000 7f77795d7700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:51:20.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: in thread 7f77795d7700 thread_name:ms_dispatch 2024-03-28T17:51:20.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 2024-03-28T17:51:20.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:20.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7782988d20] 2024-03-28T17:51:20.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 2: gsignal() 2024-03-28T17:51:20.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 3: abort() 2024-03-28T17:51:20.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7783c014fb] 2024-03-28T17:51:20.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7783c01667] 2024-03-28T17:51:20.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e9b45dc223] 2024-03-28T17:51:20.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e9b44565d6] 2024-03-28T17:51:20.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e9b4456da1] 2024-03-28T17:51:20.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e9b44592d5] 2024-03-28T17:51:20.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e9b44a9233] 2024-03-28T17:51:20.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e9b44ab197] 2024-03-28T17:51:20.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e9b43aa11b] 2024-03-28T17:51:20.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e9b43aa6bc] 2024-03-28T17:51:20.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e9b43941ff] 2024-03-28T17:51:20.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7783e93488] 2024-03-28T17:51:20.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 16: (DispatchQueue::entry()+0x51f) [0x7f7783e9060f] 2024-03-28T17:51:20.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7783f63e81] 2024-03-28T17:51:20.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f778297e1ca] 2024-03-28T17:51:20.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 19: clone() 2024-03-28T17:51:20.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:51:20.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 2024-03-28T17:51:20.326 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: -1> 2024-03-28T17:51:19.942+0000 7f77795d7700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f77795d7700 time 2024-03-28T17:51:19.941957+0000 2024-03-28T17:51:20.327 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:51:20.327 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 2024-03-28T17:51:20.327 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:20.327 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7783c014a1] 2024-03-28T17:51:20.327 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7783c01667] 2024-03-28T17:51:20.327 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e9b45dc223] 2024-03-28T17:51:20.327 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e9b44565d6] 2024-03-28T17:51:20.327 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e9b4456da1] 2024-03-28T17:51:20.327 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e9b44592d5] 2024-03-28T17:51:20.327 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e9b44a9233] 2024-03-28T17:51:20.327 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e9b44ab197] 2024-03-28T17:51:20.327 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e9b43aa11b] 2024-03-28T17:51:20.327 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e9b43aa6bc] 2024-03-28T17:51:20.328 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e9b43941ff] 2024-03-28T17:51:20.328 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7783e93488] 2024-03-28T17:51:20.328 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 13: (DispatchQueue::entry()+0x51f) [0x7f7783e9060f] 2024-03-28T17:51:20.328 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7783f63e81] 2024-03-28T17:51:20.328 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f778297e1ca] 2024-03-28T17:51:20.328 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 16: clone() 2024-03-28T17:51:20.328 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 2024-03-28T17:51:20.328 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 0> 2024-03-28T17:51:19.944+0000 7f77795d7700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:51:20.328 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: in thread 7f77795d7700 thread_name:ms_dispatch 2024-03-28T17:51:20.328 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 2024-03-28T17:51:20.328 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:20.328 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7782988d20] 2024-03-28T17:51:20.328 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 2: gsignal() 2024-03-28T17:51:20.328 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 3: abort() 2024-03-28T17:51:20.328 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7783c014fb] 2024-03-28T17:51:20.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7783c01667] 2024-03-28T17:51:20.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e9b45dc223] 2024-03-28T17:51:20.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e9b44565d6] 2024-03-28T17:51:20.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e9b4456da1] 2024-03-28T17:51:20.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e9b44592d5] 2024-03-28T17:51:20.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e9b44a9233] 2024-03-28T17:51:20.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e9b44ab197] 2024-03-28T17:51:20.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e9b43aa11b] 2024-03-28T17:51:20.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e9b43aa6bc] 2024-03-28T17:51:20.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e9b43941ff] 2024-03-28T17:51:20.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7783e93488] 2024-03-28T17:51:20.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 16: (DispatchQueue::entry()+0x51f) [0x7f7783e9060f] 2024-03-28T17:51:20.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7783f63e81] 2024-03-28T17:51:20.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f778297e1ca] 2024-03-28T17:51:20.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 19: clone() 2024-03-28T17:51:20.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:51:20.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 2024-03-28T17:51:20.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: -1> 2024-03-28T17:51:19.942+0000 7f77795d7700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f77795d7700 time 2024-03-28T17:51:19.941957+0000 2024-03-28T17:51:20.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:51:20.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 2024-03-28T17:51:20.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:20.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7783c014a1] 2024-03-28T17:51:20.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7783c01667] 2024-03-28T17:51:20.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e9b45dc223] 2024-03-28T17:51:20.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e9b44565d6] 2024-03-28T17:51:20.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e9b4456da1] 2024-03-28T17:51:20.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e9b44592d5] 2024-03-28T17:51:20.330 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e9b44a9233] 2024-03-28T17:51:20.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e9b44ab197] 2024-03-28T17:51:20.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e9b43aa11b] 2024-03-28T17:51:20.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e9b43aa6bc] 2024-03-28T17:51:20.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e9b43941ff] 2024-03-28T17:51:20.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7783e93488] 2024-03-28T17:51:20.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 13: (DispatchQueue::entry()+0x51f) [0x7f7783e9060f] 2024-03-28T17:51:20.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7783f63e81] 2024-03-28T17:51:20.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f778297e1ca] 2024-03-28T17:51:20.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 16: clone() 2024-03-28T17:51:20.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 2024-03-28T17:51:20.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 0> 2024-03-28T17:51:19.944+0000 7f77795d7700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:51:20.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: in thread 7f77795d7700 thread_name:ms_dispatch 2024-03-28T17:51:20.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 2024-03-28T17:51:20.331 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:20.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7782988d20] 2024-03-28T17:51:20.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 2: gsignal() 2024-03-28T17:51:20.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 3: abort() 2024-03-28T17:51:20.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7783c014fb] 2024-03-28T17:51:20.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7783c01667] 2024-03-28T17:51:20.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e9b45dc223] 2024-03-28T17:51:20.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e9b44565d6] 2024-03-28T17:51:20.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e9b4456da1] 2024-03-28T17:51:20.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e9b44592d5] 2024-03-28T17:51:20.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e9b44a9233] 2024-03-28T17:51:20.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e9b44ab197] 2024-03-28T17:51:20.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e9b43aa11b] 2024-03-28T17:51:20.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e9b43aa6bc] 2024-03-28T17:51:20.332 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e9b43941ff] 2024-03-28T17:51:20.333 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7783e93488] 2024-03-28T17:51:20.333 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 16: (DispatchQueue::entry()+0x51f) [0x7f7783e9060f] 2024-03-28T17:51:20.333 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7783f63e81] 2024-03-28T17:51:20.333 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f778297e1ca] 2024-03-28T17:51:20.333 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 19: clone() 2024-03-28T17:51:20.333 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:51:20.333 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71082]: 2024-03-28T17:51:20.333 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:19 smithi151 podman[71419]: 2024-03-28 17:51:19.99258092 +0000 UTC m=+0.020824562 container died 0a51be78584ea4109b0507707af5ec218cfc1f46173b487a100a46a706e7992d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-28T17:51:20.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:19 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/1934350429,v1:172.21.15.151:6835/1934350429] up:rejoin 2024-03-28T17:51:20.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:19 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:20.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:19 smithi045 ceph-mon[31773]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-28T17:51:20.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:19 smithi045 ceph-mon[31773]: pgmap v76: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:51:20.839 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.12 MB/sec warmup 53 sec latency 48735.930 ms 2024-03-28T17:51:20.928 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:20 smithi151 podman[71419]: 2024-03-28 17:51:20.645251166 +0000 UTC m=+0.673494812 container remove 0a51be78584ea4109b0507707af5ec218cfc1f46173b487a100a46a706e7992d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T17:51:20.928 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:20 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:51:20.929 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:20 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-28T17:51:21.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:20 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:51:21.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:20 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:51:21.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:20 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/1934350429,v1:172.21.15.151:6835/1934350429] up:active 2024-03-28T17:51:21.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:20 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:21.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:20 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:51:21.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:20 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:51:21.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:20 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:51:21.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:20 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/1934350429,v1:172.21.15.151:6835/1934350429] up:active 2024-03-28T17:51:21.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:20 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:21.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:20 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:51:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:20 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:51:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:20 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:51:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:20 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/1934350429,v1:172.21.15.151:6835/1934350429] up:active 2024-03-28T17:51:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:20 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:20 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:51:21.839 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.10 MB/sec warmup 54 sec latency 49736.094 ms 2024-03-28T17:51:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:21 smithi071 ceph-mon[36597]: pgmap v77: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-03-28T17:51:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:21 smithi151 ceph-mon[36481]: pgmap v77: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-03-28T17:51:22.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:21 smithi045 ceph-mon[31773]: pgmap v77: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-03-28T17:51:22.839 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.08 MB/sec warmup 55 sec latency 50736.245 ms 2024-03-28T17:51:23.839 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.06 MB/sec warmup 56 sec latency 51736.371 ms 2024-03-28T17:51:24.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:24 smithi071 ceph-mon[36597]: pgmap v78: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-03-28T17:51:24.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:24 smithi151 ceph-mon[36481]: pgmap v78: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-03-28T17:51:24.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:24 smithi045 ceph-mon[31773]: pgmap v78: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-03-28T17:51:24.839 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.04 MB/sec warmup 57 sec latency 52736.522 ms 2024-03-28T17:51:25.840 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.02 MB/sec warmup 58 sec latency 53736.666 ms 2024-03-28T17:51:26.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:26 smithi071 ceph-mon[36597]: pgmap v79: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-03-28T17:51:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:26 smithi151 ceph-mon[36481]: pgmap v79: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-03-28T17:51:26.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:26 smithi045 ceph-mon[31773]: pgmap v79: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-03-28T17:51:26.840 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 1.00 MB/sec warmup 59 sec latency 54736.800 ms 2024-03-28T17:51:27.840 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.99 MB/sec warmup 60 sec latency 55736.889 ms 2024-03-28T17:51:28.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:28 smithi071 ceph-mon[36597]: pgmap v80: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-03-28T17:51:28.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:28 smithi151 ceph-mon[36481]: pgmap v80: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-03-28T17:51:28.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:28 smithi045 ceph-mon[31773]: pgmap v80: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-03-28T17:51:28.840 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.97 MB/sec warmup 61 sec latency 56737.049 ms 2024-03-28T17:51:29.840 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.95 MB/sec warmup 62 sec latency 57737.160 ms 2024-03-28T17:51:30.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:30 smithi071 ceph-mon[36597]: pgmap v81: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-28T17:51:30.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:30 smithi151 ceph-mon[36481]: pgmap v81: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-28T17:51:30.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:30 smithi045 ceph-mon[31773]: pgmap v81: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-28T17:51:30.841 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.94 MB/sec warmup 63 sec latency 58737.321 ms 2024-03-28T17:51:31.328 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:31 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 2. 2024-03-28T17:51:31.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:31 smithi151 systemd[1]: Stopped Ceph mds.f for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:51:31.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:31 smithi151 systemd[1]: Starting Ceph mds.f for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:51:31.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:31 smithi151 podman[71517]: 2024-03-28 17:51:31.212603234 +0000 UTC m=+0.018587549 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:51:31.329 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:31 smithi151 podman[71517]: 2024-03-28 17:51:31.329128264 +0000 UTC m=+0.135112615 container create 80be162d30c757c5605a4d191c6cc74af519d7fa51fdd306b74f77783c11c5f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-28T17:51:31.597 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:31 smithi151 podman[71517]: 2024-03-28 17:51:31.573414808 +0000 UTC m=+0.379399120 container init 80be162d30c757c5605a4d191c6cc74af519d7fa51fdd306b74f77783c11c5f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-28T17:51:31.598 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:31 smithi151 podman[71517]: 2024-03-28 17:51:31.576035057 +0000 UTC m=+0.382019368 container start 80be162d30c757c5605a4d191c6cc74af519d7fa51fdd306b74f77783c11c5f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T17:51:31.598 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:31 smithi151 ceph-mds[71536]: -- 172.21.15.151:0/1065245683 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561167323a40 con 0x5611672f7000 2024-03-28T17:51:31.840 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.93 MB/sec warmup 64 sec latency 59737.442 ms 2024-03-28T17:51:32.026 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:31 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: starting mds.f at 2024-03-28T17:51:32.026 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:31 smithi151 bash[71517]: 80be162d30c757c5605a4d191c6cc74af519d7fa51fdd306b74f77783c11c5f8 2024-03-28T17:51:32.026 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:31 smithi151 systemd[1]: Started Ceph mds.f for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:51:32.098 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:51:32.098 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:51:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:32 smithi071 ceph-mon[36597]: pgmap v82: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-28T17:51:32.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:32 smithi071 ceph-mon[36597]: daemon mds.f restarted 2024-03-28T17:51:32.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:32 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:51:32.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:32 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T17:51:32.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:32 smithi071 ceph-mon[36597]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T17:51:32.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:32 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/4265871690,v1:172.21.15.151:6835/4265871690] up:boot 2024-03-28T17:51:32.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:32 smithi071 ceph-mon[36597]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-03-28T17:51:32.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:32 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:51:32.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:32 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:32 smithi151 ceph-mon[36481]: pgmap v82: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-28T17:51:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:32 smithi151 ceph-mon[36481]: daemon mds.f restarted 2024-03-28T17:51:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:32 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:51:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:32 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T17:51:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:32 smithi151 ceph-mon[36481]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T17:51:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:32 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/4265871690,v1:172.21.15.151:6835/4265871690] up:boot 2024-03-28T17:51:32.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:32 smithi151 ceph-mon[36481]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-03-28T17:51:32.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:32 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:51:32.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:32 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:32.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.336+0000 7f7091a00640 1 -- 172.21.15.45:0/1866381112 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f708c10a6a0 msgr2=0x7f708c10ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:32.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.336+0000 7f7091a00640 1 --2- 172.21.15.45:0/1866381112 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f708c10a6a0 0x7f708c10ca90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7084009ca0 tx=0x7f708402c100 comp rx=0 tx=0).stop 2024-03-28T17:51:32.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.336+0000 7f7091a00640 1 -- 172.21.15.45:0/1866381112 shutdown_connections 2024-03-28T17:51:32.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.336+0000 7f7091a00640 1 --2- 172.21.15.45:0/1866381112 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f708c10a6a0 0x7f708c10ca90 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.336+0000 7f7091a00640 1 --2- 172.21.15.45:0/1866381112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f708c106110 0x7f708c0fffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.336+0000 7f7091a00640 1 --2- 172.21.15.45:0/1866381112 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f708c067b20 0x7f708c0ffa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.336+0000 7f7091a00640 1 -- 172.21.15.45:0/1866381112 >> 172.21.15.45:0/1866381112 conn(0x7f708c075240 msgr2=0x7f708c075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:32.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.336+0000 7f7091a00640 1 -- 172.21.15.45:0/1866381112 shutdown_connections 2024-03-28T17:51:32.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.336+0000 7f7091a00640 1 -- 172.21.15.45:0/1866381112 wait complete. 2024-03-28T17:51:32.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.337+0000 7f7091a00640 1 Processor -- start 2024-03-28T17:51:32.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.337+0000 7f7091a00640 1 -- start start 2024-03-28T17:51:32.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.337+0000 7f7091a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f708c067b20 0x7f708c19fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:32.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.337+0000 7f7091a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f708c106110 0x7f708c1a02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:32.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.338+0000 7f7091a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f708c10a6a0 0x7f708c109380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:32.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.338+0000 7f7091a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c1119b0 con 0x7f708c067b20 2024-03-28T17:51:32.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.338+0000 7f7091a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c111830 con 0x7f708c106110 2024-03-28T17:51:32.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.338+0000 7f7091a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c111b30 con 0x7f708c10a6a0 2024-03-28T17:51:32.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.338+0000 7f708b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f708c106110 0x7f708c1a02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:32.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.338+0000 7f708be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f708c067b20 0x7f708c19fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:32.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.338+0000 7f708b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f708c106110 0x7f708c1a02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:43918/0 (socket says 172.21.15.45:43918) 2024-03-28T17:51:32.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.338+0000 7f708b400640 1 -- 172.21.15.45:0/3588058673 learned_addr learned my addr 172.21.15.45:0/3588058673 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:32.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.338+0000 7f7091000640 1 --2- 172.21.15.45:0/3588058673 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f708c10a6a0 0x7f708c109380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:32.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.338+0000 7f708b400640 1 -- 172.21.15.45:0/3588058673 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f708c10a6a0 msgr2=0x7f708c109380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:32.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.338+0000 7f708b400640 1 --2- 172.21.15.45:0/3588058673 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f708c10a6a0 0x7f708c109380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.338+0000 7f708b400640 1 -- 172.21.15.45:0/3588058673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f708c067b20 msgr2=0x7f708c19fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:32.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.338+0000 7f708b400640 1 --2- 172.21.15.45:0/3588058673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f708c067b20 0x7f708c19fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.338+0000 7f708b400640 1 -- 172.21.15.45:0/3588058673 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f708c109b00 con 0x7f708c106110 2024-03-28T17:51:32.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.339+0000 7f708b400640 1 --2- 172.21.15.45:0/3588058673 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f708c106110 0x7f708c1a02a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f707800d740 tx=0x7f707800dc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:32.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.339+0000 7f7088c00640 1 -- 172.21.15.45:0/3588058673 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7078017070 con 0x7f708c106110 2024-03-28T17:51:32.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.339+0000 7f7088c00640 1 -- 172.21.15.45:0/3588058673 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f707800de50 con 0x7f708c106110 2024-03-28T17:51:32.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.339+0000 7f7088c00640 1 -- 172.21.15.45:0/3588058673 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707800f6b0 con 0x7f708c106110 2024-03-28T17:51:32.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.339+0000 7f7091a00640 1 -- 172.21.15.45:0/3588058673 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f708c109df0 con 0x7f708c106110 2024-03-28T17:51:32.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.339+0000 7f7091a00640 1 -- 172.21.15.45:0/3588058673 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f708c1a6bb0 con 0x7f708c106110 2024-03-28T17:51:32.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.342+0000 7f7095c05740 1 Processor -- start 2024-03-28T17:51:32.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.342+0000 7f7095c05740 1 -- start start 2024-03-28T17:51:32.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.343+0000 7f7088c00640 1 -- 172.21.15.45:0/3588058673 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f707801a070 con 0x7f708c106110 2024-03-28T17:51:32.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.343+0000 7f7095c05740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d809dd0 0x56438d808b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:32.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.343+0000 7f7095c05740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d8090f0 0x56438d809570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:32.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.343+0000 7f7095c05740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d812460 0x56438d814850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:32.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.343+0000 7f7095c05740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56438d76d250 con 0x56438d812460 2024-03-28T17:51:32.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.343+0000 7f7095c05740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x56438d715a30 con 0x56438d8090f0 2024-03-28T17:51:32.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.343+0000 7f7095c05740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x56438d7692f0 con 0x56438d809dd0 2024-03-28T17:51:32.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.343+0000 7f7091000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d809dd0 0x56438d808b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:32.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.343+0000 7f708be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d8090f0 0x56438d809570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:32.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.343+0000 7f7091000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d809dd0 0x56438d808b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:42662/0 (socket says 172.21.15.45:42662) 2024-03-28T17:51:32.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.343+0000 7f7091000640 1 -- 172.21.15.45:0/1013379809 learned_addr learned my addr 172.21.15.45:0/1013379809 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:32.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.343+0000 7f708b400640 1 --2- 172.21.15.45:0/1013379809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d812460 0x56438d814850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:32.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.343+0000 7f7091000640 1 -- 172.21.15.45:0/1013379809 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d8090f0 msgr2=0x56438d809570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:32.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.344+0000 7f7091000640 1 --2- 172.21.15.45:0/1013379809 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d8090f0 0x56438d809570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.344+0000 7f7091000640 1 -- 172.21.15.45:0/1013379809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d812460 msgr2=0x56438d814850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:32.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.344+0000 7f7091000640 1 --2- 172.21.15.45:0/1013379809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d812460 0x56438d814850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.344+0000 7f7091000640 1 -- 172.21.15.45:0/1013379809 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56438d700690 con 0x56438d809dd0 2024-03-28T17:51:32.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.344+0000 7f708be00640 1 --2- 172.21.15.45:0/1013379809 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d8090f0 0x56438d809570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:51:32.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.344+0000 7f7091000640 1 --2- 172.21.15.45:0/1013379809 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d809dd0 0x56438d808b20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7084004af0 tx=0x7f70840098a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58a37d1a48481aae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:32.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.344+0000 7f7088c00640 1 --2- 172.21.15.45:0/3588058673 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f70700a29f0 0x7f70700a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:32.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.344+0000 7f7081600640 1 -- 172.21.15.45:0/1013379809 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7084043070 con 0x56438d809dd0 2024-03-28T17:51:32.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.344+0000 7f7088c00640 1 -- 172.21.15.45:0/3588058673 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8856+0+0 (secure 0 0 0) 0x7f70780ce810 con 0x7f708c106110 2024-03-28T17:51:32.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.344+0000 7f7081600640 1 -- 172.21.15.45:0/1013379809 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7084032c90 con 0x56438d809dd0 2024-03-28T17:51:32.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.344+0000 7f7081600640 1 -- 172.21.15.45:0/1013379809 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7084039040 con 0x56438d809dd0 2024-03-28T17:51:32.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.344+0000 7f708be00640 1 --2- 172.21.15.45:0/3588058673 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f70700a29f0 0x7f70700a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:32.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.345+0000 7f7095c05740 1 -- 172.21.15.45:0/1013379809 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d809dd0 msgr2=0x56438d808b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:32.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.345+0000 7f7095c05740 1 --2- 172.21.15.45:0/1013379809 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d809dd0 0x56438d808b20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7084004af0 tx=0x7f70840098a0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.345+0000 7f708be00640 1 --2- 172.21.15.45:0/3588058673 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f70700a29f0 0x7f70700a4eb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x56438d80a320 tx=0x7f707c00b390 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:32.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.345+0000 7f7095c05740 1 -- 172.21.15.45:0/1013379809 shutdown_connections 2024-03-28T17:51:32.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.345+0000 7f7095c05740 1 --2- 172.21.15.45:0/1013379809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d812460 0x56438d814850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.345+0000 7f7095c05740 1 --2- 172.21.15.45:0/1013379809 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d8090f0 0x56438d809570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.345+0000 7f7095c05740 1 --2- 172.21.15.45:0/1013379809 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d809dd0 0x56438d808b20 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.345+0000 7f7095c05740 1 -- 172.21.15.45:0/1013379809 >> 172.21.15.45:0/1013379809 conn(0x56438d7d0200 msgr2=0x56438d803b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:32.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.345+0000 7f7095c05740 1 -- 172.21.15.45:0/1013379809 shutdown_connections 2024-03-28T17:51:32.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.345+0000 7f7095c05740 1 -- 172.21.15.45:0/1013379809 wait complete. 2024-03-28T17:51:32.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.346+0000 7f7095c05740 1 -- start start 2024-03-28T17:51:32.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.346+0000 7f7095c05740 1 Processor -- start 2024-03-28T17:51:32.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.346+0000 7f7095c05740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d8090f0 0x56438d811ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:32.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.346+0000 7f7095c05740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d809dd0 0x56438d80ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:32.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.347+0000 7f7095c05740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d812460 0x56438d80f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:32.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.347+0000 7f7095c05740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56438d815270 con 0x56438d812460 2024-03-28T17:51:32.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.347+0000 7f7095c05740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x56438d728e80 con 0x56438d809dd0 2024-03-28T17:51:32.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.347+0000 7f7095c05740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x56438d8153f0 con 0x56438d8090f0 2024-03-28T17:51:32.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.347+0000 7f7095c05740 10 client.0 _ll_register_callbacks cb 0x56438d73b130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:51:32.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.347+0000 7f708be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d8090f0 0x56438d811ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:32.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.347+0000 7f708b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d809dd0 0x56438d80ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:32.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.347+0000 7f708be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d8090f0 0x56438d811ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:42676/0 (socket says 172.21.15.45:42676) 2024-03-28T17:51:32.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.347+0000 7f708be00640 1 -- 172.21.15.45:0/1661567565 learned_addr learned my addr 172.21.15.45:0/1661567565 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:32.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.347+0000 7f7091000640 1 --2- 172.21.15.45:0/1661567565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d812460 0x56438d80f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:32.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.347+0000 7f708be00640 1 -- 172.21.15.45:0/1661567565 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d809dd0 msgr2=0x56438d80ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:32.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.347+0000 7f708be00640 1 --2- 172.21.15.45:0/1661567565 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d809dd0 0x56438d80ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.347+0000 7f708be00640 1 -- 172.21.15.45:0/1661567565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d812460 msgr2=0x56438d80f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:32.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.347+0000 7f708be00640 1 --2- 172.21.15.45:0/1661567565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d812460 0x56438d80f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.347+0000 7f708be00640 1 -- 172.21.15.45:0/1661567565 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56438d54b7d0 con 0x56438d8090f0 2024-03-28T17:51:32.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.347+0000 7f708b400640 1 --2- 172.21.15.45:0/1661567565 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d809dd0 0x56438d80ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:51:32.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.348+0000 7f708be00640 1 --2- 172.21.15.45:0/1661567565 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d8090f0 0x56438d811ba0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f707c003d00 tx=0x7f707c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40f4e10015f0671 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:32.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.348+0000 7f7076000640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:51:32.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.348+0000 7f7076000640 1 -- 172.21.15.45:0/1661567565 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707c02c060 con 0x56438d8090f0 2024-03-28T17:51:32.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.348+0000 7f7095c05740 1 -- 172.21.15.45:0/1661567565 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7084032c90 con 0x56438d8090f0 2024-03-28T17:51:32.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.348+0000 7f7076000640 1 -- 172.21.15.45:0/1661567565 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8856+0+0 (secure 0 0 0) 0x7f707c017030 con 0x56438d8090f0 2024-03-28T17:51:32.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.348+0000 7f7076000640 1 -- 172.21.15.45:0/1661567565 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f707c003d30 con 0x56438d8090f0 2024-03-28T17:51:32.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.348+0000 7f7095c05740 10 client.25921 fetch_fsmap learned FSMap version 47 2024-03-28T17:51:32.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.348+0000 7f7095c05740 1 -- 172.21.15.45:0/1661567565 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x56438d4813f0 con 0x56438d8090f0 2024-03-28T17:51:32.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.349+0000 7f7076000640 1 -- 172.21.15.45:0/1661567565 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7084032c90 con 0x56438d8090f0 2024-03-28T17:51:32.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.349+0000 7f7076000640 1 -- 172.21.15.45:0/1661567565 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 47) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f707c011600 con 0x56438d8090f0 2024-03-28T17:51:32.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.349+0000 7f7095c05740 10 client.25921 fetch_fsmap finished waiting for FSMap version 47 2024-03-28T17:51:32.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.349+0000 7f7095c05740 10 client.25921 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:51:32.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.349+0000 7f7095c05740 20 client.25921 populate_metadata read hostname 'smithi045' 2024-03-28T17:51:32.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.349+0000 7f7095c05740 1 --2- 172.21.15.45:0/1661567565 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56438d822370 0x56438d8247a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:32.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.349+0000 7f7095c05740 4 client.25921 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:51:32.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.349+0000 7f7095c05740 1 -- 172.21.15.45:0/1661567565 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56438d77a790 con 0x56438d822370 2024-03-28T17:51:32.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.349+0000 7f708b400640 1 --2- 172.21.15.45:0/1661567565 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56438d822370 0x56438d8247a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:32.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.350+0000 7f708b400640 1 --2- 172.21.15.45:0/1661567565 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56438d822370 0x56438d8247a0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=529a300cc729e616 server_cookie=ae2ca21b2fdd6ca7 in_seq=0 out_seq=0 2024-03-28T17:51:32.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.350+0000 7f7076000640 10 client.25921 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:51:32.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.350+0000 7f7076000640 1 -- 172.21.15.45:0/1661567565 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56438d77a790 con 0x56438d822370 2024-03-28T17:51:32.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.350+0000 7f7076000640 10 client.25921 handle_command_reply: tid=0 2024-03-28T17:51:32.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.350+0000 7f7095c05740 1 client.25921 shutdown 2024-03-28T17:51:32.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.351+0000 7f7095c05740 1 -- 172.21.15.45:0/1661567565 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d8090f0 msgr2=0x56438d811ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:32.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.351+0000 7f7095c05740 1 --2- 172.21.15.45:0/1661567565 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d8090f0 0x56438d811ba0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f707c003d00 tx=0x7f707c004730 comp rx=0 tx=0).stop 2024-03-28T17:51:32.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.351+0000 7f7095c05740 1 -- 172.21.15.45:0/1661567565 shutdown_connections 2024-03-28T17:51:32.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.351+0000 7f7095c05740 1 --2- 172.21.15.45:0/1661567565 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56438d822370 0x56438d8247a0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.351+0000 7f7095c05740 1 --2- 172.21.15.45:0/1661567565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d812460 0x56438d80f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.351+0000 7f7095c05740 1 --2- 172.21.15.45:0/1661567565 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d809dd0 0x56438d80ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.351+0000 7f7076000640 0 client.25921 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:51:32.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.351+0000 7f7095c05740 1 --2- 172.21.15.45:0/1661567565 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d8090f0 0x56438d811ba0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.351+0000 7f7095c05740 1 -- 172.21.15.45:0/1661567565 >> 172.21.15.45:0/1661567565 conn(0x56438d7d0200 msgr2=0x56438d805890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:32.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.351+0000 7f7095c05740 1 -- 172.21.15.45:0/1661567565 shutdown_connections 2024-03-28T17:51:32.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.351+0000 7f7095c05740 1 -- 172.21.15.45:0/1661567565 wait complete. 2024-03-28T17:51:32.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.351+0000 7f7095c05740 20 client.25921 trim_cache size 0 max 16384 2024-03-28T17:51:32.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.369+0000 7f7095c05740 1 Processor -- start 2024-03-28T17:51:32.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.369+0000 7f7095c05740 1 -- start start 2024-03-28T17:51:32.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.369+0000 7f7095c05740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d8090f0 0x56438d811660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:32.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.369+0000 7f7095c05740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d809dd0 0x56438d811ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:32.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.369+0000 7f7095c05740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d812460 0x56438d813850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:32.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.369+0000 7f7095c05740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c1afc60 con 0x56438d809dd0 2024-03-28T17:51:32.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.369+0000 7f7095c05740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c1ad6a0 con 0x56438d8090f0 2024-03-28T17:51:32.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.369+0000 7f7095c05740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c1af600 con 0x56438d812460 2024-03-28T17:51:32.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.369+0000 7f708b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d809dd0 0x56438d811ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:32.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.369+0000 7f708be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d8090f0 0x56438d811660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:32.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.369+0000 7f708b400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d809dd0 0x56438d811ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45324/0 (socket says 172.21.15.45:45324) 2024-03-28T17:51:32.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.369+0000 7f708b400640 1 -- 172.21.15.45:0/2171974754 learned_addr learned my addr 172.21.15.45:0/2171974754 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:32.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.370+0000 7f7091000640 1 --2- 172.21.15.45:0/2171974754 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d812460 0x56438d813850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:32.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.370+0000 7f708b400640 1 -- 172.21.15.45:0/2171974754 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d812460 msgr2=0x56438d813850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:32.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.370+0000 7f708b400640 1 --2- 172.21.15.45:0/2171974754 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d812460 0x56438d813850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.370+0000 7f708b400640 1 -- 172.21.15.45:0/2171974754 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d8090f0 msgr2=0x56438d811660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:32.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.370+0000 7f708b400640 1 --2- 172.21.15.45:0/2171974754 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d8090f0 0x56438d811660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.370+0000 7f708b400640 1 -- 172.21.15.45:0/2171974754 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56438d4410f0 con 0x56438d809dd0 2024-03-28T17:51:32.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.370+0000 7f708be00640 1 --2- 172.21.15.45:0/2171974754 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d8090f0 0x56438d811660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:51:32.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.370+0000 7f708b400640 1 --2- 172.21.15.45:0/2171974754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d809dd0 0x56438d811ba0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7078097230 tx=0x7f70780d0d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bd278a3de937ba8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:32.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.370+0000 7f7080c00640 1 -- 172.21.15.45:0/2171974754 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7078021040 con 0x56438d809dd0 2024-03-28T17:51:32.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.371+0000 7f7080c00640 1 -- 172.21.15.45:0/2171974754 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f707809add0 con 0x56438d809dd0 2024-03-28T17:51:32.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.371+0000 7f7080c00640 1 -- 172.21.15.45:0/2171974754 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7078099160 con 0x56438d809dd0 2024-03-28T17:51:32.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.371+0000 7f7095c05740 1 -- 172.21.15.45:0/2171974754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d809dd0 msgr2=0x56438d811ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:32.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.371+0000 7f7095c05740 1 --2- 172.21.15.45:0/2171974754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d809dd0 0x56438d811ba0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7078097230 tx=0x7f70780d0d20 comp rx=0 tx=0).stop 2024-03-28T17:51:32.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.371+0000 7f7095c05740 1 -- 172.21.15.45:0/2171974754 shutdown_connections 2024-03-28T17:51:32.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.371+0000 7f7095c05740 1 --2- 172.21.15.45:0/2171974754 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d812460 0x56438d813850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.371+0000 7f7095c05740 1 --2- 172.21.15.45:0/2171974754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d809dd0 0x56438d811ba0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.371+0000 7f7095c05740 1 --2- 172.21.15.45:0/2171974754 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d8090f0 0x56438d811660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.371+0000 7f7095c05740 1 -- 172.21.15.45:0/2171974754 >> 172.21.15.45:0/2171974754 conn(0x56438d7d0200 msgr2=0x56438d803670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:32.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.371+0000 7f7095c05740 1 -- 172.21.15.45:0/2171974754 shutdown_connections 2024-03-28T17:51:32.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.371+0000 7f7095c05740 1 -- 172.21.15.45:0/2171974754 wait complete. 2024-03-28T17:51:32.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.372+0000 7f7095c05740 1 -- start start 2024-03-28T17:51:32.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.372+0000 7f7095c05740 1 Processor -- start 2024-03-28T17:51:32.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.372+0000 7f7095c05740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d8090f0 0x56438d81cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:32.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.372+0000 7f7095c05740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d809dd0 0x56438d81d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:32.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.372+0000 7f7095c05740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d812460 0x56438d819680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:32.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.372+0000 7f7095c05740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c1aefa0 con 0x56438d8090f0 2024-03-28T17:51:32.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.372+0000 7f7095c05740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c1adca0 con 0x56438d809dd0 2024-03-28T17:51:32.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.372+0000 7f7095c05740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c1ae2e0 con 0x56438d812460 2024-03-28T17:51:32.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.373+0000 7f7095c05740 10 client.0 _ll_register_callbacks cb 0x7f707c003bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:51:32.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.373+0000 7f708be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d8090f0 0x56438d81cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:32.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.373+0000 7f708b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d809dd0 0x56438d81d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:32.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.373+0000 7f708be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d8090f0 0x56438d81cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45340/0 (socket says 172.21.15.45:45340) 2024-03-28T17:51:32.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.373+0000 7f708be00640 1 -- 172.21.15.45:0/3283141869 learned_addr learned my addr 172.21.15.45:0/3283141869 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:51:32.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.373+0000 7f7091000640 1 --2- 172.21.15.45:0/3283141869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d812460 0x56438d819680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:32.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.373+0000 7f708b400640 1 -- 172.21.15.45:0/3283141869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d812460 msgr2=0x56438d819680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:32.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.373+0000 7f708b400640 1 --2- 172.21.15.45:0/3283141869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d812460 0x56438d819680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.373+0000 7f708b400640 1 -- 172.21.15.45:0/3283141869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d8090f0 msgr2=0x56438d81cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:51:32.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.373+0000 7f708b400640 1 --2- 172.21.15.45:0/3283141869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d8090f0 0x56438d81cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.373+0000 7f708b400640 1 -- 172.21.15.45:0/3283141869 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56438d819c70 con 0x56438d809dd0 2024-03-28T17:51:32.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.373+0000 7f7091000640 1 --2- 172.21.15.45:0/3283141869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d812460 0x56438d819680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:51:32.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.373+0000 7f708be00640 1 --2- 172.21.15.45:0/3283141869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d8090f0 0x56438d81cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:51:32.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.373+0000 7f708b400640 1 --2- 172.21.15.45:0/3283141869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d809dd0 0x56438d81d400 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f70780977a0 tx=0x7f7078010cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dddab0e709c10d7a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:51:32.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.373+0000 7f7076000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:51:32.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.374+0000 7f7076000640 1 -- 172.21.15.45:0/3283141869 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7078021070 con 0x56438d809dd0 2024-03-28T17:51:32.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.374+0000 7f7076000640 1 -- 172.21.15.45:0/3283141869 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8856+0+0 (secure 0 0 0) 0x7f7078015070 con 0x56438d809dd0 2024-03-28T17:51:32.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.374+0000 7f7095c05740 1 -- 172.21.15.45:0/3283141869 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f707809add0 con 0x56438d809dd0 2024-03-28T17:51:32.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.374+0000 7f7076000640 1 -- 172.21.15.45:0/3283141869 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f707809a730 con 0x56438d809dd0 2024-03-28T17:51:32.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.374+0000 7f7095c05740 10 client.25961 fetch_fsmap learned FSMap version 47 2024-03-28T17:51:32.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.374+0000 7f7095c05740 1 -- 172.21.15.45:0/3283141869 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x56438d87e0a0 con 0x56438d809dd0 2024-03-28T17:51:32.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.374+0000 7f7076000640 1 -- 172.21.15.45:0/3283141869 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f707809add0 con 0x56438d809dd0 2024-03-28T17:51:32.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.374+0000 7f7076000640 1 -- 172.21.15.45:0/3283141869 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 47) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f707809a2b0 con 0x56438d809dd0 2024-03-28T17:51:32.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.374+0000 7f7095c05740 10 client.25961 fetch_fsmap finished waiting for FSMap version 47 2024-03-28T17:51:32.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.374+0000 7f7095c05740 10 client.25961 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:51:32.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.374+0000 7f7095c05740 20 client.25961 populate_metadata read hostname 'smithi045' 2024-03-28T17:51:32.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.374+0000 7f7095c05740 1 --2- 172.21.15.45:0/3283141869 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56438d87e7b0 0x56438d880bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:51:32.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.374+0000 7f7095c05740 4 client.25961 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:51:32.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.374+0000 7f7095c05740 1 -- 172.21.15.45:0/3283141869 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56438d774ae0 con 0x56438d87e7b0 2024-03-28T17:51:32.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.375+0000 7f708be00640 1 --2- 172.21.15.45:0/3283141869 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56438d87e7b0 0x56438d880bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:51:32.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.375+0000 7f708be00640 1 --2- 172.21.15.45:0/3283141869 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56438d87e7b0 0x56438d880bd0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dee9b91f894e5865 server_cookie=a886a8dd1f58a0f0 in_seq=0 out_seq=0 2024-03-28T17:51:32.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.375+0000 7f7076000640 10 client.25961 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:51:32.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.375+0000 7f7076000640 1 -- 172.21.15.45:0/3283141869 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56438d774ae0 con 0x56438d87e7b0 2024-03-28T17:51:32.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.375+0000 7f7076000640 10 client.25961 handle_command_reply: tid=0 2024-03-28T17:51:32.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.375+0000 7f7095c05740 1 client.25961 shutdown 2024-03-28T17:51:32.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.375+0000 7f7095c05740 1 -- 172.21.15.45:0/3283141869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d809dd0 msgr2=0x56438d81d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:32.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.375+0000 7f7095c05740 1 --2- 172.21.15.45:0/3283141869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d809dd0 0x56438d81d400 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f70780977a0 tx=0x7f7078010cb0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.375+0000 7f7095c05740 1 -- 172.21.15.45:0/3283141869 shutdown_connections 2024-03-28T17:51:32.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.375+0000 7f7095c05740 1 --2- 172.21.15.45:0/3283141869 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56438d87e7b0 0x56438d880bd0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.375+0000 7f7076000640 0 client.25961 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:51:32.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.375+0000 7f7095c05740 1 --2- 172.21.15.45:0/3283141869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56438d812460 0x56438d819680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.375+0000 7f7095c05740 1 --2- 172.21.15.45:0/3283141869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56438d809dd0 0x56438d81d400 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.375+0000 7f7095c05740 1 --2- 172.21.15.45:0/3283141869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56438d8090f0 0x56438d81cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.375+0000 7f7095c05740 1 -- 172.21.15.45:0/3283141869 >> 172.21.15.45:0/3283141869 conn(0x56438d7d0200 msgr2=0x56438d80d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:32.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.376+0000 7f7095c05740 1 -- 172.21.15.45:0/3283141869 shutdown_connections 2024-03-28T17:51:32.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.376+0000 7f7095c05740 1 -- 172.21.15.45:0/3283141869 wait complete. 2024-03-28T17:51:32.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.376+0000 7f7095c05740 20 client.25961 trim_cache size 0 max 16384 2024-03-28T17:51:32.383 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:51:32.383 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T17:51:32.383 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T17:51:32.383 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T17:51:32.383 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T17:51:32.383 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T17:51:32.383 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T17:51:32.383 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:51:32.384 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:51:32.384 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:51:32.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.376+0000 7f7091a00640 1 -- 172.21.15.45:0/3588058673 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f70700a29f0 msgr2=0x7f70700a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:32.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.376+0000 7f7091a00640 1 --2- 172.21.15.45:0/3588058673 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f70700a29f0 0x7f70700a4eb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x56438d80a320 tx=0x7f707c00b390 comp rx=0 tx=0).stop 2024-03-28T17:51:32.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.376+0000 7f7091a00640 1 -- 172.21.15.45:0/3588058673 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f708c106110 msgr2=0x7f708c1a02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:51:32.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.376+0000 7f7091a00640 1 --2- 172.21.15.45:0/3588058673 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f708c106110 0x7f708c1a02a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f707800d740 tx=0x7f707800dc00 comp rx=0 tx=0).stop 2024-03-28T17:51:32.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.376+0000 7f7091a00640 1 -- 172.21.15.45:0/3588058673 shutdown_connections 2024-03-28T17:51:32.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.376+0000 7f7091a00640 1 --2- 172.21.15.45:0/3588058673 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f70700a29f0 0x7f70700a4eb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.376+0000 7f7091a00640 1 --2- 172.21.15.45:0/3588058673 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f708c10a6a0 0x7f708c109380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.376+0000 7f7091a00640 1 --2- 172.21.15.45:0/3588058673 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f708c106110 0x7f708c1a02a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.376+0000 7f7091a00640 1 --2- 172.21.15.45:0/3588058673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f708c067b20 0x7f708c19fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:51:32.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.376+0000 7f7091a00640 1 -- 172.21.15.45:0/3588058673 >> 172.21.15.45:0/3588058673 conn(0x7f708c075240 msgr2=0x7f708c0fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:51:32.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.377+0000 7f7091a00640 1 -- 172.21.15.45:0/3588058673 shutdown_connections 2024-03-28T17:51:32.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:51:32.377+0000 7f7091a00640 1 -- 172.21.15.45:0/3588058673 wait complete. 2024-03-28T17:51:32.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:32 smithi045 ceph-mon[31773]: pgmap v82: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-28T17:51:32.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:32 smithi045 ceph-mon[31773]: daemon mds.f restarted 2024-03-28T17:51:32.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:32 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:51:32.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:32 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T17:51:32.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:32 smithi045 ceph-mon[31773]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T17:51:32.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:32 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/4265871690,v1:172.21.15.151:6835/4265871690] up:boot 2024-03-28T17:51:32.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:32 smithi045 ceph-mon[31773]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-03-28T17:51:32.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:32 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:51:32.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:32 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:32.397 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T17:51:32.841 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.91 MB/sec warmup 65 sec latency 60737.600 ms 2024-03-28T17:51:33.841 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.90 MB/sec warmup 66 sec latency 61737.808 ms 2024-03-28T17:51:33.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:33 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6838/4005741328,v1:172.21.15.151:6839/4005741328] up:resolve 2024-03-28T17:51:33.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:33 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:34.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:33 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6838/4005741328,v1:172.21.15.151:6839/4005741328] up:resolve 2024-03-28T17:51:34.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:33 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:34.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:33 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6838/4005741328,v1:172.21.15.151:6839/4005741328] up:resolve 2024-03-28T17:51:34.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:33 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:34.841 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.88 MB/sec warmup 67 sec latency 62737.957 ms 2024-03-28T17:51:34.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:34 smithi045 ceph-mon[31773]: pgmap v84: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 9 op/s 2024-03-28T17:51:34.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:34 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:51:34.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:34 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6838/4005741328,v1:172.21.15.151:6839/4005741328] up:reconnect 2024-03-28T17:51:34.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:34 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:34.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:34 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999988 2024-03-28T17:51:35.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:34 smithi071 ceph-mon[36597]: pgmap v84: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 9 op/s 2024-03-28T17:51:35.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:34 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:51:35.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:34 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6838/4005741328,v1:172.21.15.151:6839/4005741328] up:reconnect 2024-03-28T17:51:35.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:34 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:35.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:34 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999988 2024-03-28T17:51:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:34 smithi151 ceph-mon[36481]: pgmap v84: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 9 op/s 2024-03-28T17:51:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:34 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:51:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:34 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6838/4005741328,v1:172.21.15.151:6839/4005741328] up:reconnect 2024-03-28T17:51:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:34 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:35.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:34 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999988 2024-03-28T17:51:35.841 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.87 MB/sec warmup 68 sec latency 63738.120 ms 2024-03-28T17:51:35.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:35 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6838/4005741328,v1:172.21.15.151:6839/4005741328] up:rejoin 2024-03-28T17:51:35.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:35 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:35.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:35 smithi045 ceph-mon[31773]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-28T17:51:35.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:35 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:51:35.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:35 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:51:35.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:35 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6838/4005741328,v1:172.21.15.151:6839/4005741328] up:active 2024-03-28T17:51:36.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:35 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6838/4005741328,v1:172.21.15.151:6839/4005741328] up:rejoin 2024-03-28T17:51:36.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:35 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:36.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:35 smithi071 ceph-mon[36597]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-28T17:51:36.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:35 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:51:36.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:35 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:51:36.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:35 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6838/4005741328,v1:172.21.15.151:6839/4005741328] up:active 2024-03-28T17:51:36.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6838/4005741328,v1:172.21.15.151:6839/4005741328] up:rejoin 2024-03-28T17:51:36.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:36.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-mon[36481]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-28T17:51:36.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:51:36.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:51:36.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6838/4005741328,v1:172.21.15.151:6839/4005741328] up:active 2024-03-28T17:51:36.074 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f079441d700 time 2024-03-28T17:51:35.669383+0000 2024-03-28T17:51:36.074 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:51:36.074 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:36.074 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f079ea474a1] 2024-03-28T17:51:36.074 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f079ea47667] 2024-03-28T17:51:36.074 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563683e44223] 2024-03-28T17:51:36.074 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563683cbe5d6] 2024-03-28T17:51:36.074 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563683cbeda1] 2024-03-28T17:51:36.075 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563683cc12d5] 2024-03-28T17:51:36.075 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563683d11233] 2024-03-28T17:51:36.075 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563683d13197] 2024-03-28T17:51:36.075 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563683c1211b] 2024-03-28T17:51:36.075 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563683c126bc] 2024-03-28T17:51:36.075 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563683bfc1ff] 2024-03-28T17:51:36.075 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f079ecd9488] 2024-03-28T17:51:36.075 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 13: (DispatchQueue::entry()+0x51f) [0x7f079ecd660f] 2024-03-28T17:51:36.075 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f079eda9e81] 2024-03-28T17:51:36.075 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f079d7c41ca] 2024-03-28T17:51:36.075 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 16: clone() 2024-03-28T17:51:36.075 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 2024-03-28T17:51:35.670+0000 7f079441d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f079441d700 time 2024-03-28T17:51:35.669383+0000 2024-03-28T17:51:36.075 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:51:36.075 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 2024-03-28T17:51:36.076 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:36.076 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f079ea474a1] 2024-03-28T17:51:36.076 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f079ea47667] 2024-03-28T17:51:36.076 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563683e44223] 2024-03-28T17:51:36.076 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563683cbe5d6] 2024-03-28T17:51:36.076 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563683cbeda1] 2024-03-28T17:51:36.076 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563683cc12d5] 2024-03-28T17:51:36.076 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563683d11233] 2024-03-28T17:51:36.076 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563683d13197] 2024-03-28T17:51:36.076 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563683c1211b] 2024-03-28T17:51:36.076 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563683c126bc] 2024-03-28T17:51:36.076 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563683bfc1ff] 2024-03-28T17:51:36.076 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f079ecd9488] 2024-03-28T17:51:36.076 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 13: (DispatchQueue::entry()+0x51f) [0x7f079ecd660f] 2024-03-28T17:51:36.077 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f079eda9e81] 2024-03-28T17:51:36.077 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f079d7c41ca] 2024-03-28T17:51:36.077 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 16: clone() 2024-03-28T17:51:36.077 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 2024-03-28T17:51:36.077 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: *** Caught signal (Aborted) ** 2024-03-28T17:51:36.077 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: in thread 7f079441d700 thread_name:ms_dispatch 2024-03-28T17:51:36.077 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:36.077 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f079d7ced20] 2024-03-28T17:51:36.078 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 2: gsignal() 2024-03-28T17:51:36.078 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 3: abort() 2024-03-28T17:51:36.078 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f079ea474fb] 2024-03-28T17:51:36.078 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f079ea47667] 2024-03-28T17:51:36.078 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563683e44223] 2024-03-28T17:51:36.078 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563683cbe5d6] 2024-03-28T17:51:36.079 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563683cbeda1] 2024-03-28T17:51:36.079 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563683cc12d5] 2024-03-28T17:51:36.079 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563683d11233] 2024-03-28T17:51:36.079 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563683d13197] 2024-03-28T17:51:36.079 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563683c1211b] 2024-03-28T17:51:36.079 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563683c126bc] 2024-03-28T17:51:36.079 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563683bfc1ff] 2024-03-28T17:51:36.079 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f079ecd9488] 2024-03-28T17:51:36.079 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 16: (DispatchQueue::entry()+0x51f) [0x7f079ecd660f] 2024-03-28T17:51:36.079 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f079eda9e81] 2024-03-28T17:51:36.079 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f079d7c41ca] 2024-03-28T17:51:36.079 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 19: clone() 2024-03-28T17:51:36.079 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 2024-03-28T17:51:35.672+0000 7f079441d700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:51:36.079 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: in thread 7f079441d700 thread_name:ms_dispatch 2024-03-28T17:51:36.080 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 2024-03-28T17:51:36.080 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:36.080 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f079d7ced20] 2024-03-28T17:51:36.080 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 2: gsignal() 2024-03-28T17:51:36.080 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 3: abort() 2024-03-28T17:51:36.080 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f079ea474fb] 2024-03-28T17:51:36.080 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f079ea47667] 2024-03-28T17:51:36.080 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563683e44223] 2024-03-28T17:51:36.080 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563683cbe5d6] 2024-03-28T17:51:36.080 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563683cbeda1] 2024-03-28T17:51:36.080 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563683cc12d5] 2024-03-28T17:51:36.080 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563683d11233] 2024-03-28T17:51:36.080 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563683d13197] 2024-03-28T17:51:36.080 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563683c1211b] 2024-03-28T17:51:36.081 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563683c126bc] 2024-03-28T17:51:36.081 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563683bfc1ff] 2024-03-28T17:51:36.081 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f079ecd9488] 2024-03-28T17:51:36.081 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 16: (DispatchQueue::entry()+0x51f) [0x7f079ecd660f] 2024-03-28T17:51:36.081 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f079eda9e81] 2024-03-28T17:51:36.081 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f079d7c41ca] 2024-03-28T17:51:36.081 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 19: clone() 2024-03-28T17:51:36.081 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:51:36.081 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 2024-03-28T17:51:36.081 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: -1> 2024-03-28T17:51:35.670+0000 7f079441d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f079441d700 time 2024-03-28T17:51:35.669383+0000 2024-03-28T17:51:36.081 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:51:36.081 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 2024-03-28T17:51:36.081 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:36.081 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f079ea474a1] 2024-03-28T17:51:36.081 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f079ea47667] 2024-03-28T17:51:36.082 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563683e44223] 2024-03-28T17:51:36.082 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563683cbe5d6] 2024-03-28T17:51:36.082 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563683cbeda1] 2024-03-28T17:51:36.082 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563683cc12d5] 2024-03-28T17:51:36.082 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563683d11233] 2024-03-28T17:51:36.082 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563683d13197] 2024-03-28T17:51:36.082 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563683c1211b] 2024-03-28T17:51:36.082 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563683c126bc] 2024-03-28T17:51:36.082 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563683bfc1ff] 2024-03-28T17:51:36.082 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f079ecd9488] 2024-03-28T17:51:36.082 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 13: (DispatchQueue::entry()+0x51f) [0x7f079ecd660f] 2024-03-28T17:51:36.082 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f079eda9e81] 2024-03-28T17:51:36.082 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f079d7c41ca] 2024-03-28T17:51:36.083 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 16: clone() 2024-03-28T17:51:36.083 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 2024-03-28T17:51:36.083 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 0> 2024-03-28T17:51:35.672+0000 7f079441d700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:51:36.083 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: in thread 7f079441d700 thread_name:ms_dispatch 2024-03-28T17:51:36.083 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 2024-03-28T17:51:36.083 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:36.083 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f079d7ced20] 2024-03-28T17:51:36.083 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 2: gsignal() 2024-03-28T17:51:36.083 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 3: abort() 2024-03-28T17:51:36.083 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f079ea474fb] 2024-03-28T17:51:36.083 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f079ea47667] 2024-03-28T17:51:36.083 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563683e44223] 2024-03-28T17:51:36.083 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563683cbe5d6] 2024-03-28T17:51:36.083 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563683cbeda1] 2024-03-28T17:51:36.084 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563683cc12d5] 2024-03-28T17:51:36.084 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563683d11233] 2024-03-28T17:51:36.084 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563683d13197] 2024-03-28T17:51:36.084 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563683c1211b] 2024-03-28T17:51:36.084 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563683c126bc] 2024-03-28T17:51:36.084 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563683bfc1ff] 2024-03-28T17:51:36.084 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f079ecd9488] 2024-03-28T17:51:36.084 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 16: (DispatchQueue::entry()+0x51f) [0x7f079ecd660f] 2024-03-28T17:51:36.084 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f079eda9e81] 2024-03-28T17:51:36.084 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f079d7c41ca] 2024-03-28T17:51:36.084 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 19: clone() 2024-03-28T17:51:36.084 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:51:36.084 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 2024-03-28T17:51:36.084 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: -1> 2024-03-28T17:51:35.670+0000 7f079441d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f079441d700 time 2024-03-28T17:51:35.669383+0000 2024-03-28T17:51:36.084 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:51:36.085 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 2024-03-28T17:51:36.085 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:36.085 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f079ea474a1] 2024-03-28T17:51:36.085 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f079ea47667] 2024-03-28T17:51:36.085 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563683e44223] 2024-03-28T17:51:36.085 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563683cbe5d6] 2024-03-28T17:51:36.085 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563683cbeda1] 2024-03-28T17:51:36.085 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563683cc12d5] 2024-03-28T17:51:36.085 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563683d11233] 2024-03-28T17:51:36.085 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563683d13197] 2024-03-28T17:51:36.085 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563683c1211b] 2024-03-28T17:51:36.085 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563683c126bc] 2024-03-28T17:51:36.085 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563683bfc1ff] 2024-03-28T17:51:36.085 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f079ecd9488] 2024-03-28T17:51:36.085 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 13: (DispatchQueue::entry()+0x51f) [0x7f079ecd660f] 2024-03-28T17:51:36.086 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f079eda9e81] 2024-03-28T17:51:36.086 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f079d7c41ca] 2024-03-28T17:51:36.086 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 16: clone() 2024-03-28T17:51:36.086 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 2024-03-28T17:51:36.086 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 0> 2024-03-28T17:51:35.672+0000 7f079441d700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:51:36.086 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: in thread 7f079441d700 thread_name:ms_dispatch 2024-03-28T17:51:36.086 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 2024-03-28T17:51:36.086 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:36.086 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f079d7ced20] 2024-03-28T17:51:36.086 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 2: gsignal() 2024-03-28T17:51:36.086 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 3: abort() 2024-03-28T17:51:36.086 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f079ea474fb] 2024-03-28T17:51:36.086 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f079ea47667] 2024-03-28T17:51:36.086 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563683e44223] 2024-03-28T17:51:36.086 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563683cbe5d6] 2024-03-28T17:51:36.087 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563683cbeda1] 2024-03-28T17:51:36.087 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563683cc12d5] 2024-03-28T17:51:36.087 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563683d11233] 2024-03-28T17:51:36.087 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563683d13197] 2024-03-28T17:51:36.087 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563683c1211b] 2024-03-28T17:51:36.087 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563683c126bc] 2024-03-28T17:51:36.087 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563683bfc1ff] 2024-03-28T17:51:36.087 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f079ecd9488] 2024-03-28T17:51:36.087 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 16: (DispatchQueue::entry()+0x51f) [0x7f079ecd660f] 2024-03-28T17:51:36.087 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f079eda9e81] 2024-03-28T17:51:36.087 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f079d7c41ca] 2024-03-28T17:51:36.087 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 19: clone() 2024-03-28T17:51:36.087 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:51:36.087 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71368]: 2024-03-28T17:51:36.087 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:35 smithi151 podman[71571]: 2024-03-28 17:51:35.720176035 +0000 UTC m=+0.020381580 container died d9cc3c2eb7c0046b41dc8d65b12364f29bca93fed4be29ce2ac8e018a7bb22cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-28T17:51:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:36 smithi151 ceph-mon[36481]: pgmap v85: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-03-28T17:51:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:36 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:36 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:51:36.823 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:36 smithi151 podman[71571]: 2024-03-28 17:51:36.42851707 +0000 UTC m=+0.728722626 container remove d9cc3c2eb7c0046b41dc8d65b12364f29bca93fed4be29ce2ac8e018a7bb22cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-28T17:51:36.824 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:36 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:51:36.824 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:36 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.i.service: Failed with result 'exit-code'. 2024-03-28T17:51:36.841 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.86 MB/sec warmup 69 sec latency 64738.260 ms 2024-03-28T17:51:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:36 smithi071 ceph-mon[36597]: pgmap v85: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-03-28T17:51:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:36 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:37.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:36 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:51:37.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:36 smithi045 ceph-mon[31773]: pgmap v85: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-03-28T17:51:37.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:36 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:37.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:36 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:51:37.841 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.85 MB/sec warmup 70 sec latency 65738.404 ms 2024-03-28T17:51:38.842 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.83 MB/sec warmup 71 sec latency 66738.556 ms 2024-03-28T17:51:39.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:38 smithi071 ceph-mon[36597]: pgmap v86: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-03-28T17:51:39.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:38 smithi151 ceph-mon[36481]: pgmap v86: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-03-28T17:51:39.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:38 smithi045 ceph-mon[31773]: pgmap v86: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-03-28T17:51:39.842 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.82 MB/sec warmup 72 sec latency 67738.693 ms 2024-03-28T17:51:40.842 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.81 MB/sec warmup 73 sec latency 68738.931 ms 2024-03-28T17:51:41.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:40 smithi071 ceph-mon[36597]: pgmap v87: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-03-28T17:51:41.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:40 smithi151 ceph-mon[36481]: pgmap v87: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-03-28T17:51:41.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:40 smithi045 ceph-mon[31773]: pgmap v87: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-03-28T17:51:41.842 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.80 MB/sec warmup 74 sec latency 69739.063 ms 2024-03-28T17:51:42.842 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.79 MB/sec warmup 75 sec latency 70739.228 ms 2024-03-28T17:51:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:42 smithi071 ceph-mon[36597]: pgmap v88: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-03-28T17:51:43.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:42 smithi151 ceph-mon[36481]: pgmap v88: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-03-28T17:51:43.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:42 smithi045 ceph-mon[31773]: pgmap v88: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-03-28T17:51:43.842 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.78 MB/sec warmup 76 sec latency 71739.360 ms 2024-03-28T17:51:44.842 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.77 MB/sec warmup 77 sec latency 72739.501 ms 2024-03-28T17:51:44.968 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:44 smithi045 ceph-mon[31773]: pgmap v89: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-03-28T17:51:45.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:44 smithi071 ceph-mon[36597]: pgmap v89: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-03-28T17:51:45.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:44 smithi151 ceph-mon[36481]: pgmap v89: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-03-28T17:51:45.843 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.76 MB/sec warmup 78 sec latency 73739.647 ms 2024-03-28T17:51:46.843 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.75 MB/sec warmup 79 sec latency 74739.796 ms 2024-03-28T17:51:47.059 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:46 smithi151 ceph-mon[36481]: pgmap v90: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-28T17:51:47.059 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:46 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.i.service: Scheduled restart job, restart counter is at 3. 2024-03-28T17:51:47.059 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:46 smithi151 systemd[1]: Stopped Ceph mds.i for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:51:47.059 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:46 smithi151 systemd[1]: Starting Ceph mds.i for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:51:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:46 smithi071 ceph-mon[36597]: pgmap v90: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-28T17:51:47.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:46 smithi045 ceph-mon[31773]: pgmap v90: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-28T17:51:47.323 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:47 smithi151 podman[71668]: 2024-03-28 17:51:46.962500972 +0000 UTC m=+0.018041663 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:51:47.323 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:47 smithi151 podman[71668]: 2024-03-28 17:51:47.087410278 +0000 UTC m=+0.142950969 container create 9d4190da2a31fe91d44bb8b7e2c30cb02a9f269bf295bd90376845a87c028890 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T17:51:47.573 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:47 smithi151 podman[71668]: 2024-03-28 17:51:47.323463135 +0000 UTC m=+0.379003826 container init 9d4190da2a31fe91d44bb8b7e2c30cb02a9f269bf295bd90376845a87c028890 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-28T17:51:47.573 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:47 smithi151 podman[71668]: 2024-03-28 17:51:47.326521771 +0000 UTC m=+0.382062460 container start 9d4190da2a31fe91d44bb8b7e2c30cb02a9f269bf295bd90376845a87c028890 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T17:51:47.573 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:47 smithi151 ceph-mds[71687]: -- 172.21.15.151:0/1944979678 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559c24d75a40 con 0x559c24d49000 2024-03-28T17:51:47.574 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:47 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: starting mds.i at 2024-03-28T17:51:47.574 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:47 smithi151 bash[71668]: 9d4190da2a31fe91d44bb8b7e2c30cb02a9f269bf295bd90376845a87c028890 2024-03-28T17:51:47.574 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:51:47 smithi151 systemd[1]: Started Ceph mds.i for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:51:47.843 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.74 MB/sec warmup 80 sec latency 75739.888 ms 2024-03-28T17:51:48.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:47 smithi071 ceph-mon[36597]: daemon mds.i restarted 2024-03-28T17:51:48.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:47 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:51:48.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:47 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T17:51:48.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:47 smithi071 ceph-mon[36597]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T17:51:48.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:47 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6838/4179697756,v1:172.21.15.151:6839/4179697756] up:boot 2024-03-28T17:51:48.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:47 smithi071 ceph-mon[36597]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-03-28T17:51:48.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:47 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:51:48.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:47 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:47 smithi151 ceph-mon[36481]: daemon mds.i restarted 2024-03-28T17:51:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:47 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:51:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:47 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T17:51:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:47 smithi151 ceph-mon[36481]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T17:51:48.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:47 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6838/4179697756,v1:172.21.15.151:6839/4179697756] up:boot 2024-03-28T17:51:48.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:47 smithi151 ceph-mon[36481]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-03-28T17:51:48.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:47 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:51:48.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:47 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:47 smithi045 ceph-mon[31773]: daemon mds.i restarted 2024-03-28T17:51:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:47 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:51:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:47 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T17:51:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:47 smithi045 ceph-mon[31773]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T17:51:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:47 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6838/4179697756,v1:172.21.15.151:6839/4179697756] up:boot 2024-03-28T17:51:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:47 smithi045 ceph-mon[31773]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-03-28T17:51:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:47 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:51:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:47 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:48.843 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.73 MB/sec warmup 81 sec latency 76739.988 ms 2024-03-28T17:51:49.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:48 smithi071 ceph-mon[36597]: pgmap v91: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-28T17:51:49.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:48 smithi151 ceph-mon[36481]: pgmap v91: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-28T17:51:49.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:48 smithi045 ceph-mon[31773]: pgmap v91: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-28T17:51:49.843 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.72 MB/sec warmup 82 sec latency 77740.110 ms 2024-03-28T17:51:50.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:49 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/4265871690,v1:172.21.15.151:6835/4265871690] up:resolve 2024-03-28T17:51:50.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:49 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:50.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:49 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/4265871690,v1:172.21.15.151:6835/4265871690] up:resolve 2024-03-28T17:51:50.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:49 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:50.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:49 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/4265871690,v1:172.21.15.151:6835/4265871690] up:resolve 2024-03-28T17:51:50.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:49 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:50.843 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.71 MB/sec warmup 83 sec latency 78740.257 ms 2024-03-28T17:51:51.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:50 smithi071 ceph-mon[36597]: pgmap v93: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-28T17:51:51.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:50 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:51:51.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:50 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/4265871690,v1:172.21.15.151:6835/4265871690] up:reconnect 2024-03-28T17:51:51.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:50 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:51.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:50 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:51:51.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:50 smithi151 ceph-mon[36481]: pgmap v93: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-28T17:51:51.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:50 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:51:51.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:50 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/4265871690,v1:172.21.15.151:6835/4265871690] up:reconnect 2024-03-28T17:51:51.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:50 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:51.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:50 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:51:51.137 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:50 smithi045 ceph-mon[31773]: pgmap v93: 97 pgs: 97 active+clean; 27 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-28T17:51:51.137 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:50 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:51:51.137 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:50 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/4265871690,v1:172.21.15.151:6835/4265871690] up:reconnect 2024-03-28T17:51:51.137 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:50 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:51.137 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:50 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:51:51.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb97cc4e700 time 2024-03-28T17:51:51.142348+0000 2024-03-28T17:51:51.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:51:51.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:51.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb9872784a1] 2024-03-28T17:51:51.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb987278667] 2024-03-28T17:51:51.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561165753223] 2024-03-28T17:51:51.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5611655cd5d6] 2024-03-28T17:51:51.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5611655cdda1] 2024-03-28T17:51:51.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5611655d02d5] 2024-03-28T17:51:51.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561165620233] 2024-03-28T17:51:51.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561165622197] 2024-03-28T17:51:51.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56116552111b] 2024-03-28T17:51:51.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5611655216bc] 2024-03-28T17:51:51.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56116550b1ff] 2024-03-28T17:51:51.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb98750a488] 2024-03-28T17:51:51.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 13: (DispatchQueue::entry()+0x51f) [0x7fb98750760f] 2024-03-28T17:51:51.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb9875dae81] 2024-03-28T17:51:51.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb985ff51ca] 2024-03-28T17:51:51.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 16: clone() 2024-03-28T17:51:51.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 2024-03-28T17:51:51.144+0000 7fb97cc4e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb97cc4e700 time 2024-03-28T17:51:51.142348+0000 2024-03-28T17:51:51.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:51:51.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 2024-03-28T17:51:51.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:51.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb9872784a1] 2024-03-28T17:51:51.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb987278667] 2024-03-28T17:51:51.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561165753223] 2024-03-28T17:51:51.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5611655cd5d6] 2024-03-28T17:51:51.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5611655cdda1] 2024-03-28T17:51:51.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5611655d02d5] 2024-03-28T17:51:51.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561165620233] 2024-03-28T17:51:51.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561165622197] 2024-03-28T17:51:51.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56116552111b] 2024-03-28T17:51:51.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5611655216bc] 2024-03-28T17:51:51.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56116550b1ff] 2024-03-28T17:51:51.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb98750a488] 2024-03-28T17:51:51.522 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 13: (DispatchQueue::entry()+0x51f) [0x7fb98750760f] 2024-03-28T17:51:51.522 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb9875dae81] 2024-03-28T17:51:51.522 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb985ff51ca] 2024-03-28T17:51:51.522 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 16: clone() 2024-03-28T17:51:51.522 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 2024-03-28T17:51:51.522 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: *** Caught signal (Aborted) ** 2024-03-28T17:51:51.522 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: in thread 7fb97cc4e700 thread_name:ms_dispatch 2024-03-28T17:51:51.522 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:51.522 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb985fffd20] 2024-03-28T17:51:51.523 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 2: gsignal() 2024-03-28T17:51:51.523 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 3: abort() 2024-03-28T17:51:51.523 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb9872784fb] 2024-03-28T17:51:51.523 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb987278667] 2024-03-28T17:51:51.523 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561165753223] 2024-03-28T17:51:51.523 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5611655cd5d6] 2024-03-28T17:51:51.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5611655cdda1] 2024-03-28T17:51:51.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5611655d02d5] 2024-03-28T17:51:51.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561165620233] 2024-03-28T17:51:51.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561165622197] 2024-03-28T17:51:51.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56116552111b] 2024-03-28T17:51:51.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5611655216bc] 2024-03-28T17:51:51.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56116550b1ff] 2024-03-28T17:51:51.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb98750a488] 2024-03-28T17:51:51.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 16: (DispatchQueue::entry()+0x51f) [0x7fb98750760f] 2024-03-28T17:51:51.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb9875dae81] 2024-03-28T17:51:51.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb985ff51ca] 2024-03-28T17:51:51.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 19: clone() 2024-03-28T17:51:51.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 2024-03-28T17:51:51.146+0000 7fb97cc4e700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:51:51.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: in thread 7fb97cc4e700 thread_name:ms_dispatch 2024-03-28T17:51:51.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 2024-03-28T17:51:51.525 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:51.525 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb985fffd20] 2024-03-28T17:51:51.525 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 2: gsignal() 2024-03-28T17:51:51.525 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 3: abort() 2024-03-28T17:51:51.525 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb9872784fb] 2024-03-28T17:51:51.525 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb987278667] 2024-03-28T17:51:51.525 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561165753223] 2024-03-28T17:51:51.525 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5611655cd5d6] 2024-03-28T17:51:51.525 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5611655cdda1] 2024-03-28T17:51:51.525 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5611655d02d5] 2024-03-28T17:51:51.525 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561165620233] 2024-03-28T17:51:51.525 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561165622197] 2024-03-28T17:51:51.525 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56116552111b] 2024-03-28T17:51:51.525 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5611655216bc] 2024-03-28T17:51:51.526 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56116550b1ff] 2024-03-28T17:51:51.526 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb98750a488] 2024-03-28T17:51:51.526 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 16: (DispatchQueue::entry()+0x51f) [0x7fb98750760f] 2024-03-28T17:51:51.526 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb9875dae81] 2024-03-28T17:51:51.526 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb985ff51ca] 2024-03-28T17:51:51.526 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 19: clone() 2024-03-28T17:51:51.526 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:51:51.526 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 2024-03-28T17:51:51.526 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: -1> 2024-03-28T17:51:51.144+0000 7fb97cc4e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb97cc4e700 time 2024-03-28T17:51:51.142348+0000 2024-03-28T17:51:51.526 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:51:51.526 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 2024-03-28T17:51:51.526 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:51.526 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb9872784a1] 2024-03-28T17:51:51.526 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb987278667] 2024-03-28T17:51:51.526 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561165753223] 2024-03-28T17:51:51.527 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5611655cd5d6] 2024-03-28T17:51:51.527 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5611655cdda1] 2024-03-28T17:51:51.527 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5611655d02d5] 2024-03-28T17:51:51.527 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561165620233] 2024-03-28T17:51:51.527 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561165622197] 2024-03-28T17:51:51.527 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56116552111b] 2024-03-28T17:51:51.527 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5611655216bc] 2024-03-28T17:51:51.527 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56116550b1ff] 2024-03-28T17:51:51.527 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb98750a488] 2024-03-28T17:51:51.527 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 13: (DispatchQueue::entry()+0x51f) [0x7fb98750760f] 2024-03-28T17:51:51.527 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb9875dae81] 2024-03-28T17:51:51.527 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb985ff51ca] 2024-03-28T17:51:51.527 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 16: clone() 2024-03-28T17:51:51.527 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 2024-03-28T17:51:51.527 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 0> 2024-03-28T17:51:51.146+0000 7fb97cc4e700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:51:51.528 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: in thread 7fb97cc4e700 thread_name:ms_dispatch 2024-03-28T17:51:51.528 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 2024-03-28T17:51:51.528 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:51.528 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb985fffd20] 2024-03-28T17:51:51.528 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 2: gsignal() 2024-03-28T17:51:51.528 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 3: abort() 2024-03-28T17:51:51.528 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb9872784fb] 2024-03-28T17:51:51.528 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb987278667] 2024-03-28T17:51:51.528 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561165753223] 2024-03-28T17:51:51.528 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5611655cd5d6] 2024-03-28T17:51:51.528 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5611655cdda1] 2024-03-28T17:51:51.528 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5611655d02d5] 2024-03-28T17:51:51.528 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561165620233] 2024-03-28T17:51:51.528 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561165622197] 2024-03-28T17:51:51.528 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56116552111b] 2024-03-28T17:51:51.528 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5611655216bc] 2024-03-28T17:51:51.529 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56116550b1ff] 2024-03-28T17:51:51.529 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb98750a488] 2024-03-28T17:51:51.529 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 16: (DispatchQueue::entry()+0x51f) [0x7fb98750760f] 2024-03-28T17:51:51.529 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb9875dae81] 2024-03-28T17:51:51.529 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb985ff51ca] 2024-03-28T17:51:51.529 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 19: clone() 2024-03-28T17:51:51.529 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:51:51.529 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 2024-03-28T17:51:51.529 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: -1> 2024-03-28T17:51:51.144+0000 7fb97cc4e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb97cc4e700 time 2024-03-28T17:51:51.142348+0000 2024-03-28T17:51:51.529 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:51:51.529 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 2024-03-28T17:51:51.529 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:51.529 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb9872784a1] 2024-03-28T17:51:51.529 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb987278667] 2024-03-28T17:51:51.529 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561165753223] 2024-03-28T17:51:51.529 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5611655cd5d6] 2024-03-28T17:51:51.530 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5611655cdda1] 2024-03-28T17:51:51.530 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5611655d02d5] 2024-03-28T17:51:51.530 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561165620233] 2024-03-28T17:51:51.530 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561165622197] 2024-03-28T17:51:51.530 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56116552111b] 2024-03-28T17:51:51.530 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5611655216bc] 2024-03-28T17:51:51.530 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56116550b1ff] 2024-03-28T17:51:51.530 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb98750a488] 2024-03-28T17:51:51.530 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 13: (DispatchQueue::entry()+0x51f) [0x7fb98750760f] 2024-03-28T17:51:51.530 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb9875dae81] 2024-03-28T17:51:51.530 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb985ff51ca] 2024-03-28T17:51:51.530 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 16: clone() 2024-03-28T17:51:51.530 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 2024-03-28T17:51:51.530 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 0> 2024-03-28T17:51:51.146+0000 7fb97cc4e700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:51:51.531 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: in thread 7fb97cc4e700 thread_name:ms_dispatch 2024-03-28T17:51:51.531 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 2024-03-28T17:51:51.531 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:51:51.531 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb985fffd20] 2024-03-28T17:51:51.531 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 2: gsignal() 2024-03-28T17:51:51.531 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 3: abort() 2024-03-28T17:51:51.531 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb9872784fb] 2024-03-28T17:51:51.531 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb987278667] 2024-03-28T17:51:51.531 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561165753223] 2024-03-28T17:51:51.531 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5611655cd5d6] 2024-03-28T17:51:51.531 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5611655cdda1] 2024-03-28T17:51:51.531 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5611655d02d5] 2024-03-28T17:51:51.531 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561165620233] 2024-03-28T17:51:51.531 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561165622197] 2024-03-28T17:51:51.532 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56116552111b] 2024-03-28T17:51:51.532 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5611655216bc] 2024-03-28T17:51:51.532 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56116550b1ff] 2024-03-28T17:51:51.532 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb98750a488] 2024-03-28T17:51:51.532 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 16: (DispatchQueue::entry()+0x51f) [0x7fb98750760f] 2024-03-28T17:51:51.532 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb9875dae81] 2024-03-28T17:51:51.532 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb985ff51ca] 2024-03-28T17:51:51.532 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 19: clone() 2024-03-28T17:51:51.532 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:51:51.532 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71532]: 2024-03-28T17:51:51.532 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 podman[71721]: 2024-03-28 17:51:51.200858031 +0000 UTC m=+0.020937729 container died 80be162d30c757c5605a4d191c6cc74af519d7fa51fdd306b74f77783c11c5f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD) 2024-03-28T17:51:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/4265871690,v1:172.21.15.151:6835/4265871690] up:rejoin 2024-03-28T17:51:51.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:51.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-mon[36481]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-28T17:51:51.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:51:51.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:51:51.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/4265871690,v1:172.21.15.151:6835/4265871690] up:active 2024-03-28T17:51:51.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:51 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:51:51.843 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.70 MB/sec warmup 84 sec latency 79740.394 ms 2024-03-28T17:51:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:51 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/4265871690,v1:172.21.15.151:6835/4265871690] up:rejoin 2024-03-28T17:51:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:51 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:52.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:51 smithi071 ceph-mon[36597]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-28T17:51:52.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:51 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:51:52.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:51 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:51:52.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:51 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/4265871690,v1:172.21.15.151:6835/4265871690] up:active 2024-03-28T17:51:52.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:51 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:51:52.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:51 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/4265871690,v1:172.21.15.151:6835/4265871690] up:rejoin 2024-03-28T17:51:52.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:51 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:51:52.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:51 smithi045 ceph-mon[31773]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-28T17:51:52.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:51 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:51:52.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:51 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:51:52.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:51 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/4265871690,v1:172.21.15.151:6835/4265871690] up:active 2024-03-28T17:51:52.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:51 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:51:52.323 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 podman[71721]: 2024-03-28 17:51:51.88719484 +0000 UTC m=+0.707274528 container remove 80be162d30c757c5605a4d191c6cc74af519d7fa51fdd306b74f77783c11c5f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-28T17:51:52.323 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:51 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:51:52.323 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:51:52 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-28T17:51:52.844 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.70 MB/sec warmup 85 sec latency 80740.541 ms 2024-03-28T17:51:53.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:52 smithi071 ceph-mon[36597]: pgmap v94: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:51:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:52 smithi151 ceph-mon[36481]: pgmap v94: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:51:53.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:52 smithi045 ceph-mon[31773]: pgmap v94: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:51:53.844 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.69 MB/sec warmup 86 sec latency 81740.697 ms 2024-03-28T17:51:54.844 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.68 MB/sec warmup 87 sec latency 82740.839 ms 2024-03-28T17:51:55.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:54 smithi071 ceph-mon[36597]: pgmap v95: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:51:55.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:54 smithi151 ceph-mon[36481]: pgmap v95: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:51:55.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:54 smithi045 ceph-mon[31773]: pgmap v95: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:51:55.844 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.67 MB/sec warmup 88 sec latency 83740.938 ms 2024-03-28T17:51:56.844 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.67 MB/sec warmup 89 sec latency 84741.070 ms 2024-03-28T17:51:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:56 smithi071 ceph-mon[36597]: pgmap v96: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-28T17:51:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:56 smithi151 ceph-mon[36481]: pgmap v96: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-28T17:51:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:56 smithi045 ceph-mon[31773]: pgmap v96: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-28T17:51:57.844 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.66 MB/sec warmup 90 sec latency 85741.206 ms 2024-03-28T17:51:58.844 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.65 MB/sec warmup 91 sec latency 86741.345 ms 2024-03-28T17:51:59.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:58 smithi071 ceph-mon[36597]: pgmap v97: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-28T17:51:59.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:58 smithi151 ceph-mon[36481]: pgmap v97: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-28T17:51:59.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:58 smithi045 ceph-mon[31773]: pgmap v97: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-28T17:51:59.844 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.64 MB/sec warmup 92 sec latency 87741.464 ms 2024-03-28T17:52:00.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:51:59 smithi071 ceph-mon[36597]: pgmap v98: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-03-28T17:52:00.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:51:59 smithi151 ceph-mon[36481]: pgmap v98: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-03-28T17:52:00.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:51:59 smithi045 ceph-mon[31773]: pgmap v98: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-03-28T17:52:00.845 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.64 MB/sec warmup 93 sec latency 88741.596 ms 2024-03-28T17:52:01.845 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.63 MB/sec warmup 94 sec latency 89741.738 ms 2024-03-28T17:52:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:02 smithi071 ceph-mon[36597]: pgmap v99: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-28T17:52:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:02 smithi151 ceph-mon[36481]: pgmap v99: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-28T17:52:02.398 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:52:02.398 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:52:02.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:02 smithi045 ceph-mon[31773]: pgmap v99: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-28T17:52:02.578 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:02 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 3. 2024-03-28T17:52:02.579 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:02 smithi151 systemd[1]: Stopped Ceph mds.f for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:52:02.579 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:02 smithi151 systemd[1]: Starting Ceph mds.f for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:52:02.579 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:02 smithi151 podman[71819]: 2024-03-28 17:52:02.462314008 +0000 UTC m=+0.018040723 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:52:02.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.636+0000 7f0e43e00640 1 -- 172.21.15.45:0/3060888594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e44109730 msgr2=0x7f0e44109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:52:02.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.636+0000 7f0e43e00640 1 --2- 172.21.15.45:0/3060888594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e44109730 0x7f0e44109b10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0e34009cd0 tx=0x7f0e3402c0f0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.636+0000 7f0e43e00640 1 -- 172.21.15.45:0/3060888594 shutdown_connections 2024-03-28T17:52:02.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.636+0000 7f0e43e00640 1 --2- 172.21.15.45:0/3060888594 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e441040e0 0x7f0e441105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.636+0000 7f0e43e00640 1 --2- 172.21.15.45:0/3060888594 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e44103720 0x7f0e44103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.636+0000 7f0e43e00640 1 --2- 172.21.15.45:0/3060888594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e44109730 0x7f0e44109b10 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.636+0000 7f0e43e00640 1 -- 172.21.15.45:0/3060888594 >> 172.21.15.45:0/3060888594 conn(0x7f0e440ff1b0 msgr2=0x7f0e441015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:52:02.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.636+0000 7f0e43e00640 1 -- 172.21.15.45:0/3060888594 shutdown_connections 2024-03-28T17:52:02.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.636+0000 7f0e43e00640 1 -- 172.21.15.45:0/3060888594 wait complete. 2024-03-28T17:52:02.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.637+0000 7f0e43e00640 1 Processor -- start 2024-03-28T17:52:02.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.638+0000 7f0e43e00640 1 -- start start 2024-03-28T17:52:02.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.638+0000 7f0e43e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e44103720 0x7f0e4419a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:02.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.638+0000 7f0e43e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e441040e0 0x7f0e4419a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:02.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.638+0000 7f0e43e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e44109730 0x7f0e4419ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:02.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.638+0000 7f0e43e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e44113b40 con 0x7f0e44103720 2024-03-28T17:52:02.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.638+0000 7f0e43e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e441139c0 con 0x7f0e441040e0 2024-03-28T17:52:02.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.638+0000 7f0e43e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e44113cc0 con 0x7f0e44109730 2024-03-28T17:52:02.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.638+0000 7f0e42000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e441040e0 0x7f0e4419a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:02.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.639+0000 7f0e42000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e441040e0 0x7f0e4419a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53358/0 (socket says 172.21.15.45:53358) 2024-03-28T17:52:02.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.639+0000 7f0e42000640 1 -- 172.21.15.45:0/3734131153 learned_addr learned my addr 172.21.15.45:0/3734131153 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:52:02.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.639+0000 7f0e42a00640 1 --2- 172.21.15.45:0/3734131153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e44103720 0x7f0e4419a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:02.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.639+0000 7f0e43400640 1 --2- 172.21.15.45:0/3734131153 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e44109730 0x7f0e4419ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:02.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.639+0000 7f0e42000640 1 -- 172.21.15.45:0/3734131153 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e44109730 msgr2=0x7f0e4419ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:52:02.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.639+0000 7f0e42000640 1 --2- 172.21.15.45:0/3734131153 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e44109730 0x7f0e4419ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.639+0000 7f0e42000640 1 -- 172.21.15.45:0/3734131153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e44103720 msgr2=0x7f0e4419a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:52:02.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.639+0000 7f0e42000640 1 --2- 172.21.15.45:0/3734131153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e44103720 0x7f0e4419a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.639+0000 7f0e42000640 1 -- 172.21.15.45:0/3734131153 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e4419f390 con 0x7f0e441040e0 2024-03-28T17:52:02.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.639+0000 7f0e42000640 1 --2- 172.21.15.45:0/3734131153 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e441040e0 0x7f0e4419a8e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00e9f0 tx=0x7f0e2c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:52:02.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.640+0000 7f0e3b400640 1 -- 172.21.15.45:0/3734131153 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c00ccc0 con 0x7f0e441040e0 2024-03-28T17:52:02.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.640+0000 7f0e3b400640 1 -- 172.21.15.45:0/3734131153 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e2c00ce60 con 0x7f0e441040e0 2024-03-28T17:52:02.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.640+0000 7f0e43e00640 1 -- 172.21.15.45:0/3734131153 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e4419f680 con 0x7f0e441040e0 2024-03-28T17:52:02.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.640+0000 7f0e3b400640 1 -- 172.21.15.45:0/3734131153 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c010690 con 0x7f0e441040e0 2024-03-28T17:52:02.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.640+0000 7f0e43e00640 1 -- 172.21.15.45:0/3734131153 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e441a6f10 con 0x7f0e441040e0 2024-03-28T17:52:02.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.643+0000 7f0e4c615740 1 Processor -- start 2024-03-28T17:52:02.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.643+0000 7f0e3b400640 1 -- 172.21.15.45:0/3734131153 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f0e2c0046c0 con 0x7f0e441040e0 2024-03-28T17:52:02.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.643+0000 7f0e4c615740 1 -- start start 2024-03-28T17:52:02.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.643+0000 7f0e4c615740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e019ddd0 0x55d5e019cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:02.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.643+0000 7f0e4c615740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55d5e019d0f0 0x55d5e019d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:02.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.644+0000 7f0e4c615740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e01a6460 0x55d5e01a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:02.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.644+0000 7f0e4c615740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e0101250 con 0x55d5e019ddd0 2024-03-28T17:52:02.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.644+0000 7f0e4c615740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e00a9a30 con 0x55d5e01a6460 2024-03-28T17:52:02.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.644+0000 7f0e4c615740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e00fd2f0 con 0x55d5e019d0f0 2024-03-28T17:52:02.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.644+0000 7f0e43400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e019ddd0 0x55d5e019cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:02.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.644+0000 7f0e42000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e01a6460 0x55d5e01a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:02.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.644+0000 7f0e43400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e019ddd0 0x55d5e019cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51106/0 (socket says 172.21.15.45:51106) 2024-03-28T17:52:02.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.644+0000 7f0e43400640 1 -- 172.21.15.45:0/1899258591 learned_addr learned my addr 172.21.15.45:0/1899258591 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:52:02.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.644+0000 7f0e42a00640 1 --2- 172.21.15.45:0/1899258591 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55d5e019d0f0 0x55d5e019d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:02.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.644+0000 7f0e43400640 1 -- 172.21.15.45:0/1899258591 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55d5e019d0f0 msgr2=0x55d5e019d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:52:02.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.644+0000 7f0e43400640 1 --2- 172.21.15.45:0/1899258591 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55d5e019d0f0 0x55d5e019d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.644+0000 7f0e43400640 1 -- 172.21.15.45:0/1899258591 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e01a6460 msgr2=0x55d5e01a8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:52:02.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.644+0000 7f0e43400640 1 --2- 172.21.15.45:0/1899258591 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e01a6460 0x55d5e01a8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.644+0000 7f0e43400640 1 -- 172.21.15.45:0/1899258591 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d5e0094690 con 0x55d5e019ddd0 2024-03-28T17:52:02.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.645+0000 7f0e3b400640 1 --2- 172.21.15.45:0/3734131153 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f0e280a29f0 0x7f0e280a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:02.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.645+0000 7f0e43400640 1 --2- 172.21.15.45:0/1899258591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e019ddd0 0x55d5e019cb20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c009c10 tx=0x7f0e3c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8db010ed0d908863 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:52:02.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.645+0000 7f0e3b400640 1 -- 172.21.15.45:0/3734131153 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8942+0+0 (secure 0 0 0) 0x7f0e2c0cf360 con 0x7f0e441040e0 2024-03-28T17:52:02.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.645+0000 7f0e33e00640 1 -- 172.21.15.45:0/1899258591 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c004230 con 0x55d5e019ddd0 2024-03-28T17:52:02.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.645+0000 7f0e42a00640 1 --2- 172.21.15.45:0/3734131153 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f0e280a29f0 0x7f0e280a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:02.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.645+0000 7f0e33e00640 1 -- 172.21.15.45:0/1899258591 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e3c0043d0 con 0x55d5e019ddd0 2024-03-28T17:52:02.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.645+0000 7f0e33e00640 1 -- 172.21.15.45:0/1899258591 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c0197a0 con 0x55d5e019ddd0 2024-03-28T17:52:02.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.646+0000 7f0e4c615740 1 -- 172.21.15.45:0/1899258591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e019ddd0 msgr2=0x55d5e019cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:52:02.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.646+0000 7f0e4c615740 1 --2- 172.21.15.45:0/1899258591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e019ddd0 0x55d5e019cb20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c009c10 tx=0x7f0e3c011810 comp rx=0 tx=0).stop 2024-03-28T17:52:02.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.646+0000 7f0e4c615740 1 -- 172.21.15.45:0/1899258591 shutdown_connections 2024-03-28T17:52:02.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.646+0000 7f0e4c615740 1 --2- 172.21.15.45:0/1899258591 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e01a6460 0x55d5e01a8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.646+0000 7f0e4c615740 1 --2- 172.21.15.45:0/1899258591 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55d5e019d0f0 0x55d5e019d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.646+0000 7f0e4c615740 1 --2- 172.21.15.45:0/1899258591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e019ddd0 0x55d5e019cb20 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.646+0000 7f0e4c615740 1 -- 172.21.15.45:0/1899258591 >> 172.21.15.45:0/1899258591 conn(0x55d5e0164200 msgr2=0x55d5e0197b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:52:02.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.646+0000 7f0e42a00640 1 --2- 172.21.15.45:0/3734131153 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f0e280a29f0 0x7f0e280a4eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0e34009ca0 tx=0x7f0e340093a0 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:52:02.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.646+0000 7f0e4c615740 1 -- 172.21.15.45:0/1899258591 shutdown_connections 2024-03-28T17:52:02.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.646+0000 7f0e4c615740 1 -- 172.21.15.45:0/1899258591 wait complete. 2024-03-28T17:52:02.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.646+0000 7f0e4c615740 1 -- start start 2024-03-28T17:52:02.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.647+0000 7f0e4c615740 1 Processor -- start 2024-03-28T17:52:02.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.647+0000 7f0e4c615740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e019d0f0 0x55d5e01a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:02.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.647+0000 7f0e4c615740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55d5e019ddd0 0x55d5e01a2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:02.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.647+0000 7f0e4c615740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e01a6460 0x55d5e01a3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:02.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.647+0000 7f0e4c615740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e01a92d0 con 0x55d5e019d0f0 2024-03-28T17:52:02.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.647+0000 7f0e4c615740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e00bce80 con 0x55d5e01a6460 2024-03-28T17:52:02.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.647+0000 7f0e4c615740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e01a9450 con 0x55d5e019ddd0 2024-03-28T17:52:02.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.647+0000 7f0e4c615740 10 client.0 _ll_register_callbacks cb 0x55d5e00cf130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:52:02.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.647+0000 7f0e42a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e019d0f0 0x55d5e01a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:02.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.647+0000 7f0e43400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e01a6460 0x55d5e01a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:02.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.648+0000 7f0e43400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e01a6460 0x55d5e01a3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53374/0 (socket says 172.21.15.45:53374) 2024-03-28T17:52:02.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.648+0000 7f0e43400640 1 -- 172.21.15.45:0/2159864595 learned_addr learned my addr 172.21.15.45:0/2159864595 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:52:02.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.648+0000 7f0e42000640 1 --2- 172.21.15.45:0/2159864595 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55d5e019ddd0 0x55d5e01a2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:02.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.648+0000 7f0e42a00640 1 -- 172.21.15.45:0/2159864595 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55d5e019ddd0 msgr2=0x55d5e01a2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:52:02.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.648+0000 7f0e42a00640 1 --2- 172.21.15.45:0/2159864595 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55d5e019ddd0 0x55d5e01a2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.648+0000 7f0e42a00640 1 -- 172.21.15.45:0/2159864595 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e01a6460 msgr2=0x55d5e01a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:52:02.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.648+0000 7f0e42a00640 1 --2- 172.21.15.45:0/2159864595 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e01a6460 0x55d5e01a3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.648+0000 7f0e42a00640 1 -- 172.21.15.45:0/2159864595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d5dfedf7d0 con 0x55d5e019d0f0 2024-03-28T17:52:02.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.648+0000 7f0e43400640 1 --2- 172.21.15.45:0/2159864595 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e01a6460 0x55d5e01a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:52:02.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.648+0000 7f0e42000640 1 --2- 172.21.15.45:0/2159864595 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55d5e019ddd0 0x55d5e01a2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:52:02.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.648+0000 7f0e42a00640 1 --2- 172.21.15.45:0/2159864595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e019d0f0 0x55d5e01a5b70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0e34009bf0 tx=0x7f0e340095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=374e42c088c8892c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:52:02.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.648+0000 7f0e30c00640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-28T17:52:02.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.649+0000 7f0e30c00640 1 -- 172.21.15.45:0/2159864595 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e34051070 con 0x55d5e019d0f0 2024-03-28T17:52:02.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.649+0000 7f0e30c00640 1 -- 172.21.15.45:0/2159864595 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8942+0+0 (secure 0 0 0) 0x7f0e34003970 con 0x55d5e019d0f0 2024-03-28T17:52:02.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.649+0000 7f0e4c615740 1 -- 172.21.15.45:0/2159864595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0e3c0043d0 con 0x55d5e019d0f0 2024-03-28T17:52:02.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.649+0000 7f0e30c00640 1 -- 172.21.15.45:0/2159864595 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e3405a050 con 0x55d5e019d0f0 2024-03-28T17:52:02.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.649+0000 7f0e4c615740 10 client.15789 fetch_fsmap learned FSMap version 58 2024-03-28T17:52:02.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.649+0000 7f0e4c615740 1 -- 172.21.15.45:0/2159864595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) v3 -- 0x55d5dfe153f0 con 0x55d5e019d0f0 2024-03-28T17:52:02.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.649+0000 7f0e30c00640 1 -- 172.21.15.45:0/2159864595 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e3c0043d0 con 0x55d5e019d0f0 2024-03-28T17:52:02.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.649+0000 7f0e30c00640 1 -- 172.21.15.45:0/2159864595 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 58) v1 ==== 7922+0+0 (secure 0 0 0) 0x7f0e34054020 con 0x55d5e019d0f0 2024-03-28T17:52:02.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.650+0000 7f0e4c615740 10 client.15789 fetch_fsmap finished waiting for FSMap version 58 2024-03-28T17:52:02.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.650+0000 7f0e4c615740 10 client.15789 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:52:02.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.650+0000 7f0e4c615740 20 client.15789 populate_metadata read hostname 'smithi045' 2024-03-28T17:52:02.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.650+0000 7f0e4c615740 1 --2- 172.21.15.45:0/2159864595 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55d5e01b63a0 0x55d5e01b87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:02.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.650+0000 7f0e4c615740 4 client.15789 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:52:02.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.650+0000 7f0e4c615740 1 -- 172.21.15.45:0/2159864595 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d5e010e790 con 0x55d5e01b63a0 2024-03-28T17:52:02.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.650+0000 7f0e42000640 1 --2- 172.21.15.45:0/2159864595 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55d5e01b63a0 0x55d5e01b87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:02.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.650+0000 7f0e42000640 1 --2- 172.21.15.45:0/2159864595 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55d5e01b63a0 0x55d5e01b87d0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3571b9dba4fbc611 server_cookie=e0dce3d78874db95 in_seq=0 out_seq=0 2024-03-28T17:52:02.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.650+0000 7f0e30c00640 10 client.15789 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:52:02.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.651+0000 7f0e30c00640 1 -- 172.21.15.45:0/2159864595 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d5e010e790 con 0x55d5e01b63a0 2024-03-28T17:52:02.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.651+0000 7f0e30c00640 10 client.15789 handle_command_reply: tid=0 2024-03-28T17:52:02.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.651+0000 7f0e4c615740 1 client.15789 shutdown 2024-03-28T17:52:02.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.651+0000 7f0e4c615740 1 -- 172.21.15.45:0/2159864595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e019d0f0 msgr2=0x55d5e01a5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:52:02.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.651+0000 7f0e4c615740 1 --2- 172.21.15.45:0/2159864595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e019d0f0 0x55d5e01a5b70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0e34009bf0 tx=0x7f0e340095c0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.651+0000 7f0e4c615740 1 -- 172.21.15.45:0/2159864595 shutdown_connections 2024-03-28T17:52:02.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.651+0000 7f0e4c615740 1 --2- 172.21.15.45:0/2159864595 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55d5e01b63a0 0x55d5e01b87d0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.652+0000 7f0e4c615740 1 --2- 172.21.15.45:0/2159864595 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e01a6460 0x55d5e01a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.652+0000 7f0e4c615740 1 --2- 172.21.15.45:0/2159864595 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55d5e019ddd0 0x55d5e01a2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.652+0000 7f0e30c00640 0 client.15789 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:52:02.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.652+0000 7f0e4c615740 1 --2- 172.21.15.45:0/2159864595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e019d0f0 0x55d5e01a5b70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.652+0000 7f0e4c615740 1 -- 172.21.15.45:0/2159864595 >> 172.21.15.45:0/2159864595 conn(0x55d5e0164200 msgr2=0x55d5e0199890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:52:02.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.652+0000 7f0e4c615740 1 -- 172.21.15.45:0/2159864595 shutdown_connections 2024-03-28T17:52:02.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.652+0000 7f0e4c615740 1 -- 172.21.15.45:0/2159864595 wait complete. 2024-03-28T17:52:02.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.652+0000 7f0e4c615740 20 client.15789 trim_cache size 0 max 16384 2024-03-28T17:52:02.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.670+0000 7f0e4c615740 1 Processor -- start 2024-03-28T17:52:02.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.670+0000 7f0e4c615740 1 -- start start 2024-03-28T17:52:02.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.670+0000 7f0e4c615740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e019d0f0 0x55d5e01a5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:02.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.670+0000 7f0e4c615740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55d5e019ddd0 0x55d5e01a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:02.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.670+0000 7f0e4c615740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e01a6460 0x55d5e01a7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:02.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.670+0000 7f0e4c615740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e441b1ab0 con 0x55d5e01a6460 2024-03-28T17:52:02.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.670+0000 7f0e4c615740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e441af580 con 0x55d5e019d0f0 2024-03-28T17:52:02.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.670+0000 7f0e4c615740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e441ad640 con 0x55d5e019ddd0 2024-03-28T17:52:02.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.671+0000 7f0e43400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e01a6460 0x55d5e01a7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:02.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.671+0000 7f0e42a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e019d0f0 0x55d5e01a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:02.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.671+0000 7f0e43400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e01a6460 0x55d5e01a7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51120/0 (socket says 172.21.15.45:51120) 2024-03-28T17:52:02.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.671+0000 7f0e43400640 1 -- 172.21.15.45:0/2437031772 learned_addr learned my addr 172.21.15.45:0/2437031772 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:52:02.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.671+0000 7f0e42000640 1 --2- 172.21.15.45:0/2437031772 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55d5e019ddd0 0x55d5e01a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:02.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.671+0000 7f0e43400640 1 -- 172.21.15.45:0/2437031772 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55d5e019ddd0 msgr2=0x55d5e01a5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:52:02.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.671+0000 7f0e43400640 1 --2- 172.21.15.45:0/2437031772 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55d5e019ddd0 0x55d5e01a5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.671+0000 7f0e43400640 1 -- 172.21.15.45:0/2437031772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e019d0f0 msgr2=0x55d5e01a5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:52:02.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.671+0000 7f0e43400640 1 --2- 172.21.15.45:0/2437031772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e019d0f0 0x55d5e01a5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.671+0000 7f0e43400640 1 -- 172.21.15.45:0/2437031772 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d5dfdd50f0 con 0x55d5e01a6460 2024-03-28T17:52:02.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.671+0000 7f0e42000640 1 --2- 172.21.15.45:0/2437031772 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55d5e019ddd0 0x55d5e01a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:52:02.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.671+0000 7f0e42a00640 1 --2- 172.21.15.45:0/2437031772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e019d0f0 0x55d5e01a5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:52:02.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.672+0000 7f0e43400640 1 --2- 172.21.15.45:0/2437031772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e01a6460 0x55d5e01a7850 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c009590 tx=0x7f0e3c003e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f9580ff823b6096 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:52:02.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.672+0000 7f0e33400640 1 -- 172.21.15.45:0/2437031772 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c029020 con 0x55d5e01a6460 2024-03-28T17:52:02.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.672+0000 7f0e33400640 1 -- 172.21.15.45:0/2437031772 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e3c009dc0 con 0x55d5e01a6460 2024-03-28T17:52:02.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.672+0000 7f0e33400640 1 -- 172.21.15.45:0/2437031772 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c0256e0 con 0x55d5e01a6460 2024-03-28T17:52:02.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.672+0000 7f0e4c615740 1 -- 172.21.15.45:0/2437031772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e01a6460 msgr2=0x55d5e01a7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:52:02.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.672+0000 7f0e4c615740 1 --2- 172.21.15.45:0/2437031772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e01a6460 0x55d5e01a7850 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c009590 tx=0x7f0e3c003e60 comp rx=0 tx=0).stop 2024-03-28T17:52:02.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.673+0000 7f0e4c615740 1 -- 172.21.15.45:0/2437031772 shutdown_connections 2024-03-28T17:52:02.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.673+0000 7f0e4c615740 1 --2- 172.21.15.45:0/2437031772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e01a6460 0x55d5e01a7850 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.673+0000 7f0e4c615740 1 --2- 172.21.15.45:0/2437031772 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55d5e019ddd0 0x55d5e01a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.673+0000 7f0e4c615740 1 --2- 172.21.15.45:0/2437031772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e019d0f0 0x55d5e01a5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.673+0000 7f0e4c615740 1 -- 172.21.15.45:0/2437031772 >> 172.21.15.45:0/2437031772 conn(0x55d5e0164200 msgr2=0x55d5e0197670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:52:02.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.673+0000 7f0e4c615740 1 -- 172.21.15.45:0/2437031772 shutdown_connections 2024-03-28T17:52:02.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.673+0000 7f0e4c615740 1 -- 172.21.15.45:0/2437031772 wait complete. 2024-03-28T17:52:02.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.673+0000 7f0e4c615740 1 -- start start 2024-03-28T17:52:02.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.673+0000 7f0e4c615740 1 Processor -- start 2024-03-28T17:52:02.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.674+0000 7f0e4c615740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e019d0f0 0x55d5e01a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:02.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.674+0000 7f0e4c615740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55d5e019ddd0 0x55d5e01a0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:02.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.674+0000 7f0e4c615740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e01a6460 0x55d5e01a6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:02.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.674+0000 7f0e4c615740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e441adc40 con 0x55d5e01a6460 2024-03-28T17:52:02.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.674+0000 7f0e4c615740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e441aef20 con 0x55d5e019d0f0 2024-03-28T17:52:02.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.674+0000 7f0e4c615740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e441ae260 con 0x55d5e019ddd0 2024-03-28T17:52:02.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.674+0000 7f0e4c615740 10 client.0 _ll_register_callbacks cb 0x7f0e34044070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:52:02.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.674+0000 7f0e42a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e019d0f0 0x55d5e01a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:02.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.674+0000 7f0e42a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e019d0f0 0x55d5e01a01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53398/0 (socket says 172.21.15.45:53398) 2024-03-28T17:52:02.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.674+0000 7f0e42a00640 1 -- 172.21.15.45:0/1704423075 learned_addr learned my addr 172.21.15.45:0/1704423075 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:52:02.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.674+0000 7f0e43400640 1 --2- 172.21.15.45:0/1704423075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e01a6460 0x55d5e01a6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:02.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.674+0000 7f0e42000640 1 --2- 172.21.15.45:0/1704423075 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55d5e019ddd0 0x55d5e01a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:02.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.674+0000 7f0e42a00640 1 -- 172.21.15.45:0/1704423075 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55d5e019ddd0 msgr2=0x55d5e01a0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:52:02.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.674+0000 7f0e42a00640 1 --2- 172.21.15.45:0/1704423075 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55d5e019ddd0 0x55d5e01a0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.674+0000 7f0e42a00640 1 -- 172.21.15.45:0/1704423075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e01a6460 msgr2=0x55d5e01a6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:52:02.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.674+0000 7f0e42a00640 1 --2- 172.21.15.45:0/1704423075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e01a6460 0x55d5e01a6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.674+0000 7f0e42a00640 1 -- 172.21.15.45:0/1704423075 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d5e01a73a0 con 0x55d5e019d0f0 2024-03-28T17:52:02.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.674+0000 7f0e42000640 1 --2- 172.21.15.45:0/1704423075 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55d5e019ddd0 0x55d5e01a0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:52:02.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.675+0000 7f0e42a00640 1 --2- 172.21.15.45:0/1704423075 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e019d0f0 0x55d5e01a01c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0e34009cd0 tx=0x7f0e3404d6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29713fd2ee6619cd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:52:02.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.675+0000 7f0e30c00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:52:02.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.675+0000 7f0e30c00640 1 -- 172.21.15.45:0/1704423075 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e34051040 con 0x55d5e019d0f0 2024-03-28T17:52:02.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.675+0000 7f0e30c00640 1 -- 172.21.15.45:0/1704423075 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8942+0+0 (secure 0 0 0) 0x7f0e34003970 con 0x55d5e019d0f0 2024-03-28T17:52:02.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.675+0000 7f0e4c615740 1 -- 172.21.15.45:0/1704423075 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0e3c009dc0 con 0x55d5e019d0f0 2024-03-28T17:52:02.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.675+0000 7f0e30c00640 1 -- 172.21.15.45:0/1704423075 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e34039040 con 0x55d5e019d0f0 2024-03-28T17:52:02.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.675+0000 7f0e4c615740 10 client.25985 fetch_fsmap learned FSMap version 58 2024-03-28T17:52:02.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.675+0000 7f0e4c615740 1 -- 172.21.15.45:0/1704423075 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) v3 -- 0x55d5e02122b0 con 0x55d5e019d0f0 2024-03-28T17:52:02.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.675+0000 7f0e30c00640 1 -- 172.21.15.45:0/1704423075 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e3c009dc0 con 0x55d5e019d0f0 2024-03-28T17:52:02.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.676+0000 7f0e30c00640 1 -- 172.21.15.45:0/1704423075 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 58) v1 ==== 7922+0+0 (secure 0 0 0) 0x7f0e34054070 con 0x55d5e019d0f0 2024-03-28T17:52:02.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.676+0000 7f0e4c615740 10 client.25985 fetch_fsmap finished waiting for FSMap version 58 2024-03-28T17:52:02.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.676+0000 7f0e4c615740 10 client.25985 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:52:02.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.676+0000 7f0e4c615740 20 client.25985 populate_metadata read hostname 'smithi045' 2024-03-28T17:52:02.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.676+0000 7f0e4c615740 1 --2- 172.21.15.45:0/1704423075 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55d5e0212a00 0x55d5e0214e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:02.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.676+0000 7f0e4c615740 4 client.25985 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:52:02.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.676+0000 7f0e4c615740 1 -- 172.21.15.45:0/1704423075 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d5e0108ae0 con 0x55d5e0212a00 2024-03-28T17:52:02.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.676+0000 7f0e42000640 1 --2- 172.21.15.45:0/1704423075 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55d5e0212a00 0x55d5e0214e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:02.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.676+0000 7f0e42000640 1 --2- 172.21.15.45:0/1704423075 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55d5e0212a00 0x55d5e0214e20 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddfdb6937fc7b555 server_cookie=b138f8aa61b9e3c3 in_seq=0 out_seq=0 2024-03-28T17:52:02.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.676+0000 7f0e30c00640 10 client.25985 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:52:02.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.677+0000 7f0e30c00640 1 -- 172.21.15.45:0/1704423075 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55d5e0108ae0 con 0x55d5e0212a00 2024-03-28T17:52:02.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.677+0000 7f0e30c00640 10 client.25985 handle_command_reply: tid=0 2024-03-28T17:52:02.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.677+0000 7f0e4c615740 1 client.25985 shutdown 2024-03-28T17:52:02.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.677+0000 7f0e4c615740 1 -- 172.21.15.45:0/1704423075 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e019d0f0 msgr2=0x55d5e01a01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:52:02.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.677+0000 7f0e4c615740 1 --2- 172.21.15.45:0/1704423075 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e019d0f0 0x55d5e01a01c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0e34009cd0 tx=0x7f0e3404d6d0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.677+0000 7f0e4c615740 1 -- 172.21.15.45:0/1704423075 shutdown_connections 2024-03-28T17:52:02.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.677+0000 7f0e4c615740 1 --2- 172.21.15.45:0/1704423075 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55d5e0212a00 0x55d5e0214e20 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.677+0000 7f0e4c615740 1 --2- 172.21.15.45:0/1704423075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e01a6460 0x55d5e01a6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.677+0000 7f0e4c615740 1 --2- 172.21.15.45:0/1704423075 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55d5e019ddd0 0x55d5e01a0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.677+0000 7f0e4c615740 1 --2- 172.21.15.45:0/1704423075 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d5e019d0f0 0x55d5e01a01c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.677+0000 7f0e30c00640 0 client.25985 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:52:02.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.677+0000 7f0e4c615740 1 -- 172.21.15.45:0/1704423075 >> 172.21.15.45:0/1704423075 conn(0x55d5e0164200 msgr2=0x55d5e01a17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:52:02.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.677+0000 7f0e4c615740 1 -- 172.21.15.45:0/1704423075 shutdown_connections 2024-03-28T17:52:02.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.677+0000 7f0e4c615740 1 -- 172.21.15.45:0/1704423075 wait complete. 2024-03-28T17:52:02.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.677+0000 7f0e4c615740 20 client.25985 trim_cache size 0 max 16384 2024-03-28T17:52:02.683 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:52:02.683 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T17:52:02.683 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T17:52:02.683 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T17:52:02.683 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T17:52:02.683 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T17:52:02.683 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T17:52:02.683 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:52:02.683 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:52:02.683 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:52:02.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.678+0000 7f0e43e00640 1 -- 172.21.15.45:0/3734131153 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f0e280a29f0 msgr2=0x7f0e280a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:52:02.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.678+0000 7f0e43e00640 1 --2- 172.21.15.45:0/3734131153 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f0e280a29f0 0x7f0e280a4eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0e34009ca0 tx=0x7f0e340093a0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.678+0000 7f0e43e00640 1 -- 172.21.15.45:0/3734131153 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e441040e0 msgr2=0x7f0e4419a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:52:02.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.678+0000 7f0e43e00640 1 --2- 172.21.15.45:0/3734131153 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e441040e0 0x7f0e4419a8e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00e9f0 tx=0x7f0e2c00edf0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.678+0000 7f0e43e00640 1 -- 172.21.15.45:0/3734131153 shutdown_connections 2024-03-28T17:52:02.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.678+0000 7f0e43e00640 1 --2- 172.21.15.45:0/3734131153 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f0e280a29f0 0x7f0e280a4eb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.678+0000 7f0e43e00640 1 --2- 172.21.15.45:0/3734131153 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e44109730 0x7f0e4419ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.678+0000 7f0e43e00640 1 --2- 172.21.15.45:0/3734131153 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e441040e0 0x7f0e4419a8e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.678+0000 7f0e43e00640 1 --2- 172.21.15.45:0/3734131153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e44103720 0x7f0e4419a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:02.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.678+0000 7f0e43e00640 1 -- 172.21.15.45:0/3734131153 >> 172.21.15.45:0/3734131153 conn(0x7f0e440ff1b0 msgr2=0x7f0e440fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:52:02.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.678+0000 7f0e43e00640 1 -- 172.21.15.45:0/3734131153 shutdown_connections 2024-03-28T17:52:02.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:02.678+0000 7f0e43e00640 1 -- 172.21.15.45:0/3734131153 wait complete. 2024-03-28T17:52:02.698 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T17:52:02.839 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:02 smithi151 podman[71819]: 2024-03-28 17:52:02.57933778 +0000 UTC m=+0.135064485 container create 55c00a0025ce1283f36ea29f9b9ebfb3e24309a8878a4cd16283b5fa5359df5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T17:52:02.845 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.62 MB/sec warmup 95 sec latency 90741.899 ms 2024-03-28T17:52:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:03 smithi071 ceph-mon[36597]: daemon mds.f restarted 2024-03-28T17:52:03.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:03 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:52:03.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:03 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T17:52:03.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:03 smithi071 ceph-mon[36597]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T17:52:03.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:03 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/2381827003,v1:172.21.15.151:6835/2381827003] up:boot 2024-03-28T17:52:03.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:03 smithi071 ceph-mon[36597]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-03-28T17:52:03.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:03 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:52:03.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:03 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:03.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:03 smithi151 ceph-mon[36481]: daemon mds.f restarted 2024-03-28T17:52:03.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:03 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:52:03.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:03 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T17:52:03.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:03 smithi151 ceph-mon[36481]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T17:52:03.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:03 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/2381827003,v1:172.21.15.151:6835/2381827003] up:boot 2024-03-28T17:52:03.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:03 smithi151 ceph-mon[36481]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-03-28T17:52:03.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:03 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:52:03.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:03 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:03.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:02 smithi151 podman[71819]: 2024-03-28 17:52:02.840515129 +0000 UTC m=+0.396241838 container init 55c00a0025ce1283f36ea29f9b9ebfb3e24309a8878a4cd16283b5fa5359df5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T17:52:03.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:02 smithi151 podman[71819]: 2024-03-28 17:52:02.843176841 +0000 UTC m=+0.398903555 container start 55c00a0025ce1283f36ea29f9b9ebfb3e24309a8878a4cd16283b5fa5359df5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-28T17:52:03.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:02 smithi151 ceph-mds[71838]: -- 172.21.15.151:0/16057576 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561c92a7fa40 con 0x561c92a52800 2024-03-28T17:52:03.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:02 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: starting mds.f at 2024-03-28T17:52:03.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:02 smithi151 bash[71819]: 55c00a0025ce1283f36ea29f9b9ebfb3e24309a8878a4cd16283b5fa5359df5b 2024-03-28T17:52:03.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:02 smithi151 systemd[1]: Started Ceph mds.f for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:52:03.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:03 smithi045 ceph-mon[31773]: daemon mds.f restarted 2024-03-28T17:52:03.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:03 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:52:03.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:03 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T17:52:03.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:03 smithi045 ceph-mon[31773]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T17:52:03.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:03 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/2381827003,v1:172.21.15.151:6835/2381827003] up:boot 2024-03-28T17:52:03.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:03 smithi045 ceph-mon[31773]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-03-28T17:52:03.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:03 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:52:03.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:03 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:03.845 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.62 MB/sec warmup 96 sec latency 91742.031 ms 2024-03-28T17:52:04.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:04 smithi071 ceph-mon[36597]: pgmap v101: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-28T17:52:04.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:04 smithi151 ceph-mon[36481]: pgmap v101: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-28T17:52:04.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:04 smithi045 ceph-mon[31773]: pgmap v101: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-28T17:52:04.845 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.61 MB/sec warmup 97 sec latency 92742.145 ms 2024-03-28T17:52:05.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:05 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6838/4179697756,v1:172.21.15.151:6839/4179697756] up:resolve 2024-03-28T17:52:05.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:05 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:05.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:05 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6838/4179697756,v1:172.21.15.151:6839/4179697756] up:resolve 2024-03-28T17:52:05.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:05 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:05.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:05 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6838/4179697756,v1:172.21.15.151:6839/4179697756] up:resolve 2024-03-28T17:52:05.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:05 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:05.845 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.60 MB/sec warmup 98 sec latency 93742.290 ms 2024-03-28T17:52:06.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:06 smithi071 ceph-mon[36597]: pgmap v102: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:52:06.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:06 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:52:06.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:06 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6838/4179697756,v1:172.21.15.151:6839/4179697756] up:reconnect 2024-03-28T17:52:06.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:06 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:06.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:06 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999988 2024-03-28T17:52:06.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-mon[36481]: pgmap v102: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:52:06.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:52:06.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6838/4179697756,v1:172.21.15.151:6839/4179697756] up:reconnect 2024-03-28T17:52:06.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:06.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999988 2024-03-28T17:52:06.324 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0613800700 time 2024-03-28T17:52:06.151512+0000 2024-03-28T17:52:06.324 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:52:06.324 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:06.324 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f061de2a4a1] 2024-03-28T17:52:06.324 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f061de2a667] 2024-03-28T17:52:06.324 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559c2333d223] 2024-03-28T17:52:06.325 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559c231b75d6] 2024-03-28T17:52:06.325 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559c231b7da1] 2024-03-28T17:52:06.325 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559c231ba2d5] 2024-03-28T17:52:06.325 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559c2320a233] 2024-03-28T17:52:06.325 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559c2320c197] 2024-03-28T17:52:06.325 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559c2310b11b] 2024-03-28T17:52:06.325 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559c2310b6bc] 2024-03-28T17:52:06.325 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559c230f51ff] 2024-03-28T17:52:06.325 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f061e0bc488] 2024-03-28T17:52:06.325 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 13: (DispatchQueue::entry()+0x51f) [0x7f061e0b960f] 2024-03-28T17:52:06.325 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f061e18ce81] 2024-03-28T17:52:06.325 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f061cba71ca] 2024-03-28T17:52:06.325 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 16: clone() 2024-03-28T17:52:06.325 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 2024-03-28T17:52:06.152+0000 7f0613800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0613800700 time 2024-03-28T17:52:06.151512+0000 2024-03-28T17:52:06.326 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:52:06.326 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 2024-03-28T17:52:06.326 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:06.326 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f061de2a4a1] 2024-03-28T17:52:06.326 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f061de2a667] 2024-03-28T17:52:06.326 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559c2333d223] 2024-03-28T17:52:06.326 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559c231b75d6] 2024-03-28T17:52:06.326 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559c231b7da1] 2024-03-28T17:52:06.326 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559c231ba2d5] 2024-03-28T17:52:06.326 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559c2320a233] 2024-03-28T17:52:06.326 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559c2320c197] 2024-03-28T17:52:06.326 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559c2310b11b] 2024-03-28T17:52:06.326 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559c2310b6bc] 2024-03-28T17:52:06.327 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559c230f51ff] 2024-03-28T17:52:06.327 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f061e0bc488] 2024-03-28T17:52:06.327 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 13: (DispatchQueue::entry()+0x51f) [0x7f061e0b960f] 2024-03-28T17:52:06.327 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f061e18ce81] 2024-03-28T17:52:06.327 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f061cba71ca] 2024-03-28T17:52:06.327 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 16: clone() 2024-03-28T17:52:06.327 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 2024-03-28T17:52:06.327 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: *** Caught signal (Aborted) ** 2024-03-28T17:52:06.327 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: in thread 7f0613800700 thread_name:ms_dispatch 2024-03-28T17:52:06.327 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:06.327 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f061cbb1d20] 2024-03-28T17:52:06.328 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 2: gsignal() 2024-03-28T17:52:06.328 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 3: abort() 2024-03-28T17:52:06.328 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f061de2a4fb] 2024-03-28T17:52:06.328 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f061de2a667] 2024-03-28T17:52:06.328 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559c2333d223] 2024-03-28T17:52:06.329 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559c231b75d6] 2024-03-28T17:52:06.329 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559c231b7da1] 2024-03-28T17:52:06.329 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559c231ba2d5] 2024-03-28T17:52:06.329 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559c2320a233] 2024-03-28T17:52:06.329 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559c2320c197] 2024-03-28T17:52:06.329 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559c2310b11b] 2024-03-28T17:52:06.329 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559c2310b6bc] 2024-03-28T17:52:06.329 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559c230f51ff] 2024-03-28T17:52:06.329 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f061e0bc488] 2024-03-28T17:52:06.329 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 16: (DispatchQueue::entry()+0x51f) [0x7f061e0b960f] 2024-03-28T17:52:06.329 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f061e18ce81] 2024-03-28T17:52:06.329 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f061cba71ca] 2024-03-28T17:52:06.329 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 19: clone() 2024-03-28T17:52:06.330 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 2024-03-28T17:52:06.154+0000 7f0613800700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:52:06.330 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: in thread 7f0613800700 thread_name:ms_dispatch 2024-03-28T17:52:06.330 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 2024-03-28T17:52:06.330 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:06.330 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f061cbb1d20] 2024-03-28T17:52:06.330 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 2: gsignal() 2024-03-28T17:52:06.330 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 3: abort() 2024-03-28T17:52:06.330 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f061de2a4fb] 2024-03-28T17:52:06.330 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f061de2a667] 2024-03-28T17:52:06.330 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559c2333d223] 2024-03-28T17:52:06.330 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559c231b75d6] 2024-03-28T17:52:06.330 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559c231b7da1] 2024-03-28T17:52:06.330 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559c231ba2d5] 2024-03-28T17:52:06.331 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559c2320a233] 2024-03-28T17:52:06.331 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559c2320c197] 2024-03-28T17:52:06.331 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559c2310b11b] 2024-03-28T17:52:06.331 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559c2310b6bc] 2024-03-28T17:52:06.331 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559c230f51ff] 2024-03-28T17:52:06.331 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f061e0bc488] 2024-03-28T17:52:06.331 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 16: (DispatchQueue::entry()+0x51f) [0x7f061e0b960f] 2024-03-28T17:52:06.331 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f061e18ce81] 2024-03-28T17:52:06.331 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f061cba71ca] 2024-03-28T17:52:06.331 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 19: clone() 2024-03-28T17:52:06.331 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:52:06.331 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 2024-03-28T17:52:06.331 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: -1> 2024-03-28T17:52:06.152+0000 7f0613800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0613800700 time 2024-03-28T17:52:06.151512+0000 2024-03-28T17:52:06.331 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:52:06.332 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 2024-03-28T17:52:06.332 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:06.332 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f061de2a4a1] 2024-03-28T17:52:06.332 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f061de2a667] 2024-03-28T17:52:06.332 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559c2333d223] 2024-03-28T17:52:06.332 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559c231b75d6] 2024-03-28T17:52:06.332 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559c231b7da1] 2024-03-28T17:52:06.332 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559c231ba2d5] 2024-03-28T17:52:06.332 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559c2320a233] 2024-03-28T17:52:06.332 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559c2320c197] 2024-03-28T17:52:06.332 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559c2310b11b] 2024-03-28T17:52:06.332 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559c2310b6bc] 2024-03-28T17:52:06.332 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559c230f51ff] 2024-03-28T17:52:06.332 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f061e0bc488] 2024-03-28T17:52:06.333 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 13: (DispatchQueue::entry()+0x51f) [0x7f061e0b960f] 2024-03-28T17:52:06.333 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f061e18ce81] 2024-03-28T17:52:06.333 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f061cba71ca] 2024-03-28T17:52:06.333 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 16: clone() 2024-03-28T17:52:06.333 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 2024-03-28T17:52:06.333 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 0> 2024-03-28T17:52:06.154+0000 7f0613800700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:52:06.333 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: in thread 7f0613800700 thread_name:ms_dispatch 2024-03-28T17:52:06.333 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 2024-03-28T17:52:06.333 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:06.333 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f061cbb1d20] 2024-03-28T17:52:06.333 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 2: gsignal() 2024-03-28T17:52:06.333 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 3: abort() 2024-03-28T17:52:06.333 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f061de2a4fb] 2024-03-28T17:52:06.333 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f061de2a667] 2024-03-28T17:52:06.334 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559c2333d223] 2024-03-28T17:52:06.334 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559c231b75d6] 2024-03-28T17:52:06.334 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559c231b7da1] 2024-03-28T17:52:06.334 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559c231ba2d5] 2024-03-28T17:52:06.334 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559c2320a233] 2024-03-28T17:52:06.334 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559c2320c197] 2024-03-28T17:52:06.334 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559c2310b11b] 2024-03-28T17:52:06.334 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559c2310b6bc] 2024-03-28T17:52:06.334 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559c230f51ff] 2024-03-28T17:52:06.334 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f061e0bc488] 2024-03-28T17:52:06.334 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 16: (DispatchQueue::entry()+0x51f) [0x7f061e0b960f] 2024-03-28T17:52:06.334 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f061e18ce81] 2024-03-28T17:52:06.334 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f061cba71ca] 2024-03-28T17:52:06.334 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 19: clone() 2024-03-28T17:52:06.335 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:52:06.335 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 2024-03-28T17:52:06.335 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: -1> 2024-03-28T17:52:06.152+0000 7f0613800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0613800700 time 2024-03-28T17:52:06.151512+0000 2024-03-28T17:52:06.335 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:52:06.335 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 2024-03-28T17:52:06.335 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:06.335 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f061de2a4a1] 2024-03-28T17:52:06.335 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f061de2a667] 2024-03-28T17:52:06.335 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559c2333d223] 2024-03-28T17:52:06.335 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559c231b75d6] 2024-03-28T17:52:06.335 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559c231b7da1] 2024-03-28T17:52:06.335 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559c231ba2d5] 2024-03-28T17:52:06.335 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559c2320a233] 2024-03-28T17:52:06.336 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559c2320c197] 2024-03-28T17:52:06.336 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559c2310b11b] 2024-03-28T17:52:06.336 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559c2310b6bc] 2024-03-28T17:52:06.336 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559c230f51ff] 2024-03-28T17:52:06.336 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f061e0bc488] 2024-03-28T17:52:06.336 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 13: (DispatchQueue::entry()+0x51f) [0x7f061e0b960f] 2024-03-28T17:52:06.336 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f061e18ce81] 2024-03-28T17:52:06.336 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f061cba71ca] 2024-03-28T17:52:06.336 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 16: clone() 2024-03-28T17:52:06.336 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 2024-03-28T17:52:06.336 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 0> 2024-03-28T17:52:06.154+0000 7f0613800700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:52:06.336 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: in thread 7f0613800700 thread_name:ms_dispatch 2024-03-28T17:52:06.336 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 2024-03-28T17:52:06.336 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:06.337 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f061cbb1d20] 2024-03-28T17:52:06.337 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 2: gsignal() 2024-03-28T17:52:06.337 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 3: abort() 2024-03-28T17:52:06.337 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f061de2a4fb] 2024-03-28T17:52:06.337 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f061de2a667] 2024-03-28T17:52:06.337 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559c2333d223] 2024-03-28T17:52:06.337 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559c231b75d6] 2024-03-28T17:52:06.337 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559c231b7da1] 2024-03-28T17:52:06.337 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559c231ba2d5] 2024-03-28T17:52:06.337 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559c2320a233] 2024-03-28T17:52:06.337 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559c2320c197] 2024-03-28T17:52:06.337 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559c2310b11b] 2024-03-28T17:52:06.337 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559c2310b6bc] 2024-03-28T17:52:06.337 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559c230f51ff] 2024-03-28T17:52:06.337 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f061e0bc488] 2024-03-28T17:52:06.338 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 16: (DispatchQueue::entry()+0x51f) [0x7f061e0b960f] 2024-03-28T17:52:06.338 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f061e18ce81] 2024-03-28T17:52:06.338 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f061cba71ca] 2024-03-28T17:52:06.338 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 19: clone() 2024-03-28T17:52:06.338 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:52:06.338 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[71683]: 2024-03-28T17:52:06.338 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 podman[71872]: 2024-03-28 17:52:06.202790534 +0000 UTC m=+0.020543554 container died 9d4190da2a31fe91d44bb8b7e2c30cb02a9f269bf295bd90376845a87c028890 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T17:52:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:06 smithi045 ceph-mon[31773]: pgmap v102: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:52:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:06 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:52:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:06 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6838/4179697756,v1:172.21.15.151:6839/4179697756] up:reconnect 2024-03-28T17:52:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:06 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:06 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999988 2024-03-28T17:52:06.845 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.60 MB/sec warmup 99 sec latency 94742.428 ms 2024-03-28T17:52:07.073 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 podman[71872]: 2024-03-28 17:52:06.820460482 +0000 UTC m=+0.638213497 container remove 9d4190da2a31fe91d44bb8b7e2c30cb02a9f269bf295bd90376845a87c028890 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T17:52:07.073 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:06 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:52:07.074 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:07 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.i.service: Failed with result 'exit-code'. 2024-03-28T17:52:07.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:07 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6838/4179697756,v1:172.21.15.151:6839/4179697756] up:rejoin 2024-03-28T17:52:07.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:07 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:07.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:07 smithi151 ceph-mon[36481]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-28T17:52:07.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:07 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:52:07.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:07 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:52:07.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:07 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6838/4179697756,v1:172.21.15.151:6839/4179697756] up:active 2024-03-28T17:52:07.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:07 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:52:07.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:07 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6838/4179697756,v1:172.21.15.151:6839/4179697756] up:rejoin 2024-03-28T17:52:07.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:07 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:07.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:07 smithi045 ceph-mon[31773]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-28T17:52:07.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:07 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:52:07.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:07 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:52:07.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:07 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6838/4179697756,v1:172.21.15.151:6839/4179697756] up:active 2024-03-28T17:52:07.346 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:07 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:52:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:07 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6838/4179697756,v1:172.21.15.151:6839/4179697756] up:rejoin 2024-03-28T17:52:07.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:07 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:07.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:07 smithi071 ceph-mon[36597]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-28T17:52:07.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:07 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:52:07.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:07 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:52:07.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:07 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6838/4179697756,v1:172.21.15.151:6839/4179697756] up:active 2024-03-28T17:52:07.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:07 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:52:07.846 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.59 MB/sec warmup 100 sec latency 95742.586 ms 2024-03-28T17:52:08.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:08 smithi045 ceph-mon[31773]: pgmap v103: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:52:08.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:08 smithi071 ceph-mon[36597]: pgmap v103: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:52:08.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:08 smithi151 ceph-mon[36481]: pgmap v103: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:52:08.846 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.59 MB/sec warmup 101 sec latency 96742.720 ms 2024-03-28T17:52:09.846 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.58 MB/sec warmup 102 sec latency 97742.915 ms 2024-03-28T17:52:10.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:10 smithi045 ceph-mon[31773]: pgmap v104: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:52:10.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:10 smithi071 ceph-mon[36597]: pgmap v104: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:52:10.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:10 smithi151 ceph-mon[36481]: pgmap v104: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:52:10.846 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.57 MB/sec warmup 103 sec latency 98743.065 ms 2024-03-28T17:52:11.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:11 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:52:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:11 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:52:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:11 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:52:11.846 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.57 MB/sec warmup 104 sec latency 99743.190 ms 2024-03-28T17:52:12.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:12 smithi045 ceph-mon[31773]: pgmap v105: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:52:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:12 smithi071 ceph-mon[36597]: pgmap v105: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:52:12.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:12 smithi151 ceph-mon[36481]: pgmap v105: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:52:12.846 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.56 MB/sec warmup 105 sec latency 100743.289 ms 2024-03-28T17:52:13.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:13 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:52:13.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:13 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:52:13.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:13 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:52:13.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:13 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:52:13.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:13 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:52:13.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:13 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:52:13.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:13 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:52:13.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:13 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:52:13.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:13 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:52:13.846 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.56 MB/sec warmup 106 sec latency 101743.451 ms 2024-03-28T17:52:14.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:14 smithi071 ceph-mon[36597]: pgmap v106: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:52:14.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:14 smithi151 ceph-mon[36481]: pgmap v106: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:52:14.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:14 smithi045 ceph-mon[31773]: pgmap v106: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:52:14.846 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.55 MB/sec warmup 107 sec latency 102743.609 ms 2024-03-28T17:52:15.847 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.55 MB/sec warmup 108 sec latency 103743.744 ms 2024-03-28T17:52:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:16 smithi071 ceph-mon[36597]: pgmap v107: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-03-28T17:52:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:16 smithi151 ceph-mon[36481]: pgmap v107: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-03-28T17:52:16.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:16 smithi045 ceph-mon[31773]: pgmap v107: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-03-28T17:52:16.847 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.54 MB/sec warmup 109 sec latency 104743.933 ms 2024-03-28T17:52:17.337 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:17 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.i.service: Scheduled restart job, restart counter is at 4. 2024-03-28T17:52:17.338 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:17 smithi151 systemd[1]: Stopped Ceph mds.i for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:52:17.338 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:17 smithi151 systemd[1]: Starting Ceph mds.i for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:52:17.338 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:17 smithi151 podman[72103]: 2024-03-28 17:52:17.21040816 +0000 UTC m=+0.018405589 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:52:17.338 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:17 smithi151 podman[72103]: 2024-03-28 17:52:17.338287383 +0000 UTC m=+0.146284816 container create feab6f97e9ca0d695f655c5a6aba3866c6115eeee67d679e766be6fff4a14d03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-28T17:52:17.847 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.54 MB/sec warmup 110 sec latency 105744.094 ms 2024-03-28T17:52:17.868 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:17 smithi151 podman[72103]: 2024-03-28 17:52:17.598959359 +0000 UTC m=+0.406956779 container init feab6f97e9ca0d695f655c5a6aba3866c6115eeee67d679e766be6fff4a14d03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-28T17:52:17.869 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:17 smithi151 podman[72103]: 2024-03-28 17:52:17.602077658 +0000 UTC m=+0.410075084 container start feab6f97e9ca0d695f655c5a6aba3866c6115eeee67d679e766be6fff4a14d03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-28T17:52:17.869 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:17 smithi151 ceph-mds[72121]: -- 172.21.15.151:0/2154517913 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5639121c1a40 con 0x563912195000 2024-03-28T17:52:17.869 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:17 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: starting mds.i at 2024-03-28T17:52:17.869 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:17 smithi151 bash[72103]: feab6f97e9ca0d695f655c5a6aba3866c6115eeee67d679e766be6fff4a14d03 2024-03-28T17:52:17.869 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:17 smithi151 systemd[1]: Started Ceph mds.i for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:52:18.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:18 smithi071 ceph-mon[36597]: pgmap v108: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:52:18.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:18 smithi071 ceph-mon[36597]: daemon mds.i restarted 2024-03-28T17:52:18.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:18 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:52:18.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:18 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T17:52:18.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:18 smithi071 ceph-mon[36597]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T17:52:18.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:18 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6838/1107407982,v1:172.21.15.151:6839/1107407982] up:boot 2024-03-28T17:52:18.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:18 smithi071 ceph-mon[36597]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-03-28T17:52:18.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:18 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:52:18.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:18 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:18 smithi151 ceph-mon[36481]: pgmap v108: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:52:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:18 smithi151 ceph-mon[36481]: daemon mds.i restarted 2024-03-28T17:52:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:18 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:52:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:18 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T17:52:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:18 smithi151 ceph-mon[36481]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T17:52:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:18 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6838/1107407982,v1:172.21.15.151:6839/1107407982] up:boot 2024-03-28T17:52:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:18 smithi151 ceph-mon[36481]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-03-28T17:52:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:18 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:52:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:18 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:18.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:18 smithi045 ceph-mon[31773]: pgmap v108: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:52:18.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:18 smithi045 ceph-mon[31773]: daemon mds.i restarted 2024-03-28T17:52:18.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:18 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:52:18.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:18 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T17:52:18.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:18 smithi045 ceph-mon[31773]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T17:52:18.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:18 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6838/1107407982,v1:172.21.15.151:6839/1107407982] up:boot 2024-03-28T17:52:18.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:18 smithi045 ceph-mon[31773]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-03-28T17:52:18.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:18 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:52:18.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:18 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:18.847 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.53 MB/sec warmup 111 sec latency 106744.256 ms 2024-03-28T17:52:19.847 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.53 MB/sec warmup 112 sec latency 107744.352 ms 2024-03-28T17:52:20.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:19 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/2381827003,v1:172.21.15.151:6835/2381827003] up:resolve 2024-03-28T17:52:20.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:19 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:20.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:19 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/2381827003,v1:172.21.15.151:6835/2381827003] up:resolve 2024-03-28T17:52:20.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:19 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:20.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:19 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/2381827003,v1:172.21.15.151:6835/2381827003] up:resolve 2024-03-28T17:52:20.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:19 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:20.848 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.52 MB/sec warmup 113 sec latency 108744.491 ms 2024-03-28T17:52:21.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:20 smithi071 ceph-mon[36597]: pgmap v110: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:52:21.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:20 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:52:21.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:20 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/2381827003,v1:172.21.15.151:6835/2381827003] up:reconnect 2024-03-28T17:52:21.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:20 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:21.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:20 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999988 2024-03-28T17:52:21.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:20 smithi151 ceph-mon[36481]: pgmap v110: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:52:21.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:20 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:52:21.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:20 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/2381827003,v1:172.21.15.151:6835/2381827003] up:reconnect 2024-03-28T17:52:21.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:20 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:21.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:20 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999988 2024-03-28T17:52:21.154 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:20 smithi045 ceph-mon[31773]: pgmap v110: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:52:21.154 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:20 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:52:21.154 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:20 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/2381827003,v1:172.21.15.151:6835/2381827003] up:reconnect 2024-03-28T17:52:21.154 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:20 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:21.154 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:20 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999988 2024-03-28T17:52:21.511 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe5b6113700 time 2024-03-28T17:52:21.160079+0000 2024-03-28T17:52:21.512 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:52:21.512 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:21.512 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe5c073d4a1] 2024-03-28T17:52:21.512 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe5c073d667] 2024-03-28T17:52:21.512 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561c9068d223] 2024-03-28T17:52:21.512 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561c905075d6] 2024-03-28T17:52:21.512 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561c90507da1] 2024-03-28T17:52:21.512 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561c9050a2d5] 2024-03-28T17:52:21.512 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561c9055a233] 2024-03-28T17:52:21.512 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561c9055c197] 2024-03-28T17:52:21.512 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561c9045b11b] 2024-03-28T17:52:21.512 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561c9045b6bc] 2024-03-28T17:52:21.512 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561c904451ff] 2024-03-28T17:52:21.513 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe5c09cf488] 2024-03-28T17:52:21.513 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 13: (DispatchQueue::entry()+0x51f) [0x7fe5c09cc60f] 2024-03-28T17:52:21.513 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe5c0a9fe81] 2024-03-28T17:52:21.513 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe5bf4ba1ca] 2024-03-28T17:52:21.513 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 16: clone() 2024-03-28T17:52:21.514 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 2024-03-28T17:52:21.160+0000 7fe5b6113700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe5b6113700 time 2024-03-28T17:52:21.160079+0000 2024-03-28T17:52:21.514 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:52:21.514 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 2024-03-28T17:52:21.514 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:21.514 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe5c073d4a1] 2024-03-28T17:52:21.514 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe5c073d667] 2024-03-28T17:52:21.514 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561c9068d223] 2024-03-28T17:52:21.514 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561c905075d6] 2024-03-28T17:52:21.514 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561c90507da1] 2024-03-28T17:52:21.514 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561c9050a2d5] 2024-03-28T17:52:21.515 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561c9055a233] 2024-03-28T17:52:21.515 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561c9055c197] 2024-03-28T17:52:21.515 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561c9045b11b] 2024-03-28T17:52:21.515 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561c9045b6bc] 2024-03-28T17:52:21.515 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561c904451ff] 2024-03-28T17:52:21.515 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe5c09cf488] 2024-03-28T17:52:21.515 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 13: (DispatchQueue::entry()+0x51f) [0x7fe5c09cc60f] 2024-03-28T17:52:21.515 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe5c0a9fe81] 2024-03-28T17:52:21.515 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe5bf4ba1ca] 2024-03-28T17:52:21.515 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 16: clone() 2024-03-28T17:52:21.515 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 2024-03-28T17:52:21.515 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: *** Caught signal (Aborted) ** 2024-03-28T17:52:21.515 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: in thread 7fe5b6113700 thread_name:ms_dispatch 2024-03-28T17:52:21.515 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:21.515 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe5bf4c4d20] 2024-03-28T17:52:21.516 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 2: gsignal() 2024-03-28T17:52:21.516 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 3: abort() 2024-03-28T17:52:21.516 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe5c073d4fb] 2024-03-28T17:52:21.516 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe5c073d667] 2024-03-28T17:52:21.516 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561c9068d223] 2024-03-28T17:52:21.516 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561c905075d6] 2024-03-28T17:52:21.516 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561c90507da1] 2024-03-28T17:52:21.516 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561c9050a2d5] 2024-03-28T17:52:21.516 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561c9055a233] 2024-03-28T17:52:21.516 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561c9055c197] 2024-03-28T17:52:21.516 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561c9045b11b] 2024-03-28T17:52:21.516 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561c9045b6bc] 2024-03-28T17:52:21.516 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561c904451ff] 2024-03-28T17:52:21.516 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe5c09cf488] 2024-03-28T17:52:21.516 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 16: (DispatchQueue::entry()+0x51f) [0x7fe5c09cc60f] 2024-03-28T17:52:21.516 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe5c0a9fe81] 2024-03-28T17:52:21.517 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe5bf4ba1ca] 2024-03-28T17:52:21.517 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 19: clone() 2024-03-28T17:52:21.517 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 2024-03-28T17:52:21.162+0000 7fe5b6113700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:52:21.517 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: in thread 7fe5b6113700 thread_name:ms_dispatch 2024-03-28T17:52:21.517 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 2024-03-28T17:52:21.517 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:21.517 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe5bf4c4d20] 2024-03-28T17:52:21.517 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 2: gsignal() 2024-03-28T17:52:21.517 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 3: abort() 2024-03-28T17:52:21.517 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe5c073d4fb] 2024-03-28T17:52:21.517 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe5c073d667] 2024-03-28T17:52:21.517 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561c9068d223] 2024-03-28T17:52:21.517 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561c905075d6] 2024-03-28T17:52:21.517 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561c90507da1] 2024-03-28T17:52:21.517 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561c9050a2d5] 2024-03-28T17:52:21.518 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561c9055a233] 2024-03-28T17:52:21.518 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561c9055c197] 2024-03-28T17:52:21.518 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561c9045b11b] 2024-03-28T17:52:21.518 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561c9045b6bc] 2024-03-28T17:52:21.518 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561c904451ff] 2024-03-28T17:52:21.518 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe5c09cf488] 2024-03-28T17:52:21.518 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 16: (DispatchQueue::entry()+0x51f) [0x7fe5c09cc60f] 2024-03-28T17:52:21.518 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe5c0a9fe81] 2024-03-28T17:52:21.518 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe5bf4ba1ca] 2024-03-28T17:52:21.518 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 19: clone() 2024-03-28T17:52:21.518 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:52:21.518 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 2024-03-28T17:52:21.518 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: -1> 2024-03-28T17:52:21.160+0000 7fe5b6113700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe5b6113700 time 2024-03-28T17:52:21.160079+0000 2024-03-28T17:52:21.518 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:52:21.518 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 2024-03-28T17:52:21.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:21.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe5c073d4a1] 2024-03-28T17:52:21.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe5c073d667] 2024-03-28T17:52:21.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561c9068d223] 2024-03-28T17:52:21.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561c905075d6] 2024-03-28T17:52:21.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561c90507da1] 2024-03-28T17:52:21.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561c9050a2d5] 2024-03-28T17:52:21.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561c9055a233] 2024-03-28T17:52:21.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561c9055c197] 2024-03-28T17:52:21.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561c9045b11b] 2024-03-28T17:52:21.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561c9045b6bc] 2024-03-28T17:52:21.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561c904451ff] 2024-03-28T17:52:21.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe5c09cf488] 2024-03-28T17:52:21.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 13: (DispatchQueue::entry()+0x51f) [0x7fe5c09cc60f] 2024-03-28T17:52:21.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe5c0a9fe81] 2024-03-28T17:52:21.519 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe5bf4ba1ca] 2024-03-28T17:52:21.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 16: clone() 2024-03-28T17:52:21.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 2024-03-28T17:52:21.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 0> 2024-03-28T17:52:21.162+0000 7fe5b6113700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:52:21.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: in thread 7fe5b6113700 thread_name:ms_dispatch 2024-03-28T17:52:21.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 2024-03-28T17:52:21.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:21.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe5bf4c4d20] 2024-03-28T17:52:21.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 2: gsignal() 2024-03-28T17:52:21.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 3: abort() 2024-03-28T17:52:21.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe5c073d4fb] 2024-03-28T17:52:21.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe5c073d667] 2024-03-28T17:52:21.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561c9068d223] 2024-03-28T17:52:21.520 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561c905075d6] 2024-03-28T17:52:21.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561c90507da1] 2024-03-28T17:52:21.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561c9050a2d5] 2024-03-28T17:52:21.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561c9055a233] 2024-03-28T17:52:21.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561c9055c197] 2024-03-28T17:52:21.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561c9045b11b] 2024-03-28T17:52:21.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561c9045b6bc] 2024-03-28T17:52:21.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561c904451ff] 2024-03-28T17:52:21.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe5c09cf488] 2024-03-28T17:52:21.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 16: (DispatchQueue::entry()+0x51f) [0x7fe5c09cc60f] 2024-03-28T17:52:21.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe5c0a9fe81] 2024-03-28T17:52:21.521 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe5bf4ba1ca] 2024-03-28T17:52:21.522 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 19: clone() 2024-03-28T17:52:21.522 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:52:21.522 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 2024-03-28T17:52:21.522 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: -1> 2024-03-28T17:52:21.160+0000 7fe5b6113700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe5b6113700 time 2024-03-28T17:52:21.160079+0000 2024-03-28T17:52:21.522 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:52:21.522 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 2024-03-28T17:52:21.522 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:21.522 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe5c073d4a1] 2024-03-28T17:52:21.522 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe5c073d667] 2024-03-28T17:52:21.522 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561c9068d223] 2024-03-28T17:52:21.522 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561c905075d6] 2024-03-28T17:52:21.522 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561c90507da1] 2024-03-28T17:52:21.522 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561c9050a2d5] 2024-03-28T17:52:21.522 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561c9055a233] 2024-03-28T17:52:21.523 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561c9055c197] 2024-03-28T17:52:21.523 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561c9045b11b] 2024-03-28T17:52:21.523 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561c9045b6bc] 2024-03-28T17:52:21.523 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561c904451ff] 2024-03-28T17:52:21.523 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe5c09cf488] 2024-03-28T17:52:21.523 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 13: (DispatchQueue::entry()+0x51f) [0x7fe5c09cc60f] 2024-03-28T17:52:21.523 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe5c0a9fe81] 2024-03-28T17:52:21.523 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe5bf4ba1ca] 2024-03-28T17:52:21.523 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 16: clone() 2024-03-28T17:52:21.523 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 2024-03-28T17:52:21.523 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 0> 2024-03-28T17:52:21.162+0000 7fe5b6113700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:52:21.523 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: in thread 7fe5b6113700 thread_name:ms_dispatch 2024-03-28T17:52:21.523 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 2024-03-28T17:52:21.523 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:21.523 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe5bf4c4d20] 2024-03-28T17:52:21.523 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 2: gsignal() 2024-03-28T17:52:21.523 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 3: abort() 2024-03-28T17:52:21.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe5c073d4fb] 2024-03-28T17:52:21.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe5c073d667] 2024-03-28T17:52:21.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561c9068d223] 2024-03-28T17:52:21.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561c905075d6] 2024-03-28T17:52:21.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561c90507da1] 2024-03-28T17:52:21.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561c9050a2d5] 2024-03-28T17:52:21.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561c9055a233] 2024-03-28T17:52:21.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561c9055c197] 2024-03-28T17:52:21.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561c9045b11b] 2024-03-28T17:52:21.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561c9045b6bc] 2024-03-28T17:52:21.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561c904451ff] 2024-03-28T17:52:21.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe5c09cf488] 2024-03-28T17:52:21.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 16: (DispatchQueue::entry()+0x51f) [0x7fe5c09cc60f] 2024-03-28T17:52:21.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe5c0a9fe81] 2024-03-28T17:52:21.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe5bf4ba1ca] 2024-03-28T17:52:21.524 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 19: clone() 2024-03-28T17:52:21.525 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:52:21.525 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[71834]: 2024-03-28T17:52:21.525 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 podman[72167]: 2024-03-28 17:52:21.207588948 +0000 UTC m=+0.020758382 container died 55c00a0025ce1283f36ea29f9b9ebfb3e24309a8878a4cd16283b5fa5359df5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-28T17:52:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/2381827003,v1:172.21.15.151:6835/2381827003] up:rejoin 2024-03-28T17:52:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-mon[36481]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-28T17:52:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:52:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:52:21.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/2381827003,v1:172.21.15.151:6835/2381827003] up:active 2024-03-28T17:52:21.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:21 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:52:21.848 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.52 MB/sec warmup 114 sec latency 109744.635 ms 2024-03-28T17:52:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:21 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/2381827003,v1:172.21.15.151:6835/2381827003] up:rejoin 2024-03-28T17:52:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:21 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:22.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:21 smithi071 ceph-mon[36597]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-28T17:52:22.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:21 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:52:22.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:21 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:52:22.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:21 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/2381827003,v1:172.21.15.151:6835/2381827003] up:active 2024-03-28T17:52:22.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:21 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:52:22.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:21 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/2381827003,v1:172.21.15.151:6835/2381827003] up:rejoin 2024-03-28T17:52:22.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:21 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:22.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:21 smithi045 ceph-mon[31773]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-28T17:52:22.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:21 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:52:22.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:21 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:52:22.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:21 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/2381827003,v1:172.21.15.151:6835/2381827003] up:active 2024-03-28T17:52:22.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:21 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:52:22.323 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 podman[72167]: 2024-03-28 17:52:21.862894446 +0000 UTC m=+0.676063891 container remove 55c00a0025ce1283f36ea29f9b9ebfb3e24309a8878a4cd16283b5fa5359df5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-28T17:52:22.323 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:21 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:52:22.323 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:22 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-28T17:52:22.848 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.51 MB/sec warmup 115 sec latency 110744.796 ms 2024-03-28T17:52:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:22 smithi071 ceph-mon[36597]: pgmap v111: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:52:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:22 smithi151 ceph-mon[36481]: pgmap v111: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:52:23.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:22 smithi045 ceph-mon[31773]: pgmap v111: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:52:23.848 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.51 MB/sec warmup 116 sec latency 111744.888 ms 2024-03-28T17:52:24.848 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.51 MB/sec warmup 117 sec latency 112745.029 ms 2024-03-28T17:52:25.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:24 smithi071 ceph-mon[36597]: pgmap v112: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:52:25.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:24 smithi151 ceph-mon[36481]: pgmap v112: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:52:25.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:24 smithi045 ceph-mon[31773]: pgmap v112: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:52:25.848 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.50 MB/sec warmup 118 sec latency 113745.182 ms 2024-03-28T17:52:26.851 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.50 MB/sec warmup 119 sec latency 114745.302 ms 2024-03-28T17:52:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:26 smithi071 ceph-mon[36597]: pgmap v113: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:52:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:26 smithi151 ceph-mon[36481]: pgmap v113: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:52:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:26 smithi045 ceph-mon[31773]: pgmap v113: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:52:28.849 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 1 sec latency 116745.547 ms 2024-03-28T17:52:29.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:28 smithi071 ceph-mon[36597]: pgmap v114: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:52:29.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:28 smithi151 ceph-mon[36481]: pgmap v114: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:52:29.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:28 smithi045 ceph-mon[31773]: pgmap v114: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:52:29.849 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 2 sec latency 117745.690 ms 2024-03-28T17:52:30.849 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 3 sec latency 118745.845 ms 2024-03-28T17:52:31.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:30 smithi071 ceph-mon[36597]: pgmap v115: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 271 B/s wr, 9 op/s 2024-03-28T17:52:31.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:30 smithi151 ceph-mon[36481]: pgmap v115: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 271 B/s wr, 9 op/s 2024-03-28T17:52:31.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:30 smithi045 ceph-mon[31773]: pgmap v115: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 271 B/s wr, 9 op/s 2024-03-28T17:52:31.849 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 4 sec latency 119745.994 ms 2024-03-28T17:52:32.579 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:32 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 4. 2024-03-28T17:52:32.579 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:32 smithi151 systemd[1]: Stopped Ceph mds.f for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:52:32.579 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:32 smithi151 systemd[1]: Starting Ceph mds.f for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:52:32.580 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:32 smithi151 podman[72258]: 2024-03-28 17:52:32.461282164 +0000 UTC m=+0.018524867 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:52:32.580 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:32 smithi151 podman[72258]: 2024-03-28 17:52:32.580032532 +0000 UTC m=+0.137275226 container create acf723f9e1e5b872ce2d21fed29d77453f2394dab2133f4c0b9b690104777206 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-28T17:52:32.700 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:52:32.700 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:52:32.832 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:32 smithi151 ceph-mon[36481]: pgmap v116: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T17:52:32.849 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 5 sec latency 120746.117 ms 2024-03-28T17:52:32.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.944+0000 7fbc3a200640 1 -- 172.21.15.45:0/3744921842 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc34073ad0 msgr2=0x7fbc34073eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:52:32.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.944+0000 7fbc3a200640 1 --2- 172.21.15.45:0/3744921842 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc34073ad0 0x7fbc34073eb0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fbc24009900 tx=0x7fbc2402bcd0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.944+0000 7fbc3a200640 1 -- 172.21.15.45:0/3744921842 shutdown_connections 2024-03-28T17:52:32.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.944+0000 7fbc3a200640 1 --2- 172.21.15.45:0/3744921842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc3410de70 0x7fbc341102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.944+0000 7fbc3a200640 1 --2- 172.21.15.45:0/3744921842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc340743f0 0x7fbc3410d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.944+0000 7fbc3a200640 1 --2- 172.21.15.45:0/3744921842 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc34073ad0 0x7fbc34073eb0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.944+0000 7fbc3a200640 1 -- 172.21.15.45:0/3744921842 >> 172.21.15.45:0/3744921842 conn(0x7fbc34078aa0 msgr2=0x7fbc34078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:52:32.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.944+0000 7fbc3a200640 1 -- 172.21.15.45:0/3744921842 shutdown_connections 2024-03-28T17:52:32.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.944+0000 7fbc3a200640 1 -- 172.21.15.45:0/3744921842 wait complete. 2024-03-28T17:52:32.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.946+0000 7fbc3a200640 1 Processor -- start 2024-03-28T17:52:32.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.946+0000 7fbc3a200640 1 -- start start 2024-03-28T17:52:32.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.946+0000 7fbc3a200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc34073ad0 0x7fbc3419a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:32.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.946+0000 7fbc3a200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc340743f0 0x7fbc3419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:32.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.946+0000 7fbc3a200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc3410de70 0x7fbc3419e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:32.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.946+0000 7fbc3a200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc34113c60 con 0x7fbc34073ad0 2024-03-28T17:52:32.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.946+0000 7fbc3a200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc34113ae0 con 0x7fbc3410de70 2024-03-28T17:52:32.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.946+0000 7fbc3a200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc34113de0 con 0x7fbc340743f0 2024-03-28T17:52:32.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.947+0000 7fbc33e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc3410de70 0x7fbc3419e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:32.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.947+0000 7fbc33e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc3410de70 0x7fbc3419e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40390/0 (socket says 172.21.15.45:40390) 2024-03-28T17:52:32.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.947+0000 7fbc33e00640 1 -- 172.21.15.45:0/224341627 learned_addr learned my addr 172.21.15.45:0/224341627 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:52:32.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.947+0000 7fbc33400640 1 --2- 172.21.15.45:0/224341627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc34073ad0 0x7fbc3419a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:32.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.947+0000 7fbc2be00640 1 --2- 172.21.15.45:0/224341627 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc340743f0 0x7fbc3419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:32.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.947+0000 7fbc33e00640 1 -- 172.21.15.45:0/224341627 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc340743f0 msgr2=0x7fbc3419a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:52:32.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.947+0000 7fbc33e00640 1 --2- 172.21.15.45:0/224341627 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc340743f0 0x7fbc3419a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.947+0000 7fbc33e00640 1 -- 172.21.15.45:0/224341627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc34073ad0 msgr2=0x7fbc3419a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:52:32.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.947+0000 7fbc33e00640 1 --2- 172.21.15.45:0/224341627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc34073ad0 0x7fbc3419a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.947+0000 7fbc33e00640 1 -- 172.21.15.45:0/224341627 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc3419f110 con 0x7fbc3410de70 2024-03-28T17:52:32.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.947+0000 7fbc33e00640 1 --2- 172.21.15.45:0/224341627 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc3410de70 0x7fbc3419e9e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c00eb10 tx=0x7fbc2c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:52:32.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.947+0000 7fbc30c00640 1 -- 172.21.15.45:0/224341627 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c004b20 con 0x7fbc3410de70 2024-03-28T17:52:32.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.948+0000 7fbc30c00640 1 -- 172.21.15.45:0/224341627 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc2c004cc0 con 0x7fbc3410de70 2024-03-28T17:52:32.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.948+0000 7fbc3a200640 1 -- 172.21.15.45:0/224341627 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc3419f400 con 0x7fbc3410de70 2024-03-28T17:52:32.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.948+0000 7fbc30c00640 1 -- 172.21.15.45:0/224341627 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c010760 con 0x7fbc3410de70 2024-03-28T17:52:32.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.948+0000 7fbc3a200640 1 -- 172.21.15.45:0/224341627 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc341a6c90 con 0x7fbc3410de70 2024-03-28T17:52:32.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.951+0000 7fbc30c00640 1 -- 172.21.15.45:0/224341627 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fbc2c005020 con 0x7fbc3410de70 2024-03-28T17:52:32.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.951+0000 7fbc3d039740 1 Processor -- start 2024-03-28T17:52:32.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.951+0000 7fbc3d039740 1 -- start start 2024-03-28T17:52:32.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.952+0000 7fbc3d039740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55818992cc00 0x55818992b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:32.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.952+0000 7fbc3d039740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55818992bf20 0x55818992c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:32.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.952+0000 7fbc3d039740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558189935290 0x558189937680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:32.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.952+0000 7fbc3d039740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558189890250 con 0x55818992bf20 2024-03-28T17:52:32.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.952+0000 7fbc3d039740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x558189838a30 con 0x558189935290 2024-03-28T17:52:32.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.952+0000 7fbc3d039740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55818988c2f0 con 0x55818992cc00 2024-03-28T17:52:32.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.952+0000 7fbc2be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55818992cc00 0x55818992b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:32.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.952+0000 7fbc2be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55818992cc00 0x55818992b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:59572/0 (socket says 172.21.15.45:59572) 2024-03-28T17:52:32.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.952+0000 7fbc2be00640 1 -- 172.21.15.45:0/347714094 learned_addr learned my addr 172.21.15.45:0/347714094 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:52:32.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.952+0000 7fbc33400640 1 --2- 172.21.15.45:0/347714094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55818992bf20 0x55818992c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:32.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.952+0000 7fbc2be00640 1 --2- 172.21.15.45:0/347714094 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558189935290 0x558189937680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:32.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.952+0000 7fbc2be00640 1 -- 172.21.15.45:0/347714094 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558189935290 msgr2=0x558189937680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:52:32.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.952+0000 7fbc2be00640 1 --2- 172.21.15.45:0/347714094 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558189935290 0x558189937680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.952+0000 7fbc2be00640 1 -- 172.21.15.45:0/347714094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55818992bf20 msgr2=0x55818992c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:52:32.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.952+0000 7fbc2be00640 1 --2- 172.21.15.45:0/347714094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55818992bf20 0x55818992c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.952+0000 7fbc30c00640 1 --2- 172.21.15.45:0/224341627 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbc180a2970 0x7fbc180a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:32.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.952+0000 7fbc2be00640 1 -- 172.21.15.45:0/347714094 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558189823690 con 0x55818992cc00 2024-03-28T17:52:32.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.953+0000 7fbc30c00640 1 -- 172.21.15.45:0/224341627 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9200+0+0 (secure 0 0 0) 0x7fbc2c0cf790 con 0x7fbc3410de70 2024-03-28T17:52:32.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.953+0000 7fbc2be00640 1 --2- 172.21.15.45:0/347714094 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55818992cc00 0x55818992b950 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c00b6c0 tx=0x7fbc1c00b650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6d4de30273e88c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:52:32.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.953+0000 7fbc33e00640 1 --2- 172.21.15.45:0/224341627 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbc180a2970 0x7fbc180a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:32.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.953+0000 7fbc23e00640 1 -- 172.21.15.45:0/347714094 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c012d00 con 0x55818992cc00 2024-03-28T17:52:32.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.953+0000 7fbc23e00640 1 -- 172.21.15.45:0/347714094 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc1c010770 con 0x55818992cc00 2024-03-28T17:52:32.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.953+0000 7fbc23e00640 1 -- 172.21.15.45:0/347714094 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c003850 con 0x55818992cc00 2024-03-28T17:52:32.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.954+0000 7fbc3d039740 1 -- 172.21.15.45:0/347714094 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55818992cc00 msgr2=0x55818992b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:52:32.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.954+0000 7fbc33e00640 1 --2- 172.21.15.45:0/224341627 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbc180a2970 0x7fbc180a4e30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c0049d0 tx=0x7fbc2c005b40 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:52:32.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.954+0000 7fbc3d039740 1 --2- 172.21.15.45:0/347714094 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55818992cc00 0x55818992b950 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c00b6c0 tx=0x7fbc1c00b650 comp rx=0 tx=0).stop 2024-03-28T17:52:32.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.954+0000 7fbc3d039740 1 -- 172.21.15.45:0/347714094 shutdown_connections 2024-03-28T17:52:32.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.954+0000 7fbc3d039740 1 --2- 172.21.15.45:0/347714094 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558189935290 0x558189937680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.954+0000 7fbc3d039740 1 --2- 172.21.15.45:0/347714094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55818992bf20 0x55818992c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.954+0000 7fbc3d039740 1 --2- 172.21.15.45:0/347714094 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55818992cc00 0x55818992b950 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.954+0000 7fbc3d039740 1 -- 172.21.15.45:0/347714094 >> 172.21.15.45:0/347714094 conn(0x5581898f3200 msgr2=0x558189926970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:52:32.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.954+0000 7fbc3d039740 1 -- 172.21.15.45:0/347714094 shutdown_connections 2024-03-28T17:52:32.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.954+0000 7fbc3d039740 1 -- 172.21.15.45:0/347714094 wait complete. 2024-03-28T17:52:32.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.954+0000 7fbc3d039740 1 -- start start 2024-03-28T17:52:32.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.955+0000 7fbc3d039740 1 Processor -- start 2024-03-28T17:52:32.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.955+0000 7fbc3d039740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55818992bf20 0x558189934970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:32.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.955+0000 7fbc3d039740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55818992cc00 0x558189931d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:32.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.955+0000 7fbc3d039740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558189935290 0x558189932240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:32.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.955+0000 7fbc3d039740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558189938100 con 0x55818992bf20 2024-03-28T17:52:32.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.955+0000 7fbc3d039740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55818984be80 con 0x558189935290 2024-03-28T17:52:32.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.955+0000 7fbc3d039740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x558189938280 con 0x55818992cc00 2024-03-28T17:52:32.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.955+0000 7fbc3d039740 10 client.0 _ll_register_callbacks cb 0x55818985e130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:52:32.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.955+0000 7fbc2be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55818992bf20 0x558189934970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:32.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.956+0000 7fbc2be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55818992bf20 0x558189934970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49036/0 (socket says 172.21.15.45:49036) 2024-03-28T17:52:32.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.956+0000 7fbc2be00640 1 -- 172.21.15.45:0/966688831 learned_addr learned my addr 172.21.15.45:0/966688831 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:52:32.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.955+0000 7fbc33400640 1 --2- 172.21.15.45:0/966688831 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55818992cc00 0x558189931d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:32.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.956+0000 7fbc2be00640 1 --2- 172.21.15.45:0/966688831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558189935290 0x558189932240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.956+0000 7fbc33400640 1 -- 172.21.15.45:0/966688831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558189935290 msgr2=0x558189932240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:52:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.956+0000 7fbc33400640 1 --2- 172.21.15.45:0/966688831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558189935290 0x558189932240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.956+0000 7fbc33400640 1 -- 172.21.15.45:0/966688831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55818992bf20 msgr2=0x558189934970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:52:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.956+0000 7fbc33400640 1 --2- 172.21.15.45:0/966688831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55818992bf20 0x558189934970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.956+0000 7fbc33400640 1 -- 172.21.15.45:0/966688831 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55818966e7d0 con 0x55818992cc00 2024-03-28T17:52:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.956+0000 7fbc2be00640 1 --2- 172.21.15.45:0/966688831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55818992bf20 0x558189934970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:52:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.956+0000 7fbc33400640 1 --2- 172.21.15.45:0/966688831 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55818992cc00 0x558189931d00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fbc2402c210 tx=0x7fbc24009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=953605a646843a3a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:52:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.956+0000 7fbc20c00640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:52:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.957+0000 7fbc20c00640 1 -- 172.21.15.45:0/966688831 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc24044070 con 0x55818992cc00 2024-03-28T17:52:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.957+0000 7fbc20c00640 1 -- 172.21.15.45:0/966688831 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9200+0+0 (secure 0 0 0) 0x7fbc240046c0 con 0x55818992cc00 2024-03-28T17:52:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.957+0000 7fbc3d039740 1 -- 172.21.15.45:0/966688831 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbc1c010770 con 0x55818992cc00 2024-03-28T17:52:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.957+0000 7fbc20c00640 1 -- 172.21.15.45:0/966688831 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbc240048c0 con 0x55818992cc00 2024-03-28T17:52:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.957+0000 7fbc3d039740 10 client.25981 fetch_fsmap learned FSMap version 72 2024-03-28T17:52:32.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.957+0000 7fbc3d039740 1 -- 172.21.15.45:0/966688831 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) v3 -- 0x5581895a43f0 con 0x55818992cc00 2024-03-28T17:52:32.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.958+0000 7fbc20c00640 1 -- 172.21.15.45:0/966688831 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc1c010770 con 0x55818992cc00 2024-03-28T17:52:32.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.958+0000 7fbc20c00640 1 -- 172.21.15.45:0/966688831 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 72) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fbc24003680 con 0x55818992cc00 2024-03-28T17:52:32.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.958+0000 7fbc3d039740 10 client.25981 fetch_fsmap finished waiting for FSMap version 72 2024-03-28T17:52:32.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.958+0000 7fbc3d039740 10 client.25981 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:52:32.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.958+0000 7fbc3d039740 20 client.25981 populate_metadata read hostname 'smithi045' 2024-03-28T17:52:32.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.958+0000 7fbc3d039740 1 --2- 172.21.15.45:0/966688831 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5581899451d0 0x5581899475c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:32.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.958+0000 7fbc3d039740 4 client.25981 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:52:32.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.958+0000 7fbc3d039740 1 -- 172.21.15.45:0/966688831 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55818989d790 con 0x5581899451d0 2024-03-28T17:52:32.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.958+0000 7fbc2be00640 1 --2- 172.21.15.45:0/966688831 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5581899451d0 0x5581899475c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:32.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.959+0000 7fbc2be00640 1 --2- 172.21.15.45:0/966688831 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5581899451d0 0x5581899475c0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d448e85d2bfe2c1f server_cookie=7fc6dae0b17e335a in_seq=0 out_seq=0 2024-03-28T17:52:32.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.959+0000 7fbc20c00640 10 client.25981 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:52:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.959+0000 7fbc20c00640 1 -- 172.21.15.45:0/966688831 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55818989d790 con 0x5581899451d0 2024-03-28T17:52:32.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.959+0000 7fbc20c00640 10 client.25981 handle_command_reply: tid=0 2024-03-28T17:52:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.959+0000 7fbc3d039740 1 client.25981 shutdown 2024-03-28T17:52:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.959+0000 7fbc3d039740 1 -- 172.21.15.45:0/966688831 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55818992cc00 msgr2=0x558189931d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:52:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.959+0000 7fbc3d039740 1 --2- 172.21.15.45:0/966688831 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55818992cc00 0x558189931d00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fbc2402c210 tx=0x7fbc24009580 comp rx=0 tx=0).stop 2024-03-28T17:52:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.960+0000 7fbc3d039740 1 -- 172.21.15.45:0/966688831 shutdown_connections 2024-03-28T17:52:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.960+0000 7fbc3d039740 1 --2- 172.21.15.45:0/966688831 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5581899451d0 0x5581899475c0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.960+0000 7fbc3d039740 1 --2- 172.21.15.45:0/966688831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558189935290 0x558189932240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.960+0000 7fbc3d039740 1 --2- 172.21.15.45:0/966688831 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55818992cc00 0x558189931d00 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.960+0000 7fbc3d039740 1 --2- 172.21.15.45:0/966688831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55818992bf20 0x558189934970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.960+0000 7fbc3d039740 1 -- 172.21.15.45:0/966688831 >> 172.21.15.45:0/966688831 conn(0x5581898f3200 msgr2=0x558189925200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:52:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.960+0000 7fbc20c00640 0 client.25981 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:52:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.960+0000 7fbc3d039740 1 -- 172.21.15.45:0/966688831 shutdown_connections 2024-03-28T17:52:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.960+0000 7fbc3d039740 1 -- 172.21.15.45:0/966688831 wait complete. 2024-03-28T17:52:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.960+0000 7fbc3d039740 20 client.25981 trim_cache size 0 max 16384 2024-03-28T17:52:32.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.977+0000 7fbc3d039740 1 Processor -- start 2024-03-28T17:52:32.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.978+0000 7fbc3d039740 1 -- start start 2024-03-28T17:52:32.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.978+0000 7fbc3d039740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55818992bf20 0x558189934430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:32.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.978+0000 7fbc3d039740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55818992cc00 0x558189934970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:32.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.978+0000 7fbc3d039740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558189935290 0x55818992ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:32.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.978+0000 7fbc3d039740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc341b01f0 con 0x55818992bf20 2024-03-28T17:52:32.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.978+0000 7fbc3d039740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc341adc10 con 0x55818992cc00 2024-03-28T17:52:32.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.978+0000 7fbc3d039740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc341afb90 con 0x558189935290 2024-03-28T17:52:32.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.978+0000 7fbc2be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55818992bf20 0x558189934430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:32.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.978+0000 7fbc33400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55818992cc00 0x558189934970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:32.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.978+0000 7fbc33400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55818992cc00 0x558189934970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40412/0 (socket says 172.21.15.45:40412) 2024-03-28T17:52:32.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.978+0000 7fbc33400640 1 -- 172.21.15.45:0/2215388185 learned_addr learned my addr 172.21.15.45:0/2215388185 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:52:32.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.978+0000 7fbc33400640 1 -- 172.21.15.45:0/2215388185 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558189935290 msgr2=0x55818992ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:52:32.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.978+0000 7fbc33400640 1 --2- 172.21.15.45:0/2215388185 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558189935290 0x55818992ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.978+0000 7fbc33400640 1 -- 172.21.15.45:0/2215388185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55818992bf20 msgr2=0x558189934430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:52:32.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.978+0000 7fbc33400640 1 --2- 172.21.15.45:0/2215388185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55818992bf20 0x558189934430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.978+0000 7fbc33400640 1 -- 172.21.15.45:0/2215388185 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581895640f0 con 0x55818992cc00 2024-03-28T17:52:32.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.979+0000 7fbc33400640 1 --2- 172.21.15.45:0/2215388185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55818992cc00 0x558189934970 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fbc2402c2e0 tx=0x7fbc24005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edaeff97c1642e3b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:52:32.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.979+0000 7fbc23400640 1 -- 172.21.15.45:0/2215388185 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2405a070 con 0x55818992cc00 2024-03-28T17:52:32.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.979+0000 7fbc23400640 1 -- 172.21.15.45:0/2215388185 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc24003b50 con 0x55818992cc00 2024-03-28T17:52:32.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.979+0000 7fbc23400640 1 -- 172.21.15.45:0/2215388185 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc24044070 con 0x55818992cc00 2024-03-28T17:52:32.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.980+0000 7fbc3d039740 1 -- 172.21.15.45:0/2215388185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55818992cc00 msgr2=0x558189934970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:52:32.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.980+0000 7fbc3d039740 1 --2- 172.21.15.45:0/2215388185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55818992cc00 0x558189934970 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fbc2402c2e0 tx=0x7fbc24005fd0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.980+0000 7fbc3d039740 1 -- 172.21.15.45:0/2215388185 shutdown_connections 2024-03-28T17:52:32.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.980+0000 7fbc3d039740 1 --2- 172.21.15.45:0/2215388185 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558189935290 0x55818992ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.980+0000 7fbc3d039740 1 --2- 172.21.15.45:0/2215388185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55818992cc00 0x558189934970 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.980+0000 7fbc3d039740 1 --2- 172.21.15.45:0/2215388185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55818992bf20 0x558189934430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.980+0000 7fbc3d039740 1 -- 172.21.15.45:0/2215388185 >> 172.21.15.45:0/2215388185 conn(0x5581898f3200 msgr2=0x558189928650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:52:32.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.980+0000 7fbc3d039740 1 -- 172.21.15.45:0/2215388185 shutdown_connections 2024-03-28T17:52:32.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.980+0000 7fbc3d039740 1 -- 172.21.15.45:0/2215388185 wait complete. 2024-03-28T17:52:32.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.980+0000 7fbc3d039740 1 -- start start 2024-03-28T17:52:32.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.980+0000 7fbc3d039740 1 Processor -- start 2024-03-28T17:52:32.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.981+0000 7fbc3d039740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55818992bf20 0x55818993d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:32.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.981+0000 7fbc3d039740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55818992cc00 0x55818993b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:32.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.981+0000 7fbc3d039740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558189935290 0x55818993bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:32.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.981+0000 7fbc3d039740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc341af530 con 0x558189935290 2024-03-28T17:52:32.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.981+0000 7fbc3d039740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc341ae210 con 0x55818992bf20 2024-03-28T17:52:32.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.981+0000 7fbc3d039740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc341ae870 con 0x55818992cc00 2024-03-28T17:52:32.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.981+0000 7fbc3d039740 10 client.0 _ll_register_callbacks cb 0x7fbc24005d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:52:32.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.981+0000 7fbc2be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55818992bf20 0x55818993d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:32.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.981+0000 7fbc2be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55818992bf20 0x55818993d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40418/0 (socket says 172.21.15.45:40418) 2024-03-28T17:52:32.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.981+0000 7fbc2be00640 1 -- 172.21.15.45:0/1723966027 learned_addr learned my addr 172.21.15.45:0/1723966027 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:52:32.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.981+0000 7fbc33400640 1 --2- 172.21.15.45:0/1723966027 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55818992cc00 0x55818993b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:32.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.981+0000 7fbc2be00640 1 --2- 172.21.15.45:0/1723966027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558189935290 0x55818993bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:32.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.982+0000 7fbc33400640 1 -- 172.21.15.45:0/1723966027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55818992bf20 msgr2=0x55818993d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:52:32.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.982+0000 7fbc33400640 1 --2- 172.21.15.45:0/1723966027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55818992bf20 0x55818993d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.982+0000 7fbc33400640 1 -- 172.21.15.45:0/1723966027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558189935290 msgr2=0x55818993bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:52:32.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.982+0000 7fbc33400640 1 --2- 172.21.15.45:0/1723966027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558189935290 0x55818993bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.982+0000 7fbc33400640 1 -- 172.21.15.45:0/1723966027 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55818993c170 con 0x55818992cc00 2024-03-28T17:52:32.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.982+0000 7fbc2be00640 1 --2- 172.21.15.45:0/1723966027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558189935290 0x55818993bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:52:32.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.982+0000 7fbc33400640 1 --2- 172.21.15.45:0/1723966027 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55818992cc00 0x55818993b570 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x5581899222d0 tx=0x7fbc240044d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d41c670158eef4ff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:52:32.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.982+0000 7fbc20c00640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:52:32.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.982+0000 7fbc20c00640 1 -- 172.21.15.45:0/1723966027 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2405a030 con 0x55818992cc00 2024-03-28T17:52:32.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.982+0000 7fbc3d039740 1 -- 172.21.15.45:0/1723966027 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbc24003b50 con 0x55818992cc00 2024-03-28T17:52:32.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.982+0000 7fbc20c00640 1 -- 172.21.15.45:0/1723966027 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9200+0+0 (secure 0 0 0) 0x7fbc240046c0 con 0x55818992cc00 2024-03-28T17:52:32.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.983+0000 7fbc20c00640 1 -- 172.21.15.45:0/1723966027 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbc24044070 con 0x55818992cc00 2024-03-28T17:52:32.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.983+0000 7fbc3d039740 10 client.25984 fetch_fsmap learned FSMap version 72 2024-03-28T17:52:32.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.983+0000 7fbc3d039740 1 -- 172.21.15.45:0/1723966027 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) v3 -- 0x5581899a1960 con 0x55818992cc00 2024-03-28T17:52:32.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.983+0000 7fbc20c00640 1 -- 172.21.15.45:0/1723966027 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc24003b50 con 0x55818992cc00 2024-03-28T17:52:32.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.983+0000 7fbc20c00640 1 -- 172.21.15.45:0/1723966027 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 72) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fbc240557d0 con 0x55818992cc00 2024-03-28T17:52:32.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.983+0000 7fbc3d039740 10 client.25984 fetch_fsmap finished waiting for FSMap version 72 2024-03-28T17:52:32.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.983+0000 7fbc3d039740 10 client.25984 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:52:32.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.983+0000 7fbc3d039740 20 client.25984 populate_metadata read hostname 'smithi045' 2024-03-28T17:52:32.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.983+0000 7fbc3d039740 1 --2- 172.21.15.45:0/1723966027 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5581899a1d50 0x5581899a4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:52:32.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.983+0000 7fbc3d039740 4 client.25984 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:52:32.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.983+0000 7fbc3d039740 1 -- 172.21.15.45:0/1723966027 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558189897ae0 con 0x5581899a1d50 2024-03-28T17:52:32.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.983+0000 7fbc2be00640 1 --2- 172.21.15.45:0/1723966027 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5581899a1d50 0x5581899a4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:52:32.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.984+0000 7fbc2be00640 1 --2- 172.21.15.45:0/1723966027 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5581899a1d50 0x5581899a4170 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f91c93d5a201d6 server_cookie=93cd63d6d378444d in_seq=0 out_seq=0 2024-03-28T17:52:32.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.984+0000 7fbc20c00640 10 client.25984 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:52:32.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.984+0000 7fbc20c00640 1 -- 172.21.15.45:0/1723966027 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x558189897ae0 con 0x5581899a1d50 2024-03-28T17:52:32.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.984+0000 7fbc20c00640 10 client.25984 handle_command_reply: tid=0 2024-03-28T17:52:32.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.984+0000 7fbc3d039740 1 client.25984 shutdown 2024-03-28T17:52:32.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.984+0000 7fbc3d039740 1 -- 172.21.15.45:0/1723966027 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55818992cc00 msgr2=0x55818993b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:52:32.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.984+0000 7fbc3d039740 1 --2- 172.21.15.45:0/1723966027 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55818992cc00 0x55818993b570 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x5581899222d0 tx=0x7fbc240044d0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.984+0000 7fbc3d039740 1 -- 172.21.15.45:0/1723966027 shutdown_connections 2024-03-28T17:52:32.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.984+0000 7fbc3d039740 1 --2- 172.21.15.45:0/1723966027 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5581899a1d50 0x5581899a4170 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.984+0000 7fbc3d039740 1 --2- 172.21.15.45:0/1723966027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558189935290 0x55818993bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.984+0000 7fbc3d039740 1 --2- 172.21.15.45:0/1723966027 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55818992cc00 0x55818993b570 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.984+0000 7fbc3d039740 1 --2- 172.21.15.45:0/1723966027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55818992bf20 0x55818993d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.984+0000 7fbc3d039740 1 -- 172.21.15.45:0/1723966027 >> 172.21.15.45:0/1723966027 conn(0x5581898f3200 msgr2=0x558189925200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:52:32.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.984+0000 7fbc20c00640 0 client.25984 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:52:32.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.985+0000 7fbc3d039740 1 -- 172.21.15.45:0/1723966027 shutdown_connections 2024-03-28T17:52:32.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.985+0000 7fbc3d039740 1 -- 172.21.15.45:0/1723966027 wait complete. 2024-03-28T17:52:32.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.985+0000 7fbc3d039740 20 client.25984 trim_cache size 0 max 16384 2024-03-28T17:52:32.990 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:52:32.990 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T17:52:32.990 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T17:52:32.990 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T17:52:32.990 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T17:52:32.990 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T17:52:32.990 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T17:52:32.990 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:52:32.990 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:52:32.991 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:52:32.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.985+0000 7fbc3a200640 1 -- 172.21.15.45:0/224341627 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbc180a2970 msgr2=0x7fbc180a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:52:32.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.985+0000 7fbc3a200640 1 --2- 172.21.15.45:0/224341627 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbc180a2970 0x7fbc180a4e30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c0049d0 tx=0x7fbc2c005b40 comp rx=0 tx=0).stop 2024-03-28T17:52:32.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.985+0000 7fbc3a200640 1 -- 172.21.15.45:0/224341627 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc3410de70 msgr2=0x7fbc3419e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:52:32.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.985+0000 7fbc3a200640 1 --2- 172.21.15.45:0/224341627 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc3410de70 0x7fbc3419e9e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c00eb10 tx=0x7fbc2c00c2d0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.986+0000 7fbc3a200640 1 -- 172.21.15.45:0/224341627 shutdown_connections 2024-03-28T17:52:32.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.986+0000 7fbc3a200640 1 --2- 172.21.15.45:0/224341627 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbc180a2970 0x7fbc180a4e30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.986+0000 7fbc3a200640 1 --2- 172.21.15.45:0/224341627 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc3410de70 0x7fbc3419e9e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.986+0000 7fbc3a200640 1 --2- 172.21.15.45:0/224341627 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc340743f0 0x7fbc3419a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.986+0000 7fbc3a200640 1 --2- 172.21.15.45:0/224341627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc34073ad0 0x7fbc3419a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:52:32.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.986+0000 7fbc3a200640 1 -- 172.21.15.45:0/224341627 >> 172.21.15.45:0/224341627 conn(0x7fbc34078aa0 msgr2=0x7fbc340fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:52:32.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.986+0000 7fbc3a200640 1 -- 172.21.15.45:0/224341627 shutdown_connections 2024-03-28T17:52:32.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:52:32.986+0000 7fbc3a200640 1 -- 172.21.15.45:0/224341627 wait complete. 2024-03-28T17:52:33.001 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:32 smithi045 ceph-mon[31773]: pgmap v116: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T17:52:33.003 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T17:52:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:32 smithi071 ceph-mon[36597]: pgmap v116: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T17:52:33.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:32 smithi151 podman[72258]: 2024-03-28 17:52:32.832963674 +0000 UTC m=+0.390206374 container init acf723f9e1e5b872ce2d21fed29d77453f2394dab2133f4c0b9b690104777206 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-28T17:52:33.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:32 smithi151 podman[72258]: 2024-03-28 17:52:32.835687092 +0000 UTC m=+0.392929794 container start acf723f9e1e5b872ce2d21fed29d77453f2394dab2133f4c0b9b690104777206 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-28T17:52:33.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:32 smithi151 ceph-mds[72276]: -- 172.21.15.151:0/1714062991 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ea37e6fa40 con 0x55ea37e42800 2024-03-28T17:52:33.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:32 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: starting mds.f at 2024-03-28T17:52:33.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:32 smithi151 bash[72258]: acf723f9e1e5b872ce2d21fed29d77453f2394dab2133f4c0b9b690104777206 2024-03-28T17:52:33.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:32 smithi151 systemd[1]: Started Ceph mds.f for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:52:33.849 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 6 sec latency 121746.201 ms 2024-03-28T17:52:34.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:33 smithi071 ceph-mon[36597]: daemon mds.f restarted 2024-03-28T17:52:34.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:33 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:52:34.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:33 smithi071 ceph-mon[36597]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T17:52:34.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:33 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/93427158,v1:172.21.15.151:6835/93427158] up:boot 2024-03-28T17:52:34.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:33 smithi071 ceph-mon[36597]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-03-28T17:52:34.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:33 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T17:52:34.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:33 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:52:34.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:33 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:34.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:33 smithi151 ceph-mon[36481]: daemon mds.f restarted 2024-03-28T17:52:34.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:33 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:52:34.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:33 smithi151 ceph-mon[36481]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T17:52:34.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:33 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/93427158,v1:172.21.15.151:6835/93427158] up:boot 2024-03-28T17:52:34.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:33 smithi151 ceph-mon[36481]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-03-28T17:52:34.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:33 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T17:52:34.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:33 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:52:34.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:33 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:34.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:33 smithi045 ceph-mon[31773]: daemon mds.f restarted 2024-03-28T17:52:34.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:33 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:52:34.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:33 smithi045 ceph-mon[31773]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T17:52:34.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:33 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/93427158,v1:172.21.15.151:6835/93427158] up:boot 2024-03-28T17:52:34.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:33 smithi045 ceph-mon[31773]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-03-28T17:52:34.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:33 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T17:52:34.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:33 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-28T17:52:34.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:33 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:replay} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:34.849 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 7 sec latency 122746.333 ms 2024-03-28T17:52:35.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:34 smithi071 ceph-mon[36597]: pgmap v118: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:52:35.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:34 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6838/1107407982,v1:172.21.15.151:6839/1107407982] up:resolve 2024-03-28T17:52:35.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:34 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:35.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:34 smithi045 ceph-mon[31773]: pgmap v118: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:52:35.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:34 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6838/1107407982,v1:172.21.15.151:6839/1107407982] up:resolve 2024-03-28T17:52:35.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:34 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:34 smithi151 ceph-mon[36481]: pgmap v118: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:52:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:34 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6838/1107407982,v1:172.21.15.151:6839/1107407982] up:resolve 2024-03-28T17:52:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:34 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:resolve} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:35.849 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 8 sec latency 123746.460 ms 2024-03-28T17:52:36.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:35 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:52:36.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:35 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6838/1107407982,v1:172.21.15.151:6839/1107407982] up:reconnect 2024-03-28T17:52:36.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:35 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:36.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:35 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:52:36.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:35 smithi045 ceph-mon[31773]: pgmap v119: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T17:52:36.168 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:35 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:52:36.168 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:35 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6838/1107407982,v1:172.21.15.151:6839/1107407982] up:reconnect 2024-03-28T17:52:36.168 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:35 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:36.168 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:35 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:52:36.168 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:35 smithi151 ceph-mon[36481]: pgmap v119: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T17:52:36.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:35 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:52:36.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:35 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6838/1107407982,v1:172.21.15.151:6839/1107407982] up:reconnect 2024-03-28T17:52:36.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:35 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:reconnect} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:36.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:35 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:52:36.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:35 smithi071 ceph-mon[36597]: pgmap v119: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T17:52:36.519 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2da026b700 time 2024-03-28T17:52:36.168693+0000 2024-03-28T17:52:36.519 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:52:36.520 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:36.520 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2daa8954a1] 2024-03-28T17:52:36.520 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2daa895667] 2024-03-28T17:52:36.520 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5639111a7223] 2024-03-28T17:52:36.520 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5639110215d6] 2024-03-28T17:52:36.520 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563911021da1] 2024-03-28T17:52:36.520 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5639110242d5] 2024-03-28T17:52:36.520 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563911074233] 2024-03-28T17:52:36.520 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563911076197] 2024-03-28T17:52:36.520 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563910f7511b] 2024-03-28T17:52:36.520 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563910f756bc] 2024-03-28T17:52:36.520 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563910f5f1ff] 2024-03-28T17:52:36.520 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2daab27488] 2024-03-28T17:52:36.521 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 13: (DispatchQueue::entry()+0x51f) [0x7f2daab2460f] 2024-03-28T17:52:36.521 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2daabf7e81] 2024-03-28T17:52:36.521 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2da96121ca] 2024-03-28T17:52:36.521 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 16: clone() 2024-03-28T17:52:36.522 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 2024-03-28T17:52:36.169+0000 7f2da026b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2da026b700 time 2024-03-28T17:52:36.168693+0000 2024-03-28T17:52:36.522 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:52:36.522 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 2024-03-28T17:52:36.522 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:36.522 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2daa8954a1] 2024-03-28T17:52:36.522 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2daa895667] 2024-03-28T17:52:36.522 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5639111a7223] 2024-03-28T17:52:36.523 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5639110215d6] 2024-03-28T17:52:36.523 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563911021da1] 2024-03-28T17:52:36.523 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5639110242d5] 2024-03-28T17:52:36.523 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563911074233] 2024-03-28T17:52:36.523 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563911076197] 2024-03-28T17:52:36.523 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563910f7511b] 2024-03-28T17:52:36.523 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563910f756bc] 2024-03-28T17:52:36.523 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563910f5f1ff] 2024-03-28T17:52:36.523 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2daab27488] 2024-03-28T17:52:36.523 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 13: (DispatchQueue::entry()+0x51f) [0x7f2daab2460f] 2024-03-28T17:52:36.524 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2daabf7e81] 2024-03-28T17:52:36.524 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2da96121ca] 2024-03-28T17:52:36.524 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 16: clone() 2024-03-28T17:52:36.524 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 2024-03-28T17:52:36.524 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: *** Caught signal (Aborted) ** 2024-03-28T17:52:36.524 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: in thread 7f2da026b700 thread_name:ms_dispatch 2024-03-28T17:52:36.524 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:36.524 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2da961cd20] 2024-03-28T17:52:36.524 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 2: gsignal() 2024-03-28T17:52:36.524 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 3: abort() 2024-03-28T17:52:36.524 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2daa8954fb] 2024-03-28T17:52:36.524 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2daa895667] 2024-03-28T17:52:36.524 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5639111a7223] 2024-03-28T17:52:36.524 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5639110215d6] 2024-03-28T17:52:36.525 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563911021da1] 2024-03-28T17:52:36.525 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5639110242d5] 2024-03-28T17:52:36.525 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563911074233] 2024-03-28T17:52:36.525 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563911076197] 2024-03-28T17:52:36.525 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563910f7511b] 2024-03-28T17:52:36.525 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563910f756bc] 2024-03-28T17:52:36.525 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563910f5f1ff] 2024-03-28T17:52:36.525 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2daab27488] 2024-03-28T17:52:36.525 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 16: (DispatchQueue::entry()+0x51f) [0x7f2daab2460f] 2024-03-28T17:52:36.525 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2daabf7e81] 2024-03-28T17:52:36.525 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2da96121ca] 2024-03-28T17:52:36.525 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 19: clone() 2024-03-28T17:52:36.526 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 2024-03-28T17:52:36.170+0000 7f2da026b700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:52:36.526 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: in thread 7f2da026b700 thread_name:ms_dispatch 2024-03-28T17:52:36.526 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 2024-03-28T17:52:36.526 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:36.526 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2da961cd20] 2024-03-28T17:52:36.526 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 2: gsignal() 2024-03-28T17:52:36.526 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 3: abort() 2024-03-28T17:52:36.526 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2daa8954fb] 2024-03-28T17:52:36.526 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2daa895667] 2024-03-28T17:52:36.526 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5639111a7223] 2024-03-28T17:52:36.526 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5639110215d6] 2024-03-28T17:52:36.526 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563911021da1] 2024-03-28T17:52:36.526 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5639110242d5] 2024-03-28T17:52:36.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563911074233] 2024-03-28T17:52:36.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563911076197] 2024-03-28T17:52:36.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563910f7511b] 2024-03-28T17:52:36.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563910f756bc] 2024-03-28T17:52:36.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563910f5f1ff] 2024-03-28T17:52:36.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2daab27488] 2024-03-28T17:52:36.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 16: (DispatchQueue::entry()+0x51f) [0x7f2daab2460f] 2024-03-28T17:52:36.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2daabf7e81] 2024-03-28T17:52:36.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2da96121ca] 2024-03-28T17:52:36.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 19: clone() 2024-03-28T17:52:36.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:52:36.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 2024-03-28T17:52:36.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: -1> 2024-03-28T17:52:36.169+0000 7f2da026b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2da026b700 time 2024-03-28T17:52:36.168693+0000 2024-03-28T17:52:36.527 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:52:36.528 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 2024-03-28T17:52:36.528 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:36.528 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2daa8954a1] 2024-03-28T17:52:36.528 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2daa895667] 2024-03-28T17:52:36.528 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5639111a7223] 2024-03-28T17:52:36.528 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5639110215d6] 2024-03-28T17:52:36.528 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563911021da1] 2024-03-28T17:52:36.528 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5639110242d5] 2024-03-28T17:52:36.528 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563911074233] 2024-03-28T17:52:36.528 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563911076197] 2024-03-28T17:52:36.528 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563910f7511b] 2024-03-28T17:52:36.528 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563910f756bc] 2024-03-28T17:52:36.528 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563910f5f1ff] 2024-03-28T17:52:36.528 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2daab27488] 2024-03-28T17:52:36.528 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 13: (DispatchQueue::entry()+0x51f) [0x7f2daab2460f] 2024-03-28T17:52:36.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2daabf7e81] 2024-03-28T17:52:36.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2da96121ca] 2024-03-28T17:52:36.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 16: clone() 2024-03-28T17:52:36.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 2024-03-28T17:52:36.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 0> 2024-03-28T17:52:36.170+0000 7f2da026b700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:52:36.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: in thread 7f2da026b700 thread_name:ms_dispatch 2024-03-28T17:52:36.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 2024-03-28T17:52:36.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:36.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2da961cd20] 2024-03-28T17:52:36.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 2: gsignal() 2024-03-28T17:52:36.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 3: abort() 2024-03-28T17:52:36.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2daa8954fb] 2024-03-28T17:52:36.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2daa895667] 2024-03-28T17:52:36.529 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5639111a7223] 2024-03-28T17:52:36.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5639110215d6] 2024-03-28T17:52:36.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563911021da1] 2024-03-28T17:52:36.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5639110242d5] 2024-03-28T17:52:36.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563911074233] 2024-03-28T17:52:36.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563911076197] 2024-03-28T17:52:36.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563910f7511b] 2024-03-28T17:52:36.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563910f756bc] 2024-03-28T17:52:36.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563910f5f1ff] 2024-03-28T17:52:36.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2daab27488] 2024-03-28T17:52:36.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 16: (DispatchQueue::entry()+0x51f) [0x7f2daab2460f] 2024-03-28T17:52:36.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2daabf7e81] 2024-03-28T17:52:36.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2da96121ca] 2024-03-28T17:52:36.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 19: clone() 2024-03-28T17:52:36.530 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:52:36.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 2024-03-28T17:52:36.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: -1> 2024-03-28T17:52:36.169+0000 7f2da026b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2da026b700 time 2024-03-28T17:52:36.168693+0000 2024-03-28T17:52:36.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:52:36.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 2024-03-28T17:52:36.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:36.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2daa8954a1] 2024-03-28T17:52:36.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2daa895667] 2024-03-28T17:52:36.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5639111a7223] 2024-03-28T17:52:36.531 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5639110215d6] 2024-03-28T17:52:36.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563911021da1] 2024-03-28T17:52:36.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5639110242d5] 2024-03-28T17:52:36.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563911074233] 2024-03-28T17:52:36.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563911076197] 2024-03-28T17:52:36.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563910f7511b] 2024-03-28T17:52:36.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563910f756bc] 2024-03-28T17:52:36.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563910f5f1ff] 2024-03-28T17:52:36.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2daab27488] 2024-03-28T17:52:36.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 13: (DispatchQueue::entry()+0x51f) [0x7f2daab2460f] 2024-03-28T17:52:36.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2daabf7e81] 2024-03-28T17:52:36.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2da96121ca] 2024-03-28T17:52:36.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 16: clone() 2024-03-28T17:52:36.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 2024-03-28T17:52:36.532 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 0> 2024-03-28T17:52:36.170+0000 7f2da026b700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:52:36.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: in thread 7f2da026b700 thread_name:ms_dispatch 2024-03-28T17:52:36.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 2024-03-28T17:52:36.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:36.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2da961cd20] 2024-03-28T17:52:36.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 2: gsignal() 2024-03-28T17:52:36.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 3: abort() 2024-03-28T17:52:36.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2daa8954fb] 2024-03-28T17:52:36.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2daa895667] 2024-03-28T17:52:36.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5639111a7223] 2024-03-28T17:52:36.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5639110215d6] 2024-03-28T17:52:36.533 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563911021da1] 2024-03-28T17:52:36.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5639110242d5] 2024-03-28T17:52:36.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563911074233] 2024-03-28T17:52:36.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563911076197] 2024-03-28T17:52:36.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563910f7511b] 2024-03-28T17:52:36.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563910f756bc] 2024-03-28T17:52:36.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563910f5f1ff] 2024-03-28T17:52:36.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2daab27488] 2024-03-28T17:52:36.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 16: (DispatchQueue::entry()+0x51f) [0x7f2daab2460f] 2024-03-28T17:52:36.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2daabf7e81] 2024-03-28T17:52:36.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2da96121ca] 2024-03-28T17:52:36.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 19: clone() 2024-03-28T17:52:36.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:52:36.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i[72117]: 2024-03-28T17:52:36.534 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 podman[72310]: 2024-03-28 17:52:36.214459081 +0000 UTC m=+0.020085852 container died feab6f97e9ca0d695f655c5a6aba3866c6115eeee67d679e766be6fff4a14d03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-28T17:52:36.850 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 9 sec latency 124746.603 ms 2024-03-28T17:52:37.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:36 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6838/1107407982,v1:172.21.15.151:6839/1107407982] up:rejoin 2024-03-28T17:52:37.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:36 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:37.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:36 smithi045 ceph-mon[31773]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-28T17:52:37.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:36 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:52:37.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:36 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:52:37.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:36 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6838/1107407982,v1:172.21.15.151:6839/1107407982] up:active 2024-03-28T17:52:37.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:36 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:52:37.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:36 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6838/1107407982,v1:172.21.15.151:6839/1107407982] up:rejoin 2024-03-28T17:52:37.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:36 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:37.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:36 smithi071 ceph-mon[36597]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-28T17:52:37.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:36 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:52:37.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:36 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:52:37.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:36 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6838/1107407982,v1:172.21.15.151:6839/1107407982] up:active 2024-03-28T17:52:37.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:36 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:52:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6838/1107407982,v1:172.21.15.151:6839/1107407982] up:rejoin 2024-03-28T17:52:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=i=up:rejoin} 2 up:standby-replay 7 up:standby 2024-03-28T17:52:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-mon[36481]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-28T17:52:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:52:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:52:37.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6838/1107407982,v1:172.21.15.151:6839/1107407982] up:active 2024-03-28T17:52:37.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:36 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:52:37.324 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 podman[72310]: 2024-03-28 17:52:36.896326512 +0000 UTC m=+0.701953266 container remove feab6f97e9ca0d695f655c5a6aba3866c6115eeee67d679e766be6fff4a14d03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-i, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-28T17:52:37.324 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:36 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:52:37.324 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:37 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.i.service: Failed with result 'exit-code'. 2024-03-28T17:52:37.850 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 10 sec latency 125746.786 ms 2024-03-28T17:52:38.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:37 smithi071 ceph-mon[36597]: pgmap v120: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T17:52:38.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:37 smithi151 ceph-mon[36481]: pgmap v120: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T17:52:38.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:37 smithi045 ceph-mon[31773]: pgmap v120: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T17:52:38.850 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 11 sec latency 126746.958 ms 2024-03-28T17:52:39.850 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 12 sec latency 127747.130 ms 2024-03-28T17:52:40.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:40 smithi071 ceph-mon[36597]: pgmap v121: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T17:52:40.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:40 smithi151 ceph-mon[36481]: pgmap v121: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T17:52:40.377 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:40 smithi045 ceph-mon[31773]: pgmap v121: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T17:52:40.850 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 13 sec latency 128747.234 ms 2024-03-28T17:52:41.850 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 14 sec latency 129747.363 ms 2024-03-28T17:52:42.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:42 smithi045 ceph-mon[31773]: pgmap v122: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:52:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:42 smithi071 ceph-mon[36597]: pgmap v122: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:52:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:42 smithi151 ceph-mon[36481]: pgmap v122: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:52:42.850 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 15 sec latency 130747.507 ms 2024-03-28T17:52:43.851 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 16 sec latency 131747.665 ms 2024-03-28T17:52:44.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:44 smithi045 ceph-mon[31773]: pgmap v123: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 202 B/s wr, 10 op/s 2024-03-28T17:52:44.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:44 smithi071 ceph-mon[36597]: pgmap v123: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 202 B/s wr, 10 op/s 2024-03-28T17:52:44.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:44 smithi151 ceph-mon[36481]: pgmap v123: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 202 B/s wr, 10 op/s 2024-03-28T17:52:44.851 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 17 sec latency 132747.917 ms 2024-03-28T17:52:45.851 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 18 sec latency 133748.078 ms 2024-03-28T17:52:46.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:46 smithi071 ceph-mon[36597]: pgmap v124: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:52:46.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:46 smithi151 ceph-mon[36481]: pgmap v124: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:52:46.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:46 smithi045 ceph-mon[31773]: pgmap v124: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:52:46.851 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 19 sec latency 134748.244 ms 2024-03-28T17:52:47.823 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:47 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.i.service: Scheduled restart job, restart counter is at 5. 2024-03-28T17:52:47.823 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:47 smithi151 systemd[1]: Stopped Ceph mds.i for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:52:47.823 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:47 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.i.service: Start request repeated too quickly. 2024-03-28T17:52:47.823 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:47 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.i.service: Failed with result 'exit-code'. 2024-03-28T17:52:47.823 INFO:journalctl@ceph.mds.i.smithi151.stdout:Mar 28 17:52:47 smithi151 systemd[1]: Failed to start Ceph mds.i for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:52:47.851 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 20 sec latency 135748.389 ms 2024-03-28T17:52:48.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:48 smithi071 ceph-mon[36597]: pgmap v125: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:52:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:48 smithi151 ceph-mon[36481]: pgmap v125: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:52:48.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:48 smithi045 ceph-mon[31773]: pgmap v125: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:52:48.851 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 21 sec latency 136748.536 ms 2024-03-28T17:52:49.852 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 22 sec latency 137748.678 ms 2024-03-28T17:52:50.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:50 smithi151 ceph-mon[36481]: pgmap v126: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:52:50.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:50 smithi045 ceph-mon[31773]: pgmap v126: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:52:50.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:50 smithi071 ceph-mon[36597]: pgmap v126: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:52:50.852 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 23 sec latency 138748.830 ms 2024-03-28T17:52:51.852 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 24 sec latency 139748.978 ms 2024-03-28T17:52:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:52 smithi045 ceph-mon[31773]: pgmap v127: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:52:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:52 smithi045 ceph-mon[31773]: Replacing daemon mds.i as rank 2 with standby daemon mds.f 2024-03-28T17:52:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:52 smithi045 ceph-mon[31773]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-03-28T17:52:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:52 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:52:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:52 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:52:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:52 smithi045 ceph-mon[31773]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T17:52:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:52 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:replay} 2 up:standby-replay 6 up:standby 2024-03-28T17:52:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:52 smithi071 ceph-mon[36597]: pgmap v127: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:52:52.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:52 smithi071 ceph-mon[36597]: Replacing daemon mds.i as rank 2 with standby daemon mds.f 2024-03-28T17:52:52.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:52 smithi071 ceph-mon[36597]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-03-28T17:52:52.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:52 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:52:52.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:52 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:52:52.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:52 smithi071 ceph-mon[36597]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T17:52:52.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:52 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:replay} 2 up:standby-replay 6 up:standby 2024-03-28T17:52:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:52 smithi151 ceph-mon[36481]: pgmap v127: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:52:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:52 smithi151 ceph-mon[36481]: Replacing daemon mds.i as rank 2 with standby daemon mds.f 2024-03-28T17:52:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:52 smithi151 ceph-mon[36481]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-03-28T17:52:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:52 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:52:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:52 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T17:52:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:52 smithi151 ceph-mon[36481]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T17:52:52.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:52 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:replay} 2 up:standby-replay 6 up:standby 2024-03-28T17:52:52.852 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 25 sec latency 140749.113 ms 2024-03-28T17:52:53.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:53 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/93427158,v1:172.21.15.151:6835/93427158] up:resolve 2024-03-28T17:52:53.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:53 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:resolve} 2 up:standby-replay 6 up:standby 2024-03-28T17:52:53.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:53 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/93427158,v1:172.21.15.151:6835/93427158] up:resolve 2024-03-28T17:52:53.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:53 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:resolve} 2 up:standby-replay 6 up:standby 2024-03-28T17:52:53.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:53 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/93427158,v1:172.21.15.151:6835/93427158] up:resolve 2024-03-28T17:52:53.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:53 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:resolve} 2 up:standby-replay 6 up:standby 2024-03-28T17:52:53.852 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 26 sec latency 141749.275 ms 2024-03-28T17:52:54.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:54 smithi045 ceph-mon[31773]: pgmap v129: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:52:54.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:54 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:52:54.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:54 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/93427158,v1:172.21.15.151:6835/93427158] up:reconnect 2024-03-28T17:52:54.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:54 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:reconnect} 2 up:standby-replay 6 up:standby 2024-03-28T17:52:54.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:54 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:52:54.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:54 smithi151 ceph-mon[36481]: pgmap v129: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:52:54.475 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:54 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:52:54.475 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:54 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/93427158,v1:172.21.15.151:6835/93427158] up:reconnect 2024-03-28T17:52:54.475 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:54 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:reconnect} 2 up:standby-replay 6 up:standby 2024-03-28T17:52:54.475 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:54 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:52:54.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:54 smithi071 ceph-mon[36597]: pgmap v129: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:52:54.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:54 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:52:54.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:54 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/93427158,v1:172.21.15.151:6835/93427158] up:reconnect 2024-03-28T17:52:54.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:54 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:reconnect} 2 up:standby-replay 6 up:standby 2024-03-28T17:52:54.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:54 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:52:54.852 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 27 sec latency 142749.406 ms 2024-03-28T17:52:55.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:55 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/93427158,v1:172.21.15.151:6835/93427158] up:rejoin 2024-03-28T17:52:55.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:55 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:rejoin} 2 up:standby-replay 6 up:standby 2024-03-28T17:52:55.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:55 smithi045 ceph-mon[31773]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-28T17:52:55.457 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/93427158,v1:172.21.15.151:6835/93427158] up:rejoin 2024-03-28T17:52:55.457 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:rejoin} 2 up:standby-replay 6 up:standby 2024-03-28T17:52:55.457 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-mon[36481]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-28T17:52:55.458 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f17a76bf700 time 2024-03-28T17:52:55.138187+0000 2024-03-28T17:52:55.458 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:52:55.458 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:55.458 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f17b1ce94a1] 2024-03-28T17:52:55.458 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f17b1ce9667] 2024-03-28T17:52:55.458 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ea3544d223] 2024-03-28T17:52:55.458 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ea352c75d6] 2024-03-28T17:52:55.458 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ea352c7da1] 2024-03-28T17:52:55.458 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ea352ca2d5] 2024-03-28T17:52:55.459 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ea3531a233] 2024-03-28T17:52:55.459 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ea3531c197] 2024-03-28T17:52:55.459 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ea3521b11b] 2024-03-28T17:52:55.459 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ea3521b6bc] 2024-03-28T17:52:55.459 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ea352051ff] 2024-03-28T17:52:55.459 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f17b1f7b488] 2024-03-28T17:52:55.459 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 13: (DispatchQueue::entry()+0x51f) [0x7f17b1f7860f] 2024-03-28T17:52:55.459 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f17b204be81] 2024-03-28T17:52:55.459 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f17b0a661ca] 2024-03-28T17:52:55.459 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 16: clone() 2024-03-28T17:52:55.459 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: *** Caught signal (Aborted) ** 2024-03-28T17:52:55.459 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: in thread 7f17a76bf700 thread_name:ms_dispatch 2024-03-28T17:52:55.460 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 2024-03-28T17:52:55.138+0000 7f17a76bf700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f17a76bf700 time 2024-03-28T17:52:55.138187+0000 2024-03-28T17:52:55.461 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:52:55.461 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 2024-03-28T17:52:55.461 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:55.461 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f17b1ce94a1] 2024-03-28T17:52:55.461 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f17b1ce9667] 2024-03-28T17:52:55.461 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ea3544d223] 2024-03-28T17:52:55.461 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ea352c75d6] 2024-03-28T17:52:55.461 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ea352c7da1] 2024-03-28T17:52:55.461 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ea352ca2d5] 2024-03-28T17:52:55.461 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ea3531a233] 2024-03-28T17:52:55.461 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ea3531c197] 2024-03-28T17:52:55.462 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ea3521b11b] 2024-03-28T17:52:55.462 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ea3521b6bc] 2024-03-28T17:52:55.462 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ea352051ff] 2024-03-28T17:52:55.462 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f17b1f7b488] 2024-03-28T17:52:55.462 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 13: (DispatchQueue::entry()+0x51f) [0x7f17b1f7860f] 2024-03-28T17:52:55.462 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f17b204be81] 2024-03-28T17:52:55.462 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f17b0a661ca] 2024-03-28T17:52:55.462 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 16: clone() 2024-03-28T17:52:55.462 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 2024-03-28T17:52:55.462 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:55.462 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f17b0a70d20] 2024-03-28T17:52:55.462 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 2: gsignal() 2024-03-28T17:52:55.463 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 3: abort() 2024-03-28T17:52:55.463 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f17b1ce94fb] 2024-03-28T17:52:55.463 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f17b1ce9667] 2024-03-28T17:52:55.463 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ea3544d223] 2024-03-28T17:52:55.463 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ea352c75d6] 2024-03-28T17:52:55.463 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ea352c7da1] 2024-03-28T17:52:55.463 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ea352ca2d5] 2024-03-28T17:52:55.463 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ea3531a233] 2024-03-28T17:52:55.463 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ea3531c197] 2024-03-28T17:52:55.463 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ea3521b11b] 2024-03-28T17:52:55.463 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ea3521b6bc] 2024-03-28T17:52:55.463 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ea352051ff] 2024-03-28T17:52:55.463 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f17b1f7b488] 2024-03-28T17:52:55.463 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 16: (DispatchQueue::entry()+0x51f) [0x7f17b1f7860f] 2024-03-28T17:52:55.463 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f17b204be81] 2024-03-28T17:52:55.463 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f17b0a661ca] 2024-03-28T17:52:55.464 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 19: clone() 2024-03-28T17:52:55.464 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 2024-03-28T17:52:55.140+0000 7f17a76bf700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:52:55.464 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: in thread 7f17a76bf700 thread_name:ms_dispatch 2024-03-28T17:52:55.464 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 2024-03-28T17:52:55.464 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:55.464 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f17b0a70d20] 2024-03-28T17:52:55.464 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 2: gsignal() 2024-03-28T17:52:55.464 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 3: abort() 2024-03-28T17:52:55.464 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f17b1ce94fb] 2024-03-28T17:52:55.464 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f17b1ce9667] 2024-03-28T17:52:55.464 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ea3544d223] 2024-03-28T17:52:55.464 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ea352c75d6] 2024-03-28T17:52:55.464 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ea352c7da1] 2024-03-28T17:52:55.465 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ea352ca2d5] 2024-03-28T17:52:55.465 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ea3531a233] 2024-03-28T17:52:55.465 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ea3531c197] 2024-03-28T17:52:55.465 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ea3521b11b] 2024-03-28T17:52:55.465 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ea3521b6bc] 2024-03-28T17:52:55.465 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ea352051ff] 2024-03-28T17:52:55.465 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f17b1f7b488] 2024-03-28T17:52:55.465 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 16: (DispatchQueue::entry()+0x51f) [0x7f17b1f7860f] 2024-03-28T17:52:55.465 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f17b204be81] 2024-03-28T17:52:55.465 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f17b0a661ca] 2024-03-28T17:52:55.465 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 19: clone() 2024-03-28T17:52:55.465 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:52:55.465 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 2024-03-28T17:52:55.466 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: -1> 2024-03-28T17:52:55.138+0000 7f17a76bf700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f17a76bf700 time 2024-03-28T17:52:55.138187+0000 2024-03-28T17:52:55.466 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:52:55.466 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 2024-03-28T17:52:55.466 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:55.466 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f17b1ce94a1] 2024-03-28T17:52:55.466 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f17b1ce9667] 2024-03-28T17:52:55.466 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ea3544d223] 2024-03-28T17:52:55.466 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ea352c75d6] 2024-03-28T17:52:55.466 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ea352c7da1] 2024-03-28T17:52:55.466 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ea352ca2d5] 2024-03-28T17:52:55.466 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ea3531a233] 2024-03-28T17:52:55.466 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ea3531c197] 2024-03-28T17:52:55.466 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ea3521b11b] 2024-03-28T17:52:55.466 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ea3521b6bc] 2024-03-28T17:52:55.467 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ea352051ff] 2024-03-28T17:52:55.467 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f17b1f7b488] 2024-03-28T17:52:55.467 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 13: (DispatchQueue::entry()+0x51f) [0x7f17b1f7860f] 2024-03-28T17:52:55.467 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f17b204be81] 2024-03-28T17:52:55.467 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f17b0a661ca] 2024-03-28T17:52:55.467 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 16: clone() 2024-03-28T17:52:55.467 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 2024-03-28T17:52:55.467 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 0> 2024-03-28T17:52:55.140+0000 7f17a76bf700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:52:55.467 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: in thread 7f17a76bf700 thread_name:ms_dispatch 2024-03-28T17:52:55.467 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 2024-03-28T17:52:55.467 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:55.467 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f17b0a70d20] 2024-03-28T17:52:55.467 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 2: gsignal() 2024-03-28T17:52:55.467 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 3: abort() 2024-03-28T17:52:55.467 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f17b1ce94fb] 2024-03-28T17:52:55.468 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f17b1ce9667] 2024-03-28T17:52:55.468 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ea3544d223] 2024-03-28T17:52:55.468 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ea352c75d6] 2024-03-28T17:52:55.468 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ea352c7da1] 2024-03-28T17:52:55.468 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ea352ca2d5] 2024-03-28T17:52:55.468 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ea3531a233] 2024-03-28T17:52:55.468 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ea3531c197] 2024-03-28T17:52:55.468 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ea3521b11b] 2024-03-28T17:52:55.468 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ea3521b6bc] 2024-03-28T17:52:55.468 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ea352051ff] 2024-03-28T17:52:55.468 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f17b1f7b488] 2024-03-28T17:52:55.468 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 16: (DispatchQueue::entry()+0x51f) [0x7f17b1f7860f] 2024-03-28T17:52:55.468 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f17b204be81] 2024-03-28T17:52:55.468 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f17b0a661ca] 2024-03-28T17:52:55.469 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 19: clone() 2024-03-28T17:52:55.469 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:52:55.469 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 2024-03-28T17:52:55.469 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: -1> 2024-03-28T17:52:55.138+0000 7f17a76bf700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f17a76bf700 time 2024-03-28T17:52:55.138187+0000 2024-03-28T17:52:55.469 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:52:55.469 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 2024-03-28T17:52:55.469 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:55.469 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f17b1ce94a1] 2024-03-28T17:52:55.469 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f17b1ce9667] 2024-03-28T17:52:55.469 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ea3544d223] 2024-03-28T17:52:55.469 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ea352c75d6] 2024-03-28T17:52:55.470 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ea352c7da1] 2024-03-28T17:52:55.470 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ea352ca2d5] 2024-03-28T17:52:55.470 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ea3531a233] 2024-03-28T17:52:55.470 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ea3531c197] 2024-03-28T17:52:55.470 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ea3521b11b] 2024-03-28T17:52:55.470 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ea3521b6bc] 2024-03-28T17:52:55.470 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ea352051ff] 2024-03-28T17:52:55.470 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f17b1f7b488] 2024-03-28T17:52:55.470 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 13: (DispatchQueue::entry()+0x51f) [0x7f17b1f7860f] 2024-03-28T17:52:55.470 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f17b204be81] 2024-03-28T17:52:55.471 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f17b0a661ca] 2024-03-28T17:52:55.471 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 16: clone() 2024-03-28T17:52:55.471 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 2024-03-28T17:52:55.471 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 0> 2024-03-28T17:52:55.140+0000 7f17a76bf700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:52:55.471 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: in thread 7f17a76bf700 thread_name:ms_dispatch 2024-03-28T17:52:55.471 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 2024-03-28T17:52:55.471 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:52:55.471 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f17b0a70d20] 2024-03-28T17:52:55.471 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 2: gsignal() 2024-03-28T17:52:55.471 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 3: abort() 2024-03-28T17:52:55.471 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f17b1ce94fb] 2024-03-28T17:52:55.471 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f17b1ce9667] 2024-03-28T17:52:55.471 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ea3544d223] 2024-03-28T17:52:55.471 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ea352c75d6] 2024-03-28T17:52:55.471 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ea352c7da1] 2024-03-28T17:52:55.472 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ea352ca2d5] 2024-03-28T17:52:55.472 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ea3531a233] 2024-03-28T17:52:55.472 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ea3531c197] 2024-03-28T17:52:55.472 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ea3521b11b] 2024-03-28T17:52:55.472 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ea3521b6bc] 2024-03-28T17:52:55.472 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ea352051ff] 2024-03-28T17:52:55.472 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f17b1f7b488] 2024-03-28T17:52:55.472 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 16: (DispatchQueue::entry()+0x51f) [0x7f17b1f7860f] 2024-03-28T17:52:55.472 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f17b204be81] 2024-03-28T17:52:55.472 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f17b0a661ca] 2024-03-28T17:52:55.472 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 19: clone() 2024-03-28T17:52:55.472 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:52:55.472 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72272]: 2024-03-28T17:52:55.472 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 podman[72358]: 2024-03-28 17:52:55.187205447 +0000 UTC m=+0.020410502 container died acf723f9e1e5b872ce2d21fed29d77453f2394dab2133f4c0b9b690104777206 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-28T17:52:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:55 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/93427158,v1:172.21.15.151:6835/93427158] up:rejoin 2024-03-28T17:52:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:55 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:rejoin} 2 up:standby-replay 6 up:standby 2024-03-28T17:52:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:55 smithi071 ceph-mon[36597]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-28T17:52:55.852 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 28 sec latency 143749.537 ms 2024-03-28T17:52:56.073 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 podman[72358]: 2024-03-28 17:52:55.792123482 +0000 UTC m=+0.625328590 container remove acf723f9e1e5b872ce2d21fed29d77453f2394dab2133f4c0b9b690104777206 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-28T17:52:56.073 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:55 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:52:56.073 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:52:56 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-28T17:52:56.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:56 smithi045 ceph-mon[31773]: pgmap v130: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:52:56.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:56 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:52:56.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:56 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:52:56.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:56 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/93427158,v1:172.21.15.151:6835/93427158] up:active 2024-03-28T17:52:56.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:56 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T17:52:56.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:56 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 3 up:standby-replay 5 up:standby 2024-03-28T17:52:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:56 smithi071 ceph-mon[36597]: pgmap v130: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:52:56.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:56 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:52:56.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:56 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:52:56.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:56 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/93427158,v1:172.21.15.151:6835/93427158] up:active 2024-03-28T17:52:56.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:56 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T17:52:56.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:56 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 3 up:standby-replay 5 up:standby 2024-03-28T17:52:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:56 smithi151 ceph-mon[36481]: pgmap v130: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:52:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:56 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:52:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:56 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:52:56.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:56 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/93427158,v1:172.21.15.151:6835/93427158] up:active 2024-03-28T17:52:56.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:56 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T17:52:56.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:56 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 3 up:standby-replay 5 up:standby 2024-03-28T17:52:56.853 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 29 sec latency 144749.699 ms 2024-03-28T17:52:57.853 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 30 sec latency 145749.852 ms 2024-03-28T17:52:58.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:52:58 smithi071 ceph-mon[36597]: pgmap v131: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:52:58.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:52:58 smithi151 ceph-mon[36481]: pgmap v131: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:52:58.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:52:58 smithi045 ceph-mon[31773]: pgmap v131: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:52:58.853 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 31 sec latency 146750.093 ms 2024-03-28T17:52:59.853 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 32 sec latency 147750.254 ms 2024-03-28T17:53:00.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:00 smithi071 ceph-mon[36597]: pgmap v132: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:53:00.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:00 smithi151 ceph-mon[36481]: pgmap v132: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:53:00.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:00 smithi045 ceph-mon[31773]: pgmap v132: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:53:00.853 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 33 sec latency 148750.351 ms 2024-03-28T17:53:01.853 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 34 sec latency 149750.509 ms 2024-03-28T17:53:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:02 smithi071 ceph-mon[36597]: pgmap v133: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:53:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:02 smithi151 ceph-mon[36481]: pgmap v133: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:53:02.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:02 smithi045 ceph-mon[31773]: pgmap v133: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:53:02.854 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 35 sec latency 150750.667 ms 2024-03-28T17:53:03.004 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:53:03.005 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:53:03.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.235+0000 7fa454a00640 1 -- 172.21.15.45:0/3681570731 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa450107e50 msgr2=0x7fa450108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:53:03.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.235+0000 7fa454a00640 1 --2- 172.21.15.45:0/3681570731 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa450107e50 0x7fa450108230 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa440009ca0 tx=0x7fa44002c0f0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.235+0000 7fa454a00640 1 -- 172.21.15.45:0/3681570731 shutdown_connections 2024-03-28T17:53:03.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.235+0000 7fa454a00640 1 --2- 172.21.15.45:0/3681570731 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa45010cea0 0x7fa450114a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.235+0000 7fa454a00640 1 --2- 172.21.15.45:0/3681570731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa450108800 0x7fa45010c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.235+0000 7fa454a00640 1 --2- 172.21.15.45:0/3681570731 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa450107e50 0x7fa450108230 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.235+0000 7fa454a00640 1 -- 172.21.15.45:0/3681570731 >> 172.21.15.45:0/3681570731 conn(0x7fa4500774c0 msgr2=0x7fa4501064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:53:03.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.235+0000 7fa454a00640 1 -- 172.21.15.45:0/3681570731 shutdown_connections 2024-03-28T17:53:03.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.235+0000 7fa454a00640 1 -- 172.21.15.45:0/3681570731 wait complete. 2024-03-28T17:53:03.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.237+0000 7fa454a00640 1 Processor -- start 2024-03-28T17:53:03.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.237+0000 7fa454a00640 1 -- start start 2024-03-28T17:53:03.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.237+0000 7fa454a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa450107e50 0x7fa45019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:03.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.237+0000 7fa454a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa45010cea0 0x7fa45019e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:03.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.237+0000 7fa454a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4501a2c60 0x7fa4501a3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:03.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.237+0000 7fa454a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa450117f50 con 0x7fa4501a2c60 2024-03-28T17:53:03.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.237+0000 7fa454a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa450117dd0 con 0x7fa450107e50 2024-03-28T17:53:03.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.237+0000 7fa454a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4501180d0 con 0x7fa45010cea0 2024-03-28T17:53:03.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.238+0000 7fa44e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa450107e50 0x7fa45019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.238+0000 7fa44e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa450107e50 0x7fa45019e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:58156/0 (socket says 172.21.15.45:58156) 2024-03-28T17:53:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.238+0000 7fa44e800640 1 -- 172.21.15.45:0/2869502933 learned_addr learned my addr 172.21.15.45:0/2869502933 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:53:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.238+0000 7fa44de00640 1 --2- 172.21.15.45:0/2869502933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa45010cea0 0x7fa45019e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.238+0000 7fa44f200640 1 --2- 172.21.15.45:0/2869502933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4501a2c60 0x7fa4501a3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.238+0000 7fa44e800640 1 -- 172.21.15.45:0/2869502933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa45010cea0 msgr2=0x7fa45019e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:53:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.238+0000 7fa44e800640 1 --2- 172.21.15.45:0/2869502933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa45010cea0 0x7fa45019e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.238+0000 7fa44e800640 1 -- 172.21.15.45:0/2869502933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4501a2c60 msgr2=0x7fa4501a3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:53:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.238+0000 7fa44e800640 1 --2- 172.21.15.45:0/2869502933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4501a2c60 0x7fa4501a3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.238+0000 7fa44e800640 1 -- 172.21.15.45:0/2869502933 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4501a3750 con 0x7fa450107e50 2024-03-28T17:53:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.238+0000 7fa44f200640 1 --2- 172.21.15.45:0/2869502933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4501a2c60 0x7fa4501a3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:53:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.238+0000 7fa44e800640 1 --2- 172.21.15.45:0/2869502933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa450107e50 0x7fa45019e3f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa440005e30 tx=0x7fa440032f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:53:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.239+0000 7fa447400640 1 -- 172.21.15.45:0/2869502933 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa440041020 con 0x7fa450107e50 2024-03-28T17:53:03.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.239+0000 7fa447400640 1 -- 172.21.15.45:0/2869502933 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa440034cd0 con 0x7fa450107e50 2024-03-28T17:53:03.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.239+0000 7fa447400640 1 -- 172.21.15.45:0/2869502933 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa440039040 con 0x7fa450107e50 2024-03-28T17:53:03.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.239+0000 7fa454a00640 1 -- 172.21.15.45:0/2869502933 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4501a39e0 con 0x7fa450107e50 2024-03-28T17:53:03.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.239+0000 7fa454a00640 1 -- 172.21.15.45:0/2869502933 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4501ab150 con 0x7fa450107e50 2024-03-28T17:53:03.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.243+0000 7fa447400640 1 -- 172.21.15.45:0/2869502933 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fa440044020 con 0x7fa450107e50 2024-03-28T17:53:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.243+0000 7fa45817b740 1 Processor -- start 2024-03-28T17:53:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.244+0000 7fa45817b740 1 -- start start 2024-03-28T17:53:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.244+0000 7fa45817b740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab7dd0 0x5644c1ab6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.244+0000 7fa45817b740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5644c1ab70f0 0x5644c1ab7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.244+0000 7fa45817b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644c1ac0460 0x5644c1ac2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.244+0000 7fa45817b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644c1a1b250 con 0x5644c1ac0460 2024-03-28T17:53:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.244+0000 7fa45817b740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644c19c3a30 con 0x5644c1ab7dd0 2024-03-28T17:53:03.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.244+0000 7fa45817b740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644c1a172f0 con 0x5644c1ab70f0 2024-03-28T17:53:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.244+0000 7fa44f200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab7dd0 0x5644c1ab6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.244+0000 7fa44e800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5644c1ab70f0 0x5644c1ab7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.244+0000 7fa44f200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab7dd0 0x5644c1ab6b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:58162/0 (socket says 172.21.15.45:58162) 2024-03-28T17:53:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.244+0000 7fa44f200640 1 -- 172.21.15.45:0/2322707265 learned_addr learned my addr 172.21.15.45:0/2322707265 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:53:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.244+0000 7fa44de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644c1ac0460 0x5644c1ac2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.245+0000 7fa44f200640 1 -- 172.21.15.45:0/2322707265 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5644c1ab70f0 msgr2=0x5644c1ab7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:53:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.245+0000 7fa44f200640 1 --2- 172.21.15.45:0/2322707265 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5644c1ab70f0 0x5644c1ab7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.245+0000 7fa44f200640 1 -- 172.21.15.45:0/2322707265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644c1ac0460 msgr2=0x5644c1ac2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:53:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.245+0000 7fa44f200640 1 --2- 172.21.15.45:0/2322707265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644c1ac0460 0x5644c1ac2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.245+0000 7fa44f200640 1 -- 172.21.15.45:0/2322707265 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644c19ae690 con 0x5644c1ab7dd0 2024-03-28T17:53:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.245+0000 7fa447400640 1 --2- 172.21.15.45:0/2869502933 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fa4340a29f0 0x7fa4340a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.245+0000 7fa44f200640 1 --2- 172.21.15.45:0/2322707265 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab7dd0 0x5644c1ab6b20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa44800bc60 tx=0x7fa44800f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e585b762c582278a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:53:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.245+0000 7fa447400640 1 -- 172.21.15.45:0/2869502933 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9286+0+0 (secure 0 0 0) 0x7fa4400f0d30 con 0x7fa450107e50 2024-03-28T17:53:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.245+0000 7fa44e800640 1 --2- 172.21.15.45:0/2869502933 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fa4340a29f0 0x7fa4340a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.245+0000 7fa43f400640 1 -- 172.21.15.45:0/2322707265 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa448015070 con 0x5644c1ab7dd0 2024-03-28T17:53:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.245+0000 7fa43f400640 1 -- 172.21.15.45:0/2322707265 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa448005500 con 0x5644c1ab7dd0 2024-03-28T17:53:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.245+0000 7fa43f400640 1 -- 172.21.15.45:0/2322707265 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4480116a0 con 0x5644c1ab7dd0 2024-03-28T17:53:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.246+0000 7fa45817b740 1 -- 172.21.15.45:0/2322707265 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab7dd0 msgr2=0x5644c1ab6b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:53:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.246+0000 7fa44e800640 1 --2- 172.21.15.45:0/2869502933 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fa4340a29f0 0x7fa4340a4eb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa4400ee340 tx=0x7fa4400f4880 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:53:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.246+0000 7fa45817b740 1 --2- 172.21.15.45:0/2322707265 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab7dd0 0x5644c1ab6b20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa44800bc60 tx=0x7fa44800f730 comp rx=0 tx=0).stop 2024-03-28T17:53:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.246+0000 7fa45817b740 1 -- 172.21.15.45:0/2322707265 shutdown_connections 2024-03-28T17:53:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.246+0000 7fa45817b740 1 --2- 172.21.15.45:0/2322707265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644c1ac0460 0x5644c1ac2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.246+0000 7fa45817b740 1 --2- 172.21.15.45:0/2322707265 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5644c1ab70f0 0x5644c1ab7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.246+0000 7fa45817b740 1 --2- 172.21.15.45:0/2322707265 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab7dd0 0x5644c1ab6b20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.246+0000 7fa45817b740 1 -- 172.21.15.45:0/2322707265 >> 172.21.15.45:0/2322707265 conn(0x5644c1a7e200 msgr2=0x5644c1ab1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:53:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.246+0000 7fa45817b740 1 -- 172.21.15.45:0/2322707265 shutdown_connections 2024-03-28T17:53:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.246+0000 7fa45817b740 1 -- 172.21.15.45:0/2322707265 wait complete. 2024-03-28T17:53:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.246+0000 7fa45817b740 1 -- start start 2024-03-28T17:53:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.247+0000 7fa45817b740 1 Processor -- start 2024-03-28T17:53:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.247+0000 7fa45817b740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab70f0 0x5644c1abfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.247+0000 7fa45817b740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5644c1ab7dd0 0x5644c1abceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.247+0000 7fa45817b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644c1ac0460 0x5644c1abd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.247+0000 7fa45817b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644c1ac32d0 con 0x5644c1ac0460 2024-03-28T17:53:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.247+0000 7fa45817b740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644c19d6e80 con 0x5644c1ab70f0 2024-03-28T17:53:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.247+0000 7fa45817b740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644c1ac3450 con 0x5644c1ab7dd0 2024-03-28T17:53:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.247+0000 7fa45817b740 10 client.0 _ll_register_callbacks cb 0x5644c19e9130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:53:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.248+0000 7fa44f200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab70f0 0x5644c1abfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.248+0000 7fa44f200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab70f0 0x5644c1abfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:58164/0 (socket says 172.21.15.45:58164) 2024-03-28T17:53:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.248+0000 7fa44f200640 1 -- 172.21.15.45:0/2245907489 learned_addr learned my addr 172.21.15.45:0/2245907489 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:53:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.248+0000 7fa44de00640 1 --2- 172.21.15.45:0/2245907489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5644c1ab7dd0 0x5644c1abceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.248+0000 7fa44f200640 1 --2- 172.21.15.45:0/2245907489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644c1ac0460 0x5644c1abd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.248+0000 7fa44f200640 1 -- 172.21.15.45:0/2245907489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5644c1ab7dd0 msgr2=0x5644c1abceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:53:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.248+0000 7fa44f200640 1 --2- 172.21.15.45:0/2245907489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5644c1ab7dd0 0x5644c1abceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.248+0000 7fa44f200640 1 -- 172.21.15.45:0/2245907489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644c1ac0460 msgr2=0x5644c1abd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:53:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.248+0000 7fa44f200640 1 --2- 172.21.15.45:0/2245907489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644c1ac0460 0x5644c1abd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.248+0000 7fa44f200640 1 -- 172.21.15.45:0/2245907489 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5644c17f97d0 con 0x5644c1ab70f0 2024-03-28T17:53:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.248+0000 7fa44f200640 1 --2- 172.21.15.45:0/2245907489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab70f0 0x5644c1abfb20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa44800fc30 tx=0x7fa448004ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24d000c11486b7b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:53:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.248+0000 7fa433e00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:53:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.248+0000 7fa433e00640 1 -- 172.21.15.45:0/2245907489 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa448015040 con 0x5644c1ab70f0 2024-03-28T17:53:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.249+0000 7fa45817b740 1 -- 172.21.15.45:0/2245907489 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa448005500 con 0x5644c1ab70f0 2024-03-28T17:53:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.249+0000 7fa433e00640 1 -- 172.21.15.45:0/2245907489 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9286+0+0 (secure 0 0 0) 0x7fa4480050f0 con 0x5644c1ab70f0 2024-03-28T17:53:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.249+0000 7fa433e00640 1 -- 172.21.15.45:0/2245907489 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4480052f0 con 0x5644c1ab70f0 2024-03-28T17:53:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.249+0000 7fa45817b740 10 client.26033 fetch_fsmap learned FSMap version 83 2024-03-28T17:53:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.249+0000 7fa45817b740 1 -- 172.21.15.45:0/2245907489 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=4+}) v3 -- 0x5644c172f3f0 con 0x5644c1ab70f0 2024-03-28T17:53:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.249+0000 7fa433e00640 1 -- 172.21.15.45:0/2245907489 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa448005500 con 0x5644c1ab70f0 2024-03-28T17:53:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.249+0000 7fa433e00640 1 -- 172.21.15.45:0/2245907489 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 83) v1 ==== 7343+0+0 (secure 0 0 0) 0x7fa448027040 con 0x5644c1ab70f0 2024-03-28T17:53:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.249+0000 7fa45817b740 10 client.26033 fetch_fsmap finished waiting for FSMap version 83 2024-03-28T17:53:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.249+0000 7fa45817b740 10 client.26033 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:53:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.249+0000 7fa45817b740 20 client.26033 populate_metadata read hostname 'smithi045' 2024-03-28T17:53:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.249+0000 7fa45817b740 1 --2- 172.21.15.45:0/2245907489 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5644c1ad0250 0x5644c1ad2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.249+0000 7fa45817b740 4 client.26033 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:53:03.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.249+0000 7fa45817b740 1 -- 172.21.15.45:0/2245907489 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5644c1a28790 con 0x5644c1ad0250 2024-03-28T17:53:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.250+0000 7fa44de00640 1 --2- 172.21.15.45:0/2245907489 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5644c1ad0250 0x5644c1ad2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.250+0000 7fa44de00640 1 --2- 172.21.15.45:0/2245907489 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5644c1ad0250 0x5644c1ad2640 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ba0a60db16a8ea2 server_cookie=d1f234f0211d8537 in_seq=0 out_seq=0 2024-03-28T17:53:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.250+0000 7fa433e00640 10 client.26033 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:53:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.251+0000 7fa433e00640 1 -- 172.21.15.45:0/2245907489 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5644c1a28790 con 0x5644c1ad0250 2024-03-28T17:53:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.251+0000 7fa433e00640 10 client.26033 handle_command_reply: tid=0 2024-03-28T17:53:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.251+0000 7fa45817b740 1 client.26033 shutdown 2024-03-28T17:53:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.251+0000 7fa45817b740 1 -- 172.21.15.45:0/2245907489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab70f0 msgr2=0x5644c1abfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:53:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.251+0000 7fa45817b740 1 --2- 172.21.15.45:0/2245907489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab70f0 0x5644c1abfb20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa44800fc30 tx=0x7fa448004ce0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.252+0000 7fa45817b740 1 -- 172.21.15.45:0/2245907489 shutdown_connections 2024-03-28T17:53:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.252+0000 7fa45817b740 1 --2- 172.21.15.45:0/2245907489 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5644c1ad0250 0x5644c1ad2640 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.252+0000 7fa45817b740 1 --2- 172.21.15.45:0/2245907489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644c1ac0460 0x5644c1abd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.252+0000 7fa433e00640 0 client.26033 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:53:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.252+0000 7fa45817b740 1 --2- 172.21.15.45:0/2245907489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5644c1ab7dd0 0x5644c1abceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.252+0000 7fa45817b740 1 --2- 172.21.15.45:0/2245907489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab70f0 0x5644c1abfb20 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.252+0000 7fa45817b740 1 -- 172.21.15.45:0/2245907489 >> 172.21.15.45:0/2245907489 conn(0x5644c1a7e200 msgr2=0x5644c1ab3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:53:03.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.252+0000 7fa45817b740 1 -- 172.21.15.45:0/2245907489 shutdown_connections 2024-03-28T17:53:03.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.252+0000 7fa45817b740 1 -- 172.21.15.45:0/2245907489 wait complete. 2024-03-28T17:53:03.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.252+0000 7fa45817b740 20 client.26033 trim_cache size 0 max 16384 2024-03-28T17:53:03.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.270+0000 7fa45817b740 1 Processor -- start 2024-03-28T17:53:03.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.270+0000 7fa45817b740 1 -- start start 2024-03-28T17:53:03.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.270+0000 7fa45817b740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5644c1ab70f0 0x5644c1abbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:03.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.270+0000 7fa45817b740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab7dd0 0x5644c1abf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:03.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.270+0000 7fa45817b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644c1ac0460 0x5644c1abfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:03.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.270+0000 7fa45817b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4501b5580 con 0x5644c1ac0460 2024-03-28T17:53:03.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.270+0000 7fa45817b740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4501b5be0 con 0x5644c1ab7dd0 2024-03-28T17:53:03.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.270+0000 7fa45817b740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4501b3720 con 0x5644c1ab70f0 2024-03-28T17:53:03.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.271+0000 7fa44f200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5644c1ab70f0 0x5644c1abbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:03.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.271+0000 7fa44de00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab7dd0 0x5644c1abf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:03.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.271+0000 7fa44f200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5644c1ab70f0 0x5644c1abbef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:43476/0 (socket says 172.21.15.45:43476) 2024-03-28T17:53:03.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.271+0000 7fa44f200640 1 -- 172.21.15.45:0/2298149943 learned_addr learned my addr 172.21.15.45:0/2298149943 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:53:03.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.271+0000 7fa44f200640 1 --2- 172.21.15.45:0/2298149943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644c1ac0460 0x5644c1abfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:03.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.271+0000 7fa44de00640 1 -- 172.21.15.45:0/2298149943 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5644c1ab70f0 msgr2=0x5644c1abbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:53:03.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.271+0000 7fa44de00640 1 --2- 172.21.15.45:0/2298149943 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5644c1ab70f0 0x5644c1abbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.271+0000 7fa44de00640 1 -- 172.21.15.45:0/2298149943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644c1ac0460 msgr2=0x5644c1abfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:53:03.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.271+0000 7fa44de00640 1 --2- 172.21.15.45:0/2298149943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644c1ac0460 0x5644c1abfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.271+0000 7fa44de00640 1 -- 172.21.15.45:0/2298149943 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644c16ef0f0 con 0x5644c1ab7dd0 2024-03-28T17:53:03.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.272+0000 7fa44de00640 1 --2- 172.21.15.45:0/2298149943 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab7dd0 0x5644c1abf5e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x5644c1abc730 tx=0x7fa438005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b5b68f52f235bc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:53:03.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.272+0000 7fa43ea00640 1 -- 172.21.15.45:0/2298149943 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43801f310 con 0x5644c1ab7dd0 2024-03-28T17:53:03.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.272+0000 7fa43ea00640 1 -- 172.21.15.45:0/2298149943 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa43801fbe0 con 0x5644c1ab7dd0 2024-03-28T17:53:03.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.272+0000 7fa43ea00640 1 -- 172.21.15.45:0/2298149943 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa438010830 con 0x5644c1ab7dd0 2024-03-28T17:53:03.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.272+0000 7fa45817b740 1 -- 172.21.15.45:0/2298149943 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab7dd0 msgr2=0x5644c1abf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:53:03.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.272+0000 7fa45817b740 1 --2- 172.21.15.45:0/2298149943 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab7dd0 0x5644c1abf5e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x5644c1abc730 tx=0x7fa438005f90 comp rx=0 tx=0).stop 2024-03-28T17:53:03.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.272+0000 7fa45817b740 1 -- 172.21.15.45:0/2298149943 shutdown_connections 2024-03-28T17:53:03.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.272+0000 7fa45817b740 1 --2- 172.21.15.45:0/2298149943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644c1ac0460 0x5644c1abfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.272+0000 7fa45817b740 1 --2- 172.21.15.45:0/2298149943 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab7dd0 0x5644c1abf5e0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.272+0000 7fa45817b740 1 --2- 172.21.15.45:0/2298149943 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5644c1ab70f0 0x5644c1abbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.272+0000 7fa45817b740 1 -- 172.21.15.45:0/2298149943 >> 172.21.15.45:0/2298149943 conn(0x5644c1a7e200 msgr2=0x5644c1abb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:53:03.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.273+0000 7fa45817b740 1 -- 172.21.15.45:0/2298149943 shutdown_connections 2024-03-28T17:53:03.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.273+0000 7fa45817b740 1 -- 172.21.15.45:0/2298149943 wait complete. 2024-03-28T17:53:03.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.273+0000 7fa45817b740 1 -- start start 2024-03-28T17:53:03.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.273+0000 7fa45817b740 1 Processor -- start 2024-03-28T17:53:03.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.274+0000 7fa45817b740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab70f0 0x5644c1ab9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:03.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.274+0000 7fa45817b740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5644c1ab7dd0 0x5644c1ab9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:03.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.274+0000 7fa45817b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644c1ac0460 0x5644c1ab9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:03.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.274+0000 7fa45817b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4501b30c0 con 0x5644c1ac0460 2024-03-28T17:53:03.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.274+0000 7fa45817b740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4501b1e20 con 0x5644c1ab70f0 2024-03-28T17:53:03.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.274+0000 7fa45817b740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4501b2420 con 0x5644c1ab7dd0 2024-03-28T17:53:03.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.274+0000 7fa45817b740 10 client.0 _ll_register_callbacks cb 0x7fa4380097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:53:03.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.274+0000 7fa44f200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab70f0 0x5644c1ab9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:03.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.274+0000 7fa44de00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5644c1ab7dd0 0x5644c1ab9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:03.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.274+0000 7fa44f200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab70f0 0x5644c1ab9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:58188/0 (socket says 172.21.15.45:58188) 2024-03-28T17:53:03.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.274+0000 7fa44f200640 1 -- 172.21.15.45:0/1811992997 learned_addr learned my addr 172.21.15.45:0/1811992997 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:53:03.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.274+0000 7fa44de00640 1 --2- 172.21.15.45:0/1811992997 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5644c1ab7dd0 0x5644c1ab9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:43484/0 (socket says 172.21.15.45:43484) 2024-03-28T17:53:03.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.274+0000 7fa44f200640 1 --2- 172.21.15.45:0/1811992997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644c1ac0460 0x5644c1ab9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:03.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.274+0000 7fa44de00640 1 -- 172.21.15.45:0/1811992997 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab70f0 msgr2=0x5644c1ab9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:53:03.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.274+0000 7fa44de00640 1 --2- 172.21.15.45:0/1811992997 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab70f0 0x5644c1ab9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.274+0000 7fa44de00640 1 -- 172.21.15.45:0/1811992997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644c1ac0460 msgr2=0x5644c1ab9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:53:03.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.274+0000 7fa44de00640 1 --2- 172.21.15.45:0/1811992997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644c1ac0460 0x5644c1ab9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.274+0000 7fa44de00640 1 -- 172.21.15.45:0/1811992997 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5644c1b2a9f0 con 0x5644c1ab7dd0 2024-03-28T17:53:03.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.275+0000 7fa44de00640 1 --2- 172.21.15.45:0/1811992997 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5644c1ab7dd0 0x5644c1ab9ab0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x5644c1abe240 tx=0x7fa438020da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1849a9e9b73e3635 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:53:03.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.275+0000 7fa433e00640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:53:03.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.275+0000 7fa433e00640 1 -- 172.21.15.45:0/1811992997 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa438003d60 con 0x5644c1ab7dd0 2024-03-28T17:53:03.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.275+0000 7fa433e00640 1 -- 172.21.15.45:0/1811992997 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9286+0+0 (secure 0 0 0) 0x7fa43801f700 con 0x5644c1ab7dd0 2024-03-28T17:53:03.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.275+0000 7fa45817b740 1 -- 172.21.15.45:0/1811992997 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa43801fbe0 con 0x5644c1ab7dd0 2024-03-28T17:53:03.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.275+0000 7fa433e00640 1 -- 172.21.15.45:0/1811992997 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa438037050 con 0x5644c1ab7dd0 2024-03-28T17:53:03.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.275+0000 7fa45817b740 10 client.26002 fetch_fsmap learned FSMap version 83 2024-03-28T17:53:03.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.276+0000 7fa45817b740 1 -- 172.21.15.45:0/1811992997 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=4+}) v3 -- 0x5644c1b2bca0 con 0x5644c1ab7dd0 2024-03-28T17:53:03.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.276+0000 7fa433e00640 1 -- 172.21.15.45:0/1811992997 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa43801fbe0 con 0x5644c1ab7dd0 2024-03-28T17:53:03.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.276+0000 7fa433e00640 1 -- 172.21.15.45:0/1811992997 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 83) v1 ==== 7343+0+0 (secure 0 0 0) 0x7fa438004960 con 0x5644c1ab7dd0 2024-03-28T17:53:03.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.276+0000 7fa45817b740 10 client.26002 fetch_fsmap finished waiting for FSMap version 83 2024-03-28T17:53:03.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.276+0000 7fa45817b740 10 client.26002 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:53:03.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.276+0000 7fa45817b740 20 client.26002 populate_metadata read hostname 'smithi045' 2024-03-28T17:53:03.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.276+0000 7fa45817b740 1 --2- 172.21.15.45:0/1811992997 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5644c1b2c370 0x5644c1b2e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:03.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.276+0000 7fa45817b740 4 client.26002 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:53:03.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.276+0000 7fa45817b740 1 -- 172.21.15.45:0/1811992997 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5644c1a22ae0 con 0x5644c1b2c370 2024-03-28T17:53:03.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.277+0000 7fa44f200640 1 --2- 172.21.15.45:0/1811992997 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5644c1b2c370 0x5644c1b2e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:03.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.277+0000 7fa44f200640 1 --2- 172.21.15.45:0/1811992997 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5644c1b2c370 0x5644c1b2e790 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4072b615d456eb6b server_cookie=a7993f3dddab7961 in_seq=0 out_seq=0 2024-03-28T17:53:03.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.277+0000 7fa433e00640 10 client.26002 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:53:03.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.277+0000 7fa433e00640 1 -- 172.21.15.45:0/1811992997 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5644c1a22ae0 con 0x5644c1b2c370 2024-03-28T17:53:03.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.277+0000 7fa433e00640 10 client.26002 handle_command_reply: tid=0 2024-03-28T17:53:03.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.277+0000 7fa45817b740 1 client.26002 shutdown 2024-03-28T17:53:03.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.277+0000 7fa45817b740 1 -- 172.21.15.45:0/1811992997 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5644c1ab7dd0 msgr2=0x5644c1ab9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:53:03.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.277+0000 7fa45817b740 1 --2- 172.21.15.45:0/1811992997 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5644c1ab7dd0 0x5644c1ab9ab0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x5644c1abe240 tx=0x7fa438020da0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.277+0000 7fa45817b740 1 -- 172.21.15.45:0/1811992997 shutdown_connections 2024-03-28T17:53:03.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.277+0000 7fa45817b740 1 --2- 172.21.15.45:0/1811992997 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5644c1b2c370 0x5644c1b2e790 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.278+0000 7fa45817b740 1 --2- 172.21.15.45:0/1811992997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644c1ac0460 0x5644c1ab9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.278+0000 7fa433e00640 0 client.26002 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:53:03.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.278+0000 7fa45817b740 1 --2- 172.21.15.45:0/1811992997 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5644c1ab7dd0 0x5644c1ab9ab0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.278+0000 7fa45817b740 1 --2- 172.21.15.45:0/1811992997 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5644c1ab70f0 0x5644c1ab9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.278+0000 7fa45817b740 1 -- 172.21.15.45:0/1811992997 >> 172.21.15.45:0/1811992997 conn(0x5644c1a7e200 msgr2=0x5644c1ab2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:53:03.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.278+0000 7fa45817b740 1 -- 172.21.15.45:0/1811992997 shutdown_connections 2024-03-28T17:53:03.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.278+0000 7fa45817b740 1 -- 172.21.15.45:0/1811992997 wait complete. 2024-03-28T17:53:03.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.278+0000 7fa45817b740 20 client.26002 trim_cache size 0 max 16384 2024-03-28T17:53:03.281 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:53:03.281 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T17:53:03.281 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T17:53:03.281 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T17:53:03.281 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T17:53:03.282 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T17:53:03.282 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T17:53:03.282 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:53:03.282 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:53:03.282 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:53:03.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.278+0000 7fa454a00640 1 -- 172.21.15.45:0/2869502933 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fa4340a29f0 msgr2=0x7fa4340a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:53:03.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.278+0000 7fa454a00640 1 --2- 172.21.15.45:0/2869502933 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fa4340a29f0 0x7fa4340a4eb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa4400ee340 tx=0x7fa4400f4880 comp rx=0 tx=0).stop 2024-03-28T17:53:03.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.278+0000 7fa454a00640 1 -- 172.21.15.45:0/2869502933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa450107e50 msgr2=0x7fa45019e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:53:03.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.278+0000 7fa454a00640 1 --2- 172.21.15.45:0/2869502933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa450107e50 0x7fa45019e3f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa440005e30 tx=0x7fa440032f40 comp rx=0 tx=0).stop 2024-03-28T17:53:03.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.279+0000 7fa454a00640 1 -- 172.21.15.45:0/2869502933 shutdown_connections 2024-03-28T17:53:03.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.279+0000 7fa454a00640 1 --2- 172.21.15.45:0/2869502933 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fa4340a29f0 0x7fa4340a4eb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.279+0000 7fa454a00640 1 --2- 172.21.15.45:0/2869502933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4501a2c60 0x7fa4501a3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.279+0000 7fa454a00640 1 --2- 172.21.15.45:0/2869502933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa45010cea0 0x7fa45019e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.279+0000 7fa454a00640 1 --2- 172.21.15.45:0/2869502933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa450107e50 0x7fa45019e3f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:03.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.279+0000 7fa454a00640 1 -- 172.21.15.45:0/2869502933 >> 172.21.15.45:0/2869502933 conn(0x7fa4500774c0 msgr2=0x7fa450077a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:53:03.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.279+0000 7fa454a00640 1 -- 172.21.15.45:0/2869502933 shutdown_connections 2024-03-28T17:53:03.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:03.279+0000 7fa454a00640 1 -- 172.21.15.45:0/2869502933 wait complete. 2024-03-28T17:53:03.297 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T17:53:03.854 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 36 sec latency 151750.805 ms 2024-03-28T17:53:04.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:04 smithi071 ceph-mon[36597]: pgmap v134: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 171 B/s wr, 8 op/s 2024-03-28T17:53:04.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:04 smithi151 ceph-mon[36481]: pgmap v134: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 171 B/s wr, 8 op/s 2024-03-28T17:53:04.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:04 smithi045 ceph-mon[31773]: pgmap v134: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 171 B/s wr, 8 op/s 2024-03-28T17:53:04.854 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 37 sec latency 152750.892 ms 2024-03-28T17:53:05.854 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 38 sec latency 153751.062 ms 2024-03-28T17:53:06.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:06 smithi151 ceph-mon[36481]: pgmap v135: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T17:53:06.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:06 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 5. 2024-03-28T17:53:06.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:06 smithi151 systemd[1]: Stopped Ceph mds.f for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:53:06.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:06 smithi151 systemd[1]: Starting Ceph mds.f for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:53:06.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:06 smithi151 podman[72455]: 2024-03-28 17:53:06.211274666 +0000 UTC m=+0.017894177 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:53:06.325 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:06 smithi151 podman[72455]: 2024-03-28 17:53:06.32562348 +0000 UTC m=+0.132242982 container create 5ae7b83083827dffe6bff11421e0e55fab896a968d1b5ce297bc85d0ac18f908 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T17:53:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:06 smithi071 ceph-mon[36597]: pgmap v135: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T17:53:06.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:06 smithi045 ceph-mon[31773]: pgmap v135: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T17:53:06.854 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 39 sec latency 154751.214 ms 2024-03-28T17:53:07.073 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:06 smithi151 podman[72455]: 2024-03-28 17:53:06.594780018 +0000 UTC m=+0.401399534 container init 5ae7b83083827dffe6bff11421e0e55fab896a968d1b5ce297bc85d0ac18f908 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-28T17:53:07.073 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:06 smithi151 podman[72455]: 2024-03-28 17:53:06.597343181 +0000 UTC m=+0.403962684 container start 5ae7b83083827dffe6bff11421e0e55fab896a968d1b5ce297bc85d0ac18f908 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-28T17:53:07.074 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:06 smithi151 ceph-mds[72473]: -- 172.21.15.151:0/26152977 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55887a55fa40 con 0x55887a533000 2024-03-28T17:53:07.074 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:06 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: starting mds.f at 2024-03-28T17:53:07.074 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:06 smithi151 bash[72455]: 5ae7b83083827dffe6bff11421e0e55fab896a968d1b5ce297bc85d0ac18f908 2024-03-28T17:53:07.074 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:06 smithi151 systemd[1]: Started Ceph mds.f for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:53:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:07 smithi071 ceph-mon[36597]: daemon mds.f restarted 2024-03-28T17:53:07.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:07 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:53:07.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:07 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T17:53:07.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:07 smithi071 ceph-mon[36597]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T17:53:07.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:07 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/121098913,v1:172.21.15.151:6835/121098913] up:boot 2024-03-28T17:53:07.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:07 smithi071 ceph-mon[36597]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-03-28T17:53:07.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:07 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-03-28T17:53:07.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:07 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:replay} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:07 smithi151 ceph-mon[36481]: daemon mds.f restarted 2024-03-28T17:53:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:07 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:53:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:07 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T17:53:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:07 smithi151 ceph-mon[36481]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T17:53:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:07 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/121098913,v1:172.21.15.151:6835/121098913] up:boot 2024-03-28T17:53:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:07 smithi151 ceph-mon[36481]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-03-28T17:53:07.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:07 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-03-28T17:53:07.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:07 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:replay} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:07 smithi045 ceph-mon[31773]: daemon mds.f restarted 2024-03-28T17:53:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:07 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:53:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:07 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T17:53:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:07 smithi045 ceph-mon[31773]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T17:53:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:07 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/121098913,v1:172.21.15.151:6835/121098913] up:boot 2024-03-28T17:53:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:07 smithi045 ceph-mon[31773]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-03-28T17:53:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:07 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-03-28T17:53:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:07 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:replay} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:07.854 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 40 sec latency 155751.381 ms 2024-03-28T17:53:08.854 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 41 sec latency 156751.515 ms 2024-03-28T17:53:08.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:08 smithi045 ceph-mon[31773]: pgmap v137: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-28T17:53:08.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:08 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/1262971887,v1:172.21.15.45:6837/1262971887] up:resolve 2024-03-28T17:53:08.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:08 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:resolve} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:09.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:08 smithi071 ceph-mon[36597]: pgmap v137: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-28T17:53:09.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:08 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/1262971887,v1:172.21.15.45:6837/1262971887] up:resolve 2024-03-28T17:53:09.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:08 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:resolve} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:09.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:08 smithi151 ceph-mon[36481]: pgmap v137: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-28T17:53:09.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:08 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/1262971887,v1:172.21.15.45:6837/1262971887] up:resolve 2024-03-28T17:53:09.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:08 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:resolve} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:09.855 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 42 sec latency 157751.678 ms 2024-03-28T17:53:10.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:10 smithi071 ceph-mon[36597]: pgmap v138: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-28T17:53:10.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:10 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:53:10.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:10 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/1262971887,v1:172.21.15.45:6837/1262971887] up:reconnect 2024-03-28T17:53:10.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:10 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:reconnect} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:10.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:10 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T17:53:10.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:10 smithi151 ceph-mon[36481]: pgmap v138: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-28T17:53:10.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:10 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:53:10.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:10 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/1262971887,v1:172.21.15.45:6837/1262971887] up:reconnect 2024-03-28T17:53:10.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:10 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:reconnect} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:10.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:10 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T17:53:10.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:10 smithi045 ceph-mon[31773]: pgmap v138: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-28T17:53:10.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:10 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:53:10.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:10 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/1262971887,v1:172.21.15.45:6837/1262971887] up:reconnect 2024-03-28T17:53:10.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:10 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:reconnect} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:10.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:10 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T17:53:10.855 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 43 sec latency 158751.834 ms 2024-03-28T17:53:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:11 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/1262971887,v1:172.21.15.45:6837/1262971887] up:rejoin 2024-03-28T17:53:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:11 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:rejoin} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:11 smithi071 ceph-mon[36597]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-03-28T17:53:11.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:11 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:53:11.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:11 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:53:11.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:11 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/1262971887,v1:172.21.15.45:6837/1262971887] up:active 2024-03-28T17:53:11.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:11 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 3 up:standby-replay 5 up:standby 2024-03-28T17:53:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:11 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/1262971887,v1:172.21.15.45:6837/1262971887] up:rejoin 2024-03-28T17:53:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:11 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:rejoin} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:11 smithi151 ceph-mon[36481]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-03-28T17:53:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:11 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:53:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:11 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:53:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:11 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/1262971887,v1:172.21.15.45:6837/1262971887] up:active 2024-03-28T17:53:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:11 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 3 up:standby-replay 5 up:standby 2024-03-28T17:53:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/1262971887,v1:172.21.15.45:6837/1262971887] up:rejoin 2024-03-28T17:53:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:rejoin} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-mon[31773]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-03-28T17:53:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:53:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:53:11.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/1262971887,v1:172.21.15.45:6837/1262971887] up:active 2024-03-28T17:53:11.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 3 up:standby-replay 5 up:standby 2024-03-28T17:53:11.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3be8200700 time 2024-03-28T17:53:11.205082+0000 2024-03-28T17:53:11.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:53:11.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:11.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3bf3b644a1] 2024-03-28T17:53:11.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3bf3b64667] 2024-03-28T17:53:11.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563fd1f53223] 2024-03-28T17:53:11.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563fd1dcd5d6] 2024-03-28T17:53:11.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563fd1dcdda1] 2024-03-28T17:53:11.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563fd1dd02d5] 2024-03-28T17:53:11.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563fd1e20233] 2024-03-28T17:53:11.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563fd1e22197] 2024-03-28T17:53:11.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563fd1d2111b] 2024-03-28T17:53:11.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563fd1d216bc] 2024-03-28T17:53:11.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563fd1d0b1ff] 2024-03-28T17:53:11.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3bf3df6488] 2024-03-28T17:53:11.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 13: (DispatchQueue::entry()+0x51f) [0x7f3bf3df360f] 2024-03-28T17:53:11.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3bf3ec6e81] 2024-03-28T17:53:11.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf28e11ca] 2024-03-28T17:53:11.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 16: clone() 2024-03-28T17:53:11.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: *** Caught signal (Aborted) ** 2024-03-28T17:53:11.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: in thread 7f3be8200700 thread_name:ms_dispatch 2024-03-28T17:53:11.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 2024-03-28T17:53:11.206+0000 7f3be8200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3be8200700 time 2024-03-28T17:53:11.205082+0000 2024-03-28T17:53:11.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:53:11.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 2024-03-28T17:53:11.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:11.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3bf3b644a1] 2024-03-28T17:53:11.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3bf3b64667] 2024-03-28T17:53:11.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563fd1f53223] 2024-03-28T17:53:11.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563fd1dcd5d6] 2024-03-28T17:53:11.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563fd1dcdda1] 2024-03-28T17:53:11.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563fd1dd02d5] 2024-03-28T17:53:11.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563fd1e20233] 2024-03-28T17:53:11.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563fd1e22197] 2024-03-28T17:53:11.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563fd1d2111b] 2024-03-28T17:53:11.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563fd1d216bc] 2024-03-28T17:53:11.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563fd1d0b1ff] 2024-03-28T17:53:11.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3bf3df6488] 2024-03-28T17:53:11.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 13: (DispatchQueue::entry()+0x51f) [0x7f3bf3df360f] 2024-03-28T17:53:11.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3bf3ec6e81] 2024-03-28T17:53:11.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf28e11ca] 2024-03-28T17:53:11.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 16: clone() 2024-03-28T17:53:11.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 2024-03-28T17:53:11.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:11.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3bf28ebd20] 2024-03-28T17:53:11.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 2: gsignal() 2024-03-28T17:53:11.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 3: abort() 2024-03-28T17:53:11.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3bf3b644fb] 2024-03-28T17:53:11.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3bf3b64667] 2024-03-28T17:53:11.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563fd1f53223] 2024-03-28T17:53:11.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563fd1dcd5d6] 2024-03-28T17:53:11.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563fd1dcdda1] 2024-03-28T17:53:11.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563fd1dd02d5] 2024-03-28T17:53:11.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563fd1e20233] 2024-03-28T17:53:11.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563fd1e22197] 2024-03-28T17:53:11.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563fd1d2111b] 2024-03-28T17:53:11.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563fd1d216bc] 2024-03-28T17:53:11.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563fd1d0b1ff] 2024-03-28T17:53:11.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3bf3df6488] 2024-03-28T17:53:11.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 16: (DispatchQueue::entry()+0x51f) [0x7f3bf3df360f] 2024-03-28T17:53:11.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3bf3ec6e81] 2024-03-28T17:53:11.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf28e11ca] 2024-03-28T17:53:11.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 19: clone() 2024-03-28T17:53:11.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 2024-03-28T17:53:11.210+0000 7f3be8200700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:53:11.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: in thread 7f3be8200700 thread_name:ms_dispatch 2024-03-28T17:53:11.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 2024-03-28T17:53:11.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:11.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3bf28ebd20] 2024-03-28T17:53:11.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 2: gsignal() 2024-03-28T17:53:11.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 3: abort() 2024-03-28T17:53:11.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3bf3b644fb] 2024-03-28T17:53:11.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3bf3b64667] 2024-03-28T17:53:11.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563fd1f53223] 2024-03-28T17:53:11.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563fd1dcd5d6] 2024-03-28T17:53:11.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563fd1dcdda1] 2024-03-28T17:53:11.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563fd1dd02d5] 2024-03-28T17:53:11.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563fd1e20233] 2024-03-28T17:53:11.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563fd1e22197] 2024-03-28T17:53:11.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563fd1d2111b] 2024-03-28T17:53:11.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563fd1d216bc] 2024-03-28T17:53:11.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563fd1d0b1ff] 2024-03-28T17:53:11.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3bf3df6488] 2024-03-28T17:53:11.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 16: (DispatchQueue::entry()+0x51f) [0x7f3bf3df360f] 2024-03-28T17:53:11.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3bf3ec6e81] 2024-03-28T17:53:11.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf28e11ca] 2024-03-28T17:53:11.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 19: clone() 2024-03-28T17:53:11.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:53:11.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 2024-03-28T17:53:11.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: -1> 2024-03-28T17:53:11.206+0000 7f3be8200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3be8200700 time 2024-03-28T17:53:11.205082+0000 2024-03-28T17:53:11.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:53:11.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 2024-03-28T17:53:11.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:11.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3bf3b644a1] 2024-03-28T17:53:11.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3bf3b64667] 2024-03-28T17:53:11.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563fd1f53223] 2024-03-28T17:53:11.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563fd1dcd5d6] 2024-03-28T17:53:11.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563fd1dcdda1] 2024-03-28T17:53:11.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563fd1dd02d5] 2024-03-28T17:53:11.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563fd1e20233] 2024-03-28T17:53:11.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563fd1e22197] 2024-03-28T17:53:11.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563fd1d2111b] 2024-03-28T17:53:11.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563fd1d216bc] 2024-03-28T17:53:11.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563fd1d0b1ff] 2024-03-28T17:53:11.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3bf3df6488] 2024-03-28T17:53:11.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 13: (DispatchQueue::entry()+0x51f) [0x7f3bf3df360f] 2024-03-28T17:53:11.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3bf3ec6e81] 2024-03-28T17:53:11.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf28e11ca] 2024-03-28T17:53:11.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 16: clone() 2024-03-28T17:53:11.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 2024-03-28T17:53:11.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 0> 2024-03-28T17:53:11.210+0000 7f3be8200700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:53:11.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: in thread 7f3be8200700 thread_name:ms_dispatch 2024-03-28T17:53:11.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 2024-03-28T17:53:11.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:11.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3bf28ebd20] 2024-03-28T17:53:11.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 2: gsignal() 2024-03-28T17:53:11.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 3: abort() 2024-03-28T17:53:11.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3bf3b644fb] 2024-03-28T17:53:11.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3bf3b64667] 2024-03-28T17:53:11.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563fd1f53223] 2024-03-28T17:53:11.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563fd1dcd5d6] 2024-03-28T17:53:11.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563fd1dcdda1] 2024-03-28T17:53:11.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563fd1dd02d5] 2024-03-28T17:53:11.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563fd1e20233] 2024-03-28T17:53:11.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563fd1e22197] 2024-03-28T17:53:11.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563fd1d2111b] 2024-03-28T17:53:11.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563fd1d216bc] 2024-03-28T17:53:11.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563fd1d0b1ff] 2024-03-28T17:53:11.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3bf3df6488] 2024-03-28T17:53:11.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 16: (DispatchQueue::entry()+0x51f) [0x7f3bf3df360f] 2024-03-28T17:53:11.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3bf3ec6e81] 2024-03-28T17:53:11.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf28e11ca] 2024-03-28T17:53:11.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 19: clone() 2024-03-28T17:53:11.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:53:11.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 2024-03-28T17:53:11.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: -1> 2024-03-28T17:53:11.206+0000 7f3be8200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3be8200700 time 2024-03-28T17:53:11.205082+0000 2024-03-28T17:53:11.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:53:11.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 2024-03-28T17:53:11.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:11.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3bf3b644a1] 2024-03-28T17:53:11.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3bf3b64667] 2024-03-28T17:53:11.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563fd1f53223] 2024-03-28T17:53:11.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563fd1dcd5d6] 2024-03-28T17:53:11.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563fd1dcdda1] 2024-03-28T17:53:11.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563fd1dd02d5] 2024-03-28T17:53:11.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563fd1e20233] 2024-03-28T17:53:11.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563fd1e22197] 2024-03-28T17:53:11.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563fd1d2111b] 2024-03-28T17:53:11.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563fd1d216bc] 2024-03-28T17:53:11.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563fd1d0b1ff] 2024-03-28T17:53:11.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3bf3df6488] 2024-03-28T17:53:11.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 13: (DispatchQueue::entry()+0x51f) [0x7f3bf3df360f] 2024-03-28T17:53:11.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3bf3ec6e81] 2024-03-28T17:53:11.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf28e11ca] 2024-03-28T17:53:11.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 16: clone() 2024-03-28T17:53:11.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 2024-03-28T17:53:11.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 0> 2024-03-28T17:53:11.210+0000 7f3be8200700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:53:11.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: in thread 7f3be8200700 thread_name:ms_dispatch 2024-03-28T17:53:11.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 2024-03-28T17:53:11.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:11.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3bf28ebd20] 2024-03-28T17:53:11.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 2: gsignal() 2024-03-28T17:53:11.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 3: abort() 2024-03-28T17:53:11.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3bf3b644fb] 2024-03-28T17:53:11.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3bf3b64667] 2024-03-28T17:53:11.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563fd1f53223] 2024-03-28T17:53:11.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563fd1dcd5d6] 2024-03-28T17:53:11.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563fd1dcdda1] 2024-03-28T17:53:11.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563fd1dd02d5] 2024-03-28T17:53:11.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563fd1e20233] 2024-03-28T17:53:11.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563fd1e22197] 2024-03-28T17:53:11.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563fd1d2111b] 2024-03-28T17:53:11.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563fd1d216bc] 2024-03-28T17:53:11.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563fd1d0b1ff] 2024-03-28T17:53:11.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3bf3df6488] 2024-03-28T17:53:11.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 16: (DispatchQueue::entry()+0x51f) [0x7f3bf3df360f] 2024-03-28T17:53:11.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3bf3ec6e81] 2024-03-28T17:53:11.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf28e11ca] 2024-03-28T17:53:11.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 19: clone() 2024-03-28T17:53:11.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:53:11.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[72790]: 2024-03-28T17:53:11.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:11 smithi045 podman[89991]: 2024-03-28 17:53:11.335143371 +0000 UTC m=+0.033836322 container died 1902e980607f67272121c61ead4881d3d01e1ccc1d681ee589bef689ff93ef2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-28T17:53:11.855 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 44 sec latency 159751.988 ms 2024-03-28T17:53:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:12 smithi071 ceph-mon[36597]: pgmap v139: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:53:12.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:12 smithi151 ceph-mon[36481]: pgmap v139: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:53:12.665 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:12 smithi045 podman[89991]: 2024-03-28 17:53:12.211060049 +0000 UTC m=+0.909753045 container remove 1902e980607f67272121c61ead4881d3d01e1ccc1d681ee589bef689ff93ef2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-28T17:53:12.666 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:12 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:53:12.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:12 smithi045 ceph-mon[31773]: pgmap v139: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:53:12.855 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 45 sec latency 160752.078 ms 2024-03-28T17:53:13.166 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:12 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-28T17:53:13.166 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:12 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Consumed 1.833s CPU time. 2024-03-28T17:53:13.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:13 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:53:13.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:13 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:53:13.656 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:13 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:53:13.855 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 46 sec latency 161752.194 ms 2024-03-28T17:53:14.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:14 smithi045 ceph-mon[31773]: pgmap v140: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:53:14.559 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:14 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:53:14.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:14 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:53:14.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:14 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:53:14.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:14 smithi071 ceph-mon[36597]: pgmap v140: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:53:14.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:14 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:53:14.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:14 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:53:14.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:14 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:53:14.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:14 smithi151 ceph-mon[36481]: pgmap v140: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:53:14.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:14 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:53:14.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:14 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:53:14.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:14 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:53:14.855 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 47 sec latency 162752.330 ms 2024-03-28T17:53:15.855 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 48 sec latency 163752.435 ms 2024-03-28T17:53:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:16 smithi071 ceph-mon[36597]: pgmap v141: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:53:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:16 smithi151 ceph-mon[36481]: pgmap v141: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:53:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:16 smithi045 ceph-mon[31773]: pgmap v141: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:53:16.856 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 49 sec latency 164752.576 ms 2024-03-28T17:53:17.856 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 50 sec latency 165752.740 ms 2024-03-28T17:53:18.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:18 smithi071 ceph-mon[36597]: pgmap v142: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 26 op/s 2024-03-28T17:53:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:18 smithi151 ceph-mon[36481]: pgmap v142: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 26 op/s 2024-03-28T17:53:18.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:18 smithi045 ceph-mon[31773]: pgmap v142: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 26 op/s 2024-03-28T17:53:18.856 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 51 sec latency 166752.909 ms 2024-03-28T17:53:19.857 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 52 sec latency 167753.077 ms 2024-03-28T17:53:20.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:20 smithi071 ceph-mon[36597]: pgmap v143: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T17:53:20.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:20 smithi151 ceph-mon[36481]: pgmap v143: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T17:53:20.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:20 smithi045 ceph-mon[31773]: pgmap v143: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T17:53:20.856 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 53 sec latency 168753.210 ms 2024-03-28T17:53:21.856 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 54 sec latency 169753.368 ms 2024-03-28T17:53:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:22 smithi071 ceph-mon[36597]: pgmap v144: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-28T17:53:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:22 smithi151 ceph-mon[36481]: pgmap v144: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-28T17:53:22.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:22 smithi045 ceph-mon[31773]: pgmap v144: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-28T17:53:22.856 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 55 sec latency 170753.512 ms 2024-03-28T17:53:23.324 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:22 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 1. 2024-03-28T17:53:23.325 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:22 smithi045 systemd[1]: Stopped Ceph mds.d for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:53:23.325 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:22 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Consumed 1.833s CPU time. 2024-03-28T17:53:23.325 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:22 smithi045 systemd[1]: Starting Ceph mds.d for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:53:23.666 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:23 smithi045 podman[90230]: 2024-03-28 17:53:23.228436419 +0000 UTC m=+0.038709280 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:53:23.666 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:23 smithi045 podman[90230]: 2024-03-28 17:53:23.375107393 +0000 UTC m=+0.185380289 container create e76f1cd28e0602de7b8a0cc06d9b4b203839a15ab018b549acaa8c9eda344bf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS) 2024-03-28T17:53:23.857 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 56 sec latency 171753.669 ms 2024-03-28T17:53:24.166 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:23 smithi045 podman[90230]: 2024-03-28 17:53:23.670833579 +0000 UTC m=+0.481106522 container init e76f1cd28e0602de7b8a0cc06d9b4b203839a15ab018b549acaa8c9eda344bf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-28T17:53:24.166 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:23 smithi045 podman[90230]: 2024-03-28 17:53:23.681521461 +0000 UTC m=+0.491794335 container start e76f1cd28e0602de7b8a0cc06d9b4b203839a15ab018b549acaa8c9eda344bf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-28T17:53:24.166 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:23 smithi045 ceph-mds[90249]: -- 172.21.15.45:0/3656250102 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557fa562da40 con 0x557fa5601000 2024-03-28T17:53:24.166 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: starting mds.d at 2024-03-28T17:53:24.167 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:23 smithi045 bash[90230]: e76f1cd28e0602de7b8a0cc06d9b4b203839a15ab018b549acaa8c9eda344bf5 2024-03-28T17:53:24.167 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:23 smithi045 systemd[1]: Started Ceph mds.d for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:53:24.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:24 smithi071 ceph-mon[36597]: pgmap v145: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T17:53:24.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:24 smithi071 ceph-mon[36597]: daemon mds.d restarted 2024-03-28T17:53:24.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:24 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:53:24.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:24 smithi071 ceph-mon[36597]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T17:53:24.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:24 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/3798594912,v1:172.21.15.45:6837/3798594912] up:boot 2024-03-28T17:53:24.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:24 smithi071 ceph-mon[36597]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-03-28T17:53:24.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:24 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-03-28T17:53:24.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:24 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T17:53:24.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:24 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:replay} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:24.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:24 smithi151 ceph-mon[36481]: pgmap v145: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T17:53:24.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:24 smithi151 ceph-mon[36481]: daemon mds.d restarted 2024-03-28T17:53:24.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:24 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:53:24.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:24 smithi151 ceph-mon[36481]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T17:53:24.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:24 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/3798594912,v1:172.21.15.45:6837/3798594912] up:boot 2024-03-28T17:53:24.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:24 smithi151 ceph-mon[36481]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-03-28T17:53:24.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:24 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-03-28T17:53:24.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:24 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T17:53:24.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:24 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:replay} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:24.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:24 smithi045 ceph-mon[31773]: pgmap v145: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T17:53:24.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:24 smithi045 ceph-mon[31773]: daemon mds.d restarted 2024-03-28T17:53:24.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:24 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:53:24.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:24 smithi045 ceph-mon[31773]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T17:53:24.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:24 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/3798594912,v1:172.21.15.45:6837/3798594912] up:boot 2024-03-28T17:53:24.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:24 smithi045 ceph-mon[31773]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-03-28T17:53:24.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:24 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-03-28T17:53:24.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:24 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T17:53:24.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:24 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:replay} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:24.857 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 57 sec latency 172753.797 ms 2024-03-28T17:53:25.857 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 58 sec latency 173753.935 ms 2024-03-28T17:53:25.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:25 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/121098913,v1:172.21.15.151:6835/121098913] up:resolve 2024-03-28T17:53:25.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:25 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:resolve} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:26.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:25 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/121098913,v1:172.21.15.151:6835/121098913] up:resolve 2024-03-28T17:53:26.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:25 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:resolve} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:26.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:25 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/121098913,v1:172.21.15.151:6835/121098913] up:resolve 2024-03-28T17:53:26.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:25 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:resolve} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:26.857 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 59 sec latency 174754.065 ms 2024-03-28T17:53:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:26 smithi071 ceph-mon[36597]: pgmap v147: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:53:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:26 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299996 2024-03-28T17:53:27.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:26 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/121098913,v1:172.21.15.151:6835/121098913] up:reconnect 2024-03-28T17:53:27.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:26 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:reconnect} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:27.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:26 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:53:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:26 smithi151 ceph-mon[36481]: pgmap v147: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:53:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:26 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299996 2024-03-28T17:53:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:26 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/121098913,v1:172.21.15.151:6835/121098913] up:reconnect 2024-03-28T17:53:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:26 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:reconnect} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:26 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:53:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:26 smithi045 ceph-mon[31773]: pgmap v147: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:53:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:26 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299996 2024-03-28T17:53:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:26 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/121098913,v1:172.21.15.151:6835/121098913] up:reconnect 2024-03-28T17:53:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:26 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:reconnect} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:26 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:53:27.857 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 60 sec latency 175754.158 ms 2024-03-28T17:53:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:27 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/121098913,v1:172.21.15.151:6835/121098913] up:rejoin 2024-03-28T17:53:28.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:27 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:rejoin} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:28.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:27 smithi071 ceph-mon[36597]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-28T17:53:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/121098913,v1:172.21.15.151:6835/121098913] up:rejoin 2024-03-28T17:53:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:rejoin} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-mon[36481]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-28T17:53:28.073 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f35197da700 time 2024-03-28T17:53:27.794972+0000 2024-03-28T17:53:28.074 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:53:28.074 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:28.074 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3523e044a1] 2024-03-28T17:53:28.074 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3523e04667] 2024-03-28T17:53:28.074 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558877d94223] 2024-03-28T17:53:28.074 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558877c0e5d6] 2024-03-28T17:53:28.074 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558877c0eda1] 2024-03-28T17:53:28.074 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558877c112d5] 2024-03-28T17:53:28.074 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558877c61233] 2024-03-28T17:53:28.074 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558877c63197] 2024-03-28T17:53:28.075 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558877b6211b] 2024-03-28T17:53:28.075 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558877b626bc] 2024-03-28T17:53:28.075 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558877b4c1ff] 2024-03-28T17:53:28.075 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3524096488] 2024-03-28T17:53:28.075 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 13: (DispatchQueue::entry()+0x51f) [0x7f352409360f] 2024-03-28T17:53:28.075 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3524166e81] 2024-03-28T17:53:28.075 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3522b811ca] 2024-03-28T17:53:28.075 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 16: clone() 2024-03-28T17:53:28.076 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 2024-03-28T17:53:27.796+0000 7f35197da700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f35197da700 time 2024-03-28T17:53:27.794972+0000 2024-03-28T17:53:28.076 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:53:28.076 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 2024-03-28T17:53:28.076 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:28.076 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3523e044a1] 2024-03-28T17:53:28.076 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3523e04667] 2024-03-28T17:53:28.076 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558877d94223] 2024-03-28T17:53:28.076 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558877c0e5d6] 2024-03-28T17:53:28.077 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558877c0eda1] 2024-03-28T17:53:28.077 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558877c112d5] 2024-03-28T17:53:28.077 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558877c61233] 2024-03-28T17:53:28.077 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558877c63197] 2024-03-28T17:53:28.077 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558877b6211b] 2024-03-28T17:53:28.077 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558877b626bc] 2024-03-28T17:53:28.077 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558877b4c1ff] 2024-03-28T17:53:28.077 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3524096488] 2024-03-28T17:53:28.077 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 13: (DispatchQueue::entry()+0x51f) [0x7f352409360f] 2024-03-28T17:53:28.077 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3524166e81] 2024-03-28T17:53:28.077 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3522b811ca] 2024-03-28T17:53:28.078 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 16: clone() 2024-03-28T17:53:28.078 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 2024-03-28T17:53:28.078 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: *** Caught signal (Aborted) ** 2024-03-28T17:53:28.078 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: in thread 7f35197da700 thread_name:ms_dispatch 2024-03-28T17:53:28.078 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:28.078 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3522b8bd20] 2024-03-28T17:53:28.078 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 2: gsignal() 2024-03-28T17:53:28.078 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 3: abort() 2024-03-28T17:53:28.078 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3523e044fb] 2024-03-28T17:53:28.078 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3523e04667] 2024-03-28T17:53:28.078 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558877d94223] 2024-03-28T17:53:28.078 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558877c0e5d6] 2024-03-28T17:53:28.078 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558877c0eda1] 2024-03-28T17:53:28.078 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558877c112d5] 2024-03-28T17:53:28.079 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558877c61233] 2024-03-28T17:53:28.079 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558877c63197] 2024-03-28T17:53:28.079 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558877b6211b] 2024-03-28T17:53:28.079 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558877b626bc] 2024-03-28T17:53:28.079 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558877b4c1ff] 2024-03-28T17:53:28.079 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3524096488] 2024-03-28T17:53:28.079 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 16: (DispatchQueue::entry()+0x51f) [0x7f352409360f] 2024-03-28T17:53:28.079 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3524166e81] 2024-03-28T17:53:28.079 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3522b811ca] 2024-03-28T17:53:28.079 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 19: clone() 2024-03-28T17:53:28.079 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 2024-03-28T17:53:27.798+0000 7f35197da700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:53:28.079 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: in thread 7f35197da700 thread_name:ms_dispatch 2024-03-28T17:53:28.079 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 2024-03-28T17:53:28.080 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:28.080 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3522b8bd20] 2024-03-28T17:53:28.080 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 2: gsignal() 2024-03-28T17:53:28.080 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 3: abort() 2024-03-28T17:53:28.080 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3523e044fb] 2024-03-28T17:53:28.080 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3523e04667] 2024-03-28T17:53:28.080 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558877d94223] 2024-03-28T17:53:28.080 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558877c0e5d6] 2024-03-28T17:53:28.080 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558877c0eda1] 2024-03-28T17:53:28.080 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558877c112d5] 2024-03-28T17:53:28.081 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558877c61233] 2024-03-28T17:53:28.081 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558877c63197] 2024-03-28T17:53:28.081 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558877b6211b] 2024-03-28T17:53:28.081 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558877b626bc] 2024-03-28T17:53:28.081 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558877b4c1ff] 2024-03-28T17:53:28.081 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3524096488] 2024-03-28T17:53:28.081 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 16: (DispatchQueue::entry()+0x51f) [0x7f352409360f] 2024-03-28T17:53:28.081 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3524166e81] 2024-03-28T17:53:28.081 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3522b811ca] 2024-03-28T17:53:28.081 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 19: clone() 2024-03-28T17:53:28.081 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:53:28.082 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 2024-03-28T17:53:28.082 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: -1> 2024-03-28T17:53:27.796+0000 7f35197da700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f35197da700 time 2024-03-28T17:53:27.794972+0000 2024-03-28T17:53:28.082 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:53:28.082 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 2024-03-28T17:53:28.082 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:28.082 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3523e044a1] 2024-03-28T17:53:28.082 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3523e04667] 2024-03-28T17:53:28.082 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558877d94223] 2024-03-28T17:53:28.082 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558877c0e5d6] 2024-03-28T17:53:28.082 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558877c0eda1] 2024-03-28T17:53:28.082 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558877c112d5] 2024-03-28T17:53:28.082 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558877c61233] 2024-03-28T17:53:28.082 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558877c63197] 2024-03-28T17:53:28.082 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558877b6211b] 2024-03-28T17:53:28.082 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558877b626bc] 2024-03-28T17:53:28.083 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558877b4c1ff] 2024-03-28T17:53:28.083 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3524096488] 2024-03-28T17:53:28.083 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 13: (DispatchQueue::entry()+0x51f) [0x7f352409360f] 2024-03-28T17:53:28.083 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3524166e81] 2024-03-28T17:53:28.083 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3522b811ca] 2024-03-28T17:53:28.083 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 16: clone() 2024-03-28T17:53:28.083 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 2024-03-28T17:53:28.083 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 0> 2024-03-28T17:53:27.798+0000 7f35197da700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:53:28.083 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: in thread 7f35197da700 thread_name:ms_dispatch 2024-03-28T17:53:28.083 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 2024-03-28T17:53:28.083 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:28.083 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3522b8bd20] 2024-03-28T17:53:28.083 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 2: gsignal() 2024-03-28T17:53:28.083 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 3: abort() 2024-03-28T17:53:28.084 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3523e044fb] 2024-03-28T17:53:28.084 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3523e04667] 2024-03-28T17:53:28.084 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558877d94223] 2024-03-28T17:53:28.084 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558877c0e5d6] 2024-03-28T17:53:28.084 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558877c0eda1] 2024-03-28T17:53:28.084 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558877c112d5] 2024-03-28T17:53:28.085 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558877c61233] 2024-03-28T17:53:28.085 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558877c63197] 2024-03-28T17:53:28.085 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558877b6211b] 2024-03-28T17:53:28.085 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558877b626bc] 2024-03-28T17:53:28.085 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558877b4c1ff] 2024-03-28T17:53:28.085 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3524096488] 2024-03-28T17:53:28.085 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 16: (DispatchQueue::entry()+0x51f) [0x7f352409360f] 2024-03-28T17:53:28.085 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3524166e81] 2024-03-28T17:53:28.085 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3522b811ca] 2024-03-28T17:53:28.085 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 19: clone() 2024-03-28T17:53:28.085 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:53:28.085 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 2024-03-28T17:53:28.085 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: -1> 2024-03-28T17:53:27.796+0000 7f35197da700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f35197da700 time 2024-03-28T17:53:27.794972+0000 2024-03-28T17:53:28.085 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:53:28.085 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 2024-03-28T17:53:28.085 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:28.086 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3523e044a1] 2024-03-28T17:53:28.086 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3523e04667] 2024-03-28T17:53:28.086 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558877d94223] 2024-03-28T17:53:28.086 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558877c0e5d6] 2024-03-28T17:53:28.086 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558877c0eda1] 2024-03-28T17:53:28.086 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558877c112d5] 2024-03-28T17:53:28.086 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558877c61233] 2024-03-28T17:53:28.086 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558877c63197] 2024-03-28T17:53:28.086 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558877b6211b] 2024-03-28T17:53:28.086 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558877b626bc] 2024-03-28T17:53:28.086 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558877b4c1ff] 2024-03-28T17:53:28.086 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3524096488] 2024-03-28T17:53:28.087 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 13: (DispatchQueue::entry()+0x51f) [0x7f352409360f] 2024-03-28T17:53:28.087 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3524166e81] 2024-03-28T17:53:28.087 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3522b811ca] 2024-03-28T17:53:28.087 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 16: clone() 2024-03-28T17:53:28.087 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 2024-03-28T17:53:28.087 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 0> 2024-03-28T17:53:27.798+0000 7f35197da700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:53:28.087 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: in thread 7f35197da700 thread_name:ms_dispatch 2024-03-28T17:53:28.087 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 2024-03-28T17:53:28.087 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:28.087 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3522b8bd20] 2024-03-28T17:53:28.087 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 2: gsignal() 2024-03-28T17:53:28.087 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 3: abort() 2024-03-28T17:53:28.087 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3523e044fb] 2024-03-28T17:53:28.087 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3523e04667] 2024-03-28T17:53:28.087 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558877d94223] 2024-03-28T17:53:28.087 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558877c0e5d6] 2024-03-28T17:53:28.088 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558877c0eda1] 2024-03-28T17:53:28.088 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558877c112d5] 2024-03-28T17:53:28.088 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558877c61233] 2024-03-28T17:53:28.088 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558877c63197] 2024-03-28T17:53:28.088 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558877b6211b] 2024-03-28T17:53:28.088 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558877b626bc] 2024-03-28T17:53:28.088 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558877b4c1ff] 2024-03-28T17:53:28.088 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3524096488] 2024-03-28T17:53:28.088 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 16: (DispatchQueue::entry()+0x51f) [0x7f352409360f] 2024-03-28T17:53:28.088 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3524166e81] 2024-03-28T17:53:28.088 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3522b811ca] 2024-03-28T17:53:28.089 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 19: clone() 2024-03-28T17:53:28.089 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:53:28.089 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f[72469]: 2024-03-28T17:53:28.089 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:27 smithi151 podman[72639]: 2024-03-28 17:53:27.853446923 +0000 UTC m=+0.021289695 container died 5ae7b83083827dffe6bff11421e0e55fab896a968d1b5ce297bc85d0ac18f908 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-28T17:53:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:27 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/121098913,v1:172.21.15.151:6835/121098913] up:rejoin 2024-03-28T17:53:28.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:27 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:rejoin} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:28.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:27 smithi045 ceph-mon[31773]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-28T17:53:28.792 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:28 smithi151 podman[72639]: 2024-03-28 17:53:28.482460964 +0000 UTC m=+0.650303718 container remove 5ae7b83083827dffe6bff11421e0e55fab896a968d1b5ce297bc85d0ac18f908 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-f, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-03-28T17:53:28.792 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:28 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:53:28.792 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:28 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-28T17:53:28.857 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 61 sec latency 176754.311 ms 2024-03-28T17:53:29.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:28 smithi071 ceph-mon[36597]: pgmap v148: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:53:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:28 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:53:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:28 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:53:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:28 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6834/121098913,v1:172.21.15.151:6835/121098913] up:active 2024-03-28T17:53:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:28 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:28 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 3 up:standby-replay 5 up:standby 2024-03-28T17:53:29.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:28 smithi151 ceph-mon[36481]: pgmap v148: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:53:29.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:28 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:53:29.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:28 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:53:29.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:28 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6834/121098913,v1:172.21.15.151:6835/121098913] up:active 2024-03-28T17:53:29.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:28 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:29.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:28 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 3 up:standby-replay 5 up:standby 2024-03-28T17:53:29.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:28 smithi045 ceph-mon[31773]: pgmap v148: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:53:29.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:28 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:53:29.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:28 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:53:29.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:28 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6834/121098913,v1:172.21.15.151:6835/121098913] up:active 2024-03-28T17:53:29.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:28 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 2 up:standby-replay 6 up:standby 2024-03-28T17:53:29.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:28 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=f=up:active} 3 up:standby-replay 5 up:standby 2024-03-28T17:53:29.857 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 62 sec latency 177754.478 ms 2024-03-28T17:53:30.858 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 63 sec latency 178754.606 ms 2024-03-28T17:53:31.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:30 smithi071 ceph-mon[36597]: pgmap v149: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:53:31.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:30 smithi045 ceph-mon[31773]: pgmap v149: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:53:31.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:30 smithi151 ceph-mon[36481]: pgmap v149: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:53:31.858 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 64 sec latency 179754.700 ms 2024-03-28T17:53:32.858 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 65 sec latency 180754.882 ms 2024-03-28T17:53:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:32 smithi071 ceph-mon[36597]: pgmap v150: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:53:33.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:32 smithi151 ceph-mon[36481]: pgmap v150: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:53:33.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:32 smithi045 ceph-mon[31773]: pgmap v150: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:53:33.299 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:53:33.299 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:53:33.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.544+0000 7f60b9800640 1 -- 172.21.15.45:0/1827903263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60b4103720 msgr2=0x7f60b4103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:53:33.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.544+0000 7f60b9800640 1 --2- 172.21.15.45:0/1827903263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60b4103720 0x7f60b4103ba0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f609c009900 tx=0x7f609c02bc40 comp rx=0 tx=0).stop 2024-03-28T17:53:33.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.544+0000 7f60b9800640 1 -- 172.21.15.45:0/1827903263 shutdown_connections 2024-03-28T17:53:33.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.544+0000 7f60b9800640 1 --2- 172.21.15.45:0/1827903263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60b41040e0 0x7f60b41105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.544+0000 7f60b9800640 1 --2- 172.21.15.45:0/1827903263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60b4103720 0x7f60b4103ba0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.544+0000 7f60b9800640 1 --2- 172.21.15.45:0/1827903263 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f60b4109730 0x7f60b4109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.544+0000 7f60b9800640 1 -- 172.21.15.45:0/1827903263 >> 172.21.15.45:0/1827903263 conn(0x7f60b40ff1d0 msgr2=0x7f60b41015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:53:33.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.544+0000 7f60b9800640 1 -- 172.21.15.45:0/1827903263 shutdown_connections 2024-03-28T17:53:33.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.544+0000 7f60b9800640 1 -- 172.21.15.45:0/1827903263 wait complete. 2024-03-28T17:53:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.545+0000 7f60b9800640 1 Processor -- start 2024-03-28T17:53:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.545+0000 7f60b9800640 1 -- start start 2024-03-28T17:53:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.545+0000 7f60b9800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f60b4103720 0x7f60b419a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.545+0000 7f60b9800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60b41040e0 0x7f60b419a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.545+0000 7f60b9800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60b4109730 0x7f60b419e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.545+0000 7f60b9800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b4113b90 con 0x7f60b4109730 2024-03-28T17:53:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.545+0000 7f60b9800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b4113a10 con 0x7f60b41040e0 2024-03-28T17:53:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.545+0000 7f60b9800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b4113d10 con 0x7f60b4103720 2024-03-28T17:53:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.546+0000 7f60ab400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60b41040e0 0x7f60b419a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.546+0000 7f60ab400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60b41040e0 0x7f60b419a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:60372/0 (socket says 172.21.15.45:60372) 2024-03-28T17:53:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.546+0000 7f60b8e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60b4109730 0x7f60b419e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.546+0000 7f60ab400640 1 -- 172.21.15.45:0/2582222287 learned_addr learned my addr 172.21.15.45:0/2582222287 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:53:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.546+0000 7f60b3e00640 1 --2- 172.21.15.45:0/2582222287 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f60b4103720 0x7f60b419a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.546+0000 7f60ab400640 1 -- 172.21.15.45:0/2582222287 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f60b4103720 msgr2=0x7f60b419a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:53:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.546+0000 7f60ab400640 1 --2- 172.21.15.45:0/2582222287 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f60b4103720 0x7f60b419a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.546+0000 7f60ab400640 1 -- 172.21.15.45:0/2582222287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60b4109730 msgr2=0x7f60b419e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:53:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.546+0000 7f60ab400640 1 --2- 172.21.15.45:0/2582222287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60b4109730 0x7f60b419e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.546+0000 7f60ab400640 1 -- 172.21.15.45:0/2582222287 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b419f090 con 0x7f60b41040e0 2024-03-28T17:53:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.546+0000 7f60ab400640 1 --2- 172.21.15.45:0/2582222287 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60b41040e0 0x7f60b419a630 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f609c005ab0 tx=0x7f609c0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:53:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.547+0000 7f60b1600640 1 -- 172.21.15.45:0/2582222287 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c041070 con 0x7f60b41040e0 2024-03-28T17:53:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.547+0000 7f60b9800640 1 -- 172.21.15.45:0/2582222287 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b419f320 con 0x7f60b41040e0 2024-03-28T17:53:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.547+0000 7f60b1600640 1 -- 172.21.15.45:0/2582222287 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f609c02ccd0 con 0x7f60b41040e0 2024-03-28T17:53:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.547+0000 7f60b1600640 1 -- 172.21.15.45:0/2582222287 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c003d10 con 0x7f60b41040e0 2024-03-28T17:53:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.547+0000 7f60b9800640 1 -- 172.21.15.45:0/2582222287 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b41a6c10 con 0x7f60b41040e0 2024-03-28T17:53:33.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.550+0000 7f60b1600640 1 -- 172.21.15.45:0/2582222287 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f609c044070 con 0x7f60b41040e0 2024-03-28T17:53:33.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.550+0000 7f60bdb94740 1 Processor -- start 2024-03-28T17:53:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.550+0000 7f60bdb94740 1 -- start start 2024-03-28T17:53:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.551+0000 7f60bdb94740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555a6fcdcdd0 0x555a6fcdbb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.551+0000 7f60bdb94740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fcdc0f0 0x555a6fcdc570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.551+0000 7f60bdb94740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fce5460 0x555a6fce7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.551+0000 7f60bdb94740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a6fc40250 con 0x555a6fcdcdd0 2024-03-28T17:53:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.551+0000 7f60bdb94740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a6fbe8a30 con 0x555a6fce5460 2024-03-28T17:53:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.551+0000 7f60bdb94740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a6fc3c2f0 con 0x555a6fcdc0f0 2024-03-28T17:53:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.551+0000 7f60b8e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555a6fcdcdd0 0x555a6fcdbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.551+0000 7f60ab400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fce5460 0x555a6fce7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.551+0000 7f60b3e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fcdc0f0 0x555a6fcdc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.551+0000 7f60b3e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fcdc0f0 0x555a6fcdc570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:60014/0 (socket says 172.21.15.45:60014) 2024-03-28T17:53:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.551+0000 7f60b3e00640 1 -- 172.21.15.45:0/4260373590 learned_addr learned my addr 172.21.15.45:0/4260373590 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:53:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.551+0000 7f60b8e00640 1 --2- 172.21.15.45:0/4260373590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555a6fcdcdd0 0x555a6fcdbb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36292/0 (socket says 172.21.15.45:36292) 2024-03-28T17:53:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.551+0000 7f60b3e00640 1 -- 172.21.15.45:0/4260373590 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fce5460 msgr2=0x555a6fce7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:53:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.551+0000 7f60b3e00640 1 --2- 172.21.15.45:0/4260373590 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fce5460 0x555a6fce7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.551+0000 7f60b3e00640 1 -- 172.21.15.45:0/4260373590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555a6fcdcdd0 msgr2=0x555a6fcdbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:53:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.551+0000 7f60b3e00640 1 --2- 172.21.15.45:0/4260373590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555a6fcdcdd0 0x555a6fcdbb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.551+0000 7f60b3e00640 1 -- 172.21.15.45:0/4260373590 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a6fbd3690 con 0x555a6fcdc0f0 2024-03-28T17:53:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.551+0000 7f60ab400640 1 --2- 172.21.15.45:0/4260373590 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fce5460 0x555a6fce7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:53:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.552+0000 7f60b1600640 1 --2- 172.21.15.45:0/2582222287 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f60980a2970 0x7f60980a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.552+0000 7f60b3e00640 1 --2- 172.21.15.45:0/4260373590 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fcdc0f0 0x555a6fcdc570 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f60a400b710 tx=0x7f60a4004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ed54f2c912e51e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:53:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.552+0000 7f60b1600640 1 -- 172.21.15.45:0/2582222287 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9458+0+0 (secure 0 0 0) 0x7f609c0f0b90 con 0x7f60b41040e0 2024-03-28T17:53:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.552+0000 7f60b8e00640 1 --2- 172.21.15.45:0/2582222287 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f60980a2970 0x7f60980a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.552+0000 7f60a9600640 1 -- 172.21.15.45:0/4260373590 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4003bd0 con 0x555a6fcdc0f0 2024-03-28T17:53:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.552+0000 7f60a9600640 1 -- 172.21.15.45:0/4260373590 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60a4015070 con 0x555a6fcdc0f0 2024-03-28T17:53:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.552+0000 7f60a9600640 1 -- 172.21.15.45:0/4260373590 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4004e40 con 0x555a6fcdc0f0 2024-03-28T17:53:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.553+0000 7f60bdb94740 1 -- 172.21.15.45:0/4260373590 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fcdc0f0 msgr2=0x555a6fcdc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:53:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.553+0000 7f60b8e00640 1 --2- 172.21.15.45:0/2582222287 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f60980a2970 0x7f60980a4e30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f60ac005fd0 tx=0x7f60ac009500 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:53:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.553+0000 7f60bdb94740 1 --2- 172.21.15.45:0/4260373590 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fcdc0f0 0x555a6fcdc570 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f60a400b710 tx=0x7f60a4004610 comp rx=0 tx=0).stop 2024-03-28T17:53:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.553+0000 7f60bdb94740 1 -- 172.21.15.45:0/4260373590 shutdown_connections 2024-03-28T17:53:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.553+0000 7f60bdb94740 1 --2- 172.21.15.45:0/4260373590 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fce5460 0x555a6fce7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.553+0000 7f60bdb94740 1 --2- 172.21.15.45:0/4260373590 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fcdc0f0 0x555a6fcdc570 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.553+0000 7f60bdb94740 1 --2- 172.21.15.45:0/4260373590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555a6fcdcdd0 0x555a6fcdbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.553+0000 7f60bdb94740 1 -- 172.21.15.45:0/4260373590 >> 172.21.15.45:0/4260373590 conn(0x555a6fca3200 msgr2=0x555a6fcd6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:53:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.553+0000 7f60bdb94740 1 -- 172.21.15.45:0/4260373590 shutdown_connections 2024-03-28T17:53:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.553+0000 7f60bdb94740 1 -- 172.21.15.45:0/4260373590 wait complete. 2024-03-28T17:53:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.554+0000 7f60bdb94740 1 -- start start 2024-03-28T17:53:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.554+0000 7f60bdb94740 1 Processor -- start 2024-03-28T17:53:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.554+0000 7f60bdb94740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fcdc0f0 0x555a6fce4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.554+0000 7f60bdb94740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555a6fcdcdd0 0x555a6fce1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.554+0000 7f60bdb94740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fce5460 0x555a6fce2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.554+0000 7f60bdb94740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a6fce82d0 con 0x555a6fcdcdd0 2024-03-28T17:53:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.554+0000 7f60bdb94740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a6fbfbe80 con 0x555a6fce5460 2024-03-28T17:53:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.554+0000 7f60bdb94740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a6fce8450 con 0x555a6fcdc0f0 2024-03-28T17:53:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.555+0000 7f60bdb94740 10 client.0 _ll_register_callbacks cb 0x555a6fc0e130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:53:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.555+0000 7f60b8e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fce5460 0x555a6fce2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.555+0000 7f60b3e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fcdc0f0 0x555a6fce4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.555+0000 7f60b8e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fce5460 0x555a6fce2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:60376/0 (socket says 172.21.15.45:60376) 2024-03-28T17:53:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.555+0000 7f60b3e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fcdc0f0 0x555a6fce4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:60018/0 (socket says 172.21.15.45:60018) 2024-03-28T17:53:33.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.555+0000 7f60b8e00640 1 -- 172.21.15.45:0/3884970925 learned_addr learned my addr 172.21.15.45:0/3884970925 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:53:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.555+0000 7f60ab400640 1 --2- 172.21.15.45:0/3884970925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555a6fcdcdd0 0x555a6fce1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.555+0000 7f60b8e00640 1 -- 172.21.15.45:0/3884970925 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fcdc0f0 msgr2=0x555a6fce4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:53:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.555+0000 7f60b8e00640 1 --2- 172.21.15.45:0/3884970925 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fcdc0f0 0x555a6fce4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.555+0000 7f60b8e00640 1 -- 172.21.15.45:0/3884970925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555a6fcdcdd0 msgr2=0x555a6fce1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:53:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.555+0000 7f60b8e00640 1 --2- 172.21.15.45:0/3884970925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555a6fcdcdd0 0x555a6fce1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.555+0000 7f60b8e00640 1 -- 172.21.15.45:0/3884970925 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a6fa1e7d0 con 0x555a6fce5460 2024-03-28T17:53:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.556+0000 7f60b8e00640 1 --2- 172.21.15.45:0/3884970925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fce5460 0x555a6fce2480 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f60ac003fd0 tx=0x7f60ac004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f068cc9c1f02c1d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:53:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.556+0000 7f60a2000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:53:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.556+0000 7f60a2000640 1 -- 172.21.15.45:0/3884970925 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac00f7b0 con 0x555a6fce5460 2024-03-28T17:53:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.556+0000 7f60bdb94740 1 -- 172.21.15.45:0/3884970925 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f60a4015070 con 0x555a6fce5460 2024-03-28T17:53:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.556+0000 7f60a2000640 1 -- 172.21.15.45:0/3884970925 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9458+0+0 (secure 0 0 0) 0x7f60ac00f950 con 0x555a6fce5460 2024-03-28T17:53:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.556+0000 7f60a2000640 1 -- 172.21.15.45:0/3884970925 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60ac00fb50 con 0x555a6fce5460 2024-03-28T17:53:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.556+0000 7f60bdb94740 10 client.26063 fetch_fsmap learned FSMap version 96 2024-03-28T17:53:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.556+0000 7f60bdb94740 1 -- 172.21.15.45:0/3884970925 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) v3 -- 0x555a6f9543f0 con 0x555a6fce5460 2024-03-28T17:53:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.556+0000 7f60a2000640 1 -- 172.21.15.45:0/3884970925 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60a4015070 con 0x555a6fce5460 2024-03-28T17:53:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.557+0000 7f60a2000640 1 -- 172.21.15.45:0/3884970925 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 96) v1 ==== 7343+0+0 (secure 0 0 0) 0x7f60ac026070 con 0x555a6fce5460 2024-03-28T17:53:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.557+0000 7f60bdb94740 10 client.26063 fetch_fsmap finished waiting for FSMap version 96 2024-03-28T17:53:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.557+0000 7f60bdb94740 10 client.26063 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:53:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.557+0000 7f60bdb94740 20 client.26063 populate_metadata read hostname 'smithi045' 2024-03-28T17:53:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.557+0000 7f60bdb94740 1 --2- 172.21.15.45:0/3884970925 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555a6fcf55b0 0x555a6fcf79f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.557+0000 7f60bdb94740 4 client.26063 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:53:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.557+0000 7f60bdb94740 1 -- 172.21.15.45:0/3884970925 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555a6fc4d790 con 0x555a6fcf55b0 2024-03-28T17:53:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.557+0000 7f60b3e00640 1 --2- 172.21.15.45:0/3884970925 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555a6fcf55b0 0x555a6fcf79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.557+0000 7f60b3e00640 1 --2- 172.21.15.45:0/3884970925 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555a6fcf55b0 0x555a6fcf79f0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36d939db8366c782 server_cookie=b9f3b339f52b48b2 in_seq=0 out_seq=0 2024-03-28T17:53:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.557+0000 7f60a2000640 10 client.26063 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:53:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.558+0000 7f60a2000640 1 -- 172.21.15.45:0/3884970925 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555a6fc4d790 con 0x555a6fcf55b0 2024-03-28T17:53:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.558+0000 7f60a2000640 10 client.26063 handle_command_reply: tid=0 2024-03-28T17:53:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.558+0000 7f60bdb94740 1 client.26063 shutdown 2024-03-28T17:53:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.558+0000 7f60bdb94740 1 -- 172.21.15.45:0/3884970925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fce5460 msgr2=0x555a6fce2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:53:33.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.558+0000 7f60bdb94740 1 --2- 172.21.15.45:0/3884970925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fce5460 0x555a6fce2480 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f60ac003fd0 tx=0x7f60ac004ab0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.558+0000 7f60bdb94740 1 -- 172.21.15.45:0/3884970925 shutdown_connections 2024-03-28T17:53:33.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.558+0000 7f60bdb94740 1 --2- 172.21.15.45:0/3884970925 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555a6fcf55b0 0x555a6fcf79f0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.558+0000 7f60bdb94740 1 --2- 172.21.15.45:0/3884970925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fce5460 0x555a6fce2480 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.558+0000 7f60a2000640 0 client.26063 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:53:33.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.558+0000 7f60bdb94740 1 --2- 172.21.15.45:0/3884970925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555a6fcdcdd0 0x555a6fce1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.558+0000 7f60bdb94740 1 --2- 172.21.15.45:0/3884970925 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fcdc0f0 0x555a6fce4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.558+0000 7f60bdb94740 1 -- 172.21.15.45:0/3884970925 >> 172.21.15.45:0/3884970925 conn(0x555a6fca3200 msgr2=0x555a6fcd8890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:53:33.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.559+0000 7f60bdb94740 1 -- 172.21.15.45:0/3884970925 shutdown_connections 2024-03-28T17:53:33.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.559+0000 7f60bdb94740 1 -- 172.21.15.45:0/3884970925 wait complete. 2024-03-28T17:53:33.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.559+0000 7f60bdb94740 20 client.26063 trim_cache size 0 max 16384 2024-03-28T17:53:33.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.576+0000 7f60bdb94740 1 Processor -- start 2024-03-28T17:53:33.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.576+0000 7f60bdb94740 1 -- start start 2024-03-28T17:53:33.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.577+0000 7f60bdb94740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fcdc0f0 0x555a6fce4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:33.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.577+0000 7f60bdb94740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fcdcdd0 0x555a6fce4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:33.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.577+0000 7f60bdb94740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555a6fce5460 0x555a6fce6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:33.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.577+0000 7f60bdb94740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b41b12c0 con 0x555a6fce5460 2024-03-28T17:53:33.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.577+0000 7f60bdb94740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b41b1920 con 0x555a6fcdc0f0 2024-03-28T17:53:33.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.577+0000 7f60bdb94740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b41af460 con 0x555a6fcdcdd0 2024-03-28T17:53:33.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.577+0000 7f60b3e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fcdc0f0 0x555a6fce4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:33.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.577+0000 7f60b8e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555a6fce5460 0x555a6fce6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:33.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.577+0000 7f60b3e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fcdc0f0 0x555a6fce4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:60380/0 (socket says 172.21.15.45:60380) 2024-03-28T17:53:33.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.577+0000 7f60b3e00640 1 -- 172.21.15.45:0/935653049 learned_addr learned my addr 172.21.15.45:0/935653049 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:53:33.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.577+0000 7f60ab400640 1 --2- 172.21.15.45:0/935653049 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fcdcdd0 0x555a6fce4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:33.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.578+0000 7f60b3e00640 1 -- 172.21.15.45:0/935653049 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fcdcdd0 msgr2=0x555a6fce4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:53:33.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.578+0000 7f60b3e00640 1 --2- 172.21.15.45:0/935653049 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fcdcdd0 0x555a6fce4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.578+0000 7f60b3e00640 1 -- 172.21.15.45:0/935653049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555a6fce5460 msgr2=0x555a6fce6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:53:33.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.578+0000 7f60b3e00640 1 --2- 172.21.15.45:0/935653049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555a6fce5460 0x555a6fce6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.578+0000 7f60b3e00640 1 -- 172.21.15.45:0/935653049 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a6f9140f0 con 0x555a6fcdc0f0 2024-03-28T17:53:33.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.578+0000 7f60ab400640 1 --2- 172.21.15.45:0/935653049 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fcdcdd0 0x555a6fce4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:53:33.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.578+0000 7f60b3e00640 1 --2- 172.21.15.45:0/935653049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fcdc0f0 0x555a6fce4670 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f60a400b710 tx=0x7f60a4004810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a286dd5bfe6ab88 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:53:33.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.578+0000 7f60a8c00640 1 -- 172.21.15.45:0/935653049 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4003bd0 con 0x555a6fcdc0f0 2024-03-28T17:53:33.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.578+0000 7f60a8c00640 1 -- 172.21.15.45:0/935653049 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60a4015030 con 0x555a6fcdc0f0 2024-03-28T17:53:33.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.578+0000 7f60a8c00640 1 -- 172.21.15.45:0/935653049 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a40116a0 con 0x555a6fcdc0f0 2024-03-28T17:53:33.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.579+0000 7f60bdb94740 1 -- 172.21.15.45:0/935653049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fcdc0f0 msgr2=0x555a6fce4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:53:33.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.579+0000 7f60bdb94740 1 --2- 172.21.15.45:0/935653049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fcdc0f0 0x555a6fce4670 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f60a400b710 tx=0x7f60a4004810 comp rx=0 tx=0).stop 2024-03-28T17:53:33.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.579+0000 7f60bdb94740 1 -- 172.21.15.45:0/935653049 shutdown_connections 2024-03-28T17:53:33.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.579+0000 7f60bdb94740 1 --2- 172.21.15.45:0/935653049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555a6fce5460 0x555a6fce6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.579+0000 7f60bdb94740 1 --2- 172.21.15.45:0/935653049 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fcdcdd0 0x555a6fce4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.579+0000 7f60bdb94740 1 --2- 172.21.15.45:0/935653049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fcdc0f0 0x555a6fce4670 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.579+0000 7f60bdb94740 1 -- 172.21.15.45:0/935653049 >> 172.21.15.45:0/935653049 conn(0x555a6fca3200 msgr2=0x555a6fce1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:53:33.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.579+0000 7f60bdb94740 1 -- 172.21.15.45:0/935653049 shutdown_connections 2024-03-28T17:53:33.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.579+0000 7f60bdb94740 1 -- 172.21.15.45:0/935653049 wait complete. 2024-03-28T17:53:33.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.579+0000 7f60bdb94740 1 -- start start 2024-03-28T17:53:33.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.580+0000 7f60bdb94740 1 Processor -- start 2024-03-28T17:53:33.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.580+0000 7f60bdb94740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555a6fcdc0f0 0x555a6fcde310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:33.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.580+0000 7f60bdb94740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fcdcdd0 0x555a6fcde850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:33.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.580+0000 7f60bdb94740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fce5460 0x555a6fcded90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:33.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.580+0000 7f60bdb94740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b41aee00 con 0x555a6fcdc0f0 2024-03-28T17:53:33.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.580+0000 7f60bdb94740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b41adb60 con 0x555a6fcdcdd0 2024-03-28T17:53:33.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.580+0000 7f60bdb94740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b41ae160 con 0x555a6fce5460 2024-03-28T17:53:33.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.580+0000 7f60bdb94740 10 client.0 _ll_register_callbacks cb 0x7f60ac009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:53:33.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.580+0000 7f60b3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555a6fcdc0f0 0x555a6fcde310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:33.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.580+0000 7f60ab400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fcdcdd0 0x555a6fcde850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:33.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.580+0000 7f60ab400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fcdcdd0 0x555a6fcde850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:60388/0 (socket says 172.21.15.45:60388) 2024-03-28T17:53:33.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.580+0000 7f60ab400640 1 -- 172.21.15.45:0/2187809707 learned_addr learned my addr 172.21.15.45:0/2187809707 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:53:33.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.580+0000 7f60b3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555a6fcdc0f0 0x555a6fcde310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36308/0 (socket says 172.21.15.45:36308) 2024-03-28T17:53:33.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.580+0000 7f60b8e00640 1 --2- 172.21.15.45:0/2187809707 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fce5460 0x555a6fcded90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:33.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.581+0000 7f60ab400640 1 -- 172.21.15.45:0/2187809707 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fce5460 msgr2=0x555a6fcded90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:53:33.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.581+0000 7f60ab400640 1 --2- 172.21.15.45:0/2187809707 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fce5460 0x555a6fcded90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.581+0000 7f60ab400640 1 -- 172.21.15.45:0/2187809707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555a6fcdc0f0 msgr2=0x555a6fcde310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:53:33.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.581+0000 7f60ab400640 1 --2- 172.21.15.45:0/2187809707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555a6fcdc0f0 0x555a6fcde310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.581+0000 7f60ab400640 1 -- 172.21.15.45:0/2187809707 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a6fcdff50 con 0x555a6fcdcdd0 2024-03-28T17:53:33.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.581+0000 7f60ab400640 1 --2- 172.21.15.45:0/2187809707 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fcdcdd0 0x555a6fcde850 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f609c0b9970 tx=0x7f609c0045a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33fbb9fe56f1b7b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:53:33.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.581+0000 7f60a2000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:53:33.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.581+0000 7f60a2000640 1 -- 172.21.15.45:0/2187809707 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c04b040 con 0x555a6fcdcdd0 2024-03-28T17:53:33.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.581+0000 7f60a2000640 1 -- 172.21.15.45:0/2187809707 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9458+0+0 (secure 0 0 0) 0x7f609c0c5200 con 0x555a6fcdcdd0 2024-03-28T17:53:33.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.581+0000 7f60bdb94740 1 -- 172.21.15.45:0/2187809707 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f60a4015030 con 0x555a6fcdcdd0 2024-03-28T17:53:33.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.581+0000 7f60a2000640 1 -- 172.21.15.45:0/2187809707 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) v2 ==== 24+0+0 (secure 0 0 0) 0x7f609c0f5040 con 0x555a6fcdcdd0 2024-03-28T17:53:33.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.581+0000 7f60bdb94740 10 client.26069 fetch_fsmap learned FSMap version 96 2024-03-28T17:53:33.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.581+0000 7f60bdb94740 1 -- 172.21.15.45:0/2187809707 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=4+}) v3 -- 0x555a6fd50f40 con 0x555a6fcdcdd0 2024-03-28T17:53:33.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.582+0000 7f60a2000640 1 -- 172.21.15.45:0/2187809707 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60a4015030 con 0x555a6fcdcdd0 2024-03-28T17:53:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.582+0000 7f60a2000640 1 -- 172.21.15.45:0/2187809707 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 96) v1 ==== 7343+0+0 (secure 0 0 0) 0x7f609c0bc3f0 con 0x555a6fcdcdd0 2024-03-28T17:53:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.582+0000 7f60bdb94740 10 client.26069 fetch_fsmap finished waiting for FSMap version 96 2024-03-28T17:53:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.582+0000 7f60bdb94740 10 client.26069 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:53:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.582+0000 7f60bdb94740 20 client.26069 populate_metadata read hostname 'smithi045' 2024-03-28T17:53:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.582+0000 7f60bdb94740 1 --2- 172.21.15.45:0/2187809707 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555a6fd516c0 0x555a6fd53ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:53:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.582+0000 7f60bdb94740 4 client.26069 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:53:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.582+0000 7f60bdb94740 1 -- 172.21.15.45:0/2187809707 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555a6fc47ae0 con 0x555a6fd516c0 2024-03-28T17:53:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.582+0000 7f60b3e00640 1 --2- 172.21.15.45:0/2187809707 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555a6fd516c0 0x555a6fd53ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:53:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.583+0000 7f60b3e00640 1 --2- 172.21.15.45:0/2187809707 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555a6fd516c0 0x555a6fd53ae0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bfb2253992a4d3b server_cookie=49d4337fedb2b554 in_seq=0 out_seq=0 2024-03-28T17:53:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.583+0000 7f60a2000640 10 client.26069 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:53:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.583+0000 7f60a2000640 1 -- 172.21.15.45:0/2187809707 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x555a6fc47ae0 con 0x555a6fd516c0 2024-03-28T17:53:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.583+0000 7f60a2000640 10 client.26069 handle_command_reply: tid=0 2024-03-28T17:53:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.583+0000 7f60bdb94740 1 client.26069 shutdown 2024-03-28T17:53:33.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.583+0000 7f60bdb94740 1 -- 172.21.15.45:0/2187809707 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fcdcdd0 msgr2=0x555a6fcde850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:53:33.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.583+0000 7f60bdb94740 1 --2- 172.21.15.45:0/2187809707 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fcdcdd0 0x555a6fcde850 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f609c0b9970 tx=0x7f609c0045a0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.583+0000 7f60bdb94740 1 -- 172.21.15.45:0/2187809707 shutdown_connections 2024-03-28T17:53:33.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.583+0000 7f60bdb94740 1 --2- 172.21.15.45:0/2187809707 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555a6fd516c0 0x555a6fd53ae0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.583+0000 7f60bdb94740 1 --2- 172.21.15.45:0/2187809707 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555a6fce5460 0x555a6fcded90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.583+0000 7f60bdb94740 1 --2- 172.21.15.45:0/2187809707 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555a6fcdcdd0 0x555a6fcde850 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.583+0000 7f60bdb94740 1 --2- 172.21.15.45:0/2187809707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555a6fcdc0f0 0x555a6fcde310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.583+0000 7f60a2000640 0 client.26069 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:53:33.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.583+0000 7f60bdb94740 1 -- 172.21.15.45:0/2187809707 >> 172.21.15.45:0/2187809707 conn(0x555a6fca3200 msgr2=0x555a6fcd5330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:53:33.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.583+0000 7f60bdb94740 1 -- 172.21.15.45:0/2187809707 shutdown_connections 2024-03-28T17:53:33.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.583+0000 7f60bdb94740 1 -- 172.21.15.45:0/2187809707 wait complete. 2024-03-28T17:53:33.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.583+0000 7f60bdb94740 20 client.26069 trim_cache size 0 max 16384 2024-03-28T17:53:33.589 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:53:33.589 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T17:53:33.589 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T17:53:33.589 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T17:53:33.589 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T17:53:33.589 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T17:53:33.589 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T17:53:33.589 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:53:33.589 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:53:33.589 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:53:33.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.584+0000 7f60b9800640 1 -- 172.21.15.45:0/2582222287 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f60980a2970 msgr2=0x7f60980a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:53:33.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.584+0000 7f60b9800640 1 --2- 172.21.15.45:0/2582222287 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f60980a2970 0x7f60980a4e30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f60ac005fd0 tx=0x7f60ac009500 comp rx=0 tx=0).stop 2024-03-28T17:53:33.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.584+0000 7f60b9800640 1 -- 172.21.15.45:0/2582222287 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60b41040e0 msgr2=0x7f60b419a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:53:33.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.584+0000 7f60b9800640 1 --2- 172.21.15.45:0/2582222287 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60b41040e0 0x7f60b419a630 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f609c005ab0 tx=0x7f609c0042f0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.584+0000 7f60b9800640 1 -- 172.21.15.45:0/2582222287 shutdown_connections 2024-03-28T17:53:33.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.584+0000 7f60b9800640 1 --2- 172.21.15.45:0/2582222287 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f60980a2970 0x7f60980a4e30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.584+0000 7f60b9800640 1 --2- 172.21.15.45:0/2582222287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60b4109730 0x7f60b419e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.584+0000 7f60b9800640 1 --2- 172.21.15.45:0/2582222287 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60b41040e0 0x7f60b419a630 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.584+0000 7f60b9800640 1 --2- 172.21.15.45:0/2582222287 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f60b4103720 0x7f60b419a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:53:33.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.584+0000 7f60b9800640 1 -- 172.21.15.45:0/2582222287 >> 172.21.15.45:0/2582222287 conn(0x7f60b40ff1d0 msgr2=0x7f60b40ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:53:33.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.584+0000 7f60b9800640 1 -- 172.21.15.45:0/2582222287 shutdown_connections 2024-03-28T17:53:33.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:53:33.584+0000 7f60b9800640 1 -- 172.21.15.45:0/2582222287 wait complete. 2024-03-28T17:53:33.602 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T17:53:33.858 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 66 sec latency 181754.977 ms 2024-03-28T17:53:34.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:33 smithi045 ceph-mon[31773]: pgmap v151: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:53:34.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:33 smithi071 ceph-mon[36597]: pgmap v151: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:53:34.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:33 smithi151 ceph-mon[36481]: pgmap v151: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:53:34.858 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 67 sec latency 182755.112 ms 2024-03-28T17:53:35.858 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 68 sec latency 183755.274 ms 2024-03-28T17:53:36.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:36 smithi151 ceph-mon[36481]: pgmap v152: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 272 B/s wr, 12 op/s 2024-03-28T17:53:36.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:36 smithi045 ceph-mon[31773]: pgmap v152: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 272 B/s wr, 12 op/s 2024-03-28T17:53:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:36 smithi071 ceph-mon[36597]: pgmap v152: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 272 B/s wr, 12 op/s 2024-03-28T17:53:36.858 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 69 sec latency 184755.415 ms 2024-03-28T17:53:37.859 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 70 sec latency 185755.565 ms 2024-03-28T17:53:38.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:38 smithi045 ceph-mon[31773]: pgmap v153: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:53:38.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:38 smithi151 ceph-mon[36481]: pgmap v153: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:53:38.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:38 smithi071 ceph-mon[36597]: pgmap v153: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:53:38.859 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 71 sec latency 186755.700 ms 2024-03-28T17:53:39.323 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:38 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 6. 2024-03-28T17:53:39.323 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:38 smithi151 systemd[1]: Stopped Ceph mds.f for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:53:39.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:38 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.f.service: Start request repeated too quickly. 2024-03-28T17:53:39.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:38 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-28T17:53:39.324 INFO:journalctl@ceph.mds.f.smithi151.stdout:Mar 28 17:53:38 smithi151 systemd[1]: Failed to start Ceph mds.f for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:53:39.859 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 72 sec latency 187755.863 ms 2024-03-28T17:53:40.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:40 smithi045 ceph-mon[31773]: pgmap v154: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:53:40.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:40 smithi071 ceph-mon[36597]: pgmap v154: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:53:40.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:40 smithi151 ceph-mon[36481]: pgmap v154: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:53:40.859 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 73 sec latency 188756.010 ms 2024-03-28T17:53:41.859 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 74 sec latency 189756.131 ms 2024-03-28T17:53:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:42 smithi045 ceph-mon[31773]: pgmap v155: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:53:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:42 smithi071 ceph-mon[36597]: pgmap v155: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:53:42.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:42 smithi151 ceph-mon[36481]: pgmap v155: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:53:42.859 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 75 sec latency 190756.297 ms 2024-03-28T17:53:43.859 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 76 sec latency 191756.460 ms 2024-03-28T17:53:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:44 smithi045 ceph-mon[31773]: pgmap v156: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:53:44.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:44 smithi071 ceph-mon[36597]: pgmap v156: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:53:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:44 smithi151 ceph-mon[36481]: pgmap v156: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T17:53:44.859 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 77 sec latency 192756.613 ms 2024-03-28T17:53:45.860 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 78 sec latency 193756.714 ms 2024-03-28T17:53:46.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:46 smithi045 ceph-mon[31773]: pgmap v157: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T17:53:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:46 smithi071 ceph-mon[36597]: pgmap v157: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T17:53:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:46 smithi151 ceph-mon[36481]: pgmap v157: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T17:53:46.860 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 79 sec latency 194756.877 ms 2024-03-28T17:53:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:47 smithi045 ceph-mon[31773]: Replacing daemon mds.f as rank 2 with standby daemon mds.d 2024-03-28T17:53:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:47 smithi045 ceph-mon[31773]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-03-28T17:53:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:47 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:53:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:47 smithi045 ceph-mon[31773]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T17:53:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:47 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:53:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:47 smithi071 ceph-mon[36597]: Replacing daemon mds.f as rank 2 with standby daemon mds.d 2024-03-28T17:53:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:47 smithi071 ceph-mon[36597]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-03-28T17:53:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:47 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:53:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:47 smithi071 ceph-mon[36597]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T17:53:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:47 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:53:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:47 smithi151 ceph-mon[36481]: Replacing daemon mds.f as rank 2 with standby daemon mds.d 2024-03-28T17:53:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:47 smithi151 ceph-mon[36481]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-03-28T17:53:47.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:47 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:53:47.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:47 smithi151 ceph-mon[36481]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T17:53:47.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:47 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:53:47.860 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 80 sec latency 195757.042 ms 2024-03-28T17:53:48.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:48 smithi071 ceph-mon[36597]: pgmap v159: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:53:48.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:48 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/3798594912,v1:172.21.15.45:6837/3798594912] up:resolve 2024-03-28T17:53:48.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:48 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:53:48.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:48 smithi151 ceph-mon[36481]: pgmap v159: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:53:48.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:48 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/3798594912,v1:172.21.15.45:6837/3798594912] up:resolve 2024-03-28T17:53:48.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:48 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:53:48.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:48 smithi045 ceph-mon[31773]: pgmap v159: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:53:48.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:48 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/3798594912,v1:172.21.15.45:6837/3798594912] up:resolve 2024-03-28T17:53:48.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:48 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:53:48.860 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 81 sec latency 196757.161 ms 2024-03-28T17:53:49.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:49 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T17:53:49.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:49 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/3798594912,v1:172.21.15.45:6837/3798594912] up:reconnect 2024-03-28T17:53:49.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:49 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:53:49.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:49 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.0109998 2024-03-28T17:53:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:49 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T17:53:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:49 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/3798594912,v1:172.21.15.45:6837/3798594912] up:reconnect 2024-03-28T17:53:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:49 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:53:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:49 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.0109998 2024-03-28T17:53:49.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:49 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T17:53:49.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:49 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/3798594912,v1:172.21.15.45:6837/3798594912] up:reconnect 2024-03-28T17:53:49.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:49 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:53:49.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:49 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.0109998 2024-03-28T17:53:49.860 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 82 sec latency 197757.313 ms 2024-03-28T17:53:50.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:50 smithi071 ceph-mon[36597]: pgmap v160: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:53:50.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:50 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/3798594912,v1:172.21.15.45:6837/3798594912] up:rejoin 2024-03-28T17:53:50.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:50 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:53:50.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:50 smithi071 ceph-mon[36597]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-03-28T17:53:50.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:50 smithi151 ceph-mon[36481]: pgmap v160: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:53:50.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:50 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/3798594912,v1:172.21.15.45:6837/3798594912] up:rejoin 2024-03-28T17:53:50.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:50 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:53:50.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:50 smithi151 ceph-mon[36481]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-03-28T17:53:50.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-mon[31773]: pgmap v160: 97 pgs: 97 active+clean; 27 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:53:50.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/3798594912,v1:172.21.15.45:6837/3798594912] up:rejoin 2024-03-28T17:53:50.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:53:50.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-mon[31773]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-03-28T17:53:50.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd8b7a00700 time 2024-03-28T17:53:50.257572+0000 2024-03-28T17:53:50.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:53:50.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:50.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd8c32d24a1] 2024-03-28T17:53:50.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd8c32d2667] 2024-03-28T17:53:50.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557fa4778223] 2024-03-28T17:53:50.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557fa45f25d6] 2024-03-28T17:53:50.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557fa45f2da1] 2024-03-28T17:53:50.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557fa45f52d5] 2024-03-28T17:53:50.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557fa4645233] 2024-03-28T17:53:50.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557fa4647197] 2024-03-28T17:53:50.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557fa454611b] 2024-03-28T17:53:50.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557fa45466bc] 2024-03-28T17:53:50.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557fa45301ff] 2024-03-28T17:53:50.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd8c3564488] 2024-03-28T17:53:50.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 13: (DispatchQueue::entry()+0x51f) [0x7fd8c356160f] 2024-03-28T17:53:50.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd8c3634e81] 2024-03-28T17:53:50.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd8c204f1ca] 2024-03-28T17:53:50.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 16: clone() 2024-03-28T17:53:50.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 2024-03-28T17:53:50.258+0000 7fd8b7a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd8b7a00700 time 2024-03-28T17:53:50.257572+0000 2024-03-28T17:53:50.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:53:50.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 2024-03-28T17:53:50.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:50.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd8c32d24a1] 2024-03-28T17:53:50.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd8c32d2667] 2024-03-28T17:53:50.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557fa4778223] 2024-03-28T17:53:50.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557fa45f25d6] 2024-03-28T17:53:50.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557fa45f2da1] 2024-03-28T17:53:50.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557fa45f52d5] 2024-03-28T17:53:50.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557fa4645233] 2024-03-28T17:53:50.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557fa4647197] 2024-03-28T17:53:50.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557fa454611b] 2024-03-28T17:53:50.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557fa45466bc] 2024-03-28T17:53:50.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557fa45301ff] 2024-03-28T17:53:50.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd8c3564488] 2024-03-28T17:53:50.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 13: (DispatchQueue::entry()+0x51f) [0x7fd8c356160f] 2024-03-28T17:53:50.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd8c3634e81] 2024-03-28T17:53:50.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd8c204f1ca] 2024-03-28T17:53:50.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 16: clone() 2024-03-28T17:53:50.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 2024-03-28T17:53:50.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: *** Caught signal (Aborted) ** 2024-03-28T17:53:50.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: in thread 7fd8b7a00700 thread_name:ms_dispatch 2024-03-28T17:53:50.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:50.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd8c2059d20] 2024-03-28T17:53:50.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 2: gsignal() 2024-03-28T17:53:50.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 3: abort() 2024-03-28T17:53:50.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd8c32d24fb] 2024-03-28T17:53:50.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd8c32d2667] 2024-03-28T17:53:50.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557fa4778223] 2024-03-28T17:53:50.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557fa45f25d6] 2024-03-28T17:53:50.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557fa45f2da1] 2024-03-28T17:53:50.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557fa45f52d5] 2024-03-28T17:53:50.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557fa4645233] 2024-03-28T17:53:50.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557fa4647197] 2024-03-28T17:53:50.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557fa454611b] 2024-03-28T17:53:50.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557fa45466bc] 2024-03-28T17:53:50.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557fa45301ff] 2024-03-28T17:53:50.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd8c3564488] 2024-03-28T17:53:50.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 16: (DispatchQueue::entry()+0x51f) [0x7fd8c356160f] 2024-03-28T17:53:50.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd8c3634e81] 2024-03-28T17:53:50.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd8c204f1ca] 2024-03-28T17:53:50.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 19: clone() 2024-03-28T17:53:50.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 2024-03-28T17:53:50.261+0000 7fd8b7a00700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:53:50.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: in thread 7fd8b7a00700 thread_name:ms_dispatch 2024-03-28T17:53:50.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 2024-03-28T17:53:50.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:50.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd8c2059d20] 2024-03-28T17:53:50.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 2: gsignal() 2024-03-28T17:53:50.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 3: abort() 2024-03-28T17:53:50.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd8c32d24fb] 2024-03-28T17:53:50.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd8c32d2667] 2024-03-28T17:53:50.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557fa4778223] 2024-03-28T17:53:50.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557fa45f25d6] 2024-03-28T17:53:50.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557fa45f2da1] 2024-03-28T17:53:50.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557fa45f52d5] 2024-03-28T17:53:50.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557fa4645233] 2024-03-28T17:53:50.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557fa4647197] 2024-03-28T17:53:50.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557fa454611b] 2024-03-28T17:53:50.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557fa45466bc] 2024-03-28T17:53:50.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557fa45301ff] 2024-03-28T17:53:50.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd8c3564488] 2024-03-28T17:53:50.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 16: (DispatchQueue::entry()+0x51f) [0x7fd8c356160f] 2024-03-28T17:53:50.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd8c3634e81] 2024-03-28T17:53:50.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd8c204f1ca] 2024-03-28T17:53:50.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 19: clone() 2024-03-28T17:53:50.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:53:50.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 2024-03-28T17:53:50.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: -1> 2024-03-28T17:53:50.258+0000 7fd8b7a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd8b7a00700 time 2024-03-28T17:53:50.257572+0000 2024-03-28T17:53:50.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:53:50.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 2024-03-28T17:53:50.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:50.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd8c32d24a1] 2024-03-28T17:53:50.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd8c32d2667] 2024-03-28T17:53:50.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557fa4778223] 2024-03-28T17:53:50.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557fa45f25d6] 2024-03-28T17:53:50.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557fa45f2da1] 2024-03-28T17:53:50.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557fa45f52d5] 2024-03-28T17:53:50.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557fa4645233] 2024-03-28T17:53:50.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557fa4647197] 2024-03-28T17:53:50.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557fa454611b] 2024-03-28T17:53:50.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557fa45466bc] 2024-03-28T17:53:50.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557fa45301ff] 2024-03-28T17:53:50.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd8c3564488] 2024-03-28T17:53:50.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 13: (DispatchQueue::entry()+0x51f) [0x7fd8c356160f] 2024-03-28T17:53:50.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd8c3634e81] 2024-03-28T17:53:50.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd8c204f1ca] 2024-03-28T17:53:50.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 16: clone() 2024-03-28T17:53:50.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 2024-03-28T17:53:50.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 0> 2024-03-28T17:53:50.261+0000 7fd8b7a00700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:53:50.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: in thread 7fd8b7a00700 thread_name:ms_dispatch 2024-03-28T17:53:50.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 2024-03-28T17:53:50.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:50.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd8c2059d20] 2024-03-28T17:53:50.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 2: gsignal() 2024-03-28T17:53:50.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 3: abort() 2024-03-28T17:53:50.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd8c32d24fb] 2024-03-28T17:53:50.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd8c32d2667] 2024-03-28T17:53:50.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557fa4778223] 2024-03-28T17:53:50.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557fa45f25d6] 2024-03-28T17:53:50.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557fa45f2da1] 2024-03-28T17:53:50.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557fa45f52d5] 2024-03-28T17:53:50.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557fa4645233] 2024-03-28T17:53:50.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557fa4647197] 2024-03-28T17:53:50.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557fa454611b] 2024-03-28T17:53:50.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557fa45466bc] 2024-03-28T17:53:50.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557fa45301ff] 2024-03-28T17:53:50.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd8c3564488] 2024-03-28T17:53:50.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 16: (DispatchQueue::entry()+0x51f) [0x7fd8c356160f] 2024-03-28T17:53:50.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd8c3634e81] 2024-03-28T17:53:50.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd8c204f1ca] 2024-03-28T17:53:50.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 19: clone() 2024-03-28T17:53:50.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:53:50.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 2024-03-28T17:53:50.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: -1> 2024-03-28T17:53:50.258+0000 7fd8b7a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd8b7a00700 time 2024-03-28T17:53:50.257572+0000 2024-03-28T17:53:50.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:53:50.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 2024-03-28T17:53:50.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:50.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd8c32d24a1] 2024-03-28T17:53:50.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd8c32d2667] 2024-03-28T17:53:50.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557fa4778223] 2024-03-28T17:53:50.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557fa45f25d6] 2024-03-28T17:53:50.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557fa45f2da1] 2024-03-28T17:53:50.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557fa45f52d5] 2024-03-28T17:53:50.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557fa4645233] 2024-03-28T17:53:50.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557fa4647197] 2024-03-28T17:53:50.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557fa454611b] 2024-03-28T17:53:50.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557fa45466bc] 2024-03-28T17:53:50.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557fa45301ff] 2024-03-28T17:53:50.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd8c3564488] 2024-03-28T17:53:50.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 13: (DispatchQueue::entry()+0x51f) [0x7fd8c356160f] 2024-03-28T17:53:50.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd8c3634e81] 2024-03-28T17:53:50.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd8c204f1ca] 2024-03-28T17:53:50.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 16: clone() 2024-03-28T17:53:50.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 2024-03-28T17:53:50.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 0> 2024-03-28T17:53:50.261+0000 7fd8b7a00700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:53:50.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: in thread 7fd8b7a00700 thread_name:ms_dispatch 2024-03-28T17:53:50.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 2024-03-28T17:53:50.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:53:50.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd8c2059d20] 2024-03-28T17:53:50.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 2: gsignal() 2024-03-28T17:53:50.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 3: abort() 2024-03-28T17:53:50.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd8c32d24fb] 2024-03-28T17:53:50.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd8c32d2667] 2024-03-28T17:53:50.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557fa4778223] 2024-03-28T17:53:50.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557fa45f25d6] 2024-03-28T17:53:50.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557fa45f2da1] 2024-03-28T17:53:50.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557fa45f52d5] 2024-03-28T17:53:50.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557fa4645233] 2024-03-28T17:53:50.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557fa4647197] 2024-03-28T17:53:50.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557fa454611b] 2024-03-28T17:53:50.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557fa45466bc] 2024-03-28T17:53:50.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557fa45301ff] 2024-03-28T17:53:50.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd8c3564488] 2024-03-28T17:53:50.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 16: (DispatchQueue::entry()+0x51f) [0x7fd8c356160f] 2024-03-28T17:53:50.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd8c3634e81] 2024-03-28T17:53:50.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd8c204f1ca] 2024-03-28T17:53:50.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 19: clone() 2024-03-28T17:53:50.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:53:50.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90245]: 2024-03-28T17:53:50.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:50 smithi045 podman[90349]: 2024-03-28 17:53:50.383687752 +0000 UTC m=+0.038609597 container died e76f1cd28e0602de7b8a0cc06d9b4b203839a15ab018b549acaa8c9eda344bf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T17:53:50.860 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 83 sec latency 198757.420 ms 2024-03-28T17:53:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:51 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:53:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:51 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:53:51.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:51 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/3798594912,v1:172.21.15.45:6837/3798594912] up:active 2024-03-28T17:53:51.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:51 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T17:53:51.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:51 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:53:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:51 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:53:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:51 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:53:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:51 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/3798594912,v1:172.21.15.45:6837/3798594912] up:active 2024-03-28T17:53:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:51 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T17:53:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:51 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:53:51.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:51 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:53:51.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:51 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:53:51.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:51 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/3798594912,v1:172.21.15.45:6837/3798594912] up:active 2024-03-28T17:53:51.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:51 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T17:53:51.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:51 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:53:51.666 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:51 smithi045 podman[90349]: 2024-03-28 17:53:51.172075481 +0000 UTC m=+0.826997384 container remove e76f1cd28e0602de7b8a0cc06d9b4b203839a15ab018b549acaa8c9eda344bf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-28T17:53:51.666 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:51 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:53:51.860 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 84 sec latency 199757.547 ms 2024-03-28T17:53:52.166 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:51 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-28T17:53:52.166 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:53:51 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Consumed 1.675s CPU time. 2024-03-28T17:53:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:52 smithi071 ceph-mon[36597]: pgmap v161: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:53:52.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:52 smithi045 ceph-mon[31773]: pgmap v161: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:53:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:52 smithi151 ceph-mon[36481]: pgmap v161: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:53:52.861 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 85 sec latency 200757.700 ms 2024-03-28T17:53:53.861 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 86 sec latency 201757.870 ms 2024-03-28T17:53:54.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:54 smithi071 ceph-mon[36597]: pgmap v162: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:53:54.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:54 smithi151 ceph-mon[36481]: pgmap v162: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:53:54.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:54 smithi045 ceph-mon[31773]: pgmap v162: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:53:54.861 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 87 sec latency 202757.965 ms 2024-03-28T17:53:55.861 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 88 sec latency 203758.119 ms 2024-03-28T17:53:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:56 smithi071 ceph-mon[36597]: pgmap v163: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:53:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:56 smithi151 ceph-mon[36481]: pgmap v163: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:53:56.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:56 smithi045 ceph-mon[31773]: pgmap v163: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:53:56.861 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 89 sec latency 204758.288 ms 2024-03-28T17:53:57.861 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 90 sec latency 205758.401 ms 2024-03-28T17:53:58.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:53:58 smithi071 ceph-mon[36597]: pgmap v164: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 189 B/s wr, 10 op/s 2024-03-28T17:53:58.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:53:58 smithi151 ceph-mon[36481]: pgmap v164: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 189 B/s wr, 10 op/s 2024-03-28T17:53:58.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:53:58 smithi045 ceph-mon[31773]: pgmap v164: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 189 B/s wr, 10 op/s 2024-03-28T17:53:58.862 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 91 sec latency 206758.540 ms 2024-03-28T17:53:59.862 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 92 sec latency 207758.691 ms 2024-03-28T17:54:00.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:00 smithi071 ceph-mon[36597]: pgmap v165: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T17:54:00.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:00 smithi151 ceph-mon[36481]: pgmap v165: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T17:54:00.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:00 smithi045 ceph-mon[31773]: pgmap v165: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T17:54:00.862 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 93 sec latency 208758.861 ms 2024-03-28T17:54:01.862 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 94 sec latency 209759.025 ms 2024-03-28T17:54:02.278 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:01 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 2. 2024-03-28T17:54:02.278 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:01 smithi045 systemd[1]: Stopped Ceph mds.d for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:54:02.278 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:01 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Consumed 1.675s CPU time. 2024-03-28T17:54:02.278 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:01 smithi045 systemd[1]: Starting Ceph mds.d for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:54:02.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:02 smithi045 ceph-mon[31773]: pgmap v166: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:54:02.534 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:02 smithi045 podman[90437]: 2024-03-28 17:54:02.23978627 +0000 UTC m=+0.033347166 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:54:02.534 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:02 smithi045 podman[90437]: 2024-03-28 17:54:02.372407208 +0000 UTC m=+0.165968141 container create 433e7e0d716bcdf7e1532c43d06edbe0f510d4a66e5d891b19342eff838faf82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T17:54:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:02 smithi071 ceph-mon[36597]: pgmap v166: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:54:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:02 smithi151 ceph-mon[36481]: pgmap v166: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:54:02.862 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 95 sec latency 210759.139 ms 2024-03-28T17:54:02.916 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:02 smithi045 podman[90437]: 2024-03-28 17:54:02.651445603 +0000 UTC m=+0.445006543 container init 433e7e0d716bcdf7e1532c43d06edbe0f510d4a66e5d891b19342eff838faf82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-28T17:54:02.916 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:02 smithi045 podman[90437]: 2024-03-28 17:54:02.663085261 +0000 UTC m=+0.456646248 container start 433e7e0d716bcdf7e1532c43d06edbe0f510d4a66e5d891b19342eff838faf82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T17:54:02.916 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:02 smithi045 ceph-mds[90456]: -- 172.21.15.45:0/3972550912 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561105717a40 con 0x5611056ea800 2024-03-28T17:54:02.916 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:02 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: starting mds.d at 2024-03-28T17:54:02.916 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:02 smithi045 bash[90437]: 433e7e0d716bcdf7e1532c43d06edbe0f510d4a66e5d891b19342eff838faf82 2024-03-28T17:54:02.916 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:02 smithi045 systemd[1]: Started Ceph mds.d for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:54:03.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:03 smithi071 ceph-mon[36597]: daemon mds.d restarted 2024-03-28T17:54:03.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:03 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:54:03.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:03 smithi071 ceph-mon[36597]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T17:54:03.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:03 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/1652447825,v1:172.21.15.45:6837/1652447825] up:boot 2024-03-28T17:54:03.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:03 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T17:54:03.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:03 smithi071 ceph-mon[36597]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-03-28T17:54:03.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:03 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:54:03.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:03 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:03 smithi151 ceph-mon[36481]: daemon mds.d restarted 2024-03-28T17:54:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:03 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:54:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:03 smithi151 ceph-mon[36481]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T17:54:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:03 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/1652447825,v1:172.21.15.45:6837/1652447825] up:boot 2024-03-28T17:54:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:03 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T17:54:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:03 smithi151 ceph-mon[36481]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-03-28T17:54:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:03 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:54:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:03 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:03.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:54:03.603 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:54:03.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:03 smithi045 ceph-mon[31773]: daemon mds.d restarted 2024-03-28T17:54:03.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:03 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:54:03.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:03 smithi045 ceph-mon[31773]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T17:54:03.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:03 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/1652447825,v1:172.21.15.45:6837/1652447825] up:boot 2024-03-28T17:54:03.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:03 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T17:54:03.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:03 smithi045 ceph-mon[31773]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-03-28T17:54:03.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:03 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:54:03.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:03 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.841+0000 7fe175a00640 1 -- 172.21.15.45:0/526829809 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe170101fd0 msgr2=0x7fe17010e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:54:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.841+0000 7fe175a00640 1 --2- 172.21.15.45:0/526829809 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe170101fd0 0x7fe17010e4e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe168009ca0 tx=0x7fe16802c100 comp rx=0 tx=0).stop 2024-03-28T17:54:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.841+0000 7fe175a00640 1 -- 172.21.15.45:0/526829809 shutdown_connections 2024-03-28T17:54:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.841+0000 7fe175a00640 1 --2- 172.21.15.45:0/526829809 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe170101fd0 0x7fe17010e4e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.841+0000 7fe175a00640 1 --2- 172.21.15.45:0/526829809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe170101610 0x7fe170101a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.841+0000 7fe175a00640 1 --2- 172.21.15.45:0/526829809 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1701076b0 0x7fe170107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.841+0000 7fe175a00640 1 -- 172.21.15.45:0/526829809 >> 172.21.15.45:0/526829809 conn(0x7fe1700fd100 msgr2=0x7fe1700ff520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:54:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.842+0000 7fe175a00640 1 -- 172.21.15.45:0/526829809 shutdown_connections 2024-03-28T17:54:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.842+0000 7fe175a00640 1 -- 172.21.15.45:0/526829809 wait complete. 2024-03-28T17:54:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.843+0000 7fe175a00640 1 Processor -- start 2024-03-28T17:54:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.843+0000 7fe175a00640 1 -- start start 2024-03-28T17:54:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.843+0000 7fe175a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe170101610 0x7fe17019a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.843+0000 7fe175a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe170101fd0 0x7fe17019a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.843+0000 7fe175a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe1701076b0 0x7fe17019e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.843+0000 7fe175a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1701119f0 con 0x7fe170101610 2024-03-28T17:54:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.843+0000 7fe175a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe170111870 con 0x7fe1701076b0 2024-03-28T17:54:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.843+0000 7fe175a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe170111b70 con 0x7fe170101fd0 2024-03-28T17:54:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.843+0000 7fe16ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe170101610 0x7fe17019a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.844+0000 7fe16ea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe170101610 0x7fe17019a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44764/0 (socket says 172.21.15.45:44764) 2024-03-28T17:54:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.844+0000 7fe16ea00640 1 -- 172.21.15.45:0/509625756 learned_addr learned my addr 172.21.15.45:0/509625756 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:54:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.844+0000 7fe16f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe1701076b0 0x7fe17019e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.844+0000 7fe16e000640 1 --2- 172.21.15.45:0/509625756 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe170101fd0 0x7fe17019a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.844+0000 7fe16f400640 1 -- 172.21.15.45:0/509625756 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe170101fd0 msgr2=0x7fe17019a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:54:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.844+0000 7fe16f400640 1 --2- 172.21.15.45:0/509625756 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe170101fd0 0x7fe17019a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.844+0000 7fe16f400640 1 -- 172.21.15.45:0/509625756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe170101610 msgr2=0x7fe17019a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:54:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.844+0000 7fe16f400640 1 --2- 172.21.15.45:0/509625756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe170101610 0x7fe17019a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.844+0000 7fe16f400640 1 -- 172.21.15.45:0/509625756 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17019f080 con 0x7fe1701076b0 2024-03-28T17:54:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.844+0000 7fe16e000640 1 --2- 172.21.15.45:0/509625756 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe170101fd0 0x7fe17019a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:54:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.844+0000 7fe16f400640 1 --2- 172.21.15.45:0/509625756 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe1701076b0 0x7fe17019e950 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe168009bf0 tx=0x7fe168034670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:54:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.845+0000 7fe167400640 1 -- 172.21.15.45:0/509625756 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe168039050 con 0x7fe1701076b0 2024-03-28T17:54:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.845+0000 7fe167400640 1 -- 172.21.15.45:0/509625756 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe168045070 con 0x7fe1701076b0 2024-03-28T17:54:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.845+0000 7fe175a00640 1 -- 172.21.15.45:0/509625756 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe17019f310 con 0x7fe1701076b0 2024-03-28T17:54:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.845+0000 7fe167400640 1 -- 172.21.15.45:0/509625756 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1680044c0 con 0x7fe1701076b0 2024-03-28T17:54:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.845+0000 7fe175a00640 1 -- 172.21.15.45:0/509625756 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1701a6c00 con 0x7fe1701076b0 2024-03-28T17:54:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.848+0000 7fe167400640 1 -- 172.21.15.45:0/509625756 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fe168004660 con 0x7fe1701076b0 2024-03-28T17:54:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.848+0000 7fe17867d740 1 Processor -- start 2024-03-28T17:54:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.848+0000 7fe17867d740 1 -- start start 2024-03-28T17:54:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.849+0000 7fe17867d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558854377c00 0x558854376950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.849+0000 7fe17867d740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854376f20 0x5588543773a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.849+0000 7fe17867d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854380290 0x558854382680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.849+0000 7fe17867d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588542db250 con 0x558854377c00 2024-03-28T17:54:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.849+0000 7fe17867d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x558854283a30 con 0x558854380290 2024-03-28T17:54:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.849+0000 7fe17867d740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588542d72f0 con 0x558854376f20 2024-03-28T17:54:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.849+0000 7fe16ea00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854376f20 0x5588543773a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.849+0000 7fe16ea00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854376f20 0x5588543773a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:47488/0 (socket says 172.21.15.45:47488) 2024-03-28T17:54:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.849+0000 7fe16e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558854377c00 0x558854376950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.849+0000 7fe16ea00640 1 -- 172.21.15.45:0/2415492219 learned_addr learned my addr 172.21.15.45:0/2415492219 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:54:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.849+0000 7fe16e000640 1 --2- 172.21.15.45:0/2415492219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854380290 0x558854382680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.849+0000 7fe16ea00640 1 -- 172.21.15.45:0/2415492219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854380290 msgr2=0x558854382680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:54:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.849+0000 7fe16ea00640 1 --2- 172.21.15.45:0/2415492219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854380290 0x558854382680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.849+0000 7fe16ea00640 1 -- 172.21.15.45:0/2415492219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558854377c00 msgr2=0x558854376950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:54:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.849+0000 7fe16ea00640 1 --2- 172.21.15.45:0/2415492219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558854377c00 0x558854376950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.849+0000 7fe16ea00640 1 -- 172.21.15.45:0/2415492219 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55885426e690 con 0x558854376f20 2024-03-28T17:54:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.849+0000 7fe16e000640 1 --2- 172.21.15.45:0/2415492219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558854377c00 0x558854376950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:54:03.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.850+0000 7fe16ea00640 1 --2- 172.21.15.45:0/2415492219 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854376f20 0x5588543773a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe160009d60 tx=0x7fe160011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e316e57e5c34ad6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:54:03.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.850+0000 7fe167400640 1 --2- 172.21.15.45:0/509625756 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fe1540a29f0 0x7fe1540a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:03.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.850+0000 7fe167400640 1 -- 172.21.15.45:0/509625756 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9630+0+0 (secure 0 0 0) 0x7fe1680f1740 con 0x7fe1701076b0 2024-03-28T17:54:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.850+0000 7fe15be00640 1 -- 172.21.15.45:0/2415492219 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe160004230 con 0x558854376f20 2024-03-28T17:54:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.850+0000 7fe16e000640 1 --2- 172.21.15.45:0/509625756 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fe1540a29f0 0x7fe1540a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.850+0000 7fe15be00640 1 -- 172.21.15.45:0/2415492219 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe1600043d0 con 0x558854376f20 2024-03-28T17:54:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.850+0000 7fe15be00640 1 -- 172.21.15.45:0/2415492219 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe160019830 con 0x558854376f20 2024-03-28T17:54:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.851+0000 7fe17867d740 1 -- 172.21.15.45:0/2415492219 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854376f20 msgr2=0x5588543773a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:54:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.851+0000 7fe17867d740 1 --2- 172.21.15.45:0/2415492219 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854376f20 0x5588543773a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe160009d60 tx=0x7fe160011810 comp rx=0 tx=0).stop 2024-03-28T17:54:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.851+0000 7fe16e000640 1 --2- 172.21.15.45:0/509625756 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fe1540a29f0 0x7fe1540a4eb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe17010e1b0 tx=0x7fe15c009620 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:54:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.851+0000 7fe17867d740 1 -- 172.21.15.45:0/2415492219 shutdown_connections 2024-03-28T17:54:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.851+0000 7fe17867d740 1 --2- 172.21.15.45:0/2415492219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854380290 0x558854382680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.851+0000 7fe17867d740 1 --2- 172.21.15.45:0/2415492219 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854376f20 0x5588543773a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.851+0000 7fe17867d740 1 --2- 172.21.15.45:0/2415492219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558854377c00 0x558854376950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.851+0000 7fe17867d740 1 -- 172.21.15.45:0/2415492219 >> 172.21.15.45:0/2415492219 conn(0x55885433e200 msgr2=0x558854371970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:54:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.851+0000 7fe17867d740 1 -- 172.21.15.45:0/2415492219 shutdown_connections 2024-03-28T17:54:03.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.851+0000 7fe17867d740 1 -- 172.21.15.45:0/2415492219 wait complete. 2024-03-28T17:54:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.851+0000 7fe17867d740 1 -- start start 2024-03-28T17:54:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.852+0000 7fe17867d740 1 Processor -- start 2024-03-28T17:54:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.852+0000 7fe17867d740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854376f20 0x55885437f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.852+0000 7fe17867d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558854377c00 0x55885437cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.852+0000 7fe17867d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854380290 0x55885437d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.852+0000 7fe17867d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558854383100 con 0x558854377c00 2024-03-28T17:54:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.852+0000 7fe17867d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x558854296e80 con 0x558854380290 2024-03-28T17:54:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.852+0000 7fe17867d740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x558854383280 con 0x558854376f20 2024-03-28T17:54:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.852+0000 7fe17867d740 10 client.0 _ll_register_callbacks cb 0x5588542a9130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:54:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.852+0000 7fe16ea00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854376f20 0x55885437f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.852+0000 7fe16e000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558854377c00 0x55885437cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.852+0000 7fe16f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854380290 0x55885437d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.852+0000 7fe16ea00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854376f20 0x55885437f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:47502/0 (socket says 172.21.15.45:47502) 2024-03-28T17:54:03.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.852+0000 7fe16ea00640 1 -- 172.21.15.45:0/2282798586 learned_addr learned my addr 172.21.15.45:0/2282798586 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:54:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.853+0000 7fe16f400640 1 -- 172.21.15.45:0/2282798586 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854376f20 msgr2=0x55885437f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:54:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.853+0000 7fe16f400640 1 --2- 172.21.15.45:0/2282798586 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854376f20 0x55885437f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.853+0000 7fe16f400640 1 -- 172.21.15.45:0/2282798586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558854377c00 msgr2=0x55885437cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:54:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.853+0000 7fe16f400640 1 --2- 172.21.15.45:0/2282798586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558854377c00 0x55885437cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.853+0000 7fe16f400640 1 -- 172.21.15.45:0/2282798586 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5588540b97d0 con 0x558854380290 2024-03-28T17:54:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.853+0000 7fe16f400640 1 --2- 172.21.15.45:0/2282798586 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854380290 0x55885437d170 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe1680eeaf0 tx=0x7fe1680bd420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8698c04b92908dee server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:54:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.853+0000 7fe158c00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:54:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.853+0000 7fe158c00640 1 -- 172.21.15.45:0/2282798586 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1680c1b50 con 0x558854380290 2024-03-28T17:54:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.853+0000 7fe158c00640 1 -- 172.21.15.45:0/2282798586 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9630+0+0 (secure 0 0 0) 0x7fe1680ba070 con 0x558854380290 2024-03-28T17:54:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.853+0000 7fe17867d740 1 -- 172.21.15.45:0/2282798586 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe1600043d0 con 0x558854380290 2024-03-28T17:54:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.854+0000 7fe158c00640 1 -- 172.21.15.45:0/2282798586 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe1680c61a0 con 0x558854380290 2024-03-28T17:54:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.854+0000 7fe17867d740 10 client.26087 fetch_fsmap learned FSMap version 104 2024-03-28T17:54:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.854+0000 7fe17867d740 1 -- 172.21.15.45:0/2282798586 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=4+}) v3 -- 0x558853fef3f0 con 0x558854380290 2024-03-28T17:54:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.854+0000 7fe158c00640 1 -- 172.21.15.45:0/2282798586 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe1600043d0 con 0x558854380290 2024-03-28T17:54:03.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.854+0000 7fe158c00640 1 -- 172.21.15.45:0/2282798586 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 104) v1 ==== 6772+0+0 (secure 0 0 0) 0x7fe168004c60 con 0x558854380290 2024-03-28T17:54:03.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.854+0000 7fe17867d740 10 client.26087 fetch_fsmap finished waiting for FSMap version 104 2024-03-28T17:54:03.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.854+0000 7fe17867d740 10 client.26087 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:54:03.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.854+0000 7fe17867d740 20 client.26087 populate_metadata read hostname 'smithi045' 2024-03-28T17:54:03.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.854+0000 7fe17867d740 1 --2- 172.21.15.45:0/2282798586 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55885438ff90 0x558854392380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:03.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.854+0000 7fe17867d740 4 client.26087 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:54:03.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.854+0000 7fe17867d740 1 -- 172.21.15.45:0/2282798586 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5588542e8790 con 0x55885438ff90 2024-03-28T17:54:03.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.854+0000 7fe16ea00640 1 --2- 172.21.15.45:0/2282798586 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55885438ff90 0x558854392380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:03.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.855+0000 7fe16ea00640 1 --2- 172.21.15.45:0/2282798586 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55885438ff90 0x558854392380 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25df8ffad94d780e server_cookie=933eb89ba89b5207 in_seq=0 out_seq=0 2024-03-28T17:54:03.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.855+0000 7fe158c00640 10 client.26087 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:54:03.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.856+0000 7fe158c00640 1 -- 172.21.15.45:0/2282798586 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5588542e8790 con 0x55885438ff90 2024-03-28T17:54:03.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.856+0000 7fe158c00640 10 client.26087 handle_command_reply: tid=0 2024-03-28T17:54:03.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.856+0000 7fe17867d740 1 client.26087 shutdown 2024-03-28T17:54:03.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.856+0000 7fe17867d740 1 -- 172.21.15.45:0/2282798586 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854380290 msgr2=0x55885437d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:54:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.856+0000 7fe17867d740 1 --2- 172.21.15.45:0/2282798586 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854380290 0x55885437d170 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe1680eeaf0 tx=0x7fe1680bd420 comp rx=0 tx=0).stop 2024-03-28T17:54:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.856+0000 7fe17867d740 1 -- 172.21.15.45:0/2282798586 shutdown_connections 2024-03-28T17:54:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.856+0000 7fe17867d740 1 --2- 172.21.15.45:0/2282798586 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55885438ff90 0x558854392380 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.856+0000 7fe17867d740 1 --2- 172.21.15.45:0/2282798586 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854380290 0x55885437d170 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.856+0000 7fe17867d740 1 --2- 172.21.15.45:0/2282798586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558854377c00 0x55885437cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.856+0000 7fe17867d740 1 --2- 172.21.15.45:0/2282798586 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854376f20 0x55885437f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.856+0000 7fe17867d740 1 -- 172.21.15.45:0/2282798586 >> 172.21.15.45:0/2282798586 conn(0x55885433e200 msgr2=0x5588543736c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:54:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.856+0000 7fe158c00640 0 client.26087 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:54:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.856+0000 7fe17867d740 1 -- 172.21.15.45:0/2282798586 shutdown_connections 2024-03-28T17:54:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.856+0000 7fe17867d740 1 -- 172.21.15.45:0/2282798586 wait complete. 2024-03-28T17:54:03.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.856+0000 7fe17867d740 20 client.26087 trim_cache size 0 max 16384 2024-03-28T17:54:03.862 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 96 sec latency 211759.219 ms 2024-03-28T17:54:03.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.874+0000 7fe17867d740 1 Processor -- start 2024-03-28T17:54:03.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.874+0000 7fe17867d740 1 -- start start 2024-03-28T17:54:03.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.874+0000 7fe17867d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854376f20 0x55885437bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:03.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.874+0000 7fe17867d740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854377c00 0x55885437f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:03.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.874+0000 7fe17867d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558854380290 0x55885437f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:03.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.874+0000 7fe17867d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1701b0e60 con 0x558854380290 2024-03-28T17:54:03.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.874+0000 7fe17867d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1701b14c0 con 0x558854376f20 2024-03-28T17:54:03.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.874+0000 7fe17867d740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1701af680 con 0x558854377c00 2024-03-28T17:54:03.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.875+0000 7fe16ea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854376f20 0x55885437bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:03.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.875+0000 7fe16ea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854376f20 0x55885437bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:35086/0 (socket says 172.21.15.45:35086) 2024-03-28T17:54:03.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.875+0000 7fe16ea00640 1 -- 172.21.15.45:0/3370874220 learned_addr learned my addr 172.21.15.45:0/3370874220 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:54:03.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.875+0000 7fe16e000640 1 --2- 172.21.15.45:0/3370874220 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854377c00 0x55885437f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:03.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.875+0000 7fe16f400640 1 --2- 172.21.15.45:0/3370874220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558854380290 0x55885437f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:03.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.875+0000 7fe16ea00640 1 -- 172.21.15.45:0/3370874220 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854377c00 msgr2=0x55885437f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:54:03.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.875+0000 7fe16ea00640 1 --2- 172.21.15.45:0/3370874220 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854377c00 0x55885437f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.875+0000 7fe16ea00640 1 -- 172.21.15.45:0/3370874220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558854380290 msgr2=0x55885437f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:54:03.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.875+0000 7fe16ea00640 1 --2- 172.21.15.45:0/3370874220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558854380290 0x55885437f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.875+0000 7fe16ea00640 1 -- 172.21.15.45:0/3370874220 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588543926f0 con 0x558854376f20 2024-03-28T17:54:03.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.875+0000 7fe16f400640 1 --2- 172.21.15.45:0/3370874220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558854380290 0x55885437f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:54:03.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.875+0000 7fe16ea00640 1 --2- 172.21.15.45:0/3370874220 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854376f20 0x55885437bc30 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe1600058c0 tx=0x7fe160004420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2902cac248c5631 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:54:03.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.875+0000 7fe15b400640 1 -- 172.21.15.45:0/3370874220 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16002a150 con 0x558854376f20 2024-03-28T17:54:03.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.875+0000 7fe15b400640 1 -- 172.21.15.45:0/3370874220 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe16002f070 con 0x558854376f20 2024-03-28T17:54:03.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.875+0000 7fe15b400640 1 -- 172.21.15.45:0/3370874220 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1600052a0 con 0x558854376f20 2024-03-28T17:54:03.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.876+0000 7fe17867d740 1 -- 172.21.15.45:0/3370874220 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854376f20 msgr2=0x55885437bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:54:03.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.876+0000 7fe17867d740 1 --2- 172.21.15.45:0/3370874220 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854376f20 0x55885437bc30 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe1600058c0 tx=0x7fe160004420 comp rx=0 tx=0).stop 2024-03-28T17:54:03.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.876+0000 7fe17867d740 1 -- 172.21.15.45:0/3370874220 shutdown_connections 2024-03-28T17:54:03.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.876+0000 7fe17867d740 1 --2- 172.21.15.45:0/3370874220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558854380290 0x55885437f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.876+0000 7fe17867d740 1 --2- 172.21.15.45:0/3370874220 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854377c00 0x55885437f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.876+0000 7fe17867d740 1 --2- 172.21.15.45:0/3370874220 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854376f20 0x55885437bc30 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.876+0000 7fe17867d740 1 -- 172.21.15.45:0/3370874220 >> 172.21.15.45:0/3370874220 conn(0x55885433e200 msgr2=0x55885437b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:54:03.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.876+0000 7fe17867d740 1 -- 172.21.15.45:0/3370874220 shutdown_connections 2024-03-28T17:54:03.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.876+0000 7fe17867d740 1 -- 172.21.15.45:0/3370874220 wait complete. 2024-03-28T17:54:03.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.877+0000 7fe17867d740 1 -- start start 2024-03-28T17:54:03.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.877+0000 7fe17867d740 1 Processor -- start 2024-03-28T17:54:03.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.877+0000 7fe17867d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854376f20 0x5588543794e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:03.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.877+0000 7fe17867d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558854377c00 0x558854379a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:03.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.877+0000 7fe17867d740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854380290 0x558854379f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:03.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.877+0000 7fe17867d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1701af020 con 0x558854377c00 2024-03-28T17:54:03.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.877+0000 7fe17867d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1701add80 con 0x558854376f20 2024-03-28T17:54:03.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.877+0000 7fe17867d740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1701ae380 con 0x558854380290 2024-03-28T17:54:03.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.877+0000 7fe17867d740 10 client.0 _ll_register_callbacks cb 0x7fe1680bd840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:54:03.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.878+0000 7fe16ea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854376f20 0x5588543794e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:03.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.878+0000 7fe16ea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854376f20 0x5588543794e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:35102/0 (socket says 172.21.15.45:35102) 2024-03-28T17:54:03.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.878+0000 7fe16ea00640 1 -- 172.21.15.45:0/3228863658 learned_addr learned my addr 172.21.15.45:0/3228863658 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:54:03.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.878+0000 7fe16f400640 1 --2- 172.21.15.45:0/3228863658 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854380290 0x558854379f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:03.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.878+0000 7fe16e000640 1 --2- 172.21.15.45:0/3228863658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558854377c00 0x558854379a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:03.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.878+0000 7fe16ea00640 1 -- 172.21.15.45:0/3228863658 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854380290 msgr2=0x558854379f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:54:03.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.878+0000 7fe16ea00640 1 --2- 172.21.15.45:0/3228863658 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854380290 0x558854379f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.878+0000 7fe16ea00640 1 -- 172.21.15.45:0/3228863658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558854377c00 msgr2=0x558854379a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:54:03.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.878+0000 7fe16ea00640 1 --2- 172.21.15.45:0/3228863658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558854377c00 0x558854379a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.878+0000 7fe16ea00640 1 -- 172.21.15.45:0/3228863658 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558853faf0f0 con 0x558854376f20 2024-03-28T17:54:03.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.878+0000 7fe16f400640 1 --2- 172.21.15.45:0/3228863658 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854380290 0x558854379f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T17:54:03.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.878+0000 7fe16e000640 1 --2- 172.21.15.45:0/3228863658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558854377c00 0x558854379a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T17:54:03.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.878+0000 7fe16ea00640 1 --2- 172.21.15.45:0/3228863658 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854376f20 0x5588543794e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe160000c00 tx=0x7fe160009530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbe534fe1a636196 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:54:03.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.878+0000 7fe158c00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:54:03.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.879+0000 7fe158c00640 1 -- 172.21.15.45:0/3228863658 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16002aa90 con 0x558854376f20 2024-03-28T17:54:03.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.879+0000 7fe17867d740 1 -- 172.21.15.45:0/3228863658 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe16002f070 con 0x558854376f20 2024-03-28T17:54:03.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.879+0000 7fe158c00640 1 -- 172.21.15.45:0/3228863658 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9630+0+0 (secure 0 0 0) 0x7fe160005040 con 0x558854376f20 2024-03-28T17:54:03.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.879+0000 7fe158c00640 1 -- 172.21.15.45:0/3228863658 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe16002ac30 con 0x558854376f20 2024-03-28T17:54:03.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.879+0000 7fe17867d740 10 client.26093 fetch_fsmap learned FSMap version 104 2024-03-28T17:54:03.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.879+0000 7fe17867d740 1 -- 172.21.15.45:0/3228863658 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=4+}) v3 -- 0x5588543eb800 con 0x558854376f20 2024-03-28T17:54:03.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.879+0000 7fe158c00640 1 -- 172.21.15.45:0/3228863658 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe16002f070 con 0x558854376f20 2024-03-28T17:54:03.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.879+0000 7fe158c00640 1 -- 172.21.15.45:0/3228863658 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 104) v1 ==== 6772+0+0 (secure 0 0 0) 0x7fe1600035f0 con 0x558854376f20 2024-03-28T17:54:03.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.879+0000 7fe17867d740 10 client.26093 fetch_fsmap finished waiting for FSMap version 104 2024-03-28T17:54:03.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.879+0000 7fe17867d740 10 client.26093 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:54:03.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.879+0000 7fe17867d740 20 client.26093 populate_metadata read hostname 'smithi045' 2024-03-28T17:54:03.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.879+0000 7fe17867d740 1 --2- 172.21.15.45:0/3228863658 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5588543ebe20 0x5588543ee210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:03.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.879+0000 7fe17867d740 4 client.26093 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:54:03.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.879+0000 7fe17867d740 1 -- 172.21.15.45:0/3228863658 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5588542e2ae0 con 0x5588543ebe20 2024-03-28T17:54:03.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.880+0000 7fe16e000640 1 --2- 172.21.15.45:0/3228863658 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5588543ebe20 0x5588543ee210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:03.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.880+0000 7fe16e000640 1 --2- 172.21.15.45:0/3228863658 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5588543ebe20 0x5588543ee210 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c949a5e6df3288bc server_cookie=fd20a7b961483de7 in_seq=0 out_seq=0 2024-03-28T17:54:03.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.880+0000 7fe158c00640 10 client.26093 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:54:03.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.880+0000 7fe158c00640 1 -- 172.21.15.45:0/3228863658 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5588542e2ae0 con 0x5588543ebe20 2024-03-28T17:54:03.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.880+0000 7fe158c00640 10 client.26093 handle_command_reply: tid=0 2024-03-28T17:54:03.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.880+0000 7fe17867d740 1 client.26093 shutdown 2024-03-28T17:54:03.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.880+0000 7fe17867d740 1 -- 172.21.15.45:0/3228863658 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854376f20 msgr2=0x5588543794e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:54:03.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.880+0000 7fe17867d740 1 --2- 172.21.15.45:0/3228863658 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854376f20 0x5588543794e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe160000c00 tx=0x7fe160009530 comp rx=0 tx=0).stop 2024-03-28T17:54:03.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.881+0000 7fe17867d740 1 -- 172.21.15.45:0/3228863658 shutdown_connections 2024-03-28T17:54:03.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.881+0000 7fe17867d740 1 --2- 172.21.15.45:0/3228863658 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5588543ebe20 0x5588543ee210 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.881+0000 7fe17867d740 1 --2- 172.21.15.45:0/3228863658 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x558854380290 0x558854379f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.881+0000 7fe17867d740 1 --2- 172.21.15.45:0/3228863658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558854377c00 0x558854379a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.881+0000 7fe17867d740 1 --2- 172.21.15.45:0/3228863658 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558854376f20 0x5588543794e0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.881+0000 7fe17867d740 1 -- 172.21.15.45:0/3228863658 >> 172.21.15.45:0/3228863658 conn(0x55885433e200 msgr2=0x558854370200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:54:03.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.881+0000 7fe158c00640 0 client.26093 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:54:03.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.881+0000 7fe17867d740 1 -- 172.21.15.45:0/3228863658 shutdown_connections 2024-03-28T17:54:03.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.881+0000 7fe17867d740 1 -- 172.21.15.45:0/3228863658 wait complete. 2024-03-28T17:54:03.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.881+0000 7fe17867d740 20 client.26093 trim_cache size 0 max 16384 2024-03-28T17:54:03.886 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:54:03.886 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T17:54:03.886 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T17:54:03.886 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T17:54:03.886 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T17:54:03.886 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T17:54:03.886 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T17:54:03.886 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:54:03.886 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:54:03.886 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:54:03.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.881+0000 7fe175a00640 1 -- 172.21.15.45:0/509625756 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fe1540a29f0 msgr2=0x7fe1540a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:54:03.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.881+0000 7fe175a00640 1 --2- 172.21.15.45:0/509625756 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fe1540a29f0 0x7fe1540a4eb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe17010e1b0 tx=0x7fe15c009620 comp rx=0 tx=0).stop 2024-03-28T17:54:03.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.881+0000 7fe175a00640 1 -- 172.21.15.45:0/509625756 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe1701076b0 msgr2=0x7fe17019e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:54:03.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.881+0000 7fe175a00640 1 --2- 172.21.15.45:0/509625756 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe1701076b0 0x7fe17019e950 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe168009bf0 tx=0x7fe168034670 comp rx=0 tx=0).stop 2024-03-28T17:54:03.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.882+0000 7fe175a00640 1 -- 172.21.15.45:0/509625756 shutdown_connections 2024-03-28T17:54:03.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.882+0000 7fe175a00640 1 --2- 172.21.15.45:0/509625756 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fe1540a29f0 0x7fe1540a4eb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.882+0000 7fe175a00640 1 --2- 172.21.15.45:0/509625756 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe1701076b0 0x7fe17019e950 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.882+0000 7fe175a00640 1 --2- 172.21.15.45:0/509625756 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe170101fd0 0x7fe17019a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.882+0000 7fe175a00640 1 --2- 172.21.15.45:0/509625756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe170101610 0x7fe17019a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:03.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.882+0000 7fe175a00640 1 -- 172.21.15.45:0/509625756 >> 172.21.15.45:0/509625756 conn(0x7fe1700fd100 msgr2=0x7fe1700fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:54:03.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.882+0000 7fe175a00640 1 -- 172.21.15.45:0/509625756 shutdown_connections 2024-03-28T17:54:03.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:03.882+0000 7fe175a00640 1 -- 172.21.15.45:0/509625756 wait complete. 2024-03-28T17:54:03.901 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T17:54:04.862 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 97 sec latency 212759.358 ms 2024-03-28T17:54:05.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:04 smithi071 ceph-mon[36597]: pgmap v168: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 10 op/s 2024-03-28T17:54:05.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:04 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/3145454285,v1:172.21.15.71:6833/3145454285] up:resolve 2024-03-28T17:54:05.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:04 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:05.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:04 smithi151 ceph-mon[36481]: pgmap v168: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 10 op/s 2024-03-28T17:54:05.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:04 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/3145454285,v1:172.21.15.71:6833/3145454285] up:resolve 2024-03-28T17:54:05.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:04 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:05.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:04 smithi045 ceph-mon[31773]: pgmap v168: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 10 op/s 2024-03-28T17:54:05.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:04 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/3145454285,v1:172.21.15.71:6833/3145454285] up:resolve 2024-03-28T17:54:05.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:04 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:05.862 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 98 sec latency 213759.518 ms 2024-03-28T17:54:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:06 smithi071 ceph-mon[36597]: pgmap v169: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:54:06.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:06 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:54:06.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:06 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/3145454285,v1:172.21.15.71:6833/3145454285] up:reconnect 2024-03-28T17:54:06.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:06 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:06.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:06 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999984 2024-03-28T17:54:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:06 smithi151 ceph-mon[36481]: pgmap v169: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:54:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:06 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:54:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:06 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/3145454285,v1:172.21.15.71:6833/3145454285] up:reconnect 2024-03-28T17:54:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:06 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:06 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999984 2024-03-28T17:54:06.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:06 smithi045 ceph-mon[31773]: pgmap v169: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:54:06.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:06 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:54:06.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:06 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/3145454285,v1:172.21.15.71:6833/3145454285] up:reconnect 2024-03-28T17:54:06.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:06 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:06.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:06 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999984 2024-03-28T17:54:06.863 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 99 sec latency 214759.678 ms 2024-03-28T17:54:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:07 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/3145454285,v1:172.21.15.71:6833/3145454285] up:rejoin 2024-03-28T17:54:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:07 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:07 smithi151 ceph-mon[36481]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-03-28T17:54:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:07 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/3145454285,v1:172.21.15.71:6833/3145454285] up:rejoin 2024-03-28T17:54:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:07 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:07 smithi045 ceph-mon[31773]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-03-28T17:54:07.666 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/3145454285,v1:172.21.15.71:6833/3145454285] up:rejoin 2024-03-28T17:54:07.666 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:07.667 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-mon[36597]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-03-28T17:54:07.667 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3d083be700 time 2024-03-28T17:54:07.326586+0000 2024-03-28T17:54:07.667 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:54:07.667 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:07.667 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3d129e84a1] 2024-03-28T17:54:07.667 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3d129e8667] 2024-03-28T17:54:07.667 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558e567b3223] 2024-03-28T17:54:07.667 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558e5662d5d6] 2024-03-28T17:54:07.667 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558e5662dda1] 2024-03-28T17:54:07.667 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558e566302d5] 2024-03-28T17:54:07.668 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558e56680233] 2024-03-28T17:54:07.668 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558e56682197] 2024-03-28T17:54:07.668 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558e5658111b] 2024-03-28T17:54:07.668 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558e565816bc] 2024-03-28T17:54:07.668 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558e5656b1ff] 2024-03-28T17:54:07.668 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3d12c7a488] 2024-03-28T17:54:07.668 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 13: (DispatchQueue::entry()+0x51f) [0x7f3d12c7760f] 2024-03-28T17:54:07.668 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3d12d4ae81] 2024-03-28T17:54:07.668 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3d117651ca] 2024-03-28T17:54:07.668 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 16: clone() 2024-03-28T17:54:07.669 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 2024-03-28T17:54:07.327+0000 7f3d083be700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3d083be700 time 2024-03-28T17:54:07.326586+0000 2024-03-28T17:54:07.669 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:54:07.669 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 2024-03-28T17:54:07.669 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:07.669 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3d129e84a1] 2024-03-28T17:54:07.669 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3d129e8667] 2024-03-28T17:54:07.669 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558e567b3223] 2024-03-28T17:54:07.670 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558e5662d5d6] 2024-03-28T17:54:07.670 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558e5662dda1] 2024-03-28T17:54:07.670 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558e566302d5] 2024-03-28T17:54:07.670 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558e56680233] 2024-03-28T17:54:07.670 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558e56682197] 2024-03-28T17:54:07.670 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558e5658111b] 2024-03-28T17:54:07.670 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558e565816bc] 2024-03-28T17:54:07.670 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558e5656b1ff] 2024-03-28T17:54:07.670 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3d12c7a488] 2024-03-28T17:54:07.670 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 13: (DispatchQueue::entry()+0x51f) [0x7f3d12c7760f] 2024-03-28T17:54:07.670 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3d12d4ae81] 2024-03-28T17:54:07.670 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3d117651ca] 2024-03-28T17:54:07.670 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 16: clone() 2024-03-28T17:54:07.670 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 2024-03-28T17:54:07.671 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: *** Caught signal (Aborted) ** 2024-03-28T17:54:07.671 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: in thread 7f3d083be700 thread_name:ms_dispatch 2024-03-28T17:54:07.671 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:07.671 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3d1176fd20] 2024-03-28T17:54:07.671 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 2: gsignal() 2024-03-28T17:54:07.671 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 3: abort() 2024-03-28T17:54:07.671 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3d129e84fb] 2024-03-28T17:54:07.671 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3d129e8667] 2024-03-28T17:54:07.671 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558e567b3223] 2024-03-28T17:54:07.671 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558e5662d5d6] 2024-03-28T17:54:07.671 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558e5662dda1] 2024-03-28T17:54:07.671 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558e566302d5] 2024-03-28T17:54:07.671 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558e56680233] 2024-03-28T17:54:07.671 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558e56682197] 2024-03-28T17:54:07.671 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558e5658111b] 2024-03-28T17:54:07.671 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558e565816bc] 2024-03-28T17:54:07.672 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558e5656b1ff] 2024-03-28T17:54:07.672 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3d12c7a488] 2024-03-28T17:54:07.672 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 16: (DispatchQueue::entry()+0x51f) [0x7f3d12c7760f] 2024-03-28T17:54:07.672 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3d12d4ae81] 2024-03-28T17:54:07.672 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3d117651ca] 2024-03-28T17:54:07.672 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 19: clone() 2024-03-28T17:54:07.672 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 2024-03-28T17:54:07.330+0000 7f3d083be700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:54:07.672 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: in thread 7f3d083be700 thread_name:ms_dispatch 2024-03-28T17:54:07.672 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 2024-03-28T17:54:07.672 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:07.672 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3d1176fd20] 2024-03-28T17:54:07.672 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 2: gsignal() 2024-03-28T17:54:07.672 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 3: abort() 2024-03-28T17:54:07.672 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3d129e84fb] 2024-03-28T17:54:07.672 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3d129e8667] 2024-03-28T17:54:07.672 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558e567b3223] 2024-03-28T17:54:07.673 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558e5662d5d6] 2024-03-28T17:54:07.673 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558e5662dda1] 2024-03-28T17:54:07.673 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558e566302d5] 2024-03-28T17:54:07.673 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558e56680233] 2024-03-28T17:54:07.673 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558e56682197] 2024-03-28T17:54:07.673 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558e5658111b] 2024-03-28T17:54:07.673 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558e565816bc] 2024-03-28T17:54:07.673 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558e5656b1ff] 2024-03-28T17:54:07.673 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3d12c7a488] 2024-03-28T17:54:07.673 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 16: (DispatchQueue::entry()+0x51f) [0x7f3d12c7760f] 2024-03-28T17:54:07.673 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3d12d4ae81] 2024-03-28T17:54:07.673 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3d117651ca] 2024-03-28T17:54:07.673 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 19: clone() 2024-03-28T17:54:07.673 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:54:07.673 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 2024-03-28T17:54:07.673 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: -1> 2024-03-28T17:54:07.327+0000 7f3d083be700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3d083be700 time 2024-03-28T17:54:07.326586+0000 2024-03-28T17:54:07.674 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:54:07.674 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 2024-03-28T17:54:07.674 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:07.674 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3d129e84a1] 2024-03-28T17:54:07.674 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3d129e8667] 2024-03-28T17:54:07.674 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558e567b3223] 2024-03-28T17:54:07.674 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558e5662d5d6] 2024-03-28T17:54:07.674 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558e5662dda1] 2024-03-28T17:54:07.674 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558e566302d5] 2024-03-28T17:54:07.674 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558e56680233] 2024-03-28T17:54:07.674 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558e56682197] 2024-03-28T17:54:07.674 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558e5658111b] 2024-03-28T17:54:07.674 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558e565816bc] 2024-03-28T17:54:07.674 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558e5656b1ff] 2024-03-28T17:54:07.675 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3d12c7a488] 2024-03-28T17:54:07.675 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 13: (DispatchQueue::entry()+0x51f) [0x7f3d12c7760f] 2024-03-28T17:54:07.675 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3d12d4ae81] 2024-03-28T17:54:07.675 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3d117651ca] 2024-03-28T17:54:07.675 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 16: clone() 2024-03-28T17:54:07.675 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 2024-03-28T17:54:07.675 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 0> 2024-03-28T17:54:07.330+0000 7f3d083be700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:54:07.675 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: in thread 7f3d083be700 thread_name:ms_dispatch 2024-03-28T17:54:07.675 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 2024-03-28T17:54:07.675 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:07.675 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3d1176fd20] 2024-03-28T17:54:07.675 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 2: gsignal() 2024-03-28T17:54:07.675 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 3: abort() 2024-03-28T17:54:07.675 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3d129e84fb] 2024-03-28T17:54:07.675 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3d129e8667] 2024-03-28T17:54:07.675 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558e567b3223] 2024-03-28T17:54:07.676 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558e5662d5d6] 2024-03-28T17:54:07.676 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558e5662dda1] 2024-03-28T17:54:07.676 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558e566302d5] 2024-03-28T17:54:07.676 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558e56680233] 2024-03-28T17:54:07.677 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558e56682197] 2024-03-28T17:54:07.677 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558e5658111b] 2024-03-28T17:54:07.677 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558e565816bc] 2024-03-28T17:54:07.677 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558e5656b1ff] 2024-03-28T17:54:07.677 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3d12c7a488] 2024-03-28T17:54:07.677 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 16: (DispatchQueue::entry()+0x51f) [0x7f3d12c7760f] 2024-03-28T17:54:07.677 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3d12d4ae81] 2024-03-28T17:54:07.677 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3d117651ca] 2024-03-28T17:54:07.677 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 19: clone() 2024-03-28T17:54:07.677 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:54:07.677 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 2024-03-28T17:54:07.677 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: -1> 2024-03-28T17:54:07.327+0000 7f3d083be700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3d083be700 time 2024-03-28T17:54:07.326586+0000 2024-03-28T17:54:07.677 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:54:07.677 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 2024-03-28T17:54:07.677 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:07.678 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3d129e84a1] 2024-03-28T17:54:07.678 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3d129e8667] 2024-03-28T17:54:07.678 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558e567b3223] 2024-03-28T17:54:07.678 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558e5662d5d6] 2024-03-28T17:54:07.678 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558e5662dda1] 2024-03-28T17:54:07.678 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558e566302d5] 2024-03-28T17:54:07.678 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558e56680233] 2024-03-28T17:54:07.678 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558e56682197] 2024-03-28T17:54:07.678 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558e5658111b] 2024-03-28T17:54:07.678 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558e565816bc] 2024-03-28T17:54:07.678 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558e5656b1ff] 2024-03-28T17:54:07.678 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3d12c7a488] 2024-03-28T17:54:07.678 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 13: (DispatchQueue::entry()+0x51f) [0x7f3d12c7760f] 2024-03-28T17:54:07.678 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3d12d4ae81] 2024-03-28T17:54:07.679 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3d117651ca] 2024-03-28T17:54:07.679 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 16: clone() 2024-03-28T17:54:07.679 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 2024-03-28T17:54:07.679 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 0> 2024-03-28T17:54:07.330+0000 7f3d083be700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:54:07.679 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: in thread 7f3d083be700 thread_name:ms_dispatch 2024-03-28T17:54:07.679 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 2024-03-28T17:54:07.679 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:07.679 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3d1176fd20] 2024-03-28T17:54:07.679 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 2: gsignal() 2024-03-28T17:54:07.679 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 3: abort() 2024-03-28T17:54:07.679 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3d129e84fb] 2024-03-28T17:54:07.679 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3d129e8667] 2024-03-28T17:54:07.679 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558e567b3223] 2024-03-28T17:54:07.679 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558e5662d5d6] 2024-03-28T17:54:07.679 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558e5662dda1] 2024-03-28T17:54:07.680 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558e566302d5] 2024-03-28T17:54:07.680 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558e56680233] 2024-03-28T17:54:07.680 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558e56682197] 2024-03-28T17:54:07.680 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558e5658111b] 2024-03-28T17:54:07.680 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558e565816bc] 2024-03-28T17:54:07.680 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558e5656b1ff] 2024-03-28T17:54:07.680 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3d12c7a488] 2024-03-28T17:54:07.680 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 16: (DispatchQueue::entry()+0x51f) [0x7f3d12c7760f] 2024-03-28T17:54:07.680 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3d12d4ae81] 2024-03-28T17:54:07.680 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3d117651ca] 2024-03-28T17:54:07.680 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 19: clone() 2024-03-28T17:54:07.680 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:54:07.680 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[65956]: 2024-03-28T17:54:07.680 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:07 smithi071 podman[70967]: 2024-03-28 17:54:07.38630443 +0000 UTC m=+0.022539108 container died 9f17d11e5016d688c6a7b93b43eebf0b6103668fb7a28d69cd5763c3f363bca0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-28T17:54:07.863 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 100 sec latency 215759.828 ms 2024-03-28T17:54:08.404 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:08 smithi071 podman[70967]: 2024-03-28 17:54:08.145138836 +0000 UTC m=+0.781373492 container remove 9f17d11e5016d688c6a7b93b43eebf0b6103668fb7a28d69cd5763c3f363bca0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T17:54:08.404 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:08 smithi071 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:54:08.405 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:08 smithi071 ceph-mon[36597]: pgmap v170: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:54:08.405 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:08 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:54:08.405 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:08 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:54:08.405 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:08 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/3145454285,v1:172.21.15.71:6833/3145454285] up:active 2024-03-28T17:54:08.405 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:08 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:08.405 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:08 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:54:08.442 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:08 smithi045 ceph-mon[31773]: pgmap v170: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:54:08.442 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:08 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:54:08.442 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:08 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:54:08.442 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:08 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/3145454285,v1:172.21.15.71:6833/3145454285] up:active 2024-03-28T17:54:08.442 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:08 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:08.442 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:08 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:54:08.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:08 smithi151 ceph-mon[36481]: pgmap v170: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:54:08.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:08 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:54:08.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:08 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:54:08.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:08 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/3145454285,v1:172.21.15.71:6833/3145454285] up:active 2024-03-28T17:54:08.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:08 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:08.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:08 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:54:08.810 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:08 smithi071 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.k.service: Failed with result 'exit-code'. 2024-03-28T17:54:08.863 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 101 sec latency 216759.908 ms 2024-03-28T17:54:09.863 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 102 sec latency 217759.967 ms 2024-03-28T17:54:10.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:10 smithi151 ceph-mon[36481]: pgmap v171: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:54:10.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:10 smithi045 ceph-mon[31773]: pgmap v171: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:54:10.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:10 smithi071 ceph-mon[36597]: pgmap v171: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:54:10.863 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 103 sec latency 218760.105 ms 2024-03-28T17:54:11.863 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 104 sec latency 219760.251 ms 2024-03-28T17:54:12.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:12 smithi151 ceph-mon[36481]: pgmap v172: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:54:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:12 smithi045 ceph-mon[31773]: pgmap v172: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:54:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:12 smithi071 ceph-mon[36597]: pgmap v172: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:54:12.863 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 105 sec latency 220760.386 ms 2024-03-28T17:54:13.863 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 106 sec latency 221760.531 ms 2024-03-28T17:54:14.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:14 smithi045 ceph-mon[31773]: pgmap v173: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 27 op/s 2024-03-28T17:54:14.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:14 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:54:14.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:14 smithi071 ceph-mon[36597]: pgmap v173: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 27 op/s 2024-03-28T17:54:14.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:14 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:54:14.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:14 smithi151 ceph-mon[36481]: pgmap v173: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 27 op/s 2024-03-28T17:54:14.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:14 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:54:14.863 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 107 sec latency 222760.657 ms 2024-03-28T17:54:15.864 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 108 sec latency 223760.815 ms 2024-03-28T17:54:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:16 smithi045 ceph-mon[31773]: pgmap v174: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-03-28T17:54:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:16 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:54:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:16 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:54:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:16 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:54:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:16 smithi071 ceph-mon[36597]: pgmap v174: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-03-28T17:54:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:16 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:54:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:16 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:54:16.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:16 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:54:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:16 smithi151 ceph-mon[36481]: pgmap v174: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-03-28T17:54:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:16 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:54:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:16 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:54:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:16 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:54:16.864 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 109 sec latency 224760.955 ms 2024-03-28T17:54:17.864 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 110 sec latency 225761.044 ms 2024-03-28T17:54:18.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:18 smithi045 ceph-mon[31773]: pgmap v175: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:54:18.787 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:18 smithi071 ceph-mon[36597]: pgmap v175: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:54:18.788 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:18 smithi071 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.k.service: Scheduled restart job, restart counter is at 1. 2024-03-28T17:54:18.788 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:18 smithi071 systemd[1]: Stopped Ceph mds.k for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:54:18.788 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:18 smithi071 systemd[1]: Starting Ceph mds.k for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:54:18.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:18 smithi151 ceph-mon[36481]: pgmap v175: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:54:18.864 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 111 sec latency 226761.203 ms 2024-03-28T17:54:19.058 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:18 smithi071 podman[71190]: 2024-03-28 17:54:18.690727266 +0000 UTC m=+0.018735037 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:54:19.058 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:18 smithi071 podman[71190]: 2024-03-28 17:54:18.812275736 +0000 UTC m=+0.140283498 container create 763a28229b07bff8b7b6cccdd739f686ae4bb1890b818ac111bd61f4cd640e7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-28T17:54:19.311 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:19 smithi071 podman[71190]: 2024-03-28 17:54:19.059269349 +0000 UTC m=+0.387277102 container init 763a28229b07bff8b7b6cccdd739f686ae4bb1890b818ac111bd61f4cd640e7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-28T17:54:19.311 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:19 smithi071 podman[71190]: 2024-03-28 17:54:19.062709255 +0000 UTC m=+0.390717018 container start 763a28229b07bff8b7b6cccdd739f686ae4bb1890b818ac111bd61f4cd640e7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-28T17:54:19.311 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:19 smithi071 ceph-mds[71209]: -- 172.21.15.71:0/1821331890 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5634446e5a40 con 0x5634446b8c00 2024-03-28T17:54:19.311 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:19 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: starting mds.k at 2024-03-28T17:54:19.311 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:19 smithi071 bash[71190]: 763a28229b07bff8b7b6cccdd739f686ae4bb1890b818ac111bd61f4cd640e7b 2024-03-28T17:54:19.311 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:19 smithi071 systemd[1]: Started Ceph mds.k for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:54:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:19 smithi045 ceph-mon[31773]: daemon mds.k restarted 2024-03-28T17:54:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:19 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:54:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:19 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T17:54:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:19 smithi045 ceph-mon[31773]: osdmap e102: 12 total, 12 up, 12 in 2024-03-28T17:54:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:19 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/2172148780,v1:172.21.15.71:6833/2172148780] up:boot 2024-03-28T17:54:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:19 smithi045 ceph-mon[31773]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-03-28T17:54:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:19 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:54:19.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:19 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:19.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:19 smithi071 ceph-mon[36597]: daemon mds.k restarted 2024-03-28T17:54:19.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:19 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:54:19.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:19 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T17:54:19.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:19 smithi071 ceph-mon[36597]: osdmap e102: 12 total, 12 up, 12 in 2024-03-28T17:54:19.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:19 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/2172148780,v1:172.21.15.71:6833/2172148780] up:boot 2024-03-28T17:54:19.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:19 smithi071 ceph-mon[36597]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-03-28T17:54:19.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:19 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:54:19.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:19 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:19.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:19 smithi151 ceph-mon[36481]: daemon mds.k restarted 2024-03-28T17:54:19.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:19 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:54:19.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:19 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T17:54:19.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:19 smithi151 ceph-mon[36481]: osdmap e102: 12 total, 12 up, 12 in 2024-03-28T17:54:19.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:19 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/2172148780,v1:172.21.15.71:6833/2172148780] up:boot 2024-03-28T17:54:19.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:19 smithi151 ceph-mon[36481]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-03-28T17:54:19.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:19 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:54:19.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:19 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:19.864 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 112 sec latency 227761.313 ms 2024-03-28T17:54:20.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:20 smithi071 ceph-mon[36597]: pgmap v176: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:54:20.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:20 smithi045 ceph-mon[31773]: pgmap v176: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:54:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:20 smithi151 ceph-mon[36481]: pgmap v176: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:54:20.864 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 113 sec latency 228761.467 ms 2024-03-28T17:54:21.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:21 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/1652447825,v1:172.21.15.45:6837/1652447825] up:resolve 2024-03-28T17:54:21.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:21 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:21 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/1652447825,v1:172.21.15.45:6837/1652447825] up:resolve 2024-03-28T17:54:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:21 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:21 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/1652447825,v1:172.21.15.45:6837/1652447825] up:resolve 2024-03-28T17:54:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:21 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:21.864 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 114 sec latency 229761.587 ms 2024-03-28T17:54:22.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:22 smithi045 ceph-mon[31773]: pgmap v178: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:54:22.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:22 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:54:22.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:22 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/1652447825,v1:172.21.15.45:6837/1652447825] up:reconnect 2024-03-28T17:54:22.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:22 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:22.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:22 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T17:54:22.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:22 smithi071 ceph-mon[36597]: pgmap v178: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:54:22.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:22 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:54:22.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:22 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/1652447825,v1:172.21.15.45:6837/1652447825] up:reconnect 2024-03-28T17:54:22.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:22 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:22.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:22 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T17:54:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:22 smithi151 ceph-mon[36481]: pgmap v178: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:54:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:22 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:54:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:22 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/1652447825,v1:172.21.15.45:6837/1652447825] up:reconnect 2024-03-28T17:54:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:22 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:22 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T17:54:22.864 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 115 sec latency 230761.713 ms 2024-03-28T17:54:23.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/1652447825,v1:172.21.15.45:6837/1652447825] up:rejoin 2024-03-28T17:54:23.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:23.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-mon[31773]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-03-28T17:54:23.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:54:23.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:54:23.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/1652447825,v1:172.21.15.45:6837/1652447825] up:active 2024-03-28T17:54:23.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb02b200700 time 2024-03-28T17:54:23.399192+0000 2024-03-28T17:54:23.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:54:23.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:23.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb036a6d4a1] 2024-03-28T17:54:23.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb036a6d667] 2024-03-28T17:54:23.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561103caf223] 2024-03-28T17:54:23.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561103b295d6] 2024-03-28T17:54:23.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561103b29da1] 2024-03-28T17:54:23.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561103b2c2d5] 2024-03-28T17:54:23.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561103b7c233] 2024-03-28T17:54:23.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561103b7e197] 2024-03-28T17:54:23.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561103a7d11b] 2024-03-28T17:54:23.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561103a7d6bc] 2024-03-28T17:54:23.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561103a671ff] 2024-03-28T17:54:23.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb036cff488] 2024-03-28T17:54:23.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 13: (DispatchQueue::entry()+0x51f) [0x7fb036cfc60f] 2024-03-28T17:54:23.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb036dcfe81] 2024-03-28T17:54:23.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb0357ea1ca] 2024-03-28T17:54:23.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 16: clone() 2024-03-28T17:54:23.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 2024-03-28T17:54:23.400+0000 7fb02b200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb02b200700 time 2024-03-28T17:54:23.399192+0000 2024-03-28T17:54:23.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:54:23.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 2024-03-28T17:54:23.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:23.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb036a6d4a1] 2024-03-28T17:54:23.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb036a6d667] 2024-03-28T17:54:23.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561103caf223] 2024-03-28T17:54:23.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561103b295d6] 2024-03-28T17:54:23.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561103b29da1] 2024-03-28T17:54:23.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561103b2c2d5] 2024-03-28T17:54:23.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561103b7c233] 2024-03-28T17:54:23.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561103b7e197] 2024-03-28T17:54:23.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561103a7d11b] 2024-03-28T17:54:23.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561103a7d6bc] 2024-03-28T17:54:23.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561103a671ff] 2024-03-28T17:54:23.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb036cff488] 2024-03-28T17:54:23.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 13: (DispatchQueue::entry()+0x51f) [0x7fb036cfc60f] 2024-03-28T17:54:23.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb036dcfe81] 2024-03-28T17:54:23.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb0357ea1ca] 2024-03-28T17:54:23.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 16: clone() 2024-03-28T17:54:23.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 2024-03-28T17:54:23.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: *** Caught signal (Aborted) ** 2024-03-28T17:54:23.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: in thread 7fb02b200700 thread_name:ms_dispatch 2024-03-28T17:54:23.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:23.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb0357f4d20] 2024-03-28T17:54:23.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 2: gsignal() 2024-03-28T17:54:23.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 3: abort() 2024-03-28T17:54:23.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb036a6d4fb] 2024-03-28T17:54:23.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb036a6d667] 2024-03-28T17:54:23.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561103caf223] 2024-03-28T17:54:23.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561103b295d6] 2024-03-28T17:54:23.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561103b29da1] 2024-03-28T17:54:23.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561103b2c2d5] 2024-03-28T17:54:23.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561103b7c233] 2024-03-28T17:54:23.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561103b7e197] 2024-03-28T17:54:23.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561103a7d11b] 2024-03-28T17:54:23.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561103a7d6bc] 2024-03-28T17:54:23.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561103a671ff] 2024-03-28T17:54:23.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb036cff488] 2024-03-28T17:54:23.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 16: (DispatchQueue::entry()+0x51f) [0x7fb036cfc60f] 2024-03-28T17:54:23.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb036dcfe81] 2024-03-28T17:54:23.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb0357ea1ca] 2024-03-28T17:54:23.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 19: clone() 2024-03-28T17:54:23.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 2024-03-28T17:54:23.403+0000 7fb02b200700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:54:23.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: in thread 7fb02b200700 thread_name:ms_dispatch 2024-03-28T17:54:23.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 2024-03-28T17:54:23.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:23.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb0357f4d20] 2024-03-28T17:54:23.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 2: gsignal() 2024-03-28T17:54:23.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 3: abort() 2024-03-28T17:54:23.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb036a6d4fb] 2024-03-28T17:54:23.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb036a6d667] 2024-03-28T17:54:23.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561103caf223] 2024-03-28T17:54:23.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561103b295d6] 2024-03-28T17:54:23.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561103b29da1] 2024-03-28T17:54:23.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561103b2c2d5] 2024-03-28T17:54:23.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561103b7c233] 2024-03-28T17:54:23.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561103b7e197] 2024-03-28T17:54:23.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561103a7d11b] 2024-03-28T17:54:23.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561103a7d6bc] 2024-03-28T17:54:23.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561103a671ff] 2024-03-28T17:54:23.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb036cff488] 2024-03-28T17:54:23.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 16: (DispatchQueue::entry()+0x51f) [0x7fb036cfc60f] 2024-03-28T17:54:23.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb036dcfe81] 2024-03-28T17:54:23.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb0357ea1ca] 2024-03-28T17:54:23.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 19: clone() 2024-03-28T17:54:23.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:54:23.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 2024-03-28T17:54:23.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: -1> 2024-03-28T17:54:23.400+0000 7fb02b200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb02b200700 time 2024-03-28T17:54:23.399192+0000 2024-03-28T17:54:23.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:54:23.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 2024-03-28T17:54:23.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:23.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb036a6d4a1] 2024-03-28T17:54:23.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb036a6d667] 2024-03-28T17:54:23.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561103caf223] 2024-03-28T17:54:23.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561103b295d6] 2024-03-28T17:54:23.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561103b29da1] 2024-03-28T17:54:23.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561103b2c2d5] 2024-03-28T17:54:23.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561103b7c233] 2024-03-28T17:54:23.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561103b7e197] 2024-03-28T17:54:23.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561103a7d11b] 2024-03-28T17:54:23.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561103a7d6bc] 2024-03-28T17:54:23.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561103a671ff] 2024-03-28T17:54:23.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb036cff488] 2024-03-28T17:54:23.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 13: (DispatchQueue::entry()+0x51f) [0x7fb036cfc60f] 2024-03-28T17:54:23.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb036dcfe81] 2024-03-28T17:54:23.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb0357ea1ca] 2024-03-28T17:54:23.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 16: clone() 2024-03-28T17:54:23.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 2024-03-28T17:54:23.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 0> 2024-03-28T17:54:23.403+0000 7fb02b200700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:54:23.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: in thread 7fb02b200700 thread_name:ms_dispatch 2024-03-28T17:54:23.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 2024-03-28T17:54:23.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:23.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb0357f4d20] 2024-03-28T17:54:23.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 2: gsignal() 2024-03-28T17:54:23.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 3: abort() 2024-03-28T17:54:23.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb036a6d4fb] 2024-03-28T17:54:23.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb036a6d667] 2024-03-28T17:54:23.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561103caf223] 2024-03-28T17:54:23.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561103b295d6] 2024-03-28T17:54:23.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561103b29da1] 2024-03-28T17:54:23.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561103b2c2d5] 2024-03-28T17:54:23.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561103b7c233] 2024-03-28T17:54:23.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561103b7e197] 2024-03-28T17:54:23.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561103a7d11b] 2024-03-28T17:54:23.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561103a7d6bc] 2024-03-28T17:54:23.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561103a671ff] 2024-03-28T17:54:23.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb036cff488] 2024-03-28T17:54:23.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 16: (DispatchQueue::entry()+0x51f) [0x7fb036cfc60f] 2024-03-28T17:54:23.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb036dcfe81] 2024-03-28T17:54:23.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb0357ea1ca] 2024-03-28T17:54:23.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 19: clone() 2024-03-28T17:54:23.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:54:23.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 2024-03-28T17:54:23.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: -1> 2024-03-28T17:54:23.400+0000 7fb02b200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb02b200700 time 2024-03-28T17:54:23.399192+0000 2024-03-28T17:54:23.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:54:23.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 2024-03-28T17:54:23.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:23.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb036a6d4a1] 2024-03-28T17:54:23.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb036a6d667] 2024-03-28T17:54:23.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561103caf223] 2024-03-28T17:54:23.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561103b295d6] 2024-03-28T17:54:23.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561103b29da1] 2024-03-28T17:54:23.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561103b2c2d5] 2024-03-28T17:54:23.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561103b7c233] 2024-03-28T17:54:23.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561103b7e197] 2024-03-28T17:54:23.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561103a7d11b] 2024-03-28T17:54:23.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561103a7d6bc] 2024-03-28T17:54:23.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561103a671ff] 2024-03-28T17:54:23.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb036cff488] 2024-03-28T17:54:23.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 13: (DispatchQueue::entry()+0x51f) [0x7fb036cfc60f] 2024-03-28T17:54:23.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb036dcfe81] 2024-03-28T17:54:23.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb0357ea1ca] 2024-03-28T17:54:23.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 16: clone() 2024-03-28T17:54:23.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 2024-03-28T17:54:23.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 0> 2024-03-28T17:54:23.403+0000 7fb02b200700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:54:23.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: in thread 7fb02b200700 thread_name:ms_dispatch 2024-03-28T17:54:23.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 2024-03-28T17:54:23.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:23.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb0357f4d20] 2024-03-28T17:54:23.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 2: gsignal() 2024-03-28T17:54:23.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 3: abort() 2024-03-28T17:54:23.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb036a6d4fb] 2024-03-28T17:54:23.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb036a6d667] 2024-03-28T17:54:23.683 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561103caf223] 2024-03-28T17:54:23.683 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561103b295d6] 2024-03-28T17:54:23.683 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561103b29da1] 2024-03-28T17:54:23.683 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561103b2c2d5] 2024-03-28T17:54:23.683 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561103b7c233] 2024-03-28T17:54:23.683 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561103b7e197] 2024-03-28T17:54:23.683 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561103a7d11b] 2024-03-28T17:54:23.683 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561103a7d6bc] 2024-03-28T17:54:23.683 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561103a671ff] 2024-03-28T17:54:23.683 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb036cff488] 2024-03-28T17:54:23.684 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 16: (DispatchQueue::entry()+0x51f) [0x7fb036cfc60f] 2024-03-28T17:54:23.684 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb036dcfe81] 2024-03-28T17:54:23.684 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb0357ea1ca] 2024-03-28T17:54:23.684 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 19: clone() 2024-03-28T17:54:23.684 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:54:23.684 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90452]: 2024-03-28T17:54:23.684 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:23 smithi045 podman[90697]: 2024-03-28 17:54:23.512672453 +0000 UTC m=+0.037035697 container died 433e7e0d716bcdf7e1532c43d06edbe0f510d4a66e5d891b19342eff838faf82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-28T17:54:23.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:23 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/1652447825,v1:172.21.15.45:6837/1652447825] up:rejoin 2024-03-28T17:54:23.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:23 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:23.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:23 smithi071 ceph-mon[36597]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-03-28T17:54:23.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:23 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:54:23.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:23 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:54:23.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:23 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/1652447825,v1:172.21.15.45:6837/1652447825] up:active 2024-03-28T17:54:23.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:23 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/1652447825,v1:172.21.15.45:6837/1652447825] up:rejoin 2024-03-28T17:54:23.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:23 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:23.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:23 smithi151 ceph-mon[36481]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-03-28T17:54:23.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:23 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:54:23.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:23 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:54:23.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:23 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/1652447825,v1:172.21.15.45:6837/1652447825] up:active 2024-03-28T17:54:23.865 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 116 sec latency 231761.861 ms 2024-03-28T17:54:24.665 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:24 smithi045 podman[90697]: 2024-03-28 17:54:24.353378057 +0000 UTC m=+0.877741337 container remove 433e7e0d716bcdf7e1532c43d06edbe0f510d4a66e5d891b19342eff838faf82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T17:54:24.666 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:24 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:54:24.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:24 smithi045 ceph-mon[31773]: pgmap v179: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:54:24.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:24 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:24.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:24 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:54:24.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:24 smithi151 ceph-mon[36481]: pgmap v179: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:54:24.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:24 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:24.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:24 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:54:24.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:24 smithi071 ceph-mon[36597]: pgmap v179: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:54:24.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:24 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:24.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:24 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:54:24.865 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 117 sec latency 232761.948 ms 2024-03-28T17:54:25.165 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:24 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-28T17:54:25.166 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:24 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Consumed 1.615s CPU time. 2024-03-28T17:54:25.865 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 118 sec latency 233762.104 ms 2024-03-28T17:54:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:26 smithi045 ceph-mon[31773]: pgmap v180: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:54:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:26 smithi071 ceph-mon[36597]: pgmap v180: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:54:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:26 smithi151 ceph-mon[36481]: pgmap v180: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:54:26.865 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 119 sec latency 234762.244 ms 2024-03-28T17:54:27.865 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 120 sec latency 235762.369 ms 2024-03-28T17:54:28.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:28 smithi045 ceph-mon[31773]: pgmap v181: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:54:28.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:28 smithi071 ceph-mon[36597]: pgmap v181: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:54:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:28 smithi151 ceph-mon[36481]: pgmap v181: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:54:28.865 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 121 sec latency 236762.501 ms 2024-03-28T17:54:29.865 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 122 sec latency 237762.641 ms 2024-03-28T17:54:30.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:30 smithi045 ceph-mon[31773]: pgmap v182: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:54:30.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:30 smithi071 ceph-mon[36597]: pgmap v182: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:54:30.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:30 smithi151 ceph-mon[36481]: pgmap v182: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:54:30.865 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 123 sec latency 238762.791 ms 2024-03-28T17:54:31.866 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 124 sec latency 239762.887 ms 2024-03-28T17:54:32.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:32 smithi045 ceph-mon[31773]: pgmap v183: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 257 B/s wr, 12 op/s 2024-03-28T17:54:32.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:32 smithi071 ceph-mon[36597]: pgmap v183: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 257 B/s wr, 12 op/s 2024-03-28T17:54:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:32 smithi151 ceph-mon[36481]: pgmap v183: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 257 B/s wr, 12 op/s 2024-03-28T17:54:32.866 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 125 sec latency 240763.030 ms 2024-03-28T17:54:33.866 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 126 sec latency 241763.179 ms 2024-03-28T17:54:33.901 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:54:33.902 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:54:34.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.141+0000 7fdcd1a00640 1 -- 172.21.15.45:0/2705057056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdccc103700 msgr2=0x7fdccc103b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:54:34.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.141+0000 7fdcd1a00640 1 --2- 172.21.15.45:0/2705057056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdccc103700 0x7fdccc103b80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4009930 tx=0x7fdcb402bc30 comp rx=0 tx=0).stop 2024-03-28T17:54:34.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.142+0000 7fdcd1a00640 1 -- 172.21.15.45:0/2705057056 shutdown_connections 2024-03-28T17:54:34.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.142+0000 7fdcd1a00640 1 --2- 172.21.15.45:0/2705057056 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdccc1040c0 0x7fdccc1105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.142+0000 7fdcd1a00640 1 --2- 172.21.15.45:0/2705057056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdccc103700 0x7fdccc103b80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.142+0000 7fdcd1a00640 1 --2- 172.21.15.45:0/2705057056 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdccc109710 0x7fdccc109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.142+0000 7fdcd1a00640 1 -- 172.21.15.45:0/2705057056 >> 172.21.15.45:0/2705057056 conn(0x7fdccc0ff1b0 msgr2=0x7fdccc1015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:54:34.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.142+0000 7fdcd1a00640 1 -- 172.21.15.45:0/2705057056 shutdown_connections 2024-03-28T17:54:34.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.142+0000 7fdcd1a00640 1 -- 172.21.15.45:0/2705057056 wait complete. 2024-03-28T17:54:34.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.143+0000 7fdcd1a00640 1 Processor -- start 2024-03-28T17:54:34.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.143+0000 7fdcd1a00640 1 -- start start 2024-03-28T17:54:34.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.143+0000 7fdcd1a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdccc103700 0x7fdccc19a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:34.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.143+0000 7fdcd1a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdccc1040c0 0x7fdccc19a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:34.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.143+0000 7fdcd1a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdccc109710 0x7fdccc19e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:34.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.143+0000 7fdcd1a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc113b70 con 0x7fdccc1040c0 2024-03-28T17:54:34.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.143+0000 7fdcd1a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc1139f0 con 0x7fdccc103700 2024-03-28T17:54:34.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.143+0000 7fdcd1a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc113cf0 con 0x7fdccc109710 2024-03-28T17:54:34.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.143+0000 7fdccbe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdccc103700 0x7fdccc19a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:34.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.144+0000 7fdccbe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdccc103700 0x7fdccc19a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:56182/0 (socket says 172.21.15.45:56182) 2024-03-28T17:54:34.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.144+0000 7fdccbe00640 1 -- 172.21.15.45:0/189004489 learned_addr learned my addr 172.21.15.45:0/189004489 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:54:34.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.144+0000 7fdcc3e00640 1 --2- 172.21.15.45:0/189004489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdccc1040c0 0x7fdccc19a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:34.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.144+0000 7fdcd1000640 1 --2- 172.21.15.45:0/189004489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdccc109710 0x7fdccc19e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:34.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.144+0000 7fdccbe00640 1 -- 172.21.15.45:0/189004489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdccc109710 msgr2=0x7fdccc19e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:54:34.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.144+0000 7fdccbe00640 1 --2- 172.21.15.45:0/189004489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdccc109710 0x7fdccc19e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.144+0000 7fdccbe00640 1 -- 172.21.15.45:0/189004489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdccc1040c0 msgr2=0x7fdccc19a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:54:34.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.144+0000 7fdccbe00640 1 --2- 172.21.15.45:0/189004489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdccc1040c0 0x7fdccc19a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.144+0000 7fdccbe00640 1 -- 172.21.15.45:0/189004489 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdccc19f000 con 0x7fdccc103700 2024-03-28T17:54:34.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.144+0000 7fdcd1000640 1 --2- 172.21.15.45:0/189004489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdccc109710 0x7fdccc19e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:54:34.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.144+0000 7fdccbe00640 1 --2- 172.21.15.45:0/189004489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdccc103700 0x7fdccc19a010 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc00e9f0 tx=0x7fdcbc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:54:34.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.144+0000 7fdcc9600640 1 -- 172.21.15.45:0/189004489 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc00cd80 con 0x7fdccc103700 2024-03-28T17:54:34.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.145+0000 7fdcc9600640 1 -- 172.21.15.45:0/189004489 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdcbc010910 con 0x7fdccc103700 2024-03-28T17:54:34.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.145+0000 7fdcc9600640 1 -- 172.21.15.45:0/189004489 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc018990 con 0x7fdccc103700 2024-03-28T17:54:34.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.145+0000 7fdcd1a00640 1 -- 172.21.15.45:0/189004489 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdccc19f2f0 con 0x7fdccc103700 2024-03-28T17:54:34.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.145+0000 7fdcd1a00640 1 -- 172.21.15.45:0/189004489 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdccc1a6b80 con 0x7fdccc103700 2024-03-28T17:54:34.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.148+0000 7fdcd59fd740 1 Processor -- start 2024-03-28T17:54:34.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.148+0000 7fdcc9600640 1 -- 172.21.15.45:0/189004489 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fdcbc010ac0 con 0x7fdccc103700 2024-03-28T17:54:34.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.148+0000 7fdcd59fd740 1 -- start start 2024-03-28T17:54:34.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.148+0000 7fdcd59fd740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55583080edd0 0x55583080db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:34.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.148+0000 7fdcd59fd740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55583080e0f0 0x55583080e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:34.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.148+0000 7fdcd59fd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555830817460 0x555830819850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:34.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.148+0000 7fdcd59fd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555830772250 con 0x555830817460 2024-03-28T17:54:34.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.148+0000 7fdcd59fd740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55583071aa30 con 0x55583080e0f0 2024-03-28T17:54:34.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.148+0000 7fdcd59fd740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55583076e2f0 con 0x55583080edd0 2024-03-28T17:54:34.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.149+0000 7fdccbe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55583080e0f0 0x55583080e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:34.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.149+0000 7fdccbe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55583080e0f0 0x55583080e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:56198/0 (socket says 172.21.15.45:56198) 2024-03-28T17:54:34.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.149+0000 7fdccbe00640 1 -- 172.21.15.45:0/943827146 learned_addr learned my addr 172.21.15.45:0/943827146 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:54:34.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.149+0000 7fdcc3e00640 1 --2- 172.21.15.45:0/943827146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555830817460 0x555830819850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:34.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.149+0000 7fdcd1000640 1 --2- 172.21.15.45:0/943827146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55583080edd0 0x55583080db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:34.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.149+0000 7fdcc3e00640 1 -- 172.21.15.45:0/943827146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55583080edd0 msgr2=0x55583080db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:54:34.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.149+0000 7fdcc3e00640 1 --2- 172.21.15.45:0/943827146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55583080edd0 0x55583080db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.149+0000 7fdcc3e00640 1 -- 172.21.15.45:0/943827146 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55583080e0f0 msgr2=0x55583080e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:54:34.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.149+0000 7fdcc3e00640 1 --2- 172.21.15.45:0/943827146 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55583080e0f0 0x55583080e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.149+0000 7fdcc3e00640 1 -- 172.21.15.45:0/943827146 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555830705690 con 0x555830817460 2024-03-28T17:54:34.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.149+0000 7fdcc3e00640 1 --2- 172.21.15.45:0/943827146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555830817460 0x555830819850 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4036040 tx=0x7fdcb40095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=104518850f8577c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:54:34.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.150+0000 7fdcc9600640 1 --2- 172.21.15.45:0/189004489 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fdcb00a2970 0x7fdcb00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:34.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.150+0000 7fdcc1600640 1 -- 172.21.15.45:0/943827146 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4045070 con 0x555830817460 2024-03-28T17:54:34.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.150+0000 7fdcc1600640 1 -- 172.21.15.45:0/943827146 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdcb402c880 con 0x555830817460 2024-03-28T17:54:34.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.150+0000 7fdcc1600640 1 -- 172.21.15.45:0/943827146 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4004b20 con 0x555830817460 2024-03-28T17:54:34.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.150+0000 7fdcc9600640 1 -- 172.21.15.45:0/189004489 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9716+0+0 (secure 0 0 0) 0x7fdcbc0cf6e0 con 0x7fdccc103700 2024-03-28T17:54:34.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.150+0000 7fdcd1000640 1 --2- 172.21.15.45:0/189004489 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fdcb00a2970 0x7fdcb00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:34.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.151+0000 7fdcd59fd740 1 -- 172.21.15.45:0/943827146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555830817460 msgr2=0x555830819850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:54:34.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.151+0000 7fdcd59fd740 1 --2- 172.21.15.45:0/943827146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555830817460 0x555830819850 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4036040 tx=0x7fdcb40095e0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.151+0000 7fdcd59fd740 1 -- 172.21.15.45:0/943827146 shutdown_connections 2024-03-28T17:54:34.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.151+0000 7fdcd59fd740 1 --2- 172.21.15.45:0/943827146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555830817460 0x555830819850 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.151+0000 7fdcd59fd740 1 --2- 172.21.15.45:0/943827146 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55583080e0f0 0x55583080e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.151+0000 7fdcd59fd740 1 --2- 172.21.15.45:0/943827146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55583080edd0 0x55583080db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.151+0000 7fdcd59fd740 1 -- 172.21.15.45:0/943827146 >> 172.21.15.45:0/943827146 conn(0x5558307d5200 msgr2=0x555830808b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:54:34.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.151+0000 7fdcd59fd740 1 -- 172.21.15.45:0/943827146 shutdown_connections 2024-03-28T17:54:34.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.151+0000 7fdcd1000640 1 --2- 172.21.15.45:0/189004489 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fdcb00a2970 0x7fdcb00a4e30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4005fd0 tx=0x7fdcc4009800 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:54:34.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.151+0000 7fdcd59fd740 1 -- 172.21.15.45:0/943827146 wait complete. 2024-03-28T17:54:34.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.151+0000 7fdcd59fd740 1 -- start start 2024-03-28T17:54:34.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.152+0000 7fdcd59fd740 1 Processor -- start 2024-03-28T17:54:34.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.152+0000 7fdcd59fd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55583080e0f0 0x555830816c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:34.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.152+0000 7fdcd59fd740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55583080edd0 0x555830813fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:34.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.152+0000 7fdcd59fd740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558308144f0 0x555830814970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:34.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.152+0000 7fdcd59fd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55583081a270 con 0x55583080e0f0 2024-03-28T17:54:34.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.152+0000 7fdcd59fd740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55583072de80 con 0x5558308144f0 2024-03-28T17:54:34.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.152+0000 7fdcd59fd740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55583081a3f0 con 0x55583080edd0 2024-03-28T17:54:34.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.152+0000 7fdcd59fd740 10 client.0 _ll_register_callbacks cb 0x555830740130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:54:34.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.152+0000 7fdccbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55583080e0f0 0x555830816c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:34.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.152+0000 7fdcd1000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558308144f0 0x555830814970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:34.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.152+0000 7fdccbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55583080e0f0 0x555830816c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51124/0 (socket says 172.21.15.45:51124) 2024-03-28T17:54:34.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.152+0000 7fdccbe00640 1 -- 172.21.15.45:0/2563441452 learned_addr learned my addr 172.21.15.45:0/2563441452 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:54:34.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.152+0000 7fdcc3e00640 1 --2- 172.21.15.45:0/2563441452 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55583080edd0 0x555830813fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:34.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.153+0000 7fdccbe00640 1 -- 172.21.15.45:0/2563441452 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55583080edd0 msgr2=0x555830813fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:54:34.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.153+0000 7fdccbe00640 1 --2- 172.21.15.45:0/2563441452 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55583080edd0 0x555830813fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.153+0000 7fdccbe00640 1 -- 172.21.15.45:0/2563441452 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558308144f0 msgr2=0x555830814970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:54:34.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.153+0000 7fdccbe00640 1 --2- 172.21.15.45:0/2563441452 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558308144f0 0x555830814970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.153+0000 7fdccbe00640 1 -- 172.21.15.45:0/2563441452 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558305507d0 con 0x55583080e0f0 2024-03-28T17:54:34.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.153+0000 7fdccbe00640 1 --2- 172.21.15.45:0/2563441452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55583080e0f0 0x555830816c50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc098200 tx=0x7fdcbc099d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64e509775e0e5122 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:54:34.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.153+0000 7fdcba000640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-28T17:54:34.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.153+0000 7fdcba000640 1 -- 172.21.15.45:0/2563441452 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc09b190 con 0x55583080e0f0 2024-03-28T17:54:34.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.153+0000 7fdcba000640 1 -- 172.21.15.45:0/2563441452 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9716+0+0 (secure 0 0 0) 0x7fdcbc09f670 con 0x55583080e0f0 2024-03-28T17:54:34.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.153+0000 7fdcd59fd740 1 -- 172.21.15.45:0/2563441452 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdcb402c880 con 0x55583080e0f0 2024-03-28T17:54:34.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.154+0000 7fdcba000640 1 -- 172.21.15.45:0/2563441452 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=116) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdcbc0ccda0 con 0x55583080e0f0 2024-03-28T17:54:34.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.154+0000 7fdcd59fd740 10 client.15888 fetch_fsmap learned FSMap version 116 2024-03-28T17:54:34.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.154+0000 7fdcd59fd740 1 -- 172.21.15.45:0/2563441452 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=116,monmap=4+}) v3 -- 0x5558304863f0 con 0x55583080e0f0 2024-03-28T17:54:34.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.154+0000 7fdcba000640 1 -- 172.21.15.45:0/2563441452 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdcb402c880 con 0x55583080e0f0 2024-03-28T17:54:34.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.154+0000 7fdcba000640 1 -- 172.21.15.45:0/2563441452 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 116) v1 ==== 6768+0+0 (secure 0 0 0) 0x7fdcbc0a3b40 con 0x55583080e0f0 2024-03-28T17:54:34.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.154+0000 7fdcd59fd740 10 client.15888 fetch_fsmap finished waiting for FSMap version 116 2024-03-28T17:54:34.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.154+0000 7fdcd59fd740 10 client.15888 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:54:34.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.154+0000 7fdcd59fd740 20 client.15888 populate_metadata read hostname 'smithi045' 2024-03-28T17:54:34.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.154+0000 7fdcd59fd740 1 --2- 172.21.15.45:0/2563441452 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555830827890 0x555830829d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:34.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.154+0000 7fdcd59fd740 4 client.15888 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:54:34.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.154+0000 7fdcd59fd740 1 -- 172.21.15.45:0/2563441452 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55583077f790 con 0x555830827890 2024-03-28T17:54:34.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.155+0000 7fdcc3e00640 1 --2- 172.21.15.45:0/2563441452 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555830827890 0x555830829d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:34.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.155+0000 7fdcc3e00640 1 --2- 172.21.15.45:0/2563441452 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555830827890 0x555830829d00 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d8dd3b1de55107c server_cookie=bbb97b9fade36a00 in_seq=0 out_seq=0 2024-03-28T17:54:34.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.155+0000 7fdcba000640 10 client.15888 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:54:34.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.156+0000 7fdcba000640 1 -- 172.21.15.45:0/2563441452 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55583077f790 con 0x555830827890 2024-03-28T17:54:34.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.156+0000 7fdcba000640 10 client.15888 handle_command_reply: tid=0 2024-03-28T17:54:34.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.156+0000 7fdcd59fd740 1 client.15888 shutdown 2024-03-28T17:54:34.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.156+0000 7fdcd59fd740 1 -- 172.21.15.45:0/2563441452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55583080e0f0 msgr2=0x555830816c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:54:34.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.156+0000 7fdcd59fd740 1 --2- 172.21.15.45:0/2563441452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55583080e0f0 0x555830816c50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc098200 tx=0x7fdcbc099d40 comp rx=0 tx=0).stop 2024-03-28T17:54:34.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.156+0000 7fdcd59fd740 1 -- 172.21.15.45:0/2563441452 shutdown_connections 2024-03-28T17:54:34.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.156+0000 7fdcd59fd740 1 --2- 172.21.15.45:0/2563441452 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555830827890 0x555830829d00 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.156+0000 7fdcd59fd740 1 --2- 172.21.15.45:0/2563441452 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558308144f0 0x555830814970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.156+0000 7fdcd59fd740 1 --2- 172.21.15.45:0/2563441452 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55583080edd0 0x555830813fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.156+0000 7fdcd59fd740 1 --2- 172.21.15.45:0/2563441452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55583080e0f0 0x555830816c50 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.156+0000 7fdcba000640 0 client.15888 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:54:34.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.156+0000 7fdcd59fd740 1 -- 172.21.15.45:0/2563441452 >> 172.21.15.45:0/2563441452 conn(0x5558307d5200 msgr2=0x55583080a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:54:34.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.156+0000 7fdcd59fd740 1 -- 172.21.15.45:0/2563441452 shutdown_connections 2024-03-28T17:54:34.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.156+0000 7fdcd59fd740 1 -- 172.21.15.45:0/2563441452 wait complete. 2024-03-28T17:54:34.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.156+0000 7fdcd59fd740 20 client.15888 trim_cache size 0 max 16384 2024-03-28T17:54:34.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.174+0000 7fdcd59fd740 1 Processor -- start 2024-03-28T17:54:34.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.174+0000 7fdcd59fd740 1 -- start start 2024-03-28T17:54:34.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.174+0000 7fdcd59fd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55583080e0f0 0x5558308167f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:34.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.174+0000 7fdcd59fd740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55583080edd0 0x555830822b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:34.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.174+0000 7fdcd59fd740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5558308144f0 0x555830823080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:34.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.174+0000 7fdcd59fd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc1b07a0 con 0x55583080e0f0 2024-03-28T17:54:34.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.174+0000 7fdcd59fd740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc1b0e00 con 0x55583080edd0 2024-03-28T17:54:34.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.174+0000 7fdcd59fd740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc1b1460 con 0x5558308144f0 2024-03-28T17:54:34.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.175+0000 7fdcc3e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55583080edd0 0x555830822b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:34.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.175+0000 7fdccbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55583080e0f0 0x5558308167f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:34.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.175+0000 7fdcc3e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55583080edd0 0x555830822b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:56224/0 (socket says 172.21.15.45:56224) 2024-03-28T17:54:34.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.175+0000 7fdcc3e00640 1 -- 172.21.15.45:0/3952245991 learned_addr learned my addr 172.21.15.45:0/3952245991 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:54:34.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.175+0000 7fdccbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55583080e0f0 0x5558308167f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51126/0 (socket says 172.21.15.45:51126) 2024-03-28T17:54:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.175+0000 7fdcd1000640 1 --2- 172.21.15.45:0/3952245991 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5558308144f0 0x555830823080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.175+0000 7fdcc3e00640 1 -- 172.21.15.45:0/3952245991 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5558308144f0 msgr2=0x555830823080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:54:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.175+0000 7fdcc3e00640 1 --2- 172.21.15.45:0/3952245991 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5558308144f0 0x555830823080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.175+0000 7fdcc3e00640 1 -- 172.21.15.45:0/3952245991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55583080e0f0 msgr2=0x5558308167f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:54:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.175+0000 7fdcc3e00640 1 --2- 172.21.15.45:0/3952245991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55583080e0f0 0x5558308167f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.175+0000 7fdcc3e00640 1 -- 172.21.15.45:0/3952245991 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558304460f0 con 0x55583080edd0 2024-03-28T17:54:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.175+0000 7fdccbe00640 1 --2- 172.21.15.45:0/3952245991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55583080e0f0 0x5558308167f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:54:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.175+0000 7fdcc3e00640 1 --2- 172.21.15.45:0/3952245991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55583080edd0 0x555830822b40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4036040 tx=0x7fdcb402bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=accdd98e49d74b7d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:54:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.176+0000 7fdcc0c00640 1 -- 172.21.15.45:0/3952245991 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4045040 con 0x55583080edd0 2024-03-28T17:54:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.176+0000 7fdcc0c00640 1 -- 172.21.15.45:0/3952245991 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdcb402c880 con 0x55583080edd0 2024-03-28T17:54:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.176+0000 7fdcc0c00640 1 -- 172.21.15.45:0/3952245991 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4004ee0 con 0x55583080edd0 2024-03-28T17:54:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.176+0000 7fdcd59fd740 1 -- 172.21.15.45:0/3952245991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55583080edd0 msgr2=0x555830822b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:54:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.176+0000 7fdcd59fd740 1 --2- 172.21.15.45:0/3952245991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55583080edd0 0x555830822b40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4036040 tx=0x7fdcb402bc30 comp rx=0 tx=0).stop 2024-03-28T17:54:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.176+0000 7fdcd59fd740 1 -- 172.21.15.45:0/3952245991 shutdown_connections 2024-03-28T17:54:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.176+0000 7fdcd59fd740 1 --2- 172.21.15.45:0/3952245991 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5558308144f0 0x555830823080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.176+0000 7fdcd59fd740 1 --2- 172.21.15.45:0/3952245991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55583080edd0 0x555830822b40 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.176+0000 7fdcd59fd740 1 --2- 172.21.15.45:0/3952245991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55583080e0f0 0x5558308167f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.176+0000 7fdcd59fd740 1 -- 172.21.15.45:0/3952245991 >> 172.21.15.45:0/3952245991 conn(0x5558307d5200 msgr2=0x55583080f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:54:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.176+0000 7fdcd59fd740 1 -- 172.21.15.45:0/3952245991 shutdown_connections 2024-03-28T17:54:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.176+0000 7fdcd59fd740 1 -- 172.21.15.45:0/3952245991 wait complete. 2024-03-28T17:54:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.177+0000 7fdcd59fd740 1 -- start start 2024-03-28T17:54:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.177+0000 7fdcd59fd740 1 Processor -- start 2024-03-28T17:54:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.177+0000 7fdcd59fd740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55583080e0f0 0x555830811cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.177+0000 7fdcd59fd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55583080edd0 0x55583081c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.177+0000 7fdcd59fd740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558308144f0 0x55583081cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.177+0000 7fdcd59fd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc1add40 con 0x55583080edd0 2024-03-28T17:54:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.177+0000 7fdcd59fd740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc1aefe0 con 0x5558308144f0 2024-03-28T17:54:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.177+0000 7fdcd59fd740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc1ae340 con 0x55583080e0f0 2024-03-28T17:54:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.178+0000 7fdcd59fd740 10 client.0 _ll_register_callbacks cb 0x7fdcbc0d5220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:54:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.178+0000 7fdcc3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55583080edd0 0x55583081c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.178+0000 7fdcd1000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558308144f0 0x55583081cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.178+0000 7fdcc3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55583080edd0 0x55583081c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51132/0 (socket says 172.21.15.45:51132) 2024-03-28T17:54:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.178+0000 7fdcc3e00640 1 -- 172.21.15.45:0/4058639863 learned_addr learned my addr 172.21.15.45:0/4058639863 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:54:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.178+0000 7fdccbe00640 1 --2- 172.21.15.45:0/4058639863 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55583080e0f0 0x555830811cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:34.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.178+0000 7fdcc3e00640 1 -- 172.21.15.45:0/4058639863 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55583080e0f0 msgr2=0x555830811cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:54:34.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.178+0000 7fdcc3e00640 1 --2- 172.21.15.45:0/4058639863 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55583080e0f0 0x555830811cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.178+0000 7fdcc3e00640 1 -- 172.21.15.45:0/4058639863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558308144f0 msgr2=0x55583081cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:54:34.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.178+0000 7fdcc3e00640 1 --2- 172.21.15.45:0/4058639863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558308144f0 0x55583081cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.178+0000 7fdcc3e00640 1 -- 172.21.15.45:0/4058639863 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55583081d370 con 0x55583080edd0 2024-03-28T17:54:34.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.178+0000 7fdcd1000640 1 --2- 172.21.15.45:0/4058639863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558308144f0 0x55583081cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:54:34.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.179+0000 7fdcc3e00640 1 --2- 172.21.15.45:0/4058639863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55583080edd0 0x55583081c650 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x5558308150c0 tx=0x7fdcb4005890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c35a96a4d34911ec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:54:34.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.179+0000 7fdcba000640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-28T17:54:34.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.179+0000 7fdcba000640 1 -- 172.21.15.45:0/4058639863 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4045070 con 0x55583080edd0 2024-03-28T17:54:34.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.179+0000 7fdcd59fd740 1 -- 172.21.15.45:0/4058639863 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdcb402c880 con 0x55583080edd0 2024-03-28T17:54:34.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.179+0000 7fdcba000640 1 -- 172.21.15.45:0/4058639863 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9716+0+0 (secure 0 0 0) 0x7fdcb4004a20 con 0x55583080edd0 2024-03-28T17:54:34.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.179+0000 7fdcba000640 1 -- 172.21.15.45:0/4058639863 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=116) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdcb4066050 con 0x55583080edd0 2024-03-28T17:54:34.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.179+0000 7fdcd59fd740 10 client.15894 fetch_fsmap learned FSMap version 116 2024-03-28T17:54:34.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.179+0000 7fdcd59fd740 1 -- 172.21.15.45:0/4058639863 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=116,monmap=4+}) v3 -- 0x5558308841c0 con 0x55583080edd0 2024-03-28T17:54:34.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.179+0000 7fdcba000640 1 -- 172.21.15.45:0/4058639863 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdcb402c880 con 0x55583080edd0 2024-03-28T17:54:34.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.180+0000 7fdcba000640 1 -- 172.21.15.45:0/4058639863 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 116) v1 ==== 6768+0+0 (secure 0 0 0) 0x7fdcb40054e0 con 0x55583080edd0 2024-03-28T17:54:34.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.180+0000 7fdcd59fd740 10 client.15894 fetch_fsmap finished waiting for FSMap version 116 2024-03-28T17:54:34.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.180+0000 7fdcd59fd740 10 client.15894 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:54:34.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.180+0000 7fdcd59fd740 20 client.15894 populate_metadata read hostname 'smithi045' 2024-03-28T17:54:34.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.180+0000 7fdcd59fd740 1 --2- 172.21.15.45:0/4058639863 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555830884570 0x555830886990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:54:34.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.180+0000 7fdcd59fd740 4 client.15894 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:54:34.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.180+0000 7fdcd59fd740 1 -- 172.21.15.45:0/4058639863 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555830779ae0 con 0x555830884570 2024-03-28T17:54:34.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.180+0000 7fdccbe00640 1 --2- 172.21.15.45:0/4058639863 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555830884570 0x555830886990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:54:34.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.181+0000 7fdcba000640 10 client.15894 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:54:34.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.181+0000 7fdccbe00640 1 --2- 172.21.15.45:0/4058639863 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555830884570 0x555830886990 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e404167200c45b2 server_cookie=c9fa79148e5fdd18 in_seq=0 out_seq=0 2024-03-28T17:54:34.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.181+0000 7fdcba000640 1 -- 172.21.15.45:0/4058639863 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x555830779ae0 con 0x555830884570 2024-03-28T17:54:34.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.181+0000 7fdcba000640 10 client.15894 handle_command_reply: tid=0 2024-03-28T17:54:34.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.181+0000 7fdcd59fd740 1 client.15894 shutdown 2024-03-28T17:54:34.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.181+0000 7fdcd59fd740 1 -- 172.21.15.45:0/4058639863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55583080edd0 msgr2=0x55583081c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:54:34.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.181+0000 7fdcd59fd740 1 --2- 172.21.15.45:0/4058639863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55583080edd0 0x55583081c650 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x5558308150c0 tx=0x7fdcb4005890 comp rx=0 tx=0).stop 2024-03-28T17:54:34.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.181+0000 7fdcd59fd740 1 -- 172.21.15.45:0/4058639863 shutdown_connections 2024-03-28T17:54:34.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.181+0000 7fdcd59fd740 1 --2- 172.21.15.45:0/4058639863 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555830884570 0x555830886990 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.181+0000 7fdcd59fd740 1 --2- 172.21.15.45:0/4058639863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558308144f0 0x55583081cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.181+0000 7fdcd59fd740 1 --2- 172.21.15.45:0/4058639863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55583080edd0 0x55583081c650 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.181+0000 7fdcd59fd740 1 --2- 172.21.15.45:0/4058639863 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55583080e0f0 0x555830811cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.181+0000 7fdcba000640 0 client.15894 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:54:34.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.181+0000 7fdcd59fd740 1 -- 172.21.15.45:0/4058639863 >> 172.21.15.45:0/4058639863 conn(0x5558307d5200 msgr2=0x555830807330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:54:34.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.181+0000 7fdcd59fd740 1 -- 172.21.15.45:0/4058639863 shutdown_connections 2024-03-28T17:54:34.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.181+0000 7fdcd59fd740 1 -- 172.21.15.45:0/4058639863 wait complete. 2024-03-28T17:54:34.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.181+0000 7fdcd59fd740 20 client.15894 trim_cache size 0 max 16384 2024-03-28T17:54:34.186 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:54:34.186 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T17:54:34.187 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T17:54:34.187 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T17:54:34.187 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T17:54:34.187 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T17:54:34.187 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T17:54:34.187 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:54:34.187 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:54:34.187 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:54:34.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.182+0000 7fdcd1a00640 1 -- 172.21.15.45:0/189004489 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fdcb00a2970 msgr2=0x7fdcb00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:54:34.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.182+0000 7fdcd1a00640 1 --2- 172.21.15.45:0/189004489 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fdcb00a2970 0x7fdcb00a4e30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4005fd0 tx=0x7fdcc4009800 comp rx=0 tx=0).stop 2024-03-28T17:54:34.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.182+0000 7fdcd1a00640 1 -- 172.21.15.45:0/189004489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdccc103700 msgr2=0x7fdccc19a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:54:34.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.182+0000 7fdcd1a00640 1 --2- 172.21.15.45:0/189004489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdccc103700 0x7fdccc19a010 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc00e9f0 tx=0x7fdcbc00eeb0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.182+0000 7fdcd1a00640 1 -- 172.21.15.45:0/189004489 shutdown_connections 2024-03-28T17:54:34.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.182+0000 7fdcd1a00640 1 --2- 172.21.15.45:0/189004489 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fdcb00a2970 0x7fdcb00a4e30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.182+0000 7fdcd1a00640 1 --2- 172.21.15.45:0/189004489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdccc109710 0x7fdccc19e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.182+0000 7fdcd1a00640 1 --2- 172.21.15.45:0/189004489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdccc1040c0 0x7fdccc19a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.182+0000 7fdcd1a00640 1 --2- 172.21.15.45:0/189004489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdccc103700 0x7fdccc19a010 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:54:34.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.182+0000 7fdcd1a00640 1 -- 172.21.15.45:0/189004489 >> 172.21.15.45:0/189004489 conn(0x7fdccc0ff1b0 msgr2=0x7fdccc0ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:54:34.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.182+0000 7fdcd1a00640 1 -- 172.21.15.45:0/189004489 shutdown_connections 2024-03-28T17:54:34.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:54:34.182+0000 7fdcd1a00640 1 -- 172.21.15.45:0/189004489 wait complete. 2024-03-28T17:54:34.199 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T17:54:34.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:34 smithi071 ceph-mon[36597]: pgmap v184: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:54:34.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:34 smithi151 ceph-mon[36481]: pgmap v184: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:54:34.866 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 127 sec latency 242763.273 ms 2024-03-28T17:54:34.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:34 smithi045 ceph-mon[31773]: pgmap v184: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:54:35.324 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:34 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 3. 2024-03-28T17:54:35.324 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:34 smithi045 systemd[1]: Stopped Ceph mds.d for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:54:35.325 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:34 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Consumed 1.615s CPU time. 2024-03-28T17:54:35.325 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:34 smithi045 systemd[1]: Starting Ceph mds.d for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:54:35.666 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:35 smithi045 podman[90861]: 2024-03-28 17:54:35.231362739 +0000 UTC m=+0.040158599 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:54:35.666 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:35 smithi045 podman[90861]: 2024-03-28 17:54:35.370450668 +0000 UTC m=+0.179246544 container create 269b6d49021166b86caee0b668d0814f5710dde7c56b7a85ee8bd0a4fc373f5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T17:54:35.866 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 128 sec latency 243763.416 ms 2024-03-28T17:54:36.166 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:35 smithi045 podman[90861]: 2024-03-28 17:54:35.683103809 +0000 UTC m=+0.491899691 container init 269b6d49021166b86caee0b668d0814f5710dde7c56b7a85ee8bd0a4fc373f5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T17:54:36.166 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:35 smithi045 podman[90861]: 2024-03-28 17:54:35.693306119 +0000 UTC m=+0.502102002 container start 269b6d49021166b86caee0b668d0814f5710dde7c56b7a85ee8bd0a4fc373f5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-28T17:54:36.166 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:35 smithi045 ceph-mds[90879]: -- 172.21.15.45:0/3995759300 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bfebf21a40 con 0x55bfebef5000 2024-03-28T17:54:36.166 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:35 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: starting mds.d at 2024-03-28T17:54:36.166 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:35 smithi045 bash[90861]: 269b6d49021166b86caee0b668d0814f5710dde7c56b7a85ee8bd0a4fc373f5e 2024-03-28T17:54:36.167 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:35 smithi045 systemd[1]: Started Ceph mds.d for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:54:36.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:36 smithi071 ceph-mon[36597]: pgmap v185: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:54:36.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:36 smithi071 ceph-mon[36597]: daemon mds.d restarted 2024-03-28T17:54:36.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:36 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:54:36.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:36 smithi071 ceph-mon[36597]: osdmap e103: 12 total, 12 up, 12 in 2024-03-28T17:54:36.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:36 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2283155,v1:172.21.15.45:6837/2283155] up:boot 2024-03-28T17:54:36.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:36 smithi071 ceph-mon[36597]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-03-28T17:54:36.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:36 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:54:36.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:36 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T17:54:36.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:36 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:36 smithi151 ceph-mon[36481]: pgmap v185: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:54:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:36 smithi151 ceph-mon[36481]: daemon mds.d restarted 2024-03-28T17:54:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:36 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:54:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:36 smithi151 ceph-mon[36481]: osdmap e103: 12 total, 12 up, 12 in 2024-03-28T17:54:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:36 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2283155,v1:172.21.15.45:6837/2283155] up:boot 2024-03-28T17:54:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:36 smithi151 ceph-mon[36481]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-03-28T17:54:36.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:36 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:54:36.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:36 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T17:54:36.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:36 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:36.866 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 129 sec latency 244763.576 ms 2024-03-28T17:54:36.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:36 smithi045 ceph-mon[31773]: pgmap v185: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:54:36.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:36 smithi045 ceph-mon[31773]: daemon mds.d restarted 2024-03-28T17:54:36.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:36 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:54:36.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:36 smithi045 ceph-mon[31773]: osdmap e103: 12 total, 12 up, 12 in 2024-03-28T17:54:36.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:36 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2283155,v1:172.21.15.45:6837/2283155] up:boot 2024-03-28T17:54:36.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:36 smithi045 ceph-mon[31773]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-03-28T17:54:36.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:36 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:54:36.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:36 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T17:54:36.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:36 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:37.867 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 130 sec latency 245763.708 ms 2024-03-28T17:54:38.721 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:38 smithi045 ceph-mon[31773]: pgmap v187: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:54:38.721 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:38 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/2172148780,v1:172.21.15.71:6833/2172148780] up:resolve 2024-03-28T17:54:38.721 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:38 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:38.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:38 smithi071 ceph-mon[36597]: pgmap v187: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:54:38.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:38 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/2172148780,v1:172.21.15.71:6833/2172148780] up:resolve 2024-03-28T17:54:38.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:38 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:38.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:38 smithi151 ceph-mon[36481]: pgmap v187: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:54:38.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:38 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/2172148780,v1:172.21.15.71:6833/2172148780] up:resolve 2024-03-28T17:54:38.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:38 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:38.866 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 131 sec latency 246763.859 ms 2024-03-28T17:54:39.751 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:39 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/2172148780,v1:172.21.15.71:6833/2172148780] up:reconnect 2024-03-28T17:54:39.751 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:39 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:39.751 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:39 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899985 2024-03-28T17:54:39.751 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:39 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00899985 2024-03-28T17:54:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:39 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/2172148780,v1:172.21.15.71:6833/2172148780] up:reconnect 2024-03-28T17:54:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:39 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:39 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899985 2024-03-28T17:54:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:39 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00899985 2024-03-28T17:54:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:39 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/2172148780,v1:172.21.15.71:6833/2172148780] up:reconnect 2024-03-28T17:54:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:39 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:39 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899985 2024-03-28T17:54:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:39 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00899985 2024-03-28T17:54:39.867 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 132 sec latency 247763.990 ms 2024-03-28T17:54:40.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-mon[36597]: pgmap v188: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:54:40.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/2172148780,v1:172.21.15.71:6833/2172148780] up:rejoin 2024-03-28T17:54:40.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-mon[36597]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-03-28T17:54:40.815 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f439ab3d700 time 2024-03-28T17:54:40.486167+0000 2024-03-28T17:54:40.815 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:54:40.815 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:40.815 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f43a51674a1] 2024-03-28T17:54:40.815 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f43a5167667] 2024-03-28T17:54:40.815 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56344344a223] 2024-03-28T17:54:40.815 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5634432c45d6] 2024-03-28T17:54:40.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5634432c4da1] 2024-03-28T17:54:40.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5634432c72d5] 2024-03-28T17:54:40.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563443317233] 2024-03-28T17:54:40.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563443319197] 2024-03-28T17:54:40.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56344321811b] 2024-03-28T17:54:40.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5634432186bc] 2024-03-28T17:54:40.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5634432021ff] 2024-03-28T17:54:40.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f43a53f9488] 2024-03-28T17:54:40.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 13: (DispatchQueue::entry()+0x51f) [0x7f43a53f660f] 2024-03-28T17:54:40.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f43a54c9e81] 2024-03-28T17:54:40.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f43a3ee41ca] 2024-03-28T17:54:40.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 16: clone() 2024-03-28T17:54:40.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 2024-03-28T17:54:40.487+0000 7f439ab3d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f439ab3d700 time 2024-03-28T17:54:40.486167+0000 2024-03-28T17:54:40.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:54:40.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 2024-03-28T17:54:40.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:40.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f43a51674a1] 2024-03-28T17:54:40.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f43a5167667] 2024-03-28T17:54:40.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56344344a223] 2024-03-28T17:54:40.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5634432c45d6] 2024-03-28T17:54:40.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5634432c4da1] 2024-03-28T17:54:40.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5634432c72d5] 2024-03-28T17:54:40.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563443317233] 2024-03-28T17:54:40.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563443319197] 2024-03-28T17:54:40.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56344321811b] 2024-03-28T17:54:40.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5634432186bc] 2024-03-28T17:54:40.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5634432021ff] 2024-03-28T17:54:40.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f43a53f9488] 2024-03-28T17:54:40.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 13: (DispatchQueue::entry()+0x51f) [0x7f43a53f660f] 2024-03-28T17:54:40.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f43a54c9e81] 2024-03-28T17:54:40.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f43a3ee41ca] 2024-03-28T17:54:40.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 16: clone() 2024-03-28T17:54:40.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 2024-03-28T17:54:40.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: *** Caught signal (Aborted) ** 2024-03-28T17:54:40.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: in thread 7f439ab3d700 thread_name:ms_dispatch 2024-03-28T17:54:40.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:40.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f43a3eeed20] 2024-03-28T17:54:40.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 2: gsignal() 2024-03-28T17:54:40.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 3: abort() 2024-03-28T17:54:40.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f43a51674fb] 2024-03-28T17:54:40.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f43a5167667] 2024-03-28T17:54:40.819 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56344344a223] 2024-03-28T17:54:40.819 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5634432c45d6] 2024-03-28T17:54:40.819 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5634432c4da1] 2024-03-28T17:54:40.819 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5634432c72d5] 2024-03-28T17:54:40.819 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563443317233] 2024-03-28T17:54:40.819 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563443319197] 2024-03-28T17:54:40.819 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56344321811b] 2024-03-28T17:54:40.819 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5634432186bc] 2024-03-28T17:54:40.819 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5634432021ff] 2024-03-28T17:54:40.819 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f43a53f9488] 2024-03-28T17:54:40.819 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 16: (DispatchQueue::entry()+0x51f) [0x7f43a53f660f] 2024-03-28T17:54:40.819 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f43a54c9e81] 2024-03-28T17:54:40.819 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f43a3ee41ca] 2024-03-28T17:54:40.819 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 19: clone() 2024-03-28T17:54:40.819 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 2024-03-28T17:54:40.489+0000 7f439ab3d700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:54:40.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: in thread 7f439ab3d700 thread_name:ms_dispatch 2024-03-28T17:54:40.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 2024-03-28T17:54:40.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:40.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f43a3eeed20] 2024-03-28T17:54:40.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 2: gsignal() 2024-03-28T17:54:40.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 3: abort() 2024-03-28T17:54:40.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f43a51674fb] 2024-03-28T17:54:40.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f43a5167667] 2024-03-28T17:54:40.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56344344a223] 2024-03-28T17:54:40.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5634432c45d6] 2024-03-28T17:54:40.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5634432c4da1] 2024-03-28T17:54:40.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5634432c72d5] 2024-03-28T17:54:40.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563443317233] 2024-03-28T17:54:40.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563443319197] 2024-03-28T17:54:40.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56344321811b] 2024-03-28T17:54:40.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5634432186bc] 2024-03-28T17:54:40.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5634432021ff] 2024-03-28T17:54:40.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f43a53f9488] 2024-03-28T17:54:40.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 16: (DispatchQueue::entry()+0x51f) [0x7f43a53f660f] 2024-03-28T17:54:40.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f43a54c9e81] 2024-03-28T17:54:40.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f43a3ee41ca] 2024-03-28T17:54:40.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 19: clone() 2024-03-28T17:54:40.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:54:40.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 2024-03-28T17:54:40.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: -1> 2024-03-28T17:54:40.487+0000 7f439ab3d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f439ab3d700 time 2024-03-28T17:54:40.486167+0000 2024-03-28T17:54:40.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:54:40.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 2024-03-28T17:54:40.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:40.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f43a51674a1] 2024-03-28T17:54:40.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f43a5167667] 2024-03-28T17:54:40.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56344344a223] 2024-03-28T17:54:40.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5634432c45d6] 2024-03-28T17:54:40.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5634432c4da1] 2024-03-28T17:54:40.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5634432c72d5] 2024-03-28T17:54:40.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563443317233] 2024-03-28T17:54:40.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563443319197] 2024-03-28T17:54:40.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56344321811b] 2024-03-28T17:54:40.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5634432186bc] 2024-03-28T17:54:40.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5634432021ff] 2024-03-28T17:54:40.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f43a53f9488] 2024-03-28T17:54:40.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 13: (DispatchQueue::entry()+0x51f) [0x7f43a53f660f] 2024-03-28T17:54:40.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f43a54c9e81] 2024-03-28T17:54:40.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f43a3ee41ca] 2024-03-28T17:54:40.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 16: clone() 2024-03-28T17:54:40.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 2024-03-28T17:54:40.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 0> 2024-03-28T17:54:40.489+0000 7f439ab3d700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:54:40.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: in thread 7f439ab3d700 thread_name:ms_dispatch 2024-03-28T17:54:40.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 2024-03-28T17:54:40.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:40.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f43a3eeed20] 2024-03-28T17:54:40.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 2: gsignal() 2024-03-28T17:54:40.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 3: abort() 2024-03-28T17:54:40.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f43a51674fb] 2024-03-28T17:54:40.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f43a5167667] 2024-03-28T17:54:40.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56344344a223] 2024-03-28T17:54:40.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5634432c45d6] 2024-03-28T17:54:40.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5634432c4da1] 2024-03-28T17:54:40.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5634432c72d5] 2024-03-28T17:54:40.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563443317233] 2024-03-28T17:54:40.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563443319197] 2024-03-28T17:54:40.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56344321811b] 2024-03-28T17:54:40.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5634432186bc] 2024-03-28T17:54:40.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5634432021ff] 2024-03-28T17:54:40.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f43a53f9488] 2024-03-28T17:54:40.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 16: (DispatchQueue::entry()+0x51f) [0x7f43a53f660f] 2024-03-28T17:54:40.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f43a54c9e81] 2024-03-28T17:54:40.825 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f43a3ee41ca] 2024-03-28T17:54:40.825 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 19: clone() 2024-03-28T17:54:40.825 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:54:40.825 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 2024-03-28T17:54:40.825 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: -1> 2024-03-28T17:54:40.487+0000 7f439ab3d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f439ab3d700 time 2024-03-28T17:54:40.486167+0000 2024-03-28T17:54:40.825 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:54:40.825 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 2024-03-28T17:54:40.825 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:40.825 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f43a51674a1] 2024-03-28T17:54:40.825 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f43a5167667] 2024-03-28T17:54:40.825 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56344344a223] 2024-03-28T17:54:40.825 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5634432c45d6] 2024-03-28T17:54:40.825 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5634432c4da1] 2024-03-28T17:54:40.826 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5634432c72d5] 2024-03-28T17:54:40.826 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563443317233] 2024-03-28T17:54:40.826 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563443319197] 2024-03-28T17:54:40.826 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56344321811b] 2024-03-28T17:54:40.826 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5634432186bc] 2024-03-28T17:54:40.826 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5634432021ff] 2024-03-28T17:54:40.826 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f43a53f9488] 2024-03-28T17:54:40.826 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 13: (DispatchQueue::entry()+0x51f) [0x7f43a53f660f] 2024-03-28T17:54:40.826 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f43a54c9e81] 2024-03-28T17:54:40.826 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f43a3ee41ca] 2024-03-28T17:54:40.826 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 16: clone() 2024-03-28T17:54:40.826 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 2024-03-28T17:54:40.827 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 0> 2024-03-28T17:54:40.489+0000 7f439ab3d700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:54:40.827 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: in thread 7f439ab3d700 thread_name:ms_dispatch 2024-03-28T17:54:40.827 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 2024-03-28T17:54:40.827 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:40.827 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f43a3eeed20] 2024-03-28T17:54:40.827 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 2: gsignal() 2024-03-28T17:54:40.827 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 3: abort() 2024-03-28T17:54:40.827 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f43a51674fb] 2024-03-28T17:54:40.827 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f43a5167667] 2024-03-28T17:54:40.827 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56344344a223] 2024-03-28T17:54:40.827 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5634432c45d6] 2024-03-28T17:54:40.827 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5634432c4da1] 2024-03-28T17:54:40.827 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5634432c72d5] 2024-03-28T17:54:40.827 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563443317233] 2024-03-28T17:54:40.827 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563443319197] 2024-03-28T17:54:40.828 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56344321811b] 2024-03-28T17:54:40.828 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5634432186bc] 2024-03-28T17:54:40.828 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5634432021ff] 2024-03-28T17:54:40.828 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f43a53f9488] 2024-03-28T17:54:40.828 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 16: (DispatchQueue::entry()+0x51f) [0x7f43a53f660f] 2024-03-28T17:54:40.828 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f43a54c9e81] 2024-03-28T17:54:40.828 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f43a3ee41ca] 2024-03-28T17:54:40.828 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 19: clone() 2024-03-28T17:54:40.828 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:54:40.828 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71205]: 2024-03-28T17:54:40.828 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:40 smithi071 podman[71257]: 2024-03-28 17:54:40.546453671 +0000 UTC m=+0.026297047 container died 763a28229b07bff8b7b6cccdd739f686ae4bb1890b818ac111bd61f4cd640e7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T17:54:40.828 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:40 smithi151 ceph-mon[36481]: pgmap v188: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:54:40.829 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:40 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/2172148780,v1:172.21.15.71:6833/2172148780] up:rejoin 2024-03-28T17:54:40.829 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:40 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:40.829 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:40 smithi151 ceph-mon[36481]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-03-28T17:54:40.867 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 133 sec latency 248764.121 ms 2024-03-28T17:54:40.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:40 smithi045 ceph-mon[31773]: pgmap v188: 97 pgs: 97 active+clean; 27 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:54:40.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:40 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/2172148780,v1:172.21.15.71:6833/2172148780] up:rejoin 2024-03-28T17:54:40.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:40 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:40.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:40 smithi045 ceph-mon[31773]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-03-28T17:54:41.539 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:41 smithi071 podman[71257]: 2024-03-28 17:54:41.288679038 +0000 UTC m=+0.768522404 container remove 763a28229b07bff8b7b6cccdd739f686ae4bb1890b818ac111bd61f4cd640e7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-28T17:54:41.539 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:41 smithi071 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:54:41.539 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:41 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:54:41.539 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:41 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:54:41.539 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:41 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/2172148780,v1:172.21.15.71:6833/2172148780] up:active 2024-03-28T17:54:41.539 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:41 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:41.539 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:41 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:54:41.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:41 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:54:41.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:41 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:54:41.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:41 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/2172148780,v1:172.21.15.71:6833/2172148780] up:active 2024-03-28T17:54:41.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:41 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:41.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:41 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:54:41.810 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:41 smithi071 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.k.service: Failed with result 'exit-code'. 2024-03-28T17:54:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:41 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:54:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:41 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:54:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:41 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/2172148780,v1:172.21.15.71:6833/2172148780] up:active 2024-03-28T17:54:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:41 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:41 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:54:41.867 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 134 sec latency 249764.281 ms 2024-03-28T17:54:42.745 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:42 smithi045 ceph-mon[31773]: pgmap v189: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:54:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:42 smithi071 ceph-mon[36597]: pgmap v189: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:54:42.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:42 smithi151 ceph-mon[36481]: pgmap v189: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:54:42.867 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 135 sec latency 250764.411 ms 2024-03-28T17:54:43.867 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 136 sec latency 251764.578 ms 2024-03-28T17:54:44.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:44 smithi071 ceph-mon[36597]: pgmap v190: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:54:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:44 smithi151 ceph-mon[36481]: pgmap v190: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:54:44.868 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 137 sec latency 252764.732 ms 2024-03-28T17:54:44.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:44 smithi045 ceph-mon[31773]: pgmap v190: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:54:45.868 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 138 sec latency 253764.873 ms 2024-03-28T17:54:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:46 smithi071 ceph-mon[36597]: pgmap v191: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:54:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:46 smithi151 ceph-mon[36481]: pgmap v191: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:54:46.868 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 139 sec latency 254765.037 ms 2024-03-28T17:54:46.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:46 smithi045 ceph-mon[31773]: pgmap v191: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:54:47.868 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 140 sec latency 255765.185 ms 2024-03-28T17:54:48.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:48 smithi071 ceph-mon[36597]: pgmap v192: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 181 B/s wr, 9 op/s 2024-03-28T17:54:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:48 smithi151 ceph-mon[36481]: pgmap v192: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 181 B/s wr, 9 op/s 2024-03-28T17:54:48.868 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 141 sec latency 256765.297 ms 2024-03-28T17:54:48.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:48 smithi045 ceph-mon[31773]: pgmap v192: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 181 B/s wr, 9 op/s 2024-03-28T17:54:49.868 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 142 sec latency 257765.462 ms 2024-03-28T17:54:50.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:50 smithi071 ceph-mon[36597]: pgmap v193: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:54:50.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:50 smithi151 ceph-mon[36481]: pgmap v193: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:54:50.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:50 smithi045 ceph-mon[31773]: pgmap v193: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:54:50.869 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 143 sec latency 258765.599 ms 2024-03-28T17:54:51.813 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:51 smithi071 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.k.service: Scheduled restart job, restart counter is at 2. 2024-03-28T17:54:51.813 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:51 smithi071 systemd[1]: Stopped Ceph mds.k for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:54:51.813 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:51 smithi071 systemd[1]: Starting Ceph mds.k for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:54:51.813 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:51 smithi071 podman[71349]: 2024-03-28 17:54:51.70236763 +0000 UTC m=+0.018844923 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:54:51.813 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:51 smithi071 podman[71349]: 2024-03-28 17:54:51.813474158 +0000 UTC m=+0.129951442 container create 48b19cff6462edebc34d47d2bfafcd7ab1657ec01a12cf0e91b993c81b8b9100 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-28T17:54:51.869 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 144 sec latency 259765.772 ms 2024-03-28T17:54:52.082 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:52 smithi071 podman[71349]: 2024-03-28 17:54:52.058140214 +0000 UTC m=+0.374617504 container init 48b19cff6462edebc34d47d2bfafcd7ab1657ec01a12cf0e91b993c81b8b9100 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T17:54:52.082 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:52 smithi071 podman[71349]: 2024-03-28 17:54:52.060743106 +0000 UTC m=+0.377220399 container start 48b19cff6462edebc34d47d2bfafcd7ab1657ec01a12cf0e91b993c81b8b9100 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T17:54:52.516 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:52 smithi071 ceph-mds[71368]: -- 172.21.15.71:0/2668899050 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555a836e9a40 con 0x555a836bcc00 2024-03-28T17:54:52.516 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:52 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: starting mds.k at 2024-03-28T17:54:52.516 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:52 smithi071 bash[71349]: 48b19cff6462edebc34d47d2bfafcd7ab1657ec01a12cf0e91b993c81b8b9100 2024-03-28T17:54:52.516 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:54:52 smithi071 systemd[1]: Started Ceph mds.k for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:54:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:52 smithi071 ceph-mon[36597]: pgmap v194: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:54:52.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:52 smithi071 ceph-mon[36597]: daemon mds.k restarted 2024-03-28T17:54:52.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:52 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:54:52.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:52 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T17:54:52.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:52 smithi071 ceph-mon[36597]: osdmap e104: 12 total, 12 up, 12 in 2024-03-28T17:54:52.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:52 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/627131461,v1:172.21.15.71:6833/627131461] up:boot 2024-03-28T17:54:52.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:52 smithi071 ceph-mon[36597]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-03-28T17:54:52.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:52 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:54:52.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:52 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:52 smithi151 ceph-mon[36481]: pgmap v194: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:54:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:52 smithi151 ceph-mon[36481]: daemon mds.k restarted 2024-03-28T17:54:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:52 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:54:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:52 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T17:54:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:52 smithi151 ceph-mon[36481]: osdmap e104: 12 total, 12 up, 12 in 2024-03-28T17:54:52.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:52 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/627131461,v1:172.21.15.71:6833/627131461] up:boot 2024-03-28T17:54:52.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:52 smithi151 ceph-mon[36481]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-03-28T17:54:52.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:52 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:54:52.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:52 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:52.869 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 145 sec latency 260765.944 ms 2024-03-28T17:54:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:52 smithi045 ceph-mon[31773]: pgmap v194: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:54:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:52 smithi045 ceph-mon[31773]: daemon mds.k restarted 2024-03-28T17:54:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:52 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:54:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:52 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T17:54:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:52 smithi045 ceph-mon[31773]: osdmap e104: 12 total, 12 up, 12 in 2024-03-28T17:54:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:52 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/627131461,v1:172.21.15.71:6833/627131461] up:boot 2024-03-28T17:54:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:52 smithi045 ceph-mon[31773]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-03-28T17:54:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:52 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:54:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:52 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:53.869 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 146 sec latency 261766.044 ms 2024-03-28T17:54:54.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:54 smithi071 ceph-mon[36597]: pgmap v196: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 10 op/s 2024-03-28T17:54:54.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:54 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2283155,v1:172.21.15.45:6837/2283155] up:resolve 2024-03-28T17:54:54.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:54 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:54.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:54 smithi151 ceph-mon[36481]: pgmap v196: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 10 op/s 2024-03-28T17:54:54.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:54 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2283155,v1:172.21.15.45:6837/2283155] up:resolve 2024-03-28T17:54:54.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:54 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:54.869 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 147 sec latency 262766.209 ms 2024-03-28T17:54:54.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:54 smithi045 ceph-mon[31773]: pgmap v196: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 10 op/s 2024-03-28T17:54:54.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:54 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2283155,v1:172.21.15.45:6837/2283155] up:resolve 2024-03-28T17:54:54.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:54 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:55.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:55 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T17:54:55.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:55 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2283155,v1:172.21.15.45:6837/2283155] up:reconnect 2024-03-28T17:54:55.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:55 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:55.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:55 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T17:54:55.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:55 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T17:54:55.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:55 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2283155,v1:172.21.15.45:6837/2283155] up:reconnect 2024-03-28T17:54:55.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:55 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:55.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:55 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T17:54:55.869 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 148 sec latency 263766.346 ms 2024-03-28T17:54:55.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:55 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T17:54:55.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:55 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2283155,v1:172.21.15.45:6837/2283155] up:reconnect 2024-03-28T17:54:55.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:55 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:55.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:55 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T17:54:56.556 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4056400700 time 2024-03-28T17:54:56.268404+0000 2024-03-28T17:54:56.556 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:54:56.557 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:56.557 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f4061d634a1] 2024-03-28T17:54:56.557 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4061d63667] 2024-03-28T17:54:56.557 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bfeae32223] 2024-03-28T17:54:56.557 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bfeacac5d6] 2024-03-28T17:54:56.557 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bfeacacda1] 2024-03-28T17:54:56.557 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bfeacaf2d5] 2024-03-28T17:54:56.557 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bfeacff233] 2024-03-28T17:54:56.557 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bfead01197] 2024-03-28T17:54:56.557 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bfeac0011b] 2024-03-28T17:54:56.557 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bfeac006bc] 2024-03-28T17:54:56.557 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bfeabea1ff] 2024-03-28T17:54:56.557 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4061ff5488] 2024-03-28T17:54:56.558 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 13: (DispatchQueue::entry()+0x51f) [0x7f4061ff260f] 2024-03-28T17:54:56.558 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40620c5e81] 2024-03-28T17:54:56.558 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4060ae01ca] 2024-03-28T17:54:56.558 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 16: clone() 2024-03-28T17:54:56.559 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 2024-03-28T17:54:56.272+0000 7f4056400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4056400700 time 2024-03-28T17:54:56.268404+0000 2024-03-28T17:54:56.559 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:54:56.559 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 2024-03-28T17:54:56.559 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:56.559 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f4061d634a1] 2024-03-28T17:54:56.559 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4061d63667] 2024-03-28T17:54:56.559 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bfeae32223] 2024-03-28T17:54:56.559 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bfeacac5d6] 2024-03-28T17:54:56.559 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bfeacacda1] 2024-03-28T17:54:56.559 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bfeacaf2d5] 2024-03-28T17:54:56.560 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bfeacff233] 2024-03-28T17:54:56.560 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bfead01197] 2024-03-28T17:54:56.560 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bfeac0011b] 2024-03-28T17:54:56.560 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bfeac006bc] 2024-03-28T17:54:56.560 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bfeabea1ff] 2024-03-28T17:54:56.560 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4061ff5488] 2024-03-28T17:54:56.560 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 13: (DispatchQueue::entry()+0x51f) [0x7f4061ff260f] 2024-03-28T17:54:56.560 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40620c5e81] 2024-03-28T17:54:56.560 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4060ae01ca] 2024-03-28T17:54:56.560 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 16: clone() 2024-03-28T17:54:56.560 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 2024-03-28T17:54:56.560 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: *** Caught signal (Aborted) ** 2024-03-28T17:54:56.560 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: in thread 7f4056400700 thread_name:ms_dispatch 2024-03-28T17:54:56.561 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:56.561 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4060aead20] 2024-03-28T17:54:56.561 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 2: gsignal() 2024-03-28T17:54:56.561 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 3: abort() 2024-03-28T17:54:56.561 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f4061d634fb] 2024-03-28T17:54:56.561 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4061d63667] 2024-03-28T17:54:56.561 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bfeae32223] 2024-03-28T17:54:56.561 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bfeacac5d6] 2024-03-28T17:54:56.561 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bfeacacda1] 2024-03-28T17:54:56.561 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bfeacaf2d5] 2024-03-28T17:54:56.561 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bfeacff233] 2024-03-28T17:54:56.562 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bfead01197] 2024-03-28T17:54:56.562 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bfeac0011b] 2024-03-28T17:54:56.562 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bfeac006bc] 2024-03-28T17:54:56.562 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bfeabea1ff] 2024-03-28T17:54:56.562 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4061ff5488] 2024-03-28T17:54:56.562 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 16: (DispatchQueue::entry()+0x51f) [0x7f4061ff260f] 2024-03-28T17:54:56.562 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40620c5e81] 2024-03-28T17:54:56.562 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4060ae01ca] 2024-03-28T17:54:56.562 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 19: clone() 2024-03-28T17:54:56.562 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 2024-03-28T17:54:56.277+0000 7f4056400700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:54:56.562 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: in thread 7f4056400700 thread_name:ms_dispatch 2024-03-28T17:54:56.562 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 2024-03-28T17:54:56.563 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:56.563 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4060aead20] 2024-03-28T17:54:56.563 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 2: gsignal() 2024-03-28T17:54:56.563 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 3: abort() 2024-03-28T17:54:56.563 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f4061d634fb] 2024-03-28T17:54:56.563 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4061d63667] 2024-03-28T17:54:56.563 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bfeae32223] 2024-03-28T17:54:56.563 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bfeacac5d6] 2024-03-28T17:54:56.563 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bfeacacda1] 2024-03-28T17:54:56.563 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bfeacaf2d5] 2024-03-28T17:54:56.563 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bfeacff233] 2024-03-28T17:54:56.564 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bfead01197] 2024-03-28T17:54:56.564 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bfeac0011b] 2024-03-28T17:54:56.564 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bfeac006bc] 2024-03-28T17:54:56.564 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bfeabea1ff] 2024-03-28T17:54:56.564 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4061ff5488] 2024-03-28T17:54:56.564 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 16: (DispatchQueue::entry()+0x51f) [0x7f4061ff260f] 2024-03-28T17:54:56.564 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40620c5e81] 2024-03-28T17:54:56.564 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4060ae01ca] 2024-03-28T17:54:56.564 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 19: clone() 2024-03-28T17:54:56.564 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:54:56.564 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 2024-03-28T17:54:56.564 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: -1> 2024-03-28T17:54:56.272+0000 7f4056400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4056400700 time 2024-03-28T17:54:56.268404+0000 2024-03-28T17:54:56.564 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:54:56.565 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 2024-03-28T17:54:56.565 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:56.565 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f4061d634a1] 2024-03-28T17:54:56.565 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4061d63667] 2024-03-28T17:54:56.565 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bfeae32223] 2024-03-28T17:54:56.566 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bfeacac5d6] 2024-03-28T17:54:56.566 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bfeacacda1] 2024-03-28T17:54:56.566 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bfeacaf2d5] 2024-03-28T17:54:56.566 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bfeacff233] 2024-03-28T17:54:56.566 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bfead01197] 2024-03-28T17:54:56.566 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bfeac0011b] 2024-03-28T17:54:56.566 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bfeac006bc] 2024-03-28T17:54:56.566 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bfeabea1ff] 2024-03-28T17:54:56.566 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4061ff5488] 2024-03-28T17:54:56.566 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 13: (DispatchQueue::entry()+0x51f) [0x7f4061ff260f] 2024-03-28T17:54:56.566 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40620c5e81] 2024-03-28T17:54:56.566 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4060ae01ca] 2024-03-28T17:54:56.566 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 16: clone() 2024-03-28T17:54:56.566 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 2024-03-28T17:54:56.567 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 0> 2024-03-28T17:54:56.277+0000 7f4056400700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:54:56.567 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: in thread 7f4056400700 thread_name:ms_dispatch 2024-03-28T17:54:56.567 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 2024-03-28T17:54:56.567 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:56.567 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4060aead20] 2024-03-28T17:54:56.567 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 2: gsignal() 2024-03-28T17:54:56.567 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 3: abort() 2024-03-28T17:54:56.567 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f4061d634fb] 2024-03-28T17:54:56.567 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4061d63667] 2024-03-28T17:54:56.567 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bfeae32223] 2024-03-28T17:54:56.567 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bfeacac5d6] 2024-03-28T17:54:56.568 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bfeacacda1] 2024-03-28T17:54:56.568 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bfeacaf2d5] 2024-03-28T17:54:56.568 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bfeacff233] 2024-03-28T17:54:56.568 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bfead01197] 2024-03-28T17:54:56.569 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bfeac0011b] 2024-03-28T17:54:56.569 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bfeac006bc] 2024-03-28T17:54:56.569 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bfeabea1ff] 2024-03-28T17:54:56.569 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4061ff5488] 2024-03-28T17:54:56.569 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 16: (DispatchQueue::entry()+0x51f) [0x7f4061ff260f] 2024-03-28T17:54:56.569 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40620c5e81] 2024-03-28T17:54:56.569 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4060ae01ca] 2024-03-28T17:54:56.569 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 19: clone() 2024-03-28T17:54:56.569 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:54:56.569 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 2024-03-28T17:54:56.569 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: -1> 2024-03-28T17:54:56.272+0000 7f4056400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4056400700 time 2024-03-28T17:54:56.268404+0000 2024-03-28T17:54:56.569 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:54:56.569 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 2024-03-28T17:54:56.569 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:56.570 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f4061d634a1] 2024-03-28T17:54:56.570 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4061d63667] 2024-03-28T17:54:56.570 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bfeae32223] 2024-03-28T17:54:56.570 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bfeacac5d6] 2024-03-28T17:54:56.571 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bfeacacda1] 2024-03-28T17:54:56.571 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bfeacaf2d5] 2024-03-28T17:54:56.571 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bfeacff233] 2024-03-28T17:54:56.571 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bfead01197] 2024-03-28T17:54:56.571 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bfeac0011b] 2024-03-28T17:54:56.571 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bfeac006bc] 2024-03-28T17:54:56.571 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bfeabea1ff] 2024-03-28T17:54:56.571 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4061ff5488] 2024-03-28T17:54:56.571 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 13: (DispatchQueue::entry()+0x51f) [0x7f4061ff260f] 2024-03-28T17:54:56.571 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40620c5e81] 2024-03-28T17:54:56.571 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4060ae01ca] 2024-03-28T17:54:56.571 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 16: clone() 2024-03-28T17:54:56.571 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 2024-03-28T17:54:56.572 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 0> 2024-03-28T17:54:56.277+0000 7f4056400700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:54:56.572 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: in thread 7f4056400700 thread_name:ms_dispatch 2024-03-28T17:54:56.572 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 2024-03-28T17:54:56.572 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:54:56.572 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4060aead20] 2024-03-28T17:54:56.572 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 2: gsignal() 2024-03-28T17:54:56.572 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 3: abort() 2024-03-28T17:54:56.572 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f4061d634fb] 2024-03-28T17:54:56.572 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4061d63667] 2024-03-28T17:54:56.572 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bfeae32223] 2024-03-28T17:54:56.572 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bfeacac5d6] 2024-03-28T17:54:56.572 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bfeacacda1] 2024-03-28T17:54:56.572 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bfeacaf2d5] 2024-03-28T17:54:56.573 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bfeacff233] 2024-03-28T17:54:56.573 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bfead01197] 2024-03-28T17:54:56.573 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bfeac0011b] 2024-03-28T17:54:56.573 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bfeac006bc] 2024-03-28T17:54:56.573 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bfeabea1ff] 2024-03-28T17:54:56.573 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4061ff5488] 2024-03-28T17:54:56.573 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 16: (DispatchQueue::entry()+0x51f) [0x7f4061ff260f] 2024-03-28T17:54:56.573 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40620c5e81] 2024-03-28T17:54:56.573 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4060ae01ca] 2024-03-28T17:54:56.573 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 19: clone() 2024-03-28T17:54:56.573 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:54:56.574 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[90875]: 2024-03-28T17:54:56.574 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:56 smithi045 podman[90925]: 2024-03-28 17:54:56.409520974 +0000 UTC m=+0.037527311 container died 269b6d49021166b86caee0b668d0814f5710dde7c56b7a85ee8bd0a4fc373f5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-28T17:54:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:56 smithi071 ceph-mon[36597]: pgmap v197: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:54:56.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:56 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2283155,v1:172.21.15.45:6837/2283155] up:rejoin 2024-03-28T17:54:56.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:56 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:56.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:56 smithi071 ceph-mon[36597]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-03-28T17:54:56.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:56 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:54:56.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:56 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:54:56.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:56 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2283155,v1:172.21.15.45:6837/2283155] up:active 2024-03-28T17:54:56.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:56 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:54:56.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-mon[31773]: pgmap v197: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:54:56.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2283155,v1:172.21.15.45:6837/2283155] up:rejoin 2024-03-28T17:54:56.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:56.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-mon[31773]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-03-28T17:54:56.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:54:56.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:54:56.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2283155,v1:172.21.15.45:6837/2283155] up:active 2024-03-28T17:54:56.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:56 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:54:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:56 smithi151 ceph-mon[36481]: pgmap v197: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:54:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:56 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2283155,v1:172.21.15.45:6837/2283155] up:rejoin 2024-03-28T17:54:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:56 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:54:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:56 smithi151 ceph-mon[36481]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-03-28T17:54:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:56 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:54:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:56 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:54:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:56 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2283155,v1:172.21.15.45:6837/2283155] up:active 2024-03-28T17:54:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:56 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:54:56.869 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 149 sec latency 264766.508 ms 2024-03-28T17:54:57.656 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:57 smithi045 podman[90925]: 2024-03-28 17:54:57.209185661 +0000 UTC m=+0.837192034 container remove 269b6d49021166b86caee0b668d0814f5710dde7c56b7a85ee8bd0a4fc373f5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319) 2024-03-28T17:54:57.656 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:57 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:54:57.870 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 150 sec latency 265766.655 ms 2024-03-28T17:54:58.165 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:57 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-28T17:54:58.166 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:54:57 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Consumed 1.612s CPU time. 2024-03-28T17:54:58.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:54:58 smithi045 ceph-mon[31773]: pgmap v198: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:54:58.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:54:58 smithi071 ceph-mon[36597]: pgmap v198: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:54:58.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:54:58 smithi151 ceph-mon[36481]: pgmap v198: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:54:58.870 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 151 sec latency 266767.271 ms 2024-03-28T17:54:59.870 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 152 sec latency 267767.401 ms 2024-03-28T17:55:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:00 smithi151 ceph-mon[36481]: pgmap v199: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:55:00.871 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 153 sec latency 268767.550 ms 2024-03-28T17:55:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:00 smithi045 ceph-mon[31773]: pgmap v199: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:55:01.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:00 smithi071 ceph-mon[36597]: pgmap v199: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:55:01.871 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 154 sec latency 269767.648 ms 2024-03-28T17:55:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:02 smithi151 ceph-mon[36481]: pgmap v200: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:55:02.871 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 155 sec latency 270767.798 ms 2024-03-28T17:55:02.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:02 smithi045 ceph-mon[31773]: pgmap v200: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:55:03.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:02 smithi071 ceph-mon[36597]: pgmap v200: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:55:03.871 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 156 sec latency 271767.907 ms 2024-03-28T17:55:04.201 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:55:04.201 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:55:04.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.443+0000 7fb675e00640 1 -- 172.21.15.45:0/52198442 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb670101530 msgr2=0x7fb6701019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:55:04.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.443+0000 7fb675e00640 1 --2- 172.21.15.45:0/52198442 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb670101530 0x7fb6701019b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb65c009930 tx=0x7fb65c02bc30 comp rx=0 tx=0).stop 2024-03-28T17:55:04.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.443+0000 7fb675e00640 1 -- 172.21.15.45:0/52198442 shutdown_connections 2024-03-28T17:55:04.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.443+0000 7fb675e00640 1 --2- 172.21.15.45:0/52198442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb670101ef0 0x7fb67010e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.443+0000 7fb675e00640 1 --2- 172.21.15.45:0/52198442 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb670101530 0x7fb6701019b0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.443+0000 7fb675e00640 1 --2- 172.21.15.45:0/52198442 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb6701075f0 0x7fb6701079d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.443+0000 7fb675e00640 1 -- 172.21.15.45:0/52198442 >> 172.21.15.45:0/52198442 conn(0x7fb6700fd000 msgr2=0x7fb6700ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:55:04.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.443+0000 7fb675e00640 1 -- 172.21.15.45:0/52198442 shutdown_connections 2024-03-28T17:55:04.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.443+0000 7fb675e00640 1 -- 172.21.15.45:0/52198442 wait complete. 2024-03-28T17:55:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.444+0000 7fb675e00640 1 Processor -- start 2024-03-28T17:55:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.444+0000 7fb675e00640 1 -- start start 2024-03-28T17:55:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.444+0000 7fb675e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb670101530 0x7fb670199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.444+0000 7fb675e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb670101ef0 0x7fb67019a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.445+0000 7fb675e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb6701075f0 0x7fb67019e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.445+0000 7fb675e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb670111900 con 0x7fb670101530 2024-03-28T17:55:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.445+0000 7fb675e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb670111780 con 0x7fb670101ef0 2024-03-28T17:55:04.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.445+0000 7fb675e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb670111a80 con 0x7fb6701075f0 2024-03-28T17:55:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.445+0000 7fb674a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb670101530 0x7fb670199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.445+0000 7fb674a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb670101530 0x7fb670199fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44152/0 (socket says 172.21.15.45:44152) 2024-03-28T17:55:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.445+0000 7fb66fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb670101ef0 0x7fb67019a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.445+0000 7fb674a00640 1 -- 172.21.15.45:0/1637713058 learned_addr learned my addr 172.21.15.45:0/1637713058 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:55:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.445+0000 7fb675400640 1 --2- 172.21.15.45:0/1637713058 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb6701075f0 0x7fb67019e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.445+0000 7fb66fe00640 1 -- 172.21.15.45:0/1637713058 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb6701075f0 msgr2=0x7fb67019e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:55:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.445+0000 7fb66fe00640 1 --2- 172.21.15.45:0/1637713058 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb6701075f0 0x7fb67019e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.445+0000 7fb66fe00640 1 -- 172.21.15.45:0/1637713058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb670101530 msgr2=0x7fb670199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:55:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.445+0000 7fb66fe00640 1 --2- 172.21.15.45:0/1637713058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb670101530 0x7fb670199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.445+0000 7fb66fe00640 1 -- 172.21.15.45:0/1637713058 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67019efb0 con 0x7fb670101ef0 2024-03-28T17:55:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.446+0000 7fb66fe00640 1 --2- 172.21.15.45:0/1637713058 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb670101ef0 0x7fb67019a500 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb65c036040 tx=0x7fb65c004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:55:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.446+0000 7fb66d600640 1 -- 172.21.15.45:0/1637713058 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65c043020 con 0x7fb670101ef0 2024-03-28T17:55:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.446+0000 7fb66d600640 1 -- 172.21.15.45:0/1637713058 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb65c0054e0 con 0x7fb670101ef0 2024-03-28T17:55:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.446+0000 7fb675e00640 1 -- 172.21.15.45:0/1637713058 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb67019f240 con 0x7fb670101ef0 2024-03-28T17:55:04.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.446+0000 7fb66d600640 1 -- 172.21.15.45:0/1637713058 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65c003b80 con 0x7fb670101ef0 2024-03-28T17:55:04.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.446+0000 7fb675e00640 1 -- 172.21.15.45:0/1637713058 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6701a6b30 con 0x7fb670101ef0 2024-03-28T17:55:04.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.449+0000 7fb67a189740 1 Processor -- start 2024-03-28T17:55:04.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.450+0000 7fb67a189740 1 -- start start 2024-03-28T17:55:04.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.450+0000 7fb66d600640 1 -- 172.21.15.45:0/1637713058 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fb65c046020 con 0x7fb670101ef0 2024-03-28T17:55:04.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.450+0000 7fb67a189740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622e13ecdd0 0x5622e13ebb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:04.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.450+0000 7fb67a189740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13ec0f0 0x5622e13ec570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:04.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.450+0000 7fb67a189740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5622e13f5460 0x5622e13f7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:04.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.450+0000 7fb67a189740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622e1350250 con 0x5622e13ecdd0 2024-03-28T17:55:04.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.450+0000 7fb67a189740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622e12f8a30 con 0x5622e13ec0f0 2024-03-28T17:55:04.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.450+0000 7fb67a189740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622e134c2f0 con 0x5622e13f5460 2024-03-28T17:55:04.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.450+0000 7fb674a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13ec0f0 0x5622e13ec570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:04.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.450+0000 7fb675400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622e13ecdd0 0x5622e13ebb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:04.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.450+0000 7fb674a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13ec0f0 0x5622e13ec570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53170/0 (socket says 172.21.15.45:53170) 2024-03-28T17:55:04.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.450+0000 7fb674a00640 1 -- 172.21.15.45:0/1389083078 learned_addr learned my addr 172.21.15.45:0/1389083078 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:55:04.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.450+0000 7fb66fe00640 1 --2- 172.21.15.45:0/1389083078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5622e13f5460 0x5622e13f7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:04.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.451+0000 7fb674a00640 1 -- 172.21.15.45:0/1389083078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5622e13f5460 msgr2=0x5622e13f7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:55:04.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.451+0000 7fb674a00640 1 --2- 172.21.15.45:0/1389083078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5622e13f5460 0x5622e13f7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.451+0000 7fb674a00640 1 -- 172.21.15.45:0/1389083078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622e13ecdd0 msgr2=0x5622e13ebb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:55:04.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.451+0000 7fb674a00640 1 --2- 172.21.15.45:0/1389083078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622e13ecdd0 0x5622e13ebb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.451+0000 7fb674a00640 1 -- 172.21.15.45:0/1389083078 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622e12e3690 con 0x5622e13ec0f0 2024-03-28T17:55:04.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.451+0000 7fb675400640 1 --2- 172.21.15.45:0/1389083078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622e13ecdd0 0x5622e13ebb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:55:04.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.451+0000 7fb674a00640 1 --2- 172.21.15.45:0/1389083078 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13ec0f0 0x5622e13ec570 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb660010560 tx=0x7fb660010a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63d9c3efcc37e065 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:55:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.451+0000 7fb665c00640 1 -- 172.21.15.45:0/1389083078 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66000e7d0 con 0x5622e13ec0f0 2024-03-28T17:55:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.451+0000 7fb665c00640 1 -- 172.21.15.45:0/1389083078 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb66000ee50 con 0x5622e13ec0f0 2024-03-28T17:55:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.451+0000 7fb665c00640 1 -- 172.21.15.45:0/1389083078 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6600193d0 con 0x5622e13ec0f0 2024-03-28T17:55:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.452+0000 7fb66d600640 1 --2- 172.21.15.45:0/1637713058 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fb6540a2970 0x7fb6540a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.452+0000 7fb67a189740 1 -- 172.21.15.45:0/1389083078 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13ec0f0 msgr2=0x5622e13ec570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:55:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.452+0000 7fb67a189740 1 --2- 172.21.15.45:0/1389083078 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13ec0f0 0x5622e13ec570 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb660010560 tx=0x7fb660010a20 comp rx=0 tx=0).stop 2024-03-28T17:55:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.452+0000 7fb66d600640 1 -- 172.21.15.45:0/1637713058 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9888+0+0 (secure 0 0 0) 0x7fb65c0f1270 con 0x7fb670101ef0 2024-03-28T17:55:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.452+0000 7fb675400640 1 --2- 172.21.15.45:0/1637713058 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fb6540a2970 0x7fb6540a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.452+0000 7fb67a189740 1 -- 172.21.15.45:0/1389083078 shutdown_connections 2024-03-28T17:55:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.452+0000 7fb67a189740 1 --2- 172.21.15.45:0/1389083078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5622e13f5460 0x5622e13f7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.452+0000 7fb67a189740 1 --2- 172.21.15.45:0/1389083078 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13ec0f0 0x5622e13ec570 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.452+0000 7fb67a189740 1 --2- 172.21.15.45:0/1389083078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622e13ecdd0 0x5622e13ebb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.452+0000 7fb67a189740 1 -- 172.21.15.45:0/1389083078 >> 172.21.15.45:0/1389083078 conn(0x5622e13b3200 msgr2=0x5622e13e6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:55:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.452+0000 7fb67a189740 1 -- 172.21.15.45:0/1389083078 shutdown_connections 2024-03-28T17:55:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.452+0000 7fb67a189740 1 -- 172.21.15.45:0/1389083078 wait complete. 2024-03-28T17:55:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.453+0000 7fb675400640 1 --2- 172.21.15.45:0/1637713058 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fb6540a2970 0x7fb6540a4e30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb668009d60 tx=0x7fb668005f90 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:55:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.453+0000 7fb67a189740 1 -- start start 2024-03-28T17:55:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.453+0000 7fb67a189740 1 Processor -- start 2024-03-28T17:55:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.453+0000 7fb67a189740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13ecdd0 0x5622e13f4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.453+0000 7fb67a189740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5622e13f5460 0x5622e13f1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:04.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.453+0000 7fb67a189740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622e13f2510 0x5622e13f2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.453+0000 7fb67a189740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622e13f82d0 con 0x5622e13f2510 2024-03-28T17:55:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.453+0000 7fb67a189740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622e130be80 con 0x5622e13ecdd0 2024-03-28T17:55:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.453+0000 7fb67a189740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622e13f8450 con 0x5622e13f5460 2024-03-28T17:55:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.454+0000 7fb674a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13ecdd0 0x5622e13f4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.454+0000 7fb675400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622e13f2510 0x5622e13f2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.454+0000 7fb674a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13ecdd0 0x5622e13f4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53186/0 (socket says 172.21.15.45:53186) 2024-03-28T17:55:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.454+0000 7fb67a189740 10 client.0 _ll_register_callbacks cb 0x5622e131e130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:55:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.454+0000 7fb674a00640 1 -- 172.21.15.45:0/4096133014 learned_addr learned my addr 172.21.15.45:0/4096133014 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:55:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.454+0000 7fb66fe00640 1 --2- 172.21.15.45:0/4096133014 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5622e13f5460 0x5622e13f1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.454+0000 7fb674a00640 1 -- 172.21.15.45:0/4096133014 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5622e13f5460 msgr2=0x5622e13f1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:55:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.454+0000 7fb674a00640 1 --2- 172.21.15.45:0/4096133014 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5622e13f5460 0x5622e13f1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.454+0000 7fb674a00640 1 -- 172.21.15.45:0/4096133014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622e13f2510 msgr2=0x5622e13f2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:55:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.454+0000 7fb674a00640 1 --2- 172.21.15.45:0/4096133014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622e13f2510 0x5622e13f2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.454+0000 7fb674a00640 1 -- 172.21.15.45:0/4096133014 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5622e112e7d0 con 0x5622e13ecdd0 2024-03-28T17:55:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.454+0000 7fb675400640 1 --2- 172.21.15.45:0/4096133014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622e13f2510 0x5622e13f2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:55:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.454+0000 7fb66fe00640 1 --2- 172.21.15.45:0/4096133014 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5622e13f5460 0x5622e13f1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:55:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.454+0000 7fb674a00640 1 --2- 172.21.15.45:0/4096133014 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13ecdd0 0x5622e13f4c70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb660000f80 tx=0x7fb6600119c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=988624c83b9ba374 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:55:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.454+0000 7fb65a000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:55:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.455+0000 7fb65a000640 1 -- 172.21.15.45:0/4096133014 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66001f040 con 0x5622e13ecdd0 2024-03-28T17:55:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.455+0000 7fb67a189740 1 -- 172.21.15.45:0/4096133014 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb66000ee50 con 0x5622e13ecdd0 2024-03-28T17:55:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.455+0000 7fb65a000640 1 -- 172.21.15.45:0/4096133014 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9888+0+0 (secure 0 0 0) 0x7fb6600116c0 con 0x5622e13ecdd0 2024-03-28T17:55:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.455+0000 7fb65a000640 1 -- 172.21.15.45:0/4096133014 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=129) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb660030050 con 0x5622e13ecdd0 2024-03-28T17:55:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.455+0000 7fb67a189740 10 client.26141 fetch_fsmap learned FSMap version 129 2024-03-28T17:55:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.455+0000 7fb67a189740 1 -- 172.21.15.45:0/4096133014 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=129,monmap=4+}) v3 -- 0x5622e10643f0 con 0x5622e13ecdd0 2024-03-28T17:55:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.455+0000 7fb65a000640 1 -- 172.21.15.45:0/4096133014 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb66000ee50 con 0x5622e13ecdd0 2024-03-28T17:55:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.455+0000 7fb65a000640 1 -- 172.21.15.45:0/4096133014 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 129) v1 ==== 6768+0+0 (secure 0 0 0) 0x7fb66002b030 con 0x5622e13ecdd0 2024-03-28T17:55:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.455+0000 7fb67a189740 10 client.26141 fetch_fsmap finished waiting for FSMap version 129 2024-03-28T17:55:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.455+0000 7fb67a189740 10 client.26141 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:55:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.455+0000 7fb67a189740 20 client.26141 populate_metadata read hostname 'smithi045' 2024-03-28T17:55:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.456+0000 7fb67a189740 1 --2- 172.21.15.45:0/4096133014 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5622e1405780 0x5622e1407b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.456+0000 7fb67a189740 4 client.26141 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:55:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.456+0000 7fb67a189740 1 -- 172.21.15.45:0/4096133014 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5622e135d790 con 0x5622e1405780 2024-03-28T17:55:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.456+0000 7fb66fe00640 1 --2- 172.21.15.45:0/4096133014 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5622e1405780 0x5622e1407b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.456+0000 7fb66fe00640 1 --2- 172.21.15.45:0/4096133014 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5622e1405780 0x5622e1407b70 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c41ebda86c0b3f3 server_cookie=7bc59552e2886a32 in_seq=0 out_seq=0 2024-03-28T17:55:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.456+0000 7fb65a000640 10 client.26141 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:55:04.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.457+0000 7fb65a000640 1 -- 172.21.15.45:0/4096133014 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5622e135d790 con 0x5622e1405780 2024-03-28T17:55:04.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.457+0000 7fb65a000640 10 client.26141 handle_command_reply: tid=0 2024-03-28T17:55:04.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.457+0000 7fb67a189740 1 client.26141 shutdown 2024-03-28T17:55:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.458+0000 7fb67a189740 1 -- 172.21.15.45:0/4096133014 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13ecdd0 msgr2=0x5622e13f4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:55:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.458+0000 7fb67a189740 1 --2- 172.21.15.45:0/4096133014 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13ecdd0 0x5622e13f4c70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb660000f80 tx=0x7fb6600119c0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.458+0000 7fb67a189740 1 -- 172.21.15.45:0/4096133014 shutdown_connections 2024-03-28T17:55:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.458+0000 7fb67a189740 1 --2- 172.21.15.45:0/4096133014 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5622e1405780 0x5622e1407b70 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.458+0000 7fb67a189740 1 --2- 172.21.15.45:0/4096133014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622e13f2510 0x5622e13f2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.458+0000 7fb67a189740 1 --2- 172.21.15.45:0/4096133014 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5622e13f5460 0x5622e13f1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.458+0000 7fb67a189740 1 --2- 172.21.15.45:0/4096133014 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13ecdd0 0x5622e13f4c70 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.458+0000 7fb65a000640 0 client.26141 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:55:04.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.458+0000 7fb67a189740 1 -- 172.21.15.45:0/4096133014 >> 172.21.15.45:0/4096133014 conn(0x5622e13b3200 msgr2=0x5622e13e8890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:55:04.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.458+0000 7fb67a189740 1 -- 172.21.15.45:0/4096133014 shutdown_connections 2024-03-28T17:55:04.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.458+0000 7fb67a189740 1 -- 172.21.15.45:0/4096133014 wait complete. 2024-03-28T17:55:04.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.458+0000 7fb67a189740 20 client.26141 trim_cache size 0 max 16384 2024-03-28T17:55:04.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.476+0000 7fb67a189740 1 Processor -- start 2024-03-28T17:55:04.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.476+0000 7fb67a189740 1 -- start start 2024-03-28T17:55:04.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.476+0000 7fb67a189740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5622e13ecdd0 0x5622e13f18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:04.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.476+0000 7fb67a189740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622e13f2510 0x5622e13e86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:04.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.476+0000 7fb67a189740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13f5460 0x5622e13f4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:04.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.476+0000 7fb67a189740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6701b0890 con 0x5622e13f2510 2024-03-28T17:55:04.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.476+0000 7fb67a189740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6701b0ef0 con 0x5622e13f5460 2024-03-28T17:55:04.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.476+0000 7fb67a189740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6701b1550 con 0x5622e13ecdd0 2024-03-28T17:55:04.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.476+0000 7fb675400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13f5460 0x5622e13f4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:04.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.476+0000 7fb66fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622e13f2510 0x5622e13e86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:04.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.476+0000 7fb675400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13f5460 0x5622e13f4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53190/0 (socket says 172.21.15.45:53190) 2024-03-28T17:55:04.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.476+0000 7fb675400640 1 -- 172.21.15.45:0/1138953576 learned_addr learned my addr 172.21.15.45:0/1138953576 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:55:04.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.476+0000 7fb674a00640 1 --2- 172.21.15.45:0/1138953576 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5622e13ecdd0 0x5622e13f18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:04.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.477+0000 7fb675400640 1 -- 172.21.15.45:0/1138953576 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5622e13ecdd0 msgr2=0x5622e13f18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:55:04.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.477+0000 7fb675400640 1 --2- 172.21.15.45:0/1138953576 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5622e13ecdd0 0x5622e13f18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.477+0000 7fb675400640 1 -- 172.21.15.45:0/1138953576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622e13f2510 msgr2=0x5622e13e86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:55:04.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.477+0000 7fb675400640 1 --2- 172.21.15.45:0/1138953576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622e13f2510 0x5622e13e86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.477+0000 7fb675400640 1 -- 172.21.15.45:0/1138953576 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622e10240f0 con 0x5622e13f5460 2024-03-28T17:55:04.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.477+0000 7fb674a00640 1 --2- 172.21.15.45:0/1138953576 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5622e13ecdd0 0x5622e13f18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:55:04.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.477+0000 7fb66fe00640 1 --2- 172.21.15.45:0/1138953576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622e13f2510 0x5622e13e86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:55:04.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.477+0000 7fb675400640 1 --2- 172.21.15.45:0/1138953576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13f5460 0x5622e13f4810 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb6680045b0 tx=0x7fb66801b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd40a9f4cbac4f23 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:55:04.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.477+0000 7fb665200640 1 -- 172.21.15.45:0/1138953576 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66801f4f0 con 0x5622e13f5460 2024-03-28T17:55:04.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.477+0000 7fb665200640 1 -- 172.21.15.45:0/1138953576 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb66801fb70 con 0x5622e13f5460 2024-03-28T17:55:04.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.477+0000 7fb665200640 1 -- 172.21.15.45:0/1138953576 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66801dd00 con 0x5622e13f5460 2024-03-28T17:55:04.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.478+0000 7fb67a189740 1 -- 172.21.15.45:0/1138953576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13f5460 msgr2=0x5622e13f4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:55:04.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.478+0000 7fb67a189740 1 --2- 172.21.15.45:0/1138953576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13f5460 0x5622e13f4810 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb6680045b0 tx=0x7fb66801b000 comp rx=0 tx=0).stop 2024-03-28T17:55:04.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.478+0000 7fb67a189740 1 -- 172.21.15.45:0/1138953576 shutdown_connections 2024-03-28T17:55:04.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.478+0000 7fb67a189740 1 --2- 172.21.15.45:0/1138953576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13f5460 0x5622e13f4810 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.478+0000 7fb67a189740 1 --2- 172.21.15.45:0/1138953576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622e13f2510 0x5622e13e86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.478+0000 7fb67a189740 1 --2- 172.21.15.45:0/1138953576 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5622e13ecdd0 0x5622e13f18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.478+0000 7fb67a189740 1 -- 172.21.15.45:0/1138953576 >> 172.21.15.45:0/1138953576 conn(0x5622e13b3200 msgr2=0x5622e13e7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:55:04.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.478+0000 7fb67a189740 1 -- 172.21.15.45:0/1138953576 shutdown_connections 2024-03-28T17:55:04.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.478+0000 7fb67a189740 1 -- 172.21.15.45:0/1138953576 wait complete. 2024-03-28T17:55:04.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.479+0000 7fb67a189740 1 -- start start 2024-03-28T17:55:04.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.479+0000 7fb67a189740 1 Processor -- start 2024-03-28T17:55:04.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.479+0000 7fb67a189740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5622e13ecdd0 0x5622e13ef980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:04.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.479+0000 7fb67a189740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13f2510 0x5622e13ee240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:04.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.479+0000 7fb67a189740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622e13f5460 0x5622e13ee780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:04.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.479+0000 7fb67a189740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6701ade10 con 0x5622e13f5460 2024-03-28T17:55:04.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.479+0000 7fb67a189740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6701af0d0 con 0x5622e13f2510 2024-03-28T17:55:04.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.479+0000 7fb67a189740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6701ae410 con 0x5622e13ecdd0 2024-03-28T17:55:04.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.479+0000 7fb67a189740 10 client.0 _ll_register_callbacks cb 0x7fb668009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:55:04.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.479+0000 7fb66fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13f2510 0x5622e13ee240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:04.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.480+0000 7fb66fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13f2510 0x5622e13ee240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53198/0 (socket says 172.21.15.45:53198) 2024-03-28T17:55:04.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.480+0000 7fb66fe00640 1 -- 172.21.15.45:0/2534440350 learned_addr learned my addr 172.21.15.45:0/2534440350 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:55:04.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.480+0000 7fb675400640 1 --2- 172.21.15.45:0/2534440350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622e13f5460 0x5622e13ee780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:04.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.480+0000 7fb674a00640 1 --2- 172.21.15.45:0/2534440350 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5622e13ecdd0 0x5622e13ef980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:04.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.480+0000 7fb66fe00640 1 -- 172.21.15.45:0/2534440350 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5622e13ecdd0 msgr2=0x5622e13ef980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:55:04.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.480+0000 7fb66fe00640 1 --2- 172.21.15.45:0/2534440350 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5622e13ecdd0 0x5622e13ef980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.480+0000 7fb66fe00640 1 -- 172.21.15.45:0/2534440350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622e13f5460 msgr2=0x5622e13ee780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:55:04.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.480+0000 7fb66fe00640 1 --2- 172.21.15.45:0/2534440350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622e13f5460 0x5622e13ee780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.480+0000 7fb66fe00640 1 -- 172.21.15.45:0/2534440350 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5622e13eecc0 con 0x5622e13f2510 2024-03-28T17:55:04.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.480+0000 7fb66fe00640 1 --2- 172.21.15.45:0/2534440350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13f2510 0x5622e13ee240 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb65c0b9e20 tx=0x7fb65c0f4d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66b19be3121dea33 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:55:04.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.480+0000 7fb65a000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:55:04.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.480+0000 7fb65a000640 1 -- 172.21.15.45:0/2534440350 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65c0bd220 con 0x5622e13f2510 2024-03-28T17:55:04.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.480+0000 7fb67a189740 1 -- 172.21.15.45:0/2534440350 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb66801fb70 con 0x5622e13f2510 2024-03-28T17:55:04.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.480+0000 7fb65a000640 1 -- 172.21.15.45:0/2534440350 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9888+0+0 (secure 0 0 0) 0x7fb65c02cb90 con 0x5622e13f2510 2024-03-28T17:55:04.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.480+0000 7fb65a000640 1 -- 172.21.15.45:0/2534440350 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=129) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb65c0bcdb0 con 0x5622e13f2510 2024-03-28T17:55:04.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.481+0000 7fb67a189740 10 client.26147 fetch_fsmap learned FSMap version 129 2024-03-28T17:55:04.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.481+0000 7fb67a189740 1 -- 172.21.15.45:0/2534440350 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=129,monmap=4+}) v3 -- 0x5622e1460cd0 con 0x5622e13f2510 2024-03-28T17:55:04.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.481+0000 7fb65a000640 1 -- 172.21.15.45:0/2534440350 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb66801fb70 con 0x5622e13f2510 2024-03-28T17:55:04.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.481+0000 7fb65a000640 1 -- 172.21.15.45:0/2534440350 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 129) v1 ==== 6768+0+0 (secure 0 0 0) 0x7fb65c0b9ac0 con 0x5622e13f2510 2024-03-28T17:55:04.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.481+0000 7fb67a189740 10 client.26147 fetch_fsmap finished waiting for FSMap version 129 2024-03-28T17:55:04.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.481+0000 7fb67a189740 10 client.26147 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:55:04.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.481+0000 7fb67a189740 20 client.26147 populate_metadata read hostname 'smithi045' 2024-03-28T17:55:04.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.481+0000 7fb67a189740 1 --2- 172.21.15.45:0/2534440350 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5622e14613b0 0x5622e14637d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:04.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.481+0000 7fb67a189740 4 client.26147 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:55:04.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.481+0000 7fb67a189740 1 -- 172.21.15.45:0/2534440350 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5622e1357ae0 con 0x5622e14613b0 2024-03-28T17:55:04.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.481+0000 7fb674a00640 1 --2- 172.21.15.45:0/2534440350 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5622e14613b0 0x5622e14637d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:04.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.482+0000 7fb674a00640 1 --2- 172.21.15.45:0/2534440350 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5622e14613b0 0x5622e14637d0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e91cfd11b7e4dccf server_cookie=187553189b6fe952 in_seq=0 out_seq=0 2024-03-28T17:55:04.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.482+0000 7fb65a000640 10 client.26147 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:55:04.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.482+0000 7fb65a000640 1 -- 172.21.15.45:0/2534440350 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5622e1357ae0 con 0x5622e14613b0 2024-03-28T17:55:04.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.482+0000 7fb65a000640 10 client.26147 handle_command_reply: tid=0 2024-03-28T17:55:04.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.482+0000 7fb67a189740 1 client.26147 shutdown 2024-03-28T17:55:04.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.482+0000 7fb67a189740 1 -- 172.21.15.45:0/2534440350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13f2510 msgr2=0x5622e13ee240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:55:04.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.482+0000 7fb67a189740 1 --2- 172.21.15.45:0/2534440350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13f2510 0x5622e13ee240 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb65c0b9e20 tx=0x7fb65c0f4d20 comp rx=0 tx=0).stop 2024-03-28T17:55:04.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.482+0000 7fb67a189740 1 -- 172.21.15.45:0/2534440350 shutdown_connections 2024-03-28T17:55:04.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.482+0000 7fb67a189740 1 --2- 172.21.15.45:0/2534440350 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5622e14613b0 0x5622e14637d0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.482+0000 7fb67a189740 1 --2- 172.21.15.45:0/2534440350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622e13f5460 0x5622e13ee780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.482+0000 7fb67a189740 1 --2- 172.21.15.45:0/2534440350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5622e13f2510 0x5622e13ee240 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.482+0000 7fb67a189740 1 --2- 172.21.15.45:0/2534440350 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5622e13ecdd0 0x5622e13ef980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.482+0000 7fb67a189740 1 -- 172.21.15.45:0/2534440350 >> 172.21.15.45:0/2534440350 conn(0x5622e13b3200 msgr2=0x5622e13e8740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:55:04.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.482+0000 7fb65a000640 0 client.26147 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:55:04.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.482+0000 7fb67a189740 1 -- 172.21.15.45:0/2534440350 shutdown_connections 2024-03-28T17:55:04.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.482+0000 7fb67a189740 1 -- 172.21.15.45:0/2534440350 wait complete. 2024-03-28T17:55:04.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.483+0000 7fb67a189740 20 client.26147 trim_cache size 0 max 16384 2024-03-28T17:55:04.487 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:55:04.487 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T17:55:04.487 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T17:55:04.488 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T17:55:04.488 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T17:55:04.488 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T17:55:04.488 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T17:55:04.488 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:55:04.488 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:55:04.488 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:55:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.483+0000 7fb675e00640 1 -- 172.21.15.45:0/1637713058 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fb6540a2970 msgr2=0x7fb6540a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:55:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.483+0000 7fb675e00640 1 --2- 172.21.15.45:0/1637713058 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fb6540a2970 0x7fb6540a4e30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb668009d60 tx=0x7fb668005f90 comp rx=0 tx=0).stop 2024-03-28T17:55:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.483+0000 7fb675e00640 1 -- 172.21.15.45:0/1637713058 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb670101ef0 msgr2=0x7fb67019a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:55:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.483+0000 7fb675e00640 1 --2- 172.21.15.45:0/1637713058 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb670101ef0 0x7fb67019a500 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb65c036040 tx=0x7fb65c004360 comp rx=0 tx=0).stop 2024-03-28T17:55:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.483+0000 7fb675e00640 1 -- 172.21.15.45:0/1637713058 shutdown_connections 2024-03-28T17:55:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.483+0000 7fb675e00640 1 --2- 172.21.15.45:0/1637713058 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fb6540a2970 0x7fb6540a4e30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.483+0000 7fb675e00640 1 --2- 172.21.15.45:0/1637713058 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb6701075f0 0x7fb67019e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.483+0000 7fb675e00640 1 --2- 172.21.15.45:0/1637713058 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb670101ef0 0x7fb67019a500 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.483+0000 7fb675e00640 1 --2- 172.21.15.45:0/1637713058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb670101530 0x7fb670199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.483+0000 7fb675e00640 1 -- 172.21.15.45:0/1637713058 >> 172.21.15.45:0/1637713058 conn(0x7fb6700fd000 msgr2=0x7fb6700fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:55:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.483+0000 7fb675e00640 1 -- 172.21.15.45:0/1637713058 shutdown_connections 2024-03-28T17:55:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:04.484+0000 7fb675e00640 1 -- 172.21.15.45:0/1637713058 wait complete. 2024-03-28T17:55:04.502 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T17:55:04.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:04 smithi151 ceph-mon[36481]: pgmap v201: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 187 B/s wr, 10 op/s 2024-03-28T17:55:04.871 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 157 sec latency 272768.053 ms 2024-03-28T17:55:04.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:04 smithi045 ceph-mon[31773]: pgmap v201: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 187 B/s wr, 10 op/s 2024-03-28T17:55:05.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:04 smithi071 ceph-mon[36597]: pgmap v201: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 187 B/s wr, 10 op/s 2024-03-28T17:55:05.871 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 158 sec latency 273768.214 ms 2024-03-28T17:55:06.862 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:06 smithi071 ceph-mon[36597]: pgmap v202: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T17:55:06.871 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 159 sec latency 274768.337 ms 2024-03-28T17:55:06.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:06 smithi045 ceph-mon[31773]: pgmap v202: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T17:55:07.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:06 smithi151 ceph-mon[36481]: pgmap v202: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T17:55:07.871 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 160 sec latency 275768.479 ms 2024-03-28T17:55:08.327 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:07 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 4. 2024-03-28T17:55:08.328 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:07 smithi045 systemd[1]: Stopped Ceph mds.d for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:55:08.328 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:07 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Consumed 1.612s CPU time. 2024-03-28T17:55:08.328 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:07 smithi045 systemd[1]: Starting Ceph mds.d for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:55:08.583 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:08 smithi045 podman[91082]: 2024-03-28 17:55:08.233431579 +0000 UTC m=+0.037538814 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:55:08.583 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:08 smithi045 podman[91082]: 2024-03-28 17:55:08.392707234 +0000 UTC m=+0.196814492 container create 54ff618292b634118f4309d5f72acd6bbeade0ae565376aeab753e21496b2ab4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2) 2024-03-28T17:55:08.872 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 161 sec latency 276768.607 ms 2024-03-28T17:55:08.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:08 smithi045 ceph-mon[31773]: pgmap v203: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-28T17:55:08.916 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:08 smithi045 podman[91082]: 2024-03-28 17:55:08.671515653 +0000 UTC m=+0.475622915 container init 54ff618292b634118f4309d5f72acd6bbeade0ae565376aeab753e21496b2ab4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T17:55:08.916 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:08 smithi045 podman[91082]: 2024-03-28 17:55:08.681598772 +0000 UTC m=+0.485706000 container start 54ff618292b634118f4309d5f72acd6bbeade0ae565376aeab753e21496b2ab4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-28T17:55:08.916 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:08 smithi045 ceph-mds[91100]: -- 172.21.15.45:0/3097562502 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fe26809a40 con 0x55fe267dd000 2024-03-28T17:55:08.916 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: starting mds.d at 2024-03-28T17:55:08.916 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:08 smithi045 bash[91082]: 54ff618292b634118f4309d5f72acd6bbeade0ae565376aeab753e21496b2ab4 2024-03-28T17:55:08.916 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:08 smithi045 systemd[1]: Started Ceph mds.d for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:55:09.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:08 smithi071 ceph-mon[36597]: pgmap v203: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-28T17:55:09.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:08 smithi151 ceph-mon[36481]: pgmap v203: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-28T17:55:09.872 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 162 sec latency 277768.799 ms 2024-03-28T17:55:09.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:09 smithi045 ceph-mon[31773]: daemon mds.d restarted 2024-03-28T17:55:09.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:09 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:55:09.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:09 smithi045 ceph-mon[31773]: osdmap e105: 12 total, 12 up, 12 in 2024-03-28T17:55:09.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:09 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/67226369,v1:172.21.15.45:6837/67226369] up:boot 2024-03-28T17:55:09.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:09 smithi045 ceph-mon[31773]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-03-28T17:55:09.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:09 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T17:55:09.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:09 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:55:09.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:09 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:10.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:09 smithi071 ceph-mon[36597]: daemon mds.d restarted 2024-03-28T17:55:10.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:09 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:55:10.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:09 smithi071 ceph-mon[36597]: osdmap e105: 12 total, 12 up, 12 in 2024-03-28T17:55:10.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:09 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/67226369,v1:172.21.15.45:6837/67226369] up:boot 2024-03-28T17:55:10.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:09 smithi071 ceph-mon[36597]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-03-28T17:55:10.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:09 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T17:55:10.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:09 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:55:10.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:09 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:09 smithi151 ceph-mon[36481]: daemon mds.d restarted 2024-03-28T17:55:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:09 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:55:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:09 smithi151 ceph-mon[36481]: osdmap e105: 12 total, 12 up, 12 in 2024-03-28T17:55:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:09 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/67226369,v1:172.21.15.45:6837/67226369] up:boot 2024-03-28T17:55:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:09 smithi151 ceph-mon[36481]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-03-28T17:55:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:09 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T17:55:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:09 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:55:10.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:09 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:10.872 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 163 sec latency 278768.924 ms 2024-03-28T17:55:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:10 smithi071 ceph-mon[36597]: pgmap v205: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-28T17:55:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:10 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/627131461,v1:172.21.15.71:6833/627131461] up:resolve 2024-03-28T17:55:11.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:10 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:11.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:10 smithi151 ceph-mon[36481]: pgmap v205: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-28T17:55:11.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:10 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/627131461,v1:172.21.15.71:6833/627131461] up:resolve 2024-03-28T17:55:11.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:10 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:11.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:10 smithi045 ceph-mon[31773]: pgmap v205: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-28T17:55:11.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:10 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/627131461,v1:172.21.15.71:6833/627131461] up:resolve 2024-03-28T17:55:11.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:10 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:11.872 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 164 sec latency 279769.055 ms 2024-03-28T17:55:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:11 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/627131461,v1:172.21.15.71:6833/627131461] up:reconnect 2024-03-28T17:55:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:11 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:12.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:11 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00999983 2024-03-28T17:55:12.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:11 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999983 2024-03-28T17:55:12.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:11 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/627131461,v1:172.21.15.71:6833/627131461] up:reconnect 2024-03-28T17:55:12.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:11 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:12.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:11 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00999983 2024-03-28T17:55:12.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:11 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999983 2024-03-28T17:55:12.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:11 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/627131461,v1:172.21.15.71:6833/627131461] up:reconnect 2024-03-28T17:55:12.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:11 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:12.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:11 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00999983 2024-03-28T17:55:12.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:11 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999983 2024-03-28T17:55:12.872 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 165 sec latency 280769.138 ms 2024-03-28T17:55:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-mon[36597]: pgmap v206: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:55:13.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/627131461,v1:172.21.15.71:6833/627131461] up:rejoin 2024-03-28T17:55:13.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:13.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-mon[36597]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-03-28T17:55:13.061 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f52c7711700 time 2024-03-28T17:55:12.800039+0000 2024-03-28T17:55:13.061 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:55:13.061 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:13.061 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f52d1d3b4a1] 2024-03-28T17:55:13.061 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f52d1d3b667] 2024-03-28T17:55:13.061 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555a80ba6223] 2024-03-28T17:55:13.062 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555a80a205d6] 2024-03-28T17:55:13.062 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555a80a20da1] 2024-03-28T17:55:13.062 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x555a80a232d5] 2024-03-28T17:55:13.062 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555a80a73233] 2024-03-28T17:55:13.062 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555a80a75197] 2024-03-28T17:55:13.062 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555a8097411b] 2024-03-28T17:55:13.062 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555a809746bc] 2024-03-28T17:55:13.062 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555a8095e1ff] 2024-03-28T17:55:13.062 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f52d1fcd488] 2024-03-28T17:55:13.062 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 13: (DispatchQueue::entry()+0x51f) [0x7f52d1fca60f] 2024-03-28T17:55:13.062 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f52d209de81] 2024-03-28T17:55:13.062 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f52d0ab81ca] 2024-03-28T17:55:13.063 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 16: clone() 2024-03-28T17:55:13.063 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 2024-03-28T17:55:12.800+0000 7f52c7711700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f52c7711700 time 2024-03-28T17:55:12.800039+0000 2024-03-28T17:55:13.063 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:55:13.063 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 2024-03-28T17:55:13.063 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:13.063 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f52d1d3b4a1] 2024-03-28T17:55:13.063 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f52d1d3b667] 2024-03-28T17:55:13.063 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555a80ba6223] 2024-03-28T17:55:13.063 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555a80a205d6] 2024-03-28T17:55:13.063 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555a80a20da1] 2024-03-28T17:55:13.063 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x555a80a232d5] 2024-03-28T17:55:13.063 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555a80a73233] 2024-03-28T17:55:13.063 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555a80a75197] 2024-03-28T17:55:13.063 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555a8097411b] 2024-03-28T17:55:13.063 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555a809746bc] 2024-03-28T17:55:13.064 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555a8095e1ff] 2024-03-28T17:55:13.064 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f52d1fcd488] 2024-03-28T17:55:13.064 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 13: (DispatchQueue::entry()+0x51f) [0x7f52d1fca60f] 2024-03-28T17:55:13.064 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f52d209de81] 2024-03-28T17:55:13.064 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f52d0ab81ca] 2024-03-28T17:55:13.064 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 16: clone() 2024-03-28T17:55:13.064 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 2024-03-28T17:55:13.064 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: *** Caught signal (Aborted) ** 2024-03-28T17:55:13.064 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: in thread 7f52c7711700 thread_name:ms_dispatch 2024-03-28T17:55:13.064 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:13.064 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f52d0ac2d20] 2024-03-28T17:55:13.065 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 2: gsignal() 2024-03-28T17:55:13.065 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 3: abort() 2024-03-28T17:55:13.065 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f52d1d3b4fb] 2024-03-28T17:55:13.065 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f52d1d3b667] 2024-03-28T17:55:13.065 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555a80ba6223] 2024-03-28T17:55:13.065 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555a80a205d6] 2024-03-28T17:55:13.066 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555a80a20da1] 2024-03-28T17:55:13.066 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x555a80a232d5] 2024-03-28T17:55:13.066 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555a80a73233] 2024-03-28T17:55:13.066 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555a80a75197] 2024-03-28T17:55:13.066 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555a8097411b] 2024-03-28T17:55:13.066 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555a809746bc] 2024-03-28T17:55:13.066 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555a8095e1ff] 2024-03-28T17:55:13.066 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f52d1fcd488] 2024-03-28T17:55:13.066 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 16: (DispatchQueue::entry()+0x51f) [0x7f52d1fca60f] 2024-03-28T17:55:13.066 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f52d209de81] 2024-03-28T17:55:13.066 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f52d0ab81ca] 2024-03-28T17:55:13.066 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 19: clone() 2024-03-28T17:55:13.066 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 2024-03-28T17:55:12.802+0000 7f52c7711700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:55:13.066 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: in thread 7f52c7711700 thread_name:ms_dispatch 2024-03-28T17:55:13.067 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 2024-03-28T17:55:13.067 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:13.067 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f52d0ac2d20] 2024-03-28T17:55:13.067 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 2: gsignal() 2024-03-28T17:55:13.067 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 3: abort() 2024-03-28T17:55:13.067 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f52d1d3b4fb] 2024-03-28T17:55:13.067 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f52d1d3b667] 2024-03-28T17:55:13.067 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555a80ba6223] 2024-03-28T17:55:13.067 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555a80a205d6] 2024-03-28T17:55:13.067 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555a80a20da1] 2024-03-28T17:55:13.067 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x555a80a232d5] 2024-03-28T17:55:13.067 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555a80a73233] 2024-03-28T17:55:13.067 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555a80a75197] 2024-03-28T17:55:13.067 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555a8097411b] 2024-03-28T17:55:13.068 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555a809746bc] 2024-03-28T17:55:13.068 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555a8095e1ff] 2024-03-28T17:55:13.068 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f52d1fcd488] 2024-03-28T17:55:13.068 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 16: (DispatchQueue::entry()+0x51f) [0x7f52d1fca60f] 2024-03-28T17:55:13.068 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f52d209de81] 2024-03-28T17:55:13.068 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f52d0ab81ca] 2024-03-28T17:55:13.068 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 19: clone() 2024-03-28T17:55:13.068 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:55:13.068 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 2024-03-28T17:55:13.068 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: -1> 2024-03-28T17:55:12.800+0000 7f52c7711700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f52c7711700 time 2024-03-28T17:55:12.800039+0000 2024-03-28T17:55:13.068 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:55:13.068 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 2024-03-28T17:55:13.068 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:13.068 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f52d1d3b4a1] 2024-03-28T17:55:13.069 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f52d1d3b667] 2024-03-28T17:55:13.069 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555a80ba6223] 2024-03-28T17:55:13.069 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555a80a205d6] 2024-03-28T17:55:13.069 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555a80a20da1] 2024-03-28T17:55:13.069 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x555a80a232d5] 2024-03-28T17:55:13.069 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555a80a73233] 2024-03-28T17:55:13.069 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555a80a75197] 2024-03-28T17:55:13.069 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555a8097411b] 2024-03-28T17:55:13.069 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555a809746bc] 2024-03-28T17:55:13.069 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555a8095e1ff] 2024-03-28T17:55:13.069 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f52d1fcd488] 2024-03-28T17:55:13.069 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 13: (DispatchQueue::entry()+0x51f) [0x7f52d1fca60f] 2024-03-28T17:55:13.069 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f52d209de81] 2024-03-28T17:55:13.069 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f52d0ab81ca] 2024-03-28T17:55:13.069 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 16: clone() 2024-03-28T17:55:13.070 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 2024-03-28T17:55:13.070 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 0> 2024-03-28T17:55:12.802+0000 7f52c7711700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:55:13.070 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: in thread 7f52c7711700 thread_name:ms_dispatch 2024-03-28T17:55:13.070 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 2024-03-28T17:55:13.070 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:13.070 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f52d0ac2d20] 2024-03-28T17:55:13.070 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 2: gsignal() 2024-03-28T17:55:13.070 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 3: abort() 2024-03-28T17:55:13.070 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f52d1d3b4fb] 2024-03-28T17:55:13.070 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f52d1d3b667] 2024-03-28T17:55:13.070 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555a80ba6223] 2024-03-28T17:55:13.071 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555a80a205d6] 2024-03-28T17:55:13.071 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555a80a20da1] 2024-03-28T17:55:13.071 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x555a80a232d5] 2024-03-28T17:55:13.071 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555a80a73233] 2024-03-28T17:55:13.071 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555a80a75197] 2024-03-28T17:55:13.071 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555a8097411b] 2024-03-28T17:55:13.071 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555a809746bc] 2024-03-28T17:55:13.071 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555a8095e1ff] 2024-03-28T17:55:13.071 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f52d1fcd488] 2024-03-28T17:55:13.071 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 16: (DispatchQueue::entry()+0x51f) [0x7f52d1fca60f] 2024-03-28T17:55:13.071 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f52d209de81] 2024-03-28T17:55:13.071 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f52d0ab81ca] 2024-03-28T17:55:13.071 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 19: clone() 2024-03-28T17:55:13.071 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:55:13.071 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 2024-03-28T17:55:13.071 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: -1> 2024-03-28T17:55:12.800+0000 7f52c7711700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f52c7711700 time 2024-03-28T17:55:12.800039+0000 2024-03-28T17:55:13.072 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:55:13.072 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 2024-03-28T17:55:13.072 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:13.072 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f52d1d3b4a1] 2024-03-28T17:55:13.072 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f52d1d3b667] 2024-03-28T17:55:13.072 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555a80ba6223] 2024-03-28T17:55:13.072 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555a80a205d6] 2024-03-28T17:55:13.072 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555a80a20da1] 2024-03-28T17:55:13.072 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x555a80a232d5] 2024-03-28T17:55:13.072 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555a80a73233] 2024-03-28T17:55:13.072 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555a80a75197] 2024-03-28T17:55:13.072 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555a8097411b] 2024-03-28T17:55:13.072 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555a809746bc] 2024-03-28T17:55:13.072 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555a8095e1ff] 2024-03-28T17:55:13.072 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f52d1fcd488] 2024-03-28T17:55:13.073 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 13: (DispatchQueue::entry()+0x51f) [0x7f52d1fca60f] 2024-03-28T17:55:13.073 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f52d209de81] 2024-03-28T17:55:13.073 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f52d0ab81ca] 2024-03-28T17:55:13.073 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 16: clone() 2024-03-28T17:55:13.073 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 2024-03-28T17:55:13.073 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 0> 2024-03-28T17:55:12.802+0000 7f52c7711700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:55:13.073 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: in thread 7f52c7711700 thread_name:ms_dispatch 2024-03-28T17:55:13.073 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 2024-03-28T17:55:13.073 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:13.073 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f52d0ac2d20] 2024-03-28T17:55:13.073 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 2: gsignal() 2024-03-28T17:55:13.073 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 3: abort() 2024-03-28T17:55:13.073 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f52d1d3b4fb] 2024-03-28T17:55:13.073 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f52d1d3b667] 2024-03-28T17:55:13.074 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555a80ba6223] 2024-03-28T17:55:13.074 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555a80a205d6] 2024-03-28T17:55:13.074 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555a80a20da1] 2024-03-28T17:55:13.074 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x555a80a232d5] 2024-03-28T17:55:13.074 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555a80a73233] 2024-03-28T17:55:13.074 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555a80a75197] 2024-03-28T17:55:13.074 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555a8097411b] 2024-03-28T17:55:13.074 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555a809746bc] 2024-03-28T17:55:13.074 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555a8095e1ff] 2024-03-28T17:55:13.074 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f52d1fcd488] 2024-03-28T17:55:13.074 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 16: (DispatchQueue::entry()+0x51f) [0x7f52d1fca60f] 2024-03-28T17:55:13.074 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f52d209de81] 2024-03-28T17:55:13.074 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f52d0ab81ca] 2024-03-28T17:55:13.074 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 19: clone() 2024-03-28T17:55:13.075 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:55:13.075 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71364]: 2024-03-28T17:55:13.075 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:12 smithi071 podman[71402]: 2024-03-28 17:55:12.845451071 +0000 UTC m=+0.018524090 container died 48b19cff6462edebc34d47d2bfafcd7ab1657ec01a12cf0e91b993c81b8b9100 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-28T17:55:13.075 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:12 smithi151 ceph-mon[36481]: pgmap v206: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:55:13.075 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:12 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/627131461,v1:172.21.15.71:6833/627131461] up:rejoin 2024-03-28T17:55:13.075 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:12 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:13.075 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:12 smithi151 ceph-mon[36481]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-03-28T17:55:13.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:12 smithi045 ceph-mon[31773]: pgmap v206: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:55:13.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:12 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/627131461,v1:172.21.15.71:6833/627131461] up:rejoin 2024-03-28T17:55:13.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:12 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:13.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:12 smithi045 ceph-mon[31773]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-03-28T17:55:13.791 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:13 smithi071 podman[71402]: 2024-03-28 17:55:13.511911037 +0000 UTC m=+0.684984048 container remove 48b19cff6462edebc34d47d2bfafcd7ab1657ec01a12cf0e91b993c81b8b9100 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-28T17:55:13.791 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:13 smithi071 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:55:13.872 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 166 sec latency 281769.293 ms 2024-03-28T17:55:14.060 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:13 smithi071 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.k.service: Failed with result 'exit-code'. 2024-03-28T17:55:14.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:13 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:55:14.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:13 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:55:14.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:13 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/627131461,v1:172.21.15.71:6833/627131461] up:active 2024-03-28T17:55:14.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:13 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:14.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:13 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:55:14.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:13 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:55:14.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:13 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:55:14.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:13 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/627131461,v1:172.21.15.71:6833/627131461] up:active 2024-03-28T17:55:14.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:13 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:14.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:13 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:55:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:13 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:55:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:13 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:55:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:13 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/627131461,v1:172.21.15.71:6833/627131461] up:active 2024-03-28T17:55:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:13 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:13 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:55:14.872 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 167 sec latency 282769.437 ms 2024-03-28T17:55:15.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:14 smithi151 ceph-mon[36481]: pgmap v207: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:55:15.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:14 smithi045 ceph-mon[31773]: pgmap v207: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:55:15.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:14 smithi071 ceph-mon[36597]: pgmap v207: 97 pgs: 97 active+clean; 27 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:55:15.872 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 168 sec latency 283769.559 ms 2024-03-28T17:55:16.873 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 169 sec latency 284769.699 ms 2024-03-28T17:55:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:16 smithi151 ceph-mon[36481]: pgmap v208: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:55:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:16 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:55:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:16 smithi045 ceph-mon[31773]: pgmap v208: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:55:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:16 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:55:17.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:16 smithi071 ceph-mon[36597]: pgmap v208: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:55:17.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:16 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:55:17.873 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 170 sec latency 285769.838 ms 2024-03-28T17:55:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:17 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:55:18.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:17 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:55:18.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:17 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:55:18.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:17 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:55:18.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:17 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:55:18.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:17 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:55:18.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:17 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:55:18.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:17 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:55:18.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:17 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:55:18.873 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 171 sec latency 286769.987 ms 2024-03-28T17:55:19.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:18 smithi045 ceph-mon[31773]: pgmap v209: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:55:19.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:18 smithi071 ceph-mon[36597]: pgmap v209: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:55:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:18 smithi151 ceph-mon[36481]: pgmap v209: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:55:19.873 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 172 sec latency 287770.095 ms 2024-03-28T17:55:20.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:19 smithi045 ceph-mon[31773]: pgmap v210: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 26 op/s 2024-03-28T17:55:20.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:19 smithi071 ceph-mon[36597]: pgmap v210: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 26 op/s 2024-03-28T17:55:20.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:19 smithi151 ceph-mon[36481]: pgmap v210: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 26 op/s 2024-03-28T17:55:20.874 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 173 sec latency 288770.226 ms 2024-03-28T17:55:21.873 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 174 sec latency 289770.317 ms 2024-03-28T17:55:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:22 smithi045 ceph-mon[31773]: pgmap v211: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-28T17:55:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:22 smithi071 ceph-mon[36597]: pgmap v211: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-28T17:55:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:22 smithi151 ceph-mon[36481]: pgmap v211: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-28T17:55:22.873 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 175 sec latency 290770.462 ms 2024-03-28T17:55:23.874 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 176 sec latency 291770.577 ms 2024-03-28T17:55:24.061 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:23 smithi071 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.k.service: Scheduled restart job, restart counter is at 3. 2024-03-28T17:55:24.061 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:23 smithi071 systemd[1]: Stopped Ceph mds.k for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:55:24.062 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:23 smithi071 systemd[1]: Starting Ceph mds.k for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:55:24.062 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:24 smithi071 podman[71626]: 2024-03-28 17:55:23.940141303 +0000 UTC m=+0.018825327 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:55:24.373 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:24 smithi071 ceph-mon[36597]: pgmap v212: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:55:24.373 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:24 smithi071 podman[71626]: 2024-03-28 17:55:24.062198923 +0000 UTC m=+0.140882939 container create ec94b0304e89a7a1069a124dad30146db183d162195c6779dfa66bcc8b3f026e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-28T17:55:24.373 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:24 smithi071 podman[71626]: 2024-03-28 17:55:24.281514814 +0000 UTC m=+0.360198837 container init ec94b0304e89a7a1069a124dad30146db183d162195c6779dfa66bcc8b3f026e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T17:55:24.373 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:24 smithi071 podman[71626]: 2024-03-28 17:55:24.284738995 +0000 UTC m=+0.363423005 container start ec94b0304e89a7a1069a124dad30146db183d162195c6779dfa66bcc8b3f026e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T17:55:24.373 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:24 smithi071 ceph-mds[71644]: -- 172.21.15.71:0/2350079264 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b50573fa40 con 0x55b505713000 2024-03-28T17:55:24.373 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:24 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: starting mds.k at 2024-03-28T17:55:24.375 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:24 smithi151 ceph-mon[36481]: pgmap v212: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:55:24.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:24 smithi045 ceph-mon[31773]: pgmap v212: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:55:24.652 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:24 smithi071 bash[71626]: ec94b0304e89a7a1069a124dad30146db183d162195c6779dfa66bcc8b3f026e 2024-03-28T17:55:24.652 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:24 smithi071 systemd[1]: Started Ceph mds.k for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:55:24.873 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 177 sec latency 292770.709 ms 2024-03-28T17:55:25.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:25 smithi045 ceph-mon[31773]: daemon mds.k restarted 2024-03-28T17:55:25.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:25 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:55:25.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:25 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T17:55:25.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:25 smithi045 ceph-mon[31773]: osdmap e106: 12 total, 12 up, 12 in 2024-03-28T17:55:25.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:25 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/888508431,v1:172.21.15.71:6833/888508431] up:boot 2024-03-28T17:55:25.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:25 smithi045 ceph-mon[31773]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-03-28T17:55:25.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:25 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:55:25.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:25 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:25.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:25 smithi071 ceph-mon[36597]: daemon mds.k restarted 2024-03-28T17:55:25.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:25 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:55:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:25 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T17:55:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:25 smithi071 ceph-mon[36597]: osdmap e106: 12 total, 12 up, 12 in 2024-03-28T17:55:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:25 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/888508431,v1:172.21.15.71:6833/888508431] up:boot 2024-03-28T17:55:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:25 smithi071 ceph-mon[36597]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-03-28T17:55:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:25 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:55:25.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:25 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:25.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:25 smithi151 ceph-mon[36481]: daemon mds.k restarted 2024-03-28T17:55:25.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:25 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:55:25.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:25 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T17:55:25.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:25 smithi151 ceph-mon[36481]: osdmap e106: 12 total, 12 up, 12 in 2024-03-28T17:55:25.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:25 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/888508431,v1:172.21.15.71:6833/888508431] up:boot 2024-03-28T17:55:25.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:25 smithi151 ceph-mon[36481]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-03-28T17:55:25.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:25 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:55:25.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:25 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:25.874 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 178 sec latency 293770.886 ms 2024-03-28T17:55:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:26 smithi045 ceph-mon[31773]: pgmap v214: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:55:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:26 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/67226369,v1:172.21.15.45:6837/67226369] up:resolve 2024-03-28T17:55:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:26 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:26 smithi071 ceph-mon[36597]: pgmap v214: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:55:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:26 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/67226369,v1:172.21.15.45:6837/67226369] up:resolve 2024-03-28T17:55:26.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:26 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:26 smithi151 ceph-mon[36481]: pgmap v214: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:55:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:26 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/67226369,v1:172.21.15.45:6837/67226369] up:resolve 2024-03-28T17:55:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:26 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:26.874 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 179 sec latency 294771.035 ms 2024-03-28T17:55:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:27 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:55:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:27 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/67226369,v1:172.21.15.45:6837/67226369] up:reconnect 2024-03-28T17:55:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:27 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:27 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T17:55:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:27 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:55:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:27 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/67226369,v1:172.21.15.45:6837/67226369] up:reconnect 2024-03-28T17:55:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:27 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:27.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:27 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T17:55:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:27 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:55:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:27 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/67226369,v1:172.21.15.45:6837/67226369] up:reconnect 2024-03-28T17:55:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:27 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:27 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T17:55:27.874 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 180 sec latency 295771.199 ms 2024-03-28T17:55:28.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-mon[31773]: pgmap v215: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:55:28.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/67226369,v1:172.21.15.45:6837/67226369] up:rejoin 2024-03-28T17:55:28.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:28.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-mon[31773]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-03-28T17:55:28.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7681400700 time 2024-03-28T17:55:28.399559+0000 2024-03-28T17:55:28.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:55:28.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:28.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f768cca94a1] 2024-03-28T17:55:28.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f768cca9667] 2024-03-28T17:55:28.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fe25867223] 2024-03-28T17:55:28.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fe256e15d6] 2024-03-28T17:55:28.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fe256e1da1] 2024-03-28T17:55:28.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fe256e42d5] 2024-03-28T17:55:28.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fe25734233] 2024-03-28T17:55:28.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fe25736197] 2024-03-28T17:55:28.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fe2563511b] 2024-03-28T17:55:28.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fe256356bc] 2024-03-28T17:55:28.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fe2561f1ff] 2024-03-28T17:55:28.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f768cf3b488] 2024-03-28T17:55:28.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 13: (DispatchQueue::entry()+0x51f) [0x7f768cf3860f] 2024-03-28T17:55:28.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f768d00be81] 2024-03-28T17:55:28.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f768ba261ca] 2024-03-28T17:55:28.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 16: clone() 2024-03-28T17:55:28.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 2024-03-28T17:55:28.401+0000 7f7681400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7681400700 time 2024-03-28T17:55:28.399559+0000 2024-03-28T17:55:28.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:55:28.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 2024-03-28T17:55:28.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:28.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f768cca94a1] 2024-03-28T17:55:28.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f768cca9667] 2024-03-28T17:55:28.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fe25867223] 2024-03-28T17:55:28.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fe256e15d6] 2024-03-28T17:55:28.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fe256e1da1] 2024-03-28T17:55:28.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fe256e42d5] 2024-03-28T17:55:28.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fe25734233] 2024-03-28T17:55:28.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fe25736197] 2024-03-28T17:55:28.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fe2563511b] 2024-03-28T17:55:28.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fe256356bc] 2024-03-28T17:55:28.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fe2561f1ff] 2024-03-28T17:55:28.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f768cf3b488] 2024-03-28T17:55:28.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 13: (DispatchQueue::entry()+0x51f) [0x7f768cf3860f] 2024-03-28T17:55:28.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f768d00be81] 2024-03-28T17:55:28.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f768ba261ca] 2024-03-28T17:55:28.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 16: clone() 2024-03-28T17:55:28.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 2024-03-28T17:55:28.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: *** Caught signal (Aborted) ** 2024-03-28T17:55:28.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: in thread 7f7681400700 thread_name:ms_dispatch 2024-03-28T17:55:28.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:28.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f768ba30d20] 2024-03-28T17:55:28.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 2: gsignal() 2024-03-28T17:55:28.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 3: abort() 2024-03-28T17:55:28.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f768cca94fb] 2024-03-28T17:55:28.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f768cca9667] 2024-03-28T17:55:28.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fe25867223] 2024-03-28T17:55:28.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fe256e15d6] 2024-03-28T17:55:28.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fe256e1da1] 2024-03-28T17:55:28.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fe256e42d5] 2024-03-28T17:55:28.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fe25734233] 2024-03-28T17:55:28.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fe25736197] 2024-03-28T17:55:28.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fe2563511b] 2024-03-28T17:55:28.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fe256356bc] 2024-03-28T17:55:28.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fe2561f1ff] 2024-03-28T17:55:28.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f768cf3b488] 2024-03-28T17:55:28.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 16: (DispatchQueue::entry()+0x51f) [0x7f768cf3860f] 2024-03-28T17:55:28.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f768d00be81] 2024-03-28T17:55:28.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f768ba261ca] 2024-03-28T17:55:28.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 19: clone() 2024-03-28T17:55:28.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 2024-03-28T17:55:28.404+0000 7f7681400700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:55:28.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: in thread 7f7681400700 thread_name:ms_dispatch 2024-03-28T17:55:28.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 2024-03-28T17:55:28.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:28.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f768ba30d20] 2024-03-28T17:55:28.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 2: gsignal() 2024-03-28T17:55:28.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 3: abort() 2024-03-28T17:55:28.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f768cca94fb] 2024-03-28T17:55:28.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f768cca9667] 2024-03-28T17:55:28.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fe25867223] 2024-03-28T17:55:28.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fe256e15d6] 2024-03-28T17:55:28.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fe256e1da1] 2024-03-28T17:55:28.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fe256e42d5] 2024-03-28T17:55:28.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fe25734233] 2024-03-28T17:55:28.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fe25736197] 2024-03-28T17:55:28.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fe2563511b] 2024-03-28T17:55:28.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fe256356bc] 2024-03-28T17:55:28.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fe2561f1ff] 2024-03-28T17:55:28.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f768cf3b488] 2024-03-28T17:55:28.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 16: (DispatchQueue::entry()+0x51f) [0x7f768cf3860f] 2024-03-28T17:55:28.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f768d00be81] 2024-03-28T17:55:28.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f768ba261ca] 2024-03-28T17:55:28.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 19: clone() 2024-03-28T17:55:28.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:55:28.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 2024-03-28T17:55:28.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: -1> 2024-03-28T17:55:28.401+0000 7f7681400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7681400700 time 2024-03-28T17:55:28.399559+0000 2024-03-28T17:55:28.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:55:28.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 2024-03-28T17:55:28.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:28.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f768cca94a1] 2024-03-28T17:55:28.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f768cca9667] 2024-03-28T17:55:28.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fe25867223] 2024-03-28T17:55:28.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fe256e15d6] 2024-03-28T17:55:28.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fe256e1da1] 2024-03-28T17:55:28.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fe256e42d5] 2024-03-28T17:55:28.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fe25734233] 2024-03-28T17:55:28.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fe25736197] 2024-03-28T17:55:28.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fe2563511b] 2024-03-28T17:55:28.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fe256356bc] 2024-03-28T17:55:28.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fe2561f1ff] 2024-03-28T17:55:28.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f768cf3b488] 2024-03-28T17:55:28.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 13: (DispatchQueue::entry()+0x51f) [0x7f768cf3860f] 2024-03-28T17:55:28.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f768d00be81] 2024-03-28T17:55:28.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f768ba261ca] 2024-03-28T17:55:28.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 16: clone() 2024-03-28T17:55:28.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 2024-03-28T17:55:28.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 0> 2024-03-28T17:55:28.404+0000 7f7681400700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:55:28.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: in thread 7f7681400700 thread_name:ms_dispatch 2024-03-28T17:55:28.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 2024-03-28T17:55:28.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:28.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f768ba30d20] 2024-03-28T17:55:28.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 2: gsignal() 2024-03-28T17:55:28.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 3: abort() 2024-03-28T17:55:28.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f768cca94fb] 2024-03-28T17:55:28.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f768cca9667] 2024-03-28T17:55:28.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fe25867223] 2024-03-28T17:55:28.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fe256e15d6] 2024-03-28T17:55:28.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fe256e1da1] 2024-03-28T17:55:28.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fe256e42d5] 2024-03-28T17:55:28.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fe25734233] 2024-03-28T17:55:28.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fe25736197] 2024-03-28T17:55:28.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fe2563511b] 2024-03-28T17:55:28.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fe256356bc] 2024-03-28T17:55:28.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fe2561f1ff] 2024-03-28T17:55:28.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f768cf3b488] 2024-03-28T17:55:28.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 16: (DispatchQueue::entry()+0x51f) [0x7f768cf3860f] 2024-03-28T17:55:28.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f768d00be81] 2024-03-28T17:55:28.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f768ba261ca] 2024-03-28T17:55:28.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 19: clone() 2024-03-28T17:55:28.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:55:28.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 2024-03-28T17:55:28.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: -1> 2024-03-28T17:55:28.401+0000 7f7681400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7681400700 time 2024-03-28T17:55:28.399559+0000 2024-03-28T17:55:28.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:55:28.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 2024-03-28T17:55:28.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:28.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f768cca94a1] 2024-03-28T17:55:28.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f768cca9667] 2024-03-28T17:55:28.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fe25867223] 2024-03-28T17:55:28.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fe256e15d6] 2024-03-28T17:55:28.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fe256e1da1] 2024-03-28T17:55:28.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fe256e42d5] 2024-03-28T17:55:28.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fe25734233] 2024-03-28T17:55:28.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fe25736197] 2024-03-28T17:55:28.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fe2563511b] 2024-03-28T17:55:28.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fe256356bc] 2024-03-28T17:55:28.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fe2561f1ff] 2024-03-28T17:55:28.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f768cf3b488] 2024-03-28T17:55:28.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 13: (DispatchQueue::entry()+0x51f) [0x7f768cf3860f] 2024-03-28T17:55:28.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f768d00be81] 2024-03-28T17:55:28.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f768ba261ca] 2024-03-28T17:55:28.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 16: clone() 2024-03-28T17:55:28.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 2024-03-28T17:55:28.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 0> 2024-03-28T17:55:28.404+0000 7f7681400700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:55:28.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: in thread 7f7681400700 thread_name:ms_dispatch 2024-03-28T17:55:28.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 2024-03-28T17:55:28.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:28.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f768ba30d20] 2024-03-28T17:55:28.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 2: gsignal() 2024-03-28T17:55:28.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 3: abort() 2024-03-28T17:55:28.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f768cca94fb] 2024-03-28T17:55:28.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f768cca9667] 2024-03-28T17:55:28.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fe25867223] 2024-03-28T17:55:28.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fe256e15d6] 2024-03-28T17:55:28.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fe256e1da1] 2024-03-28T17:55:28.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fe256e42d5] 2024-03-28T17:55:28.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fe25734233] 2024-03-28T17:55:28.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fe25736197] 2024-03-28T17:55:28.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fe2563511b] 2024-03-28T17:55:28.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fe256356bc] 2024-03-28T17:55:28.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fe2561f1ff] 2024-03-28T17:55:28.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f768cf3b488] 2024-03-28T17:55:28.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 16: (DispatchQueue::entry()+0x51f) [0x7f768cf3860f] 2024-03-28T17:55:28.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f768d00be81] 2024-03-28T17:55:28.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f768ba261ca] 2024-03-28T17:55:28.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 19: clone() 2024-03-28T17:55:28.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:55:28.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91096]: 2024-03-28T17:55:28.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:28 smithi045 podman[91264]: 2024-03-28 17:55:28.509626573 +0000 UTC m=+0.035000800 container died 54ff618292b634118f4309d5f72acd6bbeade0ae565376aeab753e21496b2ab4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-28T17:55:28.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:28 smithi071 ceph-mon[36597]: pgmap v215: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:55:28.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:28 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/67226369,v1:172.21.15.45:6837/67226369] up:rejoin 2024-03-28T17:55:28.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:28 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:28.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:28 smithi071 ceph-mon[36597]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-03-28T17:55:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:28 smithi151 ceph-mon[36481]: pgmap v215: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:55:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:28 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/67226369,v1:172.21.15.45:6837/67226369] up:rejoin 2024-03-28T17:55:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:28 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:28 smithi151 ceph-mon[36481]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-03-28T17:55:28.875 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 181 sec latency 296772.060 ms 2024-03-28T17:55:29.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:29 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:55:29.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:29 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:55:29.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:29 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/67226369,v1:172.21.15.45:6837/67226369] up:active 2024-03-28T17:55:29.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:29 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:29.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:29 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:55:29.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:29 smithi045 podman[91264]: 2024-03-28 17:55:29.171507591 +0000 UTC m=+0.696881843 container remove 54ff618292b634118f4309d5f72acd6bbeade0ae565376aeab753e21496b2ab4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-28T17:55:29.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:29 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:55:29.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:29 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:55:29.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:29 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:55:29.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:29 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/67226369,v1:172.21.15.45:6837/67226369] up:active 2024-03-28T17:55:29.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:29 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:29.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:29 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:55:29.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:29 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:55:29.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:29 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:55:29.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:29 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/67226369,v1:172.21.15.45:6837/67226369] up:active 2024-03-28T17:55:29.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:29 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:29.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:29 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:55:29.875 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 182 sec latency 297772.233 ms 2024-03-28T17:55:30.165 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:29 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-28T17:55:30.166 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:29 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Consumed 1.623s CPU time. 2024-03-28T17:55:30.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:30 smithi045 ceph-mon[31773]: pgmap v216: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:55:30.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:30 smithi071 ceph-mon[36597]: pgmap v216: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:55:30.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:30 smithi151 ceph-mon[36481]: pgmap v216: 97 pgs: 97 active+clean; 27 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:55:30.875 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 183 sec latency 298772.390 ms 2024-03-28T17:55:31.875 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 184 sec latency 299772.531 ms 2024-03-28T17:55:32.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:32 smithi045 ceph-mon[31773]: pgmap v217: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:55:32.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:32 smithi071 ceph-mon[36597]: pgmap v217: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:55:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:32 smithi151 ceph-mon[36481]: pgmap v217: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:55:32.875 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 185 sec latency 300772.663 ms 2024-03-28T17:55:33.876 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 186 sec latency 301772.820 ms 2024-03-28T17:55:34.503 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:55:34.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:55:34.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:34 smithi045 ceph-mon[31773]: pgmap v218: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:55:34.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.739+0000 7f9239800640 1 -- 172.21.15.45:0/1143739782 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9234074370 msgr2=0x7f923410d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:55:34.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.739+0000 7f9239800640 1 --2- 172.21.15.45:0/1143739782 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9234074370 0x7f923410d890 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f921c009900 tx=0x7f921c02bcc0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.739+0000 7f9239800640 1 -- 172.21.15.45:0/1143739782 shutdown_connections 2024-03-28T17:55:34.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.739+0000 7f9239800640 1 --2- 172.21.15.45:0/1143739782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f923410ddd0 0x7f92341101c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.739+0000 7f9239800640 1 --2- 172.21.15.45:0/1143739782 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9234074370 0x7f923410d890 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.739+0000 7f9239800640 1 --2- 172.21.15.45:0/1143739782 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9234073a50 0x7f9234073e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.739+0000 7f9239800640 1 -- 172.21.15.45:0/1143739782 >> 172.21.15.45:0/1143739782 conn(0x7f92340fd040 msgr2=0x7f92340ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:55:34.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.739+0000 7f9239800640 1 -- 172.21.15.45:0/1143739782 shutdown_connections 2024-03-28T17:55:34.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.739+0000 7f9239800640 1 -- 172.21.15.45:0/1143739782 wait complete. 2024-03-28T17:55:34.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.740+0000 7f9239800640 1 Processor -- start 2024-03-28T17:55:34.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.740+0000 7f9239800640 1 -- start start 2024-03-28T17:55:34.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.741+0000 7f9239800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9234073a50 0x7f923419a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:34.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.741+0000 7f9239800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9234074370 0x7f923419a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:34.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.741+0000 7f9239800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f923410ddd0 0x7f923419e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:34.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.741+0000 7f9239800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9234113ab0 con 0x7f923410ddd0 2024-03-28T17:55:34.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.741+0000 7f9239800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9234113930 con 0x7f9234074370 2024-03-28T17:55:34.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.741+0000 7f9239800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9234113c30 con 0x7f9234073a50 2024-03-28T17:55:34.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.741+0000 7f9233400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9234074370 0x7f923419a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:34.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.741+0000 7f9233400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9234074370 0x7f923419a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:42012/0 (socket says 172.21.15.45:42012) 2024-03-28T17:55:34.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.741+0000 7f9238e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f923410ddd0 0x7f923419e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:34.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.741+0000 7f9233400640 1 -- 172.21.15.45:0/2568712165 learned_addr learned my addr 172.21.15.45:0/2568712165 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:55:34.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.741+0000 7f9233e00640 1 --2- 172.21.15.45:0/2568712165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9234073a50 0x7f923419a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:34.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.741+0000 7f9233400640 1 -- 172.21.15.45:0/2568712165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9234073a50 msgr2=0x7f923419a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:55:34.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.741+0000 7f9233400640 1 --2- 172.21.15.45:0/2568712165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9234073a50 0x7f923419a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.741+0000 7f9233400640 1 -- 172.21.15.45:0/2568712165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f923410ddd0 msgr2=0x7f923419e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:55:34.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.741+0000 7f9233400640 1 --2- 172.21.15.45:0/2568712165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f923410ddd0 0x7f923419e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.741+0000 7f9233400640 1 -- 172.21.15.45:0/2568712165 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f923419f020 con 0x7f9234074370 2024-03-28T17:55:34.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.741+0000 7f9238e00640 1 --2- 172.21.15.45:0/2568712165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f923410ddd0 0x7f923419e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:55:34.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.742+0000 7f9233400640 1 --2- 172.21.15.45:0/2568712165 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9234074370 0x7f923419a5c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f921c036040 tx=0x7f921c02cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:55:34.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.742+0000 7f9230c00640 1 -- 172.21.15.45:0/2568712165 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c043020 con 0x7f9234074370 2024-03-28T17:55:34.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.742+0000 7f9230c00640 1 -- 172.21.15.45:0/2568712165 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f921c0054e0 con 0x7f9234074370 2024-03-28T17:55:34.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.742+0000 7f9239800640 1 -- 172.21.15.45:0/2568712165 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f923419f2b0 con 0x7f9234074370 2024-03-28T17:55:34.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.742+0000 7f9230c00640 1 -- 172.21.15.45:0/2568712165 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c003c20 con 0x7f9234074370 2024-03-28T17:55:34.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.742+0000 7f9239800640 1 -- 172.21.15.45:0/2568712165 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92341a6ba0 con 0x7f9234074370 2024-03-28T17:55:34.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.745+0000 7f9230c00640 1 -- 172.21.15.45:0/2568712165 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f921c046020 con 0x7f9234074370 2024-03-28T17:55:34.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.746+0000 7f923da74740 1 Processor -- start 2024-03-28T17:55:34.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.746+0000 7f923da74740 1 -- start start 2024-03-28T17:55:34.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.746+0000 7f923da74740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5634db172dd0 0x5634db171b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:34.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.746+0000 7f923da74740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5634db1720f0 0x5634db172570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:34.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.746+0000 7f923da74740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db17b460 0x5634db17d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:34.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.746+0000 7f923da74740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634db0d6250 con 0x5634db1720f0 2024-03-28T17:55:34.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.746+0000 7f923da74740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634db07ea30 con 0x5634db17b460 2024-03-28T17:55:34.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.746+0000 7f923da74740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634db0d22f0 con 0x5634db172dd0 2024-03-28T17:55:34.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.747+0000 7f9238e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5634db172dd0 0x5634db171b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:34.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.747+0000 7f9233400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db17b460 0x5634db17d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:34.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.747+0000 7f9233e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5634db1720f0 0x5634db172570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:34.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.747+0000 7f9238e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5634db172dd0 0x5634db171b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:56700/0 (socket says 172.21.15.45:56700) 2024-03-28T17:55:34.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.747+0000 7f9238e00640 1 -- 172.21.15.45:0/2808241318 learned_addr learned my addr 172.21.15.45:0/2808241318 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:55:34.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.747+0000 7f9230c00640 1 --2- 172.21.15.45:0/2568712165 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f92140a2970 0x7f92140a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:34.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.747+0000 7f9233400640 1 -- 172.21.15.45:0/2808241318 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5634db172dd0 msgr2=0x5634db171b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:55:34.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.747+0000 7f9233400640 1 --2- 172.21.15.45:0/2808241318 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5634db172dd0 0x5634db171b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.747+0000 7f9233400640 1 -- 172.21.15.45:0/2808241318 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5634db1720f0 msgr2=0x5634db172570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:55:34.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.747+0000 7f9233400640 1 --2- 172.21.15.45:0/2808241318 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5634db1720f0 0x5634db172570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.747+0000 7f9233400640 1 -- 172.21.15.45:0/2808241318 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634db069690 con 0x5634db17b460 2024-03-28T17:55:34.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.747+0000 7f9230c00640 1 -- 172.21.15.45:0/2568712165 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 10060+0+0 (secure 0 0 0) 0x7f921c0f1560 con 0x7f9234074370 2024-03-28T17:55:34.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.747+0000 7f9238e00640 1 --2- 172.21.15.45:0/2568712165 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f92140a2970 0x7f92140a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:34.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.748+0000 7f9233400640 1 --2- 172.21.15.45:0/2808241318 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db17b460 0x5634db17d850 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f921c0ee760 tx=0x7f921c04f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1062f256de9ce49c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:55:34.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.748+0000 7f9229600640 1 -- 172.21.15.45:0/2808241318 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c0dd070 con 0x5634db17b460 2024-03-28T17:55:34.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.748+0000 7f9229600640 1 -- 172.21.15.45:0/2808241318 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f921c005690 con 0x5634db17b460 2024-03-28T17:55:34.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.748+0000 7f9229600640 1 -- 172.21.15.45:0/2808241318 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c0bdd70 con 0x5634db17b460 2024-03-28T17:55:34.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.748+0000 7f923da74740 1 -- 172.21.15.45:0/2808241318 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db17b460 msgr2=0x5634db17d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:55:34.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.748+0000 7f923da74740 1 --2- 172.21.15.45:0/2808241318 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db17b460 0x5634db17d850 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f921c0ee760 tx=0x7f921c04f000 comp rx=0 tx=0).stop 2024-03-28T17:55:34.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.748+0000 7f9238e00640 1 --2- 172.21.15.45:0/2568712165 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f92140a2970 0x7f92140a4e30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f922c005c40 tx=0x7f922c004a40 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:55:34.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.749+0000 7f923da74740 1 -- 172.21.15.45:0/2808241318 shutdown_connections 2024-03-28T17:55:34.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.749+0000 7f923da74740 1 --2- 172.21.15.45:0/2808241318 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db17b460 0x5634db17d850 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.749+0000 7f923da74740 1 --2- 172.21.15.45:0/2808241318 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5634db1720f0 0x5634db172570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.749+0000 7f923da74740 1 --2- 172.21.15.45:0/2808241318 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5634db172dd0 0x5634db171b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.749+0000 7f923da74740 1 -- 172.21.15.45:0/2808241318 >> 172.21.15.45:0/2808241318 conn(0x5634db139200 msgr2=0x5634db16cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:55:34.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.749+0000 7f923da74740 1 -- 172.21.15.45:0/2808241318 shutdown_connections 2024-03-28T17:55:34.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.749+0000 7f923da74740 1 -- 172.21.15.45:0/2808241318 wait complete. 2024-03-28T17:55:34.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.749+0000 7f923da74740 1 -- start start 2024-03-28T17:55:34.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.749+0000 7f923da74740 1 Processor -- start 2024-03-28T17:55:34.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.750+0000 7f923da74740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db1720f0 0x5634db17aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:34.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.750+0000 7f923da74740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5634db172dd0 0x5634db177e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:34.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.750+0000 7f923da74740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5634db17b460 0x5634db178360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:34.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.750+0000 7f923da74740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634db17e090 con 0x5634db17b460 2024-03-28T17:55:34.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.750+0000 7f923da74740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634db091e80 con 0x5634db1720f0 2024-03-28T17:55:34.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.750+0000 7f923da74740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634db17e210 con 0x5634db172dd0 2024-03-28T17:55:34.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.750+0000 7f923da74740 10 client.0 _ll_register_callbacks cb 0x5634db0a4130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:55:34.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.750+0000 7f9233e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db1720f0 0x5634db17aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:34.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.750+0000 7f9233e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db1720f0 0x5634db17aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:42030/0 (socket says 172.21.15.45:42030) 2024-03-28T17:55:34.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.750+0000 7f9233e00640 1 -- 172.21.15.45:0/4106062893 learned_addr learned my addr 172.21.15.45:0/4106062893 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:55:34.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.750+0000 7f9238e00640 1 --2- 172.21.15.45:0/4106062893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5634db17b460 0x5634db178360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:34.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.750+0000 7f9233400640 1 --2- 172.21.15.45:0/4106062893 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5634db172dd0 0x5634db177e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:34.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.750+0000 7f9233e00640 1 -- 172.21.15.45:0/4106062893 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5634db172dd0 msgr2=0x5634db177e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:55:34.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.750+0000 7f9233e00640 1 --2- 172.21.15.45:0/4106062893 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5634db172dd0 0x5634db177e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.750+0000 7f9233e00640 1 -- 172.21.15.45:0/4106062893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5634db17b460 msgr2=0x5634db178360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:55:34.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.750+0000 7f9233e00640 1 --2- 172.21.15.45:0/4106062893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5634db17b460 0x5634db178360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.750+0000 7f9233e00640 1 -- 172.21.15.45:0/4106062893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5634daeb47d0 con 0x5634db1720f0 2024-03-28T17:55:34.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.750+0000 7f9238e00640 1 --2- 172.21.15.45:0/4106062893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5634db17b460 0x5634db178360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:55:34.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.751+0000 7f9233e00640 1 --2- 172.21.15.45:0/4106062893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db1720f0 0x5634db17aa90 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f92240126b0 tx=0x7f9224012b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9969095118238d51 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:55:34.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.751+0000 7f9222000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:55:34.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.751+0000 7f9222000640 1 -- 172.21.15.45:0/4106062893 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9224004260 con 0x5634db1720f0 2024-03-28T17:55:34.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.751+0000 7f9222000640 1 -- 172.21.15.45:0/4106062893 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 10060+0+0 (secure 0 0 0) 0x7f9224004400 con 0x5634db1720f0 2024-03-28T17:55:34.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.751+0000 7f923da74740 1 -- 172.21.15.45:0/4106062893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5634db0e3790 con 0x5634db1720f0 2024-03-28T17:55:34.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.751+0000 7f9222000640 1 -- 172.21.15.45:0/4106062893 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=143) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9224025050 con 0x5634db1720f0 2024-03-28T17:55:34.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.751+0000 7f923da74740 10 client.26171 fetch_fsmap learned FSMap version 143 2024-03-28T17:55:34.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.751+0000 7f923da74740 1 -- 172.21.15.45:0/4106062893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=143,monmap=4+}) v3 -- 0x5634dadea3f0 con 0x5634db1720f0 2024-03-28T17:55:34.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.752+0000 7f9222000640 1 -- 172.21.15.45:0/4106062893 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5634db0e3790 con 0x5634db1720f0 2024-03-28T17:55:34.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.752+0000 7f9222000640 1 -- 172.21.15.45:0/4106062893 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 143) v1 ==== 6768+0+0 (secure 0 0 0) 0x7f922400f440 con 0x5634db1720f0 2024-03-28T17:55:34.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.752+0000 7f923da74740 10 client.26171 fetch_fsmap finished waiting for FSMap version 143 2024-03-28T17:55:34.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.752+0000 7f923da74740 10 client.26171 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:55:34.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.752+0000 7f923da74740 20 client.26171 populate_metadata read hostname 'smithi045' 2024-03-28T17:55:34.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.752+0000 7f923da74740 1 --2- 172.21.15.45:0/4106062893 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5634db18b010 0x5634db18d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:34.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.752+0000 7f923da74740 4 client.26171 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:55:34.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.752+0000 7f923da74740 1 -- 172.21.15.45:0/4106062893 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5634db0ddae0 con 0x5634db18b010 2024-03-28T17:55:34.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.752+0000 7f9233400640 1 --2- 172.21.15.45:0/4106062893 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5634db18b010 0x5634db18d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:34.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.752+0000 7f9233400640 1 --2- 172.21.15.45:0/4106062893 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5634db18b010 0x5634db18d400 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7a06f7cad710146 server_cookie=e4278c0f301de580 in_seq=0 out_seq=0 2024-03-28T17:55:34.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.752+0000 7f9222000640 10 client.26171 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:55:34.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.753+0000 7f9222000640 1 -- 172.21.15.45:0/4106062893 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5634db0ddae0 con 0x5634db18b010 2024-03-28T17:55:34.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.753+0000 7f9222000640 10 client.26171 handle_command_reply: tid=0 2024-03-28T17:55:34.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.753+0000 7f923da74740 1 client.26171 shutdown 2024-03-28T17:55:34.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.754+0000 7f923da74740 1 -- 172.21.15.45:0/4106062893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db1720f0 msgr2=0x5634db17aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:55:34.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.754+0000 7f923da74740 1 --2- 172.21.15.45:0/4106062893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db1720f0 0x5634db17aa90 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f92240126b0 tx=0x7f9224012b70 comp rx=0 tx=0).stop 2024-03-28T17:55:34.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.754+0000 7f923da74740 1 -- 172.21.15.45:0/4106062893 shutdown_connections 2024-03-28T17:55:34.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.754+0000 7f923da74740 1 --2- 172.21.15.45:0/4106062893 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5634db18b010 0x5634db18d400 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.754+0000 7f923da74740 1 --2- 172.21.15.45:0/4106062893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5634db17b460 0x5634db178360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.754+0000 7f923da74740 1 --2- 172.21.15.45:0/4106062893 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5634db172dd0 0x5634db177e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.754+0000 7f923da74740 1 --2- 172.21.15.45:0/4106062893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db1720f0 0x5634db17aa90 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.754+0000 7f9222000640 0 client.26171 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:55:34.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.754+0000 7f923da74740 1 -- 172.21.15.45:0/4106062893 >> 172.21.15.45:0/4106062893 conn(0x5634db139200 msgr2=0x5634db16e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:55:34.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.754+0000 7f923da74740 1 -- 172.21.15.45:0/4106062893 shutdown_connections 2024-03-28T17:55:34.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.754+0000 7f923da74740 1 -- 172.21.15.45:0/4106062893 wait complete. 2024-03-28T17:55:34.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.754+0000 7f923da74740 20 client.26171 trim_cache size 0 max 16384 2024-03-28T17:55:34.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.772+0000 7f923da74740 1 Processor -- start 2024-03-28T17:55:34.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.772+0000 7f923da74740 1 -- start start 2024-03-28T17:55:34.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.772+0000 7f923da74740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5634db1720f0 0x5634db176e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:34.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.772+0000 7f923da74740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5634db172dd0 0x5634db17c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:34.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.772+0000 7f923da74740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db17b460 0x5634db17a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:34.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.772+0000 7f923da74740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92341b0aa0 con 0x5634db172dd0 2024-03-28T17:55:34.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.772+0000 7f923da74740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92341b1100 con 0x5634db17b460 2024-03-28T17:55:34.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.772+0000 7f923da74740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92341b1760 con 0x5634db1720f0 2024-03-28T17:55:34.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.772+0000 7f9233e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5634db1720f0 0x5634db176e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:34.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.772+0000 7f9238e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db17b460 0x5634db17a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:34.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.772+0000 7f9233e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5634db1720f0 0x5634db176e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:56724/0 (socket says 172.21.15.45:56724) 2024-03-28T17:55:34.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.772+0000 7f9233e00640 1 -- 172.21.15.45:0/4046407684 learned_addr learned my addr 172.21.15.45:0/4046407684 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:55:34.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.772+0000 7f9233400640 1 --2- 172.21.15.45:0/4046407684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5634db172dd0 0x5634db17c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:34.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.773+0000 7f9233e00640 1 -- 172.21.15.45:0/4046407684 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db17b460 msgr2=0x5634db17a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:55:34.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.773+0000 7f9233e00640 1 --2- 172.21.15.45:0/4046407684 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db17b460 0x5634db17a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.773+0000 7f9233e00640 1 -- 172.21.15.45:0/4046407684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5634db172dd0 msgr2=0x5634db17c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:55:34.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.773+0000 7f9233e00640 1 --2- 172.21.15.45:0/4046407684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5634db172dd0 0x5634db17c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.773+0000 7f9233e00640 1 -- 172.21.15.45:0/4046407684 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634db18d770 con 0x5634db1720f0 2024-03-28T17:55:34.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.773+0000 7f9233e00640 1 --2- 172.21.15.45:0/4046407684 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5634db1720f0 0x5634db176e00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9224011900 tx=0x7f9224009640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23937030bb07e74f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:55:34.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.773+0000 7f9228c00640 1 -- 172.21.15.45:0/4046407684 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922402d020 con 0x5634db1720f0 2024-03-28T17:55:34.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.774+0000 7f9228c00640 1 -- 172.21.15.45:0/4046407684 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f922400f9b0 con 0x5634db1720f0 2024-03-28T17:55:34.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.774+0000 7f9228c00640 1 -- 172.21.15.45:0/4046407684 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9224022040 con 0x5634db1720f0 2024-03-28T17:55:34.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.774+0000 7f923da74740 1 -- 172.21.15.45:0/4046407684 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5634db1720f0 msgr2=0x5634db176e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:55:34.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.774+0000 7f923da74740 1 --2- 172.21.15.45:0/4046407684 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5634db1720f0 0x5634db176e00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9224011900 tx=0x7f9224009640 comp rx=0 tx=0).stop 2024-03-28T17:55:34.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.774+0000 7f923da74740 1 -- 172.21.15.45:0/4046407684 shutdown_connections 2024-03-28T17:55:34.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.774+0000 7f923da74740 1 --2- 172.21.15.45:0/4046407684 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db17b460 0x5634db17a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.774+0000 7f923da74740 1 --2- 172.21.15.45:0/4046407684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5634db172dd0 0x5634db17c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.774+0000 7f923da74740 1 --2- 172.21.15.45:0/4046407684 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5634db1720f0 0x5634db176e00 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.774+0000 7f923da74740 1 -- 172.21.15.45:0/4046407684 >> 172.21.15.45:0/4046407684 conn(0x5634db139200 msgr2=0x5634db1766f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:55:34.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.774+0000 7f923da74740 1 -- 172.21.15.45:0/4046407684 shutdown_connections 2024-03-28T17:55:34.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.775+0000 7f923da74740 1 -- 172.21.15.45:0/4046407684 wait complete. 2024-03-28T17:55:34.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.775+0000 7f923da74740 1 -- start start 2024-03-28T17:55:34.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.775+0000 7f923da74740 1 Processor -- start 2024-03-28T17:55:34.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.776+0000 7f923da74740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5634db1720f0 0x5634db174270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:34.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.776+0000 7f923da74740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5634db172dd0 0x5634db1747b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:34.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.776+0000 7f923da74740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db17b460 0x5634db174cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.776+0000 7f923da74740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92341ae040 con 0x5634db1720f0 2024-03-28T17:55:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.776+0000 7f923da74740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92341af2e0 con 0x5634db17b460 2024-03-28T17:55:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.776+0000 7f923da74740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92341ae640 con 0x5634db172dd0 2024-03-28T17:55:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.776+0000 7f923da74740 10 client.0 _ll_register_callbacks cb 0x7f92240097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:55:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.776+0000 7f9238e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db17b460 0x5634db174cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.776+0000 7f9233e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5634db1720f0 0x5634db174270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.776+0000 7f9238e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db17b460 0x5634db174cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:42048/0 (socket says 172.21.15.45:42048) 2024-03-28T17:55:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.776+0000 7f9238e00640 1 -- 172.21.15.45:0/3820066133 learned_addr learned my addr 172.21.15.45:0/3820066133 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:55:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.776+0000 7f9233400640 1 --2- 172.21.15.45:0/3820066133 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5634db172dd0 0x5634db1747b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.776+0000 7f9238e00640 1 -- 172.21.15.45:0/3820066133 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5634db172dd0 msgr2=0x5634db1747b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:55:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.776+0000 7f9238e00640 1 --2- 172.21.15.45:0/3820066133 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5634db172dd0 0x5634db1747b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.776+0000 7f9238e00640 1 -- 172.21.15.45:0/3820066133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5634db1720f0 msgr2=0x5634db174270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:55:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.776+0000 7f9238e00640 1 --2- 172.21.15.45:0/3820066133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5634db1720f0 0x5634db174270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.776+0000 7f9238e00640 1 -- 172.21.15.45:0/3820066133 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5634dadaa0f0 con 0x5634db17b460 2024-03-28T17:55:34.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.777+0000 7f9238e00640 1 --2- 172.21.15.45:0/3820066133 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db17b460 0x5634db174cf0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f922c010ed0 tx=0x7f922c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2228e8f8cf0b27ab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:55:34.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.777+0000 7f9222000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:55:34.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.777+0000 7f9222000640 1 -- 172.21.15.45:0/3820066133 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c01e440 con 0x5634db17b460 2024-03-28T17:55:34.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.777+0000 7f9222000640 1 -- 172.21.15.45:0/3820066133 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 10060+0+0 (secure 0 0 0) 0x7f922c01e5e0 con 0x5634db17b460 2024-03-28T17:55:34.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.777+0000 7f923da74740 1 -- 172.21.15.45:0/3820066133 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f922400f9b0 con 0x5634db17b460 2024-03-28T17:55:34.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.777+0000 7f9222000640 1 -- 172.21.15.45:0/3820066133 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=143) v2 ==== 24+0+0 (secure 0 0 0) 0x7f922c01e7e0 con 0x5634db17b460 2024-03-28T17:55:34.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.777+0000 7f923da74740 10 client.26177 fetch_fsmap learned FSMap version 143 2024-03-28T17:55:34.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.777+0000 7f923da74740 1 -- 172.21.15.45:0/3820066133 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=143,monmap=4+}) v3 -- 0x5634db1e67e0 con 0x5634db17b460 2024-03-28T17:55:34.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.777+0000 7f9222000640 1 -- 172.21.15.45:0/3820066133 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f922400f9b0 con 0x5634db17b460 2024-03-28T17:55:34.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.777+0000 7f9222000640 1 -- 172.21.15.45:0/3820066133 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 143) v1 ==== 6768+0+0 (secure 0 0 0) 0x7f922c02aac0 con 0x5634db17b460 2024-03-28T17:55:34.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.778+0000 7f923da74740 10 client.26177 fetch_fsmap finished waiting for FSMap version 143 2024-03-28T17:55:34.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.778+0000 7f923da74740 10 client.26177 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:55:34.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.778+0000 7f923da74740 20 client.26177 populate_metadata read hostname 'smithi045' 2024-03-28T17:55:34.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.778+0000 7f923da74740 1 --2- 172.21.15.45:0/3820066133 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5634db1e6f40 0x5634db1e9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:55:34.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.778+0000 7f923da74740 4 client.26177 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:55:34.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.778+0000 7f923da74740 1 -- 172.21.15.45:0/3820066133 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5634db1786f0 con 0x5634db1e6f40 2024-03-28T17:55:34.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.778+0000 7f9233e00640 1 --2- 172.21.15.45:0/3820066133 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5634db1e6f40 0x5634db1e9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:55:34.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.778+0000 7f9233e00640 1 --2- 172.21.15.45:0/3820066133 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5634db1e6f40 0x5634db1e9360 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebacf5e7a1d84f4c server_cookie=9cb1ea8bd675dcbe in_seq=0 out_seq=0 2024-03-28T17:55:34.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.778+0000 7f9222000640 10 client.26177 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:55:34.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.778+0000 7f9222000640 1 -- 172.21.15.45:0/3820066133 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5634db1786f0 con 0x5634db1e6f40 2024-03-28T17:55:34.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.778+0000 7f9222000640 10 client.26177 handle_command_reply: tid=0 2024-03-28T17:55:34.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.778+0000 7f923da74740 1 client.26177 shutdown 2024-03-28T17:55:34.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.779+0000 7f923da74740 1 -- 172.21.15.45:0/3820066133 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db17b460 msgr2=0x5634db174cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:55:34.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.779+0000 7f923da74740 1 --2- 172.21.15.45:0/3820066133 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db17b460 0x5634db174cf0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f922c010ed0 tx=0x7f922c004ab0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.779+0000 7f923da74740 1 -- 172.21.15.45:0/3820066133 shutdown_connections 2024-03-28T17:55:34.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.779+0000 7f923da74740 1 --2- 172.21.15.45:0/3820066133 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5634db1e6f40 0x5634db1e9360 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.779+0000 7f923da74740 1 --2- 172.21.15.45:0/3820066133 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5634db17b460 0x5634db174cf0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.779+0000 7f923da74740 1 --2- 172.21.15.45:0/3820066133 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5634db172dd0 0x5634db1747b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.779+0000 7f923da74740 1 --2- 172.21.15.45:0/3820066133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5634db1720f0 0x5634db174270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.779+0000 7f9222000640 0 client.26177 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:55:34.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.779+0000 7f923da74740 1 -- 172.21.15.45:0/3820066133 >> 172.21.15.45:0/3820066133 conn(0x5634db139200 msgr2=0x5634db176f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:55:34.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.779+0000 7f923da74740 1 -- 172.21.15.45:0/3820066133 shutdown_connections 2024-03-28T17:55:34.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.779+0000 7f923da74740 1 -- 172.21.15.45:0/3820066133 wait complete. 2024-03-28T17:55:34.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.779+0000 7f923da74740 20 client.26177 trim_cache size 0 max 16384 2024-03-28T17:55:34.783 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:55:34.783 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T17:55:34.784 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T17:55:34.784 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T17:55:34.784 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T17:55:34.784 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T17:55:34.784 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T17:55:34.784 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:55:34.784 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:55:34.784 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:55:34.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.780+0000 7f9239800640 1 -- 172.21.15.45:0/2568712165 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f92140a2970 msgr2=0x7f92140a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:55:34.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.780+0000 7f9239800640 1 --2- 172.21.15.45:0/2568712165 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f92140a2970 0x7f92140a4e30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f922c005c40 tx=0x7f922c004a40 comp rx=0 tx=0).stop 2024-03-28T17:55:34.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.780+0000 7f9239800640 1 -- 172.21.15.45:0/2568712165 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9234074370 msgr2=0x7f923419a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:55:34.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.780+0000 7f9239800640 1 --2- 172.21.15.45:0/2568712165 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9234074370 0x7f923419a5c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f921c036040 tx=0x7f921c02cbc0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.780+0000 7f9239800640 1 -- 172.21.15.45:0/2568712165 shutdown_connections 2024-03-28T17:55:34.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.780+0000 7f9239800640 1 --2- 172.21.15.45:0/2568712165 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f92140a2970 0x7f92140a4e30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.780+0000 7f9239800640 1 --2- 172.21.15.45:0/2568712165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f923410ddd0 0x7f923419e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.780+0000 7f9239800640 1 --2- 172.21.15.45:0/2568712165 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9234074370 0x7f923419a5c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.780+0000 7f9239800640 1 --2- 172.21.15.45:0/2568712165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9234073a50 0x7f923419a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:55:34.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.780+0000 7f9239800640 1 -- 172.21.15.45:0/2568712165 >> 172.21.15.45:0/2568712165 conn(0x7f92340fd040 msgr2=0x7f92340fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:55:34.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.780+0000 7f9239800640 1 -- 172.21.15.45:0/2568712165 shutdown_connections 2024-03-28T17:55:34.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:55:34.780+0000 7f9239800640 1 -- 172.21.15.45:0/2568712165 wait complete. 2024-03-28T17:55:34.797 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T17:55:34.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:34 smithi071 ceph-mon[36597]: pgmap v218: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:55:34.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:34 smithi151 ceph-mon[36481]: pgmap v218: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:55:34.876 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 187 sec latency 302772.919 ms 2024-03-28T17:55:35.876 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 188 sec latency 303773.072 ms 2024-03-28T17:55:36.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:36 smithi045 ceph-mon[31773]: pgmap v219: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 286 B/s wr, 13 op/s 2024-03-28T17:55:36.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:36 smithi071 ceph-mon[36597]: pgmap v219: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 286 B/s wr, 13 op/s 2024-03-28T17:55:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:36 smithi151 ceph-mon[36481]: pgmap v219: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 286 B/s wr, 13 op/s 2024-03-28T17:55:36.876 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 189 sec latency 304773.215 ms 2024-03-28T17:55:37.876 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 190 sec latency 305773.368 ms 2024-03-28T17:55:38.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:38 smithi071 ceph-mon[36597]: pgmap v220: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:55:38.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:38 smithi151 ceph-mon[36481]: pgmap v220: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:55:38.876 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 191 sec latency 306773.462 ms 2024-03-28T17:55:38.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:38 smithi045 ceph-mon[31773]: pgmap v220: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:55:39.876 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 192 sec latency 307773.556 ms 2024-03-28T17:55:40.322 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:39 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 5. 2024-03-28T17:55:40.322 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:39 smithi045 systemd[1]: Stopped Ceph mds.d for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:55:40.322 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:39 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Consumed 1.623s CPU time. 2024-03-28T17:55:40.322 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:39 smithi045 systemd[1]: Starting Ceph mds.d for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:55:40.635 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:40 smithi045 ceph-mon[31773]: pgmap v221: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:55:40.635 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:40 smithi045 podman[91421]: 2024-03-28 17:55:40.231786931 +0000 UTC m=+0.044070619 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:55:40.635 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:40 smithi045 podman[91421]: 2024-03-28 17:55:40.363391625 +0000 UTC m=+0.175675307 container create 437610febe631b6b7d7658b54c04e8d7723afc51b897a47de074b595a79e7eaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-28T17:55:40.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:40 smithi071 ceph-mon[36597]: pgmap v221: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:55:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:40 smithi151 ceph-mon[36481]: pgmap v221: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:55:40.877 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 193 sec latency 308773.714 ms 2024-03-28T17:55:40.916 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:40 smithi045 podman[91421]: 2024-03-28 17:55:40.635421314 +0000 UTC m=+0.447705003 container init 437610febe631b6b7d7658b54c04e8d7723afc51b897a47de074b595a79e7eaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-28T17:55:40.916 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:40 smithi045 podman[91421]: 2024-03-28 17:55:40.646141907 +0000 UTC m=+0.458425592 container start 437610febe631b6b7d7658b54c04e8d7723afc51b897a47de074b595a79e7eaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-28T17:55:40.916 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:40 smithi045 ceph-mds[91439]: -- 172.21.15.45:0/2819825688 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557e91c9ba40 con 0x557e91c6f000 2024-03-28T17:55:40.916 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: starting mds.d at 2024-03-28T17:55:40.916 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:40 smithi045 bash[91421]: 437610febe631b6b7d7658b54c04e8d7723afc51b897a47de074b595a79e7eaf 2024-03-28T17:55:40.917 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:55:40 smithi045 systemd[1]: Started Ceph mds.d for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:55:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:41 smithi071 ceph-mon[36597]: daemon mds.d restarted 2024-03-28T17:55:41.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:41 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:55:41.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:41 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T17:55:41.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:41 smithi071 ceph-mon[36597]: osdmap e107: 12 total, 12 up, 12 in 2024-03-28T17:55:41.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:41 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2011326175,v1:172.21.15.45:6837/2011326175] up:boot 2024-03-28T17:55:41.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:41 smithi071 ceph-mon[36597]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-03-28T17:55:41.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:41 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:55:41.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:41 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:41 smithi151 ceph-mon[36481]: daemon mds.d restarted 2024-03-28T17:55:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:41 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:55:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:41 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T17:55:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:41 smithi151 ceph-mon[36481]: osdmap e107: 12 total, 12 up, 12 in 2024-03-28T17:55:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:41 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2011326175,v1:172.21.15.45:6837/2011326175] up:boot 2024-03-28T17:55:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:41 smithi151 ceph-mon[36481]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-03-28T17:55:41.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:41 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:55:41.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:41 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:41.877 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 194 sec latency 309773.914 ms 2024-03-28T17:55:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:41 smithi045 ceph-mon[31773]: daemon mds.d restarted 2024-03-28T17:55:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:41 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:55:41.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:41 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T17:55:41.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:41 smithi045 ceph-mon[31773]: osdmap e107: 12 total, 12 up, 12 in 2024-03-28T17:55:41.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:41 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2011326175,v1:172.21.15.45:6837/2011326175] up:boot 2024-03-28T17:55:41.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:41 smithi045 ceph-mon[31773]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-03-28T17:55:41.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:41 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:55:41.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:41 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:42.877 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 195 sec latency 310774.090 ms 2024-03-28T17:55:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:42 smithi071 ceph-mon[36597]: pgmap v223: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:55:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:42 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/888508431,v1:172.21.15.71:6833/888508431] up:resolve 2024-03-28T17:55:43.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:42 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:43.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:42 smithi151 ceph-mon[36481]: pgmap v223: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:55:43.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:42 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/888508431,v1:172.21.15.71:6833/888508431] up:resolve 2024-03-28T17:55:43.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:42 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:43.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:42 smithi045 ceph-mon[31773]: pgmap v223: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:55:43.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:42 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/888508431,v1:172.21.15.71:6833/888508431] up:resolve 2024-03-28T17:55:43.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:42 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:43.877 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 196 sec latency 311774.230 ms 2024-03-28T17:55:44.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:44 smithi071 ceph-mon[36597]: pgmap v224: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:55:44.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:44 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/888508431,v1:172.21.15.71:6833/888508431] up:reconnect 2024-03-28T17:55:44.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:44 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:44.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:44 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00999983 2024-03-28T17:55:44.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:44 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999983 2024-03-28T17:55:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:44 smithi151 ceph-mon[36481]: pgmap v224: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:55:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:44 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/888508431,v1:172.21.15.71:6833/888508431] up:reconnect 2024-03-28T17:55:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:44 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:44 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00999983 2024-03-28T17:55:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:44 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999983 2024-03-28T17:55:44.877 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 197 sec latency 312774.395 ms 2024-03-28T17:55:44.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:44 smithi045 ceph-mon[31773]: pgmap v224: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:55:44.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:44 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/888508431,v1:172.21.15.71:6833/888508431] up:reconnect 2024-03-28T17:55:44.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:44 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:44.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:44 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00999983 2024-03-28T17:55:44.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:44 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999983 2024-03-28T17:55:45.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:45 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/888508431,v1:172.21.15.71:6833/888508431] up:rejoin 2024-03-28T17:55:45.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:45 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:45.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:45 smithi045 ceph-mon[31773]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-03-28T17:55:45.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:45 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:55:45.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:45 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:55:45.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:45 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/888508431,v1:172.21.15.71:6833/888508431] up:active 2024-03-28T17:55:45.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:45 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:45.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/888508431,v1:172.21.15.71:6833/888508431] up:rejoin 2024-03-28T17:55:45.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:45.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-mon[36597]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-03-28T17:55:45.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:55:45.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:55:45.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/888508431,v1:172.21.15.71:6833/888508431] up:active 2024-03-28T17:55:45.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:45.812 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7e00150700 time 2024-03-28T17:55:45.446919+0000 2024-03-28T17:55:45.812 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:55:45.812 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:45.812 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7e0a77a4a1] 2024-03-28T17:55:45.812 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7e0a77a667] 2024-03-28T17:55:45.812 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55b503fc0223] 2024-03-28T17:55:45.812 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55b503e3a5d6] 2024-03-28T17:55:45.812 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55b503e3ada1] 2024-03-28T17:55:45.812 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55b503e3d2d5] 2024-03-28T17:55:45.812 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55b503e8d233] 2024-03-28T17:55:45.813 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b503e8f197] 2024-03-28T17:55:45.813 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b503d8e11b] 2024-03-28T17:55:45.813 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b503d8e6bc] 2024-03-28T17:55:45.813 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55b503d781ff] 2024-03-28T17:55:45.813 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7e0aa0c488] 2024-03-28T17:55:45.813 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 13: (DispatchQueue::entry()+0x51f) [0x7f7e0aa0960f] 2024-03-28T17:55:45.813 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7e0aadce81] 2024-03-28T17:55:45.813 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7e094f71ca] 2024-03-28T17:55:45.813 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 16: clone() 2024-03-28T17:55:45.813 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 2024-03-28T17:55:45.447+0000 7f7e00150700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7e00150700 time 2024-03-28T17:55:45.446919+0000 2024-03-28T17:55:45.813 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:55:45.813 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 2024-03-28T17:55:45.813 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:45.813 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7e0a77a4a1] 2024-03-28T17:55:45.814 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7e0a77a667] 2024-03-28T17:55:45.814 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55b503fc0223] 2024-03-28T17:55:45.814 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55b503e3a5d6] 2024-03-28T17:55:45.814 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55b503e3ada1] 2024-03-28T17:55:45.814 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55b503e3d2d5] 2024-03-28T17:55:45.814 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55b503e8d233] 2024-03-28T17:55:45.814 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b503e8f197] 2024-03-28T17:55:45.814 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b503d8e11b] 2024-03-28T17:55:45.814 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b503d8e6bc] 2024-03-28T17:55:45.814 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55b503d781ff] 2024-03-28T17:55:45.814 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7e0aa0c488] 2024-03-28T17:55:45.814 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 13: (DispatchQueue::entry()+0x51f) [0x7f7e0aa0960f] 2024-03-28T17:55:45.814 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7e0aadce81] 2024-03-28T17:55:45.814 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7e094f71ca] 2024-03-28T17:55:45.815 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 16: clone() 2024-03-28T17:55:45.815 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 2024-03-28T17:55:45.815 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: *** Caught signal (Aborted) ** 2024-03-28T17:55:45.815 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: in thread 7f7e00150700 thread_name:ms_dispatch 2024-03-28T17:55:45.815 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:45.815 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7e09501d20] 2024-03-28T17:55:45.815 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 2: gsignal() 2024-03-28T17:55:45.815 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 3: abort() 2024-03-28T17:55:45.815 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7e0a77a4fb] 2024-03-28T17:55:45.815 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7e0a77a667] 2024-03-28T17:55:45.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55b503fc0223] 2024-03-28T17:55:45.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55b503e3a5d6] 2024-03-28T17:55:45.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55b503e3ada1] 2024-03-28T17:55:45.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55b503e3d2d5] 2024-03-28T17:55:45.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55b503e8d233] 2024-03-28T17:55:45.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b503e8f197] 2024-03-28T17:55:45.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b503d8e11b] 2024-03-28T17:55:45.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b503d8e6bc] 2024-03-28T17:55:45.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55b503d781ff] 2024-03-28T17:55:45.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7e0aa0c488] 2024-03-28T17:55:45.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 16: (DispatchQueue::entry()+0x51f) [0x7f7e0aa0960f] 2024-03-28T17:55:45.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7e0aadce81] 2024-03-28T17:55:45.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7e094f71ca] 2024-03-28T17:55:45.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 19: clone() 2024-03-28T17:55:45.816 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 2024-03-28T17:55:45.449+0000 7f7e00150700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:55:45.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: in thread 7f7e00150700 thread_name:ms_dispatch 2024-03-28T17:55:45.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 2024-03-28T17:55:45.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:45.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7e09501d20] 2024-03-28T17:55:45.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 2: gsignal() 2024-03-28T17:55:45.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 3: abort() 2024-03-28T17:55:45.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7e0a77a4fb] 2024-03-28T17:55:45.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7e0a77a667] 2024-03-28T17:55:45.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55b503fc0223] 2024-03-28T17:55:45.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55b503e3a5d6] 2024-03-28T17:55:45.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55b503e3ada1] 2024-03-28T17:55:45.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55b503e3d2d5] 2024-03-28T17:55:45.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55b503e8d233] 2024-03-28T17:55:45.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b503e8f197] 2024-03-28T17:55:45.817 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b503d8e11b] 2024-03-28T17:55:45.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b503d8e6bc] 2024-03-28T17:55:45.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55b503d781ff] 2024-03-28T17:55:45.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7e0aa0c488] 2024-03-28T17:55:45.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 16: (DispatchQueue::entry()+0x51f) [0x7f7e0aa0960f] 2024-03-28T17:55:45.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7e0aadce81] 2024-03-28T17:55:45.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7e094f71ca] 2024-03-28T17:55:45.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 19: clone() 2024-03-28T17:55:45.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:55:45.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 2024-03-28T17:55:45.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: -1> 2024-03-28T17:55:45.447+0000 7f7e00150700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7e00150700 time 2024-03-28T17:55:45.446919+0000 2024-03-28T17:55:45.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:55:45.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 2024-03-28T17:55:45.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:45.818 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7e0a77a4a1] 2024-03-28T17:55:45.819 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7e0a77a667] 2024-03-28T17:55:45.819 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55b503fc0223] 2024-03-28T17:55:45.819 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55b503e3a5d6] 2024-03-28T17:55:45.819 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55b503e3ada1] 2024-03-28T17:55:45.819 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55b503e3d2d5] 2024-03-28T17:55:45.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55b503e8d233] 2024-03-28T17:55:45.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b503e8f197] 2024-03-28T17:55:45.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b503d8e11b] 2024-03-28T17:55:45.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b503d8e6bc] 2024-03-28T17:55:45.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55b503d781ff] 2024-03-28T17:55:45.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7e0aa0c488] 2024-03-28T17:55:45.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 13: (DispatchQueue::entry()+0x51f) [0x7f7e0aa0960f] 2024-03-28T17:55:45.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7e0aadce81] 2024-03-28T17:55:45.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7e094f71ca] 2024-03-28T17:55:45.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 16: clone() 2024-03-28T17:55:45.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 2024-03-28T17:55:45.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 0> 2024-03-28T17:55:45.449+0000 7f7e00150700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:55:45.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: in thread 7f7e00150700 thread_name:ms_dispatch 2024-03-28T17:55:45.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 2024-03-28T17:55:45.820 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:45.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7e09501d20] 2024-03-28T17:55:45.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 2: gsignal() 2024-03-28T17:55:45.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 3: abort() 2024-03-28T17:55:45.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7e0a77a4fb] 2024-03-28T17:55:45.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7e0a77a667] 2024-03-28T17:55:45.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55b503fc0223] 2024-03-28T17:55:45.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55b503e3a5d6] 2024-03-28T17:55:45.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55b503e3ada1] 2024-03-28T17:55:45.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55b503e3d2d5] 2024-03-28T17:55:45.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55b503e8d233] 2024-03-28T17:55:45.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b503e8f197] 2024-03-28T17:55:45.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b503d8e11b] 2024-03-28T17:55:45.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b503d8e6bc] 2024-03-28T17:55:45.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55b503d781ff] 2024-03-28T17:55:45.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7e0aa0c488] 2024-03-28T17:55:45.821 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 16: (DispatchQueue::entry()+0x51f) [0x7f7e0aa0960f] 2024-03-28T17:55:45.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7e0aadce81] 2024-03-28T17:55:45.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7e094f71ca] 2024-03-28T17:55:45.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 19: clone() 2024-03-28T17:55:45.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:55:45.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 2024-03-28T17:55:45.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: -1> 2024-03-28T17:55:45.447+0000 7f7e00150700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7e00150700 time 2024-03-28T17:55:45.446919+0000 2024-03-28T17:55:45.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:55:45.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 2024-03-28T17:55:45.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:45.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7e0a77a4a1] 2024-03-28T17:55:45.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7e0a77a667] 2024-03-28T17:55:45.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55b503fc0223] 2024-03-28T17:55:45.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55b503e3a5d6] 2024-03-28T17:55:45.822 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55b503e3ada1] 2024-03-28T17:55:45.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55b503e3d2d5] 2024-03-28T17:55:45.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55b503e8d233] 2024-03-28T17:55:45.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b503e8f197] 2024-03-28T17:55:45.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b503d8e11b] 2024-03-28T17:55:45.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b503d8e6bc] 2024-03-28T17:55:45.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55b503d781ff] 2024-03-28T17:55:45.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7e0aa0c488] 2024-03-28T17:55:45.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 13: (DispatchQueue::entry()+0x51f) [0x7f7e0aa0960f] 2024-03-28T17:55:45.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7e0aadce81] 2024-03-28T17:55:45.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7e094f71ca] 2024-03-28T17:55:45.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 16: clone() 2024-03-28T17:55:45.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 2024-03-28T17:55:45.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 0> 2024-03-28T17:55:45.449+0000 7f7e00150700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:55:45.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: in thread 7f7e00150700 thread_name:ms_dispatch 2024-03-28T17:55:45.823 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 2024-03-28T17:55:45.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:55:45.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7e09501d20] 2024-03-28T17:55:45.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 2: gsignal() 2024-03-28T17:55:45.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 3: abort() 2024-03-28T17:55:45.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7e0a77a4fb] 2024-03-28T17:55:45.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7e0a77a667] 2024-03-28T17:55:45.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55b503fc0223] 2024-03-28T17:55:45.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55b503e3a5d6] 2024-03-28T17:55:45.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55b503e3ada1] 2024-03-28T17:55:45.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55b503e3d2d5] 2024-03-28T17:55:45.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55b503e8d233] 2024-03-28T17:55:45.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b503e8f197] 2024-03-28T17:55:45.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b503d8e11b] 2024-03-28T17:55:45.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b503d8e6bc] 2024-03-28T17:55:45.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55b503d781ff] 2024-03-28T17:55:45.824 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7e0aa0c488] 2024-03-28T17:55:45.825 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 16: (DispatchQueue::entry()+0x51f) [0x7f7e0aa0960f] 2024-03-28T17:55:45.825 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7e0aadce81] 2024-03-28T17:55:45.825 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7e094f71ca] 2024-03-28T17:55:45.825 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 19: clone() 2024-03-28T17:55:45.825 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:55:45.825 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71640]: 2024-03-28T17:55:45.825 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:45 smithi071 podman[71690]: 2024-03-28 17:55:45.494241637 +0000 UTC m=+0.018982328 container died ec94b0304e89a7a1069a124dad30146db183d162195c6779dfa66bcc8b3f026e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-28T17:55:45.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:45 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/888508431,v1:172.21.15.71:6833/888508431] up:rejoin 2024-03-28T17:55:45.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:45 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:45.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:45 smithi151 ceph-mon[36481]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-03-28T17:55:45.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:45 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:55:45.826 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:45 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:55:45.826 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:45 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/888508431,v1:172.21.15.71:6833/888508431] up:active 2024-03-28T17:55:45.826 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:45 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:45.878 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 198 sec latency 313774.522 ms 2024-03-28T17:55:46.453 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:46 smithi071 podman[71690]: 2024-03-28 17:55:46.16048246 +0000 UTC m=+0.685223150 container remove ec94b0304e89a7a1069a124dad30146db183d162195c6779dfa66bcc8b3f026e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-28T17:55:46.453 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:46 smithi071 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:55:46.453 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:46 smithi071 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.k.service: Failed with result 'exit-code'. 2024-03-28T17:55:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:46 smithi071 ceph-mon[36597]: pgmap v225: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:55:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:46 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:55:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:46 smithi151 ceph-mon[36481]: pgmap v225: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:55:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:46 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:55:46.877 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 199 sec latency 314774.677 ms 2024-03-28T17:55:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:46 smithi045 ceph-mon[31773]: pgmap v225: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:55:46.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:46 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:55:47.878 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 200 sec latency 315774.838 ms 2024-03-28T17:55:48.793 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:48 smithi045 ceph-mon[31773]: pgmap v226: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:55:48.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:48 smithi071 ceph-mon[36597]: pgmap v226: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:55:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:48 smithi151 ceph-mon[36481]: pgmap v226: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:55:48.878 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 201 sec latency 316775.001 ms 2024-03-28T17:55:49.878 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 202 sec latency 317775.090 ms 2024-03-28T17:55:50.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:50 smithi071 ceph-mon[36597]: pgmap v227: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:55:50.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:50 smithi151 ceph-mon[36481]: pgmap v227: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:55:50.878 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 203 sec latency 318775.237 ms 2024-03-28T17:55:50.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:50 smithi045 ceph-mon[31773]: pgmap v227: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:55:51.878 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 204 sec latency 319775.396 ms 2024-03-28T17:55:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:52 smithi071 ceph-mon[36597]: pgmap v228: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 197 B/s wr, 10 op/s 2024-03-28T17:55:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:52 smithi151 ceph-mon[36481]: pgmap v228: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 197 B/s wr, 10 op/s 2024-03-28T17:55:52.878 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 205 sec latency 320775.542 ms 2024-03-28T17:55:52.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:52 smithi045 ceph-mon[31773]: pgmap v228: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 197 B/s wr, 10 op/s 2024-03-28T17:55:53.879 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 206 sec latency 321775.701 ms 2024-03-28T17:55:54.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:54 smithi071 ceph-mon[36597]: pgmap v229: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T17:55:54.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:54 smithi151 ceph-mon[36481]: pgmap v229: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T17:55:54.879 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 207 sec latency 322775.836 ms 2024-03-28T17:55:54.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:54 smithi045 ceph-mon[31773]: pgmap v229: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T17:55:55.879 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 208 sec latency 323776.003 ms 2024-03-28T17:55:56.792 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:56 smithi071 ceph-mon[36597]: pgmap v230: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:55:56.792 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:56 smithi071 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.k.service: Scheduled restart job, restart counter is at 4. 2024-03-28T17:55:56.792 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:56 smithi071 systemd[1]: Stopped Ceph mds.k for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:55:56.792 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:56 smithi071 systemd[1]: Starting Ceph mds.k for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:55:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:56 smithi151 ceph-mon[36481]: pgmap v230: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:55:56.879 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 209 sec latency 324776.167 ms 2024-03-28T17:55:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:56 smithi045 ceph-mon[31773]: pgmap v230: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:55:57.046 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:56 smithi071 podman[71788]: 2024-03-28 17:55:56.696037929 +0000 UTC m=+0.018196867 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:55:57.046 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:56 smithi071 podman[71788]: 2024-03-28 17:55:56.81041691 +0000 UTC m=+0.132575842 container create 9e5364e23a694d79fd4cb653142ba4a95612691a408bf1c2521e2478cd286f89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True) 2024-03-28T17:55:57.313 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:57 smithi071 podman[71788]: 2024-03-28 17:55:57.046577405 +0000 UTC m=+0.368736357 container init 9e5364e23a694d79fd4cb653142ba4a95612691a408bf1c2521e2478cd286f89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319) 2024-03-28T17:55:57.313 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:57 smithi071 podman[71788]: 2024-03-28 17:55:57.049247713 +0000 UTC m=+0.371406643 container start 9e5364e23a694d79fd4cb653142ba4a95612691a408bf1c2521e2478cd286f89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-28T17:55:57.313 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:57 smithi071 ceph-mds[71807]: -- 172.21.15.71:0/1256046237 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5578b3433a40 con 0x5578b3407000 2024-03-28T17:55:57.313 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:57 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: starting mds.k at 2024-03-28T17:55:57.313 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:57 smithi071 bash[71788]: 9e5364e23a694d79fd4cb653142ba4a95612691a408bf1c2521e2478cd286f89 2024-03-28T17:55:57.313 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:55:57 smithi071 systemd[1]: Started Ceph mds.k for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:55:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:57 smithi071 ceph-mon[36597]: daemon mds.k restarted 2024-03-28T17:55:57.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:57 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:55:57.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:57 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T17:55:57.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:57 smithi071 ceph-mon[36597]: osdmap e108: 12 total, 12 up, 12 in 2024-03-28T17:55:57.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:57 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/1799595971,v1:172.21.15.71:6833/1799595971] up:boot 2024-03-28T17:55:57.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:57 smithi071 ceph-mon[36597]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-03-28T17:55:57.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:57 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:55:57.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:57 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:57 smithi151 ceph-mon[36481]: daemon mds.k restarted 2024-03-28T17:55:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:57 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:55:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:57 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T17:55:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:57 smithi151 ceph-mon[36481]: osdmap e108: 12 total, 12 up, 12 in 2024-03-28T17:55:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:57 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/1799595971,v1:172.21.15.71:6833/1799595971] up:boot 2024-03-28T17:55:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:57 smithi151 ceph-mon[36481]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-03-28T17:55:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:57 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:55:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:57 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:57.879 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 210 sec latency 325776.300 ms 2024-03-28T17:55:57.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:57 smithi045 ceph-mon[31773]: daemon mds.k restarted 2024-03-28T17:55:57.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:57 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:55:57.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:57 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T17:55:57.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:57 smithi045 ceph-mon[31773]: osdmap e108: 12 total, 12 up, 12 in 2024-03-28T17:55:57.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:57 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/1799595971,v1:172.21.15.71:6833/1799595971] up:boot 2024-03-28T17:55:57.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:57 smithi045 ceph-mon[31773]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-03-28T17:55:57.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:57 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-28T17:55:57.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:57 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:replay} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:58.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:58 smithi071 ceph-mon[36597]: pgmap v231: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9 op/s 2024-03-28T17:55:58.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:58 smithi151 ceph-mon[36481]: pgmap v231: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9 op/s 2024-03-28T17:55:58.879 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 211 sec latency 326776.409 ms 2024-03-28T17:55:58.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:58 smithi045 ceph-mon[31773]: pgmap v231: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9 op/s 2024-03-28T17:55:59.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:59 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2011326175,v1:172.21.15.45:6837/2011326175] up:resolve 2024-03-28T17:55:59.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:55:59 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:59.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:59 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2011326175,v1:172.21.15.45:6837/2011326175] up:resolve 2024-03-28T17:55:59.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:55:59 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:55:59.879 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 212 sec latency 327776.555 ms 2024-03-28T17:55:59.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:59 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2011326175,v1:172.21.15.45:6837/2011326175] up:resolve 2024-03-28T17:55:59.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:55:59 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:resolve} 2 up:standby-replay 5 up:standby 2024-03-28T17:56:00.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:00 smithi071 ceph-mon[36597]: pgmap v233: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 10 op/s 2024-03-28T17:56:00.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:00 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:56:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:00 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2011326175,v1:172.21.15.45:6837/2011326175] up:reconnect 2024-03-28T17:56:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:00 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:56:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:00 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00799988 2024-03-28T17:56:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:00 smithi151 ceph-mon[36481]: pgmap v233: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 10 op/s 2024-03-28T17:56:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:00 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:56:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:00 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2011326175,v1:172.21.15.45:6837/2011326175] up:reconnect 2024-03-28T17:56:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:00 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:56:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:00 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00799988 2024-03-28T17:56:00.880 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 213 sec latency 328776.661 ms 2024-03-28T17:56:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:00 smithi045 ceph-mon[31773]: pgmap v233: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 10 op/s 2024-03-28T17:56:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:00 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:56:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:00 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2011326175,v1:172.21.15.45:6837/2011326175] up:reconnect 2024-03-28T17:56:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:00 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:reconnect} 2 up:standby-replay 5 up:standby 2024-03-28T17:56:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:00 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00799988 2024-03-28T17:56:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2011326175,v1:172.21.15.45:6837/2011326175] up:rejoin 2024-03-28T17:56:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:56:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-mon[31773]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-03-28T17:56:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:56:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:56:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2011326175,v1:172.21.15.45:6837/2011326175] up:active 2024-03-28T17:56:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:56:01.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f28fc400700 time 2024-03-28T17:56:01.310917+0000 2024-03-28T17:56:01.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:56:01.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:01.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2907c4d4a1] 2024-03-28T17:56:01.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2907c4d667] 2024-03-28T17:56:01.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557e90763223] 2024-03-28T17:56:01.667 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557e905dd5d6] 2024-03-28T17:56:01.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557e905ddda1] 2024-03-28T17:56:01.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557e905e02d5] 2024-03-28T17:56:01.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557e90630233] 2024-03-28T17:56:01.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557e90632197] 2024-03-28T17:56:01.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557e9053111b] 2024-03-28T17:56:01.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557e905316bc] 2024-03-28T17:56:01.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557e9051b1ff] 2024-03-28T17:56:01.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2907edf488] 2024-03-28T17:56:01.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 13: (DispatchQueue::entry()+0x51f) [0x7f2907edc60f] 2024-03-28T17:56:01.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2907fafe81] 2024-03-28T17:56:01.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f29069ca1ca] 2024-03-28T17:56:01.668 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 16: clone() 2024-03-28T17:56:01.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 2024-03-28T17:56:01.312+0000 7f28fc400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f28fc400700 time 2024-03-28T17:56:01.310917+0000 2024-03-28T17:56:01.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:56:01.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 2024-03-28T17:56:01.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:01.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2907c4d4a1] 2024-03-28T17:56:01.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2907c4d667] 2024-03-28T17:56:01.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557e90763223] 2024-03-28T17:56:01.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557e905dd5d6] 2024-03-28T17:56:01.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557e905ddda1] 2024-03-28T17:56:01.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557e905e02d5] 2024-03-28T17:56:01.669 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557e90630233] 2024-03-28T17:56:01.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557e90632197] 2024-03-28T17:56:01.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557e9053111b] 2024-03-28T17:56:01.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557e905316bc] 2024-03-28T17:56:01.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557e9051b1ff] 2024-03-28T17:56:01.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2907edf488] 2024-03-28T17:56:01.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 13: (DispatchQueue::entry()+0x51f) [0x7f2907edc60f] 2024-03-28T17:56:01.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2907fafe81] 2024-03-28T17:56:01.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f29069ca1ca] 2024-03-28T17:56:01.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 16: clone() 2024-03-28T17:56:01.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 2024-03-28T17:56:01.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: *** Caught signal (Aborted) ** 2024-03-28T17:56:01.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: in thread 7f28fc400700 thread_name:ms_dispatch 2024-03-28T17:56:01.670 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:01.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f29069d4d20] 2024-03-28T17:56:01.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 2: gsignal() 2024-03-28T17:56:01.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 3: abort() 2024-03-28T17:56:01.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2907c4d4fb] 2024-03-28T17:56:01.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2907c4d667] 2024-03-28T17:56:01.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557e90763223] 2024-03-28T17:56:01.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557e905dd5d6] 2024-03-28T17:56:01.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557e905ddda1] 2024-03-28T17:56:01.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557e905e02d5] 2024-03-28T17:56:01.671 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557e90630233] 2024-03-28T17:56:01.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557e90632197] 2024-03-28T17:56:01.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557e9053111b] 2024-03-28T17:56:01.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557e905316bc] 2024-03-28T17:56:01.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557e9051b1ff] 2024-03-28T17:56:01.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2907edf488] 2024-03-28T17:56:01.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 16: (DispatchQueue::entry()+0x51f) [0x7f2907edc60f] 2024-03-28T17:56:01.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2907fafe81] 2024-03-28T17:56:01.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f29069ca1ca] 2024-03-28T17:56:01.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 19: clone() 2024-03-28T17:56:01.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 2024-03-28T17:56:01.314+0000 7f28fc400700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:56:01.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: in thread 7f28fc400700 thread_name:ms_dispatch 2024-03-28T17:56:01.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 2024-03-28T17:56:01.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:01.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f29069d4d20] 2024-03-28T17:56:01.672 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 2: gsignal() 2024-03-28T17:56:01.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 3: abort() 2024-03-28T17:56:01.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2907c4d4fb] 2024-03-28T17:56:01.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2907c4d667] 2024-03-28T17:56:01.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557e90763223] 2024-03-28T17:56:01.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557e905dd5d6] 2024-03-28T17:56:01.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557e905ddda1] 2024-03-28T17:56:01.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557e905e02d5] 2024-03-28T17:56:01.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557e90630233] 2024-03-28T17:56:01.673 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557e90632197] 2024-03-28T17:56:01.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557e9053111b] 2024-03-28T17:56:01.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557e905316bc] 2024-03-28T17:56:01.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557e9051b1ff] 2024-03-28T17:56:01.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2907edf488] 2024-03-28T17:56:01.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 16: (DispatchQueue::entry()+0x51f) [0x7f2907edc60f] 2024-03-28T17:56:01.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2907fafe81] 2024-03-28T17:56:01.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f29069ca1ca] 2024-03-28T17:56:01.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 19: clone() 2024-03-28T17:56:01.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:56:01.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 2024-03-28T17:56:01.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: -1> 2024-03-28T17:56:01.312+0000 7f28fc400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f28fc400700 time 2024-03-28T17:56:01.310917+0000 2024-03-28T17:56:01.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:56:01.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 2024-03-28T17:56:01.674 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:01.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2907c4d4a1] 2024-03-28T17:56:01.675 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2907c4d667] 2024-03-28T17:56:01.676 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557e90763223] 2024-03-28T17:56:01.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557e905dd5d6] 2024-03-28T17:56:01.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557e905ddda1] 2024-03-28T17:56:01.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557e905e02d5] 2024-03-28T17:56:01.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557e90630233] 2024-03-28T17:56:01.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557e90632197] 2024-03-28T17:56:01.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557e9053111b] 2024-03-28T17:56:01.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557e905316bc] 2024-03-28T17:56:01.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557e9051b1ff] 2024-03-28T17:56:01.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2907edf488] 2024-03-28T17:56:01.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 13: (DispatchQueue::entry()+0x51f) [0x7f2907edc60f] 2024-03-28T17:56:01.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2907fafe81] 2024-03-28T17:56:01.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f29069ca1ca] 2024-03-28T17:56:01.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 16: clone() 2024-03-28T17:56:01.677 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 2024-03-28T17:56:01.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 0> 2024-03-28T17:56:01.314+0000 7f28fc400700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:56:01.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: in thread 7f28fc400700 thread_name:ms_dispatch 2024-03-28T17:56:01.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 2024-03-28T17:56:01.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:01.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f29069d4d20] 2024-03-28T17:56:01.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 2: gsignal() 2024-03-28T17:56:01.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 3: abort() 2024-03-28T17:56:01.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2907c4d4fb] 2024-03-28T17:56:01.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2907c4d667] 2024-03-28T17:56:01.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557e90763223] 2024-03-28T17:56:01.678 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557e905dd5d6] 2024-03-28T17:56:01.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557e905ddda1] 2024-03-28T17:56:01.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557e905e02d5] 2024-03-28T17:56:01.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557e90630233] 2024-03-28T17:56:01.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557e90632197] 2024-03-28T17:56:01.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557e9053111b] 2024-03-28T17:56:01.679 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557e905316bc] 2024-03-28T17:56:01.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557e9051b1ff] 2024-03-28T17:56:01.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2907edf488] 2024-03-28T17:56:01.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 16: (DispatchQueue::entry()+0x51f) [0x7f2907edc60f] 2024-03-28T17:56:01.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2907fafe81] 2024-03-28T17:56:01.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f29069ca1ca] 2024-03-28T17:56:01.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 19: clone() 2024-03-28T17:56:01.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:56:01.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 2024-03-28T17:56:01.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: -1> 2024-03-28T17:56:01.312+0000 7f28fc400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f28fc400700 time 2024-03-28T17:56:01.310917+0000 2024-03-28T17:56:01.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:56:01.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 2024-03-28T17:56:01.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:01.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2907c4d4a1] 2024-03-28T17:56:01.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2907c4d667] 2024-03-28T17:56:01.680 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557e90763223] 2024-03-28T17:56:01.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557e905dd5d6] 2024-03-28T17:56:01.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557e905ddda1] 2024-03-28T17:56:01.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557e905e02d5] 2024-03-28T17:56:01.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557e90630233] 2024-03-28T17:56:01.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557e90632197] 2024-03-28T17:56:01.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557e9053111b] 2024-03-28T17:56:01.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557e905316bc] 2024-03-28T17:56:01.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557e9051b1ff] 2024-03-28T17:56:01.681 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2907edf488] 2024-03-28T17:56:01.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 13: (DispatchQueue::entry()+0x51f) [0x7f2907edc60f] 2024-03-28T17:56:01.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2907fafe81] 2024-03-28T17:56:01.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f29069ca1ca] 2024-03-28T17:56:01.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 16: clone() 2024-03-28T17:56:01.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 2024-03-28T17:56:01.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 0> 2024-03-28T17:56:01.314+0000 7f28fc400700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:56:01.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: in thread 7f28fc400700 thread_name:ms_dispatch 2024-03-28T17:56:01.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 2024-03-28T17:56:01.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:01.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f29069d4d20] 2024-03-28T17:56:01.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 2: gsignal() 2024-03-28T17:56:01.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 3: abort() 2024-03-28T17:56:01.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2907c4d4fb] 2024-03-28T17:56:01.682 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2907c4d667] 2024-03-28T17:56:01.683 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557e90763223] 2024-03-28T17:56:01.683 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557e905dd5d6] 2024-03-28T17:56:01.683 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557e905ddda1] 2024-03-28T17:56:01.683 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557e905e02d5] 2024-03-28T17:56:01.683 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557e90630233] 2024-03-28T17:56:01.683 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557e90632197] 2024-03-28T17:56:01.683 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557e9053111b] 2024-03-28T17:56:01.683 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557e905316bc] 2024-03-28T17:56:01.683 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557e9051b1ff] 2024-03-28T17:56:01.683 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2907edf488] 2024-03-28T17:56:01.683 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 16: (DispatchQueue::entry()+0x51f) [0x7f2907edc60f] 2024-03-28T17:56:01.684 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2907fafe81] 2024-03-28T17:56:01.684 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f29069ca1ca] 2024-03-28T17:56:01.684 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 19: clone() 2024-03-28T17:56:01.684 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:56:01.684 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d[91435]: 2024-03-28T17:56:01.684 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:01 smithi045 podman[91486]: 2024-03-28 17:56:01.420945397 +0000 UTC m=+0.035523686 container died 437610febe631b6b7d7658b54c04e8d7723afc51b897a47de074b595a79e7eaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-28T17:56:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:01 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2011326175,v1:172.21.15.45:6837/2011326175] up:rejoin 2024-03-28T17:56:01.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:01 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:56:01.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:01 smithi071 ceph-mon[36597]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-03-28T17:56:01.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:01 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:56:01.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:01 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:56:01.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:01 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2011326175,v1:172.21.15.45:6837/2011326175] up:active 2024-03-28T17:56:01.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:01 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:56:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:01 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2011326175,v1:172.21.15.45:6837/2011326175] up:rejoin 2024-03-28T17:56:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:01 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=d=up:rejoin} 2 up:standby-replay 5 up:standby 2024-03-28T17:56:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:01 smithi151 ceph-mon[36481]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-03-28T17:56:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:01 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:56:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:01 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:56:01.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:01 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2011326175,v1:172.21.15.45:6837/2011326175] up:active 2024-03-28T17:56:01.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:01 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=d=up:active} 3 up:standby-replay 4 up:standby 2024-03-28T17:56:01.880 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 214 sec latency 329776.793 ms 2024-03-28T17:56:02.416 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:02 smithi045 podman[91486]: 2024-03-28 17:56:02.070606417 +0000 UTC m=+0.685184748 container remove 437610febe631b6b7d7658b54c04e8d7723afc51b897a47de074b595a79e7eaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-d, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-28T17:56:02.416 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:02 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:56:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:02 smithi071 ceph-mon[36597]: pgmap v234: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:56:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:02 smithi151 ceph-mon[36481]: pgmap v234: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:56:02.880 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 215 sec latency 330776.898 ms 2024-03-28T17:56:02.915 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:02 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-28T17:56:02.915 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:02 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Consumed 1.563s CPU time. 2024-03-28T17:56:02.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:02 smithi045 ceph-mon[31773]: pgmap v234: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:56:03.880 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 216 sec latency 331777.027 ms 2024-03-28T17:56:04.797 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:56:04.798 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:56:04.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:04 smithi071 ceph-mon[36597]: pgmap v235: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:56:04.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:04 smithi151 ceph-mon[36481]: pgmap v235: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:56:04.880 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 217 sec latency 332777.118 ms 2024-03-28T17:56:04.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:04 smithi045 ceph-mon[31773]: pgmap v235: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T17:56:05.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.038+0000 7f8cb5800640 1 -- 172.21.15.45:0/3758367099 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8cb0073a40 msgr2=0x7f8cb0073e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:56:05.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.038+0000 7f8cb5800640 1 --2- 172.21.15.45:0/3758367099 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8cb0073a40 0x7f8cb0073e20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8009ca0 tx=0x7f8ca802c180 comp rx=0 tx=0).stop 2024-03-28T17:56:05.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.038+0000 7f8cb5800640 1 -- 172.21.15.45:0/3758367099 shutdown_connections 2024-03-28T17:56:05.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.038+0000 7f8cb5800640 1 --2- 172.21.15.45:0/3758367099 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8cb0112640 0x7f8cb0114a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.038+0000 7f8cb5800640 1 --2- 172.21.15.45:0/3758367099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cb0074360 0x7f8cb0112100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.038+0000 7f8cb5800640 1 --2- 172.21.15.45:0/3758367099 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8cb0073a40 0x7f8cb0073e20 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.038+0000 7f8cb5800640 1 -- 172.21.15.45:0/3758367099 >> 172.21.15.45:0/3758367099 conn(0x7f8cb0078aa0 msgr2=0x7f8cb0078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:56:05.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.038+0000 7f8cb5800640 1 -- 172.21.15.45:0/3758367099 shutdown_connections 2024-03-28T17:56:05.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.039+0000 7f8cb5800640 1 -- 172.21.15.45:0/3758367099 wait complete. 2024-03-28T17:56:05.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.040+0000 7f8cb5800640 1 Processor -- start 2024-03-28T17:56:05.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.040+0000 7f8cb5800640 1 -- start start 2024-03-28T17:56:05.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.040+0000 7f8cb5800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cb0073a40 0x7f8cb019e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:05.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.041+0000 7f8cb5800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8cb0074360 0x7f8cb019eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:05.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.041+0000 7f8cb5800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8cb0112640 0x7f8cb01a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:05.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.041+0000 7f8cb5800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb0118040 con 0x7f8cb0073a40 2024-03-28T17:56:05.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.041+0000 7f8cb5800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb0117ec0 con 0x7f8cb0112640 2024-03-28T17:56:05.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.041+0000 7f8cb5800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb01181c0 con 0x7f8cb0074360 2024-03-28T17:56:05.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.041+0000 7f8caf400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8cb0112640 0x7f8cb01a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:05.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.041+0000 7f8caf400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8cb0112640 0x7f8cb01a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:52278/0 (socket says 172.21.15.45:52278) 2024-03-28T17:56:05.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.041+0000 7f8caf400640 1 -- 172.21.15.45:0/1074367559 learned_addr learned my addr 172.21.15.45:0/1074367559 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:56:05.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.041+0000 7f8caea00640 1 --2- 172.21.15.45:0/1074367559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cb0073a40 0x7f8cb019e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:05.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.041+0000 7f8cae000640 1 --2- 172.21.15.45:0/1074367559 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8cb0074360 0x7f8cb019eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:05.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.041+0000 7f8caf400640 1 -- 172.21.15.45:0/1074367559 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8cb0074360 msgr2=0x7f8cb019eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:56:05.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.041+0000 7f8caf400640 1 --2- 172.21.15.45:0/1074367559 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8cb0074360 0x7f8cb019eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.041+0000 7f8caf400640 1 -- 172.21.15.45:0/1074367559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cb0073a40 msgr2=0x7f8cb019e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:56:05.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.041+0000 7f8caf400640 1 --2- 172.21.15.45:0/1074367559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cb0073a40 0x7f8cb019e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.041+0000 7f8caf400640 1 -- 172.21.15.45:0/1074367559 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb01a3550 con 0x7f8cb0112640 2024-03-28T17:56:05.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.041+0000 7f8caea00640 1 --2- 172.21.15.45:0/1074367559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cb0073a40 0x7f8cb019e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:56:05.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.042+0000 7f8caf400640 1 --2- 172.21.15.45:0/1074367559 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8cb0112640 0x7f8cb01a2e20 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f8ca000d950 tx=0x7f8ca000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:56:05.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.042+0000 7f8ca7400640 1 -- 172.21.15.45:0/1074367559 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0017070 con 0x7f8cb0112640 2024-03-28T17:56:05.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.042+0000 7f8ca7400640 1 -- 172.21.15.45:0/1074367559 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ca0009c30 con 0x7f8cb0112640 2024-03-28T17:56:05.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.042+0000 7f8ca7400640 1 -- 172.21.15.45:0/1074367559 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0003d60 con 0x7f8cb0112640 2024-03-28T17:56:05.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.042+0000 7f8cb5800640 1 -- 172.21.15.45:0/1074367559 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb01a3840 con 0x7f8cb0112640 2024-03-28T17:56:05.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.042+0000 7f8cb5800640 1 -- 172.21.15.45:0/1074367559 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb01ab0d0 con 0x7f8cb0112640 2024-03-28T17:56:05.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.047+0000 7f8ca7400640 1 -- 172.21.15.45:0/1074367559 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f8ca0003f00 con 0x7f8cb0112640 2024-03-28T17:56:05.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.047+0000 7f8cb8462740 1 Processor -- start 2024-03-28T17:56:05.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.047+0000 7f8cb8462740 1 -- start start 2024-03-28T17:56:05.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.047+0000 7f8cb8462740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126770c00 0x55b12676f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:05.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.047+0000 7f8cb8462740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b12676ff20 0x55b1267703a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:05.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.047+0000 7f8cb8462740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b126779290 0x55b12677b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:05.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.047+0000 7f8cb8462740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1266d4250 con 0x55b12676ff20 2024-03-28T17:56:05.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.047+0000 7f8cb8462740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b12667ca30 con 0x55b126770c00 2024-03-28T17:56:05.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.047+0000 7f8cb8462740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1266d02f0 con 0x55b126779290 2024-03-28T17:56:05.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.048+0000 7f8cae000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126770c00 0x55b12676f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:05.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.048+0000 7f8cae000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126770c00 0x55b12676f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:52284/0 (socket says 172.21.15.45:52284) 2024-03-28T17:56:05.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.048+0000 7f8cae000640 1 -- 172.21.15.45:0/1798946000 learned_addr learned my addr 172.21.15.45:0/1798946000 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:56:05.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.048+0000 7f8caea00640 1 --2- 172.21.15.45:0/1798946000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b12676ff20 0x55b1267703a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:05.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.048+0000 7f8cae000640 1 --2- 172.21.15.45:0/1798946000 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b126779290 0x55b12677b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:05.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.048+0000 7f8cae000640 1 -- 172.21.15.45:0/1798946000 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b126779290 msgr2=0x55b12677b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:56:05.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.048+0000 7f8cae000640 1 --2- 172.21.15.45:0/1798946000 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b126779290 0x55b12677b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.048+0000 7f8cae000640 1 -- 172.21.15.45:0/1798946000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b12676ff20 msgr2=0x55b1267703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:56:05.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.048+0000 7f8cae000640 1 --2- 172.21.15.45:0/1798946000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b12676ff20 0x55b1267703a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.048+0000 7f8cae000640 1 -- 172.21.15.45:0/1798946000 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b126667690 con 0x55b126770c00 2024-03-28T17:56:05.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.048+0000 7f8cae000640 1 --2- 172.21.15.45:0/1798946000 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126770c00 0x55b12676f950 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8c98013560 tx=0x7f8c98005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7e0feedc980449a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:56:05.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.049+0000 7f8c9fe00640 1 -- 172.21.15.45:0/1798946000 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c98004250 con 0x55b126770c00 2024-03-28T17:56:05.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.049+0000 7f8ca7400640 1 --2- 172.21.15.45:0/1074367559 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8c940a2970 0x7f8c940a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:05.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.049+0000 7f8c9fe00640 1 -- 172.21.15.45:0/1798946000 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8c980043f0 con 0x55b126770c00 2024-03-28T17:56:05.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.049+0000 7f8c9fe00640 1 -- 172.21.15.45:0/1798946000 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c980119c0 con 0x55b126770c00 2024-03-28T17:56:05.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.049+0000 7f8ca7400640 1 -- 172.21.15.45:0/1074367559 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 10232+0+0 (secure 0 0 0) 0x7f8ca00cedf0 con 0x7f8cb0112640 2024-03-28T17:56:05.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.049+0000 7f8caea00640 1 --2- 172.21.15.45:0/1074367559 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8c940a2970 0x7f8c940a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:05.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.049+0000 7f8cb8462740 1 -- 172.21.15.45:0/1798946000 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126770c00 msgr2=0x55b12676f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:56:05.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.049+0000 7f8cb8462740 1 --2- 172.21.15.45:0/1798946000 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126770c00 0x55b12676f950 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8c98013560 tx=0x7f8c98005850 comp rx=0 tx=0).stop 2024-03-28T17:56:05.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.049+0000 7f8cb8462740 1 -- 172.21.15.45:0/1798946000 shutdown_connections 2024-03-28T17:56:05.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.049+0000 7f8cb8462740 1 --2- 172.21.15.45:0/1798946000 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b126779290 0x55b12677b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.049+0000 7f8cb8462740 1 --2- 172.21.15.45:0/1798946000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b12676ff20 0x55b1267703a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.049+0000 7f8cb8462740 1 --2- 172.21.15.45:0/1798946000 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126770c00 0x55b12676f950 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.049+0000 7f8cb8462740 1 -- 172.21.15.45:0/1798946000 >> 172.21.15.45:0/1798946000 conn(0x55b126737200 msgr2=0x55b12676a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:56:05.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.050+0000 7f8cb8462740 1 -- 172.21.15.45:0/1798946000 shutdown_connections 2024-03-28T17:56:05.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.050+0000 7f8cb8462740 1 -- 172.21.15.45:0/1798946000 wait complete. 2024-03-28T17:56:05.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.050+0000 7f8caea00640 1 --2- 172.21.15.45:0/1074367559 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8c940a2970 0x7f8c940a4e30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8006010 tx=0x7f8ca802e040 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:56:05.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.050+0000 7f8cb8462740 1 -- start start 2024-03-28T17:56:05.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.050+0000 7f8cb8462740 1 Processor -- start 2024-03-28T17:56:05.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.051+0000 7f8cb8462740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b12676ff20 0x55b126778970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:05.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.051+0000 7f8cb8462740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b126770c00 0x55b126775d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:05.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.051+0000 7f8cb8462740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126779290 0x55b126776240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:05.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.051+0000 7f8cb8462740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b12677c100 con 0x55b12676ff20 2024-03-28T17:56:05.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.051+0000 7f8cb8462740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b12668fe80 con 0x55b126779290 2024-03-28T17:56:05.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.051+0000 7f8cb8462740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b12677c280 con 0x55b126770c00 2024-03-28T17:56:05.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.051+0000 7f8cb8462740 10 client.0 _ll_register_callbacks cb 0x55b1266a2130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:56:05.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.051+0000 7f8caf400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126779290 0x55b126776240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:05.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.051+0000 7f8caea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b12676ff20 0x55b126778970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:05.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.051+0000 7f8cae000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b126770c00 0x55b126775d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:05.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.051+0000 7f8caf400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126779290 0x55b126776240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:52290/0 (socket says 172.21.15.45:52290) 2024-03-28T17:56:05.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.051+0000 7f8caf400640 1 -- 172.21.15.45:0/827668177 learned_addr learned my addr 172.21.15.45:0/827668177 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:56:05.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.051+0000 7f8caf400640 1 -- 172.21.15.45:0/827668177 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b126770c00 msgr2=0x55b126775d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:56:05.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.051+0000 7f8caf400640 1 --2- 172.21.15.45:0/827668177 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b126770c00 0x55b126775d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.051+0000 7f8caf400640 1 -- 172.21.15.45:0/827668177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b12676ff20 msgr2=0x55b126778970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:56:05.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.051+0000 7f8caf400640 1 --2- 172.21.15.45:0/827668177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b12676ff20 0x55b126778970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.051+0000 7f8caf400640 1 -- 172.21.15.45:0/827668177 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b1264b27d0 con 0x55b126779290 2024-03-28T17:56:05.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.052+0000 7f8caf400640 1 --2- 172.21.15.45:0/827668177 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126779290 0x55b126776240 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8ca00a3010 tx=0x7f8ca009a830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15639d55a84e92ce server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:56:05.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.052+0000 7f8c9cc00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:56:05.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.052+0000 7f8c9cc00640 1 -- 172.21.15.45:0/827668177 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca009e020 con 0x55b126779290 2024-03-28T17:56:05.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.052+0000 7f8c9cc00640 1 -- 172.21.15.45:0/827668177 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 10232+0+0 (secure 0 0 0) 0x7f8ca00041f0 con 0x55b126779290 2024-03-28T17:56:05.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.052+0000 7f8cb8462740 1 -- 172.21.15.45:0/827668177 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8c980043f0 con 0x55b126779290 2024-03-28T17:56:05.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.052+0000 7f8c9cc00640 1 -- 172.21.15.45:0/827668177 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=156) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ca0020040 con 0x55b126779290 2024-03-28T17:56:05.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.052+0000 7f8cb8462740 10 client.26201 fetch_fsmap learned FSMap version 156 2024-03-28T17:56:05.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.052+0000 7f8cb8462740 1 -- 172.21.15.45:0/827668177 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=156,monmap=4+}) v3 -- 0x55b1263e83f0 con 0x55b126779290 2024-03-28T17:56:05.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.052+0000 7f8c9cc00640 1 -- 172.21.15.45:0/827668177 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8c980043f0 con 0x55b126779290 2024-03-28T17:56:05.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.053+0000 7f8c9cc00640 1 -- 172.21.15.45:0/827668177 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 156) v1 ==== 6768+0+0 (secure 0 0 0) 0x7f8ca0079ce0 con 0x55b126779290 2024-03-28T17:56:05.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.053+0000 7f8cb8462740 10 client.26201 fetch_fsmap finished waiting for FSMap version 156 2024-03-28T17:56:05.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.053+0000 7f8cb8462740 10 client.26201 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:56:05.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.053+0000 7f8cb8462740 20 client.26201 populate_metadata read hostname 'smithi045' 2024-03-28T17:56:05.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.053+0000 7f8cb8462740 1 --2- 172.21.15.45:0/827668177 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55b1267890f0 0x55b12678b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:05.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.053+0000 7f8cb8462740 4 client.26201 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:56:05.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.053+0000 7f8cb8462740 1 -- 172.21.15.45:0/827668177 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b1266e1790 con 0x55b1267890f0 2024-03-28T17:56:05.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.053+0000 7f8caea00640 1 --2- 172.21.15.45:0/827668177 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55b1267890f0 0x55b12678b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:05.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.053+0000 7f8caea00640 1 --2- 172.21.15.45:0/827668177 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55b1267890f0 0x55b12678b570 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b33f8f7d56d2cd5 server_cookie=2526bbb00f3dfa48 in_seq=0 out_seq=0 2024-03-28T17:56:05.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.053+0000 7f8c9cc00640 10 client.26201 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:56:05.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.054+0000 7f8c9cc00640 1 -- 172.21.15.45:0/827668177 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b1266e1790 con 0x55b1267890f0 2024-03-28T17:56:05.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.054+0000 7f8c9cc00640 10 client.26201 handle_command_reply: tid=0 2024-03-28T17:56:05.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.055+0000 7f8cb8462740 1 client.26201 shutdown 2024-03-28T17:56:05.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.055+0000 7f8cb8462740 1 -- 172.21.15.45:0/827668177 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126779290 msgr2=0x55b126776240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:56:05.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.055+0000 7f8cb8462740 1 --2- 172.21.15.45:0/827668177 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126779290 0x55b126776240 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8ca00a3010 tx=0x7f8ca009a830 comp rx=0 tx=0).stop 2024-03-28T17:56:05.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.055+0000 7f8cb8462740 1 -- 172.21.15.45:0/827668177 shutdown_connections 2024-03-28T17:56:05.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.055+0000 7f8cb8462740 1 --2- 172.21.15.45:0/827668177 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55b1267890f0 0x55b12678b570 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.055+0000 7f8cb8462740 1 --2- 172.21.15.45:0/827668177 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126779290 0x55b126776240 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.055+0000 7f8cb8462740 1 --2- 172.21.15.45:0/827668177 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b126770c00 0x55b126775d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.055+0000 7f8cb8462740 1 --2- 172.21.15.45:0/827668177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b12676ff20 0x55b126778970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.055+0000 7f8cb8462740 1 -- 172.21.15.45:0/827668177 >> 172.21.15.45:0/827668177 conn(0x55b126737200 msgr2=0x55b12676c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:56:05.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.055+0000 7f8c9cc00640 0 client.26201 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:56:05.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.055+0000 7f8cb8462740 1 -- 172.21.15.45:0/827668177 shutdown_connections 2024-03-28T17:56:05.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.055+0000 7f8cb8462740 1 -- 172.21.15.45:0/827668177 wait complete. 2024-03-28T17:56:05.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.055+0000 7f8cb8462740 20 client.26201 trim_cache size 0 max 16384 2024-03-28T17:56:05.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.073+0000 7f8cb8462740 1 Processor -- start 2024-03-28T17:56:05.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.073+0000 7f8cb8462740 1 -- start start 2024-03-28T17:56:05.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.074+0000 7f8cb8462740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b12676ff20 0x55b126774ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:05.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.074+0000 7f8cb8462740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126770c00 0x55b126778430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:05.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.074+0000 7f8cb8462740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b126779290 0x55b126778970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:05.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.074+0000 7f8cb8462740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb01b5160 con 0x55b126779290 2024-03-28T17:56:05.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.074+0000 7f8cb8462740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb01b57c0 con 0x55b126770c00 2024-03-28T17:56:05.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.074+0000 7f8cb8462740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb01b5e20 con 0x55b12676ff20 2024-03-28T17:56:05.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.074+0000 7f8cae000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126770c00 0x55b126778430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:05.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.074+0000 7f8cae000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126770c00 0x55b126778430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:52304/0 (socket says 172.21.15.45:52304) 2024-03-28T17:56:05.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.074+0000 7f8cae000640 1 -- 172.21.15.45:0/2616882693 learned_addr learned my addr 172.21.15.45:0/2616882693 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:56:05.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.074+0000 7f8caea00640 1 --2- 172.21.15.45:0/2616882693 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b12676ff20 0x55b126774ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:05.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.074+0000 7f8caf400640 1 --2- 172.21.15.45:0/2616882693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b126779290 0x55b126778970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:05.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.074+0000 7f8cae000640 1 -- 172.21.15.45:0/2616882693 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b12676ff20 msgr2=0x55b126774ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:56:05.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.074+0000 7f8cae000640 1 --2- 172.21.15.45:0/2616882693 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b12676ff20 0x55b126774ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.074+0000 7f8cae000640 1 -- 172.21.15.45:0/2616882693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b126779290 msgr2=0x55b126778970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:56:05.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.074+0000 7f8cae000640 1 --2- 172.21.15.45:0/2616882693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b126779290 0x55b126778970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.074+0000 7f8cae000640 1 -- 172.21.15.45:0/2616882693 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b1263a80f0 con 0x55b126770c00 2024-03-28T17:56:05.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.074+0000 7f8caea00640 1 --2- 172.21.15.45:0/2616882693 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b12676ff20 0x55b126774ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:56:05.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.074+0000 7f8caf400640 1 --2- 172.21.15.45:0/2616882693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b126779290 0x55b126778970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T17:56:05.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.075+0000 7f8cae000640 1 --2- 172.21.15.45:0/2616882693 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126770c00 0x55b126778430 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8c98013eb0 tx=0x7f8c980097f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f74faba2c1dfd5ef server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:56:05.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.075+0000 7f8c9f400640 1 -- 172.21.15.45:0/2616882693 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9802f070 con 0x55b126770c00 2024-03-28T17:56:05.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.075+0000 7f8c9f400640 1 -- 172.21.15.45:0/2616882693 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8c98009c50 con 0x55b126770c00 2024-03-28T17:56:05.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.075+0000 7f8c9f400640 1 -- 172.21.15.45:0/2616882693 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c98011590 con 0x55b126770c00 2024-03-28T17:56:05.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.076+0000 7f8cb8462740 1 -- 172.21.15.45:0/2616882693 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126770c00 msgr2=0x55b126778430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:56:05.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.076+0000 7f8cb8462740 1 --2- 172.21.15.45:0/2616882693 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126770c00 0x55b126778430 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8c98013eb0 tx=0x7f8c980097f0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.076+0000 7f8cb8462740 1 -- 172.21.15.45:0/2616882693 shutdown_connections 2024-03-28T17:56:05.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.076+0000 7f8cb8462740 1 --2- 172.21.15.45:0/2616882693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b126779290 0x55b126778970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.076+0000 7f8cb8462740 1 --2- 172.21.15.45:0/2616882693 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126770c00 0x55b126778430 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.076+0000 7f8cb8462740 1 --2- 172.21.15.45:0/2616882693 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b12676ff20 0x55b126774ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.076+0000 7f8cb8462740 1 -- 172.21.15.45:0/2616882693 >> 172.21.15.45:0/2616882693 conn(0x55b126737200 msgr2=0x55b1267745d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:56:05.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.076+0000 7f8cb8462740 1 -- 172.21.15.45:0/2616882693 shutdown_connections 2024-03-28T17:56:05.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.076+0000 7f8cb8462740 1 -- 172.21.15.45:0/2616882693 wait complete. 2024-03-28T17:56:05.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.076+0000 7f8cb8462740 1 -- start start 2024-03-28T17:56:05.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.076+0000 7f8cb8462740 1 Processor -- start 2024-03-28T17:56:05.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.077+0000 7f8cb8462740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b12676ff20 0x55b12677cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:05.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.077+0000 7f8cb8462740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b126770c00 0x55b12677d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:05.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.077+0000 7f8cb8462740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126779290 0x55b1267e3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.077+0000 7f8cb8462740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb01b2700 con 0x55b126770c00 2024-03-28T17:56:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.077+0000 7f8cb8462740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb01b39a0 con 0x55b126779290 2024-03-28T17:56:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.077+0000 7f8cb8462740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb01b2d00 con 0x55b12676ff20 2024-03-28T17:56:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.077+0000 7f8cb8462740 10 client.0 _ll_register_callbacks cb 0x7f8ca0005d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:56:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.077+0000 7f8caf400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126779290 0x55b1267e3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.077+0000 7f8caf400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126779290 0x55b1267e3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:52314/0 (socket says 172.21.15.45:52314) 2024-03-28T17:56:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.077+0000 7f8caf400640 1 -- 172.21.15.45:0/2794275817 learned_addr learned my addr 172.21.15.45:0/2794275817 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:56:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.077+0000 7f8cae000640 1 --2- 172.21.15.45:0/2794275817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b126770c00 0x55b12677d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.077+0000 7f8caea00640 1 --2- 172.21.15.45:0/2794275817 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b12676ff20 0x55b12677cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.078+0000 7f8caf400640 1 -- 172.21.15.45:0/2794275817 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b12676ff20 msgr2=0x55b12677cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:56:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.078+0000 7f8caf400640 1 --2- 172.21.15.45:0/2794275817 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b12676ff20 0x55b12677cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.078+0000 7f8caf400640 1 -- 172.21.15.45:0/2794275817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b126770c00 msgr2=0x55b12677d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:56:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.078+0000 7f8caf400640 1 --2- 172.21.15.45:0/2794275817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b126770c00 0x55b12677d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.078+0000 7f8caf400640 1 -- 172.21.15.45:0/2794275817 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b1267e3f40 con 0x55b126779290 2024-03-28T17:56:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.078+0000 7f8caf400640 1 --2- 172.21.15.45:0/2794275817 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126779290 0x55b1267e3a00 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f8ca009aaa0 tx=0x7f8ca009a4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=668116d22fffc017 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:56:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.078+0000 7f8c9cc00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:56:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.078+0000 7f8c9cc00640 1 -- 172.21.15.45:0/2794275817 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca009e050 con 0x55b126779290 2024-03-28T17:56:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.078+0000 7f8cb8462740 1 -- 172.21.15.45:0/2794275817 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8c98009c50 con 0x55b126779290 2024-03-28T17:56:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.078+0000 7f8c9cc00640 1 -- 172.21.15.45:0/2794275817 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 10232+0+0 (secure 0 0 0) 0x7f8ca0079bc0 con 0x55b126779290 2024-03-28T17:56:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.078+0000 7f8c9cc00640 1 -- 172.21.15.45:0/2794275817 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=156) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ca00b8020 con 0x55b126779290 2024-03-28T17:56:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.078+0000 7f8cb8462740 10 client.26207 fetch_fsmap learned FSMap version 156 2024-03-28T17:56:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.078+0000 7f8cb8462740 1 -- 172.21.15.45:0/2794275817 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=156,monmap=4+}) v3 -- 0x55b1267e5000 con 0x55b126779290 2024-03-28T17:56:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.079+0000 7f8c9cc00640 1 -- 172.21.15.45:0/2794275817 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8c98009c50 con 0x55b126779290 2024-03-28T17:56:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.079+0000 7f8c9cc00640 1 -- 172.21.15.45:0/2794275817 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 156) v1 ==== 6768+0+0 (secure 0 0 0) 0x7f8ca0099c60 con 0x55b126779290 2024-03-28T17:56:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.079+0000 7f8cb8462740 10 client.26207 fetch_fsmap finished waiting for FSMap version 156 2024-03-28T17:56:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.079+0000 7f8cb8462740 10 client.26207 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:56:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.079+0000 7f8cb8462740 20 client.26207 populate_metadata read hostname 'smithi045' 2024-03-28T17:56:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.079+0000 7f8cb8462740 1 --2- 172.21.15.45:0/2794275817 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55b1267e5540 0x55b1267e7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.079+0000 7f8cb8462740 4 client.26207 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:56:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.079+0000 7f8cb8462740 1 -- 172.21.15.45:0/2794275817 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b1266dbae0 con 0x55b1267e5540 2024-03-28T17:56:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.079+0000 7f8caea00640 1 --2- 172.21.15.45:0/2794275817 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55b1267e5540 0x55b1267e7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.079+0000 7f8caea00640 1 --2- 172.21.15.45:0/2794275817 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55b1267e5540 0x55b1267e7930 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bb071298aeab99d server_cookie=1282d8ae4eb44295 in_seq=0 out_seq=0 2024-03-28T17:56:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.079+0000 7f8c9cc00640 10 client.26207 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:56:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.080+0000 7f8c9cc00640 1 -- 172.21.15.45:0/2794275817 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55b1266dbae0 con 0x55b1267e5540 2024-03-28T17:56:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.080+0000 7f8c9cc00640 10 client.26207 handle_command_reply: tid=0 2024-03-28T17:56:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.080+0000 7f8cb8462740 1 client.26207 shutdown 2024-03-28T17:56:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.080+0000 7f8cb8462740 1 -- 172.21.15.45:0/2794275817 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126779290 msgr2=0x55b1267e3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:56:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.080+0000 7f8cb8462740 1 --2- 172.21.15.45:0/2794275817 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126779290 0x55b1267e3a00 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f8ca009aaa0 tx=0x7f8ca009a4c0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.080+0000 7f8cb8462740 1 -- 172.21.15.45:0/2794275817 shutdown_connections 2024-03-28T17:56:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.080+0000 7f8cb8462740 1 --2- 172.21.15.45:0/2794275817 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55b1267e5540 0x55b1267e7930 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.080+0000 7f8cb8462740 1 --2- 172.21.15.45:0/2794275817 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b126779290 0x55b1267e3a00 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.080+0000 7f8cb8462740 1 --2- 172.21.15.45:0/2794275817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b126770c00 0x55b12677d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.080+0000 7f8c9cc00640 0 client.26207 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:56:05.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.080+0000 7f8cb8462740 1 --2- 172.21.15.45:0/2794275817 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b12676ff20 0x55b12677cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.080+0000 7f8cb8462740 1 -- 172.21.15.45:0/2794275817 >> 172.21.15.45:0/2794275817 conn(0x55b126737200 msgr2=0x55b126774e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:56:05.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.080+0000 7f8cb8462740 1 -- 172.21.15.45:0/2794275817 shutdown_connections 2024-03-28T17:56:05.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.080+0000 7f8cb8462740 1 -- 172.21.15.45:0/2794275817 wait complete. 2024-03-28T17:56:05.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.080+0000 7f8cb8462740 20 client.26207 trim_cache size 0 max 16384 2024-03-28T17:56:05.085 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:56:05.085 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T17:56:05.085 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T17:56:05.085 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T17:56:05.085 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T17:56:05.085 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T17:56:05.086 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T17:56:05.086 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:56:05.086 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:56:05.086 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:56:05.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.081+0000 7f8cb5800640 1 -- 172.21.15.45:0/1074367559 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8c940a2970 msgr2=0x7f8c940a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:56:05.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.081+0000 7f8cb5800640 1 --2- 172.21.15.45:0/1074367559 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8c940a2970 0x7f8c940a4e30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8006010 tx=0x7f8ca802e040 comp rx=0 tx=0).stop 2024-03-28T17:56:05.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.081+0000 7f8cb5800640 1 -- 172.21.15.45:0/1074367559 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8cb0112640 msgr2=0x7f8cb01a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:56:05.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.081+0000 7f8cb5800640 1 --2- 172.21.15.45:0/1074367559 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8cb0112640 0x7f8cb01a2e20 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f8ca000d950 tx=0x7f8ca000de10 comp rx=0 tx=0).stop 2024-03-28T17:56:05.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.081+0000 7f8cb5800640 1 -- 172.21.15.45:0/1074367559 shutdown_connections 2024-03-28T17:56:05.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.081+0000 7f8cb5800640 1 --2- 172.21.15.45:0/1074367559 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8c940a2970 0x7f8c940a4e30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.081+0000 7f8cb5800640 1 --2- 172.21.15.45:0/1074367559 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8cb0112640 0x7f8cb01a2e20 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.081+0000 7f8cb5800640 1 --2- 172.21.15.45:0/1074367559 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8cb0074360 0x7f8cb019eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.081+0000 7f8cb5800640 1 --2- 172.21.15.45:0/1074367559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cb0073a40 0x7f8cb019e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:05.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.081+0000 7f8cb5800640 1 -- 172.21.15.45:0/1074367559 >> 172.21.15.45:0/1074367559 conn(0x7f8cb0078aa0 msgr2=0x7f8cb01115f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:56:05.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.081+0000 7f8cb5800640 1 -- 172.21.15.45:0/1074367559 shutdown_connections 2024-03-28T17:56:05.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:05.081+0000 7f8cb5800640 1 -- 172.21.15.45:0/1074367559 wait complete. 2024-03-28T17:56:05.099 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T17:56:05.880 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 218 sec latency 333777.255 ms 2024-03-28T17:56:06.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:06 smithi071 ceph-mon[36597]: pgmap v236: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:56:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:06 smithi151 ceph-mon[36481]: pgmap v236: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:56:06.880 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 219 sec latency 334777.414 ms 2024-03-28T17:56:06.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:06 smithi045 ceph-mon[31773]: pgmap v236: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:56:07.881 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 220 sec latency 335777.567 ms 2024-03-28T17:56:08.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:08 smithi071 ceph-mon[36597]: pgmap v237: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:56:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:08 smithi151 ceph-mon[36481]: pgmap v237: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:56:08.881 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 221 sec latency 336777.712 ms 2024-03-28T17:56:08.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:08 smithi045 ceph-mon[31773]: pgmap v237: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:56:09.881 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 222 sec latency 337777.892 ms 2024-03-28T17:56:10.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:10 smithi071 ceph-mon[36597]: pgmap v238: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-03-28T17:56:10.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:10 smithi151 ceph-mon[36481]: pgmap v238: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-03-28T17:56:10.881 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 223 sec latency 338777.984 ms 2024-03-28T17:56:10.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:10 smithi045 ceph-mon[31773]: pgmap v238: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-03-28T17:56:11.881 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 224 sec latency 339778.115 ms 2024-03-28T17:56:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:12 smithi071 ceph-mon[36597]: pgmap v239: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-03-28T17:56:12.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:12 smithi151 ceph-mon[36481]: pgmap v239: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-03-28T17:56:12.881 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 225 sec latency 340778.267 ms 2024-03-28T17:56:12.911 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:12 smithi045 ceph-mon[31773]: pgmap v239: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-03-28T17:56:12.912 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:12 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 6. 2024-03-28T17:56:12.912 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:12 smithi045 systemd[1]: Stopped Ceph mds.d for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:56:12.912 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:12 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Consumed 1.563s CPU time. 2024-03-28T17:56:12.912 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:12 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Start request repeated too quickly. 2024-03-28T17:56:12.912 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:12 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-28T17:56:12.912 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 28 17:56:12 smithi045 systemd[1]: Failed to start Ceph mds.d for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:56:13.881 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 226 sec latency 341778.432 ms 2024-03-28T17:56:14.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:14 smithi151 ceph-mon[36481]: pgmap v240: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-03-28T17:56:14.881 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 227 sec latency 342778.585 ms 2024-03-28T17:56:14.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:14 smithi045 ceph-mon[31773]: pgmap v240: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-03-28T17:56:15.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:14 smithi071 ceph-mon[36597]: pgmap v240: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-03-28T17:56:15.882 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 228 sec latency 343778.688 ms 2024-03-28T17:56:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:16 smithi151 ceph-mon[36481]: pgmap v241: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-03-28T17:56:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:16 smithi151 ceph-mon[36481]: Replacing daemon mds.d as rank 2 with standby daemon mds.k 2024-03-28T17:56:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:16 smithi151 ceph-mon[36481]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-03-28T17:56:16.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:16 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:56:16.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:16 smithi151 ceph-mon[36481]: osdmap e109: 12 total, 12 up, 12 in 2024-03-28T17:56:16.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:16 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:replay} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:16.881 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 229 sec latency 344778.808 ms 2024-03-28T17:56:16.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:16 smithi045 ceph-mon[31773]: pgmap v241: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-03-28T17:56:16.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:16 smithi045 ceph-mon[31773]: Replacing daemon mds.d as rank 2 with standby daemon mds.k 2024-03-28T17:56:16.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:16 smithi045 ceph-mon[31773]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-03-28T17:56:16.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:16 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:56:16.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:16 smithi045 ceph-mon[31773]: osdmap e109: 12 total, 12 up, 12 in 2024-03-28T17:56:16.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:16 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:replay} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:16 smithi071 ceph-mon[36597]: pgmap v241: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-03-28T17:56:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:16 smithi071 ceph-mon[36597]: Replacing daemon mds.d as rank 2 with standby daemon mds.k 2024-03-28T17:56:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:16 smithi071 ceph-mon[36597]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-03-28T17:56:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:16 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:56:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:16 smithi071 ceph-mon[36597]: osdmap e109: 12 total, 12 up, 12 in 2024-03-28T17:56:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:16 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:replay} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:17.882 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 230 sec latency 345778.928 ms 2024-03-28T17:56:18.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:18 smithi045 ceph-mon[31773]: pgmap v243: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:56:18.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:18 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/1799595971,v1:172.21.15.71:6833/1799595971] up:resolve 2024-03-28T17:56:18.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:18 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:resolve} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:18.569 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:18 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:56:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:18 smithi151 ceph-mon[36481]: pgmap v243: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:56:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:18 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/1799595971,v1:172.21.15.71:6833/1799595971] up:resolve 2024-03-28T17:56:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:18 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:resolve} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:18 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:56:18.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:18 smithi071 ceph-mon[36597]: pgmap v243: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:56:18.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:18 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/1799595971,v1:172.21.15.71:6833/1799595971] up:resolve 2024-03-28T17:56:18.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:18 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:resolve} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:18.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:18 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:56:18.882 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 231 sec latency 346779.054 ms 2024-03-28T17:56:19.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:19 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/1799595971,v1:172.21.15.71:6833/1799595971] up:reconnect 2024-03-28T17:56:19.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:19 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:reconnect} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:19.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:19 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00899985 2024-03-28T17:56:19.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:19 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999983 2024-03-28T17:56:19.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:19 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:56:19.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:19 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:56:19.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:19 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:56:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:19 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/1799595971,v1:172.21.15.71:6833/1799595971] up:reconnect 2024-03-28T17:56:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:19 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:reconnect} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:19 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00899985 2024-03-28T17:56:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:19 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999983 2024-03-28T17:56:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:19 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:56:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:19 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:56:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:19 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:56:19.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:19 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/1799595971,v1:172.21.15.71:6833/1799595971] up:reconnect 2024-03-28T17:56:19.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:19 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:reconnect} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:19.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:19 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00899985 2024-03-28T17:56:19.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:19 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999983 2024-03-28T17:56:19.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:19 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:56:19.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:19 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:56:19.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:19 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:56:19.882 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 232 sec latency 347779.194 ms 2024-03-28T17:56:20.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:20 smithi045 ceph-mon[31773]: pgmap v244: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:56:20.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:20 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/1799595971,v1:172.21.15.71:6833/1799595971] up:rejoin 2024-03-28T17:56:20.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:20 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:rejoin} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:20.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:20 smithi045 ceph-mon[31773]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-03-28T17:56:20.697 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-mon[36597]: pgmap v244: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:56:20.698 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/1799595971,v1:172.21.15.71:6833/1799595971] up:rejoin 2024-03-28T17:56:20.698 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:rejoin} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:20.698 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-mon[36597]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-03-28T17:56:20.698 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcb98388700 time 2024-03-28T17:56:20.344969+0000 2024-03-28T17:56:20.698 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:56:20.698 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:20.698 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcba29b24a1] 2024-03-28T17:56:20.699 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcba29b2667] 2024-03-28T17:56:20.699 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5578b1eb9223] 2024-03-28T17:56:20.699 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5578b1d335d6] 2024-03-28T17:56:20.699 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5578b1d33da1] 2024-03-28T17:56:20.699 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5578b1d362d5] 2024-03-28T17:56:20.699 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5578b1d86233] 2024-03-28T17:56:20.699 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5578b1d88197] 2024-03-28T17:56:20.699 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5578b1c8711b] 2024-03-28T17:56:20.699 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5578b1c876bc] 2024-03-28T17:56:20.699 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5578b1c711ff] 2024-03-28T17:56:20.699 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcba2c44488] 2024-03-28T17:56:20.699 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 13: (DispatchQueue::entry()+0x51f) [0x7fcba2c4160f] 2024-03-28T17:56:20.699 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcba2d14e81] 2024-03-28T17:56:20.699 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcba172f1ca] 2024-03-28T17:56:20.699 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 16: clone() 2024-03-28T17:56:20.700 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 2024-03-28T17:56:20.346+0000 7fcb98388700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcb98388700 time 2024-03-28T17:56:20.344969+0000 2024-03-28T17:56:20.700 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:56:20.700 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 2024-03-28T17:56:20.700 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:20.700 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcba29b24a1] 2024-03-28T17:56:20.700 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcba29b2667] 2024-03-28T17:56:20.700 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5578b1eb9223] 2024-03-28T17:56:20.701 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5578b1d335d6] 2024-03-28T17:56:20.701 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5578b1d33da1] 2024-03-28T17:56:20.701 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5578b1d362d5] 2024-03-28T17:56:20.701 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5578b1d86233] 2024-03-28T17:56:20.701 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5578b1d88197] 2024-03-28T17:56:20.701 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5578b1c8711b] 2024-03-28T17:56:20.701 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5578b1c876bc] 2024-03-28T17:56:20.701 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5578b1c711ff] 2024-03-28T17:56:20.701 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcba2c44488] 2024-03-28T17:56:20.701 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 13: (DispatchQueue::entry()+0x51f) [0x7fcba2c4160f] 2024-03-28T17:56:20.701 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcba2d14e81] 2024-03-28T17:56:20.701 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcba172f1ca] 2024-03-28T17:56:20.701 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 16: clone() 2024-03-28T17:56:20.701 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 2024-03-28T17:56:20.702 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: *** Caught signal (Aborted) ** 2024-03-28T17:56:20.702 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: in thread 7fcb98388700 thread_name:ms_dispatch 2024-03-28T17:56:20.702 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:20.702 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcba1739d20] 2024-03-28T17:56:20.702 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 2: gsignal() 2024-03-28T17:56:20.702 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 3: abort() 2024-03-28T17:56:20.702 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcba29b24fb] 2024-03-28T17:56:20.702 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcba29b2667] 2024-03-28T17:56:20.702 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5578b1eb9223] 2024-03-28T17:56:20.702 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5578b1d335d6] 2024-03-28T17:56:20.702 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5578b1d33da1] 2024-03-28T17:56:20.702 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5578b1d362d5] 2024-03-28T17:56:20.702 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5578b1d86233] 2024-03-28T17:56:20.702 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5578b1d88197] 2024-03-28T17:56:20.703 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5578b1c8711b] 2024-03-28T17:56:20.703 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5578b1c876bc] 2024-03-28T17:56:20.703 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5578b1c711ff] 2024-03-28T17:56:20.703 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcba2c44488] 2024-03-28T17:56:20.703 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 16: (DispatchQueue::entry()+0x51f) [0x7fcba2c4160f] 2024-03-28T17:56:20.703 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcba2d14e81] 2024-03-28T17:56:20.703 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcba172f1ca] 2024-03-28T17:56:20.703 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 19: clone() 2024-03-28T17:56:20.703 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 2024-03-28T17:56:20.348+0000 7fcb98388700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:56:20.703 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: in thread 7fcb98388700 thread_name:ms_dispatch 2024-03-28T17:56:20.703 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 2024-03-28T17:56:20.703 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:20.703 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcba1739d20] 2024-03-28T17:56:20.703 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 2: gsignal() 2024-03-28T17:56:20.703 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 3: abort() 2024-03-28T17:56:20.704 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcba29b24fb] 2024-03-28T17:56:20.704 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcba29b2667] 2024-03-28T17:56:20.704 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5578b1eb9223] 2024-03-28T17:56:20.705 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5578b1d335d6] 2024-03-28T17:56:20.705 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5578b1d33da1] 2024-03-28T17:56:20.705 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5578b1d362d5] 2024-03-28T17:56:20.705 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5578b1d86233] 2024-03-28T17:56:20.705 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5578b1d88197] 2024-03-28T17:56:20.705 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5578b1c8711b] 2024-03-28T17:56:20.705 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5578b1c876bc] 2024-03-28T17:56:20.705 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5578b1c711ff] 2024-03-28T17:56:20.705 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcba2c44488] 2024-03-28T17:56:20.705 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 16: (DispatchQueue::entry()+0x51f) [0x7fcba2c4160f] 2024-03-28T17:56:20.705 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcba2d14e81] 2024-03-28T17:56:20.705 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcba172f1ca] 2024-03-28T17:56:20.705 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 19: clone() 2024-03-28T17:56:20.706 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:56:20.706 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 2024-03-28T17:56:20.706 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: -1> 2024-03-28T17:56:20.346+0000 7fcb98388700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcb98388700 time 2024-03-28T17:56:20.344969+0000 2024-03-28T17:56:20.706 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:56:20.706 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 2024-03-28T17:56:20.706 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:20.706 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcba29b24a1] 2024-03-28T17:56:20.706 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcba29b2667] 2024-03-28T17:56:20.706 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5578b1eb9223] 2024-03-28T17:56:20.706 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5578b1d335d6] 2024-03-28T17:56:20.706 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5578b1d33da1] 2024-03-28T17:56:20.706 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5578b1d362d5] 2024-03-28T17:56:20.706 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5578b1d86233] 2024-03-28T17:56:20.706 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5578b1d88197] 2024-03-28T17:56:20.706 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5578b1c8711b] 2024-03-28T17:56:20.707 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5578b1c876bc] 2024-03-28T17:56:20.707 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5578b1c711ff] 2024-03-28T17:56:20.707 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcba2c44488] 2024-03-28T17:56:20.707 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 13: (DispatchQueue::entry()+0x51f) [0x7fcba2c4160f] 2024-03-28T17:56:20.707 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcba2d14e81] 2024-03-28T17:56:20.707 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcba172f1ca] 2024-03-28T17:56:20.707 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 16: clone() 2024-03-28T17:56:20.707 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 2024-03-28T17:56:20.707 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 0> 2024-03-28T17:56:20.348+0000 7fcb98388700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:56:20.707 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: in thread 7fcb98388700 thread_name:ms_dispatch 2024-03-28T17:56:20.707 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 2024-03-28T17:56:20.707 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:20.707 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcba1739d20] 2024-03-28T17:56:20.707 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 2: gsignal() 2024-03-28T17:56:20.707 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 3: abort() 2024-03-28T17:56:20.708 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcba29b24fb] 2024-03-28T17:56:20.708 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcba29b2667] 2024-03-28T17:56:20.708 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5578b1eb9223] 2024-03-28T17:56:20.708 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5578b1d335d6] 2024-03-28T17:56:20.708 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5578b1d33da1] 2024-03-28T17:56:20.708 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5578b1d362d5] 2024-03-28T17:56:20.708 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5578b1d86233] 2024-03-28T17:56:20.708 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5578b1d88197] 2024-03-28T17:56:20.708 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5578b1c8711b] 2024-03-28T17:56:20.708 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5578b1c876bc] 2024-03-28T17:56:20.708 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5578b1c711ff] 2024-03-28T17:56:20.708 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcba2c44488] 2024-03-28T17:56:20.708 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 16: (DispatchQueue::entry()+0x51f) [0x7fcba2c4160f] 2024-03-28T17:56:20.708 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcba2d14e81] 2024-03-28T17:56:20.709 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcba172f1ca] 2024-03-28T17:56:20.709 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 19: clone() 2024-03-28T17:56:20.709 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:56:20.709 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 2024-03-28T17:56:20.709 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: -1> 2024-03-28T17:56:20.346+0000 7fcb98388700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcb98388700 time 2024-03-28T17:56:20.344969+0000 2024-03-28T17:56:20.709 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:56:20.709 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 2024-03-28T17:56:20.709 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:20.709 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcba29b24a1] 2024-03-28T17:56:20.709 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcba29b2667] 2024-03-28T17:56:20.709 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5578b1eb9223] 2024-03-28T17:56:20.709 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5578b1d335d6] 2024-03-28T17:56:20.709 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5578b1d33da1] 2024-03-28T17:56:20.709 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5578b1d362d5] 2024-03-28T17:56:20.709 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5578b1d86233] 2024-03-28T17:56:20.710 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5578b1d88197] 2024-03-28T17:56:20.710 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5578b1c8711b] 2024-03-28T17:56:20.710 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5578b1c876bc] 2024-03-28T17:56:20.710 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5578b1c711ff] 2024-03-28T17:56:20.710 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcba2c44488] 2024-03-28T17:56:20.710 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 13: (DispatchQueue::entry()+0x51f) [0x7fcba2c4160f] 2024-03-28T17:56:20.710 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcba2d14e81] 2024-03-28T17:56:20.710 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcba172f1ca] 2024-03-28T17:56:20.710 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 16: clone() 2024-03-28T17:56:20.710 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 2024-03-28T17:56:20.710 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 0> 2024-03-28T17:56:20.348+0000 7fcb98388700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:56:20.710 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: in thread 7fcb98388700 thread_name:ms_dispatch 2024-03-28T17:56:20.710 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 2024-03-28T17:56:20.710 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:20.710 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcba1739d20] 2024-03-28T17:56:20.711 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 2: gsignal() 2024-03-28T17:56:20.711 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 3: abort() 2024-03-28T17:56:20.711 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcba29b24fb] 2024-03-28T17:56:20.711 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcba29b2667] 2024-03-28T17:56:20.711 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5578b1eb9223] 2024-03-28T17:56:20.711 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5578b1d335d6] 2024-03-28T17:56:20.711 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5578b1d33da1] 2024-03-28T17:56:20.711 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5578b1d362d5] 2024-03-28T17:56:20.711 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5578b1d86233] 2024-03-28T17:56:20.711 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5578b1d88197] 2024-03-28T17:56:20.711 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5578b1c8711b] 2024-03-28T17:56:20.711 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5578b1c876bc] 2024-03-28T17:56:20.711 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5578b1c711ff] 2024-03-28T17:56:20.711 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcba2c44488] 2024-03-28T17:56:20.712 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 16: (DispatchQueue::entry()+0x51f) [0x7fcba2c4160f] 2024-03-28T17:56:20.712 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcba2d14e81] 2024-03-28T17:56:20.712 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcba172f1ca] 2024-03-28T17:56:20.712 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 19: clone() 2024-03-28T17:56:20.712 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:56:20.712 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[71803]: 2024-03-28T17:56:20.712 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:20 smithi071 podman[71973]: 2024-03-28 17:56:20.407388447 +0000 UTC m=+0.024804461 container died 9e5364e23a694d79fd4cb653142ba4a95612691a408bf1c2521e2478cd286f89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True) 2024-03-28T17:56:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:20 smithi151 ceph-mon[36481]: pgmap v244: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:56:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:20 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/1799595971,v1:172.21.15.71:6833/1799595971] up:rejoin 2024-03-28T17:56:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:20 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:rejoin} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:20 smithi151 ceph-mon[36481]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-03-28T17:56:20.882 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 233 sec latency 348779.297 ms 2024-03-28T17:56:21.349 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:21 smithi071 podman[71973]: 2024-03-28 17:56:21.092069386 +0000 UTC m=+0.709485379 container remove 9e5364e23a694d79fd4cb653142ba4a95612691a408bf1c2521e2478cd286f89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T17:56:21.349 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:21 smithi071 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:56:21.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:21 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:56:21.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:21 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:56:21.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:21 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/1799595971,v1:172.21.15.71:6833/1799595971] up:active 2024-03-28T17:56:21.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:21 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:21.562 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:21 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T17:56:21.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:21 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:56:21.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:21 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:56:21.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:21 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/1799595971,v1:172.21.15.71:6833/1799595971] up:active 2024-03-28T17:56:21.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:21 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:21.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:21 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T17:56:21.812 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:21 smithi071 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.k.service: Failed with result 'exit-code'. 2024-03-28T17:56:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:21 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:56:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:21 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:56:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:21 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/1799595971,v1:172.21.15.71:6833/1799595971] up:active 2024-03-28T17:56:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:21 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:21 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T17:56:21.882 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 234 sec latency 349779.451 ms 2024-03-28T17:56:22.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:22 smithi045 ceph-mon[31773]: pgmap v245: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 6 op/s 2024-03-28T17:56:22.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:22 smithi071 ceph-mon[36597]: pgmap v245: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 6 op/s 2024-03-28T17:56:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:22 smithi151 ceph-mon[36481]: pgmap v245: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 6 op/s 2024-03-28T17:56:22.882 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 235 sec latency 350779.605 ms 2024-03-28T17:56:23.883 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 236 sec latency 351779.794 ms 2024-03-28T17:56:24.625 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:24 smithi151 ceph-mon[36481]: pgmap v246: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 6 op/s 2024-03-28T17:56:24.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:24 smithi045 ceph-mon[31773]: pgmap v246: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 6 op/s 2024-03-28T17:56:24.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:24 smithi071 ceph-mon[36597]: pgmap v246: 97 pgs: 97 active+clean; 27 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 6 op/s 2024-03-28T17:56:24.883 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 237 sec latency 352779.897 ms 2024-03-28T17:56:25.883 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 238 sec latency 353780.037 ms 2024-03-28T17:56:26.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:26 smithi045 ceph-mon[31773]: pgmap v247: 97 pgs: 97 active+clean; 27 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:56:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:26 smithi071 ceph-mon[36597]: pgmap v247: 97 pgs: 97 active+clean; 27 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:56:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:26 smithi151 ceph-mon[36481]: pgmap v247: 97 pgs: 97 active+clean; 27 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:56:26.883 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 239 sec latency 354780.204 ms 2024-03-28T17:56:27.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:27 smithi045 ceph-mon[31773]: osdmap e110: 12 total, 12 up, 12 in 2024-03-28T17:56:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:27 smithi071 ceph-mon[36597]: osdmap e110: 12 total, 12 up, 12 in 2024-03-28T17:56:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:27 smithi151 ceph-mon[36481]: osdmap e110: 12 total, 12 up, 12 in 2024-03-28T17:56:27.883 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 240 sec latency 355780.321 ms 2024-03-28T17:56:28.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:28 smithi045 ceph-mon[31773]: pgmap v249: 97 pgs: 97 active+clean; 27 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:56:28.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:28 smithi071 ceph-mon[36597]: pgmap v249: 97 pgs: 97 active+clean; 27 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:56:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:28 smithi151 ceph-mon[36481]: pgmap v249: 97 pgs: 97 active+clean; 27 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:56:28.883 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 241 sec latency 356780.438 ms 2024-03-28T17:56:29.883 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 242 sec latency 357780.575 ms 2024-03-28T17:56:30.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:30 smithi045 ceph-mon[31773]: pgmap v250: 97 pgs: 97 active+clean; 27 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:56:30.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:30 smithi071 ceph-mon[36597]: pgmap v250: 97 pgs: 97 active+clean; 27 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:56:30.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:30 smithi151 ceph-mon[36481]: pgmap v250: 97 pgs: 97 active+clean; 27 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:56:30.883 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 243 sec latency 358780.741 ms 2024-03-28T17:56:31.884 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 244 sec latency 359780.945 ms 2024-03-28T17:56:31.911 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:31 smithi071 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.k.service: Scheduled restart job, restart counter is at 5. 2024-03-28T17:56:31.911 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:31 smithi071 systemd[1]: Stopped Ceph mds.k for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:56:31.911 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:31 smithi071 systemd[1]: Starting Ceph mds.k for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:56:31.911 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:31 smithi071 podman[72061]: 2024-03-28 17:56:31.68829259 +0000 UTC m=+0.018136604 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:56:31.911 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:31 smithi071 podman[72061]: 2024-03-28 17:56:31.808829544 +0000 UTC m=+0.138673535 container create 96111c0ee808121987f06f7ca9bdb8a341fd7977ee1fcb495b792f73dd323c8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-28T17:56:32.162 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:32 smithi071 podman[72061]: 2024-03-28 17:56:32.053068082 +0000 UTC m=+0.382912097 container init 96111c0ee808121987f06f7ca9bdb8a341fd7977ee1fcb495b792f73dd323c8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-03-28T17:56:32.162 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:32 smithi071 podman[72061]: 2024-03-28 17:56:32.055683919 +0000 UTC m=+0.385527918 container start 96111c0ee808121987f06f7ca9bdb8a341fd7977ee1fcb495b792f73dd323c8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T17:56:32.162 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:32 smithi071 ceph-mds[72080]: -- 172.21.15.71:0/3299772153 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f8f3a3fa40 con 0x55f8f3a12c00 2024-03-28T17:56:32.162 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:32 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: starting mds.k at 2024-03-28T17:56:32.162 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:32 smithi071 bash[72061]: 96111c0ee808121987f06f7ca9bdb8a341fd7977ee1fcb495b792f73dd323c8e 2024-03-28T17:56:32.162 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:32 smithi071 systemd[1]: Started Ceph mds.k for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:56:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:32 smithi045 ceph-mon[31773]: pgmap v251: 97 pgs: 97 active+clean; 27 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:56:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:32 smithi045 ceph-mon[31773]: daemon mds.k restarted 2024-03-28T17:56:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:32 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:56:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:32 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T17:56:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:32 smithi045 ceph-mon[31773]: osdmap e111: 12 total, 12 up, 12 in 2024-03-28T17:56:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:32 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/1081652393,v1:172.21.15.71:6833/1081652393] up:boot 2024-03-28T17:56:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:32 smithi045 ceph-mon[31773]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-03-28T17:56:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:32 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-28T17:56:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:32 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:replay} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:32.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:32 smithi071 ceph-mon[36597]: pgmap v251: 97 pgs: 97 active+clean; 27 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:56:32.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:32 smithi071 ceph-mon[36597]: daemon mds.k restarted 2024-03-28T17:56:32.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:32 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:56:32.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:32 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T17:56:32.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:32 smithi071 ceph-mon[36597]: osdmap e111: 12 total, 12 up, 12 in 2024-03-28T17:56:32.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:32 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/1081652393,v1:172.21.15.71:6833/1081652393] up:boot 2024-03-28T17:56:32.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:32 smithi071 ceph-mon[36597]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-03-28T17:56:32.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:32 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-28T17:56:32.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:32 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:replay} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:32 smithi151 ceph-mon[36481]: pgmap v251: 97 pgs: 97 active+clean; 27 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:56:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:32 smithi151 ceph-mon[36481]: daemon mds.k restarted 2024-03-28T17:56:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:32 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:56:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:32 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T17:56:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:32 smithi151 ceph-mon[36481]: osdmap e111: 12 total, 12 up, 12 in 2024-03-28T17:56:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:32 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/1081652393,v1:172.21.15.71:6833/1081652393] up:boot 2024-03-28T17:56:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:32 smithi151 ceph-mon[36481]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-03-28T17:56:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:32 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-28T17:56:32.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:32 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:replay} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:32.884 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 245 sec latency 360781.098 ms 2024-03-28T17:56:33.884 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 246 sec latency 361781.252 ms 2024-03-28T17:56:34.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:34 smithi045 ceph-mon[31773]: pgmap v253: 97 pgs: 97 active+clean; 27 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T17:56:34.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:34 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/1898596562,v1:172.21.15.151:6833/1898596562] up:resolve 2024-03-28T17:56:34.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:34 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:resolve} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:34.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:34 smithi071 ceph-mon[36597]: pgmap v253: 97 pgs: 97 active+clean; 27 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T17:56:34.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:34 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/1898596562,v1:172.21.15.151:6833/1898596562] up:resolve 2024-03-28T17:56:34.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:34 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:resolve} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:34.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:34 smithi151 ceph-mon[36481]: pgmap v253: 97 pgs: 97 active+clean; 27 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T17:56:34.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:34 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/1898596562,v1:172.21.15.151:6833/1898596562] up:resolve 2024-03-28T17:56:34.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:34 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:resolve} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:34.884 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 247 sec latency 362781.402 ms 2024-03-28T17:56:35.100 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:56:35.100 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:56:35.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.339+0000 7fcffbe00640 1 -- 172.21.15.45:0/2322089536 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcffc1040e0 msgr2=0x7fcffc1105f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:56:35.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.339+0000 7fcffbe00640 1 --2- 172.21.15.45:0/2322089536 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcffc1040e0 0x7fcffc1105f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fcff4009ca0 tx=0x7fcff402c100 comp rx=0 tx=0).stop 2024-03-28T17:56:35.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.339+0000 7fcffbe00640 1 -- 172.21.15.45:0/2322089536 shutdown_connections 2024-03-28T17:56:35.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.339+0000 7fcffbe00640 1 --2- 172.21.15.45:0/2322089536 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcffc1040e0 0x7fcffc1105f0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.339+0000 7fcffbe00640 1 --2- 172.21.15.45:0/2322089536 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcffc103720 0x7fcffc103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.339+0000 7fcffbe00640 1 --2- 172.21.15.45:0/2322089536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcffc109730 0x7fcffc109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.339+0000 7fcffbe00640 1 -- 172.21.15.45:0/2322089536 >> 172.21.15.45:0/2322089536 conn(0x7fcffc0ff1b0 msgr2=0x7fcffc1015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:56:35.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.339+0000 7fcffbe00640 1 -- 172.21.15.45:0/2322089536 shutdown_connections 2024-03-28T17:56:35.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.339+0000 7fcffbe00640 1 -- 172.21.15.45:0/2322089536 wait complete. 2024-03-28T17:56:35.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.341+0000 7fcffbe00640 1 Processor -- start 2024-03-28T17:56:35.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.341+0000 7fcffbe00640 1 -- start start 2024-03-28T17:56:35.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.342+0000 7fcffbe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcffc103720 0x7fcffc199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:35.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.342+0000 7fcffbe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcffc1040e0 0x7fcffc19a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:35.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.342+0000 7fcffbe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcffc109730 0x7fcffc19e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:35.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.342+0000 7fcffbe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc113b40 con 0x7fcffc109730 2024-03-28T17:56:35.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.342+0000 7fcffbe00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc1139c0 con 0x7fcffc1040e0 2024-03-28T17:56:35.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.342+0000 7fcffbe00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc113cc0 con 0x7fcffc103720 2024-03-28T17:56:35.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.342+0000 7fcffaa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcffc103720 0x7fcffc199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:35.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.342+0000 7fcff3e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcffc1040e0 0x7fcffc19a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:35.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.342+0000 7fcffaa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcffc103720 0x7fcffc199fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:52848/0 (socket says 172.21.15.45:52848) 2024-03-28T17:56:35.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.342+0000 7fcff3e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcffc1040e0 0x7fcffc19a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:51552/0 (socket says 172.21.15.45:51552) 2024-03-28T17:56:35.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.342+0000 7fcffaa00640 1 -- 172.21.15.45:0/985570412 learned_addr learned my addr 172.21.15.45:0/985570412 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:56:35.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.342+0000 7fcffb400640 1 --2- 172.21.15.45:0/985570412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcffc109730 0x7fcffc19e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:35.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.342+0000 7fcff3e00640 1 -- 172.21.15.45:0/985570412 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcffc103720 msgr2=0x7fcffc199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:56:35.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.342+0000 7fcff3e00640 1 --2- 172.21.15.45:0/985570412 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcffc103720 0x7fcffc199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.342+0000 7fcff3e00640 1 -- 172.21.15.45:0/985570412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcffc109730 msgr2=0x7fcffc19e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:56:35.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.342+0000 7fcff3e00640 1 --2- 172.21.15.45:0/985570412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcffc109730 0x7fcffc19e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.342+0000 7fcff3e00640 1 -- 172.21.15.45:0/985570412 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcffc19efb0 con 0x7fcffc1040e0 2024-03-28T17:56:35.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.343+0000 7fcff3e00640 1 --2- 172.21.15.45:0/985570412 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcffc1040e0 0x7fcffc19a500 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400e9f0 tx=0x7fcfe400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:56:35.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.343+0000 7fcff3400640 1 -- 172.21.15.45:0/985570412 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe400ccc0 con 0x7fcffc1040e0 2024-03-28T17:56:35.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.343+0000 7fcff3400640 1 -- 172.21.15.45:0/985570412 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfe4005020 con 0x7fcffc1040e0 2024-03-28T17:56:35.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.343+0000 7fcffbe00640 1 -- 172.21.15.45:0/985570412 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcffc19f2a0 con 0x7fcffc1040e0 2024-03-28T17:56:35.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.343+0000 7fcff3400640 1 -- 172.21.15.45:0/985570412 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe40049e0 con 0x7fcffc1040e0 2024-03-28T17:56:35.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.343+0000 7fcffbe00640 1 -- 172.21.15.45:0/985570412 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcffc1a6b30 con 0x7fcffc1040e0 2024-03-28T17:56:35.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.346+0000 7fcff3400640 1 -- 172.21.15.45:0/985570412 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fcfe4005560 con 0x7fcffc1040e0 2024-03-28T17:56:35.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.346+0000 7fd00486a740 1 Processor -- start 2024-03-28T17:56:35.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.347+0000 7fd00486a740 1 -- start start 2024-03-28T17:56:35.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.347+0000 7fd00486a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562009748dd0 0x562009747b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:35.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.347+0000 7fd00486a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5620097480f0 0x562009748570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:35.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.347+0000 7fd00486a740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562009751460 0x562009753850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:35.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.347+0000 7fd00486a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620096ac250 con 0x562009748dd0 2024-03-28T17:56:35.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.347+0000 7fd00486a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x562009654a30 con 0x5620097480f0 2024-03-28T17:56:35.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.347+0000 7fd00486a740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620096a82f0 con 0x562009751460 2024-03-28T17:56:35.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.347+0000 7fcffb400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562009748dd0 0x562009747b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:35.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.347+0000 7fcffaa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5620097480f0 0x562009748570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:35.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.347+0000 7fcffaa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5620097480f0 0x562009748570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:51558/0 (socket says 172.21.15.45:51558) 2024-03-28T17:56:35.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.347+0000 7fcffaa00640 1 -- 172.21.15.45:0/4238819651 learned_addr learned my addr 172.21.15.45:0/4238819651 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:56:35.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.348+0000 7fcff3e00640 1 --2- 172.21.15.45:0/4238819651 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562009751460 0x562009753850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:35.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.348+0000 7fcffaa00640 1 -- 172.21.15.45:0/4238819651 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562009751460 msgr2=0x562009753850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:56:35.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.348+0000 7fcffaa00640 1 --2- 172.21.15.45:0/4238819651 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562009751460 0x562009753850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.348+0000 7fcffaa00640 1 -- 172.21.15.45:0/4238819651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562009748dd0 msgr2=0x562009747b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:56:35.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.348+0000 7fcffaa00640 1 --2- 172.21.15.45:0/4238819651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562009748dd0 0x562009747b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.348+0000 7fcffaa00640 1 -- 172.21.15.45:0/4238819651 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56200963f690 con 0x5620097480f0 2024-03-28T17:56:35.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.348+0000 7fcffaa00640 1 --2- 172.21.15.45:0/4238819651 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5620097480f0 0x562009748570 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009d40 tx=0x7fcfec011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=823bdbe146d4032b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:56:35.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.348+0000 7fcfebe00640 1 -- 172.21.15.45:0/4238819651 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec004220 con 0x5620097480f0 2024-03-28T17:56:35.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.348+0000 7fcfebe00640 1 -- 172.21.15.45:0/4238819651 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfec0043c0 con 0x5620097480f0 2024-03-28T17:56:35.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.348+0000 7fcfebe00640 1 -- 172.21.15.45:0/4238819651 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec0035f0 con 0x5620097480f0 2024-03-28T17:56:35.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.348+0000 7fcff3400640 1 --2- 172.21.15.45:0/985570412 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fcfe00a2970 0x7fcfe00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:35.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.349+0000 7fcff3400640 1 -- 172.21.15.45:0/985570412 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 10448+0+0 (secure 0 0 0) 0x7fcfe40cf5f0 con 0x7fcffc1040e0 2024-03-28T17:56:35.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.349+0000 7fcffb400640 1 --2- 172.21.15.45:0/985570412 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fcfe00a2970 0x7fcfe00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:35.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.349+0000 7fd00486a740 1 -- 172.21.15.45:0/4238819651 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5620097480f0 msgr2=0x562009748570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:56:35.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.349+0000 7fd00486a740 1 --2- 172.21.15.45:0/4238819651 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5620097480f0 0x562009748570 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009d40 tx=0x7fcfec011810 comp rx=0 tx=0).stop 2024-03-28T17:56:35.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.349+0000 7fd00486a740 1 -- 172.21.15.45:0/4238819651 shutdown_connections 2024-03-28T17:56:35.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.349+0000 7fd00486a740 1 --2- 172.21.15.45:0/4238819651 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562009751460 0x562009753850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.349+0000 7fd00486a740 1 --2- 172.21.15.45:0/4238819651 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5620097480f0 0x562009748570 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.349+0000 7fd00486a740 1 --2- 172.21.15.45:0/4238819651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562009748dd0 0x562009747b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.350+0000 7fd00486a740 1 -- 172.21.15.45:0/4238819651 >> 172.21.15.45:0/4238819651 conn(0x56200970f200 msgr2=0x562009742b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:56:35.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.350+0000 7fd00486a740 1 -- 172.21.15.45:0/4238819651 shutdown_connections 2024-03-28T17:56:35.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.350+0000 7fcffb400640 1 --2- 172.21.15.45:0/985570412 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fcfe00a2970 0x7fcfe00a4e30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcff4005e30 tx=0x7fcff4005d80 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:56:35.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.350+0000 7fd00486a740 1 -- 172.21.15.45:0/4238819651 wait complete. 2024-03-28T17:56:35.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.350+0000 7fd00486a740 1 -- start start 2024-03-28T17:56:35.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.350+0000 7fd00486a740 1 Processor -- start 2024-03-28T17:56:35.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.351+0000 7fd00486a740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5620097480f0 0x562009750b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:35.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.351+0000 7fd00486a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562009748dd0 0x56200974dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:35.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.351+0000 7fd00486a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562009751460 0x56200974e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:35.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.351+0000 7fd00486a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620097542d0 con 0x562009748dd0 2024-03-28T17:56:35.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.351+0000 7fd00486a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x562009667e80 con 0x562009751460 2024-03-28T17:56:35.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.351+0000 7fd00486a740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x562009754450 con 0x5620097480f0 2024-03-28T17:56:35.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.351+0000 7fd00486a740 10 client.0 _ll_register_callbacks cb 0x56200967a130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:56:35.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.351+0000 7fcffaa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5620097480f0 0x562009750b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:35.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.351+0000 7fcffaa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5620097480f0 0x562009750b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:52864/0 (socket says 172.21.15.45:52864) 2024-03-28T17:56:35.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.351+0000 7fcffaa00640 1 -- 172.21.15.45:0/1688810066 learned_addr learned my addr 172.21.15.45:0/1688810066 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:56:35.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.351+0000 7fcff3e00640 1 --2- 172.21.15.45:0/1688810066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562009748dd0 0x56200974dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:35.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.351+0000 7fcffb400640 1 --2- 172.21.15.45:0/1688810066 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562009751460 0x56200974e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:35.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.352+0000 7fcffaa00640 1 -- 172.21.15.45:0/1688810066 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562009751460 msgr2=0x56200974e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:56:35.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.352+0000 7fcffaa00640 1 --2- 172.21.15.45:0/1688810066 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562009751460 0x56200974e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.352+0000 7fcffaa00640 1 -- 172.21.15.45:0/1688810066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562009748dd0 msgr2=0x56200974dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:56:35.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.352+0000 7fcffaa00640 1 --2- 172.21.15.45:0/1688810066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562009748dd0 0x56200974dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.352+0000 7fcffaa00640 1 -- 172.21.15.45:0/1688810066 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56200948a7d0 con 0x5620097480f0 2024-03-28T17:56:35.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.352+0000 7fcffaa00640 1 --2- 172.21.15.45:0/1688810066 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5620097480f0 0x562009750b50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcfec000c00 tx=0x7fcfec0052a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20eef6364d1172b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:56:35.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.352+0000 7fcfe8c00640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:56:35.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.352+0000 7fcfe8c00640 1 -- 172.21.15.45:0/1688810066 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec028020 con 0x5620097480f0 2024-03-28T17:56:35.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.352+0000 7fcfe8c00640 1 -- 172.21.15.45:0/1688810066 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 10448+0+0 (secure 0 0 0) 0x7fcfec005410 con 0x5620097480f0 2024-03-28T17:56:35.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.352+0000 7fd00486a740 1 -- 172.21.15.45:0/1688810066 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcfec0043c0 con 0x5620097480f0 2024-03-28T17:56:35.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.353+0000 7fcfe8c00640 1 -- 172.21.15.45:0/1688810066 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=166) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfec033050 con 0x5620097480f0 2024-03-28T17:56:35.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.353+0000 7fd00486a740 10 client.26167 fetch_fsmap learned FSMap version 166 2024-03-28T17:56:35.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.353+0000 7fd00486a740 1 -- 172.21.15.45:0/1688810066 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=166,monmap=4+}) v3 -- 0x5620093c03f0 con 0x5620097480f0 2024-03-28T17:56:35.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.353+0000 7fcfe8c00640 1 -- 172.21.15.45:0/1688810066 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfec0043c0 con 0x5620097480f0 2024-03-28T17:56:35.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.353+0000 7fcfe8c00640 1 -- 172.21.15.45:0/1688810066 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 166) v1 ==== 6197+0+0 (secure 0 0 0) 0x7fcfec023070 con 0x5620097480f0 2024-03-28T17:56:35.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.353+0000 7fd00486a740 10 client.26167 fetch_fsmap finished waiting for FSMap version 166 2024-03-28T17:56:35.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.353+0000 7fd00486a740 10 client.26167 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:56:35.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.353+0000 7fd00486a740 20 client.26167 populate_metadata read hostname 'smithi045' 2024-03-28T17:56:35.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.353+0000 7fd00486a740 1 --2- 172.21.15.45:0/1688810066 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x562009761400 0x562009763850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:35.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.353+0000 7fd00486a740 4 client.26167 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:56:35.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.353+0000 7fd00486a740 1 -- 172.21.15.45:0/1688810066 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5620096b9790 con 0x562009761400 2024-03-28T17:56:35.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.354+0000 7fcff3e00640 1 --2- 172.21.15.45:0/1688810066 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x562009761400 0x562009763850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:35.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.354+0000 7fcff3e00640 1 --2- 172.21.15.45:0/1688810066 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x562009761400 0x562009763850 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1805c8d5e7aa133 server_cookie=a2b6eb9aa372de41 in_seq=0 out_seq=0 2024-03-28T17:56:35.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.354+0000 7fcfe8c00640 10 client.26167 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:56:35.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.355+0000 7fcfe8c00640 1 -- 172.21.15.45:0/1688810066 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5620096b9790 con 0x562009761400 2024-03-28T17:56:35.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.355+0000 7fcfe8c00640 10 client.26167 handle_command_reply: tid=0 2024-03-28T17:56:35.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.355+0000 7fd00486a740 1 client.26167 shutdown 2024-03-28T17:56:35.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.355+0000 7fd00486a740 1 -- 172.21.15.45:0/1688810066 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5620097480f0 msgr2=0x562009750b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:56:35.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.355+0000 7fd00486a740 1 --2- 172.21.15.45:0/1688810066 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5620097480f0 0x562009750b50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcfec000c00 tx=0x7fcfec0052a0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.355+0000 7fd00486a740 1 -- 172.21.15.45:0/1688810066 shutdown_connections 2024-03-28T17:56:35.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.355+0000 7fd00486a740 1 --2- 172.21.15.45:0/1688810066 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x562009761400 0x562009763850 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.355+0000 7fd00486a740 1 --2- 172.21.15.45:0/1688810066 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562009751460 0x56200974e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.355+0000 7fd00486a740 1 --2- 172.21.15.45:0/1688810066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562009748dd0 0x56200974dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.355+0000 7fd00486a740 1 --2- 172.21.15.45:0/1688810066 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5620097480f0 0x562009750b50 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.355+0000 7fcfe8c00640 0 client.26167 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:56:35.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.355+0000 7fd00486a740 1 -- 172.21.15.45:0/1688810066 >> 172.21.15.45:0/1688810066 conn(0x56200970f200 msgr2=0x562009744890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:56:35.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.355+0000 7fd00486a740 1 -- 172.21.15.45:0/1688810066 shutdown_connections 2024-03-28T17:56:35.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.356+0000 7fd00486a740 1 -- 172.21.15.45:0/1688810066 wait complete. 2024-03-28T17:56:35.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.356+0000 7fd00486a740 20 client.26167 trim_cache size 0 max 16384 2024-03-28T17:56:35.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.373+0000 7fd00486a740 1 Processor -- start 2024-03-28T17:56:35.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.373+0000 7fd00486a740 1 -- start start 2024-03-28T17:56:35.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.374+0000 7fd00486a740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5620097480f0 0x562009750610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:35.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.374+0000 7fd00486a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562009748dd0 0x562009750b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:35.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.374+0000 7fd00486a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562009751460 0x562009752850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:35.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.374+0000 7fd00486a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc1b1d90 con 0x562009748dd0 2024-03-28T17:56:35.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.374+0000 7fd00486a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc1b10d0 con 0x562009751460 2024-03-28T17:56:35.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.374+0000 7fd00486a740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc1b1730 con 0x5620097480f0 2024-03-28T17:56:35.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.374+0000 7fcff3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562009748dd0 0x562009750b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:35.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.374+0000 7fcffb400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562009751460 0x562009752850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:35.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.374+0000 7fcff3e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562009748dd0 0x562009750b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57602/0 (socket says 172.21.15.45:57602) 2024-03-28T17:56:35.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.374+0000 7fcffb400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562009751460 0x562009752850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:51580/0 (socket says 172.21.15.45:51580) 2024-03-28T17:56:35.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.374+0000 7fcff3e00640 1 -- 172.21.15.45:0/2252733542 learned_addr learned my addr 172.21.15.45:0/2252733542 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:56:35.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.374+0000 7fcffaa00640 1 --2- 172.21.15.45:0/2252733542 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5620097480f0 0x562009750610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:35.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.374+0000 7fcffb400640 1 -- 172.21.15.45:0/2252733542 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5620097480f0 msgr2=0x562009750610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:56:35.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.374+0000 7fcffb400640 1 --2- 172.21.15.45:0/2252733542 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5620097480f0 0x562009750610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.374+0000 7fcffb400640 1 -- 172.21.15.45:0/2252733542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562009748dd0 msgr2=0x562009750b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:56:35.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.374+0000 7fcffb400640 1 --2- 172.21.15.45:0/2252733542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562009748dd0 0x562009750b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.374+0000 7fcffb400640 1 -- 172.21.15.45:0/2252733542 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5620093800f0 con 0x562009751460 2024-03-28T17:56:35.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.375+0000 7fcffb400640 1 --2- 172.21.15.45:0/2252733542 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562009751460 0x562009752850 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fcff4009bf0 tx=0x7fcff4041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0a9b9f5cb3b1374 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:56:35.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.375+0000 7fcfeb400640 1 -- 172.21.15.45:0/2252733542 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff4053070 con 0x562009751460 2024-03-28T17:56:35.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.375+0000 7fcfeb400640 1 -- 172.21.15.45:0/2252733542 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcff403a040 con 0x562009751460 2024-03-28T17:56:35.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.375+0000 7fcfeb400640 1 -- 172.21.15.45:0/2252733542 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff4039040 con 0x562009751460 2024-03-28T17:56:35.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.375+0000 7fd00486a740 1 -- 172.21.15.45:0/2252733542 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562009751460 msgr2=0x562009752850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:56:35.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.375+0000 7fd00486a740 1 --2- 172.21.15.45:0/2252733542 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562009751460 0x562009752850 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fcff4009bf0 tx=0x7fcff4041000 comp rx=0 tx=0).stop 2024-03-28T17:56:35.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.376+0000 7fd00486a740 1 -- 172.21.15.45:0/2252733542 shutdown_connections 2024-03-28T17:56:35.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.376+0000 7fd00486a740 1 --2- 172.21.15.45:0/2252733542 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562009751460 0x562009752850 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.376+0000 7fd00486a740 1 --2- 172.21.15.45:0/2252733542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562009748dd0 0x562009750b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.376+0000 7fd00486a740 1 --2- 172.21.15.45:0/2252733542 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5620097480f0 0x562009750610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.376+0000 7fd00486a740 1 -- 172.21.15.45:0/2252733542 >> 172.21.15.45:0/2252733542 conn(0x56200970f200 msgr2=0x56200974dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:56:35.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.376+0000 7fd00486a740 1 -- 172.21.15.45:0/2252733542 shutdown_connections 2024-03-28T17:56:35.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.376+0000 7fd00486a740 1 -- 172.21.15.45:0/2252733542 wait complete. 2024-03-28T17:56:35.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.376+0000 7fd00486a740 1 -- start start 2024-03-28T17:56:35.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.376+0000 7fd00486a740 1 Processor -- start 2024-03-28T17:56:35.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.377+0000 7fd00486a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5620097480f0 0x562009759500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:35.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.377+0000 7fd00486a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562009748dd0 0x562009754ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:35.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.377+0000 7fd00486a740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562009751460 0x562009755010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:35.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.377+0000 7fd00486a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc1ae670 con 0x5620097480f0 2024-03-28T17:56:35.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.377+0000 7fd00486a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc1af930 con 0x562009748dd0 2024-03-28T17:56:35.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.377+0000 7fd00486a740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc1aec70 con 0x562009751460 2024-03-28T17:56:35.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.377+0000 7fd00486a740 10 client.0 _ll_register_callbacks cb 0x7fcfec005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:56:35.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.377+0000 7fcff3e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562009748dd0 0x562009754ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:35.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.377+0000 7fcffaa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5620097480f0 0x562009759500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:35.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.377+0000 7fcff3e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562009748dd0 0x562009754ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:51584/0 (socket says 172.21.15.45:51584) 2024-03-28T17:56:35.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.377+0000 7fcff3e00640 1 -- 172.21.15.45:0/1096955849 learned_addr learned my addr 172.21.15.45:0/1096955849 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:56:35.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.377+0000 7fcffb400640 1 --2- 172.21.15.45:0/1096955849 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562009751460 0x562009755010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:35.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.377+0000 7fcff3e00640 1 -- 172.21.15.45:0/1096955849 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562009751460 msgr2=0x562009755010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:56:35.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.378+0000 7fcff3e00640 1 --2- 172.21.15.45:0/1096955849 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562009751460 0x562009755010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.378+0000 7fcff3e00640 1 -- 172.21.15.45:0/1096955849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5620097480f0 msgr2=0x562009759500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:56:35.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.378+0000 7fcff3e00640 1 --2- 172.21.15.45:0/1096955849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5620097480f0 0x562009759500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.378+0000 7fcff3e00640 1 -- 172.21.15.45:0/1096955849 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562009755840 con 0x562009748dd0 2024-03-28T17:56:35.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.378+0000 7fcff3e00640 1 --2- 172.21.15.45:0/1096955849 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562009748dd0 0x562009754ab0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40978f0 tx=0x7fcfe4097040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84d54af98b07ac6b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:56:35.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.378+0000 7fcfe8c00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:56:35.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.378+0000 7fcfe8c00640 1 -- 172.21.15.45:0/1096955849 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe401c040 con 0x562009748dd0 2024-03-28T17:56:35.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.378+0000 7fd00486a740 1 -- 172.21.15.45:0/1096955849 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcff403a040 con 0x562009748dd0 2024-03-28T17:56:35.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.378+0000 7fcfe8c00640 1 -- 172.21.15.45:0/1096955849 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 10448+0+0 (secure 0 0 0) 0x7fcfe40cc800 con 0x562009748dd0 2024-03-28T17:56:35.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.378+0000 7fcfe8c00640 1 -- 172.21.15.45:0/1096955849 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=166) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfe40cca00 con 0x562009748dd0 2024-03-28T17:56:35.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.378+0000 7fd00486a740 10 client.26231 fetch_fsmap learned FSMap version 166 2024-03-28T17:56:35.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.378+0000 7fd00486a740 1 -- 172.21.15.45:0/1096955849 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=166,monmap=4+}) v3 -- 0x5620097bcd60 con 0x562009748dd0 2024-03-28T17:56:35.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.378+0000 7fcfe8c00640 1 -- 172.21.15.45:0/1096955849 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcff403a040 con 0x562009748dd0 2024-03-28T17:56:35.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.379+0000 7fcfe8c00640 1 -- 172.21.15.45:0/1096955849 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 166) v1 ==== 6197+0+0 (secure 0 0 0) 0x7fcfe4011cd0 con 0x562009748dd0 2024-03-28T17:56:35.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.379+0000 7fd00486a740 10 client.26231 fetch_fsmap finished waiting for FSMap version 166 2024-03-28T17:56:35.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.379+0000 7fd00486a740 10 client.26231 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:56:35.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.379+0000 7fd00486a740 20 client.26231 populate_metadata read hostname 'smithi045' 2024-03-28T17:56:35.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.379+0000 7fd00486a740 1 --2- 172.21.15.45:0/1096955849 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5620097bd460 0x5620097bf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:56:35.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.379+0000 7fd00486a740 4 client.26231 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:56:35.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.379+0000 7fd00486a740 1 -- 172.21.15.45:0/1096955849 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5620096b3ae0 con 0x5620097bd460 2024-03-28T17:56:35.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.379+0000 7fcffaa00640 1 --2- 172.21.15.45:0/1096955849 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5620097bd460 0x5620097bf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:56:35.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.379+0000 7fcffaa00640 1 --2- 172.21.15.45:0/1096955849 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5620097bd460 0x5620097bf880 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48db3a4fbe36dacf server_cookie=6c9c654d6d0da88e in_seq=0 out_seq=0 2024-03-28T17:56:35.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.379+0000 7fcfe8c00640 10 client.26231 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:56:35.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.380+0000 7fcfe8c00640 1 -- 172.21.15.45:0/1096955849 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5620096b3ae0 con 0x5620097bd460 2024-03-28T17:56:35.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.380+0000 7fcfe8c00640 10 client.26231 handle_command_reply: tid=0 2024-03-28T17:56:35.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.380+0000 7fd00486a740 1 client.26231 shutdown 2024-03-28T17:56:35.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.380+0000 7fd00486a740 1 -- 172.21.15.45:0/1096955849 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562009748dd0 msgr2=0x562009754ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:56:35.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.380+0000 7fd00486a740 1 --2- 172.21.15.45:0/1096955849 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562009748dd0 0x562009754ab0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40978f0 tx=0x7fcfe4097040 comp rx=0 tx=0).stop 2024-03-28T17:56:35.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.380+0000 7fd00486a740 1 -- 172.21.15.45:0/1096955849 shutdown_connections 2024-03-28T17:56:35.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.380+0000 7fd00486a740 1 --2- 172.21.15.45:0/1096955849 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5620097bd460 0x5620097bf880 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.380+0000 7fd00486a740 1 --2- 172.21.15.45:0/1096955849 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562009751460 0x562009755010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.380+0000 7fcfe8c00640 0 client.26231 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:56:35.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.380+0000 7fd00486a740 1 --2- 172.21.15.45:0/1096955849 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562009748dd0 0x562009754ab0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.380+0000 7fd00486a740 1 --2- 172.21.15.45:0/1096955849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5620097480f0 0x562009759500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.380+0000 7fd00486a740 1 -- 172.21.15.45:0/1096955849 >> 172.21.15.45:0/1096955849 conn(0x56200970f200 msgr2=0x56200974c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:56:35.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.380+0000 7fd00486a740 1 -- 172.21.15.45:0/1096955849 shutdown_connections 2024-03-28T17:56:35.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.380+0000 7fd00486a740 1 -- 172.21.15.45:0/1096955849 wait complete. 2024-03-28T17:56:35.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.380+0000 7fd00486a740 20 client.26231 trim_cache size 0 max 16384 2024-03-28T17:56:35.386 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:56:35.386 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T17:56:35.386 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T17:56:35.386 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T17:56:35.386 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T17:56:35.386 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T17:56:35.386 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T17:56:35.386 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:56:35.386 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:56:35.387 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:56:35.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.381+0000 7fcffbe00640 1 -- 172.21.15.45:0/985570412 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fcfe00a2970 msgr2=0x7fcfe00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:56:35.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.381+0000 7fcffbe00640 1 --2- 172.21.15.45:0/985570412 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fcfe00a2970 0x7fcfe00a4e30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcff4005e30 tx=0x7fcff4005d80 comp rx=0 tx=0).stop 2024-03-28T17:56:35.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.381+0000 7fcffbe00640 1 -- 172.21.15.45:0/985570412 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcffc1040e0 msgr2=0x7fcffc19a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:56:35.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.381+0000 7fcffbe00640 1 --2- 172.21.15.45:0/985570412 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcffc1040e0 0x7fcffc19a500 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400e9f0 tx=0x7fcfe400eeb0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.381+0000 7fcffbe00640 1 -- 172.21.15.45:0/985570412 shutdown_connections 2024-03-28T17:56:35.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.381+0000 7fcffbe00640 1 --2- 172.21.15.45:0/985570412 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fcfe00a2970 0x7fcfe00a4e30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.381+0000 7fcffbe00640 1 --2- 172.21.15.45:0/985570412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcffc109730 0x7fcffc19e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.381+0000 7fcffbe00640 1 --2- 172.21.15.45:0/985570412 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcffc1040e0 0x7fcffc19a500 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.381+0000 7fcffbe00640 1 --2- 172.21.15.45:0/985570412 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcffc103720 0x7fcffc199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:56:35.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.381+0000 7fcffbe00640 1 -- 172.21.15.45:0/985570412 >> 172.21.15.45:0/985570412 conn(0x7fcffc0ff1b0 msgr2=0x7fcffc0ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:56:35.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.381+0000 7fcffbe00640 1 -- 172.21.15.45:0/985570412 shutdown_connections 2024-03-28T17:56:35.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:56:35.381+0000 7fcffbe00640 1 -- 172.21.15.45:0/985570412 wait complete. 2024-03-28T17:56:35.398 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T17:56:35.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:35 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:56:35.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:35 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/1898596562,v1:172.21.15.151:6833/1898596562] up:reconnect 2024-03-28T17:56:35.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:35 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:reconnect} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:35.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:35 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:56:35.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:35 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:56:35.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:35 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/1898596562,v1:172.21.15.151:6833/1898596562] up:reconnect 2024-03-28T17:56:35.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:35 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:reconnect} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:35.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:35 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:56:35.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:35 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:56:35.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:35 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/1898596562,v1:172.21.15.151:6833/1898596562] up:reconnect 2024-03-28T17:56:35.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:35 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:reconnect} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:35.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:35 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:56:35.884 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 248 sec latency 363781.563 ms 2024-03-28T17:56:36.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:36 smithi045 ceph-mon[31773]: pgmap v254: 97 pgs: 97 active+clean; 27 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 234 B/s wr, 7 op/s 2024-03-28T17:56:36.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:36 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/1898596562,v1:172.21.15.151:6833/1898596562] up:rejoin 2024-03-28T17:56:36.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:36 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:rejoin} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:36.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:36 smithi045 ceph-mon[31773]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-03-28T17:56:36.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:36 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:56:36.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:36 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:56:36.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:36 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/1898596562,v1:172.21.15.151:6833/1898596562] up:active 2024-03-28T17:56:36.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:36 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T17:56:36.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:36 smithi045 ceph-mon[31773]: osdmap e112: 12 total, 12 up, 12 in 2024-03-28T17:56:36.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:36 smithi071 ceph-mon[36597]: pgmap v254: 97 pgs: 97 active+clean; 27 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 234 B/s wr, 7 op/s 2024-03-28T17:56:36.712 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:36 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/1898596562,v1:172.21.15.151:6833/1898596562] up:rejoin 2024-03-28T17:56:36.712 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:36 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:rejoin} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:36.712 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:36 smithi071 ceph-mon[36597]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-03-28T17:56:36.712 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:36 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:56:36.712 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:36 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:56:36.712 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:36 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/1898596562,v1:172.21.15.151:6833/1898596562] up:active 2024-03-28T17:56:36.712 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:36 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T17:56:36.712 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:36 smithi071 ceph-mon[36597]: osdmap e112: 12 total, 12 up, 12 in 2024-03-28T17:56:36.809 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-mon[36481]: pgmap v254: 97 pgs: 97 active+clean; 27 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 234 B/s wr, 7 op/s 2024-03-28T17:56:36.810 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/1898596562,v1:172.21.15.151:6833/1898596562] up:rejoin 2024-03-28T17:56:36.810 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:rejoin} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:36.810 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-mon[36481]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-03-28T17:56:36.810 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:56:36.810 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:56:36.810 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/1898596562,v1:172.21.15.151:6833/1898596562] up:active 2024-03-28T17:56:36.810 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T17:56:36.810 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-mon[36481]: osdmap e112: 12 total, 12 up, 12 in 2024-03-28T17:56:36.811 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1ad5d75700 time 2024-03-28T17:56:36.335065+0000 2024-03-28T17:56:36.811 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:56:36.811 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:36.811 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1ae039f4a1] 2024-03-28T17:56:36.811 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1ae039f667] 2024-03-28T17:56:36.811 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x564f5808d223] 2024-03-28T17:56:36.811 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x564f57f075d6] 2024-03-28T17:56:36.811 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x564f57f07da1] 2024-03-28T17:56:36.811 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x564f57f0a2d5] 2024-03-28T17:56:36.812 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x564f57f5a233] 2024-03-28T17:56:36.812 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564f57f5c197] 2024-03-28T17:56:36.812 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564f57e5b11b] 2024-03-28T17:56:36.812 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564f57e5b6bc] 2024-03-28T17:56:36.812 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x564f57e451ff] 2024-03-28T17:56:36.812 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1ae0631488] 2024-03-28T17:56:36.812 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 13: (DispatchQueue::entry()+0x51f) [0x7f1ae062e60f] 2024-03-28T17:56:36.812 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1ae0701e81] 2024-03-28T17:56:36.812 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1adf11c1ca] 2024-03-28T17:56:36.812 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 16: clone() 2024-03-28T17:56:36.812 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 2024-03-28T17:56:36.335+0000 7f1ad5d75700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1ad5d75700 time 2024-03-28T17:56:36.335065+0000 2024-03-28T17:56:36.812 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:56:36.813 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 2024-03-28T17:56:36.813 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:36.813 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1ae039f4a1] 2024-03-28T17:56:36.813 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1ae039f667] 2024-03-28T17:56:36.813 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x564f5808d223] 2024-03-28T17:56:36.813 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x564f57f075d6] 2024-03-28T17:56:36.813 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x564f57f07da1] 2024-03-28T17:56:36.813 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x564f57f0a2d5] 2024-03-28T17:56:36.813 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x564f57f5a233] 2024-03-28T17:56:36.813 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564f57f5c197] 2024-03-28T17:56:36.813 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564f57e5b11b] 2024-03-28T17:56:36.813 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564f57e5b6bc] 2024-03-28T17:56:36.813 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x564f57e451ff] 2024-03-28T17:56:36.813 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1ae0631488] 2024-03-28T17:56:36.814 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 13: (DispatchQueue::entry()+0x51f) [0x7f1ae062e60f] 2024-03-28T17:56:36.814 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1ae0701e81] 2024-03-28T17:56:36.814 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1adf11c1ca] 2024-03-28T17:56:36.814 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 16: clone() 2024-03-28T17:56:36.814 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 2024-03-28T17:56:36.814 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: *** Caught signal (Aborted) ** 2024-03-28T17:56:36.814 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: in thread 7f1ad5d75700 thread_name:ms_dispatch 2024-03-28T17:56:36.814 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:36.814 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1adf126d20] 2024-03-28T17:56:36.815 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 2: gsignal() 2024-03-28T17:56:36.815 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 3: abort() 2024-03-28T17:56:36.815 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1ae039f4fb] 2024-03-28T17:56:36.816 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1ae039f667] 2024-03-28T17:56:36.816 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x564f5808d223] 2024-03-28T17:56:36.816 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x564f57f075d6] 2024-03-28T17:56:36.816 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x564f57f07da1] 2024-03-28T17:56:36.816 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x564f57f0a2d5] 2024-03-28T17:56:36.816 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x564f57f5a233] 2024-03-28T17:56:36.816 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564f57f5c197] 2024-03-28T17:56:36.816 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564f57e5b11b] 2024-03-28T17:56:36.816 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564f57e5b6bc] 2024-03-28T17:56:36.816 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x564f57e451ff] 2024-03-28T17:56:36.816 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1ae0631488] 2024-03-28T17:56:36.816 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 16: (DispatchQueue::entry()+0x51f) [0x7f1ae062e60f] 2024-03-28T17:56:36.816 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1ae0701e81] 2024-03-28T17:56:36.817 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1adf11c1ca] 2024-03-28T17:56:36.817 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 19: clone() 2024-03-28T17:56:36.817 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 2024-03-28T17:56:36.337+0000 7f1ad5d75700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:56:36.817 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: in thread 7f1ad5d75700 thread_name:ms_dispatch 2024-03-28T17:56:36.817 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 2024-03-28T17:56:36.817 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:36.817 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1adf126d20] 2024-03-28T17:56:36.817 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 2: gsignal() 2024-03-28T17:56:36.817 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 3: abort() 2024-03-28T17:56:36.817 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1ae039f4fb] 2024-03-28T17:56:36.817 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1ae039f667] 2024-03-28T17:56:36.817 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x564f5808d223] 2024-03-28T17:56:36.817 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x564f57f075d6] 2024-03-28T17:56:36.818 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x564f57f07da1] 2024-03-28T17:56:36.818 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x564f57f0a2d5] 2024-03-28T17:56:36.818 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x564f57f5a233] 2024-03-28T17:56:36.818 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564f57f5c197] 2024-03-28T17:56:36.818 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564f57e5b11b] 2024-03-28T17:56:36.818 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564f57e5b6bc] 2024-03-28T17:56:36.818 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x564f57e451ff] 2024-03-28T17:56:36.818 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1ae0631488] 2024-03-28T17:56:36.818 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 16: (DispatchQueue::entry()+0x51f) [0x7f1ae062e60f] 2024-03-28T17:56:36.818 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1ae0701e81] 2024-03-28T17:56:36.818 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1adf11c1ca] 2024-03-28T17:56:36.819 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 19: clone() 2024-03-28T17:56:36.819 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:56:36.819 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 2024-03-28T17:56:36.819 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: -1> 2024-03-28T17:56:36.335+0000 7f1ad5d75700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1ad5d75700 time 2024-03-28T17:56:36.335065+0000 2024-03-28T17:56:36.819 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:56:36.819 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 2024-03-28T17:56:36.819 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:36.819 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1ae039f4a1] 2024-03-28T17:56:36.819 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1ae039f667] 2024-03-28T17:56:36.819 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x564f5808d223] 2024-03-28T17:56:36.819 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x564f57f075d6] 2024-03-28T17:56:36.819 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x564f57f07da1] 2024-03-28T17:56:36.819 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x564f57f0a2d5] 2024-03-28T17:56:36.819 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x564f57f5a233] 2024-03-28T17:56:36.820 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564f57f5c197] 2024-03-28T17:56:36.820 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564f57e5b11b] 2024-03-28T17:56:36.820 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564f57e5b6bc] 2024-03-28T17:56:36.820 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x564f57e451ff] 2024-03-28T17:56:36.820 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1ae0631488] 2024-03-28T17:56:36.820 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 13: (DispatchQueue::entry()+0x51f) [0x7f1ae062e60f] 2024-03-28T17:56:36.820 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1ae0701e81] 2024-03-28T17:56:36.820 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1adf11c1ca] 2024-03-28T17:56:36.820 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 16: clone() 2024-03-28T17:56:36.820 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 2024-03-28T17:56:36.820 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 0> 2024-03-28T17:56:36.337+0000 7f1ad5d75700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:56:36.820 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: in thread 7f1ad5d75700 thread_name:ms_dispatch 2024-03-28T17:56:36.820 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 2024-03-28T17:56:36.820 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:36.820 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1adf126d20] 2024-03-28T17:56:36.820 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 2: gsignal() 2024-03-28T17:56:36.821 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 3: abort() 2024-03-28T17:56:36.821 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1ae039f4fb] 2024-03-28T17:56:36.821 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1ae039f667] 2024-03-28T17:56:36.821 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x564f5808d223] 2024-03-28T17:56:36.821 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x564f57f075d6] 2024-03-28T17:56:36.821 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x564f57f07da1] 2024-03-28T17:56:36.821 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x564f57f0a2d5] 2024-03-28T17:56:36.821 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x564f57f5a233] 2024-03-28T17:56:36.821 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564f57f5c197] 2024-03-28T17:56:36.821 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564f57e5b11b] 2024-03-28T17:56:36.821 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564f57e5b6bc] 2024-03-28T17:56:36.821 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x564f57e451ff] 2024-03-28T17:56:36.821 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1ae0631488] 2024-03-28T17:56:36.821 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 16: (DispatchQueue::entry()+0x51f) [0x7f1ae062e60f] 2024-03-28T17:56:36.821 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1ae0701e81] 2024-03-28T17:56:36.822 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1adf11c1ca] 2024-03-28T17:56:36.822 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 19: clone() 2024-03-28T17:56:36.822 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:56:36.822 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 2024-03-28T17:56:36.822 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: -1> 2024-03-28T17:56:36.335+0000 7f1ad5d75700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1ad5d75700 time 2024-03-28T17:56:36.335065+0000 2024-03-28T17:56:36.822 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:56:36.822 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 2024-03-28T17:56:36.822 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:36.822 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1ae039f4a1] 2024-03-28T17:56:36.822 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1ae039f667] 2024-03-28T17:56:36.822 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x564f5808d223] 2024-03-28T17:56:36.822 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x564f57f075d6] 2024-03-28T17:56:36.822 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x564f57f07da1] 2024-03-28T17:56:36.822 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x564f57f0a2d5] 2024-03-28T17:56:36.823 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x564f57f5a233] 2024-03-28T17:56:36.823 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564f57f5c197] 2024-03-28T17:56:36.823 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564f57e5b11b] 2024-03-28T17:56:36.823 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564f57e5b6bc] 2024-03-28T17:56:36.823 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x564f57e451ff] 2024-03-28T17:56:36.823 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1ae0631488] 2024-03-28T17:56:36.823 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 13: (DispatchQueue::entry()+0x51f) [0x7f1ae062e60f] 2024-03-28T17:56:36.823 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1ae0701e81] 2024-03-28T17:56:36.823 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1adf11c1ca] 2024-03-28T17:56:36.823 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 16: clone() 2024-03-28T17:56:36.823 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 2024-03-28T17:56:36.823 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 0> 2024-03-28T17:56:36.337+0000 7f1ad5d75700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:56:36.823 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: in thread 7f1ad5d75700 thread_name:ms_dispatch 2024-03-28T17:56:36.823 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 2024-03-28T17:56:36.823 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:36.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1adf126d20] 2024-03-28T17:56:36.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 2: gsignal() 2024-03-28T17:56:36.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 3: abort() 2024-03-28T17:56:36.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1ae039f4fb] 2024-03-28T17:56:36.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1ae039f667] 2024-03-28T17:56:36.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x564f5808d223] 2024-03-28T17:56:36.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x564f57f075d6] 2024-03-28T17:56:36.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x564f57f07da1] 2024-03-28T17:56:36.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x564f57f0a2d5] 2024-03-28T17:56:36.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x564f57f5a233] 2024-03-28T17:56:36.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x564f57f5c197] 2024-03-28T17:56:36.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x564f57e5b11b] 2024-03-28T17:56:36.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564f57e5b6bc] 2024-03-28T17:56:36.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x564f57e451ff] 2024-03-28T17:56:36.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1ae0631488] 2024-03-28T17:56:36.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 16: (DispatchQueue::entry()+0x51f) [0x7f1ae062e60f] 2024-03-28T17:56:36.825 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1ae0701e81] 2024-03-28T17:56:36.825 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1adf11c1ca] 2024-03-28T17:56:36.825 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 19: clone() 2024-03-28T17:56:36.825 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:56:36.825 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[66276]: 2024-03-28T17:56:36.825 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:36 smithi151 podman[73113]: 2024-03-28 17:56:36.391698133 +0000 UTC m=+0.020768313 container died 3ac9123a664969c9ebb0f696fb14b9252ce0f4a7de41fbfcd3cc78ef0548ad47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-28T17:56:36.885 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 249 sec latency 364781.723 ms 2024-03-28T17:56:37.573 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:37 smithi151 podman[73113]: 2024-03-28 17:56:37.235312014 +0000 UTC m=+0.864382177 container remove 3ac9123a664969c9ebb0f696fb14b9252ce0f4a7de41fbfcd3cc78ef0548ad47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-28T17:56:37.573 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:37 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:56:37.573 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:37 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-28T17:56:37.885 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 250 sec latency 365781.855 ms 2024-03-28T17:56:38.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:38 smithi071 ceph-mon[36597]: pgmap v256: 97 pgs: 97 active+clean; 27 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T17:56:38.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:38 smithi151 ceph-mon[36481]: pgmap v256: 97 pgs: 97 active+clean; 27 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T17:56:38.885 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 251 sec latency 366782.013 ms 2024-03-28T17:56:38.897 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:38 smithi045 ceph-mon[31773]: pgmap v256: 97 pgs: 97 active+clean; 27 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T17:56:39.885 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 252 sec latency 367782.176 ms 2024-03-28T17:56:40.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:40 smithi071 ceph-mon[36597]: pgmap v257: 97 pgs: 97 active+clean; 27 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T17:56:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:40 smithi151 ceph-mon[36481]: pgmap v257: 97 pgs: 97 active+clean; 27 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T17:56:40.881 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:40 smithi045 ceph-mon[31773]: pgmap v257: 97 pgs: 97 active+clean; 27 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T17:56:40.885 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 253 sec latency 368782.313 ms 2024-03-28T17:56:41.885 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 254 sec latency 369782.471 ms 2024-03-28T17:56:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:42 smithi071 ceph-mon[36597]: pgmap v258: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 341 B/s wr, 12 op/s 2024-03-28T17:56:42.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:42 smithi151 ceph-mon[36481]: pgmap v258: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 341 B/s wr, 12 op/s 2024-03-28T17:56:42.885 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 255 sec latency 370782.561 ms 2024-03-28T17:56:42.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:42 smithi045 ceph-mon[31773]: pgmap v258: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 341 B/s wr, 12 op/s 2024-03-28T17:56:43.886 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 256 sec latency 371782.723 ms 2024-03-28T17:56:44.718 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:44 smithi045 ceph-mon[31773]: pgmap v259: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T17:56:44.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:44 smithi071 ceph-mon[36597]: pgmap v259: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T17:56:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:44 smithi151 ceph-mon[36481]: pgmap v259: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T17:56:44.886 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 257 sec latency 372782.933 ms 2024-03-28T17:56:45.886 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 258 sec latency 373783.069 ms 2024-03-28T17:56:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:46 smithi071 ceph-mon[36597]: pgmap v260: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:56:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:46 smithi151 ceph-mon[36481]: pgmap v260: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:56:46.886 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 259 sec latency 374783.206 ms 2024-03-28T17:56:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:46 smithi045 ceph-mon[31773]: pgmap v260: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:56:47.826 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:47 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 1. 2024-03-28T17:56:47.826 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:47 smithi151 systemd[1]: Stopped Ceph mds.l for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:56:47.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:47 smithi151 systemd[1]: Starting Ceph mds.l for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:56:47.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:47 smithi151 podman[73214]: 2024-03-28 17:56:47.709770658 +0000 UTC m=+0.017522982 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:56:47.886 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 260 sec latency 375783.352 ms 2024-03-28T17:56:48.077 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:47 smithi151 podman[73214]: 2024-03-28 17:56:47.827447297 +0000 UTC m=+0.135199659 container create 0fb90d16f13382d09293905a5f31af16d7380f6542019750ec801598db142f59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-28T17:56:48.077 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:48 smithi151 podman[73214]: 2024-03-28 17:56:48.075636665 +0000 UTC m=+0.383388989 container init 0fb90d16f13382d09293905a5f31af16d7380f6542019750ec801598db142f59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T17:56:48.078 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:48 smithi151 podman[73214]: 2024-03-28 17:56:48.078310703 +0000 UTC m=+0.386063013 container start 0fb90d16f13382d09293905a5f31af16d7380f6542019750ec801598db142f59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-28T17:56:48.463 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:48 smithi151 ceph-mds[73234]: -- 172.21.15.151:0/4102631274 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563ee9ef1a40 con 0x563ee9ec4800 2024-03-28T17:56:48.463 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: starting mds.l at 2024-03-28T17:56:48.463 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:48 smithi151 bash[73214]: 0fb90d16f13382d09293905a5f31af16d7380f6542019750ec801598db142f59 2024-03-28T17:56:48.463 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:56:48 smithi151 systemd[1]: Started Ceph mds.l for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:56:48.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:48 smithi071 ceph-mon[36597]: pgmap v261: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 190 B/s wr, 9 op/s 2024-03-28T17:56:48.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:48 smithi071 ceph-mon[36597]: daemon mds.l restarted 2024-03-28T17:56:48.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:48 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:56:48.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:48 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T17:56:48.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:48 smithi071 ceph-mon[36597]: osdmap e113: 12 total, 12 up, 12 in 2024-03-28T17:56:48.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:48 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/797518908,v1:172.21.15.151:6833/797518908] up:boot 2024-03-28T17:56:48.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:48 smithi071 ceph-mon[36597]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-03-28T17:56:48.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:48 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-28T17:56:48.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:48 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:replay} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:48 smithi151 ceph-mon[36481]: pgmap v261: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 190 B/s wr, 9 op/s 2024-03-28T17:56:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:48 smithi151 ceph-mon[36481]: daemon mds.l restarted 2024-03-28T17:56:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:48 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:56:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:48 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T17:56:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:48 smithi151 ceph-mon[36481]: osdmap e113: 12 total, 12 up, 12 in 2024-03-28T17:56:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:48 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/797518908,v1:172.21.15.151:6833/797518908] up:boot 2024-03-28T17:56:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:48 smithi151 ceph-mon[36481]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-03-28T17:56:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:48 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-28T17:56:48.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:48 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:replay} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:48.886 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 261 sec latency 376783.487 ms 2024-03-28T17:56:48.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:48 smithi045 ceph-mon[31773]: pgmap v261: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 190 B/s wr, 9 op/s 2024-03-28T17:56:48.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:48 smithi045 ceph-mon[31773]: daemon mds.l restarted 2024-03-28T17:56:48.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:48 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:56:48.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:48 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T17:56:48.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:48 smithi045 ceph-mon[31773]: osdmap e113: 12 total, 12 up, 12 in 2024-03-28T17:56:48.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:48 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/797518908,v1:172.21.15.151:6833/797518908] up:boot 2024-03-28T17:56:48.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:48 smithi045 ceph-mon[31773]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-03-28T17:56:48.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:48 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-28T17:56:48.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:48 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:replay} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:49.886 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 262 sec latency 377783.627 ms 2024-03-28T17:56:50.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:50 smithi045 ceph-mon[31773]: pgmap v263: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:56:50.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:50 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/1081652393,v1:172.21.15.71:6833/1081652393] up:resolve 2024-03-28T17:56:50.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:50 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:resolve} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:50.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:50 smithi071 ceph-mon[36597]: pgmap v263: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:56:50.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:50 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/1081652393,v1:172.21.15.71:6833/1081652393] up:resolve 2024-03-28T17:56:50.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:50 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:resolve} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:50.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:50 smithi151 ceph-mon[36481]: pgmap v263: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:56:50.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:50 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/1081652393,v1:172.21.15.71:6833/1081652393] up:resolve 2024-03-28T17:56:50.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:50 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:resolve} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:50.886 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 263 sec latency 378783.779 ms 2024-03-28T17:56:51.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:51 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:56:51.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:51 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/1081652393,v1:172.21.15.71:6833/1081652393] up:reconnect 2024-03-28T17:56:51.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:51 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:reconnect} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:51.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:51 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00799987 2024-03-28T17:56:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:56:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/1081652393,v1:172.21.15.71:6833/1081652393] up:reconnect 2024-03-28T17:56:51.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:reconnect} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:51.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00799987 2024-03-28T17:56:51.561 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f802f44b700 time 2024-03-28T17:56:51.336878+0000 2024-03-28T17:56:51.561 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:56:51.561 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:51.561 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8039a754a1] 2024-03-28T17:56:51.561 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8039a75667] 2024-03-28T17:56:51.561 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f8f1045223] 2024-03-28T17:56:51.561 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f8f0ebf5d6] 2024-03-28T17:56:51.562 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f8f0ebfda1] 2024-03-28T17:56:51.562 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f8f0ec22d5] 2024-03-28T17:56:51.562 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f8f0f12233] 2024-03-28T17:56:51.562 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f8f0f14197] 2024-03-28T17:56:51.562 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f8f0e1311b] 2024-03-28T17:56:51.562 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f8f0e136bc] 2024-03-28T17:56:51.562 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f8f0dfd1ff] 2024-03-28T17:56:51.562 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8039d07488] 2024-03-28T17:56:51.562 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 13: (DispatchQueue::entry()+0x51f) [0x7f8039d0460f] 2024-03-28T17:56:51.562 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8039dd7e81] 2024-03-28T17:56:51.562 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f80387f21ca] 2024-03-28T17:56:51.562 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 16: clone() 2024-03-28T17:56:51.563 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 2024-03-28T17:56:51.337+0000 7f802f44b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f802f44b700 time 2024-03-28T17:56:51.336878+0000 2024-03-28T17:56:51.563 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:56:51.563 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 2024-03-28T17:56:51.563 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:51.563 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8039a754a1] 2024-03-28T17:56:51.563 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8039a75667] 2024-03-28T17:56:51.563 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f8f1045223] 2024-03-28T17:56:51.563 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f8f0ebf5d6] 2024-03-28T17:56:51.563 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f8f0ebfda1] 2024-03-28T17:56:51.564 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f8f0ec22d5] 2024-03-28T17:56:51.564 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f8f0f12233] 2024-03-28T17:56:51.564 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f8f0f14197] 2024-03-28T17:56:51.564 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f8f0e1311b] 2024-03-28T17:56:51.564 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f8f0e136bc] 2024-03-28T17:56:51.564 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f8f0dfd1ff] 2024-03-28T17:56:51.564 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8039d07488] 2024-03-28T17:56:51.564 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 13: (DispatchQueue::entry()+0x51f) [0x7f8039d0460f] 2024-03-28T17:56:51.564 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8039dd7e81] 2024-03-28T17:56:51.564 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f80387f21ca] 2024-03-28T17:56:51.564 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 16: clone() 2024-03-28T17:56:51.564 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 2024-03-28T17:56:51.564 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: *** Caught signal (Aborted) ** 2024-03-28T17:56:51.565 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: in thread 7f802f44b700 thread_name:ms_dispatch 2024-03-28T17:56:51.565 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:51.565 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f80387fcd20] 2024-03-28T17:56:51.565 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 2: gsignal() 2024-03-28T17:56:51.565 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 3: abort() 2024-03-28T17:56:51.565 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8039a754fb] 2024-03-28T17:56:51.565 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8039a75667] 2024-03-28T17:56:51.565 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f8f1045223] 2024-03-28T17:56:51.565 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f8f0ebf5d6] 2024-03-28T17:56:51.565 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f8f0ebfda1] 2024-03-28T17:56:51.565 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f8f0ec22d5] 2024-03-28T17:56:51.565 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f8f0f12233] 2024-03-28T17:56:51.565 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f8f0f14197] 2024-03-28T17:56:51.565 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f8f0e1311b] 2024-03-28T17:56:51.565 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f8f0e136bc] 2024-03-28T17:56:51.565 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f8f0dfd1ff] 2024-03-28T17:56:51.566 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8039d07488] 2024-03-28T17:56:51.566 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 16: (DispatchQueue::entry()+0x51f) [0x7f8039d0460f] 2024-03-28T17:56:51.566 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8039dd7e81] 2024-03-28T17:56:51.566 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f80387f21ca] 2024-03-28T17:56:51.566 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 19: clone() 2024-03-28T17:56:51.566 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 2024-03-28T17:56:51.340+0000 7f802f44b700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:56:51.566 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: in thread 7f802f44b700 thread_name:ms_dispatch 2024-03-28T17:56:51.566 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 2024-03-28T17:56:51.566 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:51.566 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f80387fcd20] 2024-03-28T17:56:51.566 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 2: gsignal() 2024-03-28T17:56:51.566 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 3: abort() 2024-03-28T17:56:51.566 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8039a754fb] 2024-03-28T17:56:51.566 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8039a75667] 2024-03-28T17:56:51.567 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f8f1045223] 2024-03-28T17:56:51.567 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f8f0ebf5d6] 2024-03-28T17:56:51.567 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f8f0ebfda1] 2024-03-28T17:56:51.568 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f8f0ec22d5] 2024-03-28T17:56:51.568 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f8f0f12233] 2024-03-28T17:56:51.568 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f8f0f14197] 2024-03-28T17:56:51.568 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f8f0e1311b] 2024-03-28T17:56:51.568 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f8f0e136bc] 2024-03-28T17:56:51.568 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f8f0dfd1ff] 2024-03-28T17:56:51.568 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8039d07488] 2024-03-28T17:56:51.568 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 16: (DispatchQueue::entry()+0x51f) [0x7f8039d0460f] 2024-03-28T17:56:51.568 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8039dd7e81] 2024-03-28T17:56:51.568 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f80387f21ca] 2024-03-28T17:56:51.568 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 19: clone() 2024-03-28T17:56:51.568 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:56:51.568 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 2024-03-28T17:56:51.568 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: -1> 2024-03-28T17:56:51.337+0000 7f802f44b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f802f44b700 time 2024-03-28T17:56:51.336878+0000 2024-03-28T17:56:51.568 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:56:51.568 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 2024-03-28T17:56:51.569 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:51.569 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8039a754a1] 2024-03-28T17:56:51.569 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8039a75667] 2024-03-28T17:56:51.569 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f8f1045223] 2024-03-28T17:56:51.569 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f8f0ebf5d6] 2024-03-28T17:56:51.569 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f8f0ebfda1] 2024-03-28T17:56:51.569 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f8f0ec22d5] 2024-03-28T17:56:51.569 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f8f0f12233] 2024-03-28T17:56:51.569 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f8f0f14197] 2024-03-28T17:56:51.569 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f8f0e1311b] 2024-03-28T17:56:51.569 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f8f0e136bc] 2024-03-28T17:56:51.569 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f8f0dfd1ff] 2024-03-28T17:56:51.569 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8039d07488] 2024-03-28T17:56:51.569 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 13: (DispatchQueue::entry()+0x51f) [0x7f8039d0460f] 2024-03-28T17:56:51.569 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8039dd7e81] 2024-03-28T17:56:51.569 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f80387f21ca] 2024-03-28T17:56:51.570 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 16: clone() 2024-03-28T17:56:51.570 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 2024-03-28T17:56:51.570 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 0> 2024-03-28T17:56:51.340+0000 7f802f44b700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:56:51.570 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: in thread 7f802f44b700 thread_name:ms_dispatch 2024-03-28T17:56:51.570 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 2024-03-28T17:56:51.570 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:51.570 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f80387fcd20] 2024-03-28T17:56:51.570 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 2: gsignal() 2024-03-28T17:56:51.570 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 3: abort() 2024-03-28T17:56:51.570 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8039a754fb] 2024-03-28T17:56:51.570 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8039a75667] 2024-03-28T17:56:51.570 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f8f1045223] 2024-03-28T17:56:51.570 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f8f0ebf5d6] 2024-03-28T17:56:51.571 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f8f0ebfda1] 2024-03-28T17:56:51.571 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f8f0ec22d5] 2024-03-28T17:56:51.571 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f8f0f12233] 2024-03-28T17:56:51.571 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f8f0f14197] 2024-03-28T17:56:51.571 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f8f0e1311b] 2024-03-28T17:56:51.571 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f8f0e136bc] 2024-03-28T17:56:51.571 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f8f0dfd1ff] 2024-03-28T17:56:51.571 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8039d07488] 2024-03-28T17:56:51.571 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 16: (DispatchQueue::entry()+0x51f) [0x7f8039d0460f] 2024-03-28T17:56:51.571 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8039dd7e81] 2024-03-28T17:56:51.571 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f80387f21ca] 2024-03-28T17:56:51.571 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 19: clone() 2024-03-28T17:56:51.571 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:56:51.571 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 2024-03-28T17:56:51.572 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: -1> 2024-03-28T17:56:51.337+0000 7f802f44b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f802f44b700 time 2024-03-28T17:56:51.336878+0000 2024-03-28T17:56:51.572 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:56:51.572 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 2024-03-28T17:56:51.572 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:51.572 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8039a754a1] 2024-03-28T17:56:51.572 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8039a75667] 2024-03-28T17:56:51.572 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f8f1045223] 2024-03-28T17:56:51.572 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f8f0ebf5d6] 2024-03-28T17:56:51.572 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f8f0ebfda1] 2024-03-28T17:56:51.572 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f8f0ec22d5] 2024-03-28T17:56:51.572 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f8f0f12233] 2024-03-28T17:56:51.572 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f8f0f14197] 2024-03-28T17:56:51.572 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f8f0e1311b] 2024-03-28T17:56:51.572 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f8f0e136bc] 2024-03-28T17:56:51.572 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f8f0dfd1ff] 2024-03-28T17:56:51.573 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8039d07488] 2024-03-28T17:56:51.573 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 13: (DispatchQueue::entry()+0x51f) [0x7f8039d0460f] 2024-03-28T17:56:51.573 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8039dd7e81] 2024-03-28T17:56:51.573 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f80387f21ca] 2024-03-28T17:56:51.573 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 16: clone() 2024-03-28T17:56:51.573 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 2024-03-28T17:56:51.573 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 0> 2024-03-28T17:56:51.340+0000 7f802f44b700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:56:51.573 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: in thread 7f802f44b700 thread_name:ms_dispatch 2024-03-28T17:56:51.573 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 2024-03-28T17:56:51.573 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:56:51.573 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f80387fcd20] 2024-03-28T17:56:51.573 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 2: gsignal() 2024-03-28T17:56:51.573 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 3: abort() 2024-03-28T17:56:51.573 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8039a754fb] 2024-03-28T17:56:51.573 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8039a75667] 2024-03-28T17:56:51.573 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f8f1045223] 2024-03-28T17:56:51.574 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f8f0ebf5d6] 2024-03-28T17:56:51.574 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f8f0ebfda1] 2024-03-28T17:56:51.574 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f8f0ec22d5] 2024-03-28T17:56:51.574 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f8f0f12233] 2024-03-28T17:56:51.574 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f8f0f14197] 2024-03-28T17:56:51.574 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f8f0e1311b] 2024-03-28T17:56:51.574 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f8f0e136bc] 2024-03-28T17:56:51.574 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f8f0dfd1ff] 2024-03-28T17:56:51.574 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8039d07488] 2024-03-28T17:56:51.574 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 16: (DispatchQueue::entry()+0x51f) [0x7f8039d0460f] 2024-03-28T17:56:51.574 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8039dd7e81] 2024-03-28T17:56:51.574 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f80387f21ca] 2024-03-28T17:56:51.574 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 19: clone() 2024-03-28T17:56:51.574 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:56:51.575 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k[72076]: 2024-03-28T17:56:51.575 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:51 smithi071 podman[72128]: 2024-03-28 17:56:51.391991471 +0000 UTC m=+0.019190086 container died 96111c0ee808121987f06f7ca9bdb8a341fd7977ee1fcb495b792f73dd323c8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-28T17:56:51.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:51 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:56:51.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:51 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/1081652393,v1:172.21.15.71:6833/1081652393] up:reconnect 2024-03-28T17:56:51.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:51 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:reconnect} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:51.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:51 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00799987 2024-03-28T17:56:51.887 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 264 sec latency 379783.937 ms 2024-03-28T17:56:52.306 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:52 smithi071 podman[72128]: 2024-03-28 17:56:52.048730562 +0000 UTC m=+0.675929162 container remove 96111c0ee808121987f06f7ca9bdb8a341fd7977ee1fcb495b792f73dd323c8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-k, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-28T17:56:52.306 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:52 smithi071 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:56:52.307 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:52 smithi071 ceph-mon[36597]: pgmap v264: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T17:56:52.307 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:52 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/1081652393,v1:172.21.15.71:6833/1081652393] up:rejoin 2024-03-28T17:56:52.307 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:52 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:rejoin} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:52.307 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:52 smithi071 ceph-mon[36597]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-03-28T17:56:52.307 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:52 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:56:52.307 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:52 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:56:52.307 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:52 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6832/1081652393,v1:172.21.15.71:6833/1081652393] up:active 2024-03-28T17:56:52.307 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:52 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T17:56:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:52 smithi045 ceph-mon[31773]: pgmap v264: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T17:56:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:52 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/1081652393,v1:172.21.15.71:6833/1081652393] up:rejoin 2024-03-28T17:56:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:52 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:rejoin} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:52 smithi045 ceph-mon[31773]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-03-28T17:56:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:52 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:56:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:52 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:56:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:52 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6832/1081652393,v1:172.21.15.71:6833/1081652393] up:active 2024-03-28T17:56:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:52 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T17:56:52.560 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:56:52 smithi071 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.k.service: Failed with result 'exit-code'. 2024-03-28T17:56:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:52 smithi151 ceph-mon[36481]: pgmap v264: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T17:56:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:52 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/1081652393,v1:172.21.15.71:6833/1081652393] up:rejoin 2024-03-28T17:56:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:52 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=k=up:rejoin} 2 up:standby-replay 4 up:standby 2024-03-28T17:56:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:52 smithi151 ceph-mon[36481]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-03-28T17:56:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:52 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:56:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:52 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:56:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:52 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6832/1081652393,v1:172.21.15.71:6833/1081652393] up:active 2024-03-28T17:56:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:52 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=k=up:active} 3 up:standby-replay 3 up:standby 2024-03-28T17:56:52.887 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 265 sec latency 380784.065 ms 2024-03-28T17:56:53.887 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 266 sec latency 381784.240 ms 2024-03-28T17:56:54.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:54 smithi045 ceph-mon[31773]: pgmap v265: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T17:56:54.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:54 smithi071 ceph-mon[36597]: pgmap v265: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T17:56:54.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:54 smithi151 ceph-mon[36481]: pgmap v265: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T17:56:54.887 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 267 sec latency 382784.374 ms 2024-03-28T17:56:55.887 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 268 sec latency 383784.514 ms 2024-03-28T17:56:56.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:56 smithi045 ceph-mon[31773]: pgmap v266: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:56:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:56 smithi071 ceph-mon[36597]: pgmap v266: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:56:56.570 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:56 smithi151 ceph-mon[36481]: pgmap v266: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:56:56.887 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 269 sec latency 384784.644 ms 2024-03-28T17:56:57.888 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 270 sec latency 385784.801 ms 2024-03-28T17:56:58.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:56:58 smithi045 ceph-mon[31773]: pgmap v267: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:56:58.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:56:58 smithi071 ceph-mon[36597]: pgmap v267: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:56:58.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:56:58 smithi151 ceph-mon[36481]: pgmap v267: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:56:58.888 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 271 sec latency 386784.959 ms 2024-03-28T17:56:59.888 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 272 sec latency 387785.056 ms 2024-03-28T17:57:00.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:00 smithi045 ceph-mon[31773]: pgmap v268: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 186 B/s wr, 9 op/s 2024-03-28T17:57:00.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:00 smithi071 ceph-mon[36597]: pgmap v268: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 186 B/s wr, 9 op/s 2024-03-28T17:57:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:00 smithi151 ceph-mon[36481]: pgmap v268: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 186 B/s wr, 9 op/s 2024-03-28T17:57:00.888 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 273 sec latency 388785.159 ms 2024-03-28T17:57:01.888 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 274 sec latency 389785.282 ms 2024-03-28T17:57:02.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:02 smithi045 ceph-mon[31773]: pgmap v269: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:57:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:02 smithi071 ceph-mon[36597]: pgmap v269: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:57:02.811 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:57:02 smithi071 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.k.service: Scheduled restart job, restart counter is at 6. 2024-03-28T17:57:02.811 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:57:02 smithi071 systemd[1]: Stopped Ceph mds.k for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:57:02.811 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:57:02 smithi071 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.k.service: Start request repeated too quickly. 2024-03-28T17:57:02.811 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:57:02 smithi071 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.k.service: Failed with result 'exit-code'. 2024-03-28T17:57:02.811 INFO:journalctl@ceph.mds.k.smithi071.stdout:Mar 28 17:57:02 smithi071 systemd[1]: Failed to start Ceph mds.k for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:57:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:02 smithi151 ceph-mon[36481]: pgmap v269: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-28T17:57:02.888 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 275 sec latency 390785.375 ms 2024-03-28T17:57:03.888 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 276 sec latency 391785.529 ms 2024-03-28T17:57:04.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:04 smithi045 ceph-mon[31773]: pgmap v270: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:57:04.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:04 smithi071 ceph-mon[36597]: pgmap v270: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:57:04.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:04 smithi151 ceph-mon[36481]: pgmap v270: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T17:57:04.888 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 277 sec latency 392785.677 ms 2024-03-28T17:57:05.400 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:57:05.400 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:57:05.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.634+0000 7fd6f5c00640 1 -- 172.21.15.45:0/1331113900 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6f0108860 msgr2=0x7fd6f010c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:57:05.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.634+0000 7fd6f5c00640 1 --2- 172.21.15.45:0/1331113900 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6f0108860 0x7fd6f010c8a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009900 tx=0x7fd6e002bc30 comp rx=0 tx=0).stop 2024-03-28T17:57:05.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.635+0000 7fd6f5c00640 1 -- 172.21.15.45:0/1331113900 shutdown_connections 2024-03-28T17:57:05.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.635+0000 7fd6f5c00640 1 --2- 172.21.15.45:0/1331113900 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6f010cf80 0x7fd6f0114b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.635+0000 7fd6f5c00640 1 --2- 172.21.15.45:0/1331113900 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6f0108860 0x7fd6f010c8a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.635+0000 7fd6f5c00640 1 --2- 172.21.15.45:0/1331113900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6f0107eb0 0x7fd6f0108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.635+0000 7fd6f5c00640 1 -- 172.21.15.45:0/1331113900 >> 172.21.15.45:0/1331113900 conn(0x7fd6f0075be0 msgr2=0x7fd6f0078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:57:05.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.635+0000 7fd6f5c00640 1 -- 172.21.15.45:0/1331113900 shutdown_connections 2024-03-28T17:57:05.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.635+0000 7fd6f5c00640 1 -- 172.21.15.45:0/1331113900 wait complete. 2024-03-28T17:57:05.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.636+0000 7fd6f5c00640 1 Processor -- start 2024-03-28T17:57:05.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.636+0000 7fd6f5c00640 1 -- start start 2024-03-28T17:57:05.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.636+0000 7fd6f5c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6f0107eb0 0x7fd6f019e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:05.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.636+0000 7fd6f5c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6f010cf80 0x7fd6f019eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:05.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.636+0000 7fd6f5c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6f01a2e60 0x7fd6f01a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:05.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.636+0000 7fd6f5c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f0118040 con 0x7fd6f0107eb0 2024-03-28T17:57:05.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.636+0000 7fd6f5c00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f0117ec0 con 0x7fd6f010cf80 2024-03-28T17:57:05.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.636+0000 7fd6f5c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f01181c0 con 0x7fd6f01a2e60 2024-03-28T17:57:05.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.637+0000 7fd6eea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6f0107eb0 0x7fd6f019e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:05.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.637+0000 7fd6eea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6f0107eb0 0x7fd6f019e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41682/0 (socket says 172.21.15.45:41682) 2024-03-28T17:57:05.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.637+0000 7fd6ee000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6f010cf80 0x7fd6f019eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:05.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.637+0000 7fd6eea00640 1 -- 172.21.15.45:0/1571386848 learned_addr learned my addr 172.21.15.45:0/1571386848 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:57:05.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.637+0000 7fd6ef400640 1 --2- 172.21.15.45:0/1571386848 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6f01a2e60 0x7fd6f01a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:05.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.637+0000 7fd6ee000640 1 -- 172.21.15.45:0/1571386848 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6f01a2e60 msgr2=0x7fd6f01a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:57:05.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.637+0000 7fd6ee000640 1 --2- 172.21.15.45:0/1571386848 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6f01a2e60 0x7fd6f01a3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.637+0000 7fd6ee000640 1 -- 172.21.15.45:0/1571386848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6f0107eb0 msgr2=0x7fd6f019e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:57:05.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.637+0000 7fd6ee000640 1 --2- 172.21.15.45:0/1571386848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6f0107eb0 0x7fd6f019e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.637+0000 7fd6ee000640 1 -- 172.21.15.45:0/1571386848 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f01a3950 con 0x7fd6f010cf80 2024-03-28T17:57:05.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.637+0000 7fd6eea00640 1 --2- 172.21.15.45:0/1571386848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6f0107eb0 0x7fd6f019e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:57:05.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.637+0000 7fd6ee000640 1 --2- 172.21.15.45:0/1571386848 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6f010cf80 0x7fd6f019eae0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0036040 tx=0x7fd6e0004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:57:05.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.638+0000 7fd6e7400640 1 -- 172.21.15.45:0/1571386848 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e0043020 con 0x7fd6f010cf80 2024-03-28T17:57:05.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.638+0000 7fd6e7400640 1 -- 172.21.15.45:0/1571386848 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6e00054e0 con 0x7fd6f010cf80 2024-03-28T17:57:05.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.638+0000 7fd6f5c00640 1 -- 172.21.15.45:0/1571386848 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f01a3be0 con 0x7fd6f010cf80 2024-03-28T17:57:05.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.638+0000 7fd6e7400640 1 -- 172.21.15.45:0/1571386848 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e0003b80 con 0x7fd6f010cf80 2024-03-28T17:57:05.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.638+0000 7fd6f5c00640 1 -- 172.21.15.45:0/1571386848 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f01ab3a0 con 0x7fd6f010cf80 2024-03-28T17:57:05.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.641+0000 7fd6e7400640 1 -- 172.21.15.45:0/1571386848 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fd6e0046020 con 0x7fd6f010cf80 2024-03-28T17:57:05.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.641+0000 7fd6f8990740 1 Processor -- start 2024-03-28T17:57:05.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.641+0000 7fd6f8990740 1 -- start start 2024-03-28T17:57:05.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.642+0000 7fd6f8990740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c42061c00 0x562c42060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:05.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.642+0000 7fd6f8990740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562c42060f20 0x562c420613a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:05.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.642+0000 7fd6f8990740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c4206a290 0x562c4206c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:05.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.642+0000 7fd6f8990740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c41fc5250 con 0x562c42061c00 2024-03-28T17:57:05.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.642+0000 7fd6f8990740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c41f6da30 con 0x562c4206a290 2024-03-28T17:57:05.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.642+0000 7fd6f8990740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c41fc12f0 con 0x562c42060f20 2024-03-28T17:57:05.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.642+0000 7fd6ef400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c42061c00 0x562c42060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:05.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.642+0000 7fd6ee000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c4206a290 0x562c4206c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:05.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.642+0000 7fd6ee000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c4206a290 0x562c4206c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:33222/0 (socket says 172.21.15.45:33222) 2024-03-28T17:57:05.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.642+0000 7fd6ef400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c42061c00 0x562c42060950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41690/0 (socket says 172.21.15.45:41690) 2024-03-28T17:57:05.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.642+0000 7fd6ee000640 1 -- 172.21.15.45:0/2880837538 learned_addr learned my addr 172.21.15.45:0/2880837538 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:57:05.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.642+0000 7fd6eea00640 1 --2- 172.21.15.45:0/2880837538 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562c42060f20 0x562c420613a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:05.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.643+0000 7fd6ee000640 1 -- 172.21.15.45:0/2880837538 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562c42060f20 msgr2=0x562c420613a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:57:05.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.643+0000 7fd6ee000640 1 --2- 172.21.15.45:0/2880837538 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562c42060f20 0x562c420613a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.643+0000 7fd6ee000640 1 -- 172.21.15.45:0/2880837538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c42061c00 msgr2=0x562c42060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:57:05.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.643+0000 7fd6ee000640 1 --2- 172.21.15.45:0/2880837538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c42061c00 0x562c42060950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.643+0000 7fd6ee000640 1 -- 172.21.15.45:0/2880837538 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562c41f58690 con 0x562c4206a290 2024-03-28T17:57:05.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.643+0000 7fd6ef400640 1 --2- 172.21.15.45:0/2880837538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c42061c00 0x562c42060950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:57:05.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.643+0000 7fd6e7400640 1 --2- 172.21.15.45:0/1571386848 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd6d40a2970 0x7fd6d40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:05.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.643+0000 7fd6ee000640 1 --2- 172.21.15.45:0/2880837538 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c4206a290 0x562c4206c680 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd6e00ee4d0 tx=0x7fd6e00f75b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=994573fc58702540 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:57:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.643+0000 7fd6e7400640 1 -- 172.21.15.45:0/1571386848 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 10490+0+0 (secure 0 0 0) 0x7fd6e00f1540 con 0x7fd6f010cf80 2024-03-28T17:57:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.643+0000 7fd6ef400640 1 --2- 172.21.15.45:0/1571386848 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd6d40a2970 0x7fd6d40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.643+0000 7fd6dfe00640 1 -- 172.21.15.45:0/2880837538 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e00b9740 con 0x562c4206a290 2024-03-28T17:57:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.643+0000 7fd6dfe00640 1 -- 172.21.15.45:0/2880837538 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6e00fe030 con 0x562c4206a290 2024-03-28T17:57:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.643+0000 7fd6dfe00640 1 -- 172.21.15.45:0/2880837538 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e00bce00 con 0x562c4206a290 2024-03-28T17:57:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.644+0000 7fd6f8990740 1 -- 172.21.15.45:0/2880837538 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c4206a290 msgr2=0x562c4206c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:57:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.644+0000 7fd6ef400640 1 --2- 172.21.15.45:0/1571386848 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd6d40a2970 0x7fd6d40a4e30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8005fd0 tx=0x7fd6e8009600 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:57:05.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.644+0000 7fd6f8990740 1 --2- 172.21.15.45:0/2880837538 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c4206a290 0x562c4206c680 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd6e00ee4d0 tx=0x7fd6e00f75b0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.644+0000 7fd6f8990740 1 -- 172.21.15.45:0/2880837538 shutdown_connections 2024-03-28T17:57:05.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.644+0000 7fd6f8990740 1 --2- 172.21.15.45:0/2880837538 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c4206a290 0x562c4206c680 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.644+0000 7fd6f8990740 1 --2- 172.21.15.45:0/2880837538 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562c42060f20 0x562c420613a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.644+0000 7fd6f8990740 1 --2- 172.21.15.45:0/2880837538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c42061c00 0x562c42060950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.644+0000 7fd6f8990740 1 -- 172.21.15.45:0/2880837538 >> 172.21.15.45:0/2880837538 conn(0x562c42028200 msgr2=0x562c4205b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:57:05.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.644+0000 7fd6f8990740 1 -- 172.21.15.45:0/2880837538 shutdown_connections 2024-03-28T17:57:05.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.644+0000 7fd6f8990740 1 -- 172.21.15.45:0/2880837538 wait complete. 2024-03-28T17:57:05.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.645+0000 7fd6f8990740 1 -- start start 2024-03-28T17:57:05.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.645+0000 7fd6f8990740 1 Processor -- start 2024-03-28T17:57:05.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.645+0000 7fd6f8990740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c42060f20 0x562c42069990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:05.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.645+0000 7fd6f8990740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562c42061c00 0x562c42066d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:05.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.645+0000 7fd6f8990740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c4206a290 0x562c42067260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:05.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.645+0000 7fd6f8990740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c4206cec0 con 0x562c42060f20 2024-03-28T17:57:05.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.646+0000 7fd6f8990740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c41f80e80 con 0x562c4206a290 2024-03-28T17:57:05.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.646+0000 7fd6f8990740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c4206d040 con 0x562c42061c00 2024-03-28T17:57:05.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.646+0000 7fd6f8990740 10 client.0 _ll_register_callbacks cb 0x562c41f93130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:57:05.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.646+0000 7fd6ef400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c4206a290 0x562c42067260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:05.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.646+0000 7fd6eea00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c42060f20 0x562c42069990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:05.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.646+0000 7fd6ef400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c4206a290 0x562c42067260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:33238/0 (socket says 172.21.15.45:33238) 2024-03-28T17:57:05.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.646+0000 7fd6ef400640 1 -- 172.21.15.45:0/2296980701 learned_addr learned my addr 172.21.15.45:0/2296980701 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:57:05.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.646+0000 7fd6ee000640 1 --2- 172.21.15.45:0/2296980701 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562c42061c00 0x562c42066d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:05.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.646+0000 7fd6ef400640 1 -- 172.21.15.45:0/2296980701 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562c42061c00 msgr2=0x562c42066d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:57:05.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.646+0000 7fd6ef400640 1 --2- 172.21.15.45:0/2296980701 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562c42061c00 0x562c42066d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.646+0000 7fd6ef400640 1 -- 172.21.15.45:0/2296980701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c42060f20 msgr2=0x562c42069990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:57:05.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.646+0000 7fd6ef400640 1 --2- 172.21.15.45:0/2296980701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c42060f20 0x562c42069990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.646+0000 7fd6ef400640 1 -- 172.21.15.45:0/2296980701 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562c41da37d0 con 0x562c4206a290 2024-03-28T17:57:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.647+0000 7fd6ef400640 1 --2- 172.21.15.45:0/2296980701 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c4206a290 0x562c42067260 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8005f40 tx=0x7fd6e8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8a880785a763b0f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:57:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.647+0000 7fd6dcc00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:57:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.647+0000 7fd6dcc00640 1 -- 172.21.15.45:0/2296980701 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e801ed30 con 0x562c4206a290 2024-03-28T17:57:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.647+0000 7fd6f8990740 1 -- 172.21.15.45:0/2296980701 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd6e00fe030 con 0x562c4206a290 2024-03-28T17:57:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.647+0000 7fd6dcc00640 1 -- 172.21.15.45:0/2296980701 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 10490+0+0 (secure 0 0 0) 0x7fd6e802a480 con 0x562c4206a290 2024-03-28T17:57:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.647+0000 7fd6dcc00640 1 -- 172.21.15.45:0/2296980701 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=174) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6e8033050 con 0x562c4206a290 2024-03-28T17:57:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.647+0000 7fd6f8990740 10 client.26249 fetch_fsmap learned FSMap version 174 2024-03-28T17:57:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.647+0000 7fd6f8990740 1 -- 172.21.15.45:0/2296980701 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=174,monmap=4+}) v3 -- 0x562c41cd93f0 con 0x562c4206a290 2024-03-28T17:57:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.647+0000 7fd6dcc00640 1 -- 172.21.15.45:0/2296980701 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6e00fe030 con 0x562c4206a290 2024-03-28T17:57:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.648+0000 7fd6dcc00640 1 -- 172.21.15.45:0/2296980701 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 174) v1 ==== 6193+0+0 (secure 0 0 0) 0x7fd6e800f040 con 0x562c4206a290 2024-03-28T17:57:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.648+0000 7fd6f8990740 10 client.26249 fetch_fsmap finished waiting for FSMap version 174 2024-03-28T17:57:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.648+0000 7fd6f8990740 10 client.26249 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:57:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.648+0000 7fd6f8990740 20 client.26249 populate_metadata read hostname 'smithi045' 2024-03-28T17:57:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.648+0000 7fd6f8990740 1 --2- 172.21.15.45:0/2296980701 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x562c4207a070 0x562c4207c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:05.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.648+0000 7fd6f8990740 4 client.26249 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:57:05.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.648+0000 7fd6f8990740 1 -- 172.21.15.45:0/2296980701 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562c41fd2790 con 0x562c4207a070 2024-03-28T17:57:05.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.648+0000 7fd6eea00640 1 --2- 172.21.15.45:0/2296980701 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x562c4207a070 0x562c4207c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:05.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.649+0000 7fd6eea00640 1 --2- 172.21.15.45:0/2296980701 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x562c4207a070 0x562c4207c4a0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3616088668bc0528 server_cookie=401d4da00732989d in_seq=0 out_seq=0 2024-03-28T17:57:05.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.649+0000 7fd6dcc00640 10 client.26249 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:57:05.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.649+0000 7fd6dcc00640 1 -- 172.21.15.45:0/2296980701 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562c41fd2790 con 0x562c4207a070 2024-03-28T17:57:05.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.649+0000 7fd6dcc00640 10 client.26249 handle_command_reply: tid=0 2024-03-28T17:57:05.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.649+0000 7fd6f8990740 1 client.26249 shutdown 2024-03-28T17:57:05.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.650+0000 7fd6f8990740 1 -- 172.21.15.45:0/2296980701 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c4206a290 msgr2=0x562c42067260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:57:05.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.650+0000 7fd6f8990740 1 --2- 172.21.15.45:0/2296980701 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c4206a290 0x562c42067260 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8005f40 tx=0x7fd6e8004a40 comp rx=0 tx=0).stop 2024-03-28T17:57:05.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.650+0000 7fd6f8990740 1 -- 172.21.15.45:0/2296980701 shutdown_connections 2024-03-28T17:57:05.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.650+0000 7fd6f8990740 1 --2- 172.21.15.45:0/2296980701 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x562c4207a070 0x562c4207c4a0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.650+0000 7fd6f8990740 1 --2- 172.21.15.45:0/2296980701 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c4206a290 0x562c42067260 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.650+0000 7fd6f8990740 1 --2- 172.21.15.45:0/2296980701 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562c42061c00 0x562c42066d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.650+0000 7fd6dcc00640 0 client.26249 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:57:05.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.650+0000 7fd6f8990740 1 --2- 172.21.15.45:0/2296980701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c42060f20 0x562c42069990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.650+0000 7fd6f8990740 1 -- 172.21.15.45:0/2296980701 >> 172.21.15.45:0/2296980701 conn(0x562c42028200 msgr2=0x562c4205d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:57:05.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.650+0000 7fd6f8990740 1 -- 172.21.15.45:0/2296980701 shutdown_connections 2024-03-28T17:57:05.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.650+0000 7fd6f8990740 1 -- 172.21.15.45:0/2296980701 wait complete. 2024-03-28T17:57:05.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.650+0000 7fd6f8990740 20 client.26249 trim_cache size 0 max 16384 2024-03-28T17:57:05.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.668+0000 7fd6f8990740 1 Processor -- start 2024-03-28T17:57:05.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.668+0000 7fd6f8990740 1 -- start start 2024-03-28T17:57:05.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.668+0000 7fd6f8990740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c42060f20 0x562c42065d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:05.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.668+0000 7fd6f8990740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562c42061c00 0x562c42069450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:05.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.668+0000 7fd6f8990740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4206a290 0x562c42069990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:05.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.668+0000 7fd6f8990740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f01b6310 con 0x562c4206a290 2024-03-28T17:57:05.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.668+0000 7fd6f8990740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f01b4ff0 con 0x562c42060f20 2024-03-28T17:57:05.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.668+0000 7fd6f8990740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f01b5650 con 0x562c42061c00 2024-03-28T17:57:05.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.669+0000 7fd6eea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c42060f20 0x562c42065d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:05.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.669+0000 7fd6eea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c42060f20 0x562c42065d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:33248/0 (socket says 172.21.15.45:33248) 2024-03-28T17:57:05.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.669+0000 7fd6eea00640 1 -- 172.21.15.45:0/833357479 learned_addr learned my addr 172.21.15.45:0/833357479 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:57:05.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.669+0000 7fd6ef400640 1 --2- 172.21.15.45:0/833357479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4206a290 0x562c42069990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:05.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.669+0000 7fd6eea00640 1 -- 172.21.15.45:0/833357479 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562c42061c00 msgr2=0x562c42069450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:57:05.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.669+0000 7fd6eea00640 1 --2- 172.21.15.45:0/833357479 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562c42061c00 0x562c42069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.669+0000 7fd6eea00640 1 -- 172.21.15.45:0/833357479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4206a290 msgr2=0x562c42069990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:57:05.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.669+0000 7fd6eea00640 1 --2- 172.21.15.45:0/833357479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4206a290 0x562c42069990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.669+0000 7fd6eea00640 1 -- 172.21.15.45:0/833357479 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562c41c990f0 con 0x562c42060f20 2024-03-28T17:57:05.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.669+0000 7fd6ef400640 1 --2- 172.21.15.45:0/833357479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4206a290 0x562c42069990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:57:05.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.669+0000 7fd6eea00640 1 --2- 172.21.15.45:0/833357479 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c42060f20 0x562c42065d20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8000c00 tx=0x7fd6d8005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc0a72335378bf2c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:57:05.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.669+0000 7fd6df400640 1 -- 172.21.15.45:0/833357479 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8012110 con 0x562c42060f20 2024-03-28T17:57:05.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.670+0000 7fd6df400640 1 -- 172.21.15.45:0/833357479 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6d8005500 con 0x562c42060f20 2024-03-28T17:57:05.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.670+0000 7fd6df400640 1 -- 172.21.15.45:0/833357479 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8004cd0 con 0x562c42060f20 2024-03-28T17:57:05.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.670+0000 7fd6f8990740 1 -- 172.21.15.45:0/833357479 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c42060f20 msgr2=0x562c42065d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:57:05.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.670+0000 7fd6f8990740 1 --2- 172.21.15.45:0/833357479 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c42060f20 0x562c42065d20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8000c00 tx=0x7fd6d8005f90 comp rx=0 tx=0).stop 2024-03-28T17:57:05.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.670+0000 7fd6f8990740 1 -- 172.21.15.45:0/833357479 shutdown_connections 2024-03-28T17:57:05.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.670+0000 7fd6f8990740 1 --2- 172.21.15.45:0/833357479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4206a290 0x562c42069990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.670+0000 7fd6f8990740 1 --2- 172.21.15.45:0/833357479 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562c42061c00 0x562c42069450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.670+0000 7fd6f8990740 1 --2- 172.21.15.45:0/833357479 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c42060f20 0x562c42065d20 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.670+0000 7fd6f8990740 1 -- 172.21.15.45:0/833357479 >> 172.21.15.45:0/833357479 conn(0x562c42028200 msgr2=0x562c42065610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:57:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.670+0000 7fd6f8990740 1 -- 172.21.15.45:0/833357479 shutdown_connections 2024-03-28T17:57:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.670+0000 7fd6f8990740 1 -- 172.21.15.45:0/833357479 wait complete. 2024-03-28T17:57:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.671+0000 7fd6f8990740 1 -- start start 2024-03-28T17:57:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.671+0000 7fd6f8990740 1 Processor -- start 2024-03-28T17:57:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.671+0000 7fd6f8990740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562c42060f20 0x562c42063fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.671+0000 7fd6f8990740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c42061c00 0x562c42064500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.671+0000 7fd6f8990740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4206a290 0x562c4206ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.671+0000 7fd6f8990740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f01b3870 con 0x562c4206a290 2024-03-28T17:57:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.671+0000 7fd6f8990740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f01b5cb0 con 0x562c42061c00 2024-03-28T17:57:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.671+0000 7fd6f8990740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f01b2c50 con 0x562c42060f20 2024-03-28T17:57:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.672+0000 7fd6f8990740 10 client.0 _ll_register_callbacks cb 0x7fd6e8009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:57:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.672+0000 7fd6ef400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4206a290 0x562c4206ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.672+0000 7fd6ef400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4206a290 0x562c4206ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41724/0 (socket says 172.21.15.45:41724) 2024-03-28T17:57:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.672+0000 7fd6ef400640 1 -- 172.21.15.45:0/1386005329 learned_addr learned my addr 172.21.15.45:0/1386005329 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:57:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.672+0000 7fd6ee000640 1 --2- 172.21.15.45:0/1386005329 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c42061c00 0x562c42064500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.672+0000 7fd6eea00640 1 --2- 172.21.15.45:0/1386005329 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562c42060f20 0x562c42063fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.672+0000 7fd6ef400640 1 -- 172.21.15.45:0/1386005329 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562c42060f20 msgr2=0x562c42063fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:57:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.672+0000 7fd6ef400640 1 --2- 172.21.15.45:0/1386005329 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562c42060f20 0x562c42063fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.672+0000 7fd6ef400640 1 -- 172.21.15.45:0/1386005329 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c42061c00 msgr2=0x562c42064500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:57:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.672+0000 7fd6ef400640 1 --2- 172.21.15.45:0/1386005329 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c42061c00 0x562c42064500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.672+0000 7fd6ef400640 1 -- 172.21.15.45:0/1386005329 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562c4206b1d0 con 0x562c4206a290 2024-03-28T17:57:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.672+0000 7fd6ee000640 1 --2- 172.21.15.45:0/1386005329 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c42061c00 0x562c42064500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:57:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.672+0000 7fd6ef400640 1 --2- 172.21.15.45:0/1386005329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4206a290 0x562c4206ac90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd6e801f790 tx=0x7fd6e801f640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad70b759d06acd1d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:57:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.673+0000 7fd6dcc00640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-28T17:57:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.673+0000 7fd6dcc00640 1 -- 172.21.15.45:0/1386005329 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e801e770 con 0x562c4206a290 2024-03-28T17:57:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.673+0000 7fd6dcc00640 1 -- 172.21.15.45:0/1386005329 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 10490+0+0 (secure 0 0 0) 0x7fd6e801eb30 con 0x562c4206a290 2024-03-28T17:57:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.673+0000 7fd6f8990740 1 -- 172.21.15.45:0/1386005329 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd6d8005500 con 0x562c4206a290 2024-03-28T17:57:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.673+0000 7fd6dcc00640 1 -- 172.21.15.45:0/1386005329 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=174) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6e8048050 con 0x562c4206a290 2024-03-28T17:57:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.673+0000 7fd6f8990740 10 client.15999 fetch_fsmap learned FSMap version 174 2024-03-28T17:57:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.673+0000 7fd6f8990740 1 -- 172.21.15.45:0/1386005329 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=174,monmap=4+}) v3 -- 0x562c420d5fb0 con 0x562c4206a290 2024-03-28T17:57:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.673+0000 7fd6dcc00640 1 -- 172.21.15.45:0/1386005329 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6d8005500 con 0x562c4206a290 2024-03-28T17:57:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.674+0000 7fd6dcc00640 1 -- 172.21.15.45:0/1386005329 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 174) v1 ==== 6193+0+0 (secure 0 0 0) 0x7fd6e800f040 con 0x562c4206a290 2024-03-28T17:57:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.674+0000 7fd6f8990740 10 client.15999 fetch_fsmap finished waiting for FSMap version 174 2024-03-28T17:57:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.674+0000 7fd6f8990740 10 client.15999 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:57:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.674+0000 7fd6f8990740 20 client.15999 populate_metadata read hostname 'smithi045' 2024-03-28T17:57:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.674+0000 7fd6f8990740 1 --2- 172.21.15.45:0/1386005329 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x562c420d6730 0x562c420d8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.674+0000 7fd6f8990740 4 client.15999 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:57:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.674+0000 7fd6f8990740 1 -- 172.21.15.45:0/1386005329 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562c41fccae0 con 0x562c420d6730 2024-03-28T17:57:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.674+0000 7fd6eea00640 1 --2- 172.21.15.45:0/1386005329 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x562c420d6730 0x562c420d8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:05.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.674+0000 7fd6eea00640 1 --2- 172.21.15.45:0/1386005329 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x562c420d6730 0x562c420d8b50 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=956d8f59c208e2d6 server_cookie=783fa10151ca1286 in_seq=0 out_seq=0 2024-03-28T17:57:05.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.674+0000 7fd6dcc00640 10 client.15999 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:57:05.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.675+0000 7fd6dcc00640 1 -- 172.21.15.45:0/1386005329 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x562c41fccae0 con 0x562c420d6730 2024-03-28T17:57:05.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.675+0000 7fd6dcc00640 10 client.15999 handle_command_reply: tid=0 2024-03-28T17:57:05.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.675+0000 7fd6f8990740 1 client.15999 shutdown 2024-03-28T17:57:05.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.675+0000 7fd6f8990740 1 -- 172.21.15.45:0/1386005329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4206a290 msgr2=0x562c4206ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:57:05.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.675+0000 7fd6f8990740 1 --2- 172.21.15.45:0/1386005329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4206a290 0x562c4206ac90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd6e801f790 tx=0x7fd6e801f640 comp rx=0 tx=0).stop 2024-03-28T17:57:05.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.675+0000 7fd6f8990740 1 -- 172.21.15.45:0/1386005329 shutdown_connections 2024-03-28T17:57:05.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.675+0000 7fd6f8990740 1 --2- 172.21.15.45:0/1386005329 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x562c420d6730 0x562c420d8b50 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.675+0000 7fd6f8990740 1 --2- 172.21.15.45:0/1386005329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4206a290 0x562c4206ac90 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.675+0000 7fd6f8990740 1 --2- 172.21.15.45:0/1386005329 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562c42061c00 0x562c42064500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.675+0000 7fd6dcc00640 0 client.15999 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:57:05.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.675+0000 7fd6f8990740 1 --2- 172.21.15.45:0/1386005329 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x562c42060f20 0x562c42063fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.675+0000 7fd6f8990740 1 -- 172.21.15.45:0/1386005329 >> 172.21.15.45:0/1386005329 conn(0x562c42028200 msgr2=0x562c42065ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:57:05.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.675+0000 7fd6f8990740 1 -- 172.21.15.45:0/1386005329 shutdown_connections 2024-03-28T17:57:05.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.675+0000 7fd6f8990740 1 -- 172.21.15.45:0/1386005329 wait complete. 2024-03-28T17:57:05.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.675+0000 7fd6f8990740 20 client.15999 trim_cache size 0 max 16384 2024-03-28T17:57:05.680 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:57:05.680 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T17:57:05.680 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T17:57:05.680 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T17:57:05.680 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T17:57:05.680 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T17:57:05.680 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T17:57:05.680 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:57:05.680 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:57:05.680 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:57:05.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.676+0000 7fd6f5c00640 1 -- 172.21.15.45:0/1571386848 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd6d40a2970 msgr2=0x7fd6d40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:57:05.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.676+0000 7fd6f5c00640 1 --2- 172.21.15.45:0/1571386848 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd6d40a2970 0x7fd6d40a4e30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8005fd0 tx=0x7fd6e8009600 comp rx=0 tx=0).stop 2024-03-28T17:57:05.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.676+0000 7fd6f5c00640 1 -- 172.21.15.45:0/1571386848 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6f010cf80 msgr2=0x7fd6f019eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:57:05.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.676+0000 7fd6f5c00640 1 --2- 172.21.15.45:0/1571386848 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6f010cf80 0x7fd6f019eae0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0036040 tx=0x7fd6e0004360 comp rx=0 tx=0).stop 2024-03-28T17:57:05.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.676+0000 7fd6f5c00640 1 -- 172.21.15.45:0/1571386848 shutdown_connections 2024-03-28T17:57:05.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.676+0000 7fd6f5c00640 1 --2- 172.21.15.45:0/1571386848 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd6d40a2970 0x7fd6d40a4e30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.676+0000 7fd6f5c00640 1 --2- 172.21.15.45:0/1571386848 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6f01a2e60 0x7fd6f01a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.676+0000 7fd6f5c00640 1 --2- 172.21.15.45:0/1571386848 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6f010cf80 0x7fd6f019eae0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.676+0000 7fd6f5c00640 1 --2- 172.21.15.45:0/1571386848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6f0107eb0 0x7fd6f019e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:05.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.676+0000 7fd6f5c00640 1 -- 172.21.15.45:0/1571386848 >> 172.21.15.45:0/1571386848 conn(0x7fd6f0075be0 msgr2=0x7fd6f0077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:57:05.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.676+0000 7fd6f5c00640 1 -- 172.21.15.45:0/1571386848 shutdown_connections 2024-03-28T17:57:05.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:05.676+0000 7fd6f5c00640 1 -- 172.21.15.45:0/1571386848 wait complete. 2024-03-28T17:57:05.696 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T17:57:05.889 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 278 sec latency 393785.818 ms 2024-03-28T17:57:06.612 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:06 smithi071 ceph-mon[36597]: pgmap v271: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T17:57:06.612 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:06 smithi071 ceph-mon[36597]: Replacing daemon mds.k as rank 2 with standby daemon mds.l 2024-03-28T17:57:06.612 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:06 smithi071 ceph-mon[36597]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-03-28T17:57:06.613 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:06 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:57:06.613 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:06 smithi071 ceph-mon[36597]: osdmap e114: 12 total, 12 up, 12 in 2024-03-28T17:57:06.613 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:06 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:06.637 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:06 smithi151 ceph-mon[36481]: pgmap v271: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T17:57:06.637 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:06 smithi151 ceph-mon[36481]: Replacing daemon mds.k as rank 2 with standby daemon mds.l 2024-03-28T17:57:06.637 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:06 smithi151 ceph-mon[36481]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-03-28T17:57:06.637 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:06 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:57:06.638 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:06 smithi151 ceph-mon[36481]: osdmap e114: 12 total, 12 up, 12 in 2024-03-28T17:57:06.638 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:06 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:06.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:06 smithi045 ceph-mon[31773]: pgmap v271: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T17:57:06.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:06 smithi045 ceph-mon[31773]: Replacing daemon mds.k as rank 2 with standby daemon mds.l 2024-03-28T17:57:06.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:06 smithi045 ceph-mon[31773]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-03-28T17:57:06.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:06 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:57:06.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:06 smithi045 ceph-mon[31773]: osdmap e114: 12 total, 12 up, 12 in 2024-03-28T17:57:06.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:06 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:06.889 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 279 sec latency 394785.977 ms 2024-03-28T17:57:07.889 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 280 sec latency 395786.107 ms 2024-03-28T17:57:08.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:08 smithi045 ceph-mon[31773]: pgmap v273: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:57:08.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:08 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/797518908,v1:172.21.15.151:6833/797518908] up:resolve 2024-03-28T17:57:08.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:08 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:08.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:08 smithi071 ceph-mon[36597]: pgmap v273: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:57:08.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:08 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/797518908,v1:172.21.15.151:6833/797518908] up:resolve 2024-03-28T17:57:08.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:08 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:08 smithi151 ceph-mon[36481]: pgmap v273: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:57:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:08 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/797518908,v1:172.21.15.151:6833/797518908] up:resolve 2024-03-28T17:57:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:08 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:08.889 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 281 sec latency 396786.237 ms 2024-03-28T17:57:09.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:09 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:57:09.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:09 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/797518908,v1:172.21.15.151:6833/797518908] up:reconnect 2024-03-28T17:57:09.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:09 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:09.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:09 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:57:09.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:09 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:57:09.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:09 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/797518908,v1:172.21.15.151:6833/797518908] up:reconnect 2024-03-28T17:57:09.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:09 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:09.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:09 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:57:09.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:09 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:57:09.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:09 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/797518908,v1:172.21.15.151:6833/797518908] up:reconnect 2024-03-28T17:57:09.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:09 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:09.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:09 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:57:09.889 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 282 sec latency 397786.373 ms 2024-03-28T17:57:10.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:10 smithi045 ceph-mon[31773]: pgmap v274: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:57:10.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:10 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/797518908,v1:172.21.15.151:6833/797518908] up:rejoin 2024-03-28T17:57:10.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:10 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:10.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:10 smithi045 ceph-mon[31773]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-03-28T17:57:10.776 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-mon[36481]: pgmap v274: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:57:10.776 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/797518908,v1:172.21.15.151:6833/797518908] up:rejoin 2024-03-28T17:57:10.776 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:10.776 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-mon[36481]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-03-28T17:57:10.777 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fca060e4700 time 2024-03-28T17:57:10.403075+0000 2024-03-28T17:57:10.777 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:57:10.777 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:10.777 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fca1070e4a1] 2024-03-28T17:57:10.777 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fca1070e667] 2024-03-28T17:57:10.777 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563ee8f5c223] 2024-03-28T17:57:10.777 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563ee8dd65d6] 2024-03-28T17:57:10.777 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563ee8dd6da1] 2024-03-28T17:57:10.777 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563ee8dd92d5] 2024-03-28T17:57:10.778 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563ee8e29233] 2024-03-28T17:57:10.778 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563ee8e2b197] 2024-03-28T17:57:10.778 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563ee8d2a11b] 2024-03-28T17:57:10.778 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563ee8d2a6bc] 2024-03-28T17:57:10.778 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563ee8d141ff] 2024-03-28T17:57:10.778 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fca109a0488] 2024-03-28T17:57:10.778 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 13: (DispatchQueue::entry()+0x51f) [0x7fca1099d60f] 2024-03-28T17:57:10.778 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fca10a70e81] 2024-03-28T17:57:10.778 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fca0f48b1ca] 2024-03-28T17:57:10.778 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 16: clone() 2024-03-28T17:57:10.779 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 2024-03-28T17:57:10.403+0000 7fca060e4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fca060e4700 time 2024-03-28T17:57:10.403075+0000 2024-03-28T17:57:10.779 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:57:10.779 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 2024-03-28T17:57:10.779 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:10.779 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fca1070e4a1] 2024-03-28T17:57:10.780 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fca1070e667] 2024-03-28T17:57:10.780 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563ee8f5c223] 2024-03-28T17:57:10.780 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563ee8dd65d6] 2024-03-28T17:57:10.780 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563ee8dd6da1] 2024-03-28T17:57:10.780 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563ee8dd92d5] 2024-03-28T17:57:10.780 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563ee8e29233] 2024-03-28T17:57:10.780 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563ee8e2b197] 2024-03-28T17:57:10.780 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563ee8d2a11b] 2024-03-28T17:57:10.780 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563ee8d2a6bc] 2024-03-28T17:57:10.780 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563ee8d141ff] 2024-03-28T17:57:10.780 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fca109a0488] 2024-03-28T17:57:10.780 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 13: (DispatchQueue::entry()+0x51f) [0x7fca1099d60f] 2024-03-28T17:57:10.780 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fca10a70e81] 2024-03-28T17:57:10.781 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fca0f48b1ca] 2024-03-28T17:57:10.781 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 16: clone() 2024-03-28T17:57:10.781 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 2024-03-28T17:57:10.781 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: *** Caught signal (Aborted) ** 2024-03-28T17:57:10.781 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: in thread 7fca060e4700 thread_name:ms_dispatch 2024-03-28T17:57:10.781 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:10.781 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fca0f495d20] 2024-03-28T17:57:10.781 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 2: gsignal() 2024-03-28T17:57:10.781 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 3: abort() 2024-03-28T17:57:10.781 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fca1070e4fb] 2024-03-28T17:57:10.781 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fca1070e667] 2024-03-28T17:57:10.781 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563ee8f5c223] 2024-03-28T17:57:10.781 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563ee8dd65d6] 2024-03-28T17:57:10.782 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563ee8dd6da1] 2024-03-28T17:57:10.782 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563ee8dd92d5] 2024-03-28T17:57:10.782 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563ee8e29233] 2024-03-28T17:57:10.782 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563ee8e2b197] 2024-03-28T17:57:10.782 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563ee8d2a11b] 2024-03-28T17:57:10.782 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563ee8d2a6bc] 2024-03-28T17:57:10.782 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563ee8d141ff] 2024-03-28T17:57:10.782 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fca109a0488] 2024-03-28T17:57:10.782 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 16: (DispatchQueue::entry()+0x51f) [0x7fca1099d60f] 2024-03-28T17:57:10.782 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fca10a70e81] 2024-03-28T17:57:10.782 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fca0f48b1ca] 2024-03-28T17:57:10.782 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 19: clone() 2024-03-28T17:57:10.782 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 2024-03-28T17:57:10.405+0000 7fca060e4700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:57:10.782 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: in thread 7fca060e4700 thread_name:ms_dispatch 2024-03-28T17:57:10.782 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 2024-03-28T17:57:10.783 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:10.783 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fca0f495d20] 2024-03-28T17:57:10.783 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 2: gsignal() 2024-03-28T17:57:10.783 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 3: abort() 2024-03-28T17:57:10.783 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fca1070e4fb] 2024-03-28T17:57:10.783 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fca1070e667] 2024-03-28T17:57:10.783 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563ee8f5c223] 2024-03-28T17:57:10.783 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563ee8dd65d6] 2024-03-28T17:57:10.783 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563ee8dd6da1] 2024-03-28T17:57:10.783 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563ee8dd92d5] 2024-03-28T17:57:10.783 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563ee8e29233] 2024-03-28T17:57:10.784 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563ee8e2b197] 2024-03-28T17:57:10.784 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563ee8d2a11b] 2024-03-28T17:57:10.784 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563ee8d2a6bc] 2024-03-28T17:57:10.784 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563ee8d141ff] 2024-03-28T17:57:10.784 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fca109a0488] 2024-03-28T17:57:10.784 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 16: (DispatchQueue::entry()+0x51f) [0x7fca1099d60f] 2024-03-28T17:57:10.784 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fca10a70e81] 2024-03-28T17:57:10.784 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fca0f48b1ca] 2024-03-28T17:57:10.784 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 19: clone() 2024-03-28T17:57:10.784 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:57:10.784 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 2024-03-28T17:57:10.784 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: -1> 2024-03-28T17:57:10.403+0000 7fca060e4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fca060e4700 time 2024-03-28T17:57:10.403075+0000 2024-03-28T17:57:10.785 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:57:10.785 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 2024-03-28T17:57:10.785 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:10.785 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fca1070e4a1] 2024-03-28T17:57:10.785 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fca1070e667] 2024-03-28T17:57:10.785 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563ee8f5c223] 2024-03-28T17:57:10.785 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563ee8dd65d6] 2024-03-28T17:57:10.785 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563ee8dd6da1] 2024-03-28T17:57:10.785 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563ee8dd92d5] 2024-03-28T17:57:10.785 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563ee8e29233] 2024-03-28T17:57:10.785 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563ee8e2b197] 2024-03-28T17:57:10.785 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563ee8d2a11b] 2024-03-28T17:57:10.785 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563ee8d2a6bc] 2024-03-28T17:57:10.785 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563ee8d141ff] 2024-03-28T17:57:10.785 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fca109a0488] 2024-03-28T17:57:10.785 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 13: (DispatchQueue::entry()+0x51f) [0x7fca1099d60f] 2024-03-28T17:57:10.786 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fca10a70e81] 2024-03-28T17:57:10.786 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fca0f48b1ca] 2024-03-28T17:57:10.786 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 16: clone() 2024-03-28T17:57:10.786 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 2024-03-28T17:57:10.786 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 0> 2024-03-28T17:57:10.405+0000 7fca060e4700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:57:10.786 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: in thread 7fca060e4700 thread_name:ms_dispatch 2024-03-28T17:57:10.786 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 2024-03-28T17:57:10.786 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:10.786 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fca0f495d20] 2024-03-28T17:57:10.786 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 2: gsignal() 2024-03-28T17:57:10.786 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 3: abort() 2024-03-28T17:57:10.786 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fca1070e4fb] 2024-03-28T17:57:10.786 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fca1070e667] 2024-03-28T17:57:10.786 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563ee8f5c223] 2024-03-28T17:57:10.786 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563ee8dd65d6] 2024-03-28T17:57:10.787 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563ee8dd6da1] 2024-03-28T17:57:10.787 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563ee8dd92d5] 2024-03-28T17:57:10.787 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563ee8e29233] 2024-03-28T17:57:10.787 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563ee8e2b197] 2024-03-28T17:57:10.787 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563ee8d2a11b] 2024-03-28T17:57:10.788 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563ee8d2a6bc] 2024-03-28T17:57:10.788 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563ee8d141ff] 2024-03-28T17:57:10.788 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fca109a0488] 2024-03-28T17:57:10.788 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 16: (DispatchQueue::entry()+0x51f) [0x7fca1099d60f] 2024-03-28T17:57:10.788 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fca10a70e81] 2024-03-28T17:57:10.788 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fca0f48b1ca] 2024-03-28T17:57:10.788 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 19: clone() 2024-03-28T17:57:10.788 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:57:10.788 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 2024-03-28T17:57:10.788 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: -1> 2024-03-28T17:57:10.403+0000 7fca060e4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fca060e4700 time 2024-03-28T17:57:10.403075+0000 2024-03-28T17:57:10.788 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:57:10.788 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 2024-03-28T17:57:10.788 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:10.788 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fca1070e4a1] 2024-03-28T17:57:10.788 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fca1070e667] 2024-03-28T17:57:10.788 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563ee8f5c223] 2024-03-28T17:57:10.789 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563ee8dd65d6] 2024-03-28T17:57:10.789 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563ee8dd6da1] 2024-03-28T17:57:10.789 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563ee8dd92d5] 2024-03-28T17:57:10.789 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563ee8e29233] 2024-03-28T17:57:10.789 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563ee8e2b197] 2024-03-28T17:57:10.789 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563ee8d2a11b] 2024-03-28T17:57:10.789 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563ee8d2a6bc] 2024-03-28T17:57:10.789 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563ee8d141ff] 2024-03-28T17:57:10.789 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fca109a0488] 2024-03-28T17:57:10.789 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 13: (DispatchQueue::entry()+0x51f) [0x7fca1099d60f] 2024-03-28T17:57:10.789 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fca10a70e81] 2024-03-28T17:57:10.789 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fca0f48b1ca] 2024-03-28T17:57:10.789 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 16: clone() 2024-03-28T17:57:10.789 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 2024-03-28T17:57:10.789 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 0> 2024-03-28T17:57:10.405+0000 7fca060e4700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:57:10.790 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: in thread 7fca060e4700 thread_name:ms_dispatch 2024-03-28T17:57:10.790 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 2024-03-28T17:57:10.790 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:10.790 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fca0f495d20] 2024-03-28T17:57:10.790 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 2: gsignal() 2024-03-28T17:57:10.790 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 3: abort() 2024-03-28T17:57:10.790 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fca1070e4fb] 2024-03-28T17:57:10.790 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fca1070e667] 2024-03-28T17:57:10.790 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563ee8f5c223] 2024-03-28T17:57:10.790 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563ee8dd65d6] 2024-03-28T17:57:10.790 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563ee8dd6da1] 2024-03-28T17:57:10.790 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563ee8dd92d5] 2024-03-28T17:57:10.790 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563ee8e29233] 2024-03-28T17:57:10.791 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563ee8e2b197] 2024-03-28T17:57:10.791 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563ee8d2a11b] 2024-03-28T17:57:10.791 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563ee8d2a6bc] 2024-03-28T17:57:10.791 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563ee8d141ff] 2024-03-28T17:57:10.791 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fca109a0488] 2024-03-28T17:57:10.791 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 16: (DispatchQueue::entry()+0x51f) [0x7fca1099d60f] 2024-03-28T17:57:10.791 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fca10a70e81] 2024-03-28T17:57:10.791 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fca0f48b1ca] 2024-03-28T17:57:10.791 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 19: clone() 2024-03-28T17:57:10.791 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:57:10.791 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73230]: 2024-03-28T17:57:10.791 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:10 smithi151 podman[73280]: 2024-03-28 17:57:10.453908778 +0000 UTC m=+0.021139771 container died 0fb90d16f13382d09293905a5f31af16d7380f6542019750ec801598db142f59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-28T17:57:10.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:10 smithi071 ceph-mon[36597]: pgmap v274: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:57:10.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:10 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/797518908,v1:172.21.15.151:6833/797518908] up:rejoin 2024-03-28T17:57:10.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:10 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:10.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:10 smithi071 ceph-mon[36597]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-03-28T17:57:10.889 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 283 sec latency 398786.492 ms 2024-03-28T17:57:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:11 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:57:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:11 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:57:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:11 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/797518908,v1:172.21.15.151:6833/797518908] up:active 2024-03-28T17:57:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:11 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:11.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:11 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:57:11.574 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:11 smithi151 podman[73280]: 2024-03-28 17:57:11.205544727 +0000 UTC m=+0.772775702 container remove 0fb90d16f13382d09293905a5f31af16d7380f6542019750ec801598db142f59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-28T17:57:11.574 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:11 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:57:11.574 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:11 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-28T17:57:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:11 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:57:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:11 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:57:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:11 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/797518908,v1:172.21.15.151:6833/797518908] up:active 2024-03-28T17:57:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:11 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:11 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:57:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:11 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:57:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:11 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:57:11.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:11 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/797518908,v1:172.21.15.151:6833/797518908] up:active 2024-03-28T17:57:11.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:11 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:11.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:11 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:57:11.889 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 284 sec latency 399786.648 ms 2024-03-28T17:57:12.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:12 smithi045 ceph-mon[31773]: pgmap v275: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 204 B/s wr, 7 op/s 2024-03-28T17:57:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:12 smithi071 ceph-mon[36597]: pgmap v275: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 204 B/s wr, 7 op/s 2024-03-28T17:57:12.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:12 smithi151 ceph-mon[36481]: pgmap v275: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 204 B/s wr, 7 op/s 2024-03-28T17:57:12.890 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 285 sec latency 400786.809 ms 2024-03-28T17:57:13.890 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 286 sec latency 401786.947 ms 2024-03-28T17:57:14.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:14 smithi071 ceph-mon[36597]: pgmap v276: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 204 B/s wr, 7 op/s 2024-03-28T17:57:14.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:14 smithi151 ceph-mon[36481]: pgmap v276: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 204 B/s wr, 7 op/s 2024-03-28T17:57:14.890 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 287 sec latency 402787.083 ms 2024-03-28T17:57:14.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:14 smithi045 ceph-mon[31773]: pgmap v276: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 204 B/s wr, 7 op/s 2024-03-28T17:57:15.890 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 288 sec latency 403787.215 ms 2024-03-28T17:57:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:16 smithi071 ceph-mon[36597]: pgmap v277: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:57:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:16 smithi151 ceph-mon[36481]: pgmap v277: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:57:16.890 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 289 sec latency 404787.334 ms 2024-03-28T17:57:16.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:16 smithi045 ceph-mon[31773]: pgmap v277: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:57:17.890 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 290 sec latency 405787.451 ms 2024-03-28T17:57:18.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:18 smithi071 ceph-mon[36597]: pgmap v278: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-03-28T17:57:18.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:18 smithi151 ceph-mon[36481]: pgmap v278: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-03-28T17:57:18.890 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 291 sec latency 406787.600 ms 2024-03-28T17:57:18.897 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:18 smithi045 ceph-mon[31773]: pgmap v278: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-03-28T17:57:19.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:19 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:57:19.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:19 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:57:19.890 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 292 sec latency 407787.726 ms 2024-03-28T17:57:19.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:19 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:57:20.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:20 smithi071 ceph-mon[36597]: pgmap v279: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:57:20.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:20 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:57:20.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:20 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:57:20.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:20 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:57:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:20 smithi151 ceph-mon[36481]: pgmap v279: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:57:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:20 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:57:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:20 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:57:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:20 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:57:20.891 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 293 sec latency 408787.883 ms 2024-03-28T17:57:20.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:20 smithi045 ceph-mon[31773]: pgmap v279: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T17:57:20.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:20 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:57:20.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:20 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:57:20.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:20 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:57:21.893 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 294 sec latency 409787.997 ms 2024-03-28T17:57:21.932 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:21 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 2. 2024-03-28T17:57:21.932 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:21 smithi151 systemd[1]: Stopped Ceph mds.l for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:57:21.932 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:21 smithi151 systemd[1]: Starting Ceph mds.l for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:57:21.932 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:21 smithi151 podman[73506]: 2024-03-28 17:57:21.712297738 +0000 UTC m=+0.017936357 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:57:21.932 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:21 smithi151 podman[73506]: 2024-03-28 17:57:21.820057193 +0000 UTC m=+0.125695804 container create b41f49335965d3b4bc673195af25b925da440c6803717f9a302a98155e31b8c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-28T17:57:22.184 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:22 smithi151 podman[73506]: 2024-03-28 17:57:22.090296057 +0000 UTC m=+0.395934679 container init b41f49335965d3b4bc673195af25b925da440c6803717f9a302a98155e31b8c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD) 2024-03-28T17:57:22.184 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:22 smithi151 podman[73506]: 2024-03-28 17:57:22.092873054 +0000 UTC m=+0.398511674 container start b41f49335965d3b4bc673195af25b925da440c6803717f9a302a98155e31b8c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T17:57:22.184 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:22 smithi151 ceph-mds[73525]: -- 172.21.15.151:0/2497695607 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ea57365a40 con 0x55ea57338800 2024-03-28T17:57:22.184 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:22 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: starting mds.l at 2024-03-28T17:57:22.184 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:22 smithi151 bash[73506]: b41f49335965d3b4bc673195af25b925da440c6803717f9a302a98155e31b8c1 2024-03-28T17:57:22.445 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:22 smithi151 systemd[1]: Started Ceph mds.l for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:57:22.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:22 smithi071 ceph-mon[36597]: pgmap v280: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-28T17:57:22.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:22 smithi071 ceph-mon[36597]: daemon mds.l restarted 2024-03-28T17:57:22.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:22 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:57:22.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:22 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T17:57:22.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:22 smithi071 ceph-mon[36597]: osdmap e115: 12 total, 12 up, 12 in 2024-03-28T17:57:22.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:22 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/775549656,v1:172.21.15.151:6833/775549656] up:boot 2024-03-28T17:57:22.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:22 smithi071 ceph-mon[36597]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-03-28T17:57:22.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:22 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:57:22.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:22 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:22 smithi151 ceph-mon[36481]: pgmap v280: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-28T17:57:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:22 smithi151 ceph-mon[36481]: daemon mds.l restarted 2024-03-28T17:57:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:22 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:57:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:22 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T17:57:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:22 smithi151 ceph-mon[36481]: osdmap e115: 12 total, 12 up, 12 in 2024-03-28T17:57:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:22 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/775549656,v1:172.21.15.151:6833/775549656] up:boot 2024-03-28T17:57:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:22 smithi151 ceph-mon[36481]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-03-28T17:57:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:22 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:57:22.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:22 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:22.891 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 295 sec latency 410788.153 ms 2024-03-28T17:57:22.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:22 smithi045 ceph-mon[31773]: pgmap v280: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-28T17:57:22.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:22 smithi045 ceph-mon[31773]: daemon mds.l restarted 2024-03-28T17:57:22.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:22 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:57:22.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:22 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T17:57:22.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:22 smithi045 ceph-mon[31773]: osdmap e115: 12 total, 12 up, 12 in 2024-03-28T17:57:22.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:22 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/775549656,v1:172.21.15.151:6833/775549656] up:boot 2024-03-28T17:57:22.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:22 smithi045 ceph-mon[31773]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-03-28T17:57:22.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:22 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:57:22.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:22 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:23.891 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 296 sec latency 411788.318 ms 2024-03-28T17:57:24.718 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:24 smithi045 ceph-mon[31773]: pgmap v282: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:57:24.718 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:24 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6840/2647972571,v1:172.21.15.45:6841/2647972571] up:resolve 2024-03-28T17:57:24.719 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:24 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:24.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:24 smithi071 ceph-mon[36597]: pgmap v282: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:57:24.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:24 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6840/2647972571,v1:172.21.15.45:6841/2647972571] up:resolve 2024-03-28T17:57:24.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:24 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:24.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:24 smithi151 ceph-mon[36481]: pgmap v282: 97 pgs: 97 active+clean; 27 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:57:24.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:24 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6840/2647972571,v1:172.21.15.45:6841/2647972571] up:resolve 2024-03-28T17:57:24.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:24 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:24.891 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 297 sec latency 412788.451 ms 2024-03-28T17:57:25.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:25 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T17:57:25.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:25 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6840/2647972571,v1:172.21.15.45:6841/2647972571] up:reconnect 2024-03-28T17:57:25.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:25 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:25.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:25 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T17:57:25.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:25 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T17:57:25.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:25 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6840/2647972571,v1:172.21.15.45:6841/2647972571] up:reconnect 2024-03-28T17:57:25.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:25 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:25.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:25 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T17:57:25.891 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 298 sec latency 413788.580 ms 2024-03-28T17:57:25.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:25 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T17:57:25.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:25 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6840/2647972571,v1:172.21.15.45:6841/2647972571] up:reconnect 2024-03-28T17:57:25.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:25 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:25.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:25 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T17:57:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-mon[31773]: pgmap v283: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T17:57:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6840/2647972571,v1:172.21.15.45:6841/2647972571] up:rejoin 2024-03-28T17:57:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-mon[31773]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-03-28T17:57:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:57:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:57:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6840/2647972571,v1:172.21.15.45:6841/2647972571] up:active 2024-03-28T17:57:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:57:26.667 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f5970e00700 time 2024-03-28T17:57:26.368942+0000 2024-03-28T17:57:26.667 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:57:26.667 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:26.667 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f597c71e4a1] 2024-03-28T17:57:26.667 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f597c71e667] 2024-03-28T17:57:26.667 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557241b5e223] 2024-03-28T17:57:26.667 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5572419d85d6] 2024-03-28T17:57:26.667 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5572419d8da1] 2024-03-28T17:57:26.667 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5572419db2d5] 2024-03-28T17:57:26.668 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557241a2b233] 2024-03-28T17:57:26.668 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557241a2d197] 2024-03-28T17:57:26.668 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55724192c11b] 2024-03-28T17:57:26.668 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55724192c6bc] 2024-03-28T17:57:26.668 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5572419161ff] 2024-03-28T17:57:26.668 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f597c9b0488] 2024-03-28T17:57:26.668 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 13: (DispatchQueue::entry()+0x51f) [0x7f597c9ad60f] 2024-03-28T17:57:26.668 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f597ca80e81] 2024-03-28T17:57:26.668 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f597b49b1ca] 2024-03-28T17:57:26.668 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 16: clone() 2024-03-28T17:57:26.669 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 2024-03-28T17:57:26.370+0000 7f5970e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f5970e00700 time 2024-03-28T17:57:26.368942+0000 2024-03-28T17:57:26.669 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:57:26.669 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 2024-03-28T17:57:26.669 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:26.669 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f597c71e4a1] 2024-03-28T17:57:26.669 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f597c71e667] 2024-03-28T17:57:26.669 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557241b5e223] 2024-03-28T17:57:26.669 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5572419d85d6] 2024-03-28T17:57:26.669 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5572419d8da1] 2024-03-28T17:57:26.669 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5572419db2d5] 2024-03-28T17:57:26.669 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557241a2b233] 2024-03-28T17:57:26.669 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557241a2d197] 2024-03-28T17:57:26.669 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55724192c11b] 2024-03-28T17:57:26.669 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55724192c6bc] 2024-03-28T17:57:26.670 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5572419161ff] 2024-03-28T17:57:26.670 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f597c9b0488] 2024-03-28T17:57:26.670 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 13: (DispatchQueue::entry()+0x51f) [0x7f597c9ad60f] 2024-03-28T17:57:26.670 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f597ca80e81] 2024-03-28T17:57:26.670 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f597b49b1ca] 2024-03-28T17:57:26.670 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 16: clone() 2024-03-28T17:57:26.670 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 2024-03-28T17:57:26.670 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: *** Caught signal (Aborted) ** 2024-03-28T17:57:26.670 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: in thread 7f5970e00700 thread_name:ms_dispatch 2024-03-28T17:57:26.670 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:26.670 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f597b4a5d20] 2024-03-28T17:57:26.671 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 2: gsignal() 2024-03-28T17:57:26.671 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 3: abort() 2024-03-28T17:57:26.671 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f597c71e4fb] 2024-03-28T17:57:26.671 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f597c71e667] 2024-03-28T17:57:26.671 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557241b5e223] 2024-03-28T17:57:26.671 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5572419d85d6] 2024-03-28T17:57:26.671 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5572419d8da1] 2024-03-28T17:57:26.671 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5572419db2d5] 2024-03-28T17:57:26.671 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557241a2b233] 2024-03-28T17:57:26.671 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557241a2d197] 2024-03-28T17:57:26.671 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55724192c11b] 2024-03-28T17:57:26.671 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55724192c6bc] 2024-03-28T17:57:26.672 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5572419161ff] 2024-03-28T17:57:26.672 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f597c9b0488] 2024-03-28T17:57:26.672 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 16: (DispatchQueue::entry()+0x51f) [0x7f597c9ad60f] 2024-03-28T17:57:26.672 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f597ca80e81] 2024-03-28T17:57:26.672 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f597b49b1ca] 2024-03-28T17:57:26.672 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 19: clone() 2024-03-28T17:57:26.672 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 2024-03-28T17:57:26.373+0000 7f5970e00700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:57:26.672 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: in thread 7f5970e00700 thread_name:ms_dispatch 2024-03-28T17:57:26.672 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 2024-03-28T17:57:26.672 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:26.672 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f597b4a5d20] 2024-03-28T17:57:26.672 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 2: gsignal() 2024-03-28T17:57:26.672 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 3: abort() 2024-03-28T17:57:26.672 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f597c71e4fb] 2024-03-28T17:57:26.672 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f597c71e667] 2024-03-28T17:57:26.673 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557241b5e223] 2024-03-28T17:57:26.673 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5572419d85d6] 2024-03-28T17:57:26.673 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5572419d8da1] 2024-03-28T17:57:26.673 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5572419db2d5] 2024-03-28T17:57:26.673 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557241a2b233] 2024-03-28T17:57:26.673 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557241a2d197] 2024-03-28T17:57:26.673 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55724192c11b] 2024-03-28T17:57:26.673 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55724192c6bc] 2024-03-28T17:57:26.673 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5572419161ff] 2024-03-28T17:57:26.673 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f597c9b0488] 2024-03-28T17:57:26.674 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 16: (DispatchQueue::entry()+0x51f) [0x7f597c9ad60f] 2024-03-28T17:57:26.674 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f597ca80e81] 2024-03-28T17:57:26.674 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f597b49b1ca] 2024-03-28T17:57:26.674 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 19: clone() 2024-03-28T17:57:26.674 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:57:26.674 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 2024-03-28T17:57:26.674 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: -1> 2024-03-28T17:57:26.370+0000 7f5970e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f5970e00700 time 2024-03-28T17:57:26.368942+0000 2024-03-28T17:57:26.674 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:57:26.674 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 2024-03-28T17:57:26.674 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:26.674 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f597c71e4a1] 2024-03-28T17:57:26.674 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f597c71e667] 2024-03-28T17:57:26.675 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557241b5e223] 2024-03-28T17:57:26.675 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5572419d85d6] 2024-03-28T17:57:26.675 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5572419d8da1] 2024-03-28T17:57:26.675 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5572419db2d5] 2024-03-28T17:57:26.675 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557241a2b233] 2024-03-28T17:57:26.675 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557241a2d197] 2024-03-28T17:57:26.675 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55724192c11b] 2024-03-28T17:57:26.675 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55724192c6bc] 2024-03-28T17:57:26.675 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5572419161ff] 2024-03-28T17:57:26.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f597c9b0488] 2024-03-28T17:57:26.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 13: (DispatchQueue::entry()+0x51f) [0x7f597c9ad60f] 2024-03-28T17:57:26.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f597ca80e81] 2024-03-28T17:57:26.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f597b49b1ca] 2024-03-28T17:57:26.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 16: clone() 2024-03-28T17:57:26.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 2024-03-28T17:57:26.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 0> 2024-03-28T17:57:26.373+0000 7f5970e00700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:57:26.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: in thread 7f5970e00700 thread_name:ms_dispatch 2024-03-28T17:57:26.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 2024-03-28T17:57:26.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:26.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f597b4a5d20] 2024-03-28T17:57:26.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 2: gsignal() 2024-03-28T17:57:26.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 3: abort() 2024-03-28T17:57:26.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f597c71e4fb] 2024-03-28T17:57:26.677 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f597c71e667] 2024-03-28T17:57:26.677 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557241b5e223] 2024-03-28T17:57:26.677 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5572419d85d6] 2024-03-28T17:57:26.677 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5572419d8da1] 2024-03-28T17:57:26.677 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5572419db2d5] 2024-03-28T17:57:26.677 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557241a2b233] 2024-03-28T17:57:26.677 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557241a2d197] 2024-03-28T17:57:26.677 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55724192c11b] 2024-03-28T17:57:26.677 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55724192c6bc] 2024-03-28T17:57:26.677 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5572419161ff] 2024-03-28T17:57:26.677 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f597c9b0488] 2024-03-28T17:57:26.677 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 16: (DispatchQueue::entry()+0x51f) [0x7f597c9ad60f] 2024-03-28T17:57:26.677 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f597ca80e81] 2024-03-28T17:57:26.677 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f597b49b1ca] 2024-03-28T17:57:26.678 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 19: clone() 2024-03-28T17:57:26.678 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:57:26.678 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 2024-03-28T17:57:26.678 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: -1> 2024-03-28T17:57:26.370+0000 7f5970e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f5970e00700 time 2024-03-28T17:57:26.368942+0000 2024-03-28T17:57:26.678 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:57:26.678 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 2024-03-28T17:57:26.678 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:26.678 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f597c71e4a1] 2024-03-28T17:57:26.678 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f597c71e667] 2024-03-28T17:57:26.678 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557241b5e223] 2024-03-28T17:57:26.678 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5572419d85d6] 2024-03-28T17:57:26.679 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5572419d8da1] 2024-03-28T17:57:26.679 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5572419db2d5] 2024-03-28T17:57:26.679 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557241a2b233] 2024-03-28T17:57:26.679 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557241a2d197] 2024-03-28T17:57:26.679 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55724192c11b] 2024-03-28T17:57:26.679 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55724192c6bc] 2024-03-28T17:57:26.679 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5572419161ff] 2024-03-28T17:57:26.679 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f597c9b0488] 2024-03-28T17:57:26.679 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 13: (DispatchQueue::entry()+0x51f) [0x7f597c9ad60f] 2024-03-28T17:57:26.679 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f597ca80e81] 2024-03-28T17:57:26.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f597b49b1ca] 2024-03-28T17:57:26.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 16: clone() 2024-03-28T17:57:26.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 2024-03-28T17:57:26.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 0> 2024-03-28T17:57:26.373+0000 7f5970e00700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:57:26.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: in thread 7f5970e00700 thread_name:ms_dispatch 2024-03-28T17:57:26.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 2024-03-28T17:57:26.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:26.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f597b4a5d20] 2024-03-28T17:57:26.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 2: gsignal() 2024-03-28T17:57:26.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 3: abort() 2024-03-28T17:57:26.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f597c71e4fb] 2024-03-28T17:57:26.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f597c71e667] 2024-03-28T17:57:26.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557241b5e223] 2024-03-28T17:57:26.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5572419d85d6] 2024-03-28T17:57:26.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5572419d8da1] 2024-03-28T17:57:26.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5572419db2d5] 2024-03-28T17:57:26.681 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557241a2b233] 2024-03-28T17:57:26.681 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557241a2d197] 2024-03-28T17:57:26.681 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55724192c11b] 2024-03-28T17:57:26.681 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55724192c6bc] 2024-03-28T17:57:26.681 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5572419161ff] 2024-03-28T17:57:26.681 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f597c9b0488] 2024-03-28T17:57:26.681 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 16: (DispatchQueue::entry()+0x51f) [0x7f597c9ad60f] 2024-03-28T17:57:26.681 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f597ca80e81] 2024-03-28T17:57:26.681 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f597b49b1ca] 2024-03-28T17:57:26.681 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 19: clone() 2024-03-28T17:57:26.681 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:57:26.681 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[73777]: 2024-03-28T17:57:26.681 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:26 smithi045 podman[92010]: 2024-03-28 17:57:26.496032366 +0000 UTC m=+0.036812289 container died 6c14419b3a77dcc47a370deeeb01fe0f2017e4cd220f8eabe3be703ce25e36bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T17:57:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:26 smithi071 ceph-mon[36597]: pgmap v283: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T17:57:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:26 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6840/2647972571,v1:172.21.15.45:6841/2647972571] up:rejoin 2024-03-28T17:57:26.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:26 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:26.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:26 smithi071 ceph-mon[36597]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-03-28T17:57:26.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:26 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:57:26.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:26 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:57:26.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:26 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6840/2647972571,v1:172.21.15.45:6841/2647972571] up:active 2024-03-28T17:57:26.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:26 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:57:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:26 smithi151 ceph-mon[36481]: pgmap v283: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T17:57:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:26 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6840/2647972571,v1:172.21.15.45:6841/2647972571] up:rejoin 2024-03-28T17:57:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:26 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:26 smithi151 ceph-mon[36481]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-03-28T17:57:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:26 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:57:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:26 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:57:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:26 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6840/2647972571,v1:172.21.15.45:6841/2647972571] up:active 2024-03-28T17:57:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:26 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:57:26.891 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 299 sec latency 414788.723 ms 2024-03-28T17:57:27.607 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:27 smithi045 podman[92010]: 2024-03-28 17:57:27.25369662 +0000 UTC m=+0.794476597 container remove 6c14419b3a77dcc47a370deeeb01fe0f2017e4cd220f8eabe3be703ce25e36bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-28T17:57:27.608 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:27 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:57:27.892 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 300 sec latency 415788.903 ms 2024-03-28T17:57:28.119 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:27 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Failed with result 'exit-code'. 2024-03-28T17:57:28.119 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:27 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Consumed 1.975s CPU time. 2024-03-28T17:57:28.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:28 smithi071 ceph-mon[36597]: pgmap v284: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T17:57:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:28 smithi151 ceph-mon[36481]: pgmap v284: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T17:57:28.892 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 301 sec latency 416789.066 ms 2024-03-28T17:57:28.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:28 smithi045 ceph-mon[31773]: pgmap v284: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T17:57:29.893 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 302 sec latency 417789.222 ms 2024-03-28T17:57:30.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:30 smithi071 ceph-mon[36597]: pgmap v285: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T17:57:30.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:30 smithi151 ceph-mon[36481]: pgmap v285: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T17:57:30.892 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 303 sec latency 418789.367 ms 2024-03-28T17:57:30.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:30 smithi045 ceph-mon[31773]: pgmap v285: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T17:57:31.892 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 304 sec latency 419789.528 ms 2024-03-28T17:57:32.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:32 smithi071 ceph-mon[36597]: pgmap v286: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:57:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:32 smithi151 ceph-mon[36481]: pgmap v286: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:57:32.893 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 305 sec latency 420789.677 ms 2024-03-28T17:57:32.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:32 smithi045 ceph-mon[31773]: pgmap v286: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:57:33.893 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 306 sec latency 421789.813 ms 2024-03-28T17:57:34.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:34 smithi071 ceph-mon[36597]: pgmap v287: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 12 op/s 2024-03-28T17:57:34.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:34 smithi151 ceph-mon[36481]: pgmap v287: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 12 op/s 2024-03-28T17:57:34.893 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 307 sec latency 422789.904 ms 2024-03-28T17:57:34.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:34 smithi045 ceph-mon[31773]: pgmap v287: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 12 op/s 2024-03-28T17:57:35.697 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:57:35.697 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:57:35.893 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 308 sec latency 423789.996 ms 2024-03-28T17:57:35.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.937+0000 7f0c7b600640 1 -- 172.21.15.45:0/3451981086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c74101ff0 msgr2=0x7f0c7410e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:57:35.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.937+0000 7f0c7b600640 1 --2- 172.21.15.45:0/3451981086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c74101ff0 0x7f0c7410e500 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c009ca0 tx=0x7f0c6c02c100 comp rx=0 tx=0).stop 2024-03-28T17:57:35.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.938+0000 7f0c7b600640 1 -- 172.21.15.45:0/3451981086 shutdown_connections 2024-03-28T17:57:35.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.938+0000 7f0c7b600640 1 --2- 172.21.15.45:0/3451981086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c74101ff0 0x7f0c7410e500 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.938+0000 7f0c7b600640 1 --2- 172.21.15.45:0/3451981086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c74101630 0x7f0c74101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.938+0000 7f0c7b600640 1 --2- 172.21.15.45:0/3451981086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0c74107640 0x7f0c74107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.938+0000 7f0c7b600640 1 -- 172.21.15.45:0/3451981086 >> 172.21.15.45:0/3451981086 conn(0x7f0c74078aa0 msgr2=0x7f0c74078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:57:35.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.938+0000 7f0c7b600640 1 -- 172.21.15.45:0/3451981086 shutdown_connections 2024-03-28T17:57:35.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.938+0000 7f0c7b600640 1 -- 172.21.15.45:0/3451981086 wait complete. 2024-03-28T17:57:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.939+0000 7f0c7b600640 1 Processor -- start 2024-03-28T17:57:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.939+0000 7f0c7b600640 1 -- start start 2024-03-28T17:57:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.939+0000 7f0c7b600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0c74101630 0x7f0c740739a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.939+0000 7f0c7b600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c74101ff0 0x7f0c74073ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.939+0000 7f0c7b600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c74107640 0x7f0c74074420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.939+0000 7f0c7b600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c74111a50 con 0x7f0c74107640 2024-03-28T17:57:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.939+0000 7f0c7b600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c741118d0 con 0x7f0c74101ff0 2024-03-28T17:57:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.939+0000 7f0c7b600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c74111bd0 con 0x7f0c74101630 2024-03-28T17:57:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.940+0000 7f0c79600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c74107640 0x7f0c74074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.940+0000 7f0c73e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c74101ff0 0x7f0c74073ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.940+0000 7f0c73e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c74101ff0 0x7f0c74073ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:43906/0 (socket says 172.21.15.45:43906) 2024-03-28T17:57:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.940+0000 7f0c73e00640 1 -- 172.21.15.45:0/2977555146 learned_addr learned my addr 172.21.15.45:0/2977555146 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:57:35.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.940+0000 7f0c78c00640 1 --2- 172.21.15.45:0/2977555146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0c74101630 0x7f0c740739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:35.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.940+0000 7f0c73e00640 1 -- 172.21.15.45:0/2977555146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0c74101630 msgr2=0x7f0c740739a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:57:35.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.940+0000 7f0c73e00640 1 --2- 172.21.15.45:0/2977555146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0c74101630 0x7f0c740739a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.940+0000 7f0c73e00640 1 -- 172.21.15.45:0/2977555146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c74107640 msgr2=0x7f0c74074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:57:35.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.940+0000 7f0c73e00640 1 --2- 172.21.15.45:0/2977555146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c74107640 0x7f0c74074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.940+0000 7f0c73e00640 1 -- 172.21.15.45:0/2977555146 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c741a6390 con 0x7f0c74101ff0 2024-03-28T17:57:35.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.940+0000 7f0c79600640 1 --2- 172.21.15.45:0/2977555146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c74107640 0x7f0c74074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:57:35.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.940+0000 7f0c73e00640 1 --2- 172.21.15.45:0/2977555146 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c74101ff0 0x7f0c74073ee0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0c6000e9f0 tx=0x7f0c6000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:57:35.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.940+0000 7f0c71600640 1 -- 172.21.15.45:0/2977555146 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6000cd80 con 0x7f0c74101ff0 2024-03-28T17:57:35.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.941+0000 7f0c7b600640 1 -- 172.21.15.45:0/2977555146 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c741a6680 con 0x7f0c74101ff0 2024-03-28T17:57:35.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.941+0000 7f0c71600640 1 -- 172.21.15.45:0/2977555146 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c60010910 con 0x7f0c74101ff0 2024-03-28T17:57:35.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.941+0000 7f0c71600640 1 -- 172.21.15.45:0/2977555146 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c60018990 con 0x7f0c74101ff0 2024-03-28T17:57:35.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.941+0000 7f0c7b600640 1 -- 172.21.15.45:0/2977555146 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c741a6bd0 con 0x7f0c74101ff0 2024-03-28T17:57:35.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.944+0000 7f0c7e3d5740 1 Processor -- start 2024-03-28T17:57:35.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.944+0000 7f0c71600640 1 -- 172.21.15.45:0/2977555146 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f0c60010ac0 con 0x7f0c74101ff0 2024-03-28T17:57:35.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.944+0000 7f0c7e3d5740 1 -- start start 2024-03-28T17:57:35.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.944+0000 7f0c7e3d5740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x564d95b26c00 0x564d95b25950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:35.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.944+0000 7f0c7e3d5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564d95b25f20 0x564d95b263a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:35.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.944+0000 7f0c7e3d5740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b2f290 0x564d95b31680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:35.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.944+0000 7f0c7e3d5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d95a8a250 con 0x564d95b25f20 2024-03-28T17:57:35.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.944+0000 7f0c7e3d5740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d95a32a30 con 0x564d95b2f290 2024-03-28T17:57:35.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.944+0000 7f0c7e3d5740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d95a862f0 con 0x564d95b26c00 2024-03-28T17:57:35.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.945+0000 7f0c78c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564d95b25f20 0x564d95b263a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:35.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.945+0000 7f0c78c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564d95b25f20 0x564d95b263a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40426/0 (socket says 172.21.15.45:40426) 2024-03-28T17:57:35.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.945+0000 7f0c78c00640 1 -- 172.21.15.45:0/75419696 learned_addr learned my addr 172.21.15.45:0/75419696 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:57:35.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.945+0000 7f0c73e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b2f290 0x564d95b31680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:35.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.945+0000 7f0c79600640 1 --2- 172.21.15.45:0/75419696 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x564d95b26c00 0x564d95b25950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:35.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.945+0000 7f0c78c00640 1 -- 172.21.15.45:0/75419696 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x564d95b26c00 msgr2=0x564d95b25950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:57:35.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.945+0000 7f0c78c00640 1 --2- 172.21.15.45:0/75419696 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x564d95b26c00 0x564d95b25950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.945+0000 7f0c78c00640 1 -- 172.21.15.45:0/75419696 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b2f290 msgr2=0x564d95b31680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:57:35.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.945+0000 7f0c78c00640 1 --2- 172.21.15.45:0/75419696 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b2f290 0x564d95b31680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.945+0000 7f0c78c00640 1 -- 172.21.15.45:0/75419696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d95a1d690 con 0x564d95b25f20 2024-03-28T17:57:35.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.945+0000 7f0c78c00640 1 --2- 172.21.15.45:0/75419696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564d95b25f20 0x564d95b263a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0c6800bc90 tx=0x7f0c680107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=987d0f1412e0417 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:57:35.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.946+0000 7f0c71600640 1 --2- 172.21.15.45:0/2977555146 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f0c5c0a2970 0x7f0c5c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:35.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.946+0000 7f0c65c00640 1 -- 172.21.15.45:0/75419696 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6801c480 con 0x564d95b25f20 2024-03-28T17:57:35.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.946+0000 7f0c65c00640 1 -- 172.21.15.45:0/75419696 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c6801cb00 con 0x564d95b25f20 2024-03-28T17:57:35.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.946+0000 7f0c65c00640 1 -- 172.21.15.45:0/75419696 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c68011bb0 con 0x564d95b25f20 2024-03-28T17:57:35.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.946+0000 7f0c71600640 1 -- 172.21.15.45:0/2977555146 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 10662+0+0 (secure 0 0 0) 0x7f0c600cfa20 con 0x7f0c74101ff0 2024-03-28T17:57:35.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.946+0000 7f0c79600640 1 --2- 172.21.15.45:0/2977555146 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f0c5c0a2970 0x7f0c5c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:35.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.946+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/75419696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564d95b25f20 msgr2=0x564d95b263a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:57:35.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.946+0000 7f0c7e3d5740 1 --2- 172.21.15.45:0/75419696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564d95b25f20 0x564d95b263a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0c6800bc90 tx=0x7f0c680107a0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.947+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/75419696 shutdown_connections 2024-03-28T17:57:35.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.947+0000 7f0c7e3d5740 1 --2- 172.21.15.45:0/75419696 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b2f290 0x564d95b31680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.947+0000 7f0c7e3d5740 1 --2- 172.21.15.45:0/75419696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564d95b25f20 0x564d95b263a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.947+0000 7f0c7e3d5740 1 --2- 172.21.15.45:0/75419696 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x564d95b26c00 0x564d95b25950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.947+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/75419696 >> 172.21.15.45:0/75419696 conn(0x564d95aed200 msgr2=0x564d95b20970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:57:35.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.947+0000 7f0c79600640 1 --2- 172.21.15.45:0/2977555146 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f0c5c0a2970 0x7f0c5c0a4e30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c005e30 tx=0x7f0c6c005d80 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:57:35.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.947+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/75419696 shutdown_connections 2024-03-28T17:57:35.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.947+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/75419696 wait complete. 2024-03-28T17:57:35.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.947+0000 7f0c7e3d5740 1 -- start start 2024-03-28T17:57:35.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.947+0000 7f0c7e3d5740 1 Processor -- start 2024-03-28T17:57:35.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.948+0000 7f0c7e3d5740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x564d95b25f20 0x564d95b2e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:35.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.948+0000 7f0c7e3d5740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b26c00 0x564d95b2bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:35.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.948+0000 7f0c7e3d5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564d95b2f290 0x564d95b2c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:35.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.948+0000 7f0c7e3d5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d95b32100 con 0x564d95b2f290 2024-03-28T17:57:35.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.948+0000 7f0c7e3d5740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d95a45e80 con 0x564d95b26c00 2024-03-28T17:57:35.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.948+0000 7f0c7e3d5740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d95b32280 con 0x564d95b25f20 2024-03-28T17:57:35.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.948+0000 7f0c7e3d5740 10 client.0 _ll_register_callbacks cb 0x564d95a58130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:57:35.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.948+0000 7f0c73e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b26c00 0x564d95b2bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:35.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.948+0000 7f0c73e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b26c00 0x564d95b2bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:43920/0 (socket says 172.21.15.45:43920) 2024-03-28T17:57:35.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.948+0000 7f0c73e00640 1 -- 172.21.15.45:0/4053981417 learned_addr learned my addr 172.21.15.45:0/4053981417 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:57:35.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.948+0000 7f0c79600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564d95b2f290 0x564d95b2c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:35.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.948+0000 7f0c78c00640 1 --2- 172.21.15.45:0/4053981417 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x564d95b25f20 0x564d95b2e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:35.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.949+0000 7f0c73e00640 1 -- 172.21.15.45:0/4053981417 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x564d95b25f20 msgr2=0x564d95b2e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:57:35.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.949+0000 7f0c73e00640 1 --2- 172.21.15.45:0/4053981417 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x564d95b25f20 0x564d95b2e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.949+0000 7f0c73e00640 1 -- 172.21.15.45:0/4053981417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564d95b2f290 msgr2=0x564d95b2c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:57:35.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.949+0000 7f0c73e00640 1 --2- 172.21.15.45:0/4053981417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564d95b2f290 0x564d95b2c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.949+0000 7f0c73e00640 1 -- 172.21.15.45:0/4053981417 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564d958687d0 con 0x564d95b26c00 2024-03-28T17:57:35.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.949+0000 7f0c73e00640 1 --2- 172.21.15.45:0/4053981417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b26c00 0x564d95b2bce0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f0c6009b5d0 tx=0x7f0c600d3160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65f6665e0f9d6201 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:57:35.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.949+0000 7f0c5a000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:57:35.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.949+0000 7f0c5a000640 1 -- 172.21.15.45:0/4053981417 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6009b230 con 0x564d95b26c00 2024-03-28T17:57:35.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.949+0000 7f0c5a000640 1 -- 172.21.15.45:0/4053981417 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(115..115 src has 1..115) v4 ==== 10662+0+0 (secure 0 0 0) 0x7f0c6009f510 con 0x564d95b26c00 2024-03-28T17:57:35.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.949+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/4053981417 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0c6801cb00 con 0x564d95b26c00 2024-03-28T17:57:35.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.949+0000 7f0c5a000640 1 -- 172.21.15.45:0/4053981417 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=186) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c6009b8d0 con 0x564d95b26c00 2024-03-28T17:57:35.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.949+0000 7f0c7e3d5740 10 client.26273 fetch_fsmap learned FSMap version 186 2024-03-28T17:57:35.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.949+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/4053981417 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=186,monmap=4+}) v3 -- 0x564d9579e3f0 con 0x564d95b26c00 2024-03-28T17:57:35.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.950+0000 7f0c5a000640 1 -- 172.21.15.45:0/4053981417 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c6801cb00 con 0x564d95b26c00 2024-03-28T17:57:35.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.950+0000 7f0c5a000640 1 -- 172.21.15.45:0/4053981417 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 186) v1 ==== 5618+0+0 (secure 0 0 0) 0x7f0c60018b30 con 0x564d95b26c00 2024-03-28T17:57:35.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.950+0000 7f0c7e3d5740 10 client.26273 fetch_fsmap finished waiting for FSMap version 186 2024-03-28T17:57:35.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.950+0000 7f0c7e3d5740 10 client.26273 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:57:35.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.950+0000 7f0c7e3d5740 20 client.26273 populate_metadata read hostname 'smithi045' 2024-03-28T17:57:35.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.950+0000 7f0c7e3d5740 1 --2- 172.21.15.45:0/4053981417 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x564d95b3f4a0 0x564d95b418c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:35.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.950+0000 7f0c7e3d5740 4 client.26273 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:57:35.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.950+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/4053981417 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564d95a97790 con 0x564d95b3f4a0 2024-03-28T17:57:35.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.950+0000 7f0c78c00640 1 --2- 172.21.15.45:0/4053981417 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x564d95b3f4a0 0x564d95b418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:35.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.951+0000 7f0c78c00640 1 --2- 172.21.15.45:0/4053981417 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x564d95b3f4a0 0x564d95b418c0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bc81c058e30a0e4 server_cookie=18c2d2d05a31fcd2 in_seq=0 out_seq=0 2024-03-28T17:57:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.951+0000 7f0c5a000640 10 client.26273 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:57:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.951+0000 7f0c5a000640 1 -- 172.21.15.45:0/4053981417 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564d95a97790 con 0x564d95b3f4a0 2024-03-28T17:57:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.951+0000 7f0c5a000640 10 client.26273 handle_command_reply: tid=0 2024-03-28T17:57:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.951+0000 7f0c7e3d5740 1 client.26273 shutdown 2024-03-28T17:57:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.952+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/4053981417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b26c00 msgr2=0x564d95b2bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:57:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.952+0000 7f0c7e3d5740 1 --2- 172.21.15.45:0/4053981417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b26c00 0x564d95b2bce0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f0c6009b5d0 tx=0x7f0c600d3160 comp rx=0 tx=0).stop 2024-03-28T17:57:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.952+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/4053981417 shutdown_connections 2024-03-28T17:57:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.952+0000 7f0c7e3d5740 1 --2- 172.21.15.45:0/4053981417 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x564d95b3f4a0 0x564d95b418c0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.952+0000 7f0c7e3d5740 1 --2- 172.21.15.45:0/4053981417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564d95b2f290 0x564d95b2c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.952+0000 7f0c7e3d5740 1 --2- 172.21.15.45:0/4053981417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b26c00 0x564d95b2bce0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.952+0000 7f0c7e3d5740 1 --2- 172.21.15.45:0/4053981417 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x564d95b25f20 0x564d95b2e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.952+0000 7f0c5a000640 0 client.26273 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:57:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.952+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/4053981417 >> 172.21.15.45:0/4053981417 conn(0x564d95aed200 msgr2=0x564d95b226c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:57:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.952+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/4053981417 shutdown_connections 2024-03-28T17:57:35.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.952+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/4053981417 wait complete. 2024-03-28T17:57:35.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.952+0000 7f0c7e3d5740 20 client.26273 trim_cache size 0 max 16384 2024-03-28T17:57:35.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.970+0000 7f0c7e3d5740 1 Processor -- start 2024-03-28T17:57:35.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.970+0000 7f0c7e3d5740 1 -- start start 2024-03-28T17:57:35.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.970+0000 7f0c7e3d5740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x564d95b25f20 0x564d95b30680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:35.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.970+0000 7f0c7e3d5740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b26c00 0x564d95b30bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:35.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.970+0000 7f0c7e3d5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564d95b2f290 0x564d95b204a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:35.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.970+0000 7f0c7e3d5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c741b0090 con 0x564d95b2f290 2024-03-28T17:57:35.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.970+0000 7f0c7e3d5740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c741b1a10 con 0x564d95b26c00 2024-03-28T17:57:35.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.970+0000 7f0c7e3d5740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c741b06f0 con 0x564d95b25f20 2024-03-28T17:57:35.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.971+0000 7f0c73e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b26c00 0x564d95b30bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:35.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.971+0000 7f0c73e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b26c00 0x564d95b30bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:43930/0 (socket says 172.21.15.45:43930) 2024-03-28T17:57:35.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.971+0000 7f0c73e00640 1 -- 172.21.15.45:0/1069761220 learned_addr learned my addr 172.21.15.45:0/1069761220 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:57:35.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.971+0000 7f0c79600640 1 --2- 172.21.15.45:0/1069761220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564d95b2f290 0x564d95b204a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:35.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.971+0000 7f0c78c00640 1 --2- 172.21.15.45:0/1069761220 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x564d95b25f20 0x564d95b30680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:35.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.971+0000 7f0c73e00640 1 -- 172.21.15.45:0/1069761220 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x564d95b25f20 msgr2=0x564d95b30680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:57:35.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.971+0000 7f0c73e00640 1 --2- 172.21.15.45:0/1069761220 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x564d95b25f20 0x564d95b30680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.971+0000 7f0c73e00640 1 -- 172.21.15.45:0/1069761220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564d95b2f290 msgr2=0x564d95b204a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:57:35.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.971+0000 7f0c73e00640 1 --2- 172.21.15.45:0/1069761220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564d95b2f290 0x564d95b204a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.971+0000 7f0c73e00640 1 -- 172.21.15.45:0/1069761220 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d9575e0f0 con 0x564d95b26c00 2024-03-28T17:57:35.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.971+0000 7f0c78c00640 1 --2- 172.21.15.45:0/1069761220 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x564d95b25f20 0x564d95b30680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T17:57:35.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.971+0000 7f0c73e00640 1 --2- 172.21.15.45:0/1069761220 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b26c00 0x564d95b30bc0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f0c600ccf10 tx=0x7f0c6000c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15560754309eb377 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:57:35.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.972+0000 7f0c65200640 1 -- 172.21.15.45:0/1069761220 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c600ba070 con 0x564d95b26c00 2024-03-28T17:57:35.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.972+0000 7f0c65200640 1 -- 172.21.15.45:0/1069761220 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c600d8070 con 0x564d95b26c00 2024-03-28T17:57:35.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.972+0000 7f0c65200640 1 -- 172.21.15.45:0/1069761220 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6009ad70 con 0x564d95b26c00 2024-03-28T17:57:35.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.972+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/1069761220 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b26c00 msgr2=0x564d95b30bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:57:35.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.972+0000 7f0c7e3d5740 1 --2- 172.21.15.45:0/1069761220 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b26c00 0x564d95b30bc0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f0c600ccf10 tx=0x7f0c6000c7d0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.972+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/1069761220 shutdown_connections 2024-03-28T17:57:35.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.972+0000 7f0c7e3d5740 1 --2- 172.21.15.45:0/1069761220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564d95b2f290 0x564d95b204a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.972+0000 7f0c7e3d5740 1 --2- 172.21.15.45:0/1069761220 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b26c00 0x564d95b30bc0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.972+0000 7f0c7e3d5740 1 --2- 172.21.15.45:0/1069761220 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x564d95b25f20 0x564d95b30680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.972+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/1069761220 >> 172.21.15.45:0/1069761220 conn(0x564d95aed200 msgr2=0x564d95b2bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:57:35.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.973+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/1069761220 shutdown_connections 2024-03-28T17:57:35.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.973+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/1069761220 wait complete. 2024-03-28T17:57:35.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.973+0000 7f0c7e3d5740 1 -- start start 2024-03-28T17:57:35.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.973+0000 7f0c7e3d5740 1 Processor -- start 2024-03-28T17:57:35.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.974+0000 7f0c7e3d5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564d95b25f20 0x564d95b28550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:35.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.974+0000 7f0c7e3d5740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x564d95b26c00 0x564d95b28a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:35.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.974+0000 7f0c7e3d5740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b2f290 0x564d95b28fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:35.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.974+0000 7f0c7e3d5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c741b13b0 con 0x564d95b25f20 2024-03-28T17:57:35.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.974+0000 7f0c7e3d5740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c741b0d50 con 0x564d95b2f290 2024-03-28T17:57:35.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.974+0000 7f0c7e3d5740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c741ae9d0 con 0x564d95b26c00 2024-03-28T17:57:35.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.974+0000 7f0c7e3d5740 10 client.0 _ll_register_callbacks cb 0x7f0c600d3260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:57:35.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.974+0000 7f0c79600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b2f290 0x564d95b28fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:35.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.974+0000 7f0c79600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b2f290 0x564d95b28fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:43932/0 (socket says 172.21.15.45:43932) 2024-03-28T17:57:35.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.974+0000 7f0c79600640 1 -- 172.21.15.45:0/1727730727 learned_addr learned my addr 172.21.15.45:0/1727730727 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:57:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.974+0000 7f0c78c00640 1 --2- 172.21.15.45:0/1727730727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564d95b25f20 0x564d95b28550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.974+0000 7f0c73e00640 1 --2- 172.21.15.45:0/1727730727 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x564d95b26c00 0x564d95b28a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.974+0000 7f0c79600640 1 -- 172.21.15.45:0/1727730727 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x564d95b26c00 msgr2=0x564d95b28a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:57:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.974+0000 7f0c79600640 1 --2- 172.21.15.45:0/1727730727 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x564d95b26c00 0x564d95b28a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.974+0000 7f0c79600640 1 -- 172.21.15.45:0/1727730727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564d95b25f20 msgr2=0x564d95b28550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:57:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.974+0000 7f0c79600640 1 --2- 172.21.15.45:0/1727730727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564d95b25f20 0x564d95b28550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.974+0000 7f0c79600640 1 -- 172.21.15.45:0/1727730727 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564d95b99f70 con 0x564d95b2f290 2024-03-28T17:57:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.974+0000 7f0c73e00640 1 --2- 172.21.15.45:0/1727730727 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x564d95b26c00 0x564d95b28a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:57:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.975+0000 7f0c79600640 1 --2- 172.21.15.45:0/1727730727 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b2f290 0x564d95b28fd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c02c710 tx=0x7f0c6c02c100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0eba26598399418 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:57:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.975+0000 7f0c5a000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:57:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.975+0000 7f0c5a000640 1 -- 172.21.15.45:0/1727730727 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c054070 con 0x564d95b2f290 2024-03-28T17:57:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.975+0000 7f0c5a000640 1 -- 172.21.15.45:0/1727730727 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(115..115 src has 1..115) v4 ==== 10662+0+0 (secure 0 0 0) 0x7f0c6c0040c0 con 0x564d95b2f290 2024-03-28T17:57:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.975+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/1727730727 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0c600d8070 con 0x564d95b2f290 2024-03-28T17:57:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.975+0000 7f0c5a000640 1 -- 172.21.15.45:0/1727730727 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=186) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c6c039040 con 0x564d95b2f290 2024-03-28T17:57:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.975+0000 7f0c7e3d5740 10 client.26279 fetch_fsmap learned FSMap version 186 2024-03-28T17:57:35.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.975+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/1727730727 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=186,monmap=4+}) v3 -- 0x564d95b9b170 con 0x564d95b2f290 2024-03-28T17:57:35.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.976+0000 7f0c5a000640 1 -- 172.21.15.45:0/1727730727 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c600d8070 con 0x564d95b2f290 2024-03-28T17:57:35.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.976+0000 7f0c5a000640 1 -- 172.21.15.45:0/1727730727 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 186) v1 ==== 5618+0+0 (secure 0 0 0) 0x7f0c6c03f040 con 0x564d95b2f290 2024-03-28T17:57:35.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.976+0000 7f0c7e3d5740 10 client.26279 fetch_fsmap finished waiting for FSMap version 186 2024-03-28T17:57:35.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.976+0000 7f0c7e3d5740 10 client.26279 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:57:35.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.976+0000 7f0c7e3d5740 20 client.26279 populate_metadata read hostname 'smithi045' 2024-03-28T17:57:35.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.976+0000 7f0c7e3d5740 1 --2- 172.21.15.45:0/1727730727 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x564d95b9b870 0x564d95b9dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:57:35.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.976+0000 7f0c7e3d5740 4 client.26279 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:57:35.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.976+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/1727730727 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564d95a91ae0 con 0x564d95b9b870 2024-03-28T17:57:35.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.976+0000 7f0c78c00640 1 --2- 172.21.15.45:0/1727730727 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x564d95b9b870 0x564d95b9dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:57:35.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.976+0000 7f0c78c00640 1 --2- 172.21.15.45:0/1727730727 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x564d95b9b870 0x564d95b9dc90 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5db672b773a7084 server_cookie=8e5c53ea65a4a58 in_seq=0 out_seq=0 2024-03-28T17:57:35.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.976+0000 7f0c5a000640 10 client.26279 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:57:35.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.977+0000 7f0c5a000640 1 -- 172.21.15.45:0/1727730727 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x564d95a91ae0 con 0x564d95b9b870 2024-03-28T17:57:35.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.977+0000 7f0c5a000640 10 client.26279 handle_command_reply: tid=0 2024-03-28T17:57:35.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.977+0000 7f0c7e3d5740 1 client.26279 shutdown 2024-03-28T17:57:35.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.977+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/1727730727 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b2f290 msgr2=0x564d95b28fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:57:35.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.977+0000 7f0c7e3d5740 1 --2- 172.21.15.45:0/1727730727 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b2f290 0x564d95b28fd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c02c710 tx=0x7f0c6c02c100 comp rx=0 tx=0).stop 2024-03-28T17:57:35.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.977+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/1727730727 shutdown_connections 2024-03-28T17:57:35.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.977+0000 7f0c7e3d5740 1 --2- 172.21.15.45:0/1727730727 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x564d95b9b870 0x564d95b9dc90 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.977+0000 7f0c7e3d5740 1 --2- 172.21.15.45:0/1727730727 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564d95b2f290 0x564d95b28fd0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.977+0000 7f0c7e3d5740 1 --2- 172.21.15.45:0/1727730727 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x564d95b26c00 0x564d95b28a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.977+0000 7f0c7e3d5740 1 --2- 172.21.15.45:0/1727730727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564d95b25f20 0x564d95b28550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.977+0000 7f0c5a000640 0 client.26279 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:57:35.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.977+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/1727730727 >> 172.21.15.45:0/1727730727 conn(0x564d95aed200 msgr2=0x564d95b204a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:57:35.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.977+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/1727730727 shutdown_connections 2024-03-28T17:57:35.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.977+0000 7f0c7e3d5740 1 -- 172.21.15.45:0/1727730727 wait complete. 2024-03-28T17:57:35.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.977+0000 7f0c7e3d5740 20 client.26279 trim_cache size 0 max 16384 2024-03-28T17:57:35.982 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:57:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T17:57:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T17:57:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T17:57:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T17:57:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T17:57:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T17:57:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:57:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:57:35.982 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:57:35.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.978+0000 7f0c7b600640 1 -- 172.21.15.45:0/2977555146 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f0c5c0a2970 msgr2=0x7f0c5c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:57:35.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.978+0000 7f0c7b600640 1 --2- 172.21.15.45:0/2977555146 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f0c5c0a2970 0x7f0c5c0a4e30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c005e30 tx=0x7f0c6c005d80 comp rx=0 tx=0).stop 2024-03-28T17:57:35.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.978+0000 7f0c7b600640 1 -- 172.21.15.45:0/2977555146 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c74101ff0 msgr2=0x7f0c74073ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:57:35.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.978+0000 7f0c7b600640 1 --2- 172.21.15.45:0/2977555146 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c74101ff0 0x7f0c74073ee0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0c6000e9f0 tx=0x7f0c6000eeb0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.978+0000 7f0c7b600640 1 -- 172.21.15.45:0/2977555146 shutdown_connections 2024-03-28T17:57:35.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.978+0000 7f0c7b600640 1 --2- 172.21.15.45:0/2977555146 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f0c5c0a2970 0x7f0c5c0a4e30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.978+0000 7f0c7b600640 1 --2- 172.21.15.45:0/2977555146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c74107640 0x7f0c74074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.978+0000 7f0c7b600640 1 --2- 172.21.15.45:0/2977555146 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c74101ff0 0x7f0c74073ee0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.978+0000 7f0c7b600640 1 --2- 172.21.15.45:0/2977555146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0c74101630 0x7f0c740739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:57:35.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.978+0000 7f0c7b600640 1 -- 172.21.15.45:0/2977555146 >> 172.21.15.45:0/2977555146 conn(0x7f0c74078aa0 msgr2=0x7f0c74103f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:57:35.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.978+0000 7f0c7b600640 1 -- 172.21.15.45:0/2977555146 shutdown_connections 2024-03-28T17:57:35.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:57:35.978+0000 7f0c7b600640 1 -- 172.21.15.45:0/2977555146 wait complete. 2024-03-28T17:57:35.995 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T17:57:36.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:36 smithi071 ceph-mon[36597]: pgmap v288: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-28T17:57:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:36 smithi151 ceph-mon[36481]: pgmap v288: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-28T17:57:36.893 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 309 sec latency 424790.123 ms 2024-03-28T17:57:36.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:36 smithi045 ceph-mon[31773]: pgmap v288: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-28T17:57:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:37 smithi071 ceph-mon[36597]: osdmap e116: 12 total, 12 up, 12 in 2024-03-28T17:57:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:37 smithi151 ceph-mon[36481]: osdmap e116: 12 total, 12 up, 12 in 2024-03-28T17:57:37.893 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 310 sec latency 425790.284 ms 2024-03-28T17:57:37.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:37 smithi045 ceph-mon[31773]: osdmap e116: 12 total, 12 up, 12 in 2024-03-28T17:57:38.321 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:37 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Scheduled restart job, restart counter is at 1. 2024-03-28T17:57:38.322 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:37 smithi045 systemd[1]: Stopped Ceph mds.a for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:57:38.322 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:37 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Consumed 1.975s CPU time. 2024-03-28T17:57:38.322 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:37 smithi045 systemd[1]: Starting Ceph mds.a for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:57:38.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:38 smithi045 ceph-mon[31773]: pgmap v290: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:57:38.666 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:38 smithi045 podman[92171]: 2024-03-28 17:57:38.228481751 +0000 UTC m=+0.037453787 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:57:38.666 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:38 smithi045 podman[92171]: 2024-03-28 17:57:38.35664793 +0000 UTC m=+0.165619960 container create 7a4a474873405a3304413de36cb9de7360e4903d87856ffb0f4ed4ba30be7054 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T17:57:38.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:38 smithi071 ceph-mon[36597]: pgmap v290: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:57:38.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:38 smithi151 ceph-mon[36481]: pgmap v290: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:57:38.893 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 311 sec latency 426790.427 ms 2024-03-28T17:57:39.166 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:38 smithi045 podman[92171]: 2024-03-28 17:57:38.674779707 +0000 UTC m=+0.483751744 container init 7a4a474873405a3304413de36cb9de7360e4903d87856ffb0f4ed4ba30be7054 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T17:57:39.166 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:38 smithi045 podman[92171]: 2024-03-28 17:57:38.6885372 +0000 UTC m=+0.497509236 container start 7a4a474873405a3304413de36cb9de7360e4903d87856ffb0f4ed4ba30be7054 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T17:57:39.166 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:38 smithi045 ceph-mds[92190]: -- 172.21.15.45:0/3773982348 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558206609a40 con 0x5582065dd000 2024-03-28T17:57:39.166 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:38 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: starting mds.a at 2024-03-28T17:57:39.166 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:38 smithi045 bash[92171]: 7a4a474873405a3304413de36cb9de7360e4903d87856ffb0f4ed4ba30be7054 2024-03-28T17:57:39.166 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:38 smithi045 systemd[1]: Started Ceph mds.a for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:57:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:39 smithi071 ceph-mon[36597]: daemon mds.a restarted 2024-03-28T17:57:39.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:39 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:57:39.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:39 smithi071 ceph-mon[36597]: osdmap e117: 12 total, 12 up, 12 in 2024-03-28T17:57:39.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:39 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2043640256,v1:172.21.15.45:6837/2043640256] up:boot 2024-03-28T17:57:39.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:39 smithi071 ceph-mon[36597]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-03-28T17:57:39.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:39 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:57:39.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:39 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T17:57:39.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:39 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:39 smithi151 ceph-mon[36481]: daemon mds.a restarted 2024-03-28T17:57:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:39 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:57:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:39 smithi151 ceph-mon[36481]: osdmap e117: 12 total, 12 up, 12 in 2024-03-28T17:57:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:39 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2043640256,v1:172.21.15.45:6837/2043640256] up:boot 2024-03-28T17:57:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:39 smithi151 ceph-mon[36481]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-03-28T17:57:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:39 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:57:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:39 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T17:57:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:39 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:39.893 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 312 sec latency 427790.579 ms 2024-03-28T17:57:39.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:39 smithi045 ceph-mon[31773]: daemon mds.a restarted 2024-03-28T17:57:39.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:39 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:57:39.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:39 smithi045 ceph-mon[31773]: osdmap e117: 12 total, 12 up, 12 in 2024-03-28T17:57:39.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:39 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2043640256,v1:172.21.15.45:6837/2043640256] up:boot 2024-03-28T17:57:39.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:39 smithi045 ceph-mon[31773]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-03-28T17:57:39.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:39 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:57:39.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:39 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T17:57:39.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:39 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:40.894 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 313 sec latency 428790.741 ms 2024-03-28T17:57:41.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:40 smithi071 ceph-mon[36597]: pgmap v292: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T17:57:41.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:40 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/775549656,v1:172.21.15.151:6833/775549656] up:resolve 2024-03-28T17:57:41.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:40 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:41.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:40 smithi151 ceph-mon[36481]: pgmap v292: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T17:57:41.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:40 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/775549656,v1:172.21.15.151:6833/775549656] up:resolve 2024-03-28T17:57:41.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:40 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:41.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:40 smithi045 ceph-mon[31773]: pgmap v292: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T17:57:41.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:40 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/775549656,v1:172.21.15.151:6833/775549656] up:resolve 2024-03-28T17:57:41.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:40 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:41.894 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 314 sec latency 429790.930 ms 2024-03-28T17:57:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:41 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:57:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:41 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/775549656,v1:172.21.15.151:6833/775549656] up:reconnect 2024-03-28T17:57:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:41 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:41 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.0109999 2024-03-28T17:57:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:41 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:57:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:41 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/775549656,v1:172.21.15.151:6833/775549656] up:reconnect 2024-03-28T17:57:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:41 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:41 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.0109999 2024-03-28T17:57:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:41 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:57:42.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:41 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/775549656,v1:172.21.15.151:6833/775549656] up:reconnect 2024-03-28T17:57:42.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:41 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:42.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:41 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.0109999 2024-03-28T17:57:42.894 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 315 sec latency 430791.062 ms 2024-03-28T17:57:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:42 smithi071 ceph-mon[36597]: pgmap v293: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T17:57:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:42 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/775549656,v1:172.21.15.151:6833/775549656] up:rejoin 2024-03-28T17:57:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:42 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:42 smithi071 ceph-mon[36597]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-03-28T17:57:43.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-mon[36481]: pgmap v293: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T17:57:43.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/775549656,v1:172.21.15.151:6833/775549656] up:rejoin 2024-03-28T17:57:43.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:43.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-mon[36481]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-03-28T17:57:43.074 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3fdf3bd700 time 2024-03-28T17:57:42.805458+0000 2024-03-28T17:57:43.074 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:57:43.074 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:43.074 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3fe99e74a1] 2024-03-28T17:57:43.074 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3fe99e7667] 2024-03-28T17:57:43.074 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ea54e72223] 2024-03-28T17:57:43.074 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ea54cec5d6] 2024-03-28T17:57:43.074 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ea54cecda1] 2024-03-28T17:57:43.074 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ea54cef2d5] 2024-03-28T17:57:43.074 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ea54d3f233] 2024-03-28T17:57:43.075 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ea54d41197] 2024-03-28T17:57:43.075 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ea54c4011b] 2024-03-28T17:57:43.075 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ea54c406bc] 2024-03-28T17:57:43.075 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ea54c2a1ff] 2024-03-28T17:57:43.075 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3fe9c79488] 2024-03-28T17:57:43.075 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 13: (DispatchQueue::entry()+0x51f) [0x7f3fe9c7660f] 2024-03-28T17:57:43.075 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3fe9d49e81] 2024-03-28T17:57:43.075 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3fe87641ca] 2024-03-28T17:57:43.075 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 16: clone() 2024-03-28T17:57:43.075 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 2024-03-28T17:57:42.806+0000 7f3fdf3bd700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3fdf3bd700 time 2024-03-28T17:57:42.805458+0000 2024-03-28T17:57:43.075 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:57:43.075 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 2024-03-28T17:57:43.075 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:43.076 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3fe99e74a1] 2024-03-28T17:57:43.076 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3fe99e7667] 2024-03-28T17:57:43.076 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ea54e72223] 2024-03-28T17:57:43.076 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ea54cec5d6] 2024-03-28T17:57:43.076 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ea54cecda1] 2024-03-28T17:57:43.076 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ea54cef2d5] 2024-03-28T17:57:43.076 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ea54d3f233] 2024-03-28T17:57:43.076 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ea54d41197] 2024-03-28T17:57:43.076 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ea54c4011b] 2024-03-28T17:57:43.076 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ea54c406bc] 2024-03-28T17:57:43.076 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ea54c2a1ff] 2024-03-28T17:57:43.076 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3fe9c79488] 2024-03-28T17:57:43.076 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 13: (DispatchQueue::entry()+0x51f) [0x7f3fe9c7660f] 2024-03-28T17:57:43.076 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3fe9d49e81] 2024-03-28T17:57:43.077 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3fe87641ca] 2024-03-28T17:57:43.077 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 16: clone() 2024-03-28T17:57:43.077 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 2024-03-28T17:57:43.077 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: *** Caught signal (Aborted) ** 2024-03-28T17:57:43.077 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: in thread 7f3fdf3bd700 thread_name:ms_dispatch 2024-03-28T17:57:43.077 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:43.077 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3fe876ed20] 2024-03-28T17:57:43.078 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 2: gsignal() 2024-03-28T17:57:43.078 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 3: abort() 2024-03-28T17:57:43.078 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3fe99e74fb] 2024-03-28T17:57:43.078 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3fe99e7667] 2024-03-28T17:57:43.078 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ea54e72223] 2024-03-28T17:57:43.078 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ea54cec5d6] 2024-03-28T17:57:43.078 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ea54cecda1] 2024-03-28T17:57:43.079 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ea54cef2d5] 2024-03-28T17:57:43.079 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ea54d3f233] 2024-03-28T17:57:43.079 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ea54d41197] 2024-03-28T17:57:43.079 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ea54c4011b] 2024-03-28T17:57:43.079 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ea54c406bc] 2024-03-28T17:57:43.079 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ea54c2a1ff] 2024-03-28T17:57:43.079 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3fe9c79488] 2024-03-28T17:57:43.079 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 16: (DispatchQueue::entry()+0x51f) [0x7f3fe9c7660f] 2024-03-28T17:57:43.079 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3fe9d49e81] 2024-03-28T17:57:43.079 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3fe87641ca] 2024-03-28T17:57:43.079 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 19: clone() 2024-03-28T17:57:43.079 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 2024-03-28T17:57:42.808+0000 7f3fdf3bd700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:57:43.079 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: in thread 7f3fdf3bd700 thread_name:ms_dispatch 2024-03-28T17:57:43.080 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 2024-03-28T17:57:43.080 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:43.080 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3fe876ed20] 2024-03-28T17:57:43.080 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 2: gsignal() 2024-03-28T17:57:43.080 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 3: abort() 2024-03-28T17:57:43.080 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3fe99e74fb] 2024-03-28T17:57:43.080 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3fe99e7667] 2024-03-28T17:57:43.080 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ea54e72223] 2024-03-28T17:57:43.080 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ea54cec5d6] 2024-03-28T17:57:43.080 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ea54cecda1] 2024-03-28T17:57:43.080 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ea54cef2d5] 2024-03-28T17:57:43.080 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ea54d3f233] 2024-03-28T17:57:43.080 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ea54d41197] 2024-03-28T17:57:43.080 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ea54c4011b] 2024-03-28T17:57:43.080 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ea54c406bc] 2024-03-28T17:57:43.080 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ea54c2a1ff] 2024-03-28T17:57:43.081 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3fe9c79488] 2024-03-28T17:57:43.081 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 16: (DispatchQueue::entry()+0x51f) [0x7f3fe9c7660f] 2024-03-28T17:57:43.081 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3fe9d49e81] 2024-03-28T17:57:43.081 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3fe87641ca] 2024-03-28T17:57:43.081 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 19: clone() 2024-03-28T17:57:43.081 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:57:43.081 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 2024-03-28T17:57:43.081 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: -1> 2024-03-28T17:57:42.806+0000 7f3fdf3bd700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3fdf3bd700 time 2024-03-28T17:57:42.805458+0000 2024-03-28T17:57:43.081 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:57:43.081 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 2024-03-28T17:57:43.081 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:43.081 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3fe99e74a1] 2024-03-28T17:57:43.081 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3fe99e7667] 2024-03-28T17:57:43.081 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ea54e72223] 2024-03-28T17:57:43.081 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ea54cec5d6] 2024-03-28T17:57:43.082 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ea54cecda1] 2024-03-28T17:57:43.082 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ea54cef2d5] 2024-03-28T17:57:43.082 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ea54d3f233] 2024-03-28T17:57:43.082 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ea54d41197] 2024-03-28T17:57:43.082 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ea54c4011b] 2024-03-28T17:57:43.082 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ea54c406bc] 2024-03-28T17:57:43.082 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ea54c2a1ff] 2024-03-28T17:57:43.082 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3fe9c79488] 2024-03-28T17:57:43.082 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 13: (DispatchQueue::entry()+0x51f) [0x7f3fe9c7660f] 2024-03-28T17:57:43.082 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3fe9d49e81] 2024-03-28T17:57:43.082 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3fe87641ca] 2024-03-28T17:57:43.082 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 16: clone() 2024-03-28T17:57:43.082 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 2024-03-28T17:57:43.082 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 0> 2024-03-28T17:57:42.808+0000 7f3fdf3bd700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:57:43.082 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: in thread 7f3fdf3bd700 thread_name:ms_dispatch 2024-03-28T17:57:43.083 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 2024-03-28T17:57:43.083 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:43.083 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3fe876ed20] 2024-03-28T17:57:43.083 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 2: gsignal() 2024-03-28T17:57:43.083 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 3: abort() 2024-03-28T17:57:43.083 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3fe99e74fb] 2024-03-28T17:57:43.083 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3fe99e7667] 2024-03-28T17:57:43.083 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ea54e72223] 2024-03-28T17:57:43.083 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ea54cec5d6] 2024-03-28T17:57:43.083 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ea54cecda1] 2024-03-28T17:57:43.083 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ea54cef2d5] 2024-03-28T17:57:43.083 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ea54d3f233] 2024-03-28T17:57:43.083 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ea54d41197] 2024-03-28T17:57:43.083 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ea54c4011b] 2024-03-28T17:57:43.083 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ea54c406bc] 2024-03-28T17:57:43.084 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ea54c2a1ff] 2024-03-28T17:57:43.084 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3fe9c79488] 2024-03-28T17:57:43.084 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 16: (DispatchQueue::entry()+0x51f) [0x7f3fe9c7660f] 2024-03-28T17:57:43.084 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3fe9d49e81] 2024-03-28T17:57:43.084 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3fe87641ca] 2024-03-28T17:57:43.084 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 19: clone() 2024-03-28T17:57:43.084 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:57:43.084 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 2024-03-28T17:57:43.084 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: -1> 2024-03-28T17:57:42.806+0000 7f3fdf3bd700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3fdf3bd700 time 2024-03-28T17:57:42.805458+0000 2024-03-28T17:57:43.084 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:57:43.084 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 2024-03-28T17:57:43.084 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:43.084 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3fe99e74a1] 2024-03-28T17:57:43.084 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3fe99e7667] 2024-03-28T17:57:43.084 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ea54e72223] 2024-03-28T17:57:43.084 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ea54cec5d6] 2024-03-28T17:57:43.085 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ea54cecda1] 2024-03-28T17:57:43.085 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ea54cef2d5] 2024-03-28T17:57:43.085 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ea54d3f233] 2024-03-28T17:57:43.085 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ea54d41197] 2024-03-28T17:57:43.085 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ea54c4011b] 2024-03-28T17:57:43.085 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ea54c406bc] 2024-03-28T17:57:43.085 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ea54c2a1ff] 2024-03-28T17:57:43.085 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3fe9c79488] 2024-03-28T17:57:43.085 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 13: (DispatchQueue::entry()+0x51f) [0x7f3fe9c7660f] 2024-03-28T17:57:43.085 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3fe9d49e81] 2024-03-28T17:57:43.085 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3fe87641ca] 2024-03-28T17:57:43.085 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 16: clone() 2024-03-28T17:57:43.085 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 2024-03-28T17:57:43.085 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 0> 2024-03-28T17:57:42.808+0000 7f3fdf3bd700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:57:43.085 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: in thread 7f3fdf3bd700 thread_name:ms_dispatch 2024-03-28T17:57:43.086 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 2024-03-28T17:57:43.086 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:43.086 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3fe876ed20] 2024-03-28T17:57:43.086 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 2: gsignal() 2024-03-28T17:57:43.086 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 3: abort() 2024-03-28T17:57:43.086 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3fe99e74fb] 2024-03-28T17:57:43.086 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3fe99e7667] 2024-03-28T17:57:43.086 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ea54e72223] 2024-03-28T17:57:43.086 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ea54cec5d6] 2024-03-28T17:57:43.086 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ea54cecda1] 2024-03-28T17:57:43.086 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ea54cef2d5] 2024-03-28T17:57:43.086 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ea54d3f233] 2024-03-28T17:57:43.086 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ea54d41197] 2024-03-28T17:57:43.086 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ea54c4011b] 2024-03-28T17:57:43.086 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ea54c406bc] 2024-03-28T17:57:43.087 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ea54c2a1ff] 2024-03-28T17:57:43.087 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3fe9c79488] 2024-03-28T17:57:43.087 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 16: (DispatchQueue::entry()+0x51f) [0x7f3fe9c7660f] 2024-03-28T17:57:43.087 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3fe9d49e81] 2024-03-28T17:57:43.087 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3fe87641ca] 2024-03-28T17:57:43.087 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 19: clone() 2024-03-28T17:57:43.087 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:57:43.087 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73521]: 2024-03-28T17:57:43.087 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:42 smithi151 podman[73563]: 2024-03-28 17:57:42.862426356 +0000 UTC m=+0.020578759 container died b41f49335965d3b4bc673195af25b925da440c6803717f9a302a98155e31b8c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T17:57:43.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:42 smithi045 ceph-mon[31773]: pgmap v293: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T17:57:43.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:42 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/775549656,v1:172.21.15.151:6833/775549656] up:rejoin 2024-03-28T17:57:43.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:42 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:43.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:42 smithi045 ceph-mon[31773]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-03-28T17:57:43.877 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:43 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:57:43.877 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:43 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:57:43.877 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:43 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/775549656,v1:172.21.15.151:6833/775549656] up:active 2024-03-28T17:57:43.877 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:43 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:43.877 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:43 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:57:43.877 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:43 smithi151 podman[73563]: 2024-03-28 17:57:43.625131472 +0000 UTC m=+0.783283906 container remove b41f49335965d3b4bc673195af25b925da440c6803717f9a302a98155e31b8c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-28T17:57:43.877 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:43 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:57:43.894 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 316 sec latency 431791.154 ms 2024-03-28T17:57:44.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:43 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:57:44.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:43 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:57:44.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:43 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/775549656,v1:172.21.15.151:6833/775549656] up:active 2024-03-28T17:57:44.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:43 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:44.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:43 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:57:44.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:43 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:57:44.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:43 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:57:44.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:43 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/775549656,v1:172.21.15.151:6833/775549656] up:active 2024-03-28T17:57:44.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:43 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:44.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:43 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:57:44.323 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:43 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-28T17:57:44.894 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 317 sec latency 432791.289 ms 2024-03-28T17:57:45.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:44 smithi071 ceph-mon[36597]: pgmap v294: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T17:57:45.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:44 smithi151 ceph-mon[36481]: pgmap v294: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T17:57:45.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:44 smithi045 ceph-mon[31773]: pgmap v294: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T17:57:45.894 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 318 sec latency 433791.396 ms 2024-03-28T17:57:46.894 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 319 sec latency 434791.513 ms 2024-03-28T17:57:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:46 smithi151 ceph-mon[36481]: pgmap v295: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 476 B/s wr, 13 op/s 2024-03-28T17:57:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:46 smithi151 ceph-mon[36481]: osdmap e118: 12 total, 12 up, 12 in 2024-03-28T17:57:47.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:46 smithi045 ceph-mon[31773]: pgmap v295: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 476 B/s wr, 13 op/s 2024-03-28T17:57:47.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:46 smithi045 ceph-mon[31773]: osdmap e118: 12 total, 12 up, 12 in 2024-03-28T17:57:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:46 smithi071 ceph-mon[36597]: pgmap v295: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 476 B/s wr, 13 op/s 2024-03-28T17:57:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:46 smithi071 ceph-mon[36597]: osdmap e118: 12 total, 12 up, 12 in 2024-03-28T17:57:47.894 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 320 sec latency 435791.650 ms 2024-03-28T17:57:48.895 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 321 sec latency 436791.801 ms 2024-03-28T17:57:49.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:48 smithi151 ceph-mon[36481]: pgmap v297: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 489 B/s wr, 13 op/s 2024-03-28T17:57:49.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:48 smithi045 ceph-mon[31773]: pgmap v297: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 489 B/s wr, 13 op/s 2024-03-28T17:57:49.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:48 smithi071 ceph-mon[36597]: pgmap v297: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 489 B/s wr, 13 op/s 2024-03-28T17:57:49.895 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 322 sec latency 437791.932 ms 2024-03-28T17:57:50.895 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 323 sec latency 438792.069 ms 2024-03-28T17:57:51.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:50 smithi045 ceph-mon[31773]: pgmap v298: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 409 B/s wr, 11 op/s 2024-03-28T17:57:51.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:50 smithi071 ceph-mon[36597]: pgmap v298: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 409 B/s wr, 11 op/s 2024-03-28T17:57:51.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:50 smithi151 ceph-mon[36481]: pgmap v298: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 409 B/s wr, 11 op/s 2024-03-28T17:57:51.895 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 324 sec latency 439792.203 ms 2024-03-28T17:57:52.895 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 325 sec latency 440792.323 ms 2024-03-28T17:57:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:52 smithi045 ceph-mon[31773]: pgmap v299: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T17:57:53.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:52 smithi071 ceph-mon[36597]: pgmap v299: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T17:57:53.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:52 smithi151 ceph-mon[36481]: pgmap v299: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T17:57:53.895 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 326 sec latency 441792.453 ms 2024-03-28T17:57:54.343 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:54 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 3. 2024-03-28T17:57:54.343 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:54 smithi151 systemd[1]: Stopped Ceph mds.l for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:57:54.343 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:54 smithi151 systemd[1]: Starting Ceph mds.l for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:57:54.344 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:54 smithi151 podman[73661]: 2024-03-28 17:57:54.211797205 +0000 UTC m=+0.018146972 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:57:54.344 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:54 smithi151 podman[73661]: 2024-03-28 17:57:54.344220303 +0000 UTC m=+0.150570051 container create bbd09cfa544741b1759e51fdceaf817c77d2ecb110929e8187d74b564539e247 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T17:57:54.647 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:54 smithi151 podman[73661]: 2024-03-28 17:57:54.564048176 +0000 UTC m=+0.370397941 container init bbd09cfa544741b1759e51fdceaf817c77d2ecb110929e8187d74b564539e247 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-28T17:57:54.648 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:54 smithi151 podman[73661]: 2024-03-28 17:57:54.566555868 +0000 UTC m=+0.372905622 container start bbd09cfa544741b1759e51fdceaf817c77d2ecb110929e8187d74b564539e247 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T17:57:54.648 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:54 smithi151 ceph-mds[73680]: -- 172.21.15.151:0/610487259 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559e020b9a40 con 0x559e0208d000 2024-03-28T17:57:54.648 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:54 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: starting mds.l at 2024-03-28T17:57:54.896 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 327 sec latency 442792.593 ms 2024-03-28T17:57:55.073 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:54 smithi151 bash[73661]: bbd09cfa544741b1759e51fdceaf817c77d2ecb110929e8187d74b564539e247 2024-03-28T17:57:55.073 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:57:54 smithi151 systemd[1]: Started Ceph mds.l for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:57:55.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:54 smithi151 ceph-mon[36481]: pgmap v300: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T17:57:55.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:54 smithi151 ceph-mon[36481]: daemon mds.l restarted 2024-03-28T17:57:55.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:54 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:57:55.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:54 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T17:57:55.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:54 smithi151 ceph-mon[36481]: osdmap e119: 12 total, 12 up, 12 in 2024-03-28T17:57:55.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:54 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/3219133319,v1:172.21.15.151:6833/3219133319] up:boot 2024-03-28T17:57:55.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:54 smithi151 ceph-mon[36481]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-03-28T17:57:55.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:54 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:57:55.075 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:54 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:55.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:54 smithi045 ceph-mon[31773]: pgmap v300: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T17:57:55.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:54 smithi045 ceph-mon[31773]: daemon mds.l restarted 2024-03-28T17:57:55.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:54 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:57:55.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:54 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T17:57:55.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:54 smithi045 ceph-mon[31773]: osdmap e119: 12 total, 12 up, 12 in 2024-03-28T17:57:55.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:54 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/3219133319,v1:172.21.15.151:6833/3219133319] up:boot 2024-03-28T17:57:55.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:54 smithi045 ceph-mon[31773]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-03-28T17:57:55.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:54 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:57:55.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:54 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:55.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:54 smithi071 ceph-mon[36597]: pgmap v300: 97 pgs: 97 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T17:57:55.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:54 smithi071 ceph-mon[36597]: daemon mds.l restarted 2024-03-28T17:57:55.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:54 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:57:55.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:54 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T17:57:55.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:54 smithi071 ceph-mon[36597]: osdmap e119: 12 total, 12 up, 12 in 2024-03-28T17:57:55.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:54 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/3219133319,v1:172.21.15.151:6833/3219133319] up:boot 2024-03-28T17:57:55.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:54 smithi071 ceph-mon[36597]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-03-28T17:57:55.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:54 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:57:55.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:54 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:55.896 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 328 sec latency 443792.724 ms 2024-03-28T17:57:56.896 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 329 sec latency 444792.861 ms 2024-03-28T17:57:57.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:56 smithi045 ceph-mon[31773]: pgmap v302: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-03-28T17:57:57.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:56 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2043640256,v1:172.21.15.45:6837/2043640256] up:resolve 2024-03-28T17:57:57.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:56 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:56 smithi071 ceph-mon[36597]: pgmap v302: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-03-28T17:57:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:56 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2043640256,v1:172.21.15.45:6837/2043640256] up:resolve 2024-03-28T17:57:57.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:56 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:56 smithi151 ceph-mon[36481]: pgmap v302: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-03-28T17:57:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:56 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2043640256,v1:172.21.15.45:6837/2043640256] up:resolve 2024-03-28T17:57:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:56 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:57.896 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 330 sec latency 445792.960 ms 2024-03-28T17:57:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:57 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:57:58.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:57 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2043640256,v1:172.21.15.45:6837/2043640256] up:reconnect 2024-03-28T17:57:58.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:57 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:58.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:57 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T17:57:58.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:57 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:57:58.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:57 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2043640256,v1:172.21.15.45:6837/2043640256] up:reconnect 2024-03-28T17:57:58.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:57 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:58.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:57 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T17:57:58.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:57 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:57:58.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:57 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2043640256,v1:172.21.15.45:6837/2043640256] up:reconnect 2024-03-28T17:57:58.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:57 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:58.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:57 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T17:57:58.896 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 331 sec latency 446793.061 ms 2024-03-28T17:57:59.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-mon[31773]: pgmap v303: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-28T17:57:59.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2043640256,v1:172.21.15.45:6837/2043640256] up:rejoin 2024-03-28T17:57:59.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:59.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-mon[31773]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-03-28T17:57:59.168 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f805f400700 time 2024-03-28T17:57:58.894945+0000 2024-03-28T17:57:59.168 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:57:59.168 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:59.168 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f806ad2f4a1] 2024-03-28T17:57:59.168 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f806ad2f667] 2024-03-28T17:57:59.168 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558203deb223] 2024-03-28T17:57:59.168 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558203c655d6] 2024-03-28T17:57:59.169 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558203c65da1] 2024-03-28T17:57:59.169 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558203c682d5] 2024-03-28T17:57:59.169 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558203cb8233] 2024-03-28T17:57:59.169 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558203cba197] 2024-03-28T17:57:59.169 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558203bb911b] 2024-03-28T17:57:59.169 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558203bb96bc] 2024-03-28T17:57:59.169 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558203ba31ff] 2024-03-28T17:57:59.169 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f806afc1488] 2024-03-28T17:57:59.169 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 13: (DispatchQueue::entry()+0x51f) [0x7f806afbe60f] 2024-03-28T17:57:59.169 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f806b091e81] 2024-03-28T17:57:59.169 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8069aac1ca] 2024-03-28T17:57:59.169 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 16: clone() 2024-03-28T17:57:59.171 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 2024-03-28T17:57:58.896+0000 7f805f400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f805f400700 time 2024-03-28T17:57:58.894945+0000 2024-03-28T17:57:59.171 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:57:59.171 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 2024-03-28T17:57:59.171 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:59.171 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f806ad2f4a1] 2024-03-28T17:57:59.172 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f806ad2f667] 2024-03-28T17:57:59.172 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558203deb223] 2024-03-28T17:57:59.172 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558203c655d6] 2024-03-28T17:57:59.172 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558203c65da1] 2024-03-28T17:57:59.172 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558203c682d5] 2024-03-28T17:57:59.172 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558203cb8233] 2024-03-28T17:57:59.172 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558203cba197] 2024-03-28T17:57:59.172 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558203bb911b] 2024-03-28T17:57:59.172 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558203bb96bc] 2024-03-28T17:57:59.172 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558203ba31ff] 2024-03-28T17:57:59.172 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f806afc1488] 2024-03-28T17:57:59.172 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 13: (DispatchQueue::entry()+0x51f) [0x7f806afbe60f] 2024-03-28T17:57:59.173 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f806b091e81] 2024-03-28T17:57:59.173 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8069aac1ca] 2024-03-28T17:57:59.173 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 16: clone() 2024-03-28T17:57:59.173 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 2024-03-28T17:57:59.173 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: *** Caught signal (Aborted) ** 2024-03-28T17:57:59.173 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: in thread 7f805f400700 thread_name:ms_dispatch 2024-03-28T17:57:59.173 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:59.173 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8069ab6d20] 2024-03-28T17:57:59.174 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 2: gsignal() 2024-03-28T17:57:59.175 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 3: abort() 2024-03-28T17:57:59.175 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f806ad2f4fb] 2024-03-28T17:57:59.175 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f806ad2f667] 2024-03-28T17:57:59.175 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558203deb223] 2024-03-28T17:57:59.175 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558203c655d6] 2024-03-28T17:57:59.175 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558203c65da1] 2024-03-28T17:57:59.175 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558203c682d5] 2024-03-28T17:57:59.175 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558203cb8233] 2024-03-28T17:57:59.175 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558203cba197] 2024-03-28T17:57:59.175 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558203bb911b] 2024-03-28T17:57:59.175 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558203bb96bc] 2024-03-28T17:57:59.175 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558203ba31ff] 2024-03-28T17:57:59.175 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f806afc1488] 2024-03-28T17:57:59.175 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 16: (DispatchQueue::entry()+0x51f) [0x7f806afbe60f] 2024-03-28T17:57:59.176 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f806b091e81] 2024-03-28T17:57:59.176 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8069aac1ca] 2024-03-28T17:57:59.176 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 19: clone() 2024-03-28T17:57:59.176 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 2024-03-28T17:57:58.900+0000 7f805f400700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:57:59.176 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: in thread 7f805f400700 thread_name:ms_dispatch 2024-03-28T17:57:59.176 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 2024-03-28T17:57:59.176 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:59.176 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8069ab6d20] 2024-03-28T17:57:59.176 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 2: gsignal() 2024-03-28T17:57:59.176 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 3: abort() 2024-03-28T17:57:59.176 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f806ad2f4fb] 2024-03-28T17:57:59.176 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f806ad2f667] 2024-03-28T17:57:59.177 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558203deb223] 2024-03-28T17:57:59.177 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558203c655d6] 2024-03-28T17:57:59.177 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558203c65da1] 2024-03-28T17:57:59.177 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558203c682d5] 2024-03-28T17:57:59.177 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558203cb8233] 2024-03-28T17:57:59.177 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558203cba197] 2024-03-28T17:57:59.177 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558203bb911b] 2024-03-28T17:57:59.177 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558203bb96bc] 2024-03-28T17:57:59.177 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558203ba31ff] 2024-03-28T17:57:59.177 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f806afc1488] 2024-03-28T17:57:59.177 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 16: (DispatchQueue::entry()+0x51f) [0x7f806afbe60f] 2024-03-28T17:57:59.178 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f806b091e81] 2024-03-28T17:57:59.178 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8069aac1ca] 2024-03-28T17:57:59.178 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 19: clone() 2024-03-28T17:57:59.178 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:57:59.178 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 2024-03-28T17:57:59.178 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: -1> 2024-03-28T17:57:58.896+0000 7f805f400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f805f400700 time 2024-03-28T17:57:58.894945+0000 2024-03-28T17:57:59.178 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:57:59.178 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 2024-03-28T17:57:59.178 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:59.178 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f806ad2f4a1] 2024-03-28T17:57:59.178 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f806ad2f667] 2024-03-28T17:57:59.179 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558203deb223] 2024-03-28T17:57:59.179 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558203c655d6] 2024-03-28T17:57:59.179 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558203c65da1] 2024-03-28T17:57:59.179 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558203c682d5] 2024-03-28T17:57:59.179 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558203cb8233] 2024-03-28T17:57:59.179 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558203cba197] 2024-03-28T17:57:59.179 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558203bb911b] 2024-03-28T17:57:59.179 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558203bb96bc] 2024-03-28T17:57:59.179 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558203ba31ff] 2024-03-28T17:57:59.179 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f806afc1488] 2024-03-28T17:57:59.179 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 13: (DispatchQueue::entry()+0x51f) [0x7f806afbe60f] 2024-03-28T17:57:59.180 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f806b091e81] 2024-03-28T17:57:59.180 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8069aac1ca] 2024-03-28T17:57:59.180 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 16: clone() 2024-03-28T17:57:59.180 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 2024-03-28T17:57:59.180 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 0> 2024-03-28T17:57:58.900+0000 7f805f400700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:57:59.180 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: in thread 7f805f400700 thread_name:ms_dispatch 2024-03-28T17:57:59.180 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 2024-03-28T17:57:59.180 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:59.180 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8069ab6d20] 2024-03-28T17:57:59.180 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 2: gsignal() 2024-03-28T17:57:59.180 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 3: abort() 2024-03-28T17:57:59.180 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f806ad2f4fb] 2024-03-28T17:57:59.180 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f806ad2f667] 2024-03-28T17:57:59.180 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558203deb223] 2024-03-28T17:57:59.181 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558203c655d6] 2024-03-28T17:57:59.181 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558203c65da1] 2024-03-28T17:57:59.181 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558203c682d5] 2024-03-28T17:57:59.181 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558203cb8233] 2024-03-28T17:57:59.181 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558203cba197] 2024-03-28T17:57:59.181 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558203bb911b] 2024-03-28T17:57:59.181 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558203bb96bc] 2024-03-28T17:57:59.181 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558203ba31ff] 2024-03-28T17:57:59.181 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f806afc1488] 2024-03-28T17:57:59.181 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 16: (DispatchQueue::entry()+0x51f) [0x7f806afbe60f] 2024-03-28T17:57:59.182 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f806b091e81] 2024-03-28T17:57:59.182 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8069aac1ca] 2024-03-28T17:57:59.182 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 19: clone() 2024-03-28T17:57:59.182 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:57:59.182 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 2024-03-28T17:57:59.182 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: -1> 2024-03-28T17:57:58.896+0000 7f805f400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f805f400700 time 2024-03-28T17:57:58.894945+0000 2024-03-28T17:57:59.182 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:57:59.182 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 2024-03-28T17:57:59.182 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:59.182 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f806ad2f4a1] 2024-03-28T17:57:59.182 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f806ad2f667] 2024-03-28T17:57:59.182 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558203deb223] 2024-03-28T17:57:59.182 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558203c655d6] 2024-03-28T17:57:59.183 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558203c65da1] 2024-03-28T17:57:59.183 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558203c682d5] 2024-03-28T17:57:59.183 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558203cb8233] 2024-03-28T17:57:59.183 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558203cba197] 2024-03-28T17:57:59.183 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558203bb911b] 2024-03-28T17:57:59.183 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558203bb96bc] 2024-03-28T17:57:59.183 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558203ba31ff] 2024-03-28T17:57:59.183 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f806afc1488] 2024-03-28T17:57:59.184 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 13: (DispatchQueue::entry()+0x51f) [0x7f806afbe60f] 2024-03-28T17:57:59.184 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f806b091e81] 2024-03-28T17:57:59.184 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8069aac1ca] 2024-03-28T17:57:59.184 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 16: clone() 2024-03-28T17:57:59.184 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 2024-03-28T17:57:59.184 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 0> 2024-03-28T17:57:58.900+0000 7f805f400700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:57:59.184 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: in thread 7f805f400700 thread_name:ms_dispatch 2024-03-28T17:57:59.184 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 2024-03-28T17:57:59.184 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:57:59.184 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8069ab6d20] 2024-03-28T17:57:59.184 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 2: gsignal() 2024-03-28T17:57:59.184 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 3: abort() 2024-03-28T17:57:59.184 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f806ad2f4fb] 2024-03-28T17:57:59.184 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f806ad2f667] 2024-03-28T17:57:59.184 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558203deb223] 2024-03-28T17:57:59.185 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558203c655d6] 2024-03-28T17:57:59.185 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558203c65da1] 2024-03-28T17:57:59.185 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558203c682d5] 2024-03-28T17:57:59.185 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558203cb8233] 2024-03-28T17:57:59.185 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558203cba197] 2024-03-28T17:57:59.185 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558203bb911b] 2024-03-28T17:57:59.185 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558203bb96bc] 2024-03-28T17:57:59.185 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558203ba31ff] 2024-03-28T17:57:59.185 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f806afc1488] 2024-03-28T17:57:59.185 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 16: (DispatchQueue::entry()+0x51f) [0x7f806afbe60f] 2024-03-28T17:57:59.185 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f806b091e81] 2024-03-28T17:57:59.185 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8069aac1ca] 2024-03-28T17:57:59.185 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 19: clone() 2024-03-28T17:57:59.185 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:57:59.185 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92186]: 2024-03-28T17:57:59.186 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:59 smithi045 podman[92236]: 2024-03-28 17:57:59.02556512 +0000 UTC m=+0.037094180 container died 7a4a474873405a3304413de36cb9de7360e4903d87856ffb0f4ed4ba30be7054 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T17:57:59.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:58 smithi071 ceph-mon[36597]: pgmap v303: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-28T17:57:59.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:58 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2043640256,v1:172.21.15.45:6837/2043640256] up:rejoin 2024-03-28T17:57:59.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:58 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:59.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:58 smithi071 ceph-mon[36597]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-03-28T17:57:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:58 smithi151 ceph-mon[36481]: pgmap v303: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-28T17:57:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:58 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2043640256,v1:172.21.15.45:6837/2043640256] up:rejoin 2024-03-28T17:57:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:58 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:57:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:58 smithi151 ceph-mon[36481]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-03-28T17:57:59.896 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 332 sec latency 447793.196 ms 2024-03-28T17:58:00.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:59 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:58:00.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:59 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:58:00.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:59 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2043640256,v1:172.21.15.45:6837/2043640256] up:active 2024-03-28T17:58:00.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:59 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:00.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:59 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:58:00.112 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:57:59 smithi045 ceph-mon[31773]: pgmap v304: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-28T17:58:00.112 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:59 smithi045 podman[92236]: 2024-03-28 17:57:59.76842886 +0000 UTC m=+0.779957964 container remove 7a4a474873405a3304413de36cb9de7360e4903d87856ffb0f4ed4ba30be7054 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=) 2024-03-28T17:58:00.112 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:57:59 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:58:00.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:59 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:58:00.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:59 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:58:00.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:59 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2043640256,v1:172.21.15.45:6837/2043640256] up:active 2024-03-28T17:58:00.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:59 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:00.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:59 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:58:00.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:57:59 smithi071 ceph-mon[36597]: pgmap v304: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-28T17:58:00.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:59 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:58:00.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:59 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:58:00.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:59 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2043640256,v1:172.21.15.45:6837/2043640256] up:active 2024-03-28T17:58:00.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:59 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:00.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:59 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:58:00.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:57:59 smithi151 ceph-mon[36481]: pgmap v304: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-28T17:58:00.665 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:00 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Failed with result 'exit-code'. 2024-03-28T17:58:00.665 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:00 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Consumed 1.620s CPU time. 2024-03-28T17:58:00.896 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 333 sec latency 448793.308 ms 2024-03-28T17:58:01.896 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 334 sec latency 449793.412 ms 2024-03-28T17:58:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:02 smithi071 ceph-mon[36597]: pgmap v305: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:58:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:02 smithi151 ceph-mon[36481]: pgmap v305: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:58:02.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:02 smithi045 ceph-mon[31773]: pgmap v305: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:58:02.896 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 335 sec latency 450793.559 ms 2024-03-28T17:58:03.896 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 336 sec latency 451793.699 ms 2024-03-28T17:58:04.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:04 smithi071 ceph-mon[36597]: pgmap v306: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:58:04.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:04 smithi151 ceph-mon[36481]: pgmap v306: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:58:04.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:04 smithi045 ceph-mon[31773]: pgmap v306: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T17:58:04.896 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 337 sec latency 452793.839 ms 2024-03-28T17:58:05.897 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 338 sec latency 453793.987 ms 2024-03-28T17:58:05.996 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:58:05.996 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:58:06.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.239+0000 7f86f3000640 1 -- 172.21.15.45:0/3085025676 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86ec101630 msgr2=0x7f86ec101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:58:06.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.239+0000 7f86f3000640 1 --2- 172.21.15.45:0/3085025676 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86ec101630 0x7f86ec101ab0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f86d4009900 tx=0x7f86d402bc40 comp rx=0 tx=0).stop 2024-03-28T17:58:06.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.239+0000 7f86f3000640 1 -- 172.21.15.45:0/3085025676 shutdown_connections 2024-03-28T17:58:06.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.239+0000 7f86f3000640 1 --2- 172.21.15.45:0/3085025676 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86ec101ff0 0x7f86ec10e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.239+0000 7f86f3000640 1 --2- 172.21.15.45:0/3085025676 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86ec101630 0x7f86ec101ab0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.239+0000 7f86f3000640 1 --2- 172.21.15.45:0/3085025676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86ec107640 0x7f86ec107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.239+0000 7f86f3000640 1 -- 172.21.15.45:0/3085025676 >> 172.21.15.45:0/3085025676 conn(0x7f86ec078aa0 msgr2=0x7f86ec078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:58:06.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.239+0000 7f86f3000640 1 -- 172.21.15.45:0/3085025676 shutdown_connections 2024-03-28T17:58:06.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.239+0000 7f86f3000640 1 -- 172.21.15.45:0/3085025676 wait complete. 2024-03-28T17:58:06.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.240+0000 7f86f3000640 1 Processor -- start 2024-03-28T17:58:06.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.241+0000 7f86f3000640 1 -- start start 2024-03-28T17:58:06.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.241+0000 7f86f3000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86ec101630 0x7f86ec197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:06.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.241+0000 7f86f3000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86ec101ff0 0x7f86ec198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:06.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.241+0000 7f86f3000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86ec107640 0x7f86ec19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:06.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.241+0000 7f86f3000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec111a60 con 0x7f86ec101630 2024-03-28T17:58:06.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.241+0000 7f86f3000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec1118e0 con 0x7f86ec101ff0 2024-03-28T17:58:06.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.241+0000 7f86f3000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec111be0 con 0x7f86ec107640 2024-03-28T17:58:06.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.241+0000 7f86ebe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86ec101630 0x7f86ec197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:06.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.241+0000 7f86e3400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86ec101ff0 0x7f86ec198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:06.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.241+0000 7f86e3400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86ec101ff0 0x7f86ec198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39354/0 (socket says 172.21.15.45:39354) 2024-03-28T17:58:06.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.241+0000 7f86e3400640 1 -- 172.21.15.45:0/2873068912 learned_addr learned my addr 172.21.15.45:0/2873068912 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:58:06.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.241+0000 7f86ebe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86ec101630 0x7f86ec197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34344/0 (socket says 172.21.15.45:34344) 2024-03-28T17:58:06.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.242+0000 7f86f1000640 1 --2- 172.21.15.45:0/2873068912 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86ec107640 0x7f86ec19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:06.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.242+0000 7f86e3400640 1 -- 172.21.15.45:0/2873068912 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86ec107640 msgr2=0x7f86ec19c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:58:06.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.242+0000 7f86e3400640 1 --2- 172.21.15.45:0/2873068912 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86ec107640 0x7f86ec19c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.242+0000 7f86e3400640 1 -- 172.21.15.45:0/2873068912 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86ec101630 msgr2=0x7f86ec197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:58:06.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.242+0000 7f86e3400640 1 --2- 172.21.15.45:0/2873068912 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86ec101630 0x7f86ec197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.242+0000 7f86e3400640 1 -- 172.21.15.45:0/2873068912 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ec19cef0 con 0x7f86ec101ff0 2024-03-28T17:58:06.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.242+0000 7f86e3400640 1 --2- 172.21.15.45:0/2873068912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86ec101ff0 0x7f86ec198440 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f86d4005ab0 tx=0x7f86d40042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:58:06.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.242+0000 7f86e9600640 1 -- 172.21.15.45:0/2873068912 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d4041070 con 0x7f86ec101ff0 2024-03-28T17:58:06.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.242+0000 7f86e9600640 1 -- 172.21.15.45:0/2873068912 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f86d402ccd0 con 0x7f86ec101ff0 2024-03-28T17:58:06.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.242+0000 7f86f3000640 1 -- 172.21.15.45:0/2873068912 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86ec19d180 con 0x7f86ec101ff0 2024-03-28T17:58:06.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.242+0000 7f86e9600640 1 -- 172.21.15.45:0/2873068912 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d4003d10 con 0x7f86ec101ff0 2024-03-28T17:58:06.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.243+0000 7f86f3000640 1 -- 172.21.15.45:0/2873068912 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86ec1a4a70 con 0x7f86ec101ff0 2024-03-28T17:58:06.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.247+0000 7f86e9600640 1 -- 172.21.15.45:0/2873068912 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f86d4044070 con 0x7f86ec101ff0 2024-03-28T17:58:06.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.247+0000 7f86f5c76740 1 Processor -- start 2024-03-28T17:58:06.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.247+0000 7f86f5c76740 1 -- start start 2024-03-28T17:58:06.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.247+0000 7f86f5c76740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56547048ec00 0x56547048d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:06.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.247+0000 7f86f5c76740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56547048df20 0x56547048e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:06.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.248+0000 7f86f5c76740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547049b6a0 0x56547049da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:06.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.248+0000 7f86f5c76740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654703f2250 con 0x56547048ec00 2024-03-28T17:58:06.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.248+0000 7f86f5c76740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x56547039aa30 con 0x56547049b6a0 2024-03-28T17:58:06.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.248+0000 7f86f5c76740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654703ee2f0 con 0x56547048df20 2024-03-28T17:58:06.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.248+0000 7f86e3400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547049b6a0 0x56547049da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:06.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.248+0000 7f86ebe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56547048df20 0x56547048e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:06.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.248+0000 7f86e3400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547049b6a0 0x56547049da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39368/0 (socket says 172.21.15.45:39368) 2024-03-28T17:58:06.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.248+0000 7f86e3400640 1 -- 172.21.15.45:0/3131116295 learned_addr learned my addr 172.21.15.45:0/3131116295 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:58:06.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.248+0000 7f86f1000640 1 --2- 172.21.15.45:0/3131116295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56547048ec00 0x56547048d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:06.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.248+0000 7f86ebe00640 1 -- 172.21.15.45:0/3131116295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547049b6a0 msgr2=0x56547049da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:58:06.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.248+0000 7f86ebe00640 1 --2- 172.21.15.45:0/3131116295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547049b6a0 0x56547049da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.248+0000 7f86ebe00640 1 -- 172.21.15.45:0/3131116295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56547048ec00 msgr2=0x56547048d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:58:06.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.248+0000 7f86ebe00640 1 --2- 172.21.15.45:0/3131116295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56547048ec00 0x56547048d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.248+0000 7f86ebe00640 1 -- 172.21.15.45:0/3131116295 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565470385690 con 0x56547048df20 2024-03-28T17:58:06.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.248+0000 7f86e3400640 1 --2- 172.21.15.45:0/3131116295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547049b6a0 0x56547049da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:58:06.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.249+0000 7f86ebe00640 1 --2- 172.21.15.45:0/3131116295 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56547048df20 0x56547048e3a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f86dc009c20 tx=0x7f86dc011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=724cbcc64ecd798 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:58:06.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.249+0000 7f86e9600640 1 --2- 172.21.15.45:0/2873068912 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f86d00a29f0 0x7f86d00a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:06.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.249+0000 7f86e9600640 1 -- 172.21.15.45:0/2873068912 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 10834+0+0 (secure 0 0 0) 0x7f86d40f10a0 con 0x7f86ec101ff0 2024-03-28T17:58:06.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.249+0000 7f86e1600640 1 -- 172.21.15.45:0/3131116295 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc004050 con 0x56547048df20 2024-03-28T17:58:06.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.249+0000 7f86e1600640 1 -- 172.21.15.45:0/3131116295 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f86dc005500 con 0x56547048df20 2024-03-28T17:58:06.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.249+0000 7f86f1000640 1 --2- 172.21.15.45:0/2873068912 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f86d00a29f0 0x7f86d00a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:06.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.249+0000 7f86e1600640 1 -- 172.21.15.45:0/3131116295 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc004db0 con 0x56547048df20 2024-03-28T17:58:06.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.250+0000 7f86f5c76740 1 -- 172.21.15.45:0/3131116295 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56547048df20 msgr2=0x56547048e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:58:06.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.250+0000 7f86f1000640 1 --2- 172.21.15.45:0/2873068912 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f86d00a29f0 0x7f86d00a4eb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f86e4005fd0 tx=0x7f86e4009580 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:58:06.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.250+0000 7f86f5c76740 1 --2- 172.21.15.45:0/3131116295 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56547048df20 0x56547048e3a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f86dc009c20 tx=0x7f86dc011810 comp rx=0 tx=0).stop 2024-03-28T17:58:06.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.250+0000 7f86f5c76740 1 -- 172.21.15.45:0/3131116295 shutdown_connections 2024-03-28T17:58:06.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.250+0000 7f86f5c76740 1 --2- 172.21.15.45:0/3131116295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547049b6a0 0x56547049da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.250+0000 7f86f5c76740 1 --2- 172.21.15.45:0/3131116295 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56547048df20 0x56547048e3a0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.250+0000 7f86f5c76740 1 --2- 172.21.15.45:0/3131116295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56547048ec00 0x56547048d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.250+0000 7f86f5c76740 1 -- 172.21.15.45:0/3131116295 >> 172.21.15.45:0/3131116295 conn(0x565470455200 msgr2=0x565470488970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:58:06.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.250+0000 7f86f5c76740 1 -- 172.21.15.45:0/3131116295 shutdown_connections 2024-03-28T17:58:06.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.250+0000 7f86f5c76740 1 -- 172.21.15.45:0/3131116295 wait complete. 2024-03-28T17:58:06.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.250+0000 7f86f5c76740 1 -- start start 2024-03-28T17:58:06.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.251+0000 7f86f5c76740 1 Processor -- start 2024-03-28T17:58:06.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.251+0000 7f86f5c76740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56547048df20 0x56547049aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:06.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.251+0000 7f86f5c76740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547048ec00 0x565470498030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:06.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.251+0000 7f86f5c76740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56547049b6a0 0x565470498570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:06.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.251+0000 7f86f5c76740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56547049e510 con 0x56547049b6a0 2024-03-28T17:58:06.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.251+0000 7f86f5c76740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654703ade80 con 0x56547048ec00 2024-03-28T17:58:06.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.251+0000 7f86f5c76740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x56547049e690 con 0x56547048df20 2024-03-28T17:58:06.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.251+0000 7f86f5c76740 10 client.0 _ll_register_callbacks cb 0x5654703c0130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:58:06.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.251+0000 7f86e3400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547048ec00 0x565470498030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:06.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.251+0000 7f86f1000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56547049b6a0 0x565470498570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:06.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.251+0000 7f86e3400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547048ec00 0x565470498030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39376/0 (socket says 172.21.15.45:39376) 2024-03-28T17:58:06.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.252+0000 7f86ebe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56547048df20 0x56547049aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:06.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.252+0000 7f86e3400640 1 -- 172.21.15.45:0/1817479500 learned_addr learned my addr 172.21.15.45:0/1817479500 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:58:06.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.252+0000 7f86e3400640 1 -- 172.21.15.45:0/1817479500 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56547048df20 msgr2=0x56547049aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:58:06.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.252+0000 7f86e3400640 1 --2- 172.21.15.45:0/1817479500 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56547048df20 0x56547049aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.252+0000 7f86e3400640 1 -- 172.21.15.45:0/1817479500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56547049b6a0 msgr2=0x565470498570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:58:06.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.252+0000 7f86e3400640 1 --2- 172.21.15.45:0/1817479500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56547049b6a0 0x565470498570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.252+0000 7f86e3400640 1 -- 172.21.15.45:0/1817479500 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5654701d07d0 con 0x56547048ec00 2024-03-28T17:58:06.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.252+0000 7f86e3400640 1 --2- 172.21.15.45:0/1817479500 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547048ec00 0x565470498030 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f86d40b9360 tx=0x7f86d4003eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7aa9ae3c495e96f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:58:06.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.252+0000 7f86da000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:58:06.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.252+0000 7f86da000640 1 -- 172.21.15.45:0/1817479500 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d404b070 con 0x56547048ec00 2024-03-28T17:58:06.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.252+0000 7f86da000640 1 -- 172.21.15.45:0/1817479500 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(119..119 src has 1..119) v4 ==== 10834+0+0 (secure 0 0 0) 0x7f86d40b9570 con 0x56547048ec00 2024-03-28T17:58:06.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.252+0000 7f86f5c76740 1 -- 172.21.15.45:0/1817479500 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f86dc005500 con 0x56547048ec00 2024-03-28T17:58:06.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.253+0000 7f86da000640 1 -- 172.21.15.45:0/1817479500 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=200) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86d40b91a0 con 0x56547048ec00 2024-03-28T17:58:06.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.253+0000 7f86f5c76740 10 client.26303 fetch_fsmap learned FSMap version 200 2024-03-28T17:58:06.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.253+0000 7f86f5c76740 1 -- 172.21.15.45:0/1817479500 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=200,monmap=4+}) v3 -- 0x5654701063f0 con 0x56547048ec00 2024-03-28T17:58:06.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.253+0000 7f86da000640 1 -- 172.21.15.45:0/1817479500 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f86dc005500 con 0x56547048ec00 2024-03-28T17:58:06.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.253+0000 7f86da000640 1 -- 172.21.15.45:0/1817479500 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 200) v1 ==== 5618+0+0 (secure 0 0 0) 0x7f86d40c5210 con 0x56547048ec00 2024-03-28T17:58:06.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.253+0000 7f86f5c76740 10 client.26303 fetch_fsmap finished waiting for FSMap version 200 2024-03-28T17:58:06.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.253+0000 7f86f5c76740 10 client.26303 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:58:06.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.253+0000 7f86f5c76740 20 client.26303 populate_metadata read hostname 'smithi045' 2024-03-28T17:58:06.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.253+0000 7f86f5c76740 1 --2- 172.21.15.45:0/1817479500 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5654704ab4f0 0x5654704ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:06.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.253+0000 7f86f5c76740 4 client.26303 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:58:06.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.253+0000 7f86f5c76740 1 -- 172.21.15.45:0/1817479500 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5654703ff790 con 0x5654704ab4f0 2024-03-28T17:58:06.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.253+0000 7f86ebe00640 1 --2- 172.21.15.45:0/1817479500 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5654704ab4f0 0x5654704ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:06.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.254+0000 7f86ebe00640 1 --2- 172.21.15.45:0/1817479500 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5654704ab4f0 0x5654704ad8e0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93fd9f258fc294af server_cookie=2e31787faaf6c609 in_seq=0 out_seq=0 2024-03-28T17:58:06.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.254+0000 7f86da000640 10 client.26303 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:58:06.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.255+0000 7f86da000640 1 -- 172.21.15.45:0/1817479500 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5654703ff790 con 0x5654704ab4f0 2024-03-28T17:58:06.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.255+0000 7f86da000640 10 client.26303 handle_command_reply: tid=0 2024-03-28T17:58:06.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.255+0000 7f86f5c76740 1 client.26303 shutdown 2024-03-28T17:58:06.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.255+0000 7f86f5c76740 1 -- 172.21.15.45:0/1817479500 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547048ec00 msgr2=0x565470498030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:58:06.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.255+0000 7f86f5c76740 1 --2- 172.21.15.45:0/1817479500 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547048ec00 0x565470498030 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f86d40b9360 tx=0x7f86d4003eb0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.255+0000 7f86f5c76740 1 -- 172.21.15.45:0/1817479500 shutdown_connections 2024-03-28T17:58:06.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.255+0000 7f86f5c76740 1 --2- 172.21.15.45:0/1817479500 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5654704ab4f0 0x5654704ad8e0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.255+0000 7f86f5c76740 1 --2- 172.21.15.45:0/1817479500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56547049b6a0 0x565470498570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.255+0000 7f86f5c76740 1 --2- 172.21.15.45:0/1817479500 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547048ec00 0x565470498030 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.255+0000 7f86f5c76740 1 --2- 172.21.15.45:0/1817479500 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56547048df20 0x56547049aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.255+0000 7f86f5c76740 1 -- 172.21.15.45:0/1817479500 >> 172.21.15.45:0/1817479500 conn(0x565470455200 msgr2=0x56547048a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:58:06.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.255+0000 7f86da000640 0 client.26303 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:58:06.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.255+0000 7f86f5c76740 1 -- 172.21.15.45:0/1817479500 shutdown_connections 2024-03-28T17:58:06.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.255+0000 7f86f5c76740 1 -- 172.21.15.45:0/1817479500 wait complete. 2024-03-28T17:58:06.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.255+0000 7f86f5c76740 20 client.26303 trim_cache size 0 max 16384 2024-03-28T17:58:06.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.273+0000 7f86f5c76740 1 Processor -- start 2024-03-28T17:58:06.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.273+0000 7f86f5c76740 1 -- start start 2024-03-28T17:58:06.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.273+0000 7f86f5c76740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56547048df20 0x56547049ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:06.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.273+0000 7f86f5c76740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547048ec00 0x56547049a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:06.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.273+0000 7f86f5c76740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56547049b6a0 0x56547049aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:06.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.273+0000 7f86f5c76740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec1abce0 con 0x56547049b6a0 2024-03-28T17:58:06.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.273+0000 7f86f5c76740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec1ad660 con 0x56547048ec00 2024-03-28T17:58:06.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.273+0000 7f86f5c76740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec1ac340 con 0x56547048df20 2024-03-28T17:58:06.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.273+0000 7f86e3400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547048ec00 0x56547049a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:06.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.273+0000 7f86e3400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547048ec00 0x56547049a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39382/0 (socket says 172.21.15.45:39382) 2024-03-28T17:58:06.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.273+0000 7f86f1000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56547049b6a0 0x56547049aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:06.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.273+0000 7f86e3400640 1 -- 172.21.15.45:0/2019876380 learned_addr learned my addr 172.21.15.45:0/2019876380 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:58:06.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.273+0000 7f86ebe00640 1 --2- 172.21.15.45:0/2019876380 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56547048df20 0x56547049ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:06.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.274+0000 7f86e3400640 1 -- 172.21.15.45:0/2019876380 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56547048df20 msgr2=0x56547049ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:58:06.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.274+0000 7f86e3400640 1 --2- 172.21.15.45:0/2019876380 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56547048df20 0x56547049ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.274+0000 7f86e3400640 1 -- 172.21.15.45:0/2019876380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56547049b6a0 msgr2=0x56547049aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:58:06.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.274+0000 7f86e3400640 1 --2- 172.21.15.45:0/2019876380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56547049b6a0 0x56547049aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.274+0000 7f86e3400640 1 -- 172.21.15.45:0/2019876380 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654700c60f0 con 0x56547048ec00 2024-03-28T17:58:06.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.274+0000 7f86e3400640 1 --2- 172.21.15.45:0/2019876380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547048ec00 0x56547049a760 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f86d40bce90 tx=0x7f86d4003840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d55744fb539f576 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:58:06.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.274+0000 7f86e0c00640 1 -- 172.21.15.45:0/2019876380 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d40d9070 con 0x56547048ec00 2024-03-28T17:58:06.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.274+0000 7f86e0c00640 1 -- 172.21.15.45:0/2019876380 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f86d4004c80 con 0x56547048ec00 2024-03-28T17:58:06.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.274+0000 7f86e0c00640 1 -- 172.21.15.45:0/2019876380 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d404b070 con 0x56547048ec00 2024-03-28T17:58:06.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.275+0000 7f86f5c76740 1 -- 172.21.15.45:0/2019876380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547048ec00 msgr2=0x56547049a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:58:06.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.275+0000 7f86f5c76740 1 --2- 172.21.15.45:0/2019876380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547048ec00 0x56547049a760 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f86d40bce90 tx=0x7f86d4003840 comp rx=0 tx=0).stop 2024-03-28T17:58:06.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.275+0000 7f86f5c76740 1 -- 172.21.15.45:0/2019876380 shutdown_connections 2024-03-28T17:58:06.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.275+0000 7f86f5c76740 1 --2- 172.21.15.45:0/2019876380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56547049b6a0 0x56547049aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.275+0000 7f86f5c76740 1 --2- 172.21.15.45:0/2019876380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547048ec00 0x56547049a760 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.275+0000 7f86f5c76740 1 --2- 172.21.15.45:0/2019876380 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56547048df20 0x56547049ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.275+0000 7f86f5c76740 1 -- 172.21.15.45:0/2019876380 >> 172.21.15.45:0/2019876380 conn(0x565470455200 msgr2=0x5654704884a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:58:06.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.275+0000 7f86f5c76740 1 -- 172.21.15.45:0/2019876380 shutdown_connections 2024-03-28T17:58:06.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.275+0000 7f86f5c76740 1 -- 172.21.15.45:0/2019876380 wait complete. 2024-03-28T17:58:06.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.276+0000 7f86f5c76740 1 -- start start 2024-03-28T17:58:06.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.276+0000 7f86f5c76740 1 Processor -- start 2024-03-28T17:58:06.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.276+0000 7f86f5c76740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56547048df20 0x5654704924e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:06.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.276+0000 7f86f5c76740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56547048ec00 0x565470492a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:06.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.276+0000 7f86f5c76740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547049b6a0 0x565470494e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:06.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.276+0000 7f86f5c76740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec1ad000 con 0x56547048ec00 2024-03-28T17:58:06.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.276+0000 7f86f5c76740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec1ac9a0 con 0x56547049b6a0 2024-03-28T17:58:06.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.276+0000 7f86f5c76740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec1aa620 con 0x56547048df20 2024-03-28T17:58:06.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.276+0000 7f86f5c76740 10 client.0 _ll_register_callbacks cb 0x7f86d4005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:58:06.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.276+0000 7f86f1000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547049b6a0 0x565470494e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:06.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.277+0000 7f86f1000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547049b6a0 0x565470494e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:39396/0 (socket says 172.21.15.45:39396) 2024-03-28T17:58:06.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.277+0000 7f86f1000640 1 -- 172.21.15.45:0/2991072624 learned_addr learned my addr 172.21.15.45:0/2991072624 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:58:06.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.277+0000 7f86e3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56547048ec00 0x565470492a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:06.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.277+0000 7f86ebe00640 1 --2- 172.21.15.45:0/2991072624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56547048df20 0x5654704924e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:06.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.277+0000 7f86f1000640 1 -- 172.21.15.45:0/2991072624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56547048df20 msgr2=0x5654704924e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:58:06.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.277+0000 7f86f1000640 1 --2- 172.21.15.45:0/2991072624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56547048df20 0x5654704924e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.277+0000 7f86f1000640 1 -- 172.21.15.45:0/2991072624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56547048ec00 msgr2=0x565470492a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:58:06.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.277+0000 7f86f1000640 1 --2- 172.21.15.45:0/2991072624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56547048ec00 0x565470492a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.277+0000 7f86f1000640 1 -- 172.21.15.45:0/2991072624 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565470495370 con 0x56547049b6a0 2024-03-28T17:58:06.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.277+0000 7f86f1000640 1 --2- 172.21.15.45:0/2991072624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547049b6a0 0x565470494e30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f86e4012960 tx=0x7f86e4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfae7d2a5280950 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:58:06.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.277+0000 7f86da000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:58:06.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.277+0000 7f86da000640 1 -- 172.21.15.45:0/2991072624 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e402e060 con 0x56547049b6a0 2024-03-28T17:58:06.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.277+0000 7f86f5c76740 1 -- 172.21.15.45:0/2991072624 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f86d4004c80 con 0x56547049b6a0 2024-03-28T17:58:06.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.278+0000 7f86da000640 1 -- 172.21.15.45:0/2991072624 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(119..119 src has 1..119) v4 ==== 10834+0+0 (secure 0 0 0) 0x7f86e4020b40 con 0x56547049b6a0 2024-03-28T17:58:06.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.278+0000 7f86da000640 1 -- 172.21.15.45:0/2991072624 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=200) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86e4035050 con 0x56547049b6a0 2024-03-28T17:58:06.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.278+0000 7f86f5c76740 10 client.26309 fetch_fsmap learned FSMap version 200 2024-03-28T17:58:06.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.278+0000 7f86f5c76740 1 -- 172.21.15.45:0/2991072624 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=200,monmap=4+}) v3 -- 0x56547050bc90 con 0x56547049b6a0 2024-03-28T17:58:06.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.278+0000 7f86da000640 1 -- 172.21.15.45:0/2991072624 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f86d4004c80 con 0x56547049b6a0 2024-03-28T17:58:06.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.278+0000 7f86da000640 1 -- 172.21.15.45:0/2991072624 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 200) v1 ==== 5618+0+0 (secure 0 0 0) 0x7f86e400f040 con 0x56547049b6a0 2024-03-28T17:58:06.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.278+0000 7f86f5c76740 10 client.26309 fetch_fsmap finished waiting for FSMap version 200 2024-03-28T17:58:06.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.278+0000 7f86f5c76740 10 client.26309 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:58:06.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.278+0000 7f86f5c76740 20 client.26309 populate_metadata read hostname 'smithi045' 2024-03-28T17:58:06.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.278+0000 7f86f5c76740 1 --2- 172.21.15.45:0/2991072624 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56547050c340 0x56547050e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:06.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.278+0000 7f86f5c76740 4 client.26309 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:58:06.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.278+0000 7f86f5c76740 1 -- 172.21.15.45:0/2991072624 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x565470493c50 con 0x56547050c340 2024-03-28T17:58:06.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.278+0000 7f86ebe00640 1 --2- 172.21.15.45:0/2991072624 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56547050c340 0x56547050e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:06.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.279+0000 7f86ebe00640 1 --2- 172.21.15.45:0/2991072624 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56547050c340 0x56547050e760 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=345e0f48c5037f1e server_cookie=1d77c5ee4ee8a252 in_seq=0 out_seq=0 2024-03-28T17:58:06.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.279+0000 7f86da000640 10 client.26309 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:58:06.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.279+0000 7f86da000640 1 -- 172.21.15.45:0/2991072624 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x565470493c50 con 0x56547050c340 2024-03-28T17:58:06.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.279+0000 7f86da000640 10 client.26309 handle_command_reply: tid=0 2024-03-28T17:58:06.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.279+0000 7f86f5c76740 1 client.26309 shutdown 2024-03-28T17:58:06.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.279+0000 7f86f5c76740 1 -- 172.21.15.45:0/2991072624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547049b6a0 msgr2=0x565470494e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:58:06.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.279+0000 7f86f5c76740 1 --2- 172.21.15.45:0/2991072624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547049b6a0 0x565470494e30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f86e4012960 tx=0x7f86e4004a40 comp rx=0 tx=0).stop 2024-03-28T17:58:06.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.279+0000 7f86f5c76740 1 -- 172.21.15.45:0/2991072624 shutdown_connections 2024-03-28T17:58:06.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.279+0000 7f86f5c76740 1 --2- 172.21.15.45:0/2991072624 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56547050c340 0x56547050e760 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.279+0000 7f86f5c76740 1 --2- 172.21.15.45:0/2991072624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56547049b6a0 0x565470494e30 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.279+0000 7f86f5c76740 1 --2- 172.21.15.45:0/2991072624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56547048ec00 0x565470492a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.279+0000 7f86f5c76740 1 --2- 172.21.15.45:0/2991072624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56547048df20 0x5654704924e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.279+0000 7f86da000640 0 client.26309 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:58:06.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.279+0000 7f86f5c76740 1 -- 172.21.15.45:0/2991072624 >> 172.21.15.45:0/2991072624 conn(0x565470455200 msgr2=0x565470487200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:58:06.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.279+0000 7f86f5c76740 1 -- 172.21.15.45:0/2991072624 shutdown_connections 2024-03-28T17:58:06.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.280+0000 7f86f5c76740 1 -- 172.21.15.45:0/2991072624 wait complete. 2024-03-28T17:58:06.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.280+0000 7f86f5c76740 20 client.26309 trim_cache size 0 max 16384 2024-03-28T17:58:06.287 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:58:06.288 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T17:58:06.288 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T17:58:06.288 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T17:58:06.288 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T17:58:06.288 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T17:58:06.288 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T17:58:06.288 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:58:06.288 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:58:06.288 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:58:06.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.280+0000 7f86f3000640 1 -- 172.21.15.45:0/2873068912 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f86d00a29f0 msgr2=0x7f86d00a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:58:06.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.280+0000 7f86f3000640 1 --2- 172.21.15.45:0/2873068912 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f86d00a29f0 0x7f86d00a4eb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f86e4005fd0 tx=0x7f86e4009580 comp rx=0 tx=0).stop 2024-03-28T17:58:06.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.280+0000 7f86f3000640 1 -- 172.21.15.45:0/2873068912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86ec101ff0 msgr2=0x7f86ec198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:58:06.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.280+0000 7f86f3000640 1 --2- 172.21.15.45:0/2873068912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86ec101ff0 0x7f86ec198440 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f86d4005ab0 tx=0x7f86d40042f0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.281+0000 7f86f3000640 1 -- 172.21.15.45:0/2873068912 shutdown_connections 2024-03-28T17:58:06.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.281+0000 7f86f3000640 1 --2- 172.21.15.45:0/2873068912 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f86d00a29f0 0x7f86d00a4eb0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.281+0000 7f86f3000640 1 --2- 172.21.15.45:0/2873068912 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86ec107640 0x7f86ec19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.281+0000 7f86f3000640 1 --2- 172.21.15.45:0/2873068912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86ec101ff0 0x7f86ec198440 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.281+0000 7f86f3000640 1 --2- 172.21.15.45:0/2873068912 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86ec101630 0x7f86ec197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:06.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.281+0000 7f86f3000640 1 -- 172.21.15.45:0/2873068912 >> 172.21.15.45:0/2873068912 conn(0x7f86ec078aa0 msgr2=0x7f86ec103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:58:06.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.281+0000 7f86f3000640 1 -- 172.21.15.45:0/2873068912 shutdown_connections 2024-03-28T17:58:06.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:06.281+0000 7f86f3000640 1 -- 172.21.15.45:0/2873068912 wait complete. 2024-03-28T17:58:06.298 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T17:58:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:06 smithi045 ceph-mon[31773]: pgmap v307: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 194 B/s wr, 14 op/s 2024-03-28T17:58:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:06 smithi071 ceph-mon[36597]: pgmap v307: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 194 B/s wr, 14 op/s 2024-03-28T17:58:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:06 smithi151 ceph-mon[36481]: pgmap v307: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 194 B/s wr, 14 op/s 2024-03-28T17:58:06.897 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 339 sec latency 454794.076 ms 2024-03-28T17:58:07.897 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 340 sec latency 455794.222 ms 2024-03-28T17:58:08.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:08 smithi071 ceph-mon[36597]: pgmap v308: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T17:58:08.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:08 smithi151 ceph-mon[36481]: pgmap v308: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T17:58:08.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:08 smithi045 ceph-mon[31773]: pgmap v308: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T17:58:08.897 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 341 sec latency 456794.379 ms 2024-03-28T17:58:09.898 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 342 sec latency 457794.506 ms 2024-03-28T17:58:10.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:10 smithi071 ceph-mon[36597]: pgmap v309: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T17:58:10.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:10 smithi151 ceph-mon[36481]: pgmap v309: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T17:58:10.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:10 smithi045 ceph-mon[31773]: pgmap v309: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T17:58:10.665 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:10 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Scheduled restart job, restart counter is at 2. 2024-03-28T17:58:10.665 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:10 smithi045 systemd[1]: Stopped Ceph mds.a for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:58:10.666 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:10 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Consumed 1.620s CPU time. 2024-03-28T17:58:10.666 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:10 smithi045 systemd[1]: Starting Ceph mds.a for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:58:10.898 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 343 sec latency 458794.645 ms 2024-03-28T17:58:11.148 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:10 smithi045 podman[92393]: 2024-03-28 17:58:10.725237741 +0000 UTC m=+0.034688863 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:58:11.148 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:10 smithi045 podman[92393]: 2024-03-28 17:58:10.870731809 +0000 UTC m=+0.180182956 container create 6405899b7c3c3459ef7d1bd443a25713af2faafb75f135de0486395153e3f70e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-28T17:58:11.416 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:11 smithi045 podman[92393]: 2024-03-28 17:58:11.148302084 +0000 UTC m=+0.457753285 container init 6405899b7c3c3459ef7d1bd443a25713af2faafb75f135de0486395153e3f70e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=) 2024-03-28T17:58:11.416 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:11 smithi045 podman[92393]: 2024-03-28 17:58:11.15884718 +0000 UTC m=+0.468298300 container start 6405899b7c3c3459ef7d1bd443a25713af2faafb75f135de0486395153e3f70e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-28T17:58:11.417 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:11 smithi045 ceph-mds[92412]: -- 172.21.15.45:0/2384797467 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556cbd425a40 con 0x556cbd3f9000 2024-03-28T17:58:11.417 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: starting mds.a at 2024-03-28T17:58:11.417 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:11 smithi045 bash[92393]: 6405899b7c3c3459ef7d1bd443a25713af2faafb75f135de0486395153e3f70e 2024-03-28T17:58:11.417 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:11 smithi045 systemd[1]: Started Ceph mds.a for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:58:11.898 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 344 sec latency 459794.801 ms 2024-03-28T17:58:12.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:12 smithi071 ceph-mon[36597]: pgmap v310: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T17:58:12.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:12 smithi071 ceph-mon[36597]: daemon mds.a restarted 2024-03-28T17:58:12.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:12 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:58:12.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:12 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T17:58:12.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:12 smithi071 ceph-mon[36597]: osdmap e120: 12 total, 12 up, 12 in 2024-03-28T17:58:12.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:12 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/181516854,v1:172.21.15.45:6837/181516854] up:boot 2024-03-28T17:58:12.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:12 smithi071 ceph-mon[36597]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-03-28T17:58:12.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:12 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:58:12.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:12 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:12.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:12 smithi151 ceph-mon[36481]: pgmap v310: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T17:58:12.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:12 smithi151 ceph-mon[36481]: daemon mds.a restarted 2024-03-28T17:58:12.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:12 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:58:12.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:12 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T17:58:12.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:12 smithi151 ceph-mon[36481]: osdmap e120: 12 total, 12 up, 12 in 2024-03-28T17:58:12.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:12 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/181516854,v1:172.21.15.45:6837/181516854] up:boot 2024-03-28T17:58:12.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:12 smithi151 ceph-mon[36481]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-03-28T17:58:12.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:12 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:58:12.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:12 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:12.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:12 smithi045 ceph-mon[31773]: pgmap v310: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T17:58:12.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:12 smithi045 ceph-mon[31773]: daemon mds.a restarted 2024-03-28T17:58:12.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:12 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:58:12.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:12 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T17:58:12.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:12 smithi045 ceph-mon[31773]: osdmap e120: 12 total, 12 up, 12 in 2024-03-28T17:58:12.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:12 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/181516854,v1:172.21.15.45:6837/181516854] up:boot 2024-03-28T17:58:12.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:12 smithi045 ceph-mon[31773]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-03-28T17:58:12.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:12 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:58:12.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:12 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:12.898 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 345 sec latency 460794.931 ms 2024-03-28T17:58:13.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:13 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/3219133319,v1:172.21.15.151:6833/3219133319] up:resolve 2024-03-28T17:58:13.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:13 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:13.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:13 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/3219133319,v1:172.21.15.151:6833/3219133319] up:resolve 2024-03-28T17:58:13.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:13 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:13.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:13 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/3219133319,v1:172.21.15.151:6833/3219133319] up:resolve 2024-03-28T17:58:13.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:13 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:13.898 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 346 sec latency 461795.027 ms 2024-03-28T17:58:14.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:14 smithi071 ceph-mon[36597]: pgmap v312: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:58:14.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:14 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0 2024-03-28T17:58:14.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:14 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/3219133319,v1:172.21.15.151:6833/3219133319] up:reconnect 2024-03-28T17:58:14.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:14 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:14.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:14 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00699992 2024-03-28T17:58:14.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:14 smithi045 ceph-mon[31773]: pgmap v312: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:58:14.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:14 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0 2024-03-28T17:58:14.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:14 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/3219133319,v1:172.21.15.151:6833/3219133319] up:reconnect 2024-03-28T17:58:14.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:14 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:14.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:14 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00699992 2024-03-28T17:58:14.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:14 smithi151 ceph-mon[36481]: pgmap v312: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T17:58:14.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:14 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0 2024-03-28T17:58:14.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:14 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/3219133319,v1:172.21.15.151:6833/3219133319] up:reconnect 2024-03-28T17:58:14.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:14 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:14.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:14 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00699992 2024-03-28T17:58:14.898 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 347 sec latency 462795.179 ms 2024-03-28T17:58:15.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:15 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/3219133319,v1:172.21.15.151:6833/3219133319] up:rejoin 2024-03-28T17:58:15.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:15 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:15.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:15 smithi071 ceph-mon[36597]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-03-28T17:58:15.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:15 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:58:15.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:15 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:58:15.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:15 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/3219133319,v1:172.21.15.151:6833/3219133319] up:active 2024-03-28T17:58:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/3219133319,v1:172.21.15.151:6833/3219133319] up:rejoin 2024-03-28T17:58:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-mon[36481]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-03-28T17:58:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:58:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:58:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/3219133319,v1:172.21.15.151:6833/3219133319] up:active 2024-03-28T17:58:15.574 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd608d90700 time 2024-03-28T17:58:15.264763+0000 2024-03-28T17:58:15.574 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:58:15.574 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:15.574 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd6133ba4a1] 2024-03-28T17:58:15.574 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd6133ba667] 2024-03-28T17:58:15.574 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559dfff46223] 2024-03-28T17:58:15.574 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559dffdc05d6] 2024-03-28T17:58:15.575 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559dffdc0da1] 2024-03-28T17:58:15.575 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559dffdc32d5] 2024-03-28T17:58:15.575 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559dffe13233] 2024-03-28T17:58:15.575 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559dffe15197] 2024-03-28T17:58:15.575 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559dffd1411b] 2024-03-28T17:58:15.575 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559dffd146bc] 2024-03-28T17:58:15.575 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559dffcfe1ff] 2024-03-28T17:58:15.575 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd61364c488] 2024-03-28T17:58:15.575 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 13: (DispatchQueue::entry()+0x51f) [0x7fd61364960f] 2024-03-28T17:58:15.575 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd61371ce81] 2024-03-28T17:58:15.575 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd6121371ca] 2024-03-28T17:58:15.575 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 16: clone() 2024-03-28T17:58:15.575 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: *** Caught signal (Aborted) ** 2024-03-28T17:58:15.575 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: in thread 7fd608d90700 thread_name:ms_dispatch 2024-03-28T17:58:15.576 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 2024-03-28T17:58:15.265+0000 7fd608d90700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd608d90700 time 2024-03-28T17:58:15.264763+0000 2024-03-28T17:58:15.576 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:58:15.576 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 2024-03-28T17:58:15.576 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:15.576 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd6133ba4a1] 2024-03-28T17:58:15.576 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd6133ba667] 2024-03-28T17:58:15.576 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559dfff46223] 2024-03-28T17:58:15.576 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559dffdc05d6] 2024-03-28T17:58:15.576 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559dffdc0da1] 2024-03-28T17:58:15.576 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559dffdc32d5] 2024-03-28T17:58:15.576 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559dffe13233] 2024-03-28T17:58:15.576 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559dffe15197] 2024-03-28T17:58:15.576 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559dffd1411b] 2024-03-28T17:58:15.576 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559dffd146bc] 2024-03-28T17:58:15.577 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559dffcfe1ff] 2024-03-28T17:58:15.577 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd61364c488] 2024-03-28T17:58:15.577 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 13: (DispatchQueue::entry()+0x51f) [0x7fd61364960f] 2024-03-28T17:58:15.577 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd61371ce81] 2024-03-28T17:58:15.577 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd6121371ca] 2024-03-28T17:58:15.577 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 16: clone() 2024-03-28T17:58:15.577 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 2024-03-28T17:58:15.577 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:15.577 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd612141d20] 2024-03-28T17:58:15.578 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 2: gsignal() 2024-03-28T17:58:15.578 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 3: abort() 2024-03-28T17:58:15.578 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd6133ba4fb] 2024-03-28T17:58:15.578 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd6133ba667] 2024-03-28T17:58:15.578 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559dfff46223] 2024-03-28T17:58:15.578 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559dffdc05d6] 2024-03-28T17:58:15.579 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559dffdc0da1] 2024-03-28T17:58:15.579 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559dffdc32d5] 2024-03-28T17:58:15.579 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559dffe13233] 2024-03-28T17:58:15.579 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559dffe15197] 2024-03-28T17:58:15.579 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559dffd1411b] 2024-03-28T17:58:15.579 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559dffd146bc] 2024-03-28T17:58:15.579 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559dffcfe1ff] 2024-03-28T17:58:15.579 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd61364c488] 2024-03-28T17:58:15.579 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 16: (DispatchQueue::entry()+0x51f) [0x7fd61364960f] 2024-03-28T17:58:15.579 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd61371ce81] 2024-03-28T17:58:15.579 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd6121371ca] 2024-03-28T17:58:15.579 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 19: clone() 2024-03-28T17:58:15.579 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 2024-03-28T17:58:15.267+0000 7fd608d90700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:58:15.579 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: in thread 7fd608d90700 thread_name:ms_dispatch 2024-03-28T17:58:15.579 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 2024-03-28T17:58:15.580 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:15.580 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd612141d20] 2024-03-28T17:58:15.580 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 2: gsignal() 2024-03-28T17:58:15.580 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 3: abort() 2024-03-28T17:58:15.580 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd6133ba4fb] 2024-03-28T17:58:15.580 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd6133ba667] 2024-03-28T17:58:15.580 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559dfff46223] 2024-03-28T17:58:15.580 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559dffdc05d6] 2024-03-28T17:58:15.580 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559dffdc0da1] 2024-03-28T17:58:15.580 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559dffdc32d5] 2024-03-28T17:58:15.580 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559dffe13233] 2024-03-28T17:58:15.580 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559dffe15197] 2024-03-28T17:58:15.580 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559dffd1411b] 2024-03-28T17:58:15.580 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559dffd146bc] 2024-03-28T17:58:15.580 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559dffcfe1ff] 2024-03-28T17:58:15.581 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd61364c488] 2024-03-28T17:58:15.581 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 16: (DispatchQueue::entry()+0x51f) [0x7fd61364960f] 2024-03-28T17:58:15.581 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd61371ce81] 2024-03-28T17:58:15.581 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd6121371ca] 2024-03-28T17:58:15.581 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 19: clone() 2024-03-28T17:58:15.581 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:58:15.581 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 2024-03-28T17:58:15.581 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: -1> 2024-03-28T17:58:15.265+0000 7fd608d90700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd608d90700 time 2024-03-28T17:58:15.264763+0000 2024-03-28T17:58:15.581 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:58:15.581 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 2024-03-28T17:58:15.581 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:15.581 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd6133ba4a1] 2024-03-28T17:58:15.581 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd6133ba667] 2024-03-28T17:58:15.581 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559dfff46223] 2024-03-28T17:58:15.581 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559dffdc05d6] 2024-03-28T17:58:15.582 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559dffdc0da1] 2024-03-28T17:58:15.582 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559dffdc32d5] 2024-03-28T17:58:15.582 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559dffe13233] 2024-03-28T17:58:15.582 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559dffe15197] 2024-03-28T17:58:15.582 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559dffd1411b] 2024-03-28T17:58:15.582 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559dffd146bc] 2024-03-28T17:58:15.582 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559dffcfe1ff] 2024-03-28T17:58:15.582 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd61364c488] 2024-03-28T17:58:15.582 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 13: (DispatchQueue::entry()+0x51f) [0x7fd61364960f] 2024-03-28T17:58:15.582 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd61371ce81] 2024-03-28T17:58:15.582 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd6121371ca] 2024-03-28T17:58:15.582 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 16: clone() 2024-03-28T17:58:15.582 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 2024-03-28T17:58:15.582 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 0> 2024-03-28T17:58:15.267+0000 7fd608d90700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:58:15.583 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: in thread 7fd608d90700 thread_name:ms_dispatch 2024-03-28T17:58:15.583 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 2024-03-28T17:58:15.583 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:15.583 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd612141d20] 2024-03-28T17:58:15.583 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 2: gsignal() 2024-03-28T17:58:15.583 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 3: abort() 2024-03-28T17:58:15.583 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd6133ba4fb] 2024-03-28T17:58:15.583 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd6133ba667] 2024-03-28T17:58:15.583 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559dfff46223] 2024-03-28T17:58:15.583 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559dffdc05d6] 2024-03-28T17:58:15.583 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559dffdc0da1] 2024-03-28T17:58:15.583 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559dffdc32d5] 2024-03-28T17:58:15.583 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559dffe13233] 2024-03-28T17:58:15.583 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559dffe15197] 2024-03-28T17:58:15.583 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559dffd1411b] 2024-03-28T17:58:15.584 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559dffd146bc] 2024-03-28T17:58:15.584 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559dffcfe1ff] 2024-03-28T17:58:15.584 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd61364c488] 2024-03-28T17:58:15.584 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 16: (DispatchQueue::entry()+0x51f) [0x7fd61364960f] 2024-03-28T17:58:15.584 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd61371ce81] 2024-03-28T17:58:15.584 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd6121371ca] 2024-03-28T17:58:15.584 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 19: clone() 2024-03-28T17:58:15.584 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:58:15.584 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 2024-03-28T17:58:15.584 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: -1> 2024-03-28T17:58:15.265+0000 7fd608d90700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd608d90700 time 2024-03-28T17:58:15.264763+0000 2024-03-28T17:58:15.584 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:58:15.584 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 2024-03-28T17:58:15.584 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:15.584 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd6133ba4a1] 2024-03-28T17:58:15.584 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd6133ba667] 2024-03-28T17:58:15.585 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559dfff46223] 2024-03-28T17:58:15.585 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559dffdc05d6] 2024-03-28T17:58:15.585 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559dffdc0da1] 2024-03-28T17:58:15.585 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559dffdc32d5] 2024-03-28T17:58:15.585 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559dffe13233] 2024-03-28T17:58:15.585 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559dffe15197] 2024-03-28T17:58:15.585 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559dffd1411b] 2024-03-28T17:58:15.585 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559dffd146bc] 2024-03-28T17:58:15.585 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559dffcfe1ff] 2024-03-28T17:58:15.585 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd61364c488] 2024-03-28T17:58:15.585 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 13: (DispatchQueue::entry()+0x51f) [0x7fd61364960f] 2024-03-28T17:58:15.585 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd61371ce81] 2024-03-28T17:58:15.585 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd6121371ca] 2024-03-28T17:58:15.585 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 16: clone() 2024-03-28T17:58:15.585 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 2024-03-28T17:58:15.585 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 0> 2024-03-28T17:58:15.267+0000 7fd608d90700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:58:15.586 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: in thread 7fd608d90700 thread_name:ms_dispatch 2024-03-28T17:58:15.586 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 2024-03-28T17:58:15.586 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:15.586 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd612141d20] 2024-03-28T17:58:15.586 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 2: gsignal() 2024-03-28T17:58:15.586 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 3: abort() 2024-03-28T17:58:15.586 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd6133ba4fb] 2024-03-28T17:58:15.586 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd6133ba667] 2024-03-28T17:58:15.586 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559dfff46223] 2024-03-28T17:58:15.586 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559dffdc05d6] 2024-03-28T17:58:15.586 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559dffdc0da1] 2024-03-28T17:58:15.586 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559dffdc32d5] 2024-03-28T17:58:15.586 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559dffe13233] 2024-03-28T17:58:15.586 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559dffe15197] 2024-03-28T17:58:15.587 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559dffd1411b] 2024-03-28T17:58:15.587 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559dffd146bc] 2024-03-28T17:58:15.587 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559dffcfe1ff] 2024-03-28T17:58:15.587 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd61364c488] 2024-03-28T17:58:15.587 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 16: (DispatchQueue::entry()+0x51f) [0x7fd61364960f] 2024-03-28T17:58:15.587 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd61371ce81] 2024-03-28T17:58:15.587 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd6121371ca] 2024-03-28T17:58:15.587 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 19: clone() 2024-03-28T17:58:15.587 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:58:15.587 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73676]: 2024-03-28T17:58:15.587 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:15 smithi151 podman[73714]: 2024-03-28 17:58:15.319492734 +0000 UTC m=+0.020705182 container died bbd09cfa544741b1759e51fdceaf817c77d2ecb110929e8187d74b564539e247 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T17:58:15.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:15 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/3219133319,v1:172.21.15.151:6833/3219133319] up:rejoin 2024-03-28T17:58:15.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:15 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:15.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:15 smithi045 ceph-mon[31773]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-03-28T17:58:15.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:15 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:58:15.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:15 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:58:15.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:15 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/3219133319,v1:172.21.15.151:6833/3219133319] up:active 2024-03-28T17:58:15.898 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 348 sec latency 463795.339 ms 2024-03-28T17:58:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:16 smithi071 ceph-mon[36597]: pgmap v313: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:58:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:16 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:16.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:16 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:58:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:16 smithi151 ceph-mon[36481]: pgmap v313: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:58:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:16 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:16 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:58:16.574 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:16 smithi151 podman[73714]: 2024-03-28 17:58:16.113662382 +0000 UTC m=+0.814874829 container remove bbd09cfa544741b1759e51fdceaf817c77d2ecb110929e8187d74b564539e247 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T17:58:16.574 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:16 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:58:16.574 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:16 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-28T17:58:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:16 smithi045 ceph-mon[31773]: pgmap v313: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:58:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:16 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:16 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:58:16.898 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 349 sec latency 464795.495 ms 2024-03-28T17:58:17.899 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 350 sec latency 465795.625 ms 2024-03-28T17:58:18.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:18 smithi071 ceph-mon[36597]: pgmap v314: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:58:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:18 smithi151 ceph-mon[36481]: pgmap v314: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:58:18.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:18 smithi045 ceph-mon[31773]: pgmap v314: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:58:18.899 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 351 sec latency 466795.786 ms 2024-03-28T17:58:19.899 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 352 sec latency 467795.913 ms 2024-03-28T17:58:20.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:20 smithi071 ceph-mon[36597]: pgmap v315: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:58:20.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:20 smithi151 ceph-mon[36481]: pgmap v315: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:58:20.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:20 smithi045 ceph-mon[31773]: pgmap v315: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:58:20.899 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 353 sec latency 468795.988 ms 2024-03-28T17:58:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:21 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:58:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:21 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:58:21.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:21 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:58:21.899 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 354 sec latency 469796.047 ms 2024-03-28T17:58:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:22 smithi071 ceph-mon[36597]: pgmap v316: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:58:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:22 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:58:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:22 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:58:22.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:22 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:58:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:22 smithi151 ceph-mon[36481]: pgmap v316: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:58:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:22 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:58:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:22 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:58:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:22 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:58:22.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:22 smithi045 ceph-mon[31773]: pgmap v316: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:58:22.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:22 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:58:22.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:22 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:58:22.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:22 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:58:22.899 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 355 sec latency 470796.185 ms 2024-03-28T17:58:23.899 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 356 sec latency 471796.349 ms 2024-03-28T17:58:24.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:24 smithi071 ceph-mon[36597]: pgmap v317: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-28T17:58:24.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:24 smithi151 ceph-mon[36481]: pgmap v317: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-28T17:58:24.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:24 smithi045 ceph-mon[31773]: pgmap v317: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-28T17:58:24.899 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 357 sec latency 472796.487 ms 2024-03-28T17:58:25.900 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 358 sec latency 473796.629 ms 2024-03-28T17:58:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:26 smithi071 ceph-mon[36597]: pgmap v318: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-28T17:58:26.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:26 smithi151 ceph-mon[36481]: pgmap v318: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-28T17:58:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:26 smithi045 ceph-mon[31773]: pgmap v318: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-28T17:58:26.900 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 359 sec latency 474796.804 ms 2024-03-28T17:58:26.945 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:26 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 4. 2024-03-28T17:58:26.946 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:26 smithi151 systemd[1]: Stopped Ceph mds.l for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:58:26.946 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:26 smithi151 systemd[1]: Starting Ceph mds.l for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:58:26.946 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:26 smithi151 podman[73956]: 2024-03-28 17:58:26.702489056 +0000 UTC m=+0.017738642 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:58:26.946 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:26 smithi151 podman[73956]: 2024-03-28 17:58:26.823655857 +0000 UTC m=+0.138905437 container create 47587274a62d62aea96afbfe5d0463bb3c1a333bfd747d7b1a3dc1025732e1d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T17:58:27.308 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:27 smithi151 podman[73956]: 2024-03-28 17:58:27.060610035 +0000 UTC m=+0.375859627 container init 47587274a62d62aea96afbfe5d0463bb3c1a333bfd747d7b1a3dc1025732e1d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True) 2024-03-28T17:58:27.308 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:27 smithi151 podman[73956]: 2024-03-28 17:58:27.063188342 +0000 UTC m=+0.378437920 container start 47587274a62d62aea96afbfe5d0463bb3c1a333bfd747d7b1a3dc1025732e1d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-28T17:58:27.308 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:27 smithi151 ceph-mds[73976]: -- 172.21.15.151:0/2292242137 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a58fad5a40 con 0x55a58faa8800 2024-03-28T17:58:27.308 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:27 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: starting mds.l at 2024-03-28T17:58:27.308 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:27 smithi151 bash[73956]: 47587274a62d62aea96afbfe5d0463bb3c1a333bfd747d7b1a3dc1025732e1d5 2024-03-28T17:58:27.308 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:27 smithi151 systemd[1]: Started Ceph mds.l for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:58:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:27 smithi071 ceph-mon[36597]: daemon mds.l restarted 2024-03-28T17:58:27.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:27 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:58:27.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:27 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T17:58:27.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:27 smithi071 ceph-mon[36597]: osdmap e121: 12 total, 12 up, 12 in 2024-03-28T17:58:27.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:27 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/4173171826,v1:172.21.15.151:6833/4173171826] up:boot 2024-03-28T17:58:27.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:27 smithi071 ceph-mon[36597]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-03-28T17:58:27.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:27 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:58:27.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:27 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:27 smithi151 ceph-mon[36481]: daemon mds.l restarted 2024-03-28T17:58:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:27 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:58:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:27 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T17:58:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:27 smithi151 ceph-mon[36481]: osdmap e121: 12 total, 12 up, 12 in 2024-03-28T17:58:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:27 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/4173171826,v1:172.21.15.151:6833/4173171826] up:boot 2024-03-28T17:58:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:27 smithi151 ceph-mon[36481]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-03-28T17:58:27.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:27 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:58:27.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:27 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:27 smithi045 ceph-mon[31773]: daemon mds.l restarted 2024-03-28T17:58:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:27 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:58:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:27 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T17:58:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:27 smithi045 ceph-mon[31773]: osdmap e121: 12 total, 12 up, 12 in 2024-03-28T17:58:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:27 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/4173171826,v1:172.21.15.151:6833/4173171826] up:boot 2024-03-28T17:58:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:27 smithi045 ceph-mon[31773]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-03-28T17:58:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:27 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:58:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:27 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:27.900 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 360 sec latency 475796.936 ms 2024-03-28T17:58:28.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:28 smithi071 ceph-mon[36597]: pgmap v320: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:58:28.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:28 smithi151 ceph-mon[36481]: pgmap v320: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:58:28.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:28 smithi045 ceph-mon[31773]: pgmap v320: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:58:28.900 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 361 sec latency 476797.076 ms 2024-03-28T17:58:29.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:29 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/181516854,v1:172.21.15.45:6837/181516854] up:resolve 2024-03-28T17:58:29.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:29 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:29.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:29 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/181516854,v1:172.21.15.45:6837/181516854] up:resolve 2024-03-28T17:58:29.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:29 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:29.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:29 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/181516854,v1:172.21.15.45:6837/181516854] up:resolve 2024-03-28T17:58:29.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:29 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:29.900 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 362 sec latency 477797.201 ms 2024-03-28T17:58:30.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:30 smithi045 ceph-mon[31773]: pgmap v321: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:58:30.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:30 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T17:58:30.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:30 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/181516854,v1:172.21.15.45:6837/181516854] up:reconnect 2024-03-28T17:58:30.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:30 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:30.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:30 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T17:58:30.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:30 smithi071 ceph-mon[36597]: pgmap v321: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:58:30.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:30 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T17:58:30.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:30 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/181516854,v1:172.21.15.45:6837/181516854] up:reconnect 2024-03-28T17:58:30.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:30 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:30.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:30 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T17:58:30.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:30 smithi151 ceph-mon[36481]: pgmap v321: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T17:58:30.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:30 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T17:58:30.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:30 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/181516854,v1:172.21.15.45:6837/181516854] up:reconnect 2024-03-28T17:58:30.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:30 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:30.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:30 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T17:58:30.900 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 363 sec latency 478797.329 ms 2024-03-28T17:58:31.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/181516854,v1:172.21.15.45:6837/181516854] up:rejoin 2024-03-28T17:58:31.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:31.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-mon[31773]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-03-28T17:58:31.667 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f9730c00700 time 2024-03-28T17:58:31.363035+0000 2024-03-28T17:58:31.667 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:58:31.668 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:31.668 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f973c41a4a1] 2024-03-28T17:58:31.668 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f973c41a667] 2024-03-28T17:58:31.668 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556cbb5a3223] 2024-03-28T17:58:31.668 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556cbb41d5d6] 2024-03-28T17:58:31.668 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556cbb41dda1] 2024-03-28T17:58:31.668 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556cbb4202d5] 2024-03-28T17:58:31.668 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556cbb470233] 2024-03-28T17:58:31.668 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556cbb472197] 2024-03-28T17:58:31.668 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556cbb37111b] 2024-03-28T17:58:31.669 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556cbb3716bc] 2024-03-28T17:58:31.669 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556cbb35b1ff] 2024-03-28T17:58:31.669 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f973c6ac488] 2024-03-28T17:58:31.669 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 13: (DispatchQueue::entry()+0x51f) [0x7f973c6a960f] 2024-03-28T17:58:31.669 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f973c77ce81] 2024-03-28T17:58:31.669 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f973b1971ca] 2024-03-28T17:58:31.669 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 16: clone() 2024-03-28T17:58:31.669 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: *** Caught signal (Aborted) ** 2024-03-28T17:58:31.669 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: in thread 7f9730c00700 thread_name:ms_dispatch 2024-03-28T17:58:31.670 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 2024-03-28T17:58:31.365+0000 7f9730c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f9730c00700 time 2024-03-28T17:58:31.363035+0000 2024-03-28T17:58:31.670 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:58:31.670 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 2024-03-28T17:58:31.670 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:31.670 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f973c41a4a1] 2024-03-28T17:58:31.671 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f973c41a667] 2024-03-28T17:58:31.671 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556cbb5a3223] 2024-03-28T17:58:31.671 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556cbb41d5d6] 2024-03-28T17:58:31.671 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556cbb41dda1] 2024-03-28T17:58:31.671 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556cbb4202d5] 2024-03-28T17:58:31.671 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556cbb470233] 2024-03-28T17:58:31.671 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556cbb472197] 2024-03-28T17:58:31.671 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556cbb37111b] 2024-03-28T17:58:31.671 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556cbb3716bc] 2024-03-28T17:58:31.671 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556cbb35b1ff] 2024-03-28T17:58:31.672 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f973c6ac488] 2024-03-28T17:58:31.672 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 13: (DispatchQueue::entry()+0x51f) [0x7f973c6a960f] 2024-03-28T17:58:31.672 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f973c77ce81] 2024-03-28T17:58:31.672 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f973b1971ca] 2024-03-28T17:58:31.672 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 16: clone() 2024-03-28T17:58:31.672 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 2024-03-28T17:58:31.672 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:31.672 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f973b1a1d20] 2024-03-28T17:58:31.673 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 2: gsignal() 2024-03-28T17:58:31.673 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 3: abort() 2024-03-28T17:58:31.673 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f973c41a4fb] 2024-03-28T17:58:31.673 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f973c41a667] 2024-03-28T17:58:31.673 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556cbb5a3223] 2024-03-28T17:58:31.674 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556cbb41d5d6] 2024-03-28T17:58:31.674 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556cbb41dda1] 2024-03-28T17:58:31.674 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556cbb4202d5] 2024-03-28T17:58:31.674 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556cbb470233] 2024-03-28T17:58:31.674 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556cbb472197] 2024-03-28T17:58:31.674 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556cbb37111b] 2024-03-28T17:58:31.674 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556cbb3716bc] 2024-03-28T17:58:31.674 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556cbb35b1ff] 2024-03-28T17:58:31.674 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f973c6ac488] 2024-03-28T17:58:31.675 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 16: (DispatchQueue::entry()+0x51f) [0x7f973c6a960f] 2024-03-28T17:58:31.675 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f973c77ce81] 2024-03-28T17:58:31.675 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f973b1971ca] 2024-03-28T17:58:31.675 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 19: clone() 2024-03-28T17:58:31.675 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 2024-03-28T17:58:31.369+0000 7f9730c00700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:58:31.675 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: in thread 7f9730c00700 thread_name:ms_dispatch 2024-03-28T17:58:31.675 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 2024-03-28T17:58:31.675 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:31.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f973b1a1d20] 2024-03-28T17:58:31.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 2: gsignal() 2024-03-28T17:58:31.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 3: abort() 2024-03-28T17:58:31.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f973c41a4fb] 2024-03-28T17:58:31.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f973c41a667] 2024-03-28T17:58:31.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556cbb5a3223] 2024-03-28T17:58:31.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556cbb41d5d6] 2024-03-28T17:58:31.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556cbb41dda1] 2024-03-28T17:58:31.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556cbb4202d5] 2024-03-28T17:58:31.676 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556cbb470233] 2024-03-28T17:58:31.677 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556cbb472197] 2024-03-28T17:58:31.677 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556cbb37111b] 2024-03-28T17:58:31.677 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556cbb3716bc] 2024-03-28T17:58:31.677 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556cbb35b1ff] 2024-03-28T17:58:31.677 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f973c6ac488] 2024-03-28T17:58:31.677 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 16: (DispatchQueue::entry()+0x51f) [0x7f973c6a960f] 2024-03-28T17:58:31.677 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f973c77ce81] 2024-03-28T17:58:31.677 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f973b1971ca] 2024-03-28T17:58:31.677 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 19: clone() 2024-03-28T17:58:31.678 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:58:31.678 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 2024-03-28T17:58:31.678 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: -1> 2024-03-28T17:58:31.365+0000 7f9730c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f9730c00700 time 2024-03-28T17:58:31.363035+0000 2024-03-28T17:58:31.678 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:58:31.678 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 2024-03-28T17:58:31.678 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:31.678 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f973c41a4a1] 2024-03-28T17:58:31.678 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f973c41a667] 2024-03-28T17:58:31.679 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556cbb5a3223] 2024-03-28T17:58:31.679 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556cbb41d5d6] 2024-03-28T17:58:31.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556cbb41dda1] 2024-03-28T17:58:31.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556cbb4202d5] 2024-03-28T17:58:31.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556cbb470233] 2024-03-28T17:58:31.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556cbb472197] 2024-03-28T17:58:31.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556cbb37111b] 2024-03-28T17:58:31.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556cbb3716bc] 2024-03-28T17:58:31.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556cbb35b1ff] 2024-03-28T17:58:31.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f973c6ac488] 2024-03-28T17:58:31.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 13: (DispatchQueue::entry()+0x51f) [0x7f973c6a960f] 2024-03-28T17:58:31.680 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f973c77ce81] 2024-03-28T17:58:31.681 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f973b1971ca] 2024-03-28T17:58:31.681 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 16: clone() 2024-03-28T17:58:31.681 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 2024-03-28T17:58:31.681 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 0> 2024-03-28T17:58:31.369+0000 7f9730c00700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:58:31.681 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: in thread 7f9730c00700 thread_name:ms_dispatch 2024-03-28T17:58:31.681 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 2024-03-28T17:58:31.681 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:31.681 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f973b1a1d20] 2024-03-28T17:58:31.681 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 2: gsignal() 2024-03-28T17:58:31.681 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 3: abort() 2024-03-28T17:58:31.682 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f973c41a4fb] 2024-03-28T17:58:31.682 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f973c41a667] 2024-03-28T17:58:31.682 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556cbb5a3223] 2024-03-28T17:58:31.682 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556cbb41d5d6] 2024-03-28T17:58:31.682 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556cbb41dda1] 2024-03-28T17:58:31.682 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556cbb4202d5] 2024-03-28T17:58:31.682 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556cbb470233] 2024-03-28T17:58:31.682 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556cbb472197] 2024-03-28T17:58:31.682 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556cbb37111b] 2024-03-28T17:58:31.682 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556cbb3716bc] 2024-03-28T17:58:31.683 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556cbb35b1ff] 2024-03-28T17:58:31.683 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f973c6ac488] 2024-03-28T17:58:31.683 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 16: (DispatchQueue::entry()+0x51f) [0x7f973c6a960f] 2024-03-28T17:58:31.683 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f973c77ce81] 2024-03-28T17:58:31.683 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f973b1971ca] 2024-03-28T17:58:31.683 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 19: clone() 2024-03-28T17:58:31.683 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:58:31.683 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 2024-03-28T17:58:31.683 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: -1> 2024-03-28T17:58:31.365+0000 7f9730c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f9730c00700 time 2024-03-28T17:58:31.363035+0000 2024-03-28T17:58:31.683 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:58:31.683 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 2024-03-28T17:58:31.683 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:31.684 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f973c41a4a1] 2024-03-28T17:58:31.684 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f973c41a667] 2024-03-28T17:58:31.684 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556cbb5a3223] 2024-03-28T17:58:31.684 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556cbb41d5d6] 2024-03-28T17:58:31.684 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556cbb41dda1] 2024-03-28T17:58:31.684 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556cbb4202d5] 2024-03-28T17:58:31.684 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556cbb470233] 2024-03-28T17:58:31.684 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556cbb472197] 2024-03-28T17:58:31.684 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556cbb37111b] 2024-03-28T17:58:31.684 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556cbb3716bc] 2024-03-28T17:58:31.685 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556cbb35b1ff] 2024-03-28T17:58:31.685 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f973c6ac488] 2024-03-28T17:58:31.685 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 13: (DispatchQueue::entry()+0x51f) [0x7f973c6a960f] 2024-03-28T17:58:31.685 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f973c77ce81] 2024-03-28T17:58:31.685 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f973b1971ca] 2024-03-28T17:58:31.685 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 16: clone() 2024-03-28T17:58:31.685 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 2024-03-28T17:58:31.685 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 0> 2024-03-28T17:58:31.369+0000 7f9730c00700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:58:31.685 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: in thread 7f9730c00700 thread_name:ms_dispatch 2024-03-28T17:58:31.685 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 2024-03-28T17:58:31.685 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:31.686 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f973b1a1d20] 2024-03-28T17:58:31.686 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 2: gsignal() 2024-03-28T17:58:31.686 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 3: abort() 2024-03-28T17:58:31.686 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f973c41a4fb] 2024-03-28T17:58:31.686 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f973c41a667] 2024-03-28T17:58:31.686 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556cbb5a3223] 2024-03-28T17:58:31.686 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556cbb41d5d6] 2024-03-28T17:58:31.686 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556cbb41dda1] 2024-03-28T17:58:31.686 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556cbb4202d5] 2024-03-28T17:58:31.687 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556cbb470233] 2024-03-28T17:58:31.687 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556cbb472197] 2024-03-28T17:58:31.687 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556cbb37111b] 2024-03-28T17:58:31.687 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556cbb3716bc] 2024-03-28T17:58:31.687 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556cbb35b1ff] 2024-03-28T17:58:31.687 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f973c6ac488] 2024-03-28T17:58:31.687 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 16: (DispatchQueue::entry()+0x51f) [0x7f973c6a960f] 2024-03-28T17:58:31.687 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f973c77ce81] 2024-03-28T17:58:31.687 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f973b1971ca] 2024-03-28T17:58:31.687 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 19: clone() 2024-03-28T17:58:31.687 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:58:31.688 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92408]: 2024-03-28T17:58:31.688 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:31 smithi045 podman[92588]: 2024-03-28 17:58:31.473544765 +0000 UTC m=+0.037974641 container died 6405899b7c3c3459ef7d1bd443a25713af2faafb75f135de0486395153e3f70e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T17:58:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:31 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/181516854,v1:172.21.15.45:6837/181516854] up:rejoin 2024-03-28T17:58:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:31 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:31 smithi071 ceph-mon[36597]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-03-28T17:58:31.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:31 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/181516854,v1:172.21.15.45:6837/181516854] up:rejoin 2024-03-28T17:58:31.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:31 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:31.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:31 smithi151 ceph-mon[36481]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-03-28T17:58:31.900 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 364 sec latency 479797.460 ms 2024-03-28T17:58:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:32 smithi045 ceph-mon[31773]: pgmap v322: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T17:58:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:32 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:58:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:32 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:58:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:32 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/181516854,v1:172.21.15.45:6837/181516854] up:active 2024-03-28T17:58:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:32 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:32 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:58:32.667 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:32 smithi045 podman[92588]: 2024-03-28 17:58:32.408558533 +0000 UTC m=+0.972988409 container remove 6405899b7c3c3459ef7d1bd443a25713af2faafb75f135de0486395153e3f70e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-28T17:58:32.667 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:32 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:58:32.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:32 smithi071 ceph-mon[36597]: pgmap v322: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T17:58:32.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:32 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:58:32.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:32 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:58:32.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:32 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/181516854,v1:172.21.15.45:6837/181516854] up:active 2024-03-28T17:58:32.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:32 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:32.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:32 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:58:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:32 smithi151 ceph-mon[36481]: pgmap v322: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T17:58:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:32 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:58:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:32 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:58:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:32 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/181516854,v1:172.21.15.45:6837/181516854] up:active 2024-03-28T17:58:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:32 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:32 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:58:32.901 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 365 sec latency 480797.586 ms 2024-03-28T17:58:33.391 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:33 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Failed with result 'exit-code'. 2024-03-28T17:58:33.392 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:33 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Consumed 1.664s CPU time. 2024-03-28T17:58:33.901 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 366 sec latency 481797.734 ms 2024-03-28T17:58:34.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:34 smithi045 ceph-mon[31773]: pgmap v323: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T17:58:34.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:34 smithi071 ceph-mon[36597]: pgmap v323: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T17:58:34.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:34 smithi151 ceph-mon[36481]: pgmap v323: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T17:58:34.901 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 367 sec latency 482797.884 ms 2024-03-28T17:58:35.901 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 368 sec latency 483798.016 ms 2024-03-28T17:58:36.299 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:58:36.300 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:58:36.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.547+0000 7f9b9f200640 1 -- 172.21.15.45:0/2855363890 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ba0074390 msgr2=0x7f9ba010d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:58:36.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.547+0000 7f9b9f200640 1 --2- 172.21.15.45:0/2855363890 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ba0074390 0x7f9ba010d870 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f9b88009900 tx=0x7f9b8802bcd0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.547+0000 7f9b9f200640 1 -- 172.21.15.45:0/2855363890 shutdown_connections 2024-03-28T17:58:36.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.547+0000 7f9b9f200640 1 --2- 172.21.15.45:0/2855363890 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ba010ddb0 0x7f9ba01101c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.547+0000 7f9b9f200640 1 --2- 172.21.15.45:0/2855363890 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ba0074390 0x7f9ba010d870 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.547+0000 7f9b9f200640 1 --2- 172.21.15.45:0/2855363890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ba0073a70 0x7f9ba0073e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.547+0000 7f9b9f200640 1 -- 172.21.15.45:0/2855363890 >> 172.21.15.45:0/2855363890 conn(0x7f9ba00fd000 msgr2=0x7f9ba00ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:58:36.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.547+0000 7f9b9f200640 1 -- 172.21.15.45:0/2855363890 shutdown_connections 2024-03-28T17:58:36.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.547+0000 7f9b9f200640 1 -- 172.21.15.45:0/2855363890 wait complete. 2024-03-28T17:58:36.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.548+0000 7f9b9f200640 1 Processor -- start 2024-03-28T17:58:36.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.548+0000 7f9b9f200640 1 -- start start 2024-03-28T17:58:36.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.548+0000 7f9b9f200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ba0073a70 0x7f9ba019a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:36.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.548+0000 7f9b9f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ba0074390 0x7f9ba019a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:36.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.548+0000 7f9b9f200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ba010ddb0 0x7f9ba019e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:36.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.548+0000 7f9b9f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba0113b60 con 0x7f9ba0074390 2024-03-28T17:58:36.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.548+0000 7f9b9f200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba01139e0 con 0x7f9ba0073a70 2024-03-28T17:58:36.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.548+0000 7f9b9f200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba0113ce0 con 0x7f9ba010ddb0 2024-03-28T17:58:36.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.549+0000 7f9b9de00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ba0073a70 0x7f9ba019a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:36.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.549+0000 7f9b95400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ba0074390 0x7f9ba019a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:36.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.549+0000 7f9b9de00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ba0073a70 0x7f9ba019a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:57052/0 (socket says 172.21.15.45:57052) 2024-03-28T17:58:36.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.549+0000 7f9b9de00640 1 -- 172.21.15.45:0/1666135652 learned_addr learned my addr 172.21.15.45:0/1666135652 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:58:36.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.549+0000 7f9b9e800640 1 --2- 172.21.15.45:0/1666135652 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ba010ddb0 0x7f9ba019e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:36.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.549+0000 7f9b9de00640 1 -- 172.21.15.45:0/1666135652 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ba010ddb0 msgr2=0x7f9ba019e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:58:36.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.549+0000 7f9b9de00640 1 --2- 172.21.15.45:0/1666135652 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ba010ddb0 0x7f9ba019e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.549+0000 7f9b9de00640 1 -- 172.21.15.45:0/1666135652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ba0074390 msgr2=0x7f9ba019a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:58:36.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.549+0000 7f9b9de00640 1 --2- 172.21.15.45:0/1666135652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ba0074390 0x7f9ba019a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.549+0000 7f9b9de00640 1 -- 172.21.15.45:0/1666135652 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba019f060 con 0x7f9ba0073a70 2024-03-28T17:58:36.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.549+0000 7f9b9de00640 1 --2- 172.21.15.45:0/1666135652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ba0073a70 0x7f9ba019a0c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9b9000ebc0 tx=0x7f9b9000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:58:36.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.549+0000 7f9b97400640 1 -- 172.21.15.45:0/1666135652 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b90010630 con 0x7f9ba0073a70 2024-03-28T17:58:36.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.549+0000 7f9b97400640 1 -- 172.21.15.45:0/1666135652 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b90010cb0 con 0x7f9ba0073a70 2024-03-28T17:58:36.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.550+0000 7f9b9f200640 1 -- 172.21.15.45:0/1666135652 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba019f350 con 0x7f9ba0073a70 2024-03-28T17:58:36.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.550+0000 7f9b97400640 1 -- 172.21.15.45:0/1666135652 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b90018bd0 con 0x7f9ba0073a70 2024-03-28T17:58:36.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.550+0000 7f9b9f200640 1 -- 172.21.15.45:0/1666135652 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba01a6be0 con 0x7f9ba0073a70 2024-03-28T17:58:36.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.553+0000 7f9b97400640 1 -- 172.21.15.45:0/1666135652 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f9b900107d0 con 0x7f9ba0073a70 2024-03-28T17:58:36.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.553+0000 7f9ba7bf4740 1 Processor -- start 2024-03-28T17:58:36.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.553+0000 7f9ba7bf4740 1 -- start start 2024-03-28T17:58:36.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.553+0000 7f9ba7bf4740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56373b0b0dd0 0x56373b0afb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:36.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.553+0000 7f9ba7bf4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b00f0 0x56373b0b0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:36.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.553+0000 7f9ba7bf4740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b9460 0x56373b0bb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:36.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.553+0000 7f9ba7bf4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56373b014250 con 0x56373b0b00f0 2024-03-28T17:58:36.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.553+0000 7f9ba7bf4740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x56373afbca30 con 0x56373b0b9460 2024-03-28T17:58:36.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.553+0000 7f9ba7bf4740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x56373b0102f0 con 0x56373b0b0dd0 2024-03-28T17:58:36.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.554+0000 7f9b9e800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56373b0b0dd0 0x56373b0afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:36.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.554+0000 7f9b95400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b9460 0x56373b0bb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:36.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.554+0000 7f9b9de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b00f0 0x56373b0b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:36.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.554+0000 7f9b9e800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56373b0b0dd0 0x56373b0afb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:42824/0 (socket says 172.21.15.45:42824) 2024-03-28T17:58:36.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.554+0000 7f9b9e800640 1 -- 172.21.15.45:0/863648069 learned_addr learned my addr 172.21.15.45:0/863648069 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:58:36.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.554+0000 7f9b95400640 1 -- 172.21.15.45:0/863648069 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56373b0b0dd0 msgr2=0x56373b0afb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:58:36.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.554+0000 7f9b95400640 1 --2- 172.21.15.45:0/863648069 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56373b0b0dd0 0x56373b0afb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.554+0000 7f9b95400640 1 -- 172.21.15.45:0/863648069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b00f0 msgr2=0x56373b0b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:58:36.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.554+0000 7f9b95400640 1 --2- 172.21.15.45:0/863648069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b00f0 0x56373b0b0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.554+0000 7f9b95400640 1 -- 172.21.15.45:0/863648069 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56373afa7690 con 0x56373b0b9460 2024-03-28T17:58:36.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.554+0000 7f9b97400640 1 --2- 172.21.15.45:0/1666135652 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f9b840a2970 0x7f9b840a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:36.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.555+0000 7f9b95400640 1 --2- 172.21.15.45:0/863648069 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b9460 0x56373b0bb850 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9b8802c210 tx=0x7f9b880095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66d476df72406cd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:58:36.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.555+0000 7f9b97400640 1 -- 172.21.15.45:0/1666135652 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 11006+0+0 (secure 0 0 0) 0x7f9b900cff10 con 0x7f9ba0073a70 2024-03-28T17:58:36.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.555+0000 7f9b9e800640 1 --2- 172.21.15.45:0/1666135652 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f9b840a2970 0x7f9b840a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:36.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.555+0000 7f9b8f400640 1 -- 172.21.15.45:0/863648069 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b88043070 con 0x56373b0b9460 2024-03-28T17:58:36.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.555+0000 7f9b8f400640 1 -- 172.21.15.45:0/863648069 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b8802c920 con 0x56373b0b9460 2024-03-28T17:58:36.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.555+0000 7f9b8f400640 1 -- 172.21.15.45:0/863648069 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b88004ac0 con 0x56373b0b9460 2024-03-28T17:58:36.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.556+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/863648069 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b9460 msgr2=0x56373b0bb850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:58:36.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.556+0000 7f9ba7bf4740 1 --2- 172.21.15.45:0/863648069 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b9460 0x56373b0bb850 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9b8802c210 tx=0x7f9b880095e0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.556+0000 7f9b9e800640 1 --2- 172.21.15.45:0/1666135652 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f9b840a2970 0x7f9b840a4e30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9b9800dc90 tx=0x7f9b9800d440 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:58:36.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.556+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/863648069 shutdown_connections 2024-03-28T17:58:36.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.556+0000 7f9ba7bf4740 1 --2- 172.21.15.45:0/863648069 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b9460 0x56373b0bb850 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.556+0000 7f9ba7bf4740 1 --2- 172.21.15.45:0/863648069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b00f0 0x56373b0b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.556+0000 7f9ba7bf4740 1 --2- 172.21.15.45:0/863648069 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56373b0b0dd0 0x56373b0afb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.556+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/863648069 >> 172.21.15.45:0/863648069 conn(0x56373b077200 msgr2=0x56373b0aab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:58:36.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.556+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/863648069 shutdown_connections 2024-03-28T17:58:36.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.556+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/863648069 wait complete. 2024-03-28T17:58:36.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.556+0000 7f9ba7bf4740 1 -- start start 2024-03-28T17:58:36.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.557+0000 7f9ba7bf4740 1 Processor -- start 2024-03-28T17:58:36.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.557+0000 7f9ba7bf4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b00f0 0x56373b0b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:36.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.557+0000 7f9ba7bf4740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56373b0b0dd0 0x56373b0b5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:36.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.557+0000 7f9ba7bf4740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b9460 0x56373b0b63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:36.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.557+0000 7f9ba7bf4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56373b0bc270 con 0x56373b0b00f0 2024-03-28T17:58:36.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.557+0000 7f9ba7bf4740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x56373afcfe80 con 0x56373b0b9460 2024-03-28T17:58:36.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.557+0000 7f9ba7bf4740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x56373b0bc3f0 con 0x56373b0b0dd0 2024-03-28T17:58:36.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.557+0000 7f9ba7bf4740 10 client.0 _ll_register_callbacks cb 0x56373afe2130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:58:36.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.557+0000 7f9b9e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b9460 0x56373b0b63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:36.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.557+0000 7f9b9e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b9460 0x56373b0b63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:57068/0 (socket says 172.21.15.45:57068) 2024-03-28T17:58:36.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.557+0000 7f9b9e800640 1 -- 172.21.15.45:0/1849818316 learned_addr learned my addr 172.21.15.45:0/1849818316 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:58:36.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.557+0000 7f9b9de00640 1 --2- 172.21.15.45:0/1849818316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b00f0 0x56373b0b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:36.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.557+0000 7f9b95400640 1 --2- 172.21.15.45:0/1849818316 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56373b0b0dd0 0x56373b0b5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:36.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.558+0000 7f9b9e800640 1 -- 172.21.15.45:0/1849818316 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56373b0b0dd0 msgr2=0x56373b0b5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:58:36.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.558+0000 7f9b9e800640 1 --2- 172.21.15.45:0/1849818316 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56373b0b0dd0 0x56373b0b5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.558+0000 7f9b9e800640 1 -- 172.21.15.45:0/1849818316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b00f0 msgr2=0x56373b0b8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:58:36.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.558+0000 7f9b9e800640 1 --2- 172.21.15.45:0/1849818316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b00f0 0x56373b0b8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.558+0000 7f9b9e800640 1 -- 172.21.15.45:0/1849818316 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56373adf27d0 con 0x56373b0b9460 2024-03-28T17:58:36.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.558+0000 7f9b9de00640 1 --2- 172.21.15.45:0/1849818316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b00f0 0x56373b0b8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:58:36.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.558+0000 7f9b9e800640 1 --2- 172.21.15.45:0/1849818316 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b9460 0x56373b0b63f0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9b980058c0 tx=0x7f9b980046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa9751f4deb1e300 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:58:36.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.558+0000 7f9b83e00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:58:36.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.558+0000 7f9b83e00640 1 -- 172.21.15.45:0/1849818316 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b98029680 con 0x56373b0b9460 2024-03-28T17:58:36.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.558+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/1849818316 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9b8802c920 con 0x56373b0b9460 2024-03-28T17:58:36.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.558+0000 7f9b83e00640 1 -- 172.21.15.45:0/1849818316 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(121..121 src has 1..121) v4 ==== 11006+0+0 (secure 0 0 0) 0x7f9b9801e410 con 0x56373b0b9460 2024-03-28T17:58:36.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.559+0000 7f9b83e00640 1 -- 172.21.15.45:0/1849818316 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=214) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b98029820 con 0x56373b0b9460 2024-03-28T17:58:36.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.559+0000 7f9ba7bf4740 10 client.26333 fetch_fsmap learned FSMap version 214 2024-03-28T17:58:36.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.559+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/1849818316 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=214,monmap=4+}) v3 -- 0x56373ad283f0 con 0x56373b0b9460 2024-03-28T17:58:36.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.559+0000 7f9b83e00640 1 -- 172.21.15.45:0/1849818316 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b8802c920 con 0x56373b0b9460 2024-03-28T17:58:36.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.559+0000 7f9b83e00640 1 -- 172.21.15.45:0/1849818316 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 214) v1 ==== 5618+0+0 (secure 0 0 0) 0x7f9b9801b390 con 0x56373b0b9460 2024-03-28T17:58:36.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.559+0000 7f9ba7bf4740 10 client.26333 fetch_fsmap finished waiting for FSMap version 214 2024-03-28T17:58:36.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.559+0000 7f9ba7bf4740 10 client.26333 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:58:36.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.559+0000 7f9ba7bf4740 20 client.26333 populate_metadata read hostname 'smithi045' 2024-03-28T17:58:36.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.559+0000 7f9ba7bf4740 1 --2- 172.21.15.45:0/1849818316 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56373b0c9530 0x56373b0cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:36.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.559+0000 7f9ba7bf4740 4 client.26333 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:58:36.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.559+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/1849818316 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56373b021790 con 0x56373b0c9530 2024-03-28T17:58:36.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.560+0000 7f9b9de00640 1 --2- 172.21.15.45:0/1849818316 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56373b0c9530 0x56373b0cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:36.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.560+0000 7f9b9de00640 1 --2- 172.21.15.45:0/1849818316 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56373b0c9530 0x56373b0cb9b0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0f5e2e1fa3104a1 server_cookie=e83b184ede88f289 in_seq=0 out_seq=0 2024-03-28T17:58:36.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.560+0000 7f9b83e00640 10 client.26333 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:58:36.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.561+0000 7f9b83e00640 1 -- 172.21.15.45:0/1849818316 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56373b021790 con 0x56373b0c9530 2024-03-28T17:58:36.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.561+0000 7f9b83e00640 10 client.26333 handle_command_reply: tid=0 2024-03-28T17:58:36.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.561+0000 7f9ba7bf4740 1 client.26333 shutdown 2024-03-28T17:58:36.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.561+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/1849818316 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b9460 msgr2=0x56373b0b63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:58:36.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.561+0000 7f9ba7bf4740 1 --2- 172.21.15.45:0/1849818316 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b9460 0x56373b0b63f0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9b980058c0 tx=0x7f9b980046c0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.561+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/1849818316 shutdown_connections 2024-03-28T17:58:36.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.561+0000 7f9ba7bf4740 1 --2- 172.21.15.45:0/1849818316 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56373b0c9530 0x56373b0cb9b0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.561+0000 7f9ba7bf4740 1 --2- 172.21.15.45:0/1849818316 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b9460 0x56373b0b63f0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.561+0000 7f9ba7bf4740 1 --2- 172.21.15.45:0/1849818316 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56373b0b0dd0 0x56373b0b5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.561+0000 7f9b83e00640 0 client.26333 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:58:36.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.561+0000 7f9ba7bf4740 1 --2- 172.21.15.45:0/1849818316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b00f0 0x56373b0b8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.561+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/1849818316 >> 172.21.15.45:0/1849818316 conn(0x56373b077200 msgr2=0x56373b0ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:58:36.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.561+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/1849818316 shutdown_connections 2024-03-28T17:58:36.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.561+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/1849818316 wait complete. 2024-03-28T17:58:36.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.561+0000 7f9ba7bf4740 20 client.26333 trim_cache size 0 max 16384 2024-03-28T17:58:36.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.579+0000 7f9ba7bf4740 1 Processor -- start 2024-03-28T17:58:36.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.580+0000 7f9ba7bf4740 1 -- start start 2024-03-28T17:58:36.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.580+0000 7f9ba7bf4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b00f0 0x56373b0b5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:36.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.580+0000 7f9ba7bf4740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b0dd0 0x56373b0ba850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:36.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.580+0000 7f9ba7bf4740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56373b0b9460 0x56373b0bad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.580+0000 7f9ba7bf4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba01b00c0 con 0x56373b0b00f0 2024-03-28T17:58:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.580+0000 7f9ba7bf4740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba01b1a40 con 0x56373b0b0dd0 2024-03-28T17:58:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.580+0000 7f9ba7bf4740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba01b0720 con 0x56373b0b9460 2024-03-28T17:58:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.580+0000 7f9b9de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b00f0 0x56373b0b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.580+0000 7f9b95400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b0dd0 0x56373b0ba850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.580+0000 7f9b9de00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b00f0 0x56373b0b5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45494/0 (socket says 172.21.15.45:45494) 2024-03-28T17:58:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.580+0000 7f9b9de00640 1 -- 172.21.15.45:0/2859265719 learned_addr learned my addr 172.21.15.45:0/2859265719 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:58:36.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.580+0000 7f9b9e800640 1 --2- 172.21.15.45:0/2859265719 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56373b0b9460 0x56373b0bad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:36.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.581+0000 7f9b95400640 1 -- 172.21.15.45:0/2859265719 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56373b0b9460 msgr2=0x56373b0bad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:58:36.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.581+0000 7f9b95400640 1 --2- 172.21.15.45:0/2859265719 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56373b0b9460 0x56373b0bad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.581+0000 7f9b95400640 1 -- 172.21.15.45:0/2859265719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b00f0 msgr2=0x56373b0b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:58:36.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.581+0000 7f9b95400640 1 --2- 172.21.15.45:0/2859265719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b00f0 0x56373b0b5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.581+0000 7f9b95400640 1 -- 172.21.15.45:0/2859265719 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56373ace80f0 con 0x56373b0b0dd0 2024-03-28T17:58:36.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.581+0000 7f9b9de00640 1 --2- 172.21.15.45:0/2859265719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b00f0 0x56373b0b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:58:36.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.581+0000 7f9b95400640 1 --2- 172.21.15.45:0/2859265719 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b0dd0 0x56373b0ba850 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x56373b0aa4f0 tx=0x7f9b88000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=559842cb28328e13 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:58:36.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.581+0000 7f9b8ea00640 1 -- 172.21.15.45:0/2859265719 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b88043040 con 0x56373b0b0dd0 2024-03-28T17:58:36.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.581+0000 7f9b8ea00640 1 -- 172.21.15.45:0/2859265719 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b8804c030 con 0x56373b0b0dd0 2024-03-28T17:58:36.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.581+0000 7f9b8ea00640 1 -- 172.21.15.45:0/2859265719 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b88005140 con 0x56373b0b0dd0 2024-03-28T17:58:36.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.582+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/2859265719 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b0dd0 msgr2=0x56373b0ba850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:58:36.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.582+0000 7f9ba7bf4740 1 --2- 172.21.15.45:0/2859265719 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b0dd0 0x56373b0ba850 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x56373b0aa4f0 tx=0x7f9b88000bc0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.582+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/2859265719 shutdown_connections 2024-03-28T17:58:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.582+0000 7f9ba7bf4740 1 --2- 172.21.15.45:0/2859265719 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56373b0b9460 0x56373b0bad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.582+0000 7f9ba7bf4740 1 --2- 172.21.15.45:0/2859265719 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b0dd0 0x56373b0ba850 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.582+0000 7f9ba7bf4740 1 --2- 172.21.15.45:0/2859265719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b00f0 0x56373b0b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.582+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/2859265719 >> 172.21.15.45:0/2859265719 conn(0x56373b077200 msgr2=0x56373b0aa670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:58:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.582+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/2859265719 shutdown_connections 2024-03-28T17:58:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.582+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/2859265719 wait complete. 2024-03-28T17:58:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.583+0000 7f9ba7bf4740 1 -- start start 2024-03-28T17:58:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.583+0000 7f9ba7bf4740 1 Processor -- start 2024-03-28T17:58:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.583+0000 7f9ba7bf4740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56373b0b00f0 0x56373b0b3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.583+0000 7f9ba7bf4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b0dd0 0x56373b0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.583+0000 7f9ba7bf4740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b9460 0x56373b0b2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:36.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.583+0000 7f9ba7bf4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba01b13e0 con 0x56373b0b0dd0 2024-03-28T17:58:36.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.583+0000 7f9ba7bf4740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba01b0d80 con 0x56373b0b9460 2024-03-28T17:58:36.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.583+0000 7f9ba7bf4740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba01aea00 con 0x56373b0b00f0 2024-03-28T17:58:36.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.583+0000 7f9ba7bf4740 10 client.0 _ll_register_callbacks cb 0x7f9b9800d8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:58:36.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.583+0000 7f9b9de00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56373b0b00f0 0x56373b0b3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:36.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.583+0000 7f9b9e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b9460 0x56373b0b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:36.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.583+0000 7f9b9de00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56373b0b00f0 0x56373b0b3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:42852/0 (socket says 172.21.15.45:42852) 2024-03-28T17:58:36.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.584+0000 7f9b9de00640 1 -- 172.21.15.45:0/2818736377 learned_addr learned my addr 172.21.15.45:0/2818736377 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:58:36.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.584+0000 7f9b95400640 1 --2- 172.21.15.45:0/2818736377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b0dd0 0x56373b0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:36.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.584+0000 7f9b9de00640 1 -- 172.21.15.45:0/2818736377 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b9460 msgr2=0x56373b0b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:58:36.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.584+0000 7f9b9de00640 1 --2- 172.21.15.45:0/2818736377 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b9460 0x56373b0b2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.584+0000 7f9b9de00640 1 -- 172.21.15.45:0/2818736377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b0dd0 msgr2=0x56373b0b2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:58:36.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.584+0000 7f9b9de00640 1 --2- 172.21.15.45:0/2818736377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b0dd0 0x56373b0b2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.584+0000 7f9b9de00640 1 -- 172.21.15.45:0/2818736377 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56373b0b2cd0 con 0x56373b0b00f0 2024-03-28T17:58:36.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.584+0000 7f9b9e800640 1 --2- 172.21.15.45:0/2818736377 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b9460 0x56373b0b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:58:36.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.584+0000 7f9b95400640 1 --2- 172.21.15.45:0/2818736377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b0dd0 0x56373b0b2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:58:36.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.584+0000 7f9b9de00640 1 --2- 172.21.15.45:0/2818736377 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56373b0b00f0 0x56373b0b3a50 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9b9000cba0 tx=0x7f9b90009480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf70634ad94ee63d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:58:36.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.584+0000 7f9b83e00640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T17:58:36.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.584+0000 7f9b83e00640 1 -- 172.21.15.45:0/2818736377 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b90010e60 con 0x56373b0b00f0 2024-03-28T17:58:36.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.584+0000 7f9b83e00640 1 -- 172.21.15.45:0/2818736377 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(121..121 src has 1..121) v4 ==== 11006+0+0 (secure 0 0 0) 0x7f9b900981d0 con 0x56373b0b00f0 2024-03-28T17:58:36.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.585+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/2818736377 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9b8804c030 con 0x56373b0b00f0 2024-03-28T17:58:36.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.585+0000 7f9b83e00640 1 -- 172.21.15.45:0/2818736377 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=214) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b9009bb60 con 0x56373b0b00f0 2024-03-28T17:58:36.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.585+0000 7f9ba7bf4740 10 client.26275 fetch_fsmap learned FSMap version 214 2024-03-28T17:58:36.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.585+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/2818736377 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=214,monmap=4+}) v3 -- 0x56373b12d550 con 0x56373b0b00f0 2024-03-28T17:58:36.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.585+0000 7f9b83e00640 1 -- 172.21.15.45:0/2818736377 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b8804c030 con 0x56373b0b00f0 2024-03-28T17:58:36.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.585+0000 7f9b83e00640 1 -- 172.21.15.45:0/2818736377 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 214) v1 ==== 5618+0+0 (secure 0 0 0) 0x7f9b900d3040 con 0x56373b0b00f0 2024-03-28T17:58:36.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.585+0000 7f9ba7bf4740 10 client.26275 fetch_fsmap finished waiting for FSMap version 214 2024-03-28T17:58:36.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.585+0000 7f9ba7bf4740 10 client.26275 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:58:36.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.585+0000 7f9ba7bf4740 20 client.26275 populate_metadata read hostname 'smithi045' 2024-03-28T17:58:36.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.585+0000 7f9ba7bf4740 1 --2- 172.21.15.45:0/2818736377 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56373b12dbd0 0x56373b12ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:58:36.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.585+0000 7f9ba7bf4740 4 client.26275 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:58:36.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.585+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/2818736377 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56373b01bae0 con 0x56373b12dbd0 2024-03-28T17:58:36.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.586+0000 7f9b95400640 1 --2- 172.21.15.45:0/2818736377 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56373b12dbd0 0x56373b12ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:58:36.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.586+0000 7f9b95400640 1 --2- 172.21.15.45:0/2818736377 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56373b12dbd0 0x56373b12ffc0 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c2a612293f5be22 server_cookie=2033ff246a97d0bc in_seq=0 out_seq=0 2024-03-28T17:58:36.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.586+0000 7f9b83e00640 10 client.26275 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:58:36.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.586+0000 7f9b83e00640 1 -- 172.21.15.45:0/2818736377 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56373b01bae0 con 0x56373b12dbd0 2024-03-28T17:58:36.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.586+0000 7f9b83e00640 10 client.26275 handle_command_reply: tid=0 2024-03-28T17:58:36.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.586+0000 7f9ba7bf4740 1 client.26275 shutdown 2024-03-28T17:58:36.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.586+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/2818736377 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56373b0b00f0 msgr2=0x56373b0b3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:58:36.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.586+0000 7f9ba7bf4740 1 --2- 172.21.15.45:0/2818736377 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56373b0b00f0 0x56373b0b3a50 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9b9000cba0 tx=0x7f9b90009480 comp rx=0 tx=0).stop 2024-03-28T17:58:36.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.586+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/2818736377 shutdown_connections 2024-03-28T17:58:36.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.586+0000 7f9ba7bf4740 1 --2- 172.21.15.45:0/2818736377 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x56373b12dbd0 0x56373b12ffc0 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.587+0000 7f9ba7bf4740 1 --2- 172.21.15.45:0/2818736377 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56373b0b9460 0x56373b0b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.587+0000 7f9ba7bf4740 1 --2- 172.21.15.45:0/2818736377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56373b0b0dd0 0x56373b0b2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.587+0000 7f9ba7bf4740 1 --2- 172.21.15.45:0/2818736377 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x56373b0b00f0 0x56373b0b3a50 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.587+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/2818736377 >> 172.21.15.45:0/2818736377 conn(0x56373b077200 msgr2=0x56373b0a9330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:58:36.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.587+0000 7f9b83e00640 0 client.26275 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:58:36.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.587+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/2818736377 shutdown_connections 2024-03-28T17:58:36.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.587+0000 7f9ba7bf4740 1 -- 172.21.15.45:0/2818736377 wait complete. 2024-03-28T17:58:36.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.587+0000 7f9ba7bf4740 20 client.26275 trim_cache size 0 max 16384 2024-03-28T17:58:36.593 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:58:36.593 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T17:58:36.593 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T17:58:36.593 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T17:58:36.594 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T17:58:36.594 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T17:58:36.594 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T17:58:36.594 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:58:36.594 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:58:36.594 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:58:36.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.587+0000 7f9b9f200640 1 -- 172.21.15.45:0/1666135652 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f9b840a2970 msgr2=0x7f9b840a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:58:36.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.587+0000 7f9b9f200640 1 --2- 172.21.15.45:0/1666135652 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f9b840a2970 0x7f9b840a4e30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9b9800dc90 tx=0x7f9b9800d440 comp rx=0 tx=0).stop 2024-03-28T17:58:36.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.587+0000 7f9b9f200640 1 -- 172.21.15.45:0/1666135652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ba0073a70 msgr2=0x7f9ba019a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:58:36.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.587+0000 7f9b9f200640 1 --2- 172.21.15.45:0/1666135652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ba0073a70 0x7f9ba019a0c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9b9000ebc0 tx=0x7f9b9000c6a0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.588+0000 7f9b9f200640 1 -- 172.21.15.45:0/1666135652 shutdown_connections 2024-03-28T17:58:36.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.588+0000 7f9b9f200640 1 --2- 172.21.15.45:0/1666135652 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f9b840a2970 0x7f9b840a4e30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.588+0000 7f9b9f200640 1 --2- 172.21.15.45:0/1666135652 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ba010ddb0 0x7f9ba019e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.588+0000 7f9b9f200640 1 --2- 172.21.15.45:0/1666135652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ba0074390 0x7f9ba019a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.588+0000 7f9b9f200640 1 --2- 172.21.15.45:0/1666135652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ba0073a70 0x7f9ba019a0c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:58:36.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.588+0000 7f9b9f200640 1 -- 172.21.15.45:0/1666135652 >> 172.21.15.45:0/1666135652 conn(0x7f9ba00fd000 msgr2=0x7f9ba00fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:58:36.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.588+0000 7f9b9f200640 1 -- 172.21.15.45:0/1666135652 shutdown_connections 2024-03-28T17:58:36.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:58:36.588+0000 7f9b9f200640 1 -- 172.21.15.45:0/1666135652 wait complete. 2024-03-28T17:58:36.605 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T17:58:36.613 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:36 smithi071 ceph-mon[36597]: pgmap v324: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:58:36.649 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:36 smithi151 ceph-mon[36481]: pgmap v324: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:58:36.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:36 smithi045 ceph-mon[31773]: pgmap v324: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:58:36.901 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 369 sec latency 484798.180 ms 2024-03-28T17:58:37.901 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 370 sec latency 485798.323 ms 2024-03-28T17:58:38.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:38 smithi045 ceph-mon[31773]: pgmap v325: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:58:38.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:38 smithi071 ceph-mon[36597]: pgmap v325: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:58:38.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:38 smithi151 ceph-mon[36481]: pgmap v325: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:58:38.901 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 371 sec latency 486798.481 ms 2024-03-28T17:58:39.902 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 372 sec latency 487798.613 ms 2024-03-28T17:58:40.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:40 smithi045 ceph-mon[31773]: pgmap v326: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-03-28T17:58:40.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:40 smithi071 ceph-mon[36597]: pgmap v326: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-03-28T17:58:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:40 smithi151 ceph-mon[36481]: pgmap v326: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-03-28T17:58:40.902 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 373 sec latency 488798.790 ms 2024-03-28T17:58:41.902 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 374 sec latency 489798.888 ms 2024-03-28T17:58:42.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:42 smithi045 ceph-mon[31773]: pgmap v327: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 14 op/s 2024-03-28T17:58:42.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:42 smithi045 ceph-mon[31773]: osdmap e122: 12 total, 12 up, 12 in 2024-03-28T17:58:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:42 smithi071 ceph-mon[36597]: pgmap v327: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 14 op/s 2024-03-28T17:58:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:42 smithi071 ceph-mon[36597]: osdmap e122: 12 total, 12 up, 12 in 2024-03-28T17:58:42.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:42 smithi151 ceph-mon[36481]: pgmap v327: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 14 op/s 2024-03-28T17:58:42.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:42 smithi151 ceph-mon[36481]: osdmap e122: 12 total, 12 up, 12 in 2024-03-28T17:58:42.902 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 375 sec latency 490798.980 ms 2024-03-28T17:58:43.569 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:43 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Scheduled restart job, restart counter is at 3. 2024-03-28T17:58:43.569 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:43 smithi045 systemd[1]: Stopped Ceph mds.a for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:58:43.569 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:43 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Consumed 1.664s CPU time. 2024-03-28T17:58:43.569 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:43 smithi045 systemd[1]: Starting Ceph mds.a for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:58:43.891 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:43 smithi045 podman[92748]: 2024-03-28 17:58:43.475389405 +0000 UTC m=+0.040712720 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:58:43.892 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:43 smithi045 podman[92748]: 2024-03-28 17:58:43.745328707 +0000 UTC m=+0.310652067 container create 86686230b557b1bfae04f5d4f323dc35ad3e1db8db176cabda9f1e5cbd112c2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-28T17:58:43.902 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 376 sec latency 491799.056 ms 2024-03-28T17:58:44.389 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:44 smithi045 podman[92748]: 2024-03-28 17:58:44.014297592 +0000 UTC m=+0.579620939 container init 86686230b557b1bfae04f5d4f323dc35ad3e1db8db176cabda9f1e5cbd112c2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS) 2024-03-28T17:58:44.390 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:44 smithi045 podman[92748]: 2024-03-28 17:58:44.026248072 +0000 UTC m=+0.591571455 container start 86686230b557b1bfae04f5d4f323dc35ad3e1db8db176cabda9f1e5cbd112c2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-28T17:58:44.390 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:44 smithi045 ceph-mds[92766]: -- 172.21.15.45:0/3104207755 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55613b7f1a40 con 0x55613b7c4c00 2024-03-28T17:58:44.390 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:44 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: starting mds.a at 2024-03-28T17:58:44.390 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:44 smithi045 bash[92748]: 86686230b557b1bfae04f5d4f323dc35ad3e1db8db176cabda9f1e5cbd112c2b 2024-03-28T17:58:44.390 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:58:44 smithi045 systemd[1]: Started Ceph mds.a for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:58:44.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:44 smithi045 ceph-mon[31773]: pgmap v329: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:58:44.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:44 smithi045 ceph-mon[31773]: daemon mds.a restarted 2024-03-28T17:58:44.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:44 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:58:44.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:44 smithi045 ceph-mon[31773]: osdmap e123: 12 total, 12 up, 12 in 2024-03-28T17:58:44.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:44 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/785732331,v1:172.21.15.45:6837/785732331] up:boot 2024-03-28T17:58:44.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:44 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T17:58:44.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:44 smithi045 ceph-mon[31773]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-03-28T17:58:44.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:44 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:58:44.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:44 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:44.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:44 smithi071 ceph-mon[36597]: pgmap v329: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:58:44.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:44 smithi071 ceph-mon[36597]: daemon mds.a restarted 2024-03-28T17:58:44.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:44 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:58:44.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:44 smithi071 ceph-mon[36597]: osdmap e123: 12 total, 12 up, 12 in 2024-03-28T17:58:44.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:44 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/785732331,v1:172.21.15.45:6837/785732331] up:boot 2024-03-28T17:58:44.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:44 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T17:58:44.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:44 smithi071 ceph-mon[36597]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-03-28T17:58:44.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:44 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:58:44.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:44 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:44 smithi151 ceph-mon[36481]: pgmap v329: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:58:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:44 smithi151 ceph-mon[36481]: daemon mds.a restarted 2024-03-28T17:58:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:44 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:58:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:44 smithi151 ceph-mon[36481]: osdmap e123: 12 total, 12 up, 12 in 2024-03-28T17:58:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:44 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/785732331,v1:172.21.15.45:6837/785732331] up:boot 2024-03-28T17:58:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:44 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T17:58:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:44 smithi151 ceph-mon[36481]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-03-28T17:58:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:44 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:58:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:44 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:44.902 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 377 sec latency 492799.196 ms 2024-03-28T17:58:45.902 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 378 sec latency 493799.319 ms 2024-03-28T17:58:46.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:46 smithi045 ceph-mon[31773]: pgmap v331: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T17:58:46.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:46 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/4173171826,v1:172.21.15.151:6833/4173171826] up:resolve 2024-03-28T17:58:46.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:46 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:46 smithi071 ceph-mon[36597]: pgmap v331: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T17:58:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:46 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/4173171826,v1:172.21.15.151:6833/4173171826] up:resolve 2024-03-28T17:58:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:46 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:46 smithi151 ceph-mon[36481]: pgmap v331: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T17:58:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:46 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/4173171826,v1:172.21.15.151:6833/4173171826] up:resolve 2024-03-28T17:58:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:46 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:46.902 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 379 sec latency 494799.416 ms 2024-03-28T17:58:47.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:47 smithi071 ceph-mon[36597]: osdmap e124: 12 total, 12 up, 12 in 2024-03-28T17:58:47.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:47 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:58:47.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:47 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/4173171826,v1:172.21.15.151:6833/4173171826] up:reconnect 2024-03-28T17:58:47.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:47 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:47.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:47 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:58:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:47 smithi151 ceph-mon[36481]: osdmap e124: 12 total, 12 up, 12 in 2024-03-28T17:58:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:47 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:58:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:47 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/4173171826,v1:172.21.15.151:6833/4173171826] up:reconnect 2024-03-28T17:58:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:47 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:47 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:58:47.902 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 380 sec latency 495799.558 ms 2024-03-28T17:58:47.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:47 smithi045 ceph-mon[31773]: osdmap e124: 12 total, 12 up, 12 in 2024-03-28T17:58:47.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:47 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199998 2024-03-28T17:58:47.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:47 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/4173171826,v1:172.21.15.151:6833/4173171826] up:reconnect 2024-03-28T17:58:47.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:47 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:47.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:47 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:58:48.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:48 smithi071 ceph-mon[36597]: pgmap v333: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T17:58:48.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:48 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/4173171826,v1:172.21.15.151:6833/4173171826] up:rejoin 2024-03-28T17:58:48.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:48 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:48.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:48 smithi071 ceph-mon[36597]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-03-28T17:58:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-mon[36481]: pgmap v333: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T17:58:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/4173171826,v1:172.21.15.151:6833/4173171826] up:rejoin 2024-03-28T17:58:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-mon[36481]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-03-28T17:58:48.823 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd3fa606700 time 2024-03-28T17:58:48.441176+0000 2024-03-28T17:58:48.823 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:58:48.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:48.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd404c304a1] 2024-03-28T17:58:48.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd404c30667] 2024-03-28T17:58:48.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a58e31c223] 2024-03-28T17:58:48.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a58e1965d6] 2024-03-28T17:58:48.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a58e196da1] 2024-03-28T17:58:48.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a58e1992d5] 2024-03-28T17:58:48.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a58e1e9233] 2024-03-28T17:58:48.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a58e1eb197] 2024-03-28T17:58:48.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a58e0ea11b] 2024-03-28T17:58:48.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a58e0ea6bc] 2024-03-28T17:58:48.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a58e0d41ff] 2024-03-28T17:58:48.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd404ec2488] 2024-03-28T17:58:48.824 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 13: (DispatchQueue::entry()+0x51f) [0x7fd404ebf60f] 2024-03-28T17:58:48.825 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd404f92e81] 2024-03-28T17:58:48.825 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd4039ad1ca] 2024-03-28T17:58:48.825 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 16: clone() 2024-03-28T17:58:48.825 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 2024-03-28T17:58:48.442+0000 7fd3fa606700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd3fa606700 time 2024-03-28T17:58:48.441176+0000 2024-03-28T17:58:48.826 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:58:48.826 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 2024-03-28T17:58:48.826 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:48.826 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd404c304a1] 2024-03-28T17:58:48.826 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd404c30667] 2024-03-28T17:58:48.826 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a58e31c223] 2024-03-28T17:58:48.826 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a58e1965d6] 2024-03-28T17:58:48.826 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a58e196da1] 2024-03-28T17:58:48.826 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a58e1992d5] 2024-03-28T17:58:48.826 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a58e1e9233] 2024-03-28T17:58:48.826 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a58e1eb197] 2024-03-28T17:58:48.826 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a58e0ea11b] 2024-03-28T17:58:48.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a58e0ea6bc] 2024-03-28T17:58:48.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a58e0d41ff] 2024-03-28T17:58:48.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd404ec2488] 2024-03-28T17:58:48.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 13: (DispatchQueue::entry()+0x51f) [0x7fd404ebf60f] 2024-03-28T17:58:48.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd404f92e81] 2024-03-28T17:58:48.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd4039ad1ca] 2024-03-28T17:58:48.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 16: clone() 2024-03-28T17:58:48.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 2024-03-28T17:58:48.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: *** Caught signal (Aborted) ** 2024-03-28T17:58:48.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: in thread 7fd3fa606700 thread_name:ms_dispatch 2024-03-28T17:58:48.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:48.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd4039b7d20] 2024-03-28T17:58:48.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 2: gsignal() 2024-03-28T17:58:48.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 3: abort() 2024-03-28T17:58:48.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd404c304fb] 2024-03-28T17:58:48.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd404c30667] 2024-03-28T17:58:48.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a58e31c223] 2024-03-28T17:58:48.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a58e1965d6] 2024-03-28T17:58:48.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a58e196da1] 2024-03-28T17:58:48.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a58e1992d5] 2024-03-28T17:58:48.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a58e1e9233] 2024-03-28T17:58:48.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a58e1eb197] 2024-03-28T17:58:48.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a58e0ea11b] 2024-03-28T17:58:48.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a58e0ea6bc] 2024-03-28T17:58:48.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a58e0d41ff] 2024-03-28T17:58:48.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd404ec2488] 2024-03-28T17:58:48.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 16: (DispatchQueue::entry()+0x51f) [0x7fd404ebf60f] 2024-03-28T17:58:48.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd404f92e81] 2024-03-28T17:58:48.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd4039ad1ca] 2024-03-28T17:58:48.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 19: clone() 2024-03-28T17:58:48.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 2024-03-28T17:58:48.444+0000 7fd3fa606700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:58:48.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: in thread 7fd3fa606700 thread_name:ms_dispatch 2024-03-28T17:58:48.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 2024-03-28T17:58:48.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:48.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd4039b7d20] 2024-03-28T17:58:48.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 2: gsignal() 2024-03-28T17:58:48.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 3: abort() 2024-03-28T17:58:48.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd404c304fb] 2024-03-28T17:58:48.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd404c30667] 2024-03-28T17:58:48.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a58e31c223] 2024-03-28T17:58:48.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a58e1965d6] 2024-03-28T17:58:48.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a58e196da1] 2024-03-28T17:58:48.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a58e1992d5] 2024-03-28T17:58:48.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a58e1e9233] 2024-03-28T17:58:48.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a58e1eb197] 2024-03-28T17:58:48.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a58e0ea11b] 2024-03-28T17:58:48.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a58e0ea6bc] 2024-03-28T17:58:48.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a58e0d41ff] 2024-03-28T17:58:48.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd404ec2488] 2024-03-28T17:58:48.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 16: (DispatchQueue::entry()+0x51f) [0x7fd404ebf60f] 2024-03-28T17:58:48.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd404f92e81] 2024-03-28T17:58:48.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd4039ad1ca] 2024-03-28T17:58:48.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 19: clone() 2024-03-28T17:58:48.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:58:48.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 2024-03-28T17:58:48.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: -1> 2024-03-28T17:58:48.442+0000 7fd3fa606700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd3fa606700 time 2024-03-28T17:58:48.441176+0000 2024-03-28T17:58:48.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:58:48.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 2024-03-28T17:58:48.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:48.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd404c304a1] 2024-03-28T17:58:48.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd404c30667] 2024-03-28T17:58:48.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a58e31c223] 2024-03-28T17:58:48.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a58e1965d6] 2024-03-28T17:58:48.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a58e196da1] 2024-03-28T17:58:48.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a58e1992d5] 2024-03-28T17:58:48.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a58e1e9233] 2024-03-28T17:58:48.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a58e1eb197] 2024-03-28T17:58:48.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a58e0ea11b] 2024-03-28T17:58:48.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a58e0ea6bc] 2024-03-28T17:58:48.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a58e0d41ff] 2024-03-28T17:58:48.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd404ec2488] 2024-03-28T17:58:48.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 13: (DispatchQueue::entry()+0x51f) [0x7fd404ebf60f] 2024-03-28T17:58:48.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd404f92e81] 2024-03-28T17:58:48.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd4039ad1ca] 2024-03-28T17:58:48.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 16: clone() 2024-03-28T17:58:48.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 2024-03-28T17:58:48.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 0> 2024-03-28T17:58:48.444+0000 7fd3fa606700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:58:48.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: in thread 7fd3fa606700 thread_name:ms_dispatch 2024-03-28T17:58:48.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 2024-03-28T17:58:48.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:48.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd4039b7d20] 2024-03-28T17:58:48.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 2: gsignal() 2024-03-28T17:58:48.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 3: abort() 2024-03-28T17:58:48.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd404c304fb] 2024-03-28T17:58:48.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd404c30667] 2024-03-28T17:58:48.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a58e31c223] 2024-03-28T17:58:48.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a58e1965d6] 2024-03-28T17:58:48.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a58e196da1] 2024-03-28T17:58:48.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a58e1992d5] 2024-03-28T17:58:48.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a58e1e9233] 2024-03-28T17:58:48.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a58e1eb197] 2024-03-28T17:58:48.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a58e0ea11b] 2024-03-28T17:58:48.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a58e0ea6bc] 2024-03-28T17:58:48.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a58e0d41ff] 2024-03-28T17:58:48.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd404ec2488] 2024-03-28T17:58:48.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 16: (DispatchQueue::entry()+0x51f) [0x7fd404ebf60f] 2024-03-28T17:58:48.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd404f92e81] 2024-03-28T17:58:48.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd4039ad1ca] 2024-03-28T17:58:48.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 19: clone() 2024-03-28T17:58:48.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:58:48.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 2024-03-28T17:58:48.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: -1> 2024-03-28T17:58:48.442+0000 7fd3fa606700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd3fa606700 time 2024-03-28T17:58:48.441176+0000 2024-03-28T17:58:48.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:58:48.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 2024-03-28T17:58:48.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:48.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd404c304a1] 2024-03-28T17:58:48.834 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd404c30667] 2024-03-28T17:58:48.834 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a58e31c223] 2024-03-28T17:58:48.834 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a58e1965d6] 2024-03-28T17:58:48.834 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a58e196da1] 2024-03-28T17:58:48.834 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a58e1992d5] 2024-03-28T17:58:48.834 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a58e1e9233] 2024-03-28T17:58:48.834 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a58e1eb197] 2024-03-28T17:58:48.834 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a58e0ea11b] 2024-03-28T17:58:48.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a58e0ea6bc] 2024-03-28T17:58:48.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a58e0d41ff] 2024-03-28T17:58:48.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd404ec2488] 2024-03-28T17:58:48.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 13: (DispatchQueue::entry()+0x51f) [0x7fd404ebf60f] 2024-03-28T17:58:48.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd404f92e81] 2024-03-28T17:58:48.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd4039ad1ca] 2024-03-28T17:58:48.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 16: clone() 2024-03-28T17:58:48.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 2024-03-28T17:58:48.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 0> 2024-03-28T17:58:48.444+0000 7fd3fa606700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:58:48.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: in thread 7fd3fa606700 thread_name:ms_dispatch 2024-03-28T17:58:48.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 2024-03-28T17:58:48.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:58:48.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd4039b7d20] 2024-03-28T17:58:48.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 2: gsignal() 2024-03-28T17:58:48.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 3: abort() 2024-03-28T17:58:48.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd404c304fb] 2024-03-28T17:58:48.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd404c30667] 2024-03-28T17:58:48.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a58e31c223] 2024-03-28T17:58:48.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a58e1965d6] 2024-03-28T17:58:48.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a58e196da1] 2024-03-28T17:58:48.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55a58e1992d5] 2024-03-28T17:58:48.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a58e1e9233] 2024-03-28T17:58:48.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a58e1eb197] 2024-03-28T17:58:48.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a58e0ea11b] 2024-03-28T17:58:48.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a58e0ea6bc] 2024-03-28T17:58:48.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a58e0d41ff] 2024-03-28T17:58:48.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd404ec2488] 2024-03-28T17:58:48.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 16: (DispatchQueue::entry()+0x51f) [0x7fd404ebf60f] 2024-03-28T17:58:48.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd404f92e81] 2024-03-28T17:58:48.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd4039ad1ca] 2024-03-28T17:58:48.837 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 19: clone() 2024-03-28T17:58:48.837 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:58:48.837 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[73971]: 2024-03-28T17:58:48.837 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:48 smithi151 podman[74014]: 2024-03-28 17:58:48.496674224 +0000 UTC m=+0.019745706 container died 47587274a62d62aea96afbfe5d0463bb3c1a333bfd747d7b1a3dc1025732e1d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-28T17:58:48.902 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 381 sec latency 496799.693 ms 2024-03-28T17:58:48.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:48 smithi045 ceph-mon[31773]: pgmap v333: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T17:58:48.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:48 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/4173171826,v1:172.21.15.151:6833/4173171826] up:rejoin 2024-03-28T17:58:48.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:48 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:48.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:48 smithi045 ceph-mon[31773]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-03-28T17:58:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:49 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:58:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:49 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:58:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:49 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/4173171826,v1:172.21.15.151:6833/4173171826] up:active 2024-03-28T17:58:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:49 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:49 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:58:49.574 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:49 smithi151 podman[74014]: 2024-03-28 17:58:49.259147886 +0000 UTC m=+0.782219368 container remove 47587274a62d62aea96afbfe5d0463bb3c1a333bfd747d7b1a3dc1025732e1d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-28T17:58:49.574 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:49 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:58:49.574 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:49 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-28T17:58:49.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:49 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:58:49.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:49 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:58:49.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:49 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/4173171826,v1:172.21.15.151:6833/4173171826] up:active 2024-03-28T17:58:49.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:49 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:49.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:49 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:58:49.902 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 382 sec latency 497799.846 ms 2024-03-28T17:58:49.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:49 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:58:49.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:49 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:58:49.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:49 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/4173171826,v1:172.21.15.151:6833/4173171826] up:active 2024-03-28T17:58:49.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:49 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T17:58:49.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:49 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:58:50.553 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:50 smithi045 ceph-mon[31773]: pgmap v334: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 131 B/s wr, 4 op/s 2024-03-28T17:58:50.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:50 smithi071 ceph-mon[36597]: pgmap v334: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 131 B/s wr, 4 op/s 2024-03-28T17:58:50.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:50 smithi151 ceph-mon[36481]: pgmap v334: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 131 B/s wr, 4 op/s 2024-03-28T17:58:50.903 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 383 sec latency 498799.954 ms 2024-03-28T17:58:51.903 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 384 sec latency 499800.056 ms 2024-03-28T17:58:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:52 smithi071 ceph-mon[36597]: pgmap v335: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 511 B/s wr, 13 op/s 2024-03-28T17:58:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:52 smithi151 ceph-mon[36481]: pgmap v335: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 511 B/s wr, 13 op/s 2024-03-28T17:58:52.903 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 385 sec latency 500800.188 ms 2024-03-28T17:58:52.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:52 smithi045 ceph-mon[31773]: pgmap v335: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 511 B/s wr, 13 op/s 2024-03-28T17:58:53.903 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 386 sec latency 501800.323 ms 2024-03-28T17:58:54.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:54 smithi071 ceph-mon[36597]: pgmap v336: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 451 B/s wr, 12 op/s 2024-03-28T17:58:54.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:54 smithi151 ceph-mon[36481]: pgmap v336: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 451 B/s wr, 12 op/s 2024-03-28T17:58:54.903 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 387 sec latency 502800.484 ms 2024-03-28T17:58:54.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:54 smithi045 ceph-mon[31773]: pgmap v336: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 451 B/s wr, 12 op/s 2024-03-28T17:58:55.903 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 388 sec latency 503800.645 ms 2024-03-28T17:58:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:56 smithi071 ceph-mon[36597]: pgmap v337: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T17:58:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:56 smithi151 ceph-mon[36481]: pgmap v337: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T17:58:56.903 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 389 sec latency 504800.812 ms 2024-03-28T17:58:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:56 smithi045 ceph-mon[31773]: pgmap v337: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T17:58:57.904 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 390 sec latency 505800.944 ms 2024-03-28T17:58:58.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:58:58 smithi071 ceph-mon[36597]: pgmap v338: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 285 B/s wr, 9 op/s 2024-03-28T17:58:58.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:58:58 smithi151 ceph-mon[36481]: pgmap v338: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 285 B/s wr, 9 op/s 2024-03-28T17:58:58.904 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 391 sec latency 506801.053 ms 2024-03-28T17:58:58.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:58:58 smithi045 ceph-mon[31773]: pgmap v338: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 285 B/s wr, 9 op/s 2024-03-28T17:58:59.834 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:59 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 5. 2024-03-28T17:58:59.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:59 smithi151 systemd[1]: Stopped Ceph mds.l for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:58:59.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:59 smithi151 systemd[1]: Starting Ceph mds.l for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:58:59.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:59 smithi151 podman[74114]: 2024-03-28 17:58:59.710185605 +0000 UTC m=+0.018162363 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:58:59.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:58:59 smithi151 podman[74114]: 2024-03-28 17:58:59.835496667 +0000 UTC m=+0.143473400 container create dc8464e8f6b83f207b23264d8f85aa8023a121bd95704f7f3123f778dfb3357f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-28T17:58:59.904 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 392 sec latency 507801.185 ms 2024-03-28T17:59:00.471 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:00 smithi151 podman[74114]: 2024-03-28 17:59:00.097220273 +0000 UTC m=+0.405197012 container init dc8464e8f6b83f207b23264d8f85aa8023a121bd95704f7f3123f778dfb3357f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-28T17:59:00.472 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:00 smithi151 podman[74114]: 2024-03-28 17:59:00.100465039 +0000 UTC m=+0.408441770 container start dc8464e8f6b83f207b23264d8f85aa8023a121bd95704f7f3123f778dfb3357f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T17:59:00.472 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:00 smithi151 ceph-mds[74135]: -- 172.21.15.151:0/2959306253 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c1d084fa40 con 0x55c1d0823000 2024-03-28T17:59:00.472 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:00 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: starting mds.l at 2024-03-28T17:59:00.472 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:00 smithi151 bash[74114]: dc8464e8f6b83f207b23264d8f85aa8023a121bd95704f7f3123f778dfb3357f 2024-03-28T17:59:00.472 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:00 smithi151 systemd[1]: Started Ceph mds.l for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:59:00.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:00 smithi071 ceph-mon[36597]: pgmap v339: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T17:59:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:00 smithi071 ceph-mon[36597]: daemon mds.l restarted 2024-03-28T17:59:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:00 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:59:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:00 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T17:59:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:00 smithi071 ceph-mon[36597]: osdmap e125: 12 total, 12 up, 12 in 2024-03-28T17:59:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:00 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/1729218296,v1:172.21.15.151:6833/1729218296] up:boot 2024-03-28T17:59:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:00 smithi071 ceph-mon[36597]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-03-28T17:59:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:00 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:59:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:00 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:00 smithi151 ceph-mon[36481]: pgmap v339: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T17:59:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:00 smithi151 ceph-mon[36481]: daemon mds.l restarted 2024-03-28T17:59:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:00 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:59:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:00 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T17:59:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:00 smithi151 ceph-mon[36481]: osdmap e125: 12 total, 12 up, 12 in 2024-03-28T17:59:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:00 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/1729218296,v1:172.21.15.151:6833/1729218296] up:boot 2024-03-28T17:59:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:00 smithi151 ceph-mon[36481]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-03-28T17:59:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:00 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:59:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:00 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:00.904 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 393 sec latency 508801.285 ms 2024-03-28T17:59:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:00 smithi045 ceph-mon[31773]: pgmap v339: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T17:59:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:00 smithi045 ceph-mon[31773]: daemon mds.l restarted 2024-03-28T17:59:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:00 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:59:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:00 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T17:59:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:00 smithi045 ceph-mon[31773]: osdmap e125: 12 total, 12 up, 12 in 2024-03-28T17:59:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:00 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/1729218296,v1:172.21.15.151:6833/1729218296] up:boot 2024-03-28T17:59:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:00 smithi045 ceph-mon[31773]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-03-28T17:59:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:00 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:59:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:00 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:01.904 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 394 sec latency 509801.438 ms 2024-03-28T17:59:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:02 smithi071 ceph-mon[36597]: pgmap v341: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:59:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:02 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/785732331,v1:172.21.15.45:6837/785732331] up:resolve 2024-03-28T17:59:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:02 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:02 smithi151 ceph-mon[36481]: pgmap v341: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:59:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:02 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/785732331,v1:172.21.15.45:6837/785732331] up:resolve 2024-03-28T17:59:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:02 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:02.904 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 395 sec latency 510801.540 ms 2024-03-28T17:59:02.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:02 smithi045 ceph-mon[31773]: pgmap v341: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:59:02.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:02 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/785732331,v1:172.21.15.45:6837/785732331] up:resolve 2024-03-28T17:59:02.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:02 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:03.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:03 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:59:03.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:03 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/785732331,v1:172.21.15.45:6837/785732331] up:reconnect 2024-03-28T17:59:03.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:03 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:03.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:03 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T17:59:03.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:03 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:59:03.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:03 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/785732331,v1:172.21.15.45:6837/785732331] up:reconnect 2024-03-28T17:59:03.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:03 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:03.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:03 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T17:59:03.904 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 396 sec latency 511801.675 ms 2024-03-28T17:59:03.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:03 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:59:03.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:03 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/785732331,v1:172.21.15.45:6837/785732331] up:reconnect 2024-03-28T17:59:03.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:03 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:03.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:03 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T17:59:04.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:04 smithi071 ceph-mon[36597]: pgmap v342: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:59:04.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:04 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/785732331,v1:172.21.15.45:6837/785732331] up:rejoin 2024-03-28T17:59:04.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:04 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:04.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:04 smithi071 ceph-mon[36597]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-03-28T17:59:04.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:04 smithi151 ceph-mon[36481]: pgmap v342: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:59:04.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:04 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/785732331,v1:172.21.15.45:6837/785732331] up:rejoin 2024-03-28T17:59:04.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:04 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:04.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:04 smithi151 ceph-mon[36481]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-03-28T17:59:04.905 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 397 sec latency 512801.811 ms 2024-03-28T17:59:04.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-mon[31773]: pgmap v342: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:59:04.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/785732331,v1:172.21.15.45:6837/785732331] up:rejoin 2024-03-28T17:59:04.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:04.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-mon[31773]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-03-28T17:59:04.917 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f20d1000700 time 2024-03-28T17:59:04.540377+0000 2024-03-28T17:59:04.917 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:59:04.917 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:04.917 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f20dc86e4a1] 2024-03-28T17:59:04.917 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f20dc86e667] 2024-03-28T17:59:04.917 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556139ebb223] 2024-03-28T17:59:04.917 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556139d355d6] 2024-03-28T17:59:04.917 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556139d35da1] 2024-03-28T17:59:04.917 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556139d382d5] 2024-03-28T17:59:04.917 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556139d88233] 2024-03-28T17:59:04.917 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556139d8a197] 2024-03-28T17:59:04.917 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556139c8911b] 2024-03-28T17:59:04.917 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556139c896bc] 2024-03-28T17:59:04.918 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556139c731ff] 2024-03-28T17:59:04.918 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f20dcb00488] 2024-03-28T17:59:04.918 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 13: (DispatchQueue::entry()+0x51f) [0x7f20dcafd60f] 2024-03-28T17:59:04.918 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f20dcbd0e81] 2024-03-28T17:59:04.918 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f20db5eb1ca] 2024-03-28T17:59:04.918 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 16: clone() 2024-03-28T17:59:04.919 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 2024-03-28T17:59:04.543+0000 7f20d1000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f20d1000700 time 2024-03-28T17:59:04.540377+0000 2024-03-28T17:59:04.919 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:59:04.919 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 2024-03-28T17:59:04.919 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:04.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f20dc86e4a1] 2024-03-28T17:59:04.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f20dc86e667] 2024-03-28T17:59:04.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556139ebb223] 2024-03-28T17:59:04.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556139d355d6] 2024-03-28T17:59:04.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556139d35da1] 2024-03-28T17:59:04.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556139d382d5] 2024-03-28T17:59:04.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556139d88233] 2024-03-28T17:59:04.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556139d8a197] 2024-03-28T17:59:04.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556139c8911b] 2024-03-28T17:59:04.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556139c896bc] 2024-03-28T17:59:04.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556139c731ff] 2024-03-28T17:59:04.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f20dcb00488] 2024-03-28T17:59:04.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 13: (DispatchQueue::entry()+0x51f) [0x7f20dcafd60f] 2024-03-28T17:59:04.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f20dcbd0e81] 2024-03-28T17:59:04.921 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f20db5eb1ca] 2024-03-28T17:59:04.921 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 16: clone() 2024-03-28T17:59:04.921 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 2024-03-28T17:59:04.921 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: *** Caught signal (Aborted) ** 2024-03-28T17:59:04.921 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: in thread 7f20d1000700 thread_name:ms_dispatch 2024-03-28T17:59:04.921 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:04.921 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f20db5f5d20] 2024-03-28T17:59:04.922 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 2: gsignal() 2024-03-28T17:59:04.923 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 3: abort() 2024-03-28T17:59:04.923 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f20dc86e4fb] 2024-03-28T17:59:04.923 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f20dc86e667] 2024-03-28T17:59:04.923 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556139ebb223] 2024-03-28T17:59:04.923 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556139d355d6] 2024-03-28T17:59:04.923 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556139d35da1] 2024-03-28T17:59:04.923 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556139d382d5] 2024-03-28T17:59:04.923 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556139d88233] 2024-03-28T17:59:04.923 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556139d8a197] 2024-03-28T17:59:04.923 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556139c8911b] 2024-03-28T17:59:04.923 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556139c896bc] 2024-03-28T17:59:04.923 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556139c731ff] 2024-03-28T17:59:04.923 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f20dcb00488] 2024-03-28T17:59:04.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 16: (DispatchQueue::entry()+0x51f) [0x7f20dcafd60f] 2024-03-28T17:59:04.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f20dcbd0e81] 2024-03-28T17:59:04.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f20db5eb1ca] 2024-03-28T17:59:04.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 19: clone() 2024-03-28T17:59:04.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 2024-03-28T17:59:04.550+0000 7f20d1000700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:59:04.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: in thread 7f20d1000700 thread_name:ms_dispatch 2024-03-28T17:59:04.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 2024-03-28T17:59:04.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:04.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f20db5f5d20] 2024-03-28T17:59:04.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 2: gsignal() 2024-03-28T17:59:04.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 3: abort() 2024-03-28T17:59:04.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f20dc86e4fb] 2024-03-28T17:59:04.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f20dc86e667] 2024-03-28T17:59:04.925 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556139ebb223] 2024-03-28T17:59:04.925 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556139d355d6] 2024-03-28T17:59:04.925 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556139d35da1] 2024-03-28T17:59:04.925 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556139d382d5] 2024-03-28T17:59:04.925 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556139d88233] 2024-03-28T17:59:04.925 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556139d8a197] 2024-03-28T17:59:04.925 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556139c8911b] 2024-03-28T17:59:04.925 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556139c896bc] 2024-03-28T17:59:04.925 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556139c731ff] 2024-03-28T17:59:04.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f20dcb00488] 2024-03-28T17:59:04.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 16: (DispatchQueue::entry()+0x51f) [0x7f20dcafd60f] 2024-03-28T17:59:04.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f20dcbd0e81] 2024-03-28T17:59:04.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f20db5eb1ca] 2024-03-28T17:59:04.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 19: clone() 2024-03-28T17:59:04.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:59:04.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 2024-03-28T17:59:04.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: -1> 2024-03-28T17:59:04.543+0000 7f20d1000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f20d1000700 time 2024-03-28T17:59:04.540377+0000 2024-03-28T17:59:04.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:59:04.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 2024-03-28T17:59:04.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:04.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f20dc86e4a1] 2024-03-28T17:59:04.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f20dc86e667] 2024-03-28T17:59:04.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556139ebb223] 2024-03-28T17:59:04.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556139d355d6] 2024-03-28T17:59:04.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556139d35da1] 2024-03-28T17:59:04.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556139d382d5] 2024-03-28T17:59:04.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556139d88233] 2024-03-28T17:59:04.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556139d8a197] 2024-03-28T17:59:04.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556139c8911b] 2024-03-28T17:59:04.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556139c896bc] 2024-03-28T17:59:04.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556139c731ff] 2024-03-28T17:59:04.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f20dcb00488] 2024-03-28T17:59:04.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 13: (DispatchQueue::entry()+0x51f) [0x7f20dcafd60f] 2024-03-28T17:59:04.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f20dcbd0e81] 2024-03-28T17:59:04.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f20db5eb1ca] 2024-03-28T17:59:04.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 16: clone() 2024-03-28T17:59:04.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 2024-03-28T17:59:04.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 0> 2024-03-28T17:59:04.550+0000 7f20d1000700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:59:04.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: in thread 7f20d1000700 thread_name:ms_dispatch 2024-03-28T17:59:04.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 2024-03-28T17:59:04.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:04.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f20db5f5d20] 2024-03-28T17:59:04.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 2: gsignal() 2024-03-28T17:59:04.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 3: abort() 2024-03-28T17:59:04.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f20dc86e4fb] 2024-03-28T17:59:04.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f20dc86e667] 2024-03-28T17:59:04.929 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556139ebb223] 2024-03-28T17:59:04.929 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556139d355d6] 2024-03-28T17:59:04.929 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556139d35da1] 2024-03-28T17:59:04.929 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556139d382d5] 2024-03-28T17:59:04.929 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556139d88233] 2024-03-28T17:59:04.929 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556139d8a197] 2024-03-28T17:59:04.929 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556139c8911b] 2024-03-28T17:59:04.929 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556139c896bc] 2024-03-28T17:59:04.929 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556139c731ff] 2024-03-28T17:59:04.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f20dcb00488] 2024-03-28T17:59:04.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 16: (DispatchQueue::entry()+0x51f) [0x7f20dcafd60f] 2024-03-28T17:59:04.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f20dcbd0e81] 2024-03-28T17:59:04.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f20db5eb1ca] 2024-03-28T17:59:04.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 19: clone() 2024-03-28T17:59:04.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:59:04.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 2024-03-28T17:59:04.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: -1> 2024-03-28T17:59:04.543+0000 7f20d1000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f20d1000700 time 2024-03-28T17:59:04.540377+0000 2024-03-28T17:59:04.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:59:04.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 2024-03-28T17:59:04.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:04.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f20dc86e4a1] 2024-03-28T17:59:04.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f20dc86e667] 2024-03-28T17:59:04.931 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556139ebb223] 2024-03-28T17:59:04.931 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556139d355d6] 2024-03-28T17:59:04.931 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556139d35da1] 2024-03-28T17:59:04.931 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556139d382d5] 2024-03-28T17:59:04.931 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556139d88233] 2024-03-28T17:59:04.931 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556139d8a197] 2024-03-28T17:59:04.932 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556139c8911b] 2024-03-28T17:59:04.932 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556139c896bc] 2024-03-28T17:59:04.932 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556139c731ff] 2024-03-28T17:59:04.932 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f20dcb00488] 2024-03-28T17:59:04.932 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 13: (DispatchQueue::entry()+0x51f) [0x7f20dcafd60f] 2024-03-28T17:59:04.932 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f20dcbd0e81] 2024-03-28T17:59:04.932 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f20db5eb1ca] 2024-03-28T17:59:04.932 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 16: clone() 2024-03-28T17:59:04.932 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 2024-03-28T17:59:04.932 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 0> 2024-03-28T17:59:04.550+0000 7f20d1000700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:59:04.932 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: in thread 7f20d1000700 thread_name:ms_dispatch 2024-03-28T17:59:04.932 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 2024-03-28T17:59:04.932 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:04.932 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f20db5f5d20] 2024-03-28T17:59:04.933 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 2: gsignal() 2024-03-28T17:59:04.933 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 3: abort() 2024-03-28T17:59:04.933 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f20dc86e4fb] 2024-03-28T17:59:04.933 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f20dc86e667] 2024-03-28T17:59:04.933 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556139ebb223] 2024-03-28T17:59:04.933 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556139d355d6] 2024-03-28T17:59:04.933 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556139d35da1] 2024-03-28T17:59:04.933 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556139d382d5] 2024-03-28T17:59:04.933 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556139d88233] 2024-03-28T17:59:04.933 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556139d8a197] 2024-03-28T17:59:04.933 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556139c8911b] 2024-03-28T17:59:04.933 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556139c896bc] 2024-03-28T17:59:04.933 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556139c731ff] 2024-03-28T17:59:04.934 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f20dcb00488] 2024-03-28T17:59:04.934 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 16: (DispatchQueue::entry()+0x51f) [0x7f20dcafd60f] 2024-03-28T17:59:04.934 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f20dcbd0e81] 2024-03-28T17:59:04.934 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f20db5eb1ca] 2024-03-28T17:59:04.934 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 19: clone() 2024-03-28T17:59:04.934 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:59:04.934 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92762]: 2024-03-28T17:59:04.934 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:04 smithi045 podman[92800]: 2024-03-28 17:59:04.670835184 +0000 UTC m=+0.033033912 container died 86686230b557b1bfae04f5d4f323dc35ad3e1db8db176cabda9f1e5cbd112c2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T17:59:05.310 INFO:journalctl@ceph.mds.b.smithi071.stdout:Mar 28 17:59:04 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-b[66900]: 2024-03-28T17:59:04.924+0000 7ff9f838b700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-28T17:59:05.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:05 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:59:05.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:05 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:59:05.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:05 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/785732331,v1:172.21.15.45:6837/785732331] up:active 2024-03-28T17:59:05.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:05 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:05.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:05 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:59:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:05 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:59:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:05 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:59:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:05 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/785732331,v1:172.21.15.45:6837/785732331] up:active 2024-03-28T17:59:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:05 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:05 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:59:05.905 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 398 sec latency 513801.932 ms 2024-03-28T17:59:05.915 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:05 smithi045 podman[92800]: 2024-03-28 17:59:05.698240842 +0000 UTC m=+1.060439608 container remove 86686230b557b1bfae04f5d4f323dc35ad3e1db8db176cabda9f1e5cbd112c2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-28T17:59:05.915 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:05 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:59:05.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:05 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:59:05.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:05 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:59:05.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:05 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/785732331,v1:172.21.15.45:6837/785732331] up:active 2024-03-28T17:59:05.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:05 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:05.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:05 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:59:06.415 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:06 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Failed with result 'exit-code'. 2024-03-28T17:59:06.415 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:06 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Consumed 1.658s CPU time. 2024-03-28T17:59:06.607 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:59:06.607 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:59:06.703 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:06 smithi045 ceph-mon[31773]: pgmap v343: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:59:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:06 smithi071 ceph-mon[36597]: pgmap v343: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:59:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:06 smithi151 ceph-mon[36481]: pgmap v343: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:59:06.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.850+0000 7f1810400640 1 -- 172.21.15.45:0/1203315723 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f180810cf80 msgr2=0x7f1808114b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:06.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.850+0000 7f1810400640 1 --2- 172.21.15.45:0/1203315723 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f180810cf80 0x7f1808114b50 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1800009900 tx=0x7f180002bc30 comp rx=0 tx=0).stop 2024-03-28T17:59:06.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.850+0000 7f1810400640 1 -- 172.21.15.45:0/1203315723 shutdown_connections 2024-03-28T17:59:06.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.850+0000 7f1810400640 1 --2- 172.21.15.45:0/1203315723 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f180810cf80 0x7f1808114b50 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.850+0000 7f1810400640 1 --2- 172.21.15.45:0/1203315723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1808108860 0x7f180810c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.850+0000 7f1810400640 1 --2- 172.21.15.45:0/1203315723 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1808107eb0 0x7f1808108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.850+0000 7f1810400640 1 -- 172.21.15.45:0/1203315723 >> 172.21.15.45:0/1203315723 conn(0x7f1808075be0 msgr2=0x7f1808078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:06.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.850+0000 7f1810400640 1 -- 172.21.15.45:0/1203315723 shutdown_connections 2024-03-28T17:59:06.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.850+0000 7f1810400640 1 -- 172.21.15.45:0/1203315723 wait complete. 2024-03-28T17:59:06.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.851+0000 7f1810400640 1 Processor -- start 2024-03-28T17:59:06.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.851+0000 7f1810400640 1 -- start start 2024-03-28T17:59:06.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.851+0000 7f1810400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1808107eb0 0x7f180819e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:06.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.851+0000 7f1810400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f180810cf80 0x7f180819ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:06.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.851+0000 7f1810400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18081a2da0 0x7f18081a3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:06.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.851+0000 7f1810400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1808118080 con 0x7f180810cf80 2024-03-28T17:59:06.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.851+0000 7f1810400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1808117f00 con 0x7f18081a2da0 2024-03-28T17:59:06.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.851+0000 7f1810400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1808118200 con 0x7f1808107eb0 2024-03-28T17:59:06.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.852+0000 7f180e400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18081a2da0 0x7f18081a3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:06.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.852+0000 7f180e400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18081a2da0 0x7f18081a3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:58576/0 (socket says 172.21.15.45:58576) 2024-03-28T17:59:06.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.852+0000 7f180e400640 1 -- 172.21.15.45:0/2771385848 learned_addr learned my addr 172.21.15.45:0/2771385848 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:59:06.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.852+0000 7f180da00640 1 --2- 172.21.15.45:0/2771385848 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1808107eb0 0x7f180819e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:06.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.852+0000 7f180d000640 1 --2- 172.21.15.45:0/2771385848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f180810cf80 0x7f180819ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:06.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.852+0000 7f180e400640 1 -- 172.21.15.45:0/2771385848 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1808107eb0 msgr2=0x7f180819e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:06.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.852+0000 7f180e400640 1 --2- 172.21.15.45:0/2771385848 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1808107eb0 0x7f180819e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.852+0000 7f180e400640 1 -- 172.21.15.45:0/2771385848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f180810cf80 msgr2=0x7f180819ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:06.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.852+0000 7f180e400640 1 --2- 172.21.15.45:0/2771385848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f180810cf80 0x7f180819ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.852+0000 7f180e400640 1 -- 172.21.15.45:0/2771385848 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18081a3890 con 0x7f18081a2da0 2024-03-28T17:59:06.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.852+0000 7f180e400640 1 --2- 172.21.15.45:0/2771385848 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18081a2da0 0x7f18081a3250 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1800036040 tx=0x7f1800004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:06.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.853+0000 7f1806000640 1 -- 172.21.15.45:0/2771385848 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1800043020 con 0x7f18081a2da0 2024-03-28T17:59:06.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.853+0000 7f1806000640 1 -- 172.21.15.45:0/2771385848 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1800004490 con 0x7f18081a2da0 2024-03-28T17:59:06.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.853+0000 7f1810400640 1 -- 172.21.15.45:0/2771385848 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18081a3b20 con 0x7f18081a2da0 2024-03-28T17:59:06.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.853+0000 7f1806000640 1 -- 172.21.15.45:0/2771385848 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1800003b50 con 0x7f18081a2da0 2024-03-28T17:59:06.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.853+0000 7f1810400640 1 -- 172.21.15.45:0/2771385848 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18081ab370 con 0x7f18081a2da0 2024-03-28T17:59:06.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.856+0000 7f1806000640 1 -- 172.21.15.45:0/2771385848 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f1800046020 con 0x7f18081a2da0 2024-03-28T17:59:06.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.856+0000 7f18131c3740 1 Processor -- start 2024-03-28T17:59:06.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.856+0000 7f18131c3740 1 -- start start 2024-03-28T17:59:06.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.857+0000 7f18131c3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573dbc00 0x557d573da950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:06.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.857+0000 7f18131c3740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557d573daf20 0x557d573db3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:06.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.857+0000 7f18131c3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d573e4290 0x557d573e6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:06.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.857+0000 7f18131c3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d5733f250 con 0x557d573e4290 2024-03-28T17:59:06.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.857+0000 7f18131c3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d572e7a30 con 0x557d573dbc00 2024-03-28T17:59:06.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.857+0000 7f18131c3740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d5733b2f0 con 0x557d573daf20 2024-03-28T17:59:06.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.857+0000 7f180d000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573dbc00 0x557d573da950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:06.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.857+0000 7f180da00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557d573daf20 0x557d573db3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:06.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.857+0000 7f180d000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573dbc00 0x557d573da950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:58590/0 (socket says 172.21.15.45:58590) 2024-03-28T17:59:06.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.857+0000 7f180d000640 1 -- 172.21.15.45:0/3180786409 learned_addr learned my addr 172.21.15.45:0/3180786409 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:59:06.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.857+0000 7f180d000640 1 --2- 172.21.15.45:0/3180786409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d573e4290 0x557d573e6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:06.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.858+0000 7f180da00640 1 -- 172.21.15.45:0/3180786409 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573dbc00 msgr2=0x557d573da950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:59:06.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.858+0000 7f180da00640 1 --2- 172.21.15.45:0/3180786409 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573dbc00 0x557d573da950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.858+0000 7f180da00640 1 -- 172.21.15.45:0/3180786409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d573e4290 msgr2=0x557d573e6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:59:06.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.858+0000 7f180da00640 1 --2- 172.21.15.45:0/3180786409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d573e4290 0x557d573e6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.858+0000 7f180da00640 1 -- 172.21.15.45:0/3180786409 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d572d2690 con 0x557d573daf20 2024-03-28T17:59:06.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.858+0000 7f1806000640 1 --2- 172.21.15.45:0/2771385848 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f17f00a29f0 0x7f17f00a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:06.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.858+0000 7f180da00640 1 --2- 172.21.15.45:0/3180786409 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557d573daf20 0x557d573db3a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f17fc009fa0 tx=0x7f17fc0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5f817436da2020a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:06.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.858+0000 7f1806000640 1 -- 172.21.15.45:0/2771385848 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 11178+0+0 (secure 0 0 0) 0x7f18000f1740 con 0x7f18081a2da0 2024-03-28T17:59:06.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.858+0000 7f180d000640 1 --2- 172.21.15.45:0/2771385848 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f17f00a29f0 0x7f17f00a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:06.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.858+0000 7f17faa00640 1 -- 172.21.15.45:0/3180786409 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17fc003c30 con 0x557d573daf20 2024-03-28T17:59:06.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.858+0000 7f17faa00640 1 -- 172.21.15.45:0/3180786409 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17fc003dd0 con 0x557d573daf20 2024-03-28T17:59:06.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.858+0000 7f17faa00640 1 -- 172.21.15.45:0/3180786409 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17fc00fac0 con 0x557d573daf20 2024-03-28T17:59:06.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.859+0000 7f18131c3740 1 -- 172.21.15.45:0/3180786409 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557d573daf20 msgr2=0x557d573db3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:06.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.859+0000 7f18131c3740 1 --2- 172.21.15.45:0/3180786409 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557d573daf20 0x557d573db3a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f17fc009fa0 tx=0x7f17fc0118d0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.859+0000 7f180d000640 1 --2- 172.21.15.45:0/2771385848 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f17f00a29f0 0x7f17f00a4eb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f17f400bfd0 tx=0x7f17f400b410 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:06.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.859+0000 7f18131c3740 1 -- 172.21.15.45:0/3180786409 shutdown_connections 2024-03-28T17:59:06.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.859+0000 7f18131c3740 1 --2- 172.21.15.45:0/3180786409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d573e4290 0x557d573e6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.859+0000 7f18131c3740 1 --2- 172.21.15.45:0/3180786409 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557d573daf20 0x557d573db3a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.859+0000 7f18131c3740 1 --2- 172.21.15.45:0/3180786409 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573dbc00 0x557d573da950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.859+0000 7f18131c3740 1 -- 172.21.15.45:0/3180786409 >> 172.21.15.45:0/3180786409 conn(0x557d573a2200 msgr2=0x557d573d5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:06.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.859+0000 7f18131c3740 1 -- 172.21.15.45:0/3180786409 shutdown_connections 2024-03-28T17:59:06.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.859+0000 7f18131c3740 1 -- 172.21.15.45:0/3180786409 wait complete. 2024-03-28T17:59:06.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.860+0000 7f18131c3740 1 -- start start 2024-03-28T17:59:06.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.860+0000 7f18131c3740 1 Processor -- start 2024-03-28T17:59:06.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.860+0000 7f18131c3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573daf20 0x557d573e3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:06.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.860+0000 7f18131c3740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557d573dbc00 0x557d573e0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:06.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.860+0000 7f18131c3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d573e4290 0x557d573e11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:06.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.860+0000 7f18131c3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d573e7100 con 0x557d573e4290 2024-03-28T17:59:06.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.860+0000 7f18131c3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d572fae80 con 0x557d573daf20 2024-03-28T17:59:06.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.860+0000 7f18131c3740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d573e7280 con 0x557d573dbc00 2024-03-28T17:59:06.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.860+0000 7f18131c3740 10 client.0 _ll_register_callbacks cb 0x557d5730d130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:59:06.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.861+0000 7f180da00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573daf20 0x557d573e3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:06.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.861+0000 7f180da00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573daf20 0x557d573e3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:58600/0 (socket says 172.21.15.45:58600) 2024-03-28T17:59:06.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.861+0000 7f180da00640 1 -- 172.21.15.45:0/2446438661 learned_addr learned my addr 172.21.15.45:0/2446438661 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:59:06.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.861+0000 7f180e400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d573e4290 0x557d573e11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:06.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.861+0000 7f180d000640 1 --2- 172.21.15.45:0/2446438661 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557d573dbc00 0x557d573e0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:06.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.861+0000 7f180da00640 1 -- 172.21.15.45:0/2446438661 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557d573dbc00 msgr2=0x557d573e0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:59:06.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.861+0000 7f180da00640 1 --2- 172.21.15.45:0/2446438661 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557d573dbc00 0x557d573e0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.861+0000 7f180da00640 1 -- 172.21.15.45:0/2446438661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d573e4290 msgr2=0x557d573e11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:59:06.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.861+0000 7f180da00640 1 --2- 172.21.15.45:0/2446438661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d573e4290 0x557d573e11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.861+0000 7f180da00640 1 -- 172.21.15.45:0/2446438661 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d5711d7d0 con 0x557d573daf20 2024-03-28T17:59:06.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.861+0000 7f180da00640 1 --2- 172.21.15.45:0/2446438661 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573daf20 0x557d573e3910 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f17fc004cf0 tx=0x7f17fc00f620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b63dadd03dec5d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:06.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.861+0000 7f17ef400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:59:06.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.862+0000 7f17ef400640 1 -- 172.21.15.45:0/2446438661 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17fc028020 con 0x557d573daf20 2024-03-28T17:59:06.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.862+0000 7f17ef400640 1 -- 172.21.15.45:0/2446438661 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(125..125 src has 1..125) v4 ==== 11178+0+0 (secure 0 0 0) 0x7f17fc00fa50 con 0x557d573daf20 2024-03-28T17:59:06.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.862+0000 7f18131c3740 1 -- 172.21.15.45:0/2446438661 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f17fc003dd0 con 0x557d573daf20 2024-03-28T17:59:06.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.862+0000 7f17ef400640 1 -- 172.21.15.45:0/2446438661 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=228) v2 ==== 24+0+0 (secure 0 0 0) 0x7f17fc033050 con 0x557d573daf20 2024-03-28T17:59:06.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.862+0000 7f18131c3740 10 client.26363 fetch_fsmap learned FSMap version 228 2024-03-28T17:59:06.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.862+0000 7f18131c3740 1 -- 172.21.15.45:0/2446438661 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=228,monmap=4+}) v3 -- 0x557d570533f0 con 0x557d573daf20 2024-03-28T17:59:06.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.862+0000 7f17ef400640 1 -- 172.21.15.45:0/2446438661 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17fc003dd0 con 0x557d573daf20 2024-03-28T17:59:06.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.862+0000 7f17ef400640 1 -- 172.21.15.45:0/2446438661 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 228) v1 ==== 5618+0+0 (secure 0 0 0) 0x7f17fc02ea50 con 0x557d573daf20 2024-03-28T17:59:06.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.862+0000 7f18131c3740 10 client.26363 fetch_fsmap finished waiting for FSMap version 228 2024-03-28T17:59:06.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.862+0000 7f18131c3740 10 client.26363 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:59:06.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.862+0000 7f18131c3740 20 client.26363 populate_metadata read hostname 'smithi045' 2024-03-28T17:59:06.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.862+0000 7f18131c3740 1 --2- 172.21.15.45:0/2446438661 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x557d573f4350 0x557d573f67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:06.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.863+0000 7f18131c3740 4 client.26363 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:59:06.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.863+0000 7f18131c3740 1 -- 172.21.15.45:0/2446438661 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557d5734c790 con 0x557d573f4350 2024-03-28T17:59:06.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.863+0000 7f180d000640 1 --2- 172.21.15.45:0/2446438661 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x557d573f4350 0x557d573f67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:06.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.863+0000 7f180d000640 1 --2- 172.21.15.45:0/2446438661 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x557d573f4350 0x557d573f67d0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f8f8a6733516adb server_cookie=ce50cf76fd8b4ec in_seq=0 out_seq=0 2024-03-28T17:59:06.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.863+0000 7f17ef400640 10 client.26363 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:59:06.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.864+0000 7f17ef400640 1 -- 172.21.15.45:0/2446438661 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557d5734c790 con 0x557d573f4350 2024-03-28T17:59:06.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.864+0000 7f17ef400640 10 client.26363 handle_command_reply: tid=0 2024-03-28T17:59:06.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.864+0000 7f18131c3740 1 client.26363 shutdown 2024-03-28T17:59:06.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.865+0000 7f18131c3740 1 -- 172.21.15.45:0/2446438661 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573daf20 msgr2=0x557d573e3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:06.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.865+0000 7f18131c3740 1 --2- 172.21.15.45:0/2446438661 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573daf20 0x557d573e3910 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f17fc004cf0 tx=0x7f17fc00f620 comp rx=0 tx=0).stop 2024-03-28T17:59:06.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.865+0000 7f18131c3740 1 -- 172.21.15.45:0/2446438661 shutdown_connections 2024-03-28T17:59:06.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.865+0000 7f18131c3740 1 --2- 172.21.15.45:0/2446438661 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x557d573f4350 0x557d573f67d0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.865+0000 7f18131c3740 1 --2- 172.21.15.45:0/2446438661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d573e4290 0x557d573e11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.865+0000 7f18131c3740 1 --2- 172.21.15.45:0/2446438661 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557d573dbc00 0x557d573e0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.865+0000 7f18131c3740 1 --2- 172.21.15.45:0/2446438661 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573daf20 0x557d573e3910 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.865+0000 7f17ef400640 0 client.26363 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:59:06.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.865+0000 7f18131c3740 1 -- 172.21.15.45:0/2446438661 >> 172.21.15.45:0/2446438661 conn(0x557d573a2200 msgr2=0x557d573d76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:06.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.865+0000 7f18131c3740 1 -- 172.21.15.45:0/2446438661 shutdown_connections 2024-03-28T17:59:06.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.865+0000 7f18131c3740 1 -- 172.21.15.45:0/2446438661 wait complete. 2024-03-28T17:59:06.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.865+0000 7f18131c3740 20 client.26363 trim_cache size 0 max 16384 2024-03-28T17:59:06.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.882+0000 7f18131c3740 1 Processor -- start 2024-03-28T17:59:06.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.883+0000 7f18131c3740 1 -- start start 2024-03-28T17:59:06.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.883+0000 7f18131c3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573daf20 0x557d573e33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:06.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.883+0000 7f18131c3740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557d573dbc00 0x557d573e3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:06.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.883+0000 7f18131c3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d573e4290 0x557d573e5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:06.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.883+0000 7f18131c3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18081b4780 con 0x557d573e4290 2024-03-28T17:59:06.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.883+0000 7f18131c3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18081b6100 con 0x557d573daf20 2024-03-28T17:59:06.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.883+0000 7f18131c3740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18081b4de0 con 0x557d573dbc00 2024-03-28T17:59:06.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.883+0000 7f180da00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573daf20 0x557d573e33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:06.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.883+0000 7f180da00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573daf20 0x557d573e33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:58614/0 (socket says 172.21.15.45:58614) 2024-03-28T17:59:06.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.883+0000 7f180da00640 1 -- 172.21.15.45:0/1992409160 learned_addr learned my addr 172.21.15.45:0/1992409160 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:59:06.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.883+0000 7f180e400640 1 --2- 172.21.15.45:0/1992409160 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d573e4290 0x557d573e5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:06.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.883+0000 7f180d000640 1 --2- 172.21.15.45:0/1992409160 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557d573dbc00 0x557d573e3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:06.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.883+0000 7f180da00640 1 -- 172.21.15.45:0/1992409160 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557d573dbc00 msgr2=0x557d573e3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:59:06.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.883+0000 7f180da00640 1 --2- 172.21.15.45:0/1992409160 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557d573dbc00 0x557d573e3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.883+0000 7f180da00640 1 -- 172.21.15.45:0/1992409160 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d573e4290 msgr2=0x557d573e5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:59:06.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.883+0000 7f180da00640 1 --2- 172.21.15.45:0/1992409160 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d573e4290 0x557d573e5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.884+0000 7f180da00640 1 -- 172.21.15.45:0/1992409160 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d570130f0 con 0x557d573daf20 2024-03-28T17:59:06.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.884+0000 7f180da00640 1 --2- 172.21.15.45:0/1992409160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573daf20 0x557d573e33d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f17fc00f810 tx=0x7f17fc00f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f14b312f39fd435 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:06.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.884+0000 7f17fa000640 1 -- 172.21.15.45:0/1992409160 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17fc028070 con 0x557d573daf20 2024-03-28T17:59:06.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.884+0000 7f17fa000640 1 -- 172.21.15.45:0/1992409160 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17fc00fa50 con 0x557d573daf20 2024-03-28T17:59:06.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.884+0000 7f17fa000640 1 -- 172.21.15.45:0/1992409160 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17fc005830 con 0x557d573daf20 2024-03-28T17:59:06.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.885+0000 7f18131c3740 1 -- 172.21.15.45:0/1992409160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573daf20 msgr2=0x557d573e33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:06.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.885+0000 7f18131c3740 1 --2- 172.21.15.45:0/1992409160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573daf20 0x557d573e33d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f17fc00f810 tx=0x7f17fc00f690 comp rx=0 tx=0).stop 2024-03-28T17:59:06.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.885+0000 7f18131c3740 1 -- 172.21.15.45:0/1992409160 shutdown_connections 2024-03-28T17:59:06.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.885+0000 7f18131c3740 1 --2- 172.21.15.45:0/1992409160 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d573e4290 0x557d573e5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.885+0000 7f18131c3740 1 --2- 172.21.15.45:0/1992409160 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557d573dbc00 0x557d573e3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.885+0000 7f18131c3740 1 --2- 172.21.15.45:0/1992409160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573daf20 0x557d573e33d0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.885+0000 7f18131c3740 1 -- 172.21.15.45:0/1992409160 >> 172.21.15.45:0/1992409160 conn(0x557d573a2200 msgr2=0x557d573df5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:06.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.885+0000 7f18131c3740 1 -- 172.21.15.45:0/1992409160 shutdown_connections 2024-03-28T17:59:06.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.885+0000 7f18131c3740 1 -- 172.21.15.45:0/1992409160 wait complete. 2024-03-28T17:59:06.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.885+0000 7f18131c3740 1 -- start start 2024-03-28T17:59:06.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.885+0000 7f18131c3740 1 Processor -- start 2024-03-28T17:59:06.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.886+0000 7f18131c3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d573daf20 0x557d573dd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:06.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.886+0000 7f18131c3740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557d573dbc00 0x557d573dd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:06.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.886+0000 7f18131c3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573e4290 0x557d573ddd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:06.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.886+0000 7f18131c3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18081b5aa0 con 0x557d573daf20 2024-03-28T17:59:06.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.886+0000 7f18131c3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18081b5440 con 0x557d573e4290 2024-03-28T17:59:06.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.886+0000 7f18131c3740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18081b30c0 con 0x557d573dbc00 2024-03-28T17:59:06.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.886+0000 7f18131c3740 10 client.0 _ll_register_callbacks cb 0x7f17f400bde0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:59:06.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.886+0000 7f180da00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d573daf20 0x557d573dd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:06.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.886+0000 7f180e400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573e4290 0x557d573ddd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:06.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.886+0000 7f180da00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d573daf20 0x557d573dd300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33182/0 (socket says 172.21.15.45:33182) 2024-03-28T17:59:06.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.886+0000 7f180da00640 1 -- 172.21.15.45:0/1615989129 learned_addr learned my addr 172.21.15.45:0/1615989129 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:59:06.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.886+0000 7f180e400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573e4290 0x557d573ddd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:58616/0 (socket says 172.21.15.45:58616) 2024-03-28T17:59:06.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.886+0000 7f180d000640 1 --2- 172.21.15.45:0/1615989129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557d573dbc00 0x557d573dd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:06.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.887+0000 7f180e400640 1 -- 172.21.15.45:0/1615989129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557d573dbc00 msgr2=0x557d573dd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:59:06.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.887+0000 7f180e400640 1 --2- 172.21.15.45:0/1615989129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557d573dbc00 0x557d573dd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.887+0000 7f180e400640 1 -- 172.21.15.45:0/1615989129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d573daf20 msgr2=0x557d573dd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:59:06.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.887+0000 7f180e400640 1 --2- 172.21.15.45:0/1615989129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d573daf20 0x557d573dd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.887+0000 7f180e400640 1 -- 172.21.15.45:0/1615989129 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d5744ec60 con 0x557d573e4290 2024-03-28T17:59:06.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.887+0000 7f180e400640 1 --2- 172.21.15.45:0/1615989129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573e4290 0x557d573ddd80 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f18000ee4e0 tx=0x7f18000bcff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=257f8aa71ad23309 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:06.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.887+0000 7f17ef400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:59:06.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.887+0000 7f17ef400640 1 -- 172.21.15.45:0/1615989129 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18000c1540 con 0x557d573e4290 2024-03-28T17:59:06.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.887+0000 7f18131c3740 1 -- 172.21.15.45:0/1615989129 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f17fc00fa50 con 0x557d573e4290 2024-03-28T17:59:06.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.887+0000 7f17ef400640 1 -- 172.21.15.45:0/1615989129 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(125..125 src has 1..125) v4 ==== 11178+0+0 (secure 0 0 0) 0x7f18000c1170 con 0x557d573e4290 2024-03-28T17:59:06.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.887+0000 7f17ef400640 1 -- 172.21.15.45:0/1615989129 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=228) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18000c5b90 con 0x557d573e4290 2024-03-28T17:59:06.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.887+0000 7f18131c3740 10 client.26369 fetch_fsmap learned FSMap version 228 2024-03-28T17:59:06.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.887+0000 7f18131c3740 1 -- 172.21.15.45:0/1615989129 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=228,monmap=4+}) v3 -- 0x557d5744fe50 con 0x557d573e4290 2024-03-28T17:59:06.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.887+0000 7f17ef400640 1 -- 172.21.15.45:0/1615989129 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17fc00fa50 con 0x557d573e4290 2024-03-28T17:59:06.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.888+0000 7f17ef400640 1 -- 172.21.15.45:0/1615989129 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 228) v1 ==== 5618+0+0 (secure 0 0 0) 0x7f18000c5700 con 0x557d573e4290 2024-03-28T17:59:06.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.888+0000 7f18131c3740 10 client.26369 fetch_fsmap finished waiting for FSMap version 228 2024-03-28T17:59:06.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.888+0000 7f18131c3740 10 client.26369 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:59:06.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.888+0000 7f18131c3740 20 client.26369 populate_metadata read hostname 'smithi045' 2024-03-28T17:59:06.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.888+0000 7f18131c3740 1 --2- 172.21.15.45:0/1615989129 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x557d57450520 0x557d57452940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:06.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.888+0000 7f18131c3740 4 client.26369 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:59:06.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.888+0000 7f18131c3740 1 -- 172.21.15.45:0/1615989129 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557d57346ae0 con 0x557d57450520 2024-03-28T17:59:06.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.888+0000 7f180da00640 1 --2- 172.21.15.45:0/1615989129 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x557d57450520 0x557d57452940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:06.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.888+0000 7f180da00640 1 --2- 172.21.15.45:0/1615989129 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x557d57450520 0x557d57452940 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d17cfa2b822de06 server_cookie=a1e4e8cf6f4fec16 in_seq=0 out_seq=0 2024-03-28T17:59:06.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.888+0000 7f17ef400640 10 client.26369 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:59:06.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.889+0000 7f17ef400640 1 -- 172.21.15.45:0/1615989129 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x557d57346ae0 con 0x557d57450520 2024-03-28T17:59:06.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.889+0000 7f17ef400640 10 client.26369 handle_command_reply: tid=0 2024-03-28T17:59:06.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.889+0000 7f18131c3740 1 client.26369 shutdown 2024-03-28T17:59:06.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.889+0000 7f18131c3740 1 -- 172.21.15.45:0/1615989129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573e4290 msgr2=0x557d573ddd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:06.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.889+0000 7f18131c3740 1 --2- 172.21.15.45:0/1615989129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573e4290 0x557d573ddd80 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f18000ee4e0 tx=0x7f18000bcff0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.889+0000 7f18131c3740 1 -- 172.21.15.45:0/1615989129 shutdown_connections 2024-03-28T17:59:06.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.889+0000 7f18131c3740 1 --2- 172.21.15.45:0/1615989129 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x557d57450520 0x557d57452940 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.889+0000 7f18131c3740 1 --2- 172.21.15.45:0/1615989129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557d573e4290 0x557d573ddd80 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.889+0000 7f18131c3740 1 --2- 172.21.15.45:0/1615989129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557d573dbc00 0x557d573dd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.889+0000 7f18131c3740 1 --2- 172.21.15.45:0/1615989129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d573daf20 0x557d573dd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.889+0000 7f17ef400640 0 client.26369 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:59:06.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.889+0000 7f18131c3740 1 -- 172.21.15.45:0/1615989129 >> 172.21.15.45:0/1615989129 conn(0x557d573a2200 msgr2=0x557d573d4a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:06.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.889+0000 7f18131c3740 1 -- 172.21.15.45:0/1615989129 shutdown_connections 2024-03-28T17:59:06.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.889+0000 7f18131c3740 1 -- 172.21.15.45:0/1615989129 wait complete. 2024-03-28T17:59:06.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.889+0000 7f18131c3740 20 client.26369 trim_cache size 0 max 16384 2024-03-28T17:59:06.895 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:59:06.895 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T17:59:06.895 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T17:59:06.895 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T17:59:06.895 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T17:59:06.895 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T17:59:06.895 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T17:59:06.895 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:59:06.895 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:59:06.895 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:59:06.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.890+0000 7f1810400640 1 -- 172.21.15.45:0/2771385848 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f17f00a29f0 msgr2=0x7f17f00a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:06.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.890+0000 7f1810400640 1 --2- 172.21.15.45:0/2771385848 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f17f00a29f0 0x7f17f00a4eb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f17f400bfd0 tx=0x7f17f400b410 comp rx=0 tx=0).stop 2024-03-28T17:59:06.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.890+0000 7f1810400640 1 -- 172.21.15.45:0/2771385848 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18081a2da0 msgr2=0x7f18081a3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:06.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.890+0000 7f1810400640 1 --2- 172.21.15.45:0/2771385848 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18081a2da0 0x7f18081a3250 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1800036040 tx=0x7f1800004270 comp rx=0 tx=0).stop 2024-03-28T17:59:06.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.890+0000 7f1810400640 1 -- 172.21.15.45:0/2771385848 shutdown_connections 2024-03-28T17:59:06.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.890+0000 7f1810400640 1 --2- 172.21.15.45:0/2771385848 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f17f00a29f0 0x7f17f00a4eb0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.890+0000 7f1810400640 1 --2- 172.21.15.45:0/2771385848 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18081a2da0 0x7f18081a3250 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.890+0000 7f1810400640 1 --2- 172.21.15.45:0/2771385848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f180810cf80 0x7f180819ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.890+0000 7f1810400640 1 --2- 172.21.15.45:0/2771385848 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1808107eb0 0x7f180819e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:06.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.890+0000 7f1810400640 1 -- 172.21.15.45:0/2771385848 >> 172.21.15.45:0/2771385848 conn(0x7f1808075be0 msgr2=0x7f18080766e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:06.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.890+0000 7f1810400640 1 -- 172.21.15.45:0/2771385848 shutdown_connections 2024-03-28T17:59:06.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:06.890+0000 7f1810400640 1 -- 172.21.15.45:0/2771385848 wait complete. 2024-03-28T17:59:06.904 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 399 sec latency 514802.005 ms 2024-03-28T17:59:06.909 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T17:59:07.906 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 400 sec latency 515802.143 ms 2024-03-28T17:59:08.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:08 smithi071 ceph-mon[36597]: pgmap v344: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:59:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:08 smithi151 ceph-mon[36481]: pgmap v344: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:59:08.905 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 401 sec latency 516802.279 ms 2024-03-28T17:59:08.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:08 smithi045 ceph-mon[31773]: pgmap v344: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:59:09.905 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 402 sec latency 517802.430 ms 2024-03-28T17:59:10.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:10 smithi071 ceph-mon[36597]: pgmap v345: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:59:10.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:10 smithi151 ceph-mon[36481]: pgmap v345: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:59:10.905 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 403 sec latency 518802.590 ms 2024-03-28T17:59:10.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:10 smithi045 ceph-mon[31773]: pgmap v345: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-28T17:59:11.906 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 404 sec latency 519802.789 ms 2024-03-28T17:59:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:12 smithi071 ceph-mon[36597]: pgmap v346: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 26 op/s 2024-03-28T17:59:12.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:12 smithi151 ceph-mon[36481]: pgmap v346: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 26 op/s 2024-03-28T17:59:12.906 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 405 sec latency 520802.941 ms 2024-03-28T17:59:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:12 smithi045 ceph-mon[31773]: pgmap v346: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 26 op/s 2024-03-28T17:59:13.906 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 406 sec latency 521803.073 ms 2024-03-28T17:59:14.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:14 smithi071 ceph-mon[36597]: pgmap v347: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-03-28T17:59:14.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:14 smithi151 ceph-mon[36481]: pgmap v347: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-03-28T17:59:14.906 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 407 sec latency 522803.215 ms 2024-03-28T17:59:14.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:14 smithi045 ceph-mon[31773]: pgmap v347: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-03-28T17:59:15.906 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 408 sec latency 523803.328 ms 2024-03-28T17:59:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:16 smithi071 ceph-mon[36597]: pgmap v348: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-03-28T17:59:16.815 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:16 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Scheduled restart job, restart counter is at 4. 2024-03-28T17:59:16.815 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:16 smithi045 systemd[1]: Stopped Ceph mds.a for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:59:16.815 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:16 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Consumed 1.658s CPU time. 2024-03-28T17:59:16.815 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:16 smithi045 systemd[1]: Starting Ceph mds.a for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T17:59:16.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:16 smithi045 ceph-mon[31773]: pgmap v348: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-03-28T17:59:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:16 smithi151 ceph-mon[36481]: pgmap v348: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-03-28T17:59:16.906 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 409 sec latency 524803.417 ms 2024-03-28T17:59:17.165 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:16 smithi045 podman[92958]: 2024-03-28 17:59:16.721041929 +0000 UTC m=+0.038762983 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T17:59:17.166 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:16 smithi045 podman[92958]: 2024-03-28 17:59:16.848409553 +0000 UTC m=+0.166130638 container create 2aadaf9109441b1e07f43d04479a4d5d224f6cda7d1eb6ee26035497328a0acb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True) 2024-03-28T17:59:17.555 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:17 smithi045 podman[92958]: 2024-03-28 17:59:17.28543984 +0000 UTC m=+0.603160945 container init 2aadaf9109441b1e07f43d04479a4d5d224f6cda7d1eb6ee26035497328a0acb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-28T17:59:17.555 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:17 smithi045 podman[92958]: 2024-03-28 17:59:17.295762927 +0000 UTC m=+0.613484042 container start 2aadaf9109441b1e07f43d04479a4d5d224f6cda7d1eb6ee26035497328a0acb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-28T17:59:17.555 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:17 smithi045 ceph-mds[92977]: -- 172.21.15.45:0/2172416718 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560d69915a40 con 0x560d698e8c00 2024-03-28T17:59:17.555 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: starting mds.a at 2024-03-28T17:59:17.555 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:17 smithi045 bash[92958]: 2aadaf9109441b1e07f43d04479a4d5d224f6cda7d1eb6ee26035497328a0acb 2024-03-28T17:59:17.555 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:17 smithi045 systemd[1]: Started Ceph mds.a for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:59:17.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:17 smithi071 ceph-mon[36597]: daemon mds.a restarted 2024-03-28T17:59:17.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:17 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:59:17.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:17 smithi071 ceph-mon[36597]: osdmap e126: 12 total, 12 up, 12 in 2024-03-28T17:59:17.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:17 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/3511256023,v1:172.21.15.45:6837/3511256023] up:boot 2024-03-28T17:59:17.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:17 smithi071 ceph-mon[36597]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-03-28T17:59:17.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:17 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:59:17.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:17 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T17:59:17.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:17 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:17 smithi151 ceph-mon[36481]: daemon mds.a restarted 2024-03-28T17:59:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:17 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:59:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:17 smithi151 ceph-mon[36481]: osdmap e126: 12 total, 12 up, 12 in 2024-03-28T17:59:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:17 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/3511256023,v1:172.21.15.45:6837/3511256023] up:boot 2024-03-28T17:59:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:17 smithi151 ceph-mon[36481]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-03-28T17:59:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:17 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:59:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:17 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T17:59:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:17 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:17.906 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 410 sec latency 525803.563 ms 2024-03-28T17:59:17.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:17 smithi045 ceph-mon[31773]: daemon mds.a restarted 2024-03-28T17:59:17.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:17 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:59:17.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:17 smithi045 ceph-mon[31773]: osdmap e126: 12 total, 12 up, 12 in 2024-03-28T17:59:17.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:17 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/3511256023,v1:172.21.15.45:6837/3511256023] up:boot 2024-03-28T17:59:17.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:17 smithi045 ceph-mon[31773]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-03-28T17:59:17.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:17 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-28T17:59:17.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:17 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T17:59:17.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:17 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:replay} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:18.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:18 smithi151 ceph-mon[36481]: pgmap v349: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T17:59:18.906 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 411 sec latency 526803.670 ms 2024-03-28T17:59:18.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:18 smithi045 ceph-mon[31773]: pgmap v349: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T17:59:19.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:18 smithi071 ceph-mon[36597]: pgmap v349: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T17:59:19.906 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 412 sec latency 527803.801 ms 2024-03-28T17:59:19.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:19 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/1729218296,v1:172.21.15.151:6833/1729218296] up:resolve 2024-03-28T17:59:19.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:19 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:20.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:19 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/1729218296,v1:172.21.15.151:6833/1729218296] up:resolve 2024-03-28T17:59:20.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:19 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:20.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:19 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/1729218296,v1:172.21.15.151:6833/1729218296] up:resolve 2024-03-28T17:59:20.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:19 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:resolve} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:20.907 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 413 sec latency 528803.965 ms 2024-03-28T17:59:20.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:20 smithi045 ceph-mon[31773]: pgmap v351: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:59:20.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:20 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/1729218296,v1:172.21.15.151:6833/1729218296] up:reconnect 2024-03-28T17:59:20.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:20 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:20.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:20 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:59:20.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:20 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00999988 2024-03-28T17:59:21.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:20 smithi071 ceph-mon[36597]: pgmap v351: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:59:21.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:20 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/1729218296,v1:172.21.15.151:6833/1729218296] up:reconnect 2024-03-28T17:59:21.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:20 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:21.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:20 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:59:21.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:20 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00999988 2024-03-28T17:59:21.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:20 smithi151 ceph-mon[36481]: pgmap v351: 97 pgs: 97 active+clean; 27 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T17:59:21.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:20 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/1729218296,v1:172.21.15.151:6833/1729218296] up:reconnect 2024-03-28T17:59:21.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:20 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:reconnect} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:21.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:20 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899989 2024-03-28T17:59:21.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:20 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00999988 2024-03-28T17:59:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/1729218296,v1:172.21.15.151:6833/1729218296] up:rejoin 2024-03-28T17:59:21.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:21.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-mon[36481]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-03-28T17:59:21.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:59:21.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-mon[36481]: Cluster is now healthy 2024-03-28T17:59:21.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.151:6832/1729218296,v1:172.21.15.151:6833/1729218296] up:active 2024-03-28T17:59:21.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:59:21.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd5f11e6700 time 2024-03-28T17:59:21.439276+0000 2024-03-28T17:59:21.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:59:21.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:21.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd5fb8104a1] 2024-03-28T17:59:21.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd5fb810667] 2024-03-28T17:59:21.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c1cf20a223] 2024-03-28T17:59:21.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c1cf0845d6] 2024-03-28T17:59:21.827 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c1cf084da1] 2024-03-28T17:59:21.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c1cf0872d5] 2024-03-28T17:59:21.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c1cf0d7233] 2024-03-28T17:59:21.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c1cf0d9197] 2024-03-28T17:59:21.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c1cefd811b] 2024-03-28T17:59:21.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c1cefd86bc] 2024-03-28T17:59:21.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c1cefc21ff] 2024-03-28T17:59:21.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd5fbaa2488] 2024-03-28T17:59:21.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 13: (DispatchQueue::entry()+0x51f) [0x7fd5fba9f60f] 2024-03-28T17:59:21.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd5fbb72e81] 2024-03-28T17:59:21.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd5fa58d1ca] 2024-03-28T17:59:21.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 16: clone() 2024-03-28T17:59:21.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 2024-03-28T17:59:21.440+0000 7fd5f11e6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd5f11e6700 time 2024-03-28T17:59:21.439276+0000 2024-03-28T17:59:21.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:59:21.828 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 2024-03-28T17:59:21.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:21.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd5fb8104a1] 2024-03-28T17:59:21.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd5fb810667] 2024-03-28T17:59:21.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c1cf20a223] 2024-03-28T17:59:21.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c1cf0845d6] 2024-03-28T17:59:21.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c1cf084da1] 2024-03-28T17:59:21.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c1cf0872d5] 2024-03-28T17:59:21.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c1cf0d7233] 2024-03-28T17:59:21.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c1cf0d9197] 2024-03-28T17:59:21.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c1cefd811b] 2024-03-28T17:59:21.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c1cefd86bc] 2024-03-28T17:59:21.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c1cefc21ff] 2024-03-28T17:59:21.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd5fbaa2488] 2024-03-28T17:59:21.829 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 13: (DispatchQueue::entry()+0x51f) [0x7fd5fba9f60f] 2024-03-28T17:59:21.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd5fbb72e81] 2024-03-28T17:59:21.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd5fa58d1ca] 2024-03-28T17:59:21.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 16: clone() 2024-03-28T17:59:21.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 2024-03-28T17:59:21.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: *** Caught signal (Aborted) ** 2024-03-28T17:59:21.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: in thread 7fd5f11e6700 thread_name:ms_dispatch 2024-03-28T17:59:21.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:21.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd5fa597d20] 2024-03-28T17:59:21.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 2: gsignal() 2024-03-28T17:59:21.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 3: abort() 2024-03-28T17:59:21.830 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd5fb8104fb] 2024-03-28T17:59:21.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd5fb810667] 2024-03-28T17:59:21.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c1cf20a223] 2024-03-28T17:59:21.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c1cf0845d6] 2024-03-28T17:59:21.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c1cf084da1] 2024-03-28T17:59:21.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c1cf0872d5] 2024-03-28T17:59:21.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c1cf0d7233] 2024-03-28T17:59:21.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c1cf0d9197] 2024-03-28T17:59:21.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c1cefd811b] 2024-03-28T17:59:21.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c1cefd86bc] 2024-03-28T17:59:21.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c1cefc21ff] 2024-03-28T17:59:21.831 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd5fbaa2488] 2024-03-28T17:59:21.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 16: (DispatchQueue::entry()+0x51f) [0x7fd5fba9f60f] 2024-03-28T17:59:21.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd5fbb72e81] 2024-03-28T17:59:21.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd5fa58d1ca] 2024-03-28T17:59:21.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 19: clone() 2024-03-28T17:59:21.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 2024-03-28T17:59:21.442+0000 7fd5f11e6700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:59:21.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: in thread 7fd5f11e6700 thread_name:ms_dispatch 2024-03-28T17:59:21.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 2024-03-28T17:59:21.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:21.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd5fa597d20] 2024-03-28T17:59:21.832 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 2: gsignal() 2024-03-28T17:59:21.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 3: abort() 2024-03-28T17:59:21.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd5fb8104fb] 2024-03-28T17:59:21.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd5fb810667] 2024-03-28T17:59:21.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c1cf20a223] 2024-03-28T17:59:21.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c1cf0845d6] 2024-03-28T17:59:21.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c1cf084da1] 2024-03-28T17:59:21.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c1cf0872d5] 2024-03-28T17:59:21.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c1cf0d7233] 2024-03-28T17:59:21.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c1cf0d9197] 2024-03-28T17:59:21.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c1cefd811b] 2024-03-28T17:59:21.833 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c1cefd86bc] 2024-03-28T17:59:21.834 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c1cefc21ff] 2024-03-28T17:59:21.834 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd5fbaa2488] 2024-03-28T17:59:21.834 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 16: (DispatchQueue::entry()+0x51f) [0x7fd5fba9f60f] 2024-03-28T17:59:21.834 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd5fbb72e81] 2024-03-28T17:59:21.834 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd5fa58d1ca] 2024-03-28T17:59:21.834 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 19: clone() 2024-03-28T17:59:21.834 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:59:21.834 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 2024-03-28T17:59:21.834 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: -1> 2024-03-28T17:59:21.440+0000 7fd5f11e6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd5f11e6700 time 2024-03-28T17:59:21.439276+0000 2024-03-28T17:59:21.834 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:59:21.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 2024-03-28T17:59:21.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:21.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd5fb8104a1] 2024-03-28T17:59:21.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd5fb810667] 2024-03-28T17:59:21.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c1cf20a223] 2024-03-28T17:59:21.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c1cf0845d6] 2024-03-28T17:59:21.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c1cf084da1] 2024-03-28T17:59:21.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c1cf0872d5] 2024-03-28T17:59:21.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c1cf0d7233] 2024-03-28T17:59:21.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c1cf0d9197] 2024-03-28T17:59:21.835 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c1cefd811b] 2024-03-28T17:59:21.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c1cefd86bc] 2024-03-28T17:59:21.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c1cefc21ff] 2024-03-28T17:59:21.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd5fbaa2488] 2024-03-28T17:59:21.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 13: (DispatchQueue::entry()+0x51f) [0x7fd5fba9f60f] 2024-03-28T17:59:21.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd5fbb72e81] 2024-03-28T17:59:21.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd5fa58d1ca] 2024-03-28T17:59:21.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 16: clone() 2024-03-28T17:59:21.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 2024-03-28T17:59:21.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 0> 2024-03-28T17:59:21.442+0000 7fd5f11e6700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:59:21.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: in thread 7fd5f11e6700 thread_name:ms_dispatch 2024-03-28T17:59:21.836 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 2024-03-28T17:59:21.837 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:21.837 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd5fa597d20] 2024-03-28T17:59:21.837 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 2: gsignal() 2024-03-28T17:59:21.837 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 3: abort() 2024-03-28T17:59:21.837 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd5fb8104fb] 2024-03-28T17:59:21.837 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd5fb810667] 2024-03-28T17:59:21.837 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c1cf20a223] 2024-03-28T17:59:21.837 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c1cf0845d6] 2024-03-28T17:59:21.837 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c1cf084da1] 2024-03-28T17:59:21.837 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c1cf0872d5] 2024-03-28T17:59:21.837 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c1cf0d7233] 2024-03-28T17:59:21.838 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c1cf0d9197] 2024-03-28T17:59:21.838 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c1cefd811b] 2024-03-28T17:59:21.838 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c1cefd86bc] 2024-03-28T17:59:21.838 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c1cefc21ff] 2024-03-28T17:59:21.838 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd5fbaa2488] 2024-03-28T17:59:21.838 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 16: (DispatchQueue::entry()+0x51f) [0x7fd5fba9f60f] 2024-03-28T17:59:21.838 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd5fbb72e81] 2024-03-28T17:59:21.838 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd5fa58d1ca] 2024-03-28T17:59:21.838 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 19: clone() 2024-03-28T17:59:21.838 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:59:21.838 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 2024-03-28T17:59:21.839 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: -1> 2024-03-28T17:59:21.440+0000 7fd5f11e6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd5f11e6700 time 2024-03-28T17:59:21.439276+0000 2024-03-28T17:59:21.839 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:59:21.839 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 2024-03-28T17:59:21.839 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:21.839 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd5fb8104a1] 2024-03-28T17:59:21.839 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd5fb810667] 2024-03-28T17:59:21.839 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c1cf20a223] 2024-03-28T17:59:21.839 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c1cf0845d6] 2024-03-28T17:59:21.839 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c1cf084da1] 2024-03-28T17:59:21.840 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c1cf0872d5] 2024-03-28T17:59:21.840 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c1cf0d7233] 2024-03-28T17:59:21.840 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c1cf0d9197] 2024-03-28T17:59:21.840 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c1cefd811b] 2024-03-28T17:59:21.840 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c1cefd86bc] 2024-03-28T17:59:21.840 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c1cefc21ff] 2024-03-28T17:59:21.840 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd5fbaa2488] 2024-03-28T17:59:21.840 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 13: (DispatchQueue::entry()+0x51f) [0x7fd5fba9f60f] 2024-03-28T17:59:21.840 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd5fbb72e81] 2024-03-28T17:59:21.840 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd5fa58d1ca] 2024-03-28T17:59:21.840 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 16: clone() 2024-03-28T17:59:21.841 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 2024-03-28T17:59:21.841 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 0> 2024-03-28T17:59:21.442+0000 7fd5f11e6700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:59:21.841 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: in thread 7fd5f11e6700 thread_name:ms_dispatch 2024-03-28T17:59:21.841 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 2024-03-28T17:59:21.841 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:21.841 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd5fa597d20] 2024-03-28T17:59:21.841 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 2: gsignal() 2024-03-28T17:59:21.841 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 3: abort() 2024-03-28T17:59:21.841 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd5fb8104fb] 2024-03-28T17:59:21.841 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd5fb810667] 2024-03-28T17:59:21.841 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c1cf20a223] 2024-03-28T17:59:21.842 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c1cf0845d6] 2024-03-28T17:59:21.842 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c1cf084da1] 2024-03-28T17:59:21.842 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c1cf0872d5] 2024-03-28T17:59:21.842 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c1cf0d7233] 2024-03-28T17:59:21.842 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c1cf0d9197] 2024-03-28T17:59:21.842 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c1cefd811b] 2024-03-28T17:59:21.842 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c1cefd86bc] 2024-03-28T17:59:21.842 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c1cefc21ff] 2024-03-28T17:59:21.842 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd5fbaa2488] 2024-03-28T17:59:21.842 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 16: (DispatchQueue::entry()+0x51f) [0x7fd5fba9f60f] 2024-03-28T17:59:21.843 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd5fbb72e81] 2024-03-28T17:59:21.843 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd5fa58d1ca] 2024-03-28T17:59:21.843 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 19: clone() 2024-03-28T17:59:21.843 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:59:21.843 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l[74131]: 2024-03-28T17:59:21.843 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:21 smithi151 podman[74172]: 2024-03-28 17:59:21.496590976 +0000 UTC m=+0.020031457 container died dc8464e8f6b83f207b23264d8f85aa8023a121bd95704f7f3123f778dfb3357f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T17:59:21.907 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 414 sec latency 529804.115 ms 2024-03-28T17:59:21.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:21 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/1729218296,v1:172.21.15.151:6833/1729218296] up:rejoin 2024-03-28T17:59:21.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:21 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:21.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:21 smithi045 ceph-mon[31773]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-03-28T17:59:21.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:21 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:59:21.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:21 smithi045 ceph-mon[31773]: Cluster is now healthy 2024-03-28T17:59:21.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:21 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.151:6832/1729218296,v1:172.21.15.151:6833/1729218296] up:active 2024-03-28T17:59:21.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:21 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:59:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:21 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/1729218296,v1:172.21.15.151:6833/1729218296] up:rejoin 2024-03-28T17:59:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:21 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=l=up:rejoin} 2 up:standby-replay 3 up:standby 2024-03-28T17:59:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:21 smithi071 ceph-mon[36597]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-03-28T17:59:22.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:21 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:59:22.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:21 smithi071 ceph-mon[36597]: Cluster is now healthy 2024-03-28T17:59:22.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:21 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.151:6832/1729218296,v1:172.21.15.151:6833/1729218296] up:active 2024-03-28T17:59:22.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:21 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 2 up:standby 2024-03-28T17:59:22.541 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:22 smithi151 podman[74172]: 2024-03-28 17:59:22.288157841 +0000 UTC m=+0.811598299 container remove dc8464e8f6b83f207b23264d8f85aa8023a121bd95704f7f3123f778dfb3357f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-l, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-28T17:59:22.541 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:22 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:59:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:22 smithi151 ceph-mon[36481]: pgmap v352: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T17:59:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:22 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:59:22.823 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:22 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-28T17:59:22.907 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 415 sec latency 530804.189 ms 2024-03-28T17:59:22.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:22 smithi045 ceph-mon[31773]: pgmap v352: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T17:59:22.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:22 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:59:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:22 smithi071 ceph-mon[36597]: pgmap v352: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T17:59:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:22 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T17:59:23.907 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 416 sec latency 531804.252 ms 2024-03-28T17:59:24.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:23 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:24.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:23 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:24.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:23 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:24.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:23 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:24.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:23 smithi071 ceph-mon[36597]: pgmap v353: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T17:59:24.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:23 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:24.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:23 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:24.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:23 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:24.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:23 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:24.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:23 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:24.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:23 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:24.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:23 smithi151 ceph-mon[36481]: pgmap v353: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T17:59:24.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:23 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:24.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:23 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:24.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:23 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:24.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:23 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:24.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:23 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:24.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:23 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:24.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:23 smithi045 ceph-mon[31773]: pgmap v353: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T17:59:24.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:23 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:24.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:23 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:24.912 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 417 sec latency 532808.938 ms 2024-03-28T17:59:25.912 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 418 sec latency 533809.049 ms 2024-03-28T17:59:26.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:25 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:26.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:25 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:26.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:25 smithi045 ceph-mon[31773]: pgmap v354: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:59:26.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:25 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:26.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:25 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:26.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:25 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:25 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:25 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:25 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:25 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:25 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:25 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:25 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:25 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T17:59:26.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:25 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T17:59:26.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:25 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:26.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:25 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:26.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:25 smithi071 ceph-mon[36597]: pgmap v354: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:59:26.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:25 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:26.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:25 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:26.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:25 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:25 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:25 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:25 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:25 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:25 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:25 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:25 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:25 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T17:59:26.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:25 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T17:59:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:25 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:25 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:25 smithi151 ceph-mon[36481]: pgmap v354: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:59:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:25 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:25 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:25 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:25 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:25 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:25 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:25 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:25 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:25 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:25 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:26.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:25 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T17:59:26.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:25 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T17:59:26.912 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 419 sec latency 534809.136 ms 2024-03-28T17:59:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:26 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:26 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:26 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:26 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:26 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:26 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:26 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:26 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:26 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T17:59:27.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:26 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T17:59:27.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:26 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:27.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:26 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:26 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:26 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:26 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:26 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:26 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:26 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:26 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:26 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:26 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T17:59:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:26 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T17:59:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:26 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:27.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:26 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:26 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:26 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:26 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:26 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:26 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:26 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:26 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:26 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:26 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T17:59:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:26 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T17:59:27.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:26 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:27.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:26 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:27.912 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 420 sec latency 535809.234 ms 2024-03-28T17:59:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:27 smithi045 ceph-mon[31773]: pgmap v355: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:59:28.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:27 smithi071 ceph-mon[36597]: pgmap v355: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:59:28.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:27 smithi151 ceph-mon[36481]: pgmap v355: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T17:59:28.912 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 421 sec latency 536809.364 ms 2024-03-28T17:59:29.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:28 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:28 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:28 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:28 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:28 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:28 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:28 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:28 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:28 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T17:59:29.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:28 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T17:59:29.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:28 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:59:29.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:28 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:59:29.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:28 smithi045 ceph-mon[31773]: pgmap v356: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-03-28T17:59:29.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:28 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:29.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:28 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:28 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:28 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:28 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:28 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:28 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:28 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:28 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:28 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T17:59:29.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:28 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T17:59:29.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:28 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:59:29.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:28 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:59:29.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:28 smithi071 ceph-mon[36597]: pgmap v356: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-03-28T17:59:29.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:28 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:29.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:28 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:28 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:28 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:28 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:28 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:28 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:28 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:28 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T17:59:29.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:28 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T17:59:29.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:28 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T17:59:29.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:28 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T17:59:29.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:28 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T17:59:29.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:28 smithi151 ceph-mon[36481]: pgmap v356: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-03-28T17:59:29.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:28 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T17:59:29.912 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 422 sec latency 537809.485 ms 2024-03-28T17:59:30.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:29 smithi045 ceph-mon[31773]: Health check failed: 5 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-28T17:59:30.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:29 smithi071 ceph-mon[36597]: Health check failed: 5 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-28T17:59:30.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:29 smithi151 ceph-mon[36481]: Health check failed: 5 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-28T17:59:30.912 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 423 sec latency 538809.602 ms 2024-03-28T17:59:31.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:30 smithi045 ceph-mon[31773]: pgmap v357: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-03-28T17:59:31.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:30 smithi071 ceph-mon[36597]: pgmap v357: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-03-28T17:59:31.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:30 smithi151 ceph-mon[36481]: pgmap v357: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-03-28T17:59:31.912 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 424 sec latency 539809.787 ms 2024-03-28T17:59:32.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:31 smithi045 ceph-mon[31773]: osdmap e127: 12 total, 12 up, 12 in 2024-03-28T17:59:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:31 smithi071 ceph-mon[36597]: osdmap e127: 12 total, 12 up, 12 in 2024-03-28T17:59:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:31 smithi151 ceph-mon[36481]: osdmap e127: 12 total, 12 up, 12 in 2024-03-28T17:59:32.912 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 425 sec latency 540809.876 ms 2024-03-28T17:59:32.920 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:32 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 6. 2024-03-28T17:59:32.920 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:32 smithi151 systemd[1]: Stopped Ceph mds.l for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:59:32.920 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:32 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.l.service: Start request repeated too quickly. 2024-03-28T17:59:32.920 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:32 smithi151 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-28T17:59:32.920 INFO:journalctl@ceph.mds.l.smithi151.stdout:Mar 28 17:59:32 smithi151 systemd[1]: Failed to start Ceph mds.l for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:59:33.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:32 smithi071 ceph-mon[36597]: pgmap v359: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 31 KiB/s wr, 15 op/s 2024-03-28T17:59:33.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:32 smithi151 ceph-mon[36481]: pgmap v359: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 31 KiB/s wr, 15 op/s 2024-03-28T17:59:33.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:32 smithi045 ceph-mon[31773]: pgmap v359: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 31 KiB/s wr, 15 op/s 2024-03-28T17:59:33.913 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 426 sec latency 541810.017 ms 2024-03-28T17:59:34.913 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 427 sec latency 542810.113 ms 2024-03-28T17:59:35.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:35 smithi071 ceph-mon[36597]: pgmap v360: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 116 B/s wr, 3 op/s 2024-03-28T17:59:35.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:35 smithi151 ceph-mon[36481]: pgmap v360: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 116 B/s wr, 3 op/s 2024-03-28T17:59:35.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:35 smithi045 ceph-mon[31773]: pgmap v360: 97 pgs: 97 active+clean; 27 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 116 B/s wr, 3 op/s 2024-03-28T17:59:35.913 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 428 sec latency 543810.262 ms 2024-03-28T17:59:36.910 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T17:59:36.911 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T17:59:36.919 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 429 sec latency 544810.409 ms 2024-03-28T17:59:37.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.151+0000 7fbb03000640 1 -- 172.21.15.45:0/2839843672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbafc103720 msgr2=0x7fbafc103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:37.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.151+0000 7fbb03000640 1 --2- 172.21.15.45:0/2839843672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbafc103720 0x7fbafc103ba0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbae8009900 tx=0x7fbae802bc30 comp rx=0 tx=0).stop 2024-03-28T17:59:37.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.151+0000 7fbb03000640 1 -- 172.21.15.45:0/2839843672 shutdown_connections 2024-03-28T17:59:37.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.151+0000 7fbb03000640 1 --2- 172.21.15.45:0/2839843672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbafc1040e0 0x7fbafc1105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.151+0000 7fbb03000640 1 --2- 172.21.15.45:0/2839843672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbafc103720 0x7fbafc103ba0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.151+0000 7fbb03000640 1 --2- 172.21.15.45:0/2839843672 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbafc109730 0x7fbafc109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.151+0000 7fbb03000640 1 -- 172.21.15.45:0/2839843672 >> 172.21.15.45:0/2839843672 conn(0x7fbafc0ff1b0 msgr2=0x7fbafc1015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:37.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.152+0000 7fbb03000640 1 -- 172.21.15.45:0/2839843672 shutdown_connections 2024-03-28T17:59:37.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.152+0000 7fbb03000640 1 -- 172.21.15.45:0/2839843672 wait complete. 2024-03-28T17:59:37.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.153+0000 7fbb03000640 1 Processor -- start 2024-03-28T17:59:37.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.153+0000 7fbb03000640 1 -- start start 2024-03-28T17:59:37.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.153+0000 7fbb03000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbafc103720 0x7fbafc19a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:37.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.153+0000 7fbb03000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbafc1040e0 0x7fbafc19a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:37.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.153+0000 7fbb03000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbafc109730 0x7fbafc19ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:37.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.153+0000 7fbb03000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc113b90 con 0x7fbafc1040e0 2024-03-28T17:59:37.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.153+0000 7fbb03000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc113a10 con 0x7fbafc103720 2024-03-28T17:59:37.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.153+0000 7fbb03000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc113d10 con 0x7fbafc109730 2024-03-28T17:59:37.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.153+0000 7fbb01c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbafc103720 0x7fbafc19a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:37.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.153+0000 7fbb01200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbafc1040e0 0x7fbafc19a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:37.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.153+0000 7fbb01c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbafc103720 0x7fbafc19a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:38098/0 (socket says 172.21.15.45:38098) 2024-03-28T17:59:37.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.153+0000 7fbb01c00640 1 -- 172.21.15.45:0/3073062916 learned_addr learned my addr 172.21.15.45:0/3073062916 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:59:37.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.153+0000 7fbb02600640 1 --2- 172.21.15.45:0/3073062916 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbafc109730 0x7fbafc19ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:37.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.154+0000 7fbb01c00640 1 -- 172.21.15.45:0/3073062916 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbafc109730 msgr2=0x7fbafc19ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:37.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.154+0000 7fbb01c00640 1 --2- 172.21.15.45:0/3073062916 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbafc109730 0x7fbafc19ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.154+0000 7fbb01c00640 1 -- 172.21.15.45:0/3073062916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbafc1040e0 msgr2=0x7fbafc19a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:37.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.154+0000 7fbb01c00640 1 --2- 172.21.15.45:0/3073062916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbafc1040e0 0x7fbafc19a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.154+0000 7fbb01c00640 1 -- 172.21.15.45:0/3073062916 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc19f3a0 con 0x7fbafc103720 2024-03-28T17:59:37.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.154+0000 7fbb01200640 1 --2- 172.21.15.45:0/3073062916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbafc1040e0 0x7fbafc19a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:59:37.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.154+0000 7fbb01c00640 1 --2- 172.21.15.45:0/3073062916 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbafc103720 0x7fbafc19a3b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbaf000e9f0 tx=0x7fbaf000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:37.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.154+0000 7fbafa000640 1 -- 172.21.15.45:0/3073062916 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf000ccc0 con 0x7fbafc103720 2024-03-28T17:59:37.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.154+0000 7fbafa000640 1 -- 172.21.15.45:0/3073062916 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbaf0010910 con 0x7fbafc103720 2024-03-28T17:59:37.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.154+0000 7fbb03000640 1 -- 172.21.15.45:0/3073062916 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbafc19f690 con 0x7fbafc103720 2024-03-28T17:59:37.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.155+0000 7fbafa000640 1 -- 172.21.15.45:0/3073062916 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf00188a0 con 0x7fbafc103720 2024-03-28T17:59:37.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.155+0000 7fbb03000640 1 -- 172.21.15.45:0/3073062916 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbafc1a6f20 con 0x7fbafc103720 2024-03-28T17:59:37.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.158+0000 7fbb073e8740 1 Processor -- start 2024-03-28T17:59:37.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.158+0000 7fbb073e8740 1 -- start start 2024-03-28T17:59:37.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.158+0000 7fbafa000640 1 -- 172.21.15.45:0/3073062916 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fbaf0010ac0 con 0x7fbafc103720 2024-03-28T17:59:37.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.158+0000 7fbb073e8740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55965da9edd0 0x55965da9db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:37.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.158+0000 7fbb073e8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9e0f0 0x55965da9e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:37.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.158+0000 7fbb073e8740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965daa7460 0x55965daa9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:37.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.158+0000 7fbb073e8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55965da02250 con 0x55965da9e0f0 2024-03-28T17:59:37.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.158+0000 7fbb073e8740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55965d9aaa30 con 0x55965daa7460 2024-03-28T17:59:37.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.158+0000 7fbb073e8740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55965d9fe2f0 con 0x55965da9edd0 2024-03-28T17:59:37.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.159+0000 7fbb01200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965daa7460 0x55965daa9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:37.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.159+0000 7fbb01c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9e0f0 0x55965da9e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:37.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.159+0000 7fbb01200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965daa7460 0x55965daa9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:38112/0 (socket says 172.21.15.45:38112) 2024-03-28T17:59:37.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.159+0000 7fbb01200640 1 -- 172.21.15.45:0/1887337462 learned_addr learned my addr 172.21.15.45:0/1887337462 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:59:37.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.159+0000 7fbb02600640 1 --2- 172.21.15.45:0/1887337462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55965da9edd0 0x55965da9db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:37.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.159+0000 7fbb01200640 1 -- 172.21.15.45:0/1887337462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55965da9edd0 msgr2=0x55965da9db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:59:37.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.159+0000 7fbb01200640 1 --2- 172.21.15.45:0/1887337462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55965da9edd0 0x55965da9db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.159+0000 7fbb01200640 1 -- 172.21.15.45:0/1887337462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9e0f0 msgr2=0x55965da9e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:59:37.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.159+0000 7fbb01200640 1 --2- 172.21.15.45:0/1887337462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9e0f0 0x55965da9e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.159+0000 7fbb01200640 1 -- 172.21.15.45:0/1887337462 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55965d995690 con 0x55965daa7460 2024-03-28T17:59:37.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.159+0000 7fbb01c00640 1 --2- 172.21.15.45:0/1887337462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9e0f0 0x55965da9e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:59:37.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.160+0000 7fbb01200640 1 --2- 172.21.15.45:0/1887337462 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965daa7460 0x55965daa9850 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fbae8009230 tx=0x7fbae8009670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0d23e8892c9a5b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:37.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.160+0000 7fbaeea00640 1 -- 172.21.15.45:0/1887337462 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae8042070 con 0x55965daa7460 2024-03-28T17:59:37.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.160+0000 7fbaeea00640 1 -- 172.21.15.45:0/1887337462 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbae8004a70 con 0x55965daa7460 2024-03-28T17:59:37.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.160+0000 7fbaeea00640 1 -- 172.21.15.45:0/1887337462 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae8005250 con 0x55965daa7460 2024-03-28T17:59:37.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.160+0000 7fbafa000640 1 --2- 172.21.15.45:0/3073062916 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbae40a29f0 0x7fbae40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:37.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.160+0000 7fbafa000640 1 -- 172.21.15.45:0/3073062916 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 11394+0+0 (secure 0 0 0) 0x7fbaf00cfcf0 con 0x7fbafc103720 2024-03-28T17:59:37.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.160+0000 7fbb02600640 1 --2- 172.21.15.45:0/3073062916 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbae40a29f0 0x7fbae40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:37.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.161+0000 7fbb073e8740 1 -- 172.21.15.45:0/1887337462 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965daa7460 msgr2=0x55965daa9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:37.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.161+0000 7fbb073e8740 1 --2- 172.21.15.45:0/1887337462 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965daa7460 0x55965daa9850 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fbae8009230 tx=0x7fbae8009670 comp rx=0 tx=0).stop 2024-03-28T17:59:37.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.161+0000 7fbb073e8740 1 -- 172.21.15.45:0/1887337462 shutdown_connections 2024-03-28T17:59:37.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.161+0000 7fbb073e8740 1 --2- 172.21.15.45:0/1887337462 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965daa7460 0x55965daa9850 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.161+0000 7fbb073e8740 1 --2- 172.21.15.45:0/1887337462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9e0f0 0x55965da9e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.161+0000 7fbb073e8740 1 --2- 172.21.15.45:0/1887337462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55965da9edd0 0x55965da9db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.161+0000 7fbb073e8740 1 -- 172.21.15.45:0/1887337462 >> 172.21.15.45:0/1887337462 conn(0x55965da65200 msgr2=0x55965da98b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:37.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.161+0000 7fbb073e8740 1 -- 172.21.15.45:0/1887337462 shutdown_connections 2024-03-28T17:59:37.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.161+0000 7fbb073e8740 1 -- 172.21.15.45:0/1887337462 wait complete. 2024-03-28T17:59:37.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.161+0000 7fbb02600640 1 --2- 172.21.15.45:0/3073062916 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbae40a29f0 0x7fbae40a4eb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4005fd0 tx=0x7fbaf4009500 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:37.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.161+0000 7fbb073e8740 1 -- start start 2024-03-28T17:59:37.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.162+0000 7fbb073e8740 1 Processor -- start 2024-03-28T17:59:37.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.162+0000 7fbb073e8740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965da9e0f0 0x55965daa6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:37.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.162+0000 7fbb073e8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9edd0 0x55965daa3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:37.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.162+0000 7fbb073e8740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55965daa7460 0x55965daa43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:37.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.162+0000 7fbb073e8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55965daaa270 con 0x55965da9edd0 2024-03-28T17:59:37.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.162+0000 7fbb073e8740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55965d9bde80 con 0x55965da9e0f0 2024-03-28T17:59:37.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.162+0000 7fbb073e8740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55965daaa3f0 con 0x55965daa7460 2024-03-28T17:59:37.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.162+0000 7fbb073e8740 10 client.0 _ll_register_callbacks cb 0x55965d9d0130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:59:37.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.162+0000 7fbb01c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965da9e0f0 0x55965daa6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:37.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.163+0000 7fbb01c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965da9e0f0 0x55965daa6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:38116/0 (socket says 172.21.15.45:38116) 2024-03-28T17:59:37.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.163+0000 7fbb01c00640 1 -- 172.21.15.45:0/1666509643 learned_addr learned my addr 172.21.15.45:0/1666509643 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:59:37.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.163+0000 7fbb01200640 1 --2- 172.21.15.45:0/1666509643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9edd0 0x55965daa3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:37.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.163+0000 7fbb02600640 1 --2- 172.21.15.45:0/1666509643 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55965daa7460 0x55965daa43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:37.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.163+0000 7fbb01c00640 1 -- 172.21.15.45:0/1666509643 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55965daa7460 msgr2=0x55965daa43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:59:37.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.163+0000 7fbb01c00640 1 --2- 172.21.15.45:0/1666509643 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55965daa7460 0x55965daa43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.163+0000 7fbb01c00640 1 -- 172.21.15.45:0/1666509643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9edd0 msgr2=0x55965daa3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:59:37.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.163+0000 7fbb01c00640 1 --2- 172.21.15.45:0/1666509643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9edd0 0x55965daa3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.163+0000 7fbb01c00640 1 -- 172.21.15.45:0/1666509643 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55965d7e07d0 con 0x55965da9e0f0 2024-03-28T17:59:37.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.163+0000 7fbb01200640 1 --2- 172.21.15.45:0/1666509643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9edd0 0x55965daa3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:59:37.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.163+0000 7fbb02600640 1 --2- 172.21.15.45:0/1666509643 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55965daa7460 0x55965daa43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T17:59:37.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.163+0000 7fbb01c00640 1 --2- 172.21.15.45:0/1666509643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965da9e0f0 0x55965daa6b00 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fbaf00cc900 tx=0x7fbaf000cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a8a5e3dba3a3aa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:37.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.163+0000 7fbae3400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:59:37.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.163+0000 7fbae3400640 1 -- 172.21.15.45:0/1666509643 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf009b110 con 0x55965da9e0f0 2024-03-28T17:59:37.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.163+0000 7fbae3400640 1 -- 172.21.15.45:0/1666509643 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(128..128 src has 1..128) v4 ==== 11394+0+0 (secure 0 0 0) 0x7fbaf00ccb20 con 0x55965da9e0f0 2024-03-28T17:59:37.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.163+0000 7fbb073e8740 1 -- 172.21.15.45:0/1666509643 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbae8004a70 con 0x55965da9e0f0 2024-03-28T17:59:37.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.164+0000 7fbae3400640 1 -- 172.21.15.45:0/1666509643 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=235) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbaf00ccd20 con 0x55965da9e0f0 2024-03-28T17:59:37.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.164+0000 7fbb073e8740 10 client.26387 fetch_fsmap learned FSMap version 235 2024-03-28T17:59:37.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.164+0000 7fbb073e8740 1 -- 172.21.15.45:0/1666509643 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=235,monmap=4+}) v3 -- 0x55965d7163f0 con 0x55965da9e0f0 2024-03-28T17:59:37.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.164+0000 7fbae3400640 1 -- 172.21.15.45:0/1666509643 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbae8004a70 con 0x55965da9e0f0 2024-03-28T17:59:37.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.164+0000 7fbae3400640 1 -- 172.21.15.45:0/1666509643 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 235) v1 ==== 5047+0+0 (secure 0 0 0) 0x7fbaf00afb90 con 0x55965da9e0f0 2024-03-28T17:59:37.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.164+0000 7fbb073e8740 10 client.26387 fetch_fsmap finished waiting for FSMap version 235 2024-03-28T17:59:37.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.164+0000 7fbb073e8740 10 client.26387 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:59:37.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.164+0000 7fbb073e8740 20 client.26387 populate_metadata read hostname 'smithi045' 2024-03-28T17:59:37.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.164+0000 7fbb073e8740 1 --2- 172.21.15.45:0/1666509643 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55965dab75d0 0x55965dab9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:37.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.164+0000 7fbb073e8740 4 client.26387 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T17:59:37.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.164+0000 7fbb073e8740 1 -- 172.21.15.45:0/1666509643 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55965da0f790 con 0x55965dab75d0 2024-03-28T17:59:37.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.164+0000 7fbb01200640 1 --2- 172.21.15.45:0/1666509643 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55965dab75d0 0x55965dab9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:37.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.165+0000 7fbb01200640 1 --2- 172.21.15.45:0/1666509643 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55965dab75d0 0x55965dab9a90 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75a0b9733f5d76ca server_cookie=157e51375a13383e in_seq=0 out_seq=0 2024-03-28T17:59:37.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.165+0000 7fbae3400640 10 client.26387 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:59:37.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.166+0000 7fbae3400640 1 -- 172.21.15.45:0/1666509643 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55965da0f790 con 0x55965dab75d0 2024-03-28T17:59:37.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.166+0000 7fbae3400640 10 client.26387 handle_command_reply: tid=0 2024-03-28T17:59:37.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.166+0000 7fbb073e8740 1 client.26387 shutdown 2024-03-28T17:59:37.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.166+0000 7fbb073e8740 1 -- 172.21.15.45:0/1666509643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965da9e0f0 msgr2=0x55965daa6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:37.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.166+0000 7fbb073e8740 1 --2- 172.21.15.45:0/1666509643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965da9e0f0 0x55965daa6b00 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fbaf00cc900 tx=0x7fbaf000cef0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.166+0000 7fbb073e8740 1 -- 172.21.15.45:0/1666509643 shutdown_connections 2024-03-28T17:59:37.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.166+0000 7fbb073e8740 1 --2- 172.21.15.45:0/1666509643 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55965dab75d0 0x55965dab9a90 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.166+0000 7fbb073e8740 1 --2- 172.21.15.45:0/1666509643 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55965daa7460 0x55965daa43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.166+0000 7fbb073e8740 1 --2- 172.21.15.45:0/1666509643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9edd0 0x55965daa3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.166+0000 7fbb073e8740 1 --2- 172.21.15.45:0/1666509643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965da9e0f0 0x55965daa6b00 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.166+0000 7fbae3400640 0 client.26387 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:59:37.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.166+0000 7fbb073e8740 1 -- 172.21.15.45:0/1666509643 >> 172.21.15.45:0/1666509643 conn(0x55965da65200 msgr2=0x55965da9a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:37.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.166+0000 7fbb073e8740 1 -- 172.21.15.45:0/1666509643 shutdown_connections 2024-03-28T17:59:37.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.167+0000 7fbb073e8740 1 -- 172.21.15.45:0/1666509643 wait complete. 2024-03-28T17:59:37.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.167+0000 7fbb073e8740 20 client.26387 trim_cache size 0 max 16384 2024-03-28T17:59:37.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.184+0000 7fbb073e8740 1 Processor -- start 2024-03-28T17:59:37.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.184+0000 7fbb073e8740 1 -- start start 2024-03-28T17:59:37.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.184+0000 7fbb073e8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9e0f0 0x55965daa65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:37.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.184+0000 7fbb073e8740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965da9edd0 0x55965daa6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:37.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.184+0000 7fbb073e8740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55965daa7460 0x55965daa8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:37.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.185+0000 7fbb073e8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc1b1d50 con 0x55965da9e0f0 2024-03-28T17:59:37.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.185+0000 7fbb073e8740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc1b03d0 con 0x55965da9edd0 2024-03-28T17:59:37.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.185+0000 7fbb073e8740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc1b16f0 con 0x55965daa7460 2024-03-28T17:59:37.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.185+0000 7fbb01200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965da9edd0 0x55965daa6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:37.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.185+0000 7fbb01200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965da9edd0 0x55965daa6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:38126/0 (socket says 172.21.15.45:38126) 2024-03-28T17:59:37.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.185+0000 7fbb01200640 1 -- 172.21.15.45:0/3163949578 learned_addr learned my addr 172.21.15.45:0/3163949578 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:59:37.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.185+0000 7fbb01c00640 1 --2- 172.21.15.45:0/3163949578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9e0f0 0x55965daa65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:37.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.185+0000 7fbb02600640 1 --2- 172.21.15.45:0/3163949578 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55965daa7460 0x55965daa8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:37.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.185+0000 7fbb01200640 1 -- 172.21.15.45:0/3163949578 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55965daa7460 msgr2=0x55965daa8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:59:37.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.185+0000 7fbb01200640 1 --2- 172.21.15.45:0/3163949578 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55965daa7460 0x55965daa8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.185+0000 7fbb01200640 1 -- 172.21.15.45:0/3163949578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9e0f0 msgr2=0x55965daa65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:59:37.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.185+0000 7fbb01200640 1 --2- 172.21.15.45:0/3163949578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9e0f0 0x55965daa65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.185+0000 7fbb01200640 1 -- 172.21.15.45:0/3163949578 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55965d6d60f0 con 0x55965da9edd0 2024-03-28T17:59:37.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.185+0000 7fbb02600640 1 --2- 172.21.15.45:0/3163949578 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55965daa7460 0x55965daa8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T17:59:37.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.185+0000 7fbb01200640 1 --2- 172.21.15.45:0/3163949578 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965da9edd0 0x55965daa6b00 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fbae8004e00 tx=0x7fbae802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7665a066ba76d477 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:37.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.186+0000 7fbaee000640 1 -- 172.21.15.45:0/3163949578 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae8042040 con 0x55965da9edd0 2024-03-28T17:59:37.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.186+0000 7fbaee000640 1 -- 172.21.15.45:0/3163949578 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbae8004a70 con 0x55965da9edd0 2024-03-28T17:59:37.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.186+0000 7fbaee000640 1 -- 172.21.15.45:0/3163949578 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae80054e0 con 0x55965da9edd0 2024-03-28T17:59:37.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.186+0000 7fbb073e8740 1 -- 172.21.15.45:0/3163949578 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965da9edd0 msgr2=0x55965daa6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:37.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.186+0000 7fbb073e8740 1 --2- 172.21.15.45:0/3163949578 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965da9edd0 0x55965daa6b00 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fbae8004e00 tx=0x7fbae802bc30 comp rx=0 tx=0).stop 2024-03-28T17:59:37.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.186+0000 7fbb073e8740 1 -- 172.21.15.45:0/3163949578 shutdown_connections 2024-03-28T17:59:37.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.186+0000 7fbb073e8740 1 --2- 172.21.15.45:0/3163949578 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55965daa7460 0x55965daa8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.186+0000 7fbb073e8740 1 --2- 172.21.15.45:0/3163949578 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965da9edd0 0x55965daa6b00 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.186+0000 7fbb073e8740 1 --2- 172.21.15.45:0/3163949578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9e0f0 0x55965daa65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.186+0000 7fbb073e8740 1 -- 172.21.15.45:0/3163949578 >> 172.21.15.45:0/3163949578 conn(0x55965da65200 msgr2=0x55965da98670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:37.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.187+0000 7fbb073e8740 1 -- 172.21.15.45:0/3163949578 shutdown_connections 2024-03-28T17:59:37.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.187+0000 7fbb073e8740 1 -- 172.21.15.45:0/3163949578 wait complete. 2024-03-28T17:59:37.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.187+0000 7fbb073e8740 1 -- start start 2024-03-28T17:59:37.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.187+0000 7fbb073e8740 1 Processor -- start 2024-03-28T17:59:37.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.188+0000 7fbb073e8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9e0f0 0x55965daa1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:37.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.188+0000 7fbb073e8740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965da9edd0 0x55965daa0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:37.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.188+0000 7fbb073e8740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55965daa7460 0x55965daa0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:37.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.188+0000 7fbb073e8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc1b1090 con 0x55965da9e0f0 2024-03-28T17:59:37.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.188+0000 7fbb073e8740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc1b0a30 con 0x55965da9edd0 2024-03-28T17:59:37.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.188+0000 7fbb073e8740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc1aed30 con 0x55965daa7460 2024-03-28T17:59:37.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.188+0000 7fbb073e8740 10 client.0 _ll_register_callbacks cb 0x7fbaf00d3490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T17:59:37.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.188+0000 7fbb01c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9e0f0 0x55965daa1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:37.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.188+0000 7fbb01200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965da9edd0 0x55965daa0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:37.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.188+0000 7fbb01c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9e0f0 0x55965daa1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35070/0 (socket says 172.21.15.45:35070) 2024-03-28T17:59:37.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.188+0000 7fbb01c00640 1 -- 172.21.15.45:0/4116563260 learned_addr learned my addr 172.21.15.45:0/4116563260 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T17:59:37.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.188+0000 7fbb02600640 1 --2- 172.21.15.45:0/4116563260 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55965daa7460 0x55965daa0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:37.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.188+0000 7fbb01200640 1 -- 172.21.15.45:0/4116563260 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55965daa7460 msgr2=0x55965daa0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:59:37.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.188+0000 7fbb01200640 1 --2- 172.21.15.45:0/4116563260 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55965daa7460 0x55965daa0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.188+0000 7fbb01200640 1 -- 172.21.15.45:0/4116563260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9e0f0 msgr2=0x55965daa1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T17:59:37.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.188+0000 7fbb01200640 1 --2- 172.21.15.45:0/4116563260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9e0f0 0x55965daa1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.188+0000 7fbb01200640 1 -- 172.21.15.45:0/4116563260 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55965daa0f60 con 0x55965da9edd0 2024-03-28T17:59:37.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.188+0000 7fbb02600640 1 --2- 172.21.15.45:0/4116563260 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55965daa7460 0x55965daa0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T17:59:37.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.188+0000 7fbb01c00640 1 --2- 172.21.15.45:0/4116563260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9e0f0 0x55965daa1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T17:59:37.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.189+0000 7fbb01200640 1 --2- 172.21.15.45:0/4116563260 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965da9edd0 0x55965daa0220 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x55965daa9f30 tx=0x7fbae8004f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efd31706d7d10529 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T17:59:37.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.189+0000 7fbae3400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T17:59:37.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.189+0000 7fbae3400640 1 -- 172.21.15.45:0/4116563260 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae8042040 con 0x55965da9edd0 2024-03-28T17:59:37.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.189+0000 7fbb073e8740 1 -- 172.21.15.45:0/4116563260 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbae8004a70 con 0x55965da9edd0 2024-03-28T17:59:37.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.189+0000 7fbae3400640 1 -- 172.21.15.45:0/4116563260 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(128..128 src has 1..128) v4 ==== 11394+0+0 (secure 0 0 0) 0x7fbae8005260 con 0x55965da9edd0 2024-03-28T17:59:37.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.189+0000 7fbae3400640 1 -- 172.21.15.45:0/4116563260 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=235) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbae8005460 con 0x55965da9edd0 2024-03-28T17:59:37.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.189+0000 7fbb073e8740 10 client.26393 fetch_fsmap learned FSMap version 235 2024-03-28T17:59:37.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.189+0000 7fbb073e8740 1 -- 172.21.15.45:0/4116563260 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=235,monmap=4+}) v3 -- 0x55965db12e70 con 0x55965da9edd0 2024-03-28T17:59:37.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.189+0000 7fbae3400640 1 -- 172.21.15.45:0/4116563260 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbae8004a70 con 0x55965da9edd0 2024-03-28T17:59:37.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.190+0000 7fbae3400640 1 -- 172.21.15.45:0/4116563260 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 235) v1 ==== 5047+0+0 (secure 0 0 0) 0x7fbae805b860 con 0x55965da9edd0 2024-03-28T17:59:37.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.190+0000 7fbb073e8740 10 client.26393 fetch_fsmap finished waiting for FSMap version 235 2024-03-28T17:59:37.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.190+0000 7fbb073e8740 10 client.26393 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T17:59:37.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.190+0000 7fbb073e8740 20 client.26393 populate_metadata read hostname 'smithi045' 2024-03-28T17:59:37.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.190+0000 7fbb073e8740 1 --2- 172.21.15.45:0/4116563260 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55965db13460 0x55965db15880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T17:59:37.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.190+0000 7fbb073e8740 4 client.26393 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T17:59:37.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.190+0000 7fbb073e8740 1 -- 172.21.15.45:0/4116563260 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55965da09ae0 con 0x55965db13460 2024-03-28T17:59:37.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.190+0000 7fbb01c00640 1 --2- 172.21.15.45:0/4116563260 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55965db13460 0x55965db15880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T17:59:37.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.190+0000 7fbb01c00640 1 --2- 172.21.15.45:0/4116563260 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55965db13460 0x55965db15880 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e2b4cf280e50841 server_cookie=35ea0cfeda4ef950 in_seq=0 out_seq=0 2024-03-28T17:59:37.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.190+0000 7fbae3400640 10 client.26393 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T17:59:37.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.191+0000 7fbae3400640 1 -- 172.21.15.45:0/4116563260 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55965da09ae0 con 0x55965db13460 2024-03-28T17:59:37.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.191+0000 7fbae3400640 10 client.26393 handle_command_reply: tid=0 2024-03-28T17:59:37.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.191+0000 7fbb073e8740 1 client.26393 shutdown 2024-03-28T17:59:37.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.191+0000 7fbb073e8740 1 -- 172.21.15.45:0/4116563260 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965da9edd0 msgr2=0x55965daa0220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:37.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.191+0000 7fbb073e8740 1 --2- 172.21.15.45:0/4116563260 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965da9edd0 0x55965daa0220 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x55965daa9f30 tx=0x7fbae8004f80 comp rx=0 tx=0).stop 2024-03-28T17:59:37.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.191+0000 7fbb073e8740 1 -- 172.21.15.45:0/4116563260 shutdown_connections 2024-03-28T17:59:37.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.191+0000 7fbb073e8740 1 --2- 172.21.15.45:0/4116563260 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55965db13460 0x55965db15880 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.191+0000 7fbb073e8740 1 --2- 172.21.15.45:0/4116563260 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55965daa7460 0x55965daa0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.191+0000 7fbb073e8740 1 --2- 172.21.15.45:0/4116563260 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55965da9edd0 0x55965daa0220 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.191+0000 7fbb073e8740 1 --2- 172.21.15.45:0/4116563260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55965da9e0f0 0x55965daa1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.191+0000 7fbb073e8740 1 -- 172.21.15.45:0/4116563260 >> 172.21.15.45:0/4116563260 conn(0x55965da65200 msgr2=0x55965daa2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:37.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.191+0000 7fbae3400640 0 client.26393 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T17:59:37.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.191+0000 7fbb073e8740 1 -- 172.21.15.45:0/4116563260 shutdown_connections 2024-03-28T17:59:37.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.191+0000 7fbb073e8740 1 -- 172.21.15.45:0/4116563260 wait complete. 2024-03-28T17:59:37.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.191+0000 7fbb073e8740 20 client.26393 trim_cache size 0 max 16384 2024-03-28T17:59:37.196 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T17:59:37.196 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T17:59:37.196 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T17:59:37.196 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T17:59:37.196 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T17:59:37.196 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T17:59:37.196 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T17:59:37.196 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:59:37.196 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T17:59:37.196 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T17:59:37.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.192+0000 7fbb03000640 1 -- 172.21.15.45:0/3073062916 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbae40a29f0 msgr2=0x7fbae40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:37.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.192+0000 7fbb03000640 1 --2- 172.21.15.45:0/3073062916 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbae40a29f0 0x7fbae40a4eb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4005fd0 tx=0x7fbaf4009500 comp rx=0 tx=0).stop 2024-03-28T17:59:37.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.192+0000 7fbb03000640 1 -- 172.21.15.45:0/3073062916 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbafc103720 msgr2=0x7fbafc19a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T17:59:37.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.192+0000 7fbb03000640 1 --2- 172.21.15.45:0/3073062916 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbafc103720 0x7fbafc19a3b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbaf000e9f0 tx=0x7fbaf000edf0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.192+0000 7fbb03000640 1 -- 172.21.15.45:0/3073062916 shutdown_connections 2024-03-28T17:59:37.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.192+0000 7fbb03000640 1 --2- 172.21.15.45:0/3073062916 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbae40a29f0 0x7fbae40a4eb0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.192+0000 7fbb03000640 1 --2- 172.21.15.45:0/3073062916 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbafc109730 0x7fbafc19ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.192+0000 7fbb03000640 1 --2- 172.21.15.45:0/3073062916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbafc1040e0 0x7fbafc19a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.192+0000 7fbb03000640 1 --2- 172.21.15.45:0/3073062916 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbafc103720 0x7fbafc19a3b0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T17:59:37.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.192+0000 7fbb03000640 1 -- 172.21.15.45:0/3073062916 >> 172.21.15.45:0/3073062916 conn(0x7fbafc0ff1b0 msgr2=0x7fbafc0fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T17:59:37.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.192+0000 7fbb03000640 1 -- 172.21.15.45:0/3073062916 shutdown_connections 2024-03-28T17:59:37.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T17:59:37.192+0000 7fbb03000640 1 -- 172.21.15.45:0/3073062916 wait complete. 2024-03-28T17:59:37.209 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T17:59:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:37 smithi071 ceph-mon[36597]: pgmap v361: 97 pgs: 97 active+clean; 27 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 233 B/s wr, 7 op/s 2024-03-28T17:59:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:37 smithi071 ceph-mon[36597]: Replacing daemon mds.l as rank 2 with standby daemon mds.a 2024-03-28T17:59:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:37 smithi071 ceph-mon[36597]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-03-28T17:59:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:37 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:59:37.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:37 smithi071 ceph-mon[36597]: osdmap e128: 12 total, 12 up, 12 in 2024-03-28T17:59:37.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:37 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:replay} 2 up:standby-replay 2 up:standby 2024-03-28T17:59:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:37 smithi151 ceph-mon[36481]: pgmap v361: 97 pgs: 97 active+clean; 27 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 233 B/s wr, 7 op/s 2024-03-28T17:59:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:37 smithi151 ceph-mon[36481]: Replacing daemon mds.l as rank 2 with standby daemon mds.a 2024-03-28T17:59:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:37 smithi151 ceph-mon[36481]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-03-28T17:59:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:37 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:59:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:37 smithi151 ceph-mon[36481]: osdmap e128: 12 total, 12 up, 12 in 2024-03-28T17:59:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:37 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:replay} 2 up:standby-replay 2 up:standby 2024-03-28T17:59:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:37 smithi045 ceph-mon[31773]: pgmap v361: 97 pgs: 97 active+clean; 27 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 233 B/s wr, 7 op/s 2024-03-28T17:59:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:37 smithi045 ceph-mon[31773]: Replacing daemon mds.l as rank 2 with standby daemon mds.a 2024-03-28T17:59:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:37 smithi045 ceph-mon[31773]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-03-28T17:59:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:37 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:59:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:37 smithi045 ceph-mon[31773]: osdmap e128: 12 total, 12 up, 12 in 2024-03-28T17:59:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:37 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:replay} 2 up:standby-replay 2 up:standby 2024-03-28T17:59:37.913 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 430 sec latency 545810.566 ms 2024-03-28T17:59:38.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:38 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/3511256023,v1:172.21.15.45:6837/3511256023] up:resolve 2024-03-28T17:59:38.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:38 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:resolve} 2 up:standby-replay 2 up:standby 2024-03-28T17:59:38.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:38 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/3511256023,v1:172.21.15.45:6837/3511256023] up:resolve 2024-03-28T17:59:38.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:38 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:resolve} 2 up:standby-replay 2 up:standby 2024-03-28T17:59:38.913 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 431 sec latency 546810.712 ms 2024-03-28T17:59:38.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:38 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/3511256023,v1:172.21.15.45:6837/3511256023] up:resolve 2024-03-28T17:59:38.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:38 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:resolve} 2 up:standby-replay 2 up:standby 2024-03-28T17:59:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:39 smithi071 ceph-mon[36597]: pgmap v363: 97 pgs: 97 active+clean; 27 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-28T17:59:39.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:39 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:59:39.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:39 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/3511256023,v1:172.21.15.45:6837/3511256023] up:reconnect 2024-03-28T17:59:39.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:39 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:reconnect} 2 up:standby-replay 2 up:standby 2024-03-28T17:59:39.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:39 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00799988 2024-03-28T17:59:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:39 smithi151 ceph-mon[36481]: pgmap v363: 97 pgs: 97 active+clean; 27 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-28T17:59:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:39 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:59:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:39 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/3511256023,v1:172.21.15.45:6837/3511256023] up:reconnect 2024-03-28T17:59:39.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:39 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:reconnect} 2 up:standby-replay 2 up:standby 2024-03-28T17:59:39.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:39 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00799988 2024-03-28T17:59:39.914 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 432 sec latency 547810.856 ms 2024-03-28T17:59:39.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:39 smithi045 ceph-mon[31773]: pgmap v363: 97 pgs: 97 active+clean; 27 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-28T17:59:39.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:39 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:59:39.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:39 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/3511256023,v1:172.21.15.45:6837/3511256023] up:reconnect 2024-03-28T17:59:39.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:39 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:reconnect} 2 up:standby-replay 2 up:standby 2024-03-28T17:59:39.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:39 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00799988 2024-03-28T17:59:40.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:40 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/3511256023,v1:172.21.15.45:6837/3511256023] up:rejoin 2024-03-28T17:59:40.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:40 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:rejoin} 2 up:standby-replay 2 up:standby 2024-03-28T17:59:40.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:40 smithi071 ceph-mon[36597]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-03-28T17:59:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:40 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/3511256023,v1:172.21.15.45:6837/3511256023] up:rejoin 2024-03-28T17:59:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:40 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:rejoin} 2 up:standby-replay 2 up:standby 2024-03-28T17:59:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:40 smithi151 ceph-mon[36481]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-03-28T17:59:40.914 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 433 sec latency 548810.986 ms 2024-03-28T17:59:40.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/3511256023,v1:172.21.15.45:6837/3511256023] up:rejoin 2024-03-28T17:59:40.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=a=up:rejoin} 2 up:standby-replay 2 up:standby 2024-03-28T17:59:40.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-mon[31773]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-03-28T17:59:40.917 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa066200700 time 2024-03-28T17:59:40.488878+0000 2024-03-28T17:59:40.917 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:59:40.917 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:40.917 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa071b654a1] 2024-03-28T17:59:40.917 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa071b65667] 2024-03-28T17:59:40.917 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d675e1223] 2024-03-28T17:59:40.917 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d6745b5d6] 2024-03-28T17:59:40.918 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d6745bda1] 2024-03-28T17:59:40.918 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d6745e2d5] 2024-03-28T17:59:40.918 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d674ae233] 2024-03-28T17:59:40.918 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d674b0197] 2024-03-28T17:59:40.918 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d673af11b] 2024-03-28T17:59:40.918 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d673af6bc] 2024-03-28T17:59:40.918 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d673991ff] 2024-03-28T17:59:40.918 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa071df7488] 2024-03-28T17:59:40.918 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 13: (DispatchQueue::entry()+0x51f) [0x7fa071df460f] 2024-03-28T17:59:40.918 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa071ec7e81] 2024-03-28T17:59:40.918 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa0708e21ca] 2024-03-28T17:59:40.918 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 16: clone() 2024-03-28T17:59:40.919 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 2024-03-28T17:59:40.490+0000 7fa066200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa066200700 time 2024-03-28T17:59:40.488878+0000 2024-03-28T17:59:40.919 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:59:40.919 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 2024-03-28T17:59:40.919 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:40.919 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa071b654a1] 2024-03-28T17:59:40.919 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa071b65667] 2024-03-28T17:59:40.919 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d675e1223] 2024-03-28T17:59:40.919 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d6745b5d6] 2024-03-28T17:59:40.919 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d6745bda1] 2024-03-28T17:59:40.919 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d6745e2d5] 2024-03-28T17:59:40.919 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d674ae233] 2024-03-28T17:59:40.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d674b0197] 2024-03-28T17:59:40.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d673af11b] 2024-03-28T17:59:40.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d673af6bc] 2024-03-28T17:59:40.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d673991ff] 2024-03-28T17:59:40.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa071df7488] 2024-03-28T17:59:40.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 13: (DispatchQueue::entry()+0x51f) [0x7fa071df460f] 2024-03-28T17:59:40.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa071ec7e81] 2024-03-28T17:59:40.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa0708e21ca] 2024-03-28T17:59:40.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 16: clone() 2024-03-28T17:59:40.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 2024-03-28T17:59:40.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: *** Caught signal (Aborted) ** 2024-03-28T17:59:40.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: in thread 7fa066200700 thread_name:ms_dispatch 2024-03-28T17:59:40.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:40.920 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa0708ecd20] 2024-03-28T17:59:40.921 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 2: gsignal() 2024-03-28T17:59:40.921 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 3: abort() 2024-03-28T17:59:40.921 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa071b654fb] 2024-03-28T17:59:40.921 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa071b65667] 2024-03-28T17:59:40.921 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d675e1223] 2024-03-28T17:59:40.921 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d6745b5d6] 2024-03-28T17:59:40.921 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d6745bda1] 2024-03-28T17:59:40.921 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d6745e2d5] 2024-03-28T17:59:40.921 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d674ae233] 2024-03-28T17:59:40.921 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d674b0197] 2024-03-28T17:59:40.922 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d673af11b] 2024-03-28T17:59:40.922 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d673af6bc] 2024-03-28T17:59:40.922 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d673991ff] 2024-03-28T17:59:40.922 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa071df7488] 2024-03-28T17:59:40.922 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 16: (DispatchQueue::entry()+0x51f) [0x7fa071df460f] 2024-03-28T17:59:40.922 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa071ec7e81] 2024-03-28T17:59:40.922 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa0708e21ca] 2024-03-28T17:59:40.922 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 19: clone() 2024-03-28T17:59:40.922 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 2024-03-28T17:59:40.492+0000 7fa066200700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:59:40.922 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: in thread 7fa066200700 thread_name:ms_dispatch 2024-03-28T17:59:40.922 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 2024-03-28T17:59:40.922 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:40.922 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa0708ecd20] 2024-03-28T17:59:40.922 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 2: gsignal() 2024-03-28T17:59:40.923 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 3: abort() 2024-03-28T17:59:40.923 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa071b654fb] 2024-03-28T17:59:40.923 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa071b65667] 2024-03-28T17:59:40.923 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d675e1223] 2024-03-28T17:59:40.923 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d6745b5d6] 2024-03-28T17:59:40.923 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d6745bda1] 2024-03-28T17:59:40.923 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d6745e2d5] 2024-03-28T17:59:40.923 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d674ae233] 2024-03-28T17:59:40.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d674b0197] 2024-03-28T17:59:40.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d673af11b] 2024-03-28T17:59:40.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d673af6bc] 2024-03-28T17:59:40.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d673991ff] 2024-03-28T17:59:40.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa071df7488] 2024-03-28T17:59:40.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 16: (DispatchQueue::entry()+0x51f) [0x7fa071df460f] 2024-03-28T17:59:40.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa071ec7e81] 2024-03-28T17:59:40.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa0708e21ca] 2024-03-28T17:59:40.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 19: clone() 2024-03-28T17:59:40.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:59:40.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 2024-03-28T17:59:40.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: -1> 2024-03-28T17:59:40.490+0000 7fa066200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa066200700 time 2024-03-28T17:59:40.488878+0000 2024-03-28T17:59:40.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:59:40.924 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 2024-03-28T17:59:40.925 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:40.925 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa071b654a1] 2024-03-28T17:59:40.925 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa071b65667] 2024-03-28T17:59:40.925 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d675e1223] 2024-03-28T17:59:40.925 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d6745b5d6] 2024-03-28T17:59:40.925 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d6745bda1] 2024-03-28T17:59:40.925 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d6745e2d5] 2024-03-28T17:59:40.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d674ae233] 2024-03-28T17:59:40.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d674b0197] 2024-03-28T17:59:40.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d673af11b] 2024-03-28T17:59:40.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d673af6bc] 2024-03-28T17:59:40.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d673991ff] 2024-03-28T17:59:40.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa071df7488] 2024-03-28T17:59:40.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 13: (DispatchQueue::entry()+0x51f) [0x7fa071df460f] 2024-03-28T17:59:40.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa071ec7e81] 2024-03-28T17:59:40.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa0708e21ca] 2024-03-28T17:59:40.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 16: clone() 2024-03-28T17:59:40.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 2024-03-28T17:59:40.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 0> 2024-03-28T17:59:40.492+0000 7fa066200700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:59:40.926 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: in thread 7fa066200700 thread_name:ms_dispatch 2024-03-28T17:59:40.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 2024-03-28T17:59:40.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:40.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa0708ecd20] 2024-03-28T17:59:40.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 2: gsignal() 2024-03-28T17:59:40.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 3: abort() 2024-03-28T17:59:40.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa071b654fb] 2024-03-28T17:59:40.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa071b65667] 2024-03-28T17:59:40.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d675e1223] 2024-03-28T17:59:40.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d6745b5d6] 2024-03-28T17:59:40.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d6745bda1] 2024-03-28T17:59:40.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d6745e2d5] 2024-03-28T17:59:40.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d674ae233] 2024-03-28T17:59:40.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d674b0197] 2024-03-28T17:59:40.927 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d673af11b] 2024-03-28T17:59:40.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d673af6bc] 2024-03-28T17:59:40.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d673991ff] 2024-03-28T17:59:40.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa071df7488] 2024-03-28T17:59:40.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 16: (DispatchQueue::entry()+0x51f) [0x7fa071df460f] 2024-03-28T17:59:40.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa071ec7e81] 2024-03-28T17:59:40.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa0708e21ca] 2024-03-28T17:59:40.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 19: clone() 2024-03-28T17:59:40.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:59:40.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 2024-03-28T17:59:40.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: -1> 2024-03-28T17:59:40.490+0000 7fa066200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa066200700 time 2024-03-28T17:59:40.488878+0000 2024-03-28T17:59:40.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T17:59:40.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 2024-03-28T17:59:40.928 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:40.929 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa071b654a1] 2024-03-28T17:59:40.929 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa071b65667] 2024-03-28T17:59:40.929 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d675e1223] 2024-03-28T17:59:40.929 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d6745b5d6] 2024-03-28T17:59:40.929 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d6745bda1] 2024-03-28T17:59:40.929 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d6745e2d5] 2024-03-28T17:59:40.929 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d674ae233] 2024-03-28T17:59:40.929 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d674b0197] 2024-03-28T17:59:40.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d673af11b] 2024-03-28T17:59:40.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d673af6bc] 2024-03-28T17:59:40.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d673991ff] 2024-03-28T17:59:40.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa071df7488] 2024-03-28T17:59:40.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 13: (DispatchQueue::entry()+0x51f) [0x7fa071df460f] 2024-03-28T17:59:40.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa071ec7e81] 2024-03-28T17:59:40.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa0708e21ca] 2024-03-28T17:59:40.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 16: clone() 2024-03-28T17:59:40.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 2024-03-28T17:59:40.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 0> 2024-03-28T17:59:40.492+0000 7fa066200700 -1 *** Caught signal (Aborted) ** 2024-03-28T17:59:40.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: in thread 7fa066200700 thread_name:ms_dispatch 2024-03-28T17:59:40.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 2024-03-28T17:59:40.930 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T17:59:40.931 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa0708ecd20] 2024-03-28T17:59:40.931 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 2: gsignal() 2024-03-28T17:59:40.931 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 3: abort() 2024-03-28T17:59:40.931 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa071b654fb] 2024-03-28T17:59:40.931 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa071b65667] 2024-03-28T17:59:40.931 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d675e1223] 2024-03-28T17:59:40.931 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d6745b5d6] 2024-03-28T17:59:40.931 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d6745bda1] 2024-03-28T17:59:40.931 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d6745e2d5] 2024-03-28T17:59:40.931 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d674ae233] 2024-03-28T17:59:40.931 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d674b0197] 2024-03-28T17:59:40.931 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d673af11b] 2024-03-28T17:59:40.931 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d673af6bc] 2024-03-28T17:59:40.931 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d673991ff] 2024-03-28T17:59:40.931 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa071df7488] 2024-03-28T17:59:40.932 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 16: (DispatchQueue::entry()+0x51f) [0x7fa071df460f] 2024-03-28T17:59:40.932 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa071ec7e81] 2024-03-28T17:59:40.932 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa0708e21ca] 2024-03-28T17:59:40.932 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 19: clone() 2024-03-28T17:59:40.932 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T17:59:40.932 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a[92973]: 2024-03-28T17:59:40.932 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:40 smithi045 podman[93648]: 2024-03-28 17:59:40.603348588 +0000 UTC m=+0.035754284 container died 2aadaf9109441b1e07f43d04479a4d5d224f6cda7d1eb6ee26035497328a0acb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0) 2024-03-28T17:59:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:41 smithi071 ceph-mon[36597]: pgmap v364: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 113 B/s wr, 3 op/s 2024-03-28T17:59:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:41 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:59:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:41 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/3511256023,v1:172.21.15.45:6837/3511256023] up:active 2024-03-28T17:59:41.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:41 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T17:59:41.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:41 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 3 up:standby-replay 1 up:standby 2024-03-28T17:59:41.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:41 smithi071 ceph-mon[36597]: osdmap e129: 12 total, 12 up, 12 in 2024-03-28T17:59:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:41 smithi151 ceph-mon[36481]: pgmap v364: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 113 B/s wr, 3 op/s 2024-03-28T17:59:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:41 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:59:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:41 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/3511256023,v1:172.21.15.45:6837/3511256023] up:active 2024-03-28T17:59:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:41 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T17:59:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:41 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 3 up:standby-replay 1 up:standby 2024-03-28T17:59:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:41 smithi151 ceph-mon[36481]: osdmap e129: 12 total, 12 up, 12 in 2024-03-28T17:59:41.914 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 434 sec latency 549811.108 ms 2024-03-28T17:59:41.916 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:41 smithi045 podman[93648]: 2024-03-28 17:59:41.483303916 +0000 UTC m=+0.915709651 container remove 2aadaf9109441b1e07f43d04479a4d5d224f6cda7d1eb6ee26035497328a0acb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-a, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-28T17:59:41.916 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:41 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-03-28T17:59:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:41 smithi045 ceph-mon[31773]: pgmap v364: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 113 B/s wr, 3 op/s 2024-03-28T17:59:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:41 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T17:59:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:41 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/3511256023,v1:172.21.15.45:6837/3511256023] up:active 2024-03-28T17:59:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:41 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 2 up:standby-replay 2 up:standby 2024-03-28T17:59:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:41 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=a=up:active} 3 up:standby-replay 1 up:standby 2024-03-28T17:59:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:41 smithi045 ceph-mon[31773]: osdmap e129: 12 total, 12 up, 12 in 2024-03-28T17:59:42.415 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:42 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Failed with result 'exit-code'. 2024-03-28T17:59:42.416 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:42 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Consumed 1.663s CPU time. 2024-03-28T17:59:42.914 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 435 sec latency 550811.270 ms 2024-03-28T17:59:43.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:43 smithi071 ceph-mon[36597]: pgmap v366: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T17:59:43.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:43 smithi151 ceph-mon[36481]: pgmap v366: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T17:59:43.914 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 436 sec latency 551811.436 ms 2024-03-28T17:59:43.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:43 smithi045 ceph-mon[31773]: pgmap v366: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T17:59:44.914 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 437 sec latency 552811.607 ms 2024-03-28T17:59:45.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:45 smithi071 ceph-mon[36597]: pgmap v367: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 383 B/s wr, 4 op/s 2024-03-28T17:59:45.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:45 smithi151 ceph-mon[36481]: pgmap v367: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 383 B/s wr, 4 op/s 2024-03-28T17:59:45.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:45 smithi045 ceph-mon[31773]: pgmap v367: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 383 B/s wr, 4 op/s 2024-03-28T17:59:45.915 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 438 sec latency 553811.737 ms 2024-03-28T17:59:46.915 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 439 sec latency 554811.911 ms 2024-03-28T17:59:47.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:47 smithi071 ceph-mon[36597]: pgmap v368: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 322 B/s wr, 11 op/s 2024-03-28T17:59:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:47 smithi151 ceph-mon[36481]: pgmap v368: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 322 B/s wr, 11 op/s 2024-03-28T17:59:47.915 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 440 sec latency 555812.039 ms 2024-03-28T17:59:47.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:47 smithi045 ceph-mon[31773]: pgmap v368: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 322 B/s wr, 11 op/s 2024-03-28T17:59:48.915 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 441 sec latency 556812.183 ms 2024-03-28T17:59:49.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:49 smithi071 ceph-mon[36597]: pgmap v369: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:59:49.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:49 smithi151 ceph-mon[36481]: pgmap v369: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:59:49.916 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 442 sec latency 557812.464 ms 2024-03-28T17:59:49.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:49 smithi045 ceph-mon[31773]: pgmap v369: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:59:50.916 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 443 sec latency 558812.642 ms 2024-03-28T17:59:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:51 smithi071 ceph-mon[36597]: pgmap v370: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:59:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:51 smithi151 ceph-mon[36481]: pgmap v370: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:59:51.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:51 smithi045 ceph-mon[31773]: pgmap v370: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T17:59:51.916 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 444 sec latency 559812.739 ms 2024-03-28T17:59:52.666 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:52 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Scheduled restart job, restart counter is at 5. 2024-03-28T17:59:52.666 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:52 smithi045 systemd[1]: Stopped Ceph mds.a for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:59:52.666 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:52 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Consumed 1.663s CPU time. 2024-03-28T17:59:52.666 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:52 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Start request repeated too quickly. 2024-03-28T17:59:52.666 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:52 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.a.service: Failed with result 'exit-code'. 2024-03-28T17:59:52.666 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 28 17:59:52 smithi045 systemd[1]: Failed to start Ceph mds.a for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T17:59:52.916 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 445 sec latency 560812.857 ms 2024-03-28T17:59:53.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:53 smithi071 ceph-mon[36597]: pgmap v371: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 10 op/s 2024-03-28T17:59:53.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:53 smithi151 ceph-mon[36481]: pgmap v371: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 10 op/s 2024-03-28T17:59:53.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:53 smithi045 ceph-mon[31773]: pgmap v371: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 10 op/s 2024-03-28T17:59:53.916 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 446 sec latency 561812.945 ms 2024-03-28T17:59:54.916 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 447 sec latency 562813.083 ms 2024-03-28T17:59:55.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:55 smithi071 ceph-mon[36597]: pgmap v372: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9 op/s 2024-03-28T17:59:55.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:55 smithi151 ceph-mon[36481]: pgmap v372: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9 op/s 2024-03-28T17:59:55.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:55 smithi045 ceph-mon[31773]: pgmap v372: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9 op/s 2024-03-28T17:59:55.916 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 448 sec latency 563813.192 ms 2024-03-28T17:59:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:56 smithi071 ceph-mon[36597]: Replacing daemon mds.a as rank 2 with standby daemon mds.g 2024-03-28T17:59:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:56 smithi071 ceph-mon[36597]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-03-28T17:59:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:56 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:59:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:56 smithi071 ceph-mon[36597]: osdmap e130: 12 total, 12 up, 12 in 2024-03-28T17:59:56.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:56 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T17:59:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:56 smithi151 ceph-mon[36481]: Replacing daemon mds.a as rank 2 with standby daemon mds.g 2024-03-28T17:59:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:56 smithi151 ceph-mon[36481]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-03-28T17:59:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:56 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:59:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:56 smithi151 ceph-mon[36481]: osdmap e130: 12 total, 12 up, 12 in 2024-03-28T17:59:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:56 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T17:59:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:56 smithi045 ceph-mon[31773]: Replacing daemon mds.a as rank 2 with standby daemon mds.g 2024-03-28T17:59:56.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:56 smithi045 ceph-mon[31773]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-03-28T17:59:56.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:56 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T17:59:56.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:56 smithi045 ceph-mon[31773]: osdmap e130: 12 total, 12 up, 12 in 2024-03-28T17:59:56.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:56 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T17:59:56.916 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 449 sec latency 564813.297 ms 2024-03-28T17:59:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:57 smithi071 ceph-mon[36597]: pgmap v373: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 op/s 2024-03-28T17:59:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:57 smithi151 ceph-mon[36481]: pgmap v373: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 op/s 2024-03-28T17:59:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:57 smithi045 ceph-mon[31773]: pgmap v373: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 op/s 2024-03-28T17:59:57.916 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 450 sec latency 565813.411 ms 2024-03-28T17:59:58.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:58 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6838/2313500150,v1:172.21.15.45:6839/2313500150] up:resolve 2024-03-28T17:59:58.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:58 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T17:59:58.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:58 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6838/2313500150,v1:172.21.15.45:6839/2313500150] up:reconnect 2024-03-28T17:59:58.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:58 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T17:59:58.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:58 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6838/2313500150,v1:172.21.15.45:6839/2313500150] up:resolve 2024-03-28T17:59:58.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:58 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T17:59:58.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:58 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6838/2313500150,v1:172.21.15.45:6839/2313500150] up:reconnect 2024-03-28T17:59:58.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:58 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T17:59:58.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:58 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6838/2313500150,v1:172.21.15.45:6839/2313500150] up:resolve 2024-03-28T17:59:58.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:58 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T17:59:58.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:58 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6838/2313500150,v1:172.21.15.45:6839/2313500150] up:reconnect 2024-03-28T17:59:58.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:58 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T17:59:58.916 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 451 sec latency 566813.562 ms 2024-03-28T17:59:59.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:59 smithi071 ceph-mon[36597]: pgmap v375: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:59:59.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:59 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:59:59.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 17:59:59 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T17:59:59.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:59 smithi151 ceph-mon[36481]: pgmap v375: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:59:59.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:59 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:59:59.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 17:59:59 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T17:59:59.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:59 smithi045 ceph-mon[31773]: pgmap v375: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T17:59:59.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:59 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T17:59:59.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 17:59:59 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T17:59:59.916 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 452 sec latency 567813.700 ms 2024-03-28T18:00:00.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6838/2313500150,v1:172.21.15.45:6839/2313500150] up:rejoin 2024-03-28T18:00:00.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:00.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-mon[31773]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-03-28T18:00:00.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-mon[31773]: Health detail: HEALTH_WARN 5 failed cephadm daemon(s); 1 filesystem is degraded 2024-03-28T18:00:00.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-mon[31773]: [WRN] CEPHADM_FAILED_DAEMON: 5 failed cephadm daemon(s) 2024-03-28T18:00:00.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-mon[31773]: daemon mds.d on smithi045 is in error state 2024-03-28T18:00:00.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-mon[31773]: daemon mds.k on smithi071 is in error state 2024-03-28T18:00:00.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-mon[31773]: daemon mds.l on smithi151 is in unknown state 2024-03-28T18:00:00.577 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-mon[31773]: daemon mds.f on smithi151 is in error state 2024-03-28T18:00:00.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-mon[31773]: daemon mds.i on smithi151 is in error state 2024-03-28T18:00:00.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-mon[31773]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-28T18:00:00.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-mon[31773]: fs cephfs is degraded 2024-03-28T18:00:00.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:00:00.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6838/2313500150,v1:172.21.15.45:6839/2313500150] up:active 2024-03-28T18:00:00.578 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:00.578 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 16: clone() 2024-03-28T18:00:00.578 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 2024-03-28T18:00:00.578 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: *** Caught signal (Aborted) ** 2024-03-28T18:00:00.578 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: in thread 7f40dc600700 thread_name:ms_dispatch 2024-03-28T18:00:00.579 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:00.579 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f40e6bb1d20] 2024-03-28T18:00:00.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:00 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6838/2313500150,v1:172.21.15.45:6839/2313500150] up:rejoin 2024-03-28T18:00:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:00 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:00 smithi071 ceph-mon[36597]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-03-28T18:00:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:00 smithi071 ceph-mon[36597]: Health detail: HEALTH_WARN 5 failed cephadm daemon(s); 1 filesystem is degraded 2024-03-28T18:00:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:00 smithi071 ceph-mon[36597]: [WRN] CEPHADM_FAILED_DAEMON: 5 failed cephadm daemon(s) 2024-03-28T18:00:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:00 smithi071 ceph-mon[36597]: daemon mds.d on smithi045 is in error state 2024-03-28T18:00:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:00 smithi071 ceph-mon[36597]: daemon mds.k on smithi071 is in error state 2024-03-28T18:00:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:00 smithi071 ceph-mon[36597]: daemon mds.l on smithi151 is in unknown state 2024-03-28T18:00:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:00 smithi071 ceph-mon[36597]: daemon mds.f on smithi151 is in error state 2024-03-28T18:00:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:00 smithi071 ceph-mon[36597]: daemon mds.i on smithi151 is in error state 2024-03-28T18:00:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:00 smithi071 ceph-mon[36597]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-28T18:00:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:00 smithi071 ceph-mon[36597]: fs cephfs is degraded 2024-03-28T18:00:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:00 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:00:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:00 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6838/2313500150,v1:172.21.15.45:6839/2313500150] up:active 2024-03-28T18:00:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:00 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:00 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6838/2313500150,v1:172.21.15.45:6839/2313500150] up:rejoin 2024-03-28T18:00:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:00 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:00 smithi151 ceph-mon[36481]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-03-28T18:00:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:00 smithi151 ceph-mon[36481]: Health detail: HEALTH_WARN 5 failed cephadm daemon(s); 1 filesystem is degraded 2024-03-28T18:00:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:00 smithi151 ceph-mon[36481]: [WRN] CEPHADM_FAILED_DAEMON: 5 failed cephadm daemon(s) 2024-03-28T18:00:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:00 smithi151 ceph-mon[36481]: daemon mds.d on smithi045 is in error state 2024-03-28T18:00:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:00 smithi151 ceph-mon[36481]: daemon mds.k on smithi071 is in error state 2024-03-28T18:00:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:00 smithi151 ceph-mon[36481]: daemon mds.l on smithi151 is in unknown state 2024-03-28T18:00:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:00 smithi151 ceph-mon[36481]: daemon mds.f on smithi151 is in error state 2024-03-28T18:00:00.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:00 smithi151 ceph-mon[36481]: daemon mds.i on smithi151 is in error state 2024-03-28T18:00:00.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:00 smithi151 ceph-mon[36481]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-28T18:00:00.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:00 smithi151 ceph-mon[36481]: fs cephfs is degraded 2024-03-28T18:00:00.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:00 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:00:00.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:00 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6838/2313500150,v1:172.21.15.45:6839/2313500150] up:active 2024-03-28T18:00:00.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:00 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:00.917 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 453 sec latency 568813.801 ms 2024-03-28T18:00:00.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 2: gsignal() 2024-03-28T18:00:00.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 3: abort() 2024-03-28T18:00:00.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f40e7e2a4fb] 2024-03-28T18:00:00.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f40e7e2a667] 2024-03-28T18:00:00.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563820ad5223] 2024-03-28T18:00:00.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56382094f5d6] 2024-03-28T18:00:00.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56382094fda1] 2024-03-28T18:00:00.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5638209522d5] 2024-03-28T18:00:00.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5638209a2233] 2024-03-28T18:00:00.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5638209a4197] 2024-03-28T18:00:00.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5638208a311b] 2024-03-28T18:00:00.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5638208a36bc] 2024-03-28T18:00:00.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56382088d1ff] 2024-03-28T18:00:00.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f40e80bc488] 2024-03-28T18:00:00.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 16: (DispatchQueue::entry()+0x51f) [0x7f40e80b960f] 2024-03-28T18:00:00.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40e818ce81] 2024-03-28T18:00:00.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f40e6ba71ca] 2024-03-28T18:00:00.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 19: clone() 2024-03-28T18:00:00.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 2024-03-28T18:00:00.567+0000 7f40dc600700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:00:00.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: in thread 7f40dc600700 thread_name:ms_dispatch 2024-03-28T18:00:00.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 2024-03-28T18:00:00.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:00.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f40e6bb1d20] 2024-03-28T18:00:00.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 2: gsignal() 2024-03-28T18:00:00.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 3: abort() 2024-03-28T18:00:00.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f40e7e2a4fb] 2024-03-28T18:00:00.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f40e7e2a667] 2024-03-28T18:00:00.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563820ad5223] 2024-03-28T18:00:00.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56382094f5d6] 2024-03-28T18:00:00.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56382094fda1] 2024-03-28T18:00:00.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5638209522d5] 2024-03-28T18:00:00.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5638209a2233] 2024-03-28T18:00:00.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5638209a4197] 2024-03-28T18:00:00.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5638208a311b] 2024-03-28T18:00:00.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5638208a36bc] 2024-03-28T18:00:00.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56382088d1ff] 2024-03-28T18:00:00.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f40e80bc488] 2024-03-28T18:00:00.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 16: (DispatchQueue::entry()+0x51f) [0x7f40e80b960f] 2024-03-28T18:00:00.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40e818ce81] 2024-03-28T18:00:00.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f40e6ba71ca] 2024-03-28T18:00:00.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 19: clone() 2024-03-28T18:00:00.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:00:00.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 2024-03-28T18:00:00.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: -1> 2024-03-28T18:00:00.565+0000 7f40dc600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f40dc600700 time 2024-03-28T18:00:00.564200+0000 2024-03-28T18:00:00.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:00:00.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 2024-03-28T18:00:00.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:00.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f40e7e2a4a1] 2024-03-28T18:00:00.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f40e7e2a667] 2024-03-28T18:00:00.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563820ad5223] 2024-03-28T18:00:00.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56382094f5d6] 2024-03-28T18:00:00.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56382094fda1] 2024-03-28T18:00:00.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5638209522d5] 2024-03-28T18:00:00.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5638209a2233] 2024-03-28T18:00:00.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5638209a4197] 2024-03-28T18:00:00.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5638208a311b] 2024-03-28T18:00:00.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5638208a36bc] 2024-03-28T18:00:00.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56382088d1ff] 2024-03-28T18:00:00.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f40e80bc488] 2024-03-28T18:00:00.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 13: (DispatchQueue::entry()+0x51f) [0x7f40e80b960f] 2024-03-28T18:00:00.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40e818ce81] 2024-03-28T18:00:00.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f40e6ba71ca] 2024-03-28T18:00:00.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 16: clone() 2024-03-28T18:00:00.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 2024-03-28T18:00:00.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 0> 2024-03-28T18:00:00.567+0000 7f40dc600700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:00:00.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: in thread 7f40dc600700 thread_name:ms_dispatch 2024-03-28T18:00:00.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 2024-03-28T18:00:00.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:00.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f40e6bb1d20] 2024-03-28T18:00:00.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 2: gsignal() 2024-03-28T18:00:00.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 3: abort() 2024-03-28T18:00:00.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f40e7e2a4fb] 2024-03-28T18:00:00.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f40e7e2a667] 2024-03-28T18:00:00.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563820ad5223] 2024-03-28T18:00:00.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56382094f5d6] 2024-03-28T18:00:00.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56382094fda1] 2024-03-28T18:00:00.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5638209522d5] 2024-03-28T18:00:00.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5638209a2233] 2024-03-28T18:00:00.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5638209a4197] 2024-03-28T18:00:00.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5638208a311b] 2024-03-28T18:00:00.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5638208a36bc] 2024-03-28T18:00:00.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56382088d1ff] 2024-03-28T18:00:00.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f40e80bc488] 2024-03-28T18:00:00.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 16: (DispatchQueue::entry()+0x51f) [0x7f40e80b960f] 2024-03-28T18:00:00.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40e818ce81] 2024-03-28T18:00:00.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f40e6ba71ca] 2024-03-28T18:00:00.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 19: clone() 2024-03-28T18:00:00.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:00:00.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 2024-03-28T18:00:00.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: -1> 2024-03-28T18:00:00.565+0000 7f40dc600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f40dc600700 time 2024-03-28T18:00:00.564200+0000 2024-03-28T18:00:00.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:00:00.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 2024-03-28T18:00:00.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:00.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f40e7e2a4a1] 2024-03-28T18:00:00.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f40e7e2a667] 2024-03-28T18:00:00.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563820ad5223] 2024-03-28T18:00:00.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56382094f5d6] 2024-03-28T18:00:00.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56382094fda1] 2024-03-28T18:00:00.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5638209522d5] 2024-03-28T18:00:00.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5638209a2233] 2024-03-28T18:00:00.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5638209a4197] 2024-03-28T18:00:00.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5638208a311b] 2024-03-28T18:00:00.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5638208a36bc] 2024-03-28T18:00:00.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56382088d1ff] 2024-03-28T18:00:00.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f40e80bc488] 2024-03-28T18:00:00.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 13: (DispatchQueue::entry()+0x51f) [0x7f40e80b960f] 2024-03-28T18:00:00.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40e818ce81] 2024-03-28T18:00:00.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f40e6ba71ca] 2024-03-28T18:00:00.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 16: clone() 2024-03-28T18:00:00.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 2024-03-28T18:00:00.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 0> 2024-03-28T18:00:00.567+0000 7f40dc600700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:00:00.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: in thread 7f40dc600700 thread_name:ms_dispatch 2024-03-28T18:00:00.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 2024-03-28T18:00:00.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:00.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f40e6bb1d20] 2024-03-28T18:00:00.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 2: gsignal() 2024-03-28T18:00:00.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 3: abort() 2024-03-28T18:00:00.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f40e7e2a4fb] 2024-03-28T18:00:00.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f40e7e2a667] 2024-03-28T18:00:00.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563820ad5223] 2024-03-28T18:00:00.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56382094f5d6] 2024-03-28T18:00:00.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56382094fda1] 2024-03-28T18:00:00.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5638209522d5] 2024-03-28T18:00:00.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5638209a2233] 2024-03-28T18:00:00.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5638209a4197] 2024-03-28T18:00:00.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5638208a311b] 2024-03-28T18:00:00.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5638208a36bc] 2024-03-28T18:00:00.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56382088d1ff] 2024-03-28T18:00:00.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f40e80bc488] 2024-03-28T18:00:00.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 16: (DispatchQueue::entry()+0x51f) [0x7f40e80b960f] 2024-03-28T18:00:00.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40e818ce81] 2024-03-28T18:00:00.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f40e6ba71ca] 2024-03-28T18:00:00.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 19: clone() 2024-03-28T18:00:00.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:00:00.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[73411]: 2024-03-28T18:00:00.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:00 smithi045 podman[93717]: 2024-03-28 18:00:00.699873636 +0000 UTC m=+0.034508893 container died 764585b7da2e237900125736646335a1b634086557347db24f4aff78bf9e666f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:00:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:01 smithi151 ceph-mon[36481]: pgmap v376: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 102 B/s wr, 7 op/s 2024-03-28T18:00:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:01 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 3 up:standby-replay 2024-03-28T18:00:01.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:01 smithi045 ceph-mon[31773]: pgmap v376: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 102 B/s wr, 7 op/s 2024-03-28T18:00:01.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:01 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 3 up:standby-replay 2024-03-28T18:00:01.916 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:01 smithi045 podman[93717]: 2024-03-28 18:00:01.569773641 +0000 UTC m=+0.904408959 container remove 764585b7da2e237900125736646335a1b634086557347db24f4aff78bf9e666f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:00:01.916 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:01 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:00:01.917 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 454 sec latency 569813.901 ms 2024-03-28T18:00:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:01 smithi071 ceph-mon[36597]: pgmap v376: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 102 B/s wr, 7 op/s 2024-03-28T18:00:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:01 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 3 up:standby-replay 2024-03-28T18:00:02.231 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:02 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-28T18:00:02.231 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:02 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Consumed 2.109s CPU time. 2024-03-28T18:00:02.917 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 455 sec latency 570814.040 ms 2024-03-28T18:00:03.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:03 smithi151 ceph-mon[36481]: pgmap v377: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:00:03.887 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:03 smithi045 ceph-mon[31773]: pgmap v377: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:00:03.918 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 456 sec latency 571814.200 ms 2024-03-28T18:00:04.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:03 smithi071 ceph-mon[36597]: pgmap v377: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:00:04.917 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 457 sec latency 572814.341 ms 2024-03-28T18:00:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:05 smithi151 ceph-mon[36481]: pgmap v378: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:00:06.104 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:05 smithi071 ceph-mon[36597]: pgmap v378: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:00:06.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:05 smithi045 ceph-mon[31773]: pgmap v378: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:00:06.104 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 458 sec latency 573814.488 ms 2024-03-28T18:00:06.917 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 459 sec latency 574814.627 ms 2024-03-28T18:00:07.210 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:00:07.211 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:00:07.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.449+0000 7fbb01000640 1 -- 172.21.15.45:0/553615200 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbafc107eb0 msgr2=0x7fbafc108290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:07.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.449+0000 7fbb01000640 1 --2- 172.21.15.45:0/553615200 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbafc107eb0 0x7fbafc108290 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fbaec009900 tx=0x7fbaec02bc30 comp rx=0 tx=0).stop 2024-03-28T18:00:07.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.449+0000 7fbb01000640 1 -- 172.21.15.45:0/553615200 shutdown_connections 2024-03-28T18:00:07.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.449+0000 7fbb01000640 1 --2- 172.21.15.45:0/553615200 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbafc10cf80 0x7fbafc114b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.449+0000 7fbb01000640 1 --2- 172.21.15.45:0/553615200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbafc108860 0x7fbafc10c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.449+0000 7fbb01000640 1 --2- 172.21.15.45:0/553615200 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbafc107eb0 0x7fbafc108290 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.449+0000 7fbb01000640 1 -- 172.21.15.45:0/553615200 >> 172.21.15.45:0/553615200 conn(0x7fbafc075be0 msgr2=0x7fbafc078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:07.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.450+0000 7fbb01000640 1 -- 172.21.15.45:0/553615200 shutdown_connections 2024-03-28T18:00:07.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.450+0000 7fbb01000640 1 -- 172.21.15.45:0/553615200 wait complete. 2024-03-28T18:00:07.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.450+0000 7fbb01000640 1 Processor -- start 2024-03-28T18:00:07.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.451+0000 7fbb01000640 1 -- start start 2024-03-28T18:00:07.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.451+0000 7fbb01000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbafc107eb0 0x7fbafc19e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:07.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.451+0000 7fbb01000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbafc10cf80 0x7fbafc19eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:07.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.451+0000 7fbb01000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbafc1a2e70 0x7fbafc1a3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:07.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.451+0000 7fbb01000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc118030 con 0x7fbafc10cf80 2024-03-28T18:00:07.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.451+0000 7fbb01000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc117eb0 con 0x7fbafc1a2e70 2024-03-28T18:00:07.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.451+0000 7fbb01000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc1181b0 con 0x7fbafc107eb0 2024-03-28T18:00:07.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.451+0000 7fbaf1400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbafc10cf80 0x7fbafc19eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:07.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.451+0000 7fbafa800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbafc1a2e70 0x7fbafc1a3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:07.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.451+0000 7fbafa800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbafc1a2e70 0x7fbafc1a3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40738/0 (socket says 172.21.15.45:40738) 2024-03-28T18:00:07.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.451+0000 7fbaf1400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbafc10cf80 0x7fbafc19eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45362/0 (socket says 172.21.15.45:45362) 2024-03-28T18:00:07.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.451+0000 7fbafa800640 1 -- 172.21.15.45:0/2173284679 learned_addr learned my addr 172.21.15.45:0/2173284679 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:00:07.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.451+0000 7fbaf9e00640 1 --2- 172.21.15.45:0/2173284679 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbafc107eb0 0x7fbafc19e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:07.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.452+0000 7fbafa800640 1 -- 172.21.15.45:0/2173284679 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbafc107eb0 msgr2=0x7fbafc19e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:07.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.452+0000 7fbafa800640 1 --2- 172.21.15.45:0/2173284679 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbafc107eb0 0x7fbafc19e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.452+0000 7fbafa800640 1 -- 172.21.15.45:0/2173284679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbafc10cf80 msgr2=0x7fbafc19eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:07.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.452+0000 7fbafa800640 1 --2- 172.21.15.45:0/2173284679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbafc10cf80 0x7fbafc19eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.452+0000 7fbafa800640 1 -- 172.21.15.45:0/2173284679 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc1a3960 con 0x7fbafc1a2e70 2024-03-28T18:00:07.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.452+0000 7fbaf1400640 1 --2- 172.21.15.45:0/2173284679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbafc10cf80 0x7fbafc19eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:00:07.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.452+0000 7fbafa800640 1 --2- 172.21.15.45:0/2173284679 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbafc1a2e70 0x7fbafc1a3320 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fbaf400eaa0 tx=0x7fbaf400ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:07.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.452+0000 7fbaf3400640 1 -- 172.21.15.45:0/2173284679 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf400cdb0 con 0x7fbafc1a2e70 2024-03-28T18:00:07.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.452+0000 7fbaf3400640 1 -- 172.21.15.45:0/2173284679 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbaf4005020 con 0x7fbafc1a2e70 2024-03-28T18:00:07.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.452+0000 7fbaf3400640 1 -- 172.21.15.45:0/2173284679 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf40116b0 con 0x7fbafc1a2e70 2024-03-28T18:00:07.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.452+0000 7fbb01000640 1 -- 172.21.15.45:0/2173284679 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbafc1a3c50 con 0x7fbafc1a2e70 2024-03-28T18:00:07.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.453+0000 7fbb01000640 1 -- 172.21.15.45:0/2173284679 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbafc1ab3f0 con 0x7fbafc1a2e70 2024-03-28T18:00:07.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.456+0000 7fbaf3400640 1 -- 172.21.15.45:0/2173284679 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fbaf4005560 con 0x7fbafc1a2e70 2024-03-28T18:00:07.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.456+0000 7fbb03d6e740 1 Processor -- start 2024-03-28T18:00:07.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.456+0000 7fbb03d6e740 1 -- start start 2024-03-28T18:00:07.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.456+0000 7fbb03d6e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f5c00 0x5640af9f4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:07.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.456+0000 7fbb03d6e740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5640af9f4f20 0x5640af9f53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:07.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.456+0000 7fbb03d6e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640af9fe290 0x5640afa00680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:07.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.456+0000 7fbb03d6e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640af959250 con 0x5640af9fe290 2024-03-28T18:00:07.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.456+0000 7fbb03d6e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640af901a30 con 0x5640af9f5c00 2024-03-28T18:00:07.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.456+0000 7fbb03d6e740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640af9552f0 con 0x5640af9f4f20 2024-03-28T18:00:07.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.457+0000 7fbaf1400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f5c00 0x5640af9f4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:07.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.457+0000 7fbaf1400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f5c00 0x5640af9f4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40750/0 (socket says 172.21.15.45:40750) 2024-03-28T18:00:07.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.457+0000 7fbaf1400640 1 -- 172.21.15.45:0/4017253051 learned_addr learned my addr 172.21.15.45:0/4017253051 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:00:07.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.457+0000 7fbaf9e00640 1 --2- 172.21.15.45:0/4017253051 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5640af9f4f20 0x5640af9f53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:07.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.457+0000 7fbaf1400640 1 --2- 172.21.15.45:0/4017253051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640af9fe290 0x5640afa00680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:07.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.457+0000 7fbaf1400640 1 -- 172.21.15.45:0/4017253051 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5640af9f4f20 msgr2=0x5640af9f53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:00:07.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.457+0000 7fbaf1400640 1 --2- 172.21.15.45:0/4017253051 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5640af9f4f20 0x5640af9f53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.457+0000 7fbaf1400640 1 -- 172.21.15.45:0/4017253051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640af9fe290 msgr2=0x5640afa00680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:00:07.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.457+0000 7fbaf1400640 1 --2- 172.21.15.45:0/4017253051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640af9fe290 0x5640afa00680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.457+0000 7fbaf1400640 1 -- 172.21.15.45:0/4017253051 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5640af8ec690 con 0x5640af9f5c00 2024-03-28T18:00:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.457+0000 7fbaf9e00640 1 --2- 172.21.15.45:0/4017253051 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5640af9f4f20 0x5640af9f53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:00:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.457+0000 7fbaf1400640 1 --2- 172.21.15.45:0/4017253051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f5c00 0x5640af9f4950 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fbae4011ba0 tx=0x7fbae4009770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b9ae85affaf8256 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.457+0000 7fbaf3400640 1 --2- 172.21.15.45:0/2173284679 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbae00a29f0 0x7fbae00a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.458+0000 7fbaeb400640 1 -- 172.21.15.45:0/4017253051 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae400fc50 con 0x5640af9f5c00 2024-03-28T18:00:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.458+0000 7fbaeb400640 1 -- 172.21.15.45:0/4017253051 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbae400fdf0 con 0x5640af9f5c00 2024-03-28T18:00:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.458+0000 7fbaeb400640 1 -- 172.21.15.45:0/4017253051 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae400e3e0 con 0x5640af9f5c00 2024-03-28T18:00:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.458+0000 7fbaf3400640 1 -- 172.21.15.45:0/2173284679 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 11436+0+0 (secure 0 0 0) 0x7fbaf40cfac0 con 0x7fbafc1a2e70 2024-03-28T18:00:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.458+0000 7fbaf9e00640 1 --2- 172.21.15.45:0/2173284679 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbae00a29f0 0x7fbae00a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.458+0000 7fbb03d6e740 1 -- 172.21.15.45:0/4017253051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f5c00 msgr2=0x5640af9f4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.458+0000 7fbb03d6e740 1 --2- 172.21.15.45:0/4017253051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f5c00 0x5640af9f4950 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fbae4011ba0 tx=0x7fbae4009770 comp rx=0 tx=0).stop 2024-03-28T18:00:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.458+0000 7fbb03d6e740 1 -- 172.21.15.45:0/4017253051 shutdown_connections 2024-03-28T18:00:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.458+0000 7fbb03d6e740 1 --2- 172.21.15.45:0/4017253051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640af9fe290 0x5640afa00680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.458+0000 7fbb03d6e740 1 --2- 172.21.15.45:0/4017253051 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5640af9f4f20 0x5640af9f53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.458+0000 7fbb03d6e740 1 --2- 172.21.15.45:0/4017253051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f5c00 0x5640af9f4950 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.458+0000 7fbb03d6e740 1 -- 172.21.15.45:0/4017253051 >> 172.21.15.45:0/4017253051 conn(0x5640af9bc200 msgr2=0x5640af9ef970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:07.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.458+0000 7fbaf9e00640 1 --2- 172.21.15.45:0/2173284679 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbae00a29f0 0x7fbae00a4eb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbaec006010 tx=0x7fbaec02e040 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:07.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.459+0000 7fbb03d6e740 1 -- 172.21.15.45:0/4017253051 shutdown_connections 2024-03-28T18:00:07.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.459+0000 7fbb03d6e740 1 -- 172.21.15.45:0/4017253051 wait complete. 2024-03-28T18:00:07.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.459+0000 7fbb03d6e740 1 -- start start 2024-03-28T18:00:07.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.459+0000 7fbb03d6e740 1 Processor -- start 2024-03-28T18:00:07.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.460+0000 7fbb03d6e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f4f20 0x5640af9fd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:07.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.460+0000 7fbb03d6e740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5640af9f5c00 0x5640af9fad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.460+0000 7fbb03d6e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640af9fe290 0x5640af9fb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.460+0000 7fbb03d6e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640afa01100 con 0x5640af9fe290 2024-03-28T18:00:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.460+0000 7fbb03d6e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640af914e80 con 0x5640af9f4f20 2024-03-28T18:00:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.460+0000 7fbb03d6e740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640afa01280 con 0x5640af9f5c00 2024-03-28T18:00:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.460+0000 7fbb03d6e740 10 client.0 _ll_register_callbacks cb 0x5640af927130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:00:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.460+0000 7fbaf9e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f4f20 0x5640af9fd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.460+0000 7fbaf1400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5640af9f5c00 0x5640af9fad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.460+0000 7fbaf9e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f4f20 0x5640af9fd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40756/0 (socket says 172.21.15.45:40756) 2024-03-28T18:00:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.460+0000 7fbaf9e00640 1 -- 172.21.15.45:0/1825683430 learned_addr learned my addr 172.21.15.45:0/1825683430 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:00:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.460+0000 7fbafa800640 1 --2- 172.21.15.45:0/1825683430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640af9fe290 0x5640af9fb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.460+0000 7fbaf9e00640 1 -- 172.21.15.45:0/1825683430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5640af9f5c00 msgr2=0x5640af9fad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:00:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.460+0000 7fbaf9e00640 1 --2- 172.21.15.45:0/1825683430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5640af9f5c00 0x5640af9fad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.460+0000 7fbaf9e00640 1 -- 172.21.15.45:0/1825683430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640af9fe290 msgr2=0x5640af9fb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:00:07.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.460+0000 7fbaf9e00640 1 --2- 172.21.15.45:0/1825683430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640af9fe290 0x5640af9fb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.460+0000 7fbaf9e00640 1 -- 172.21.15.45:0/1825683430 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5640af7377d0 con 0x5640af9f4f20 2024-03-28T18:00:07.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.461+0000 7fbaf9e00640 1 --2- 172.21.15.45:0/1825683430 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f4f20 0x5640af9fd9b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fbaec02c170 tx=0x7fbaec02bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0164c8acfa002fe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:07.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.461+0000 7fbadfe00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T18:00:07.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.461+0000 7fbadfe00640 1 -- 172.21.15.45:0/1825683430 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec04e070 con 0x5640af9f4f20 2024-03-28T18:00:07.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.461+0000 7fbadfe00640 1 -- 172.21.15.45:0/1825683430 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(130..130 src has 1..130) v4 ==== 11436+0+0 (secure 0 0 0) 0x7fbaec004460 con 0x5640af9f4f20 2024-03-28T18:00:07.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.461+0000 7fbb03d6e740 1 -- 172.21.15.45:0/1825683430 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbae400fdf0 con 0x5640af9f4f20 2024-03-28T18:00:07.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.461+0000 7fbadfe00640 1 -- 172.21.15.45:0/1825683430 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=246) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbaec004ec0 con 0x5640af9f4f20 2024-03-28T18:00:07.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.461+0000 7fbb03d6e740 10 client.26405 fetch_fsmap learned FSMap version 246 2024-03-28T18:00:07.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.461+0000 7fbb03d6e740 1 -- 172.21.15.45:0/1825683430 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=246,monmap=4+}) v3 -- 0x5640af66d3f0 con 0x5640af9f4f20 2024-03-28T18:00:07.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.461+0000 7fbadfe00640 1 -- 172.21.15.45:0/1825683430 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbae400fdf0 con 0x5640af9f4f20 2024-03-28T18:00:07.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.462+0000 7fbadfe00640 1 -- 172.21.15.45:0/1825683430 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 246) v1 ==== 4468+0+0 (secure 0 0 0) 0x7fbaec003bb0 con 0x5640af9f4f20 2024-03-28T18:00:07.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.462+0000 7fbb03d6e740 10 client.26405 fetch_fsmap finished waiting for FSMap version 246 2024-03-28T18:00:07.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.462+0000 7fbb03d6e740 10 client.26405 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:00:07.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.462+0000 7fbb03d6e740 20 client.26405 populate_metadata read hostname 'smithi045' 2024-03-28T18:00:07.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.462+0000 7fbb03d6e740 1 --2- 172.21.15.45:0/1825683430 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5640afa0e530 0x5640afa10990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:07.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.462+0000 7fbb03d6e740 4 client.26405 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:00:07.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.462+0000 7fbb03d6e740 1 -- 172.21.15.45:0/1825683430 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5640af966790 con 0x5640afa0e530 2024-03-28T18:00:07.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.462+0000 7fbaf1400640 1 --2- 172.21.15.45:0/1825683430 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5640afa0e530 0x5640afa10990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:07.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.463+0000 7fbaf1400640 1 --2- 172.21.15.45:0/1825683430 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5640afa0e530 0x5640afa10990 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4226a96667dd73ef server_cookie=ad5cd5751e1351d6 in_seq=0 out_seq=0 2024-03-28T18:00:07.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.463+0000 7fbadfe00640 10 client.26405 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:00:07.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.463+0000 7fbadfe00640 1 -- 172.21.15.45:0/1825683430 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5640af966790 con 0x5640afa0e530 2024-03-28T18:00:07.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.463+0000 7fbadfe00640 10 client.26405 handle_command_reply: tid=0 2024-03-28T18:00:07.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.463+0000 7fbb03d6e740 1 client.26405 shutdown 2024-03-28T18:00:07.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.463+0000 7fbb03d6e740 1 -- 172.21.15.45:0/1825683430 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f4f20 msgr2=0x5640af9fd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:07.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.463+0000 7fbb03d6e740 1 --2- 172.21.15.45:0/1825683430 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f4f20 0x5640af9fd9b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fbaec02c170 tx=0x7fbaec02bc30 comp rx=0 tx=0).stop 2024-03-28T18:00:07.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.463+0000 7fbb03d6e740 1 -- 172.21.15.45:0/1825683430 shutdown_connections 2024-03-28T18:00:07.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.464+0000 7fbb03d6e740 1 --2- 172.21.15.45:0/1825683430 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5640afa0e530 0x5640afa10990 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.464+0000 7fbb03d6e740 1 --2- 172.21.15.45:0/1825683430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640af9fe290 0x5640af9fb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.464+0000 7fbb03d6e740 1 --2- 172.21.15.45:0/1825683430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5640af9f5c00 0x5640af9fad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.464+0000 7fbb03d6e740 1 --2- 172.21.15.45:0/1825683430 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f4f20 0x5640af9fd9b0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.464+0000 7fbadfe00640 0 client.26405 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:00:07.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.464+0000 7fbb03d6e740 1 -- 172.21.15.45:0/1825683430 >> 172.21.15.45:0/1825683430 conn(0x5640af9bc200 msgr2=0x5640af9f0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:07.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.464+0000 7fbb03d6e740 1 -- 172.21.15.45:0/1825683430 shutdown_connections 2024-03-28T18:00:07.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.464+0000 7fbb03d6e740 1 -- 172.21.15.45:0/1825683430 wait complete. 2024-03-28T18:00:07.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.464+0000 7fbb03d6e740 20 client.26405 trim_cache size 0 max 16384 2024-03-28T18:00:07.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.481+0000 7fbb03d6e740 1 Processor -- start 2024-03-28T18:00:07.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.481+0000 7fbb03d6e740 1 -- start start 2024-03-28T18:00:07.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.482+0000 7fbb03d6e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640af9f4f20 0x5640af9fb160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:07.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.482+0000 7fbb03d6e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f5c00 0x5640af9fd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:07.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.482+0000 7fbb03d6e740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5640af9fe290 0x5640af9fd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:07.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.482+0000 7fbb03d6e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc1b5b80 con 0x5640af9f4f20 2024-03-28T18:00:07.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.482+0000 7fbb03d6e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc1b3ba0 con 0x5640af9f5c00 2024-03-28T18:00:07.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.482+0000 7fbb03d6e740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc1b5520 con 0x5640af9fe290 2024-03-28T18:00:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.482+0000 7fbaf1400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f5c00 0x5640af9fd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.482+0000 7fbaf1400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f5c00 0x5640af9fd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40768/0 (socket says 172.21.15.45:40768) 2024-03-28T18:00:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.482+0000 7fbaf1400640 1 -- 172.21.15.45:0/1755893145 learned_addr learned my addr 172.21.15.45:0/1755893145 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:00:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.482+0000 7fbaf9e00640 1 --2- 172.21.15.45:0/1755893145 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640af9f4f20 0x5640af9fb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.482+0000 7fbaf1400640 1 -- 172.21.15.45:0/1755893145 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5640af9fe290 msgr2=0x5640af9fd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:00:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.482+0000 7fbaf1400640 1 --2- 172.21.15.45:0/1755893145 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5640af9fe290 0x5640af9fd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.482+0000 7fbaf1400640 1 -- 172.21.15.45:0/1755893145 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640af9f4f20 msgr2=0x5640af9fb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:00:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.482+0000 7fbaf1400640 1 --2- 172.21.15.45:0/1755893145 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640af9f4f20 0x5640af9fb160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.482+0000 7fbaf1400640 1 -- 172.21.15.45:0/1755893145 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5640af62d0f0 con 0x5640af9f5c00 2024-03-28T18:00:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.482+0000 7fbaf9e00640 1 --2- 172.21.15.45:0/1755893145 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640af9f4f20 0x5640af9fb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:00:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.482+0000 7fbaf1400640 1 --2- 172.21.15.45:0/1755893145 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f5c00 0x5640af9fd470 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fbae40097e0 tx=0x7fbae400ffa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c162e92983eaf95a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.483+0000 7fbaeaa00640 1 -- 172.21.15.45:0/1755893145 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae4009e60 con 0x5640af9f5c00 2024-03-28T18:00:07.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.483+0000 7fbaeaa00640 1 -- 172.21.15.45:0/1755893145 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbae400fd40 con 0x5640af9f5c00 2024-03-28T18:00:07.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.483+0000 7fbaeaa00640 1 -- 172.21.15.45:0/1755893145 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae402da80 con 0x5640af9f5c00 2024-03-28T18:00:07.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.483+0000 7fbb03d6e740 1 -- 172.21.15.45:0/1755893145 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f5c00 msgr2=0x5640af9fd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:07.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.483+0000 7fbb03d6e740 1 --2- 172.21.15.45:0/1755893145 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f5c00 0x5640af9fd470 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fbae40097e0 tx=0x7fbae400ffa0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.483+0000 7fbb03d6e740 1 -- 172.21.15.45:0/1755893145 shutdown_connections 2024-03-28T18:00:07.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.483+0000 7fbb03d6e740 1 --2- 172.21.15.45:0/1755893145 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5640af9fe290 0x5640af9fd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.483+0000 7fbb03d6e740 1 --2- 172.21.15.45:0/1755893145 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f5c00 0x5640af9fd470 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.483+0000 7fbb03d6e740 1 --2- 172.21.15.45:0/1755893145 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640af9f4f20 0x5640af9fb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.483+0000 7fbb03d6e740 1 -- 172.21.15.45:0/1755893145 >> 172.21.15.45:0/1755893145 conn(0x5640af9bc200 msgr2=0x5640af9f9610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:07.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.483+0000 7fbb03d6e740 1 -- 172.21.15.45:0/1755893145 shutdown_connections 2024-03-28T18:00:07.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.483+0000 7fbb03d6e740 1 -- 172.21.15.45:0/1755893145 wait complete. 2024-03-28T18:00:07.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.484+0000 7fbb03d6e740 1 -- start start 2024-03-28T18:00:07.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.484+0000 7fbb03d6e740 1 Processor -- start 2024-03-28T18:00:07.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.484+0000 7fbb03d6e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f4f20 0x5640af9f7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:07.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.484+0000 7fbb03d6e740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5640af9f5c00 0x5640af9f7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:07.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.484+0000 7fbb03d6e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640af9fe290 0x5640af9f8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:07.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.484+0000 7fbb03d6e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc1b4860 con 0x5640af9fe290 2024-03-28T18:00:07.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.484+0000 7fbb03d6e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc1b4200 con 0x5640af9f4f20 2024-03-28T18:00:07.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.484+0000 7fbb03d6e740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc1b4ec0 con 0x5640af9f5c00 2024-03-28T18:00:07.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.485+0000 7fbb03d6e740 10 client.0 _ll_register_callbacks cb 0x7fbaec02c5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:00:07.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.485+0000 7fbaf9e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f4f20 0x5640af9f7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:07.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.485+0000 7fbafa800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640af9fe290 0x5640af9f8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:07.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.485+0000 7fbaf9e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f4f20 0x5640af9f7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40778/0 (socket says 172.21.15.45:40778) 2024-03-28T18:00:07.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.485+0000 7fbaf9e00640 1 -- 172.21.15.45:0/3542759349 learned_addr learned my addr 172.21.15.45:0/3542759349 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:00:07.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.485+0000 7fbaf1400640 1 --2- 172.21.15.45:0/3542759349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5640af9f5c00 0x5640af9f7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:07.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.485+0000 7fbaf9e00640 1 -- 172.21.15.45:0/3542759349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5640af9f5c00 msgr2=0x5640af9f7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:00:07.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.485+0000 7fbaf9e00640 1 --2- 172.21.15.45:0/3542759349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5640af9f5c00 0x5640af9f7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.485+0000 7fbaf9e00640 1 -- 172.21.15.45:0/3542759349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640af9fe290 msgr2=0x5640af9f8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:00:07.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.485+0000 7fbaf9e00640 1 --2- 172.21.15.45:0/3542759349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640af9fe290 0x5640af9f8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.485+0000 7fbaf9e00640 1 -- 172.21.15.45:0/3542759349 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5640afa69100 con 0x5640af9f4f20 2024-03-28T18:00:07.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.485+0000 7fbafa800640 1 --2- 172.21.15.45:0/3542759349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640af9fe290 0x5640af9f8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:00:07.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.485+0000 7fbaf1400640 1 --2- 172.21.15.45:0/3542759349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5640af9f5c00 0x5640af9f7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:00:07.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.485+0000 7fbaf9e00640 1 --2- 172.21.15.45:0/3542759349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f4f20 0x5640af9f7690 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fbaec000c00 tx=0x7fbaec03f6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a485e477129fa559 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:07.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.485+0000 7fbadfe00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T18:00:07.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.486+0000 7fbadfe00640 1 -- 172.21.15.45:0/3542759349 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec04e040 con 0x5640af9f4f20 2024-03-28T18:00:07.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.486+0000 7fbb03d6e740 1 -- 172.21.15.45:0/3542759349 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbae400fd40 con 0x5640af9f4f20 2024-03-28T18:00:07.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.486+0000 7fbadfe00640 1 -- 172.21.15.45:0/3542759349 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(130..130 src has 1..130) v4 ==== 11436+0+0 (secure 0 0 0) 0x7fbaec054040 con 0x5640af9f4f20 2024-03-28T18:00:07.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.486+0000 7fbadfe00640 1 -- 172.21.15.45:0/3542759349 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=246) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbaec06e050 con 0x5640af9f4f20 2024-03-28T18:00:07.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.486+0000 7fbb03d6e740 10 client.26411 fetch_fsmap learned FSMap version 246 2024-03-28T18:00:07.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.486+0000 7fbb03d6e740 1 -- 172.21.15.45:0/3542759349 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=246,monmap=4+}) v3 -- 0x5640afa6a360 con 0x5640af9f4f20 2024-03-28T18:00:07.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.486+0000 7fbadfe00640 1 -- 172.21.15.45:0/3542759349 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbae400fd40 con 0x5640af9f4f20 2024-03-28T18:00:07.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.486+0000 7fbadfe00640 1 -- 172.21.15.45:0/3542759349 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 246) v1 ==== 4468+0+0 (secure 0 0 0) 0x7fbaec054240 con 0x5640af9f4f20 2024-03-28T18:00:07.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.486+0000 7fbb03d6e740 10 client.26411 fetch_fsmap finished waiting for FSMap version 246 2024-03-28T18:00:07.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.486+0000 7fbb03d6e740 10 client.26411 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:00:07.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.486+0000 7fbb03d6e740 20 client.26411 populate_metadata read hostname 'smithi045' 2024-03-28T18:00:07.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.486+0000 7fbb03d6e740 1 --2- 172.21.15.45:0/3542759349 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5640afa6aa60 0x5640afa6ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:07.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.486+0000 7fbb03d6e740 4 client.26411 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:00:07.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.486+0000 7fbb03d6e740 1 -- 172.21.15.45:0/3542759349 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5640af960ae0 con 0x5640afa6aa60 2024-03-28T18:00:07.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.487+0000 7fbaf1400640 1 --2- 172.21.15.45:0/3542759349 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5640afa6aa60 0x5640afa6ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:07.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.487+0000 7fbaf1400640 1 --2- 172.21.15.45:0/3542759349 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5640afa6aa60 0x5640afa6ce80 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93e015d7e9f63baa server_cookie=4f9ba47e282b5ead in_seq=0 out_seq=0 2024-03-28T18:00:07.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.487+0000 7fbadfe00640 10 client.26411 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:00:07.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.487+0000 7fbadfe00640 1 -- 172.21.15.45:0/3542759349 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5640af960ae0 con 0x5640afa6aa60 2024-03-28T18:00:07.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.487+0000 7fbadfe00640 10 client.26411 handle_command_reply: tid=0 2024-03-28T18:00:07.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.487+0000 7fbb03d6e740 1 client.26411 shutdown 2024-03-28T18:00:07.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.488+0000 7fbb03d6e740 1 -- 172.21.15.45:0/3542759349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f4f20 msgr2=0x5640af9f7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:07.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.488+0000 7fbb03d6e740 1 --2- 172.21.15.45:0/3542759349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f4f20 0x5640af9f7690 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fbaec000c00 tx=0x7fbaec03f6c0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.488+0000 7fbb03d6e740 1 -- 172.21.15.45:0/3542759349 shutdown_connections 2024-03-28T18:00:07.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.488+0000 7fbb03d6e740 1 --2- 172.21.15.45:0/3542759349 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5640afa6aa60 0x5640afa6ce80 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.488+0000 7fbb03d6e740 1 --2- 172.21.15.45:0/3542759349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640af9fe290 0x5640af9f8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.488+0000 7fbb03d6e740 1 --2- 172.21.15.45:0/3542759349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5640af9f5c00 0x5640af9f7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.488+0000 7fbadfe00640 0 client.26411 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:00:07.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.488+0000 7fbb03d6e740 1 --2- 172.21.15.45:0/3542759349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5640af9f4f20 0x5640af9f7690 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.488+0000 7fbb03d6e740 1 -- 172.21.15.45:0/3542759349 >> 172.21.15.45:0/3542759349 conn(0x5640af9bc200 msgr2=0x5640af9f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:07.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.488+0000 7fbb03d6e740 1 -- 172.21.15.45:0/3542759349 shutdown_connections 2024-03-28T18:00:07.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.488+0000 7fbb03d6e740 1 -- 172.21.15.45:0/3542759349 wait complete. 2024-03-28T18:00:07.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.488+0000 7fbb03d6e740 20 client.26411 trim_cache size 0 max 16384 2024-03-28T18:00:07.493 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T18:00:07.493 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:00:07.493 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T18:00:07.493 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T18:00:07.493 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T18:00:07.493 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T18:00:07.493 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T18:00:07.493 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:00:07.493 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:00:07.493 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T18:00:07.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.489+0000 7fbb01000640 1 -- 172.21.15.45:0/2173284679 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbae00a29f0 msgr2=0x7fbae00a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:07.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.489+0000 7fbb01000640 1 --2- 172.21.15.45:0/2173284679 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbae00a29f0 0x7fbae00a4eb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbaec006010 tx=0x7fbaec02e040 comp rx=0 tx=0).stop 2024-03-28T18:00:07.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.489+0000 7fbb01000640 1 -- 172.21.15.45:0/2173284679 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbafc1a2e70 msgr2=0x7fbafc1a3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:07.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.489+0000 7fbb01000640 1 --2- 172.21.15.45:0/2173284679 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbafc1a2e70 0x7fbafc1a3320 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fbaf400eaa0 tx=0x7fbaf400ef60 comp rx=0 tx=0).stop 2024-03-28T18:00:07.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.489+0000 7fbb01000640 1 -- 172.21.15.45:0/2173284679 shutdown_connections 2024-03-28T18:00:07.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.489+0000 7fbb01000640 1 --2- 172.21.15.45:0/2173284679 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fbae00a29f0 0x7fbae00a4eb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.489+0000 7fbb01000640 1 --2- 172.21.15.45:0/2173284679 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbafc1a2e70 0x7fbafc1a3320 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.489+0000 7fbb01000640 1 --2- 172.21.15.45:0/2173284679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbafc10cf80 0x7fbafc19eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.489+0000 7fbb01000640 1 --2- 172.21.15.45:0/2173284679 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbafc107eb0 0x7fbafc19e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:07.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.489+0000 7fbb01000640 1 -- 172.21.15.45:0/2173284679 >> 172.21.15.45:0/2173284679 conn(0x7fbafc075be0 msgr2=0x7fbafc077f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:07.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.489+0000 7fbb01000640 1 -- 172.21.15.45:0/2173284679 shutdown_connections 2024-03-28T18:00:07.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:07.489+0000 7fbb01000640 1 -- 172.21.15.45:0/2173284679 wait complete. 2024-03-28T18:00:07.508 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T18:00:07.862 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:07 smithi071 ceph-mon[36597]: pgmap v379: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-28T18:00:07.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:07 smithi045 ceph-mon[31773]: pgmap v379: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-28T18:00:07.917 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 460 sec latency 575814.799 ms 2024-03-28T18:00:08.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:07 smithi151 ceph-mon[36481]: pgmap v379: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-28T18:00:08.918 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 461 sec latency 576814.946 ms 2024-03-28T18:00:09.903 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:09 smithi045 ceph-mon[31773]: pgmap v380: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 177 B/s wr, 10 op/s 2024-03-28T18:00:09.918 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 462 sec latency 577815.074 ms 2024-03-28T18:00:10.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:09 smithi071 ceph-mon[36597]: pgmap v380: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 177 B/s wr, 10 op/s 2024-03-28T18:00:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:09 smithi151 ceph-mon[36481]: pgmap v380: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 177 B/s wr, 10 op/s 2024-03-28T18:00:10.918 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 463 sec latency 578815.166 ms 2024-03-28T18:00:11.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:11 smithi045 ceph-mon[31773]: pgmap v381: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T18:00:11.918 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 464 sec latency 579815.316 ms 2024-03-28T18:00:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:11 smithi071 ceph-mon[36597]: pgmap v381: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T18:00:12.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:11 smithi151 ceph-mon[36481]: pgmap v381: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T18:00:12.586 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:12 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 1. 2024-03-28T18:00:12.586 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:12 smithi045 systemd[1]: Stopped Ceph mds.g for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:00:12.586 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:12 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Consumed 2.109s CPU time. 2024-03-28T18:00:12.586 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:12 smithi045 systemd[1]: Starting Ceph mds.g for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T18:00:12.916 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:12 smithi045 podman[93875]: 2024-03-28 18:00:12.494244404 +0000 UTC m=+0.039622167 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:00:12.916 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:12 smithi045 podman[93875]: 2024-03-28 18:00:12.820902392 +0000 UTC m=+0.366280170 container create 09caf3eb991f701df802715b4403cc12179bec1bba098facb75328323dd44fc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-28T18:00:12.918 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 465 sec latency 580815.437 ms 2024-03-28T18:00:13.416 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:13 smithi045 podman[93875]: 2024-03-28 18:00:13.091161828 +0000 UTC m=+0.636539649 container init 09caf3eb991f701df802715b4403cc12179bec1bba098facb75328323dd44fc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-28T18:00:13.416 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:13 smithi045 podman[93875]: 2024-03-28 18:00:13.10230277 +0000 UTC m=+0.647680527 container start 09caf3eb991f701df802715b4403cc12179bec1bba098facb75328323dd44fc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-28T18:00:13.416 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:13 smithi045 ceph-mds[93893]: -- 172.21.15.45:0/1240146611 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5591578eba40 con 0x5591578be800 2024-03-28T18:00:13.417 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:13 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: starting mds.g at 2024-03-28T18:00:13.417 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:13 smithi045 bash[93875]: 09caf3eb991f701df802715b4403cc12179bec1bba098facb75328323dd44fc6 2024-03-28T18:00:13.417 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:13 smithi045 systemd[1]: Started Ceph mds.g for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:00:13.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:13 smithi045 ceph-mon[31773]: pgmap v382: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 85 B/s wr, 11 op/s 2024-03-28T18:00:13.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:13 smithi045 ceph-mon[31773]: daemon mds.g restarted 2024-03-28T18:00:13.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:13 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:00:13.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:13 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:00:13.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:13 smithi045 ceph-mon[31773]: osdmap e131: 12 total, 12 up, 12 in 2024-03-28T18:00:13.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:13 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2233601202,v1:172.21.15.45:6837/2233601202] up:boot 2024-03-28T18:00:13.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:13 smithi045 ceph-mon[31773]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-03-28T18:00:13.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:13 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:00:13.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:13 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:13.918 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 466 sec latency 581815.532 ms 2024-03-28T18:00:14.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:13 smithi071 ceph-mon[36597]: pgmap v382: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 85 B/s wr, 11 op/s 2024-03-28T18:00:14.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:13 smithi071 ceph-mon[36597]: daemon mds.g restarted 2024-03-28T18:00:14.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:13 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:00:14.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:13 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:00:14.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:13 smithi071 ceph-mon[36597]: osdmap e131: 12 total, 12 up, 12 in 2024-03-28T18:00:14.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:13 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2233601202,v1:172.21.15.45:6837/2233601202] up:boot 2024-03-28T18:00:14.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:13 smithi071 ceph-mon[36597]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-03-28T18:00:14.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:13 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:00:14.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:13 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:13 smithi151 ceph-mon[36481]: pgmap v382: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 85 B/s wr, 11 op/s 2024-03-28T18:00:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:13 smithi151 ceph-mon[36481]: daemon mds.g restarted 2024-03-28T18:00:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:13 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:00:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:13 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:00:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:13 smithi151 ceph-mon[36481]: osdmap e131: 12 total, 12 up, 12 in 2024-03-28T18:00:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:13 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2233601202,v1:172.21.15.45:6837/2233601202] up:boot 2024-03-28T18:00:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:13 smithi151 ceph-mon[36481]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-03-28T18:00:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:13 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:00:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:13 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:14.919 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 467 sec latency 582815.671 ms 2024-03-28T18:00:15.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:15 smithi045 ceph-mon[31773]: pgmap v384: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-28T18:00:15.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:15 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/333952790,v1:172.21.15.45:6835/333952790] up:resolve 2024-03-28T18:00:15.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:15 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:15.918 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 468 sec latency 583815.779 ms 2024-03-28T18:00:16.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:15 smithi071 ceph-mon[36597]: pgmap v384: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-28T18:00:16.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:15 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/333952790,v1:172.21.15.45:6835/333952790] up:resolve 2024-03-28T18:00:16.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:15 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:16.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:15 smithi151 ceph-mon[36481]: pgmap v384: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-28T18:00:16.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:15 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/333952790,v1:172.21.15.45:6835/333952790] up:resolve 2024-03-28T18:00:16.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:15 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:16.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:16 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:00:16.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:16 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/333952790,v1:172.21.15.45:6835/333952790] up:reconnect 2024-03-28T18:00:16.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:16 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:16.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:16 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T18:00:16.918 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 469 sec latency 584815.902 ms 2024-03-28T18:00:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:16 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:00:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:16 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/333952790,v1:172.21.15.45:6835/333952790] up:reconnect 2024-03-28T18:00:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:16 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:16 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T18:00:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:16 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:00:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:16 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/333952790,v1:172.21.15.45:6835/333952790] up:reconnect 2024-03-28T18:00:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:16 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:16 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T18:00:17.705 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-mon[31773]: pgmap v385: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T18:00:17.705 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/333952790,v1:172.21.15.45:6835/333952790] up:rejoin 2024-03-28T18:00:17.705 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:17.705 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-mon[31773]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-03-28T18:00:17.706 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0f44800700 time 2024-03-28T18:00:17.665905+0000 2024-03-28T18:00:17.706 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:00:17.706 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:17.706 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0f5016a4a1] 2024-03-28T18:00:17.706 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f5016a667] 2024-03-28T18:00:17.706 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c446c53223] 2024-03-28T18:00:17.706 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c446acd5d6] 2024-03-28T18:00:17.706 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c446acdda1] 2024-03-28T18:00:17.706 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c446ad02d5] 2024-03-28T18:00:17.706 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c446b20233] 2024-03-28T18:00:17.706 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c446b22197] 2024-03-28T18:00:17.706 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c446a2111b] 2024-03-28T18:00:17.706 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c446a216bc] 2024-03-28T18:00:17.707 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c446a0b1ff] 2024-03-28T18:00:17.707 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f503fc488] 2024-03-28T18:00:17.707 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 13: (DispatchQueue::entry()+0x51f) [0x7f0f503f960f] 2024-03-28T18:00:17.707 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f504cce81] 2024-03-28T18:00:17.707 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0f4eee71ca] 2024-03-28T18:00:17.707 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 16: clone() 2024-03-28T18:00:17.707 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 2024-03-28T18:00:17.667+0000 7f0f44800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0f44800700 time 2024-03-28T18:00:17.665905+0000 2024-03-28T18:00:17.708 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:00:17.708 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 2024-03-28T18:00:17.708 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:17.708 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0f5016a4a1] 2024-03-28T18:00:17.708 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f5016a667] 2024-03-28T18:00:17.708 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c446c53223] 2024-03-28T18:00:17.708 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c446acd5d6] 2024-03-28T18:00:17.708 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c446acdda1] 2024-03-28T18:00:17.708 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c446ad02d5] 2024-03-28T18:00:17.708 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c446b20233] 2024-03-28T18:00:17.708 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c446b22197] 2024-03-28T18:00:17.708 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c446a2111b] 2024-03-28T18:00:17.709 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c446a216bc] 2024-03-28T18:00:17.709 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c446a0b1ff] 2024-03-28T18:00:17.709 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f503fc488] 2024-03-28T18:00:17.709 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 13: (DispatchQueue::entry()+0x51f) [0x7f0f503f960f] 2024-03-28T18:00:17.709 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f504cce81] 2024-03-28T18:00:17.709 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0f4eee71ca] 2024-03-28T18:00:17.709 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 16: clone() 2024-03-28T18:00:17.709 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 2024-03-28T18:00:17.709 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: *** Caught signal (Aborted) ** 2024-03-28T18:00:17.709 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: in thread 7f0f44800700 thread_name:ms_dispatch 2024-03-28T18:00:17.709 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:17.709 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0f4eef1d20] 2024-03-28T18:00:17.710 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 2: gsignal() 2024-03-28T18:00:17.710 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 3: abort() 2024-03-28T18:00:17.710 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0f5016a4fb] 2024-03-28T18:00:17.710 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f5016a667] 2024-03-28T18:00:17.710 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c446c53223] 2024-03-28T18:00:17.710 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c446acd5d6] 2024-03-28T18:00:17.710 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c446acdda1] 2024-03-28T18:00:17.710 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c446ad02d5] 2024-03-28T18:00:17.710 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c446b20233] 2024-03-28T18:00:17.710 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c446b22197] 2024-03-28T18:00:17.710 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c446a2111b] 2024-03-28T18:00:17.710 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c446a216bc] 2024-03-28T18:00:17.710 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c446a0b1ff] 2024-03-28T18:00:17.710 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f503fc488] 2024-03-28T18:00:17.710 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 16: (DispatchQueue::entry()+0x51f) [0x7f0f503f960f] 2024-03-28T18:00:17.711 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f504cce81] 2024-03-28T18:00:17.711 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0f4eee71ca] 2024-03-28T18:00:17.711 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 19: clone() 2024-03-28T18:00:17.711 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 2024-03-28T18:00:17.670+0000 7f0f44800700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:00:17.711 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: in thread 7f0f44800700 thread_name:ms_dispatch 2024-03-28T18:00:17.711 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 2024-03-28T18:00:17.711 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:17.711 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0f4eef1d20] 2024-03-28T18:00:17.711 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 2: gsignal() 2024-03-28T18:00:17.711 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 3: abort() 2024-03-28T18:00:17.711 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0f5016a4fb] 2024-03-28T18:00:17.711 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f5016a667] 2024-03-28T18:00:17.712 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c446c53223] 2024-03-28T18:00:17.712 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c446acd5d6] 2024-03-28T18:00:17.712 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c446acdda1] 2024-03-28T18:00:17.712 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c446ad02d5] 2024-03-28T18:00:17.712 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c446b20233] 2024-03-28T18:00:17.712 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c446b22197] 2024-03-28T18:00:17.712 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c446a2111b] 2024-03-28T18:00:17.712 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c446a216bc] 2024-03-28T18:00:17.712 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c446a0b1ff] 2024-03-28T18:00:17.713 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f503fc488] 2024-03-28T18:00:17.713 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 16: (DispatchQueue::entry()+0x51f) [0x7f0f503f960f] 2024-03-28T18:00:17.713 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f504cce81] 2024-03-28T18:00:17.713 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0f4eee71ca] 2024-03-28T18:00:17.713 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 19: clone() 2024-03-28T18:00:17.713 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:00:17.713 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 2024-03-28T18:00:17.713 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: -1> 2024-03-28T18:00:17.667+0000 7f0f44800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0f44800700 time 2024-03-28T18:00:17.665905+0000 2024-03-28T18:00:17.713 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:00:17.713 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 2024-03-28T18:00:17.713 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:17.713 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0f5016a4a1] 2024-03-28T18:00:17.713 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f5016a667] 2024-03-28T18:00:17.713 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c446c53223] 2024-03-28T18:00:17.713 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c446acd5d6] 2024-03-28T18:00:17.714 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c446acdda1] 2024-03-28T18:00:17.714 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c446ad02d5] 2024-03-28T18:00:17.714 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c446b20233] 2024-03-28T18:00:17.714 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c446b22197] 2024-03-28T18:00:17.714 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c446a2111b] 2024-03-28T18:00:17.714 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c446a216bc] 2024-03-28T18:00:17.714 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c446a0b1ff] 2024-03-28T18:00:17.714 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f503fc488] 2024-03-28T18:00:17.714 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 13: (DispatchQueue::entry()+0x51f) [0x7f0f503f960f] 2024-03-28T18:00:17.714 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f504cce81] 2024-03-28T18:00:17.714 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0f4eee71ca] 2024-03-28T18:00:17.714 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 16: clone() 2024-03-28T18:00:17.714 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 2024-03-28T18:00:17.714 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 0> 2024-03-28T18:00:17.670+0000 7f0f44800700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:00:17.715 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: in thread 7f0f44800700 thread_name:ms_dispatch 2024-03-28T18:00:17.715 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 2024-03-28T18:00:17.715 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:17.715 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0f4eef1d20] 2024-03-28T18:00:17.715 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 2: gsignal() 2024-03-28T18:00:17.715 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 3: abort() 2024-03-28T18:00:17.715 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0f5016a4fb] 2024-03-28T18:00:17.715 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f5016a667] 2024-03-28T18:00:17.715 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c446c53223] 2024-03-28T18:00:17.715 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c446acd5d6] 2024-03-28T18:00:17.715 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c446acdda1] 2024-03-28T18:00:17.715 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c446ad02d5] 2024-03-28T18:00:17.715 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c446b20233] 2024-03-28T18:00:17.715 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c446b22197] 2024-03-28T18:00:17.716 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c446a2111b] 2024-03-28T18:00:17.716 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c446a216bc] 2024-03-28T18:00:17.716 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c446a0b1ff] 2024-03-28T18:00:17.716 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f503fc488] 2024-03-28T18:00:17.716 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 16: (DispatchQueue::entry()+0x51f) [0x7f0f503f960f] 2024-03-28T18:00:17.716 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f504cce81] 2024-03-28T18:00:17.716 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0f4eee71ca] 2024-03-28T18:00:17.716 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 19: clone() 2024-03-28T18:00:17.716 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:00:17.716 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 2024-03-28T18:00:17.919 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 470 sec latency 585816.023 ms 2024-03-28T18:00:18.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:17 smithi071 ceph-mon[36597]: pgmap v385: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T18:00:18.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:17 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/333952790,v1:172.21.15.45:6835/333952790] up:rejoin 2024-03-28T18:00:18.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:17 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:18.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:17 smithi071 ceph-mon[36597]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-03-28T18:00:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:17 smithi151 ceph-mon[36481]: pgmap v385: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T18:00:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:17 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/333952790,v1:172.21.15.45:6835/333952790] up:rejoin 2024-03-28T18:00:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:17 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:17 smithi151 ceph-mon[36481]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-03-28T18:00:18.159 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: -1> 2024-03-28T18:00:17.667+0000 7f0f44800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0f44800700 time 2024-03-28T18:00:17.665905+0000 2024-03-28T18:00:18.159 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:00:18.159 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 2024-03-28T18:00:18.159 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:18.159 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0f5016a4a1] 2024-03-28T18:00:18.159 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f5016a667] 2024-03-28T18:00:18.159 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c446c53223] 2024-03-28T18:00:18.159 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c446acd5d6] 2024-03-28T18:00:18.159 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c446acdda1] 2024-03-28T18:00:18.160 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c446ad02d5] 2024-03-28T18:00:18.160 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c446b20233] 2024-03-28T18:00:18.160 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c446b22197] 2024-03-28T18:00:18.160 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c446a2111b] 2024-03-28T18:00:18.160 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c446a216bc] 2024-03-28T18:00:18.160 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c446a0b1ff] 2024-03-28T18:00:18.160 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f503fc488] 2024-03-28T18:00:18.160 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 13: (DispatchQueue::entry()+0x51f) [0x7f0f503f960f] 2024-03-28T18:00:18.160 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f504cce81] 2024-03-28T18:00:18.160 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0f4eee71ca] 2024-03-28T18:00:18.160 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 16: clone() 2024-03-28T18:00:18.160 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 2024-03-28T18:00:18.160 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 0> 2024-03-28T18:00:17.670+0000 7f0f44800700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:00:18.161 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: in thread 7f0f44800700 thread_name:ms_dispatch 2024-03-28T18:00:18.161 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 2024-03-28T18:00:18.161 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:18.161 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0f4eef1d20] 2024-03-28T18:00:18.161 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 2: gsignal() 2024-03-28T18:00:18.161 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 3: abort() 2024-03-28T18:00:18.161 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0f5016a4fb] 2024-03-28T18:00:18.161 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f5016a667] 2024-03-28T18:00:18.161 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c446c53223] 2024-03-28T18:00:18.162 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c446acd5d6] 2024-03-28T18:00:18.162 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c446acdda1] 2024-03-28T18:00:18.162 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c446ad02d5] 2024-03-28T18:00:18.162 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c446b20233] 2024-03-28T18:00:18.162 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c446b22197] 2024-03-28T18:00:18.162 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c446a2111b] 2024-03-28T18:00:18.162 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c446a216bc] 2024-03-28T18:00:18.162 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c446a0b1ff] 2024-03-28T18:00:18.162 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f503fc488] 2024-03-28T18:00:18.162 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 16: (DispatchQueue::entry()+0x51f) [0x7f0f503f960f] 2024-03-28T18:00:18.162 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f504cce81] 2024-03-28T18:00:18.162 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0f4eee71ca] 2024-03-28T18:00:18.162 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 19: clone() 2024-03-28T18:00:18.162 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:00:18.163 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[72425]: 2024-03-28T18:00:18.163 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:17 smithi045 podman[93937]: 2024-03-28 18:00:17.789900874 +0000 UTC m=+0.033426009 container died 5768fb00f3eaf9dfef9cf13875ae9a09bb016a213aeb0bd17d960838939136a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-03-28T18:00:18.640 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:18 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:00:18.640 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:18 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/333952790,v1:172.21.15.45:6835/333952790] up:active 2024-03-28T18:00:18.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:18 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:00:18.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:18 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/333952790,v1:172.21.15.45:6835/333952790] up:active 2024-03-28T18:00:18.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:18 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:18.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:18 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 3 up:standby-replay 2024-03-28T18:00:18.916 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:18 smithi045 podman[93937]: 2024-03-28 18:00:18.510217221 +0000 UTC m=+0.753742426 container remove 5768fb00f3eaf9dfef9cf13875ae9a09bb016a213aeb0bd17d960838939136a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-28T18:00:18.916 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:18 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:00:18.919 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 471 sec latency 586816.095 ms 2024-03-28T18:00:19.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:18 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:19.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:18 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 3 up:standby-replay 2024-03-28T18:00:19.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:18 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:00:19.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:18 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/333952790,v1:172.21.15.45:6835/333952790] up:active 2024-03-28T18:00:19.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:18 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:19.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:18 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 3 up:standby-replay 2024-03-28T18:00:19.416 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:19 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-28T18:00:19.416 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:19 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Consumed 2.086s CPU time. 2024-03-28T18:00:19.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:19 smithi045 ceph-mon[31773]: pgmap v386: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T18:00:19.919 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 472 sec latency 587816.198 ms 2024-03-28T18:00:20.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:19 smithi071 ceph-mon[36597]: pgmap v386: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T18:00:20.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:19 smithi151 ceph-mon[36481]: pgmap v386: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-28T18:00:20.919 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 473 sec latency 588816.337 ms 2024-03-28T18:00:21.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:21 smithi045 ceph-mon[31773]: pgmap v387: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-28T18:00:21.919 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 474 sec latency 589816.469 ms 2024-03-28T18:00:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:21 smithi071 ceph-mon[36597]: pgmap v387: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-28T18:00:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:21 smithi151 ceph-mon[36481]: pgmap v387: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-28T18:00:22.919 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 475 sec latency 590816.626 ms 2024-03-28T18:00:23.922 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 476 sec latency 591816.805 ms 2024-03-28T18:00:24.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:23 smithi071 ceph-mon[36597]: pgmap v388: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T18:00:24.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:23 smithi151 ceph-mon[36481]: pgmap v388: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T18:00:24.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:23 smithi045 ceph-mon[31773]: pgmap v388: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T18:00:24.920 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 477 sec latency 592816.966 ms 2024-03-28T18:00:25.920 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 478 sec latency 593817.124 ms 2024-03-28T18:00:26.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:25 smithi071 ceph-mon[36597]: pgmap v389: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-03-28T18:00:26.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:25 smithi151 ceph-mon[36481]: pgmap v389: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-03-28T18:00:26.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:25 smithi045 ceph-mon[31773]: pgmap v389: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-03-28T18:00:26.920 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 479 sec latency 594817.240 ms 2024-03-28T18:00:27.920 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 480 sec latency 595817.394 ms 2024-03-28T18:00:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:27 smithi071 ceph-mon[36597]: pgmap v390: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-28T18:00:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:27 smithi151 ceph-mon[36481]: pgmap v390: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-28T18:00:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:27 smithi045 ceph-mon[31773]: pgmap v390: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-28T18:00:28.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:28 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:00:28.920 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 481 sec latency 596817.518 ms 2024-03-28T18:00:29.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:28 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:00:29.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:28 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:00:29.569 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:29 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 1. 2024-03-28T18:00:29.569 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:29 smithi045 systemd[1]: Stopped Ceph mds.j for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:00:29.570 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:29 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Consumed 2.086s CPU time. 2024-03-28T18:00:29.570 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:29 smithi045 systemd[1]: Starting Ceph mds.j for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T18:00:29.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:29 smithi045 ceph-mon[31773]: pgmap v391: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T18:00:29.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:29 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:00:29.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:29 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:00:29.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:29 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:00:29.882 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:29 smithi045 podman[94129]: 2024-03-28 18:00:29.474389909 +0000 UTC m=+0.038507793 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:00:29.882 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:29 smithi045 podman[94129]: 2024-03-28 18:00:29.616167457 +0000 UTC m=+0.180285324 container create af47547fe6999d32b3880a614e0e4bdf3ddfbad506059f9963821cec3cf1e373 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-28T18:00:29.882 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:29 smithi045 podman[94129]: 2024-03-28 18:00:29.881473039 +0000 UTC m=+0.445590989 container init af47547fe6999d32b3880a614e0e4bdf3ddfbad506059f9963821cec3cf1e373 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-28T18:00:29.920 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 482 sec latency 597817.607 ms 2024-03-28T18:00:30.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:29 smithi071 ceph-mon[36597]: pgmap v391: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T18:00:30.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:29 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:00:30.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:29 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:00:30.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:29 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:00:30.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:29 smithi151 ceph-mon[36481]: pgmap v391: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T18:00:30.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:29 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:00:30.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:29 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:00:30.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:29 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:00:30.166 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:29 smithi045 podman[94129]: 2024-03-28 18:00:29.892982181 +0000 UTC m=+0.457100059 container start af47547fe6999d32b3880a614e0e4bdf3ddfbad506059f9963821cec3cf1e373 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:00:30.166 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:29 smithi045 ceph-mds[94180]: -- 172.21.15.45:0/1665833440 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5590bb853a40 con 0x5590bb827000 2024-03-28T18:00:30.166 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:29 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: starting mds.j at 2024-03-28T18:00:30.166 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:29 smithi045 bash[94129]: af47547fe6999d32b3880a614e0e4bdf3ddfbad506059f9963821cec3cf1e373 2024-03-28T18:00:30.166 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:30 smithi045 systemd[1]: Started Ceph mds.j for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:00:30.920 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 483 sec latency 598817.774 ms 2024-03-28T18:00:31.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:30 smithi071 ceph-mon[36597]: pgmap v392: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T18:00:31.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:30 smithi071 ceph-mon[36597]: pgmap v393: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 22 op/s 2024-03-28T18:00:31.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:30 smithi071 ceph-mon[36597]: daemon mds.j restarted 2024-03-28T18:00:31.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:30 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:00:31.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:30 smithi071 ceph-mon[36597]: osdmap e132: 12 total, 12 up, 12 in 2024-03-28T18:00:31.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:30 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/3392572128,v1:172.21.15.45:6835/3392572128] up:boot 2024-03-28T18:00:31.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:30 smithi071 ceph-mon[36597]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-03-28T18:00:31.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:30 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:00:31.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:30 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:00:31.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:30 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:31.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:30 smithi151 ceph-mon[36481]: pgmap v392: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T18:00:31.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:30 smithi151 ceph-mon[36481]: pgmap v393: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 22 op/s 2024-03-28T18:00:31.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:30 smithi151 ceph-mon[36481]: daemon mds.j restarted 2024-03-28T18:00:31.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:30 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:00:31.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:30 smithi151 ceph-mon[36481]: osdmap e132: 12 total, 12 up, 12 in 2024-03-28T18:00:31.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:30 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/3392572128,v1:172.21.15.45:6835/3392572128] up:boot 2024-03-28T18:00:31.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:30 smithi151 ceph-mon[36481]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-03-28T18:00:31.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:30 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:00:31.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:30 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:00:31.075 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:30 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:30 smithi045 ceph-mon[31773]: pgmap v392: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T18:00:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:30 smithi045 ceph-mon[31773]: pgmap v393: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 22 op/s 2024-03-28T18:00:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:30 smithi045 ceph-mon[31773]: daemon mds.j restarted 2024-03-28T18:00:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:30 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:00:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:30 smithi045 ceph-mon[31773]: osdmap e132: 12 total, 12 up, 12 in 2024-03-28T18:00:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:30 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/3392572128,v1:172.21.15.45:6835/3392572128] up:boot 2024-03-28T18:00:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:30 smithi045 ceph-mon[31773]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-03-28T18:00:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:30 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:00:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:30 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:00:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:30 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:31.921 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 484 sec latency 599817.868 ms 2024-03-28T18:00:32.921 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 485 sec latency 600817.967 ms 2024-03-28T18:00:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:32 smithi071 ceph-mon[36597]: pgmap v395: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 271 B/s wr, 8 op/s 2024-03-28T18:00:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:32 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2233601202,v1:172.21.15.45:6837/2233601202] up:resolve 2024-03-28T18:00:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:32 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:33.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:32 smithi151 ceph-mon[36481]: pgmap v395: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 271 B/s wr, 8 op/s 2024-03-28T18:00:33.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:32 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2233601202,v1:172.21.15.45:6837/2233601202] up:resolve 2024-03-28T18:00:33.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:32 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:33.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:32 smithi045 ceph-mon[31773]: pgmap v395: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 271 B/s wr, 8 op/s 2024-03-28T18:00:33.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:32 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2233601202,v1:172.21.15.45:6837/2233601202] up:resolve 2024-03-28T18:00:33.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:32 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:33.921 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 486 sec latency 601818.136 ms 2024-03-28T18:00:34.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:33 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:00:34.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:33 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2233601202,v1:172.21.15.45:6837/2233601202] up:reconnect 2024-03-28T18:00:34.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:33 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:34.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:33 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T18:00:34.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:33 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:00:34.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:33 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2233601202,v1:172.21.15.45:6837/2233601202] up:reconnect 2024-03-28T18:00:34.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:33 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:34.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:33 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T18:00:34.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:33 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:00:34.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:33 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2233601202,v1:172.21.15.45:6837/2233601202] up:reconnect 2024-03-28T18:00:34.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:33 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:34.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:33 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T18:00:34.921 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 487 sec latency 602818.243 ms 2024-03-28T18:00:35.060 INFO:journalctl@ceph.mds.b.smithi071.stdout:Mar 28 18:00:34 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-b[66900]: 2024-03-28T18:00:34.928+0000 7ff9f838b700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-28T18:00:35.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:34 smithi071 ceph-mon[36597]: pgmap v396: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 135 B/s wr, 4 op/s 2024-03-28T18:00:35.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:34 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2233601202,v1:172.21.15.45:6837/2233601202] up:rejoin 2024-03-28T18:00:35.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:34 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:35.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:34 smithi071 ceph-mon[36597]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-03-28T18:00:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:34 smithi151 ceph-mon[36481]: pgmap v396: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 135 B/s wr, 4 op/s 2024-03-28T18:00:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:34 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2233601202,v1:172.21.15.45:6837/2233601202] up:rejoin 2024-03-28T18:00:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:34 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:34 smithi151 ceph-mon[36481]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-03-28T18:00:35.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-mon[31773]: pgmap v396: 97 pgs: 97 active+clean; 27 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 135 B/s wr, 4 op/s 2024-03-28T18:00:35.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2233601202,v1:172.21.15.45:6837/2233601202] up:rejoin 2024-03-28T18:00:35.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:35.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-mon[31773]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-03-28T18:00:35.167 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fdbde200700 time 2024-03-28T18:00:34.773079+0000 2024-03-28T18:00:35.167 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:00:35.167 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:35.167 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fdbe9b994a1] 2024-03-28T18:00:35.167 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdbe9b99667] 2024-03-28T18:00:35.167 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559156835223] 2024-03-28T18:00:35.167 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5591566af5d6] 2024-03-28T18:00:35.167 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5591566afda1] 2024-03-28T18:00:35.167 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5591566b22d5] 2024-03-28T18:00:35.167 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559156702233] 2024-03-28T18:00:35.167 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559156704197] 2024-03-28T18:00:35.167 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55915660311b] 2024-03-28T18:00:35.167 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5591566036bc] 2024-03-28T18:00:35.168 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5591565ed1ff] 2024-03-28T18:00:35.168 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdbe9e2b488] 2024-03-28T18:00:35.168 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 13: (DispatchQueue::entry()+0x51f) [0x7fdbe9e2860f] 2024-03-28T18:00:35.168 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdbe9efbe81] 2024-03-28T18:00:35.168 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fdbe89161ca] 2024-03-28T18:00:35.168 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 16: clone() 2024-03-28T18:00:35.168 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 2024-03-28T18:00:34.774+0000 7fdbde200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fdbde200700 time 2024-03-28T18:00:34.773079+0000 2024-03-28T18:00:35.168 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:00:35.169 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 2024-03-28T18:00:35.169 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:35.169 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fdbe9b994a1] 2024-03-28T18:00:35.169 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdbe9b99667] 2024-03-28T18:00:35.169 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559156835223] 2024-03-28T18:00:35.169 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5591566af5d6] 2024-03-28T18:00:35.169 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5591566afda1] 2024-03-28T18:00:35.169 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5591566b22d5] 2024-03-28T18:00:35.169 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559156702233] 2024-03-28T18:00:35.169 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559156704197] 2024-03-28T18:00:35.169 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55915660311b] 2024-03-28T18:00:35.169 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5591566036bc] 2024-03-28T18:00:35.169 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5591565ed1ff] 2024-03-28T18:00:35.169 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdbe9e2b488] 2024-03-28T18:00:35.170 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 13: (DispatchQueue::entry()+0x51f) [0x7fdbe9e2860f] 2024-03-28T18:00:35.170 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdbe9efbe81] 2024-03-28T18:00:35.170 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fdbe89161ca] 2024-03-28T18:00:35.170 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 16: clone() 2024-03-28T18:00:35.170 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 2024-03-28T18:00:35.170 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: *** Caught signal (Aborted) ** 2024-03-28T18:00:35.170 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: in thread 7fdbde200700 thread_name:ms_dispatch 2024-03-28T18:00:35.170 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:35.170 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fdbe8920d20] 2024-03-28T18:00:35.170 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 2: gsignal() 2024-03-28T18:00:35.171 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 3: abort() 2024-03-28T18:00:35.171 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fdbe9b994fb] 2024-03-28T18:00:35.171 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdbe9b99667] 2024-03-28T18:00:35.171 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559156835223] 2024-03-28T18:00:35.171 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5591566af5d6] 2024-03-28T18:00:35.171 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5591566afda1] 2024-03-28T18:00:35.171 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5591566b22d5] 2024-03-28T18:00:35.171 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559156702233] 2024-03-28T18:00:35.171 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559156704197] 2024-03-28T18:00:35.171 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55915660311b] 2024-03-28T18:00:35.171 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5591566036bc] 2024-03-28T18:00:35.171 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5591565ed1ff] 2024-03-28T18:00:35.171 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdbe9e2b488] 2024-03-28T18:00:35.171 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 16: (DispatchQueue::entry()+0x51f) [0x7fdbe9e2860f] 2024-03-28T18:00:35.172 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdbe9efbe81] 2024-03-28T18:00:35.172 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fdbe89161ca] 2024-03-28T18:00:35.172 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 19: clone() 2024-03-28T18:00:35.172 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 2024-03-28T18:00:34.777+0000 7fdbde200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:00:35.172 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: in thread 7fdbde200700 thread_name:ms_dispatch 2024-03-28T18:00:35.172 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 2024-03-28T18:00:35.172 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:35.172 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fdbe8920d20] 2024-03-28T18:00:35.172 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 2: gsignal() 2024-03-28T18:00:35.172 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 3: abort() 2024-03-28T18:00:35.172 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fdbe9b994fb] 2024-03-28T18:00:35.172 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdbe9b99667] 2024-03-28T18:00:35.173 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559156835223] 2024-03-28T18:00:35.173 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5591566af5d6] 2024-03-28T18:00:35.173 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5591566afda1] 2024-03-28T18:00:35.173 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5591566b22d5] 2024-03-28T18:00:35.173 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559156702233] 2024-03-28T18:00:35.173 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559156704197] 2024-03-28T18:00:35.173 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55915660311b] 2024-03-28T18:00:35.173 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5591566036bc] 2024-03-28T18:00:35.173 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5591565ed1ff] 2024-03-28T18:00:35.173 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdbe9e2b488] 2024-03-28T18:00:35.173 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 16: (DispatchQueue::entry()+0x51f) [0x7fdbe9e2860f] 2024-03-28T18:00:35.173 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdbe9efbe81] 2024-03-28T18:00:35.174 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fdbe89161ca] 2024-03-28T18:00:35.174 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 19: clone() 2024-03-28T18:00:35.174 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:00:35.174 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 2024-03-28T18:00:35.174 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: -1> 2024-03-28T18:00:34.774+0000 7fdbde200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fdbde200700 time 2024-03-28T18:00:34.773079+0000 2024-03-28T18:00:35.174 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:00:35.174 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 2024-03-28T18:00:35.174 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:35.174 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fdbe9b994a1] 2024-03-28T18:00:35.174 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdbe9b99667] 2024-03-28T18:00:35.175 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559156835223] 2024-03-28T18:00:35.175 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5591566af5d6] 2024-03-28T18:00:35.175 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5591566afda1] 2024-03-28T18:00:35.175 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5591566b22d5] 2024-03-28T18:00:35.175 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559156702233] 2024-03-28T18:00:35.175 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559156704197] 2024-03-28T18:00:35.175 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55915660311b] 2024-03-28T18:00:35.175 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5591566036bc] 2024-03-28T18:00:35.176 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5591565ed1ff] 2024-03-28T18:00:35.176 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdbe9e2b488] 2024-03-28T18:00:35.176 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 13: (DispatchQueue::entry()+0x51f) [0x7fdbe9e2860f] 2024-03-28T18:00:35.176 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdbe9efbe81] 2024-03-28T18:00:35.176 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fdbe89161ca] 2024-03-28T18:00:35.176 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 16: clone() 2024-03-28T18:00:35.176 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 2024-03-28T18:00:35.176 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 0> 2024-03-28T18:00:34.777+0000 7fdbde200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:00:35.176 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: in thread 7fdbde200700 thread_name:ms_dispatch 2024-03-28T18:00:35.176 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 2024-03-28T18:00:35.176 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:35.176 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fdbe8920d20] 2024-03-28T18:00:35.177 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 2: gsignal() 2024-03-28T18:00:35.177 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 3: abort() 2024-03-28T18:00:35.177 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fdbe9b994fb] 2024-03-28T18:00:35.177 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdbe9b99667] 2024-03-28T18:00:35.177 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559156835223] 2024-03-28T18:00:35.177 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5591566af5d6] 2024-03-28T18:00:35.177 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5591566afda1] 2024-03-28T18:00:35.177 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5591566b22d5] 2024-03-28T18:00:35.177 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559156702233] 2024-03-28T18:00:35.177 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559156704197] 2024-03-28T18:00:35.177 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55915660311b] 2024-03-28T18:00:35.177 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5591566036bc] 2024-03-28T18:00:35.177 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5591565ed1ff] 2024-03-28T18:00:35.177 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdbe9e2b488] 2024-03-28T18:00:35.178 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 16: (DispatchQueue::entry()+0x51f) [0x7fdbe9e2860f] 2024-03-28T18:00:35.178 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdbe9efbe81] 2024-03-28T18:00:35.178 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fdbe89161ca] 2024-03-28T18:00:35.178 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 19: clone() 2024-03-28T18:00:35.178 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:00:35.178 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 2024-03-28T18:00:35.178 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: -1> 2024-03-28T18:00:34.774+0000 7fdbde200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fdbde200700 time 2024-03-28T18:00:34.773079+0000 2024-03-28T18:00:35.178 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:00:35.178 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 2024-03-28T18:00:35.178 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:35.178 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fdbe9b994a1] 2024-03-28T18:00:35.178 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdbe9b99667] 2024-03-28T18:00:35.178 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559156835223] 2024-03-28T18:00:35.178 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5591566af5d6] 2024-03-28T18:00:35.179 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5591566afda1] 2024-03-28T18:00:35.179 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5591566b22d5] 2024-03-28T18:00:35.179 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559156702233] 2024-03-28T18:00:35.179 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559156704197] 2024-03-28T18:00:35.179 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55915660311b] 2024-03-28T18:00:35.179 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5591566036bc] 2024-03-28T18:00:35.179 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5591565ed1ff] 2024-03-28T18:00:35.179 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdbe9e2b488] 2024-03-28T18:00:35.179 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 13: (DispatchQueue::entry()+0x51f) [0x7fdbe9e2860f] 2024-03-28T18:00:35.180 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdbe9efbe81] 2024-03-28T18:00:35.180 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fdbe89161ca] 2024-03-28T18:00:35.180 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 16: clone() 2024-03-28T18:00:35.180 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 2024-03-28T18:00:35.180 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 0> 2024-03-28T18:00:34.777+0000 7fdbde200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:00:35.180 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: in thread 7fdbde200700 thread_name:ms_dispatch 2024-03-28T18:00:35.180 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 2024-03-28T18:00:35.180 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:35.180 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fdbe8920d20] 2024-03-28T18:00:35.180 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 2: gsignal() 2024-03-28T18:00:35.180 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 3: abort() 2024-03-28T18:00:35.180 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fdbe9b994fb] 2024-03-28T18:00:35.180 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdbe9b99667] 2024-03-28T18:00:35.180 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559156835223] 2024-03-28T18:00:35.181 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5591566af5d6] 2024-03-28T18:00:35.181 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5591566afda1] 2024-03-28T18:00:35.181 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5591566b22d5] 2024-03-28T18:00:35.181 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559156702233] 2024-03-28T18:00:35.181 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559156704197] 2024-03-28T18:00:35.181 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55915660311b] 2024-03-28T18:00:35.181 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5591566036bc] 2024-03-28T18:00:35.181 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5591565ed1ff] 2024-03-28T18:00:35.181 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdbe9e2b488] 2024-03-28T18:00:35.181 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 16: (DispatchQueue::entry()+0x51f) [0x7fdbe9e2860f] 2024-03-28T18:00:35.181 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdbe9efbe81] 2024-03-28T18:00:35.181 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fdbe89161ca] 2024-03-28T18:00:35.181 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 19: clone() 2024-03-28T18:00:35.181 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:00:35.181 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[93889]: 2024-03-28T18:00:35.181 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:34 smithi045 podman[94214]: 2024-03-28 18:00:34.869275701 +0000 UTC m=+0.033591477 container died 09caf3eb991f701df802715b4403cc12179bec1bba098facb75328323dd44fc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-28T18:00:35.921 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 488 sec latency 603818.317 ms 2024-03-28T18:00:36.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:35 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:00:36.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:35 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2233601202,v1:172.21.15.45:6837/2233601202] up:active 2024-03-28T18:00:36.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:35 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:36.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:35 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 3 up:standby-replay 2024-03-28T18:00:36.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:35 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:00:36.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:35 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2233601202,v1:172.21.15.45:6837/2233601202] up:active 2024-03-28T18:00:36.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:35 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:36.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:35 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 3 up:standby-replay 2024-03-28T18:00:36.165 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:35 smithi045 podman[94214]: 2024-03-28 18:00:35.842877851 +0000 UTC m=+1.007193674 container remove 09caf3eb991f701df802715b4403cc12179bec1bba098facb75328323dd44fc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-28T18:00:36.165 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:35 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:00:36.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:35 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:00:36.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:35 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2233601202,v1:172.21.15.45:6837/2233601202] up:active 2024-03-28T18:00:36.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:35 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:36.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:35 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 3 up:standby-replay 2024-03-28T18:00:36.666 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:36 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-28T18:00:36.666 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:36 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Consumed 1.585s CPU time. 2024-03-28T18:00:36.921 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 489 sec latency 604818.437 ms 2024-03-28T18:00:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:36 smithi071 ceph-mon[36597]: pgmap v397: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 36 KiB/s wr, 18 op/s 2024-03-28T18:00:37.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:36 smithi151 ceph-mon[36481]: pgmap v397: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 36 KiB/s wr, 18 op/s 2024-03-28T18:00:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:36 smithi045 ceph-mon[31773]: pgmap v397: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 36 KiB/s wr, 18 op/s 2024-03-28T18:00:37.509 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:00:37.510 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:00:37.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.752+0000 7fd2e3400640 1 -- 172.21.15.45:0/1745204491 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2dc105890 msgr2=0x7fd2dc105c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:37.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.752+0000 7fd2e3400640 1 --2- 172.21.15.45:0/1745204491 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2dc105890 0x7fd2dc105c70 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc009cd0 tx=0x7fd2cc02c0e0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.753+0000 7fd2e3400640 1 -- 172.21.15.45:0/1745204491 shutdown_connections 2024-03-28T18:00:37.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.753+0000 7fd2e3400640 1 --2- 172.21.15.45:0/1745204491 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2dc0ffdc0 0x7fd2dc100270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.753+0000 7fd2e3400640 1 --2- 172.21.15.45:0/1745204491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2dc067c00 0x7fd2dc0ff880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.753+0000 7fd2e3400640 1 --2- 172.21.15.45:0/1745204491 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2dc105890 0x7fd2dc105c70 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.753+0000 7fd2e3400640 1 -- 172.21.15.45:0/1745204491 >> 172.21.15.45:0/1745204491 conn(0x7fd2dc0766f0 msgr2=0x7fd2dc0fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:37.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.753+0000 7fd2e3400640 1 -- 172.21.15.45:0/1745204491 shutdown_connections 2024-03-28T18:00:37.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.753+0000 7fd2e3400640 1 -- 172.21.15.45:0/1745204491 wait complete. 2024-03-28T18:00:37.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.754+0000 7fd2e3400640 1 Processor -- start 2024-03-28T18:00:37.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.754+0000 7fd2e3400640 1 -- start start 2024-03-28T18:00:37.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.754+0000 7fd2e3400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2dc067c00 0x7fd2dc197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:37.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.754+0000 7fd2e3400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2dc0ffdc0 0x7fd2dc198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:37.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.754+0000 7fd2e3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2dc105890 0x7fd2dc19c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:37.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.754+0000 7fd2e3400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc110960 con 0x7fd2dc105890 2024-03-28T18:00:37.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.754+0000 7fd2e3400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc1107e0 con 0x7fd2dc0ffdc0 2024-03-28T18:00:37.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.754+0000 7fd2e3400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc110ae0 con 0x7fd2dc067c00 2024-03-28T18:00:37.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.755+0000 7fd2dbe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2dc0ffdc0 0x7fd2dc198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:37.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.755+0000 7fd2dbe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2dc0ffdc0 0x7fd2dc198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:56976/0 (socket says 172.21.15.45:56976) 2024-03-28T18:00:37.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.755+0000 7fd2dbe00640 1 -- 172.21.15.45:0/1998533364 learned_addr learned my addr 172.21.15.45:0/1998533364 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:00:37.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.755+0000 7fd2e0a00640 1 --2- 172.21.15.45:0/1998533364 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2dc067c00 0x7fd2dc197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:37.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.755+0000 7fd2e1400640 1 --2- 172.21.15.45:0/1998533364 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2dc105890 0x7fd2dc19c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:37.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.755+0000 7fd2dbe00640 1 -- 172.21.15.45:0/1998533364 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2dc067c00 msgr2=0x7fd2dc197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:37.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.755+0000 7fd2dbe00640 1 --2- 172.21.15.45:0/1998533364 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2dc067c00 0x7fd2dc197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.755+0000 7fd2dbe00640 1 -- 172.21.15.45:0/1998533364 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2dc105890 msgr2=0x7fd2dc19c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:37.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.755+0000 7fd2dbe00640 1 --2- 172.21.15.45:0/1998533364 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2dc105890 0x7fd2dc19c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.755+0000 7fd2dbe00640 1 -- 172.21.15.45:0/1998533364 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2dc19cfc0 con 0x7fd2dc0ffdc0 2024-03-28T18:00:37.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.755+0000 7fd2e1400640 1 --2- 172.21.15.45:0/1998533364 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2dc105890 0x7fd2dc19c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:00:37.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.755+0000 7fd2dbe00640 1 --2- 172.21.15.45:0/1998533364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2dc0ffdc0 0x7fd2dc198510 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fd2c800e970 tx=0x7fd2c800ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:37.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.756+0000 7fd2d9600640 1 -- 172.21.15.45:0/1998533364 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c800cd80 con 0x7fd2dc0ffdc0 2024-03-28T18:00:37.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.756+0000 7fd2d9600640 1 -- 172.21.15.45:0/1998533364 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2c80035f0 con 0x7fd2dc0ffdc0 2024-03-28T18:00:37.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.756+0000 7fd2e3400640 1 -- 172.21.15.45:0/1998533364 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2dc19d2b0 con 0x7fd2dc0ffdc0 2024-03-28T18:00:37.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.756+0000 7fd2d9600640 1 -- 172.21.15.45:0/1998533364 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c8010690 con 0x7fd2dc0ffdc0 2024-03-28T18:00:37.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.756+0000 7fd2e3400640 1 -- 172.21.15.45:0/1998533364 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2dc1a4b40 con 0x7fd2dc0ffdc0 2024-03-28T18:00:37.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.759+0000 7fd2e61e3740 1 Processor -- start 2024-03-28T18:00:37.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.759+0000 7fd2e61e3740 1 -- start start 2024-03-28T18:00:37.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.760+0000 7fd2e61e3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9e3010 0x5574fb9e1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:37.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.760+0000 7fd2d9600640 1 -- 172.21.15.45:0/1998533364 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fd2c80037a0 con 0x7fd2dc0ffdc0 2024-03-28T18:00:37.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.760+0000 7fd2e61e3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574fb9e2330 0x5574fb9e27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:37.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.760+0000 7fd2e61e3740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5574fb9eb6a0 0x5574fb9eda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:37.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.760+0000 7fd2e61e3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574fb942250 con 0x5574fb9e2330 2024-03-28T18:00:37.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.760+0000 7fd2e61e3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574fb8eaa30 con 0x5574fb9e3010 2024-03-28T18:00:37.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.760+0000 7fd2e61e3740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574fb93e2f0 con 0x5574fb9eb6a0 2024-03-28T18:00:37.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.760+0000 7fd2e1400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9e3010 0x5574fb9e1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:37.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.760+0000 7fd2dbe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5574fb9eb6a0 0x5574fb9eda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:37.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.760+0000 7fd2e1400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9e3010 0x5574fb9e1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:56988/0 (socket says 172.21.15.45:56988) 2024-03-28T18:00:37.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.760+0000 7fd2e1400640 1 -- 172.21.15.45:0/3672768770 learned_addr learned my addr 172.21.15.45:0/3672768770 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:00:37.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.760+0000 7fd2e0a00640 1 --2- 172.21.15.45:0/3672768770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574fb9e2330 0x5574fb9e27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:37.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.760+0000 7fd2e1400640 1 -- 172.21.15.45:0/3672768770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5574fb9eb6a0 msgr2=0x5574fb9eda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:00:37.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.760+0000 7fd2e1400640 1 --2- 172.21.15.45:0/3672768770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5574fb9eb6a0 0x5574fb9eda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.760+0000 7fd2e1400640 1 -- 172.21.15.45:0/3672768770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574fb9e2330 msgr2=0x5574fb9e27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:00:37.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.760+0000 7fd2e1400640 1 --2- 172.21.15.45:0/3672768770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574fb9e2330 0x5574fb9e27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.761+0000 7fd2e1400640 1 -- 172.21.15.45:0/3672768770 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574fb8d5690 con 0x5574fb9e3010 2024-03-28T18:00:37.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.761+0000 7fd2e1400640 1 --2- 172.21.15.45:0/3672768770 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9e3010 0x5574fb9e1d60 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fd2d4009c20 tx=0x7fd2d40117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=401894ef9b52e413 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:37.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.761+0000 7fd2d1c00640 1 -- 172.21.15.45:0/3672768770 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4012450 con 0x5574fb9e3010 2024-03-28T18:00:37.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.761+0000 7fd2d1c00640 1 -- 172.21.15.45:0/3672768770 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2d4012ad0 con 0x5574fb9e3010 2024-03-28T18:00:37.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.761+0000 7fd2d1c00640 1 -- 172.21.15.45:0/3672768770 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d400fc10 con 0x5574fb9e3010 2024-03-28T18:00:37.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.761+0000 7fd2d9600640 1 --2- 172.21.15.45:0/1998533364 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd2c00a29f0 0x7fd2c00a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:37.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.762+0000 7fd2d9600640 1 -- 172.21.15.45:0/1998533364 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 11608+0+0 (secure 0 0 0) 0x7fd2c80cfd40 con 0x7fd2dc0ffdc0 2024-03-28T18:00:37.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.762+0000 7fd2e0a00640 1 --2- 172.21.15.45:0/1998533364 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd2c00a29f0 0x7fd2c00a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:37.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.762+0000 7fd2e61e3740 1 -- 172.21.15.45:0/3672768770 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9e3010 msgr2=0x5574fb9e1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:37.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.762+0000 7fd2e61e3740 1 --2- 172.21.15.45:0/3672768770 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9e3010 0x5574fb9e1d60 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fd2d4009c20 tx=0x7fd2d40117b0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.762+0000 7fd2e61e3740 1 -- 172.21.15.45:0/3672768770 shutdown_connections 2024-03-28T18:00:37.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.762+0000 7fd2e61e3740 1 --2- 172.21.15.45:0/3672768770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5574fb9eb6a0 0x5574fb9eda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.762+0000 7fd2e61e3740 1 --2- 172.21.15.45:0/3672768770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574fb9e2330 0x5574fb9e27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.762+0000 7fd2e61e3740 1 --2- 172.21.15.45:0/3672768770 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9e3010 0x5574fb9e1d60 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.762+0000 7fd2e61e3740 1 -- 172.21.15.45:0/3672768770 >> 172.21.15.45:0/3672768770 conn(0x5574fb9a5200 msgr2=0x5574fb9dcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:37.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.762+0000 7fd2e61e3740 1 -- 172.21.15.45:0/3672768770 shutdown_connections 2024-03-28T18:00:37.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.762+0000 7fd2e61e3740 1 -- 172.21.15.45:0/3672768770 wait complete. 2024-03-28T18:00:37.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.763+0000 7fd2e0a00640 1 --2- 172.21.15.45:0/1998533364 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd2c00a29f0 0x7fd2c00a4eb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc009ca0 tx=0x7fd2cc0093a0 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:37.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.763+0000 7fd2e61e3740 1 -- start start 2024-03-28T18:00:37.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.763+0000 7fd2e61e3740 1 Processor -- start 2024-03-28T18:00:37.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.764+0000 7fd2e61e3740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5574fb9e2330 0x5574fb9ead50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:37.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.764+0000 7fd2e61e3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9e3010 0x5574fb9e80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:37.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.764+0000 7fd2e61e3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574fb9eb6a0 0x5574fb9e8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:37.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.764+0000 7fd2e61e3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574fb9ee510 con 0x5574fb9eb6a0 2024-03-28T18:00:37.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.764+0000 7fd2e61e3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574fb8fde80 con 0x5574fb9e3010 2024-03-28T18:00:37.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.764+0000 7fd2e61e3740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574fb9ee690 con 0x5574fb9e2330 2024-03-28T18:00:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.764+0000 7fd2e61e3740 10 client.0 _ll_register_callbacks cb 0x5574fb910130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:00:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.764+0000 7fd2dbe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9e3010 0x5574fb9e80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.764+0000 7fd2e0a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5574fb9e2330 0x5574fb9ead50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.764+0000 7fd2dbe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9e3010 0x5574fb9e80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:57004/0 (socket says 172.21.15.45:57004) 2024-03-28T18:00:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.764+0000 7fd2dbe00640 1 -- 172.21.15.45:0/3271757970 learned_addr learned my addr 172.21.15.45:0/3271757970 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:00:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.764+0000 7fd2e1400640 1 --2- 172.21.15.45:0/3271757970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574fb9eb6a0 0x5574fb9e8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.764+0000 7fd2dbe00640 1 -- 172.21.15.45:0/3271757970 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5574fb9e2330 msgr2=0x5574fb9ead50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:00:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.764+0000 7fd2dbe00640 1 --2- 172.21.15.45:0/3271757970 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5574fb9e2330 0x5574fb9ead50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.764+0000 7fd2dbe00640 1 -- 172.21.15.45:0/3271757970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574fb9eb6a0 msgr2=0x5574fb9e8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:00:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.764+0000 7fd2dbe00640 1 --2- 172.21.15.45:0/3271757970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574fb9eb6a0 0x5574fb9e8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.764+0000 7fd2dbe00640 1 -- 172.21.15.45:0/3271757970 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5574fb7207d0 con 0x5574fb9e3010 2024-03-28T18:00:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.764+0000 7fd2e1400640 1 --2- 172.21.15.45:0/3271757970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574fb9eb6a0 0x5574fb9e8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:00:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.765+0000 7fd2dbe00640 1 --2- 172.21.15.45:0/3271757970 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9e3010 0x5574fb9e80e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fd2c80cc8e0 tx=0x7fd2c80973d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d51a0659eada3bf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.765+0000 7fd2c6000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T18:00:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.765+0000 7fd2c6000640 1 -- 172.21.15.45:0/3271757970 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c809b2d0 con 0x5574fb9e3010 2024-03-28T18:00:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.765+0000 7fd2c6000640 1 -- 172.21.15.45:0/3271757970 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(132..132 src has 1..132) v4 ==== 11608+0+0 (secure 0 0 0) 0x7fd2c8097e00 con 0x5574fb9e3010 2024-03-28T18:00:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.765+0000 7fd2e61e3740 1 -- 172.21.15.45:0/3271757970 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd2d4012ad0 con 0x5574fb9e3010 2024-03-28T18:00:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.765+0000 7fd2c6000640 1 -- 172.21.15.45:0/3271757970 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=260) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2c809b0f0 con 0x5574fb9e3010 2024-03-28T18:00:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.765+0000 7fd2e61e3740 10 client.26435 fetch_fsmap learned FSMap version 260 2024-03-28T18:00:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.765+0000 7fd2e61e3740 1 -- 172.21.15.45:0/3271757970 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=260,monmap=4+}) v3 -- 0x5574fb6563f0 con 0x5574fb9e3010 2024-03-28T18:00:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.766+0000 7fd2c6000640 1 -- 172.21.15.45:0/3271757970 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2d4012ad0 con 0x5574fb9e3010 2024-03-28T18:00:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.766+0000 7fd2c6000640 1 -- 172.21.15.45:0/3271757970 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 260) v1 ==== 4468+0+0 (secure 0 0 0) 0x7fd2c809b9f0 con 0x5574fb9e3010 2024-03-28T18:00:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.766+0000 7fd2e61e3740 10 client.26435 fetch_fsmap finished waiting for FSMap version 260 2024-03-28T18:00:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.766+0000 7fd2e61e3740 10 client.26435 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:00:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.766+0000 7fd2e61e3740 20 client.26435 populate_metadata read hostname 'smithi045' 2024-03-28T18:00:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.766+0000 7fd2e61e3740 1 --2- 172.21.15.45:0/3271757970 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5574fb9fba20 0x5574fb9fde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.766+0000 7fd2e61e3740 4 client.26435 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:00:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.766+0000 7fd2e61e3740 1 -- 172.21.15.45:0/3271757970 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5574fb94f790 con 0x5574fb9fba20 2024-03-28T18:00:37.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.766+0000 7fd2e0a00640 1 --2- 172.21.15.45:0/3271757970 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5574fb9fba20 0x5574fb9fde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:37.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.767+0000 7fd2e0a00640 1 --2- 172.21.15.45:0/3271757970 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5574fb9fba20 0x5574fb9fde60 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7641b3b2cdf44c4a server_cookie=c09523997c56eb41 in_seq=0 out_seq=0 2024-03-28T18:00:37.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.767+0000 7fd2c6000640 10 client.26435 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:00:37.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.767+0000 7fd2c6000640 1 -- 172.21.15.45:0/3271757970 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5574fb94f790 con 0x5574fb9fba20 2024-03-28T18:00:37.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.767+0000 7fd2c6000640 10 client.26435 handle_command_reply: tid=0 2024-03-28T18:00:37.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.767+0000 7fd2e61e3740 1 client.26435 shutdown 2024-03-28T18:00:37.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.768+0000 7fd2e61e3740 1 -- 172.21.15.45:0/3271757970 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9e3010 msgr2=0x5574fb9e80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:37.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.768+0000 7fd2e61e3740 1 --2- 172.21.15.45:0/3271757970 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9e3010 0x5574fb9e80e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fd2c80cc8e0 tx=0x7fd2c80973d0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.768+0000 7fd2e61e3740 1 -- 172.21.15.45:0/3271757970 shutdown_connections 2024-03-28T18:00:37.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.768+0000 7fd2e61e3740 1 --2- 172.21.15.45:0/3271757970 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5574fb9fba20 0x5574fb9fde60 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.768+0000 7fd2e61e3740 1 --2- 172.21.15.45:0/3271757970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574fb9eb6a0 0x5574fb9e8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.768+0000 7fd2e61e3740 1 --2- 172.21.15.45:0/3271757970 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9e3010 0x5574fb9e80e0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.768+0000 7fd2c6000640 0 client.26435 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:00:37.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.768+0000 7fd2e61e3740 1 --2- 172.21.15.45:0/3271757970 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5574fb9e2330 0x5574fb9ead50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.768+0000 7fd2e61e3740 1 -- 172.21.15.45:0/3271757970 >> 172.21.15.45:0/3271757970 conn(0x5574fb9a5200 msgr2=0x5574fb9d7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:37.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.768+0000 7fd2e61e3740 1 -- 172.21.15.45:0/3271757970 shutdown_connections 2024-03-28T18:00:37.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.768+0000 7fd2e61e3740 1 -- 172.21.15.45:0/3271757970 wait complete. 2024-03-28T18:00:37.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.768+0000 7fd2e61e3740 20 client.26435 trim_cache size 0 max 16384 2024-03-28T18:00:37.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.786+0000 7fd2e61e3740 1 Processor -- start 2024-03-28T18:00:37.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.786+0000 7fd2e61e3740 1 -- start start 2024-03-28T18:00:37.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.786+0000 7fd2e61e3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574fb9e2330 0x5574fb9ea810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:37.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.786+0000 7fd2e61e3740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5574fb9eb6a0 0x5574fb9ead50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:37.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.786+0000 7fd2e61e3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9f3910 0x5574fb9eca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:37.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.786+0000 7fd2e61e3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc1ad5c0 con 0x5574fb9e2330 2024-03-28T18:00:37.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.786+0000 7fd2e61e3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc1ab5e0 con 0x5574fb9f3910 2024-03-28T18:00:37.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.786+0000 7fd2e61e3740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc1acf60 con 0x5574fb9eb6a0 2024-03-28T18:00:37.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.786+0000 7fd2e0a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574fb9e2330 0x5574fb9ea810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:37.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.786+0000 7fd2e1400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9f3910 0x5574fb9eca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:37.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.786+0000 7fd2e1400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9f3910 0x5574fb9eca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:57012/0 (socket says 172.21.15.45:57012) 2024-03-28T18:00:37.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.786+0000 7fd2e1400640 1 -- 172.21.15.45:0/1571824940 learned_addr learned my addr 172.21.15.45:0/1571824940 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:00:37.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.787+0000 7fd2e0a00640 1 -- 172.21.15.45:0/1571824940 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5574fb9eb6a0 msgr2=0x5574fb9ead50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:00:37.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.787+0000 7fd2dbe00640 1 --2- 172.21.15.45:0/1571824940 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5574fb9eb6a0 0x5574fb9ead50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:37.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.787+0000 7fd2e0a00640 1 --2- 172.21.15.45:0/1571824940 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5574fb9eb6a0 0x5574fb9ead50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.787+0000 7fd2e0a00640 1 -- 172.21.15.45:0/1571824940 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9f3910 msgr2=0x5574fb9eca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:00:37.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.787+0000 7fd2e0a00640 1 --2- 172.21.15.45:0/1571824940 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9f3910 0x5574fb9eca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.787+0000 7fd2e0a00640 1 -- 172.21.15.45:0/1571824940 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574fb6160f0 con 0x5574fb9e2330 2024-03-28T18:00:37.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.787+0000 7fd2e1400640 1 --2- 172.21.15.45:0/1571824940 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9f3910 0x5574fb9eca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:00:37.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.787+0000 7fd2dbe00640 1 --2- 172.21.15.45:0/1571824940 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5574fb9eb6a0 0x5574fb9ead50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:00:37.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.787+0000 7fd2e0a00640 1 --2- 172.21.15.45:0/1571824940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574fb9e2330 0x5574fb9ea810 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc009bf0 tx=0x7fd2cc02e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8082a94c826964fa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:37.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.787+0000 7fd2d1200640 1 -- 172.21.15.45:0/1571824940 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc039040 con 0x5574fb9e2330 2024-03-28T18:00:37.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.787+0000 7fd2d1200640 1 -- 172.21.15.45:0/1571824940 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2cc053070 con 0x5574fb9e2330 2024-03-28T18:00:37.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.788+0000 7fd2d1200640 1 -- 172.21.15.45:0/1571824940 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc053da0 con 0x5574fb9e2330 2024-03-28T18:00:37.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.788+0000 7fd2e61e3740 1 -- 172.21.15.45:0/1571824940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574fb9e2330 msgr2=0x5574fb9ea810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:37.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.788+0000 7fd2e61e3740 1 --2- 172.21.15.45:0/1571824940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574fb9e2330 0x5574fb9ea810 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc009bf0 tx=0x7fd2cc02e050 comp rx=0 tx=0).stop 2024-03-28T18:00:37.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.788+0000 7fd2e61e3740 1 -- 172.21.15.45:0/1571824940 shutdown_connections 2024-03-28T18:00:37.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.788+0000 7fd2e61e3740 1 --2- 172.21.15.45:0/1571824940 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9f3910 0x5574fb9eca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.788+0000 7fd2e61e3740 1 --2- 172.21.15.45:0/1571824940 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5574fb9eb6a0 0x5574fb9ead50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.788+0000 7fd2e61e3740 1 --2- 172.21.15.45:0/1571824940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574fb9e2330 0x5574fb9ea810 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.788+0000 7fd2e61e3740 1 -- 172.21.15.45:0/1571824940 >> 172.21.15.45:0/1571824940 conn(0x5574fb9a5200 msgr2=0x5574fb9e69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:37.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.788+0000 7fd2e61e3740 1 -- 172.21.15.45:0/1571824940 shutdown_connections 2024-03-28T18:00:37.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.788+0000 7fd2e61e3740 1 -- 172.21.15.45:0/1571824940 wait complete. 2024-03-28T18:00:37.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.789+0000 7fd2e61e3740 1 -- start start 2024-03-28T18:00:37.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.789+0000 7fd2e61e3740 1 Processor -- start 2024-03-28T18:00:37.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.790+0000 7fd2e61e3740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5574fb9e2330 0x5574fb9e5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:37.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.790+0000 7fd2e61e3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9eb6a0 0x5574fb9e5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:37.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.790+0000 7fd2e61e3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574fb9f3910 0x5574fb9e5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:37.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.790+0000 7fd2e61e3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc1ac2a0 con 0x5574fb9f3910 2024-03-28T18:00:37.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.790+0000 7fd2e61e3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc1abc40 con 0x5574fb9eb6a0 2024-03-28T18:00:37.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.790+0000 7fd2e61e3740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc1ac900 con 0x5574fb9e2330 2024-03-28T18:00:37.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.790+0000 7fd2e61e3740 10 client.0 _ll_register_callbacks cb 0x7fd2c80d3580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:00:37.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.790+0000 7fd2dbe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9eb6a0 0x5574fb9e5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:37.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.790+0000 7fd2dbe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9eb6a0 0x5574fb9e5880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:57028/0 (socket says 172.21.15.45:57028) 2024-03-28T18:00:37.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.790+0000 7fd2dbe00640 1 -- 172.21.15.45:0/3627651450 learned_addr learned my addr 172.21.15.45:0/3627651450 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:00:37.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.790+0000 7fd2e1400640 1 --2- 172.21.15.45:0/3627651450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574fb9f3910 0x5574fb9e5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:37.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.790+0000 7fd2e0a00640 1 --2- 172.21.15.45:0/3627651450 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5574fb9e2330 0x5574fb9e5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:37.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.790+0000 7fd2dbe00640 1 -- 172.21.15.45:0/3627651450 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5574fb9e2330 msgr2=0x5574fb9e5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:00:37.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.790+0000 7fd2dbe00640 1 --2- 172.21.15.45:0/3627651450 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5574fb9e2330 0x5574fb9e5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.790+0000 7fd2dbe00640 1 -- 172.21.15.45:0/3627651450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574fb9f3910 msgr2=0x5574fb9e5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:00:37.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.790+0000 7fd2dbe00640 1 --2- 172.21.15.45:0/3627651450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574fb9f3910 0x5574fb9e5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.790+0000 7fd2dbe00640 1 -- 172.21.15.45:0/3627651450 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5574fba57390 con 0x5574fb9eb6a0 2024-03-28T18:00:37.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.790+0000 7fd2e1400640 1 --2- 172.21.15.45:0/3627651450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574fb9f3910 0x5574fb9e5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:00:37.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.790+0000 7fd2dbe00640 1 --2- 172.21.15.45:0/3627651450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9eb6a0 0x5574fb9e5880 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd2c809b720 tx=0x7fd2c80bb510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bb37c06ba5a81a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:00:37.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.790+0000 7fd2c6000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T18:00:37.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.791+0000 7fd2c6000640 1 -- 172.21.15.45:0/3627651450 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c80b9020 con 0x5574fb9eb6a0 2024-03-28T18:00:37.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.791+0000 7fd2c6000640 1 -- 172.21.15.45:0/3627651450 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(132..132 src has 1..132) v4 ==== 11608+0+0 (secure 0 0 0) 0x7fd2c8097e00 con 0x5574fb9eb6a0 2024-03-28T18:00:37.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.791+0000 7fd2e61e3740 1 -- 172.21.15.45:0/3627651450 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd2cc053070 con 0x5574fb9eb6a0 2024-03-28T18:00:37.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.791+0000 7fd2c6000640 1 -- 172.21.15.45:0/3627651450 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=260) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2c80d8070 con 0x5574fb9eb6a0 2024-03-28T18:00:37.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.791+0000 7fd2e61e3740 10 client.26441 fetch_fsmap learned FSMap version 260 2024-03-28T18:00:37.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.791+0000 7fd2e61e3740 1 -- 172.21.15.45:0/3627651450 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=260,monmap=4+}) v3 -- 0x5574fba585f0 con 0x5574fb9eb6a0 2024-03-28T18:00:37.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.791+0000 7fd2c6000640 1 -- 172.21.15.45:0/3627651450 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2cc053070 con 0x5574fb9eb6a0 2024-03-28T18:00:37.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.791+0000 7fd2c6000640 1 -- 172.21.15.45:0/3627651450 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 260) v1 ==== 4468+0+0 (secure 0 0 0) 0x7fd2c80a3aa0 con 0x5574fb9eb6a0 2024-03-28T18:00:37.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.791+0000 7fd2e61e3740 10 client.26441 fetch_fsmap finished waiting for FSMap version 260 2024-03-28T18:00:37.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.791+0000 7fd2e61e3740 10 client.26441 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:00:37.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.791+0000 7fd2e61e3740 20 client.26441 populate_metadata read hostname 'smithi045' 2024-03-28T18:00:37.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.791+0000 7fd2e61e3740 1 --2- 172.21.15.45:0/3627651450 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5574fba58c60 0x5574fba5b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:00:37.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.791+0000 7fd2e61e3740 4 client.26441 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:00:37.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.791+0000 7fd2e61e3740 1 -- 172.21.15.45:0/3627651450 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5574fb949ae0 con 0x5574fba58c60 2024-03-28T18:00:37.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.792+0000 7fd2e0a00640 1 --2- 172.21.15.45:0/3627651450 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5574fba58c60 0x5574fba5b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:00:37.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.792+0000 7fd2e0a00640 1 --2- 172.21.15.45:0/3627651450 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5574fba58c60 0x5574fba5b080 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1cdac1d048a5123 server_cookie=79cf8ce9eb1c3ed5 in_seq=0 out_seq=0 2024-03-28T18:00:37.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.792+0000 7fd2c6000640 10 client.26441 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:00:37.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.792+0000 7fd2c6000640 1 -- 172.21.15.45:0/3627651450 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5574fb949ae0 con 0x5574fba58c60 2024-03-28T18:00:37.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.792+0000 7fd2c6000640 10 client.26441 handle_command_reply: tid=0 2024-03-28T18:00:37.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.792+0000 7fd2e61e3740 1 client.26441 shutdown 2024-03-28T18:00:37.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.792+0000 7fd2e61e3740 1 -- 172.21.15.45:0/3627651450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9eb6a0 msgr2=0x5574fb9e5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:37.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.792+0000 7fd2e61e3740 1 --2- 172.21.15.45:0/3627651450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9eb6a0 0x5574fb9e5880 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd2c809b720 tx=0x7fd2c80bb510 comp rx=0 tx=0).stop 2024-03-28T18:00:37.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.792+0000 7fd2e61e3740 1 -- 172.21.15.45:0/3627651450 shutdown_connections 2024-03-28T18:00:37.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.792+0000 7fd2e61e3740 1 --2- 172.21.15.45:0/3627651450 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5574fba58c60 0x5574fba5b080 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.792+0000 7fd2e61e3740 1 --2- 172.21.15.45:0/3627651450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574fb9f3910 0x5574fb9e5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.792+0000 7fd2e61e3740 1 --2- 172.21.15.45:0/3627651450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5574fb9eb6a0 0x5574fb9e5880 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.793+0000 7fd2e61e3740 1 --2- 172.21.15.45:0/3627651450 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5574fb9e2330 0x5574fb9e5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.793+0000 7fd2e61e3740 1 -- 172.21.15.45:0/3627651450 >> 172.21.15.45:0/3627651450 conn(0x5574fb9a5200 msgr2=0x5574fb9d7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:37.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.792+0000 7fd2c6000640 0 client.26441 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:00:37.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.793+0000 7fd2e61e3740 1 -- 172.21.15.45:0/3627651450 shutdown_connections 2024-03-28T18:00:37.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.793+0000 7fd2e61e3740 1 -- 172.21.15.45:0/3627651450 wait complete. 2024-03-28T18:00:37.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.793+0000 7fd2e61e3740 20 client.26441 trim_cache size 0 max 16384 2024-03-28T18:00:37.797 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T18:00:37.797 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:00:37.798 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T18:00:37.798 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T18:00:37.798 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T18:00:37.798 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T18:00:37.798 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T18:00:37.798 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:00:37.798 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:00:37.798 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T18:00:37.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.793+0000 7fd2e3400640 1 -- 172.21.15.45:0/1998533364 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd2c00a29f0 msgr2=0x7fd2c00a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:37.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.793+0000 7fd2e3400640 1 --2- 172.21.15.45:0/1998533364 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd2c00a29f0 0x7fd2c00a4eb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc009ca0 tx=0x7fd2cc0093a0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.793+0000 7fd2e3400640 1 -- 172.21.15.45:0/1998533364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2dc0ffdc0 msgr2=0x7fd2dc198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:00:37.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.793+0000 7fd2e3400640 1 --2- 172.21.15.45:0/1998533364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2dc0ffdc0 0x7fd2dc198510 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fd2c800e970 tx=0x7fd2c800ee30 comp rx=0 tx=0).stop 2024-03-28T18:00:37.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.793+0000 7fd2e3400640 1 -- 172.21.15.45:0/1998533364 shutdown_connections 2024-03-28T18:00:37.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.793+0000 7fd2e3400640 1 --2- 172.21.15.45:0/1998533364 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd2c00a29f0 0x7fd2c00a4eb0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.794+0000 7fd2e3400640 1 --2- 172.21.15.45:0/1998533364 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2dc105890 0x7fd2dc19c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.794+0000 7fd2e3400640 1 --2- 172.21.15.45:0/1998533364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2dc0ffdc0 0x7fd2dc198510 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.794+0000 7fd2e3400640 1 --2- 172.21.15.45:0/1998533364 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2dc067c00 0x7fd2dc197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:00:37.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.794+0000 7fd2e3400640 1 -- 172.21.15.45:0/1998533364 >> 172.21.15.45:0/1998533364 conn(0x7fd2dc0766f0 msgr2=0x7fd2dc0fba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:00:37.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.794+0000 7fd2e3400640 1 -- 172.21.15.45:0/1998533364 shutdown_connections 2024-03-28T18:00:37.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:00:37.794+0000 7fd2e3400640 1 -- 172.21.15.45:0/1998533364 wait complete. 2024-03-28T18:00:37.814 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T18:00:37.921 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 490 sec latency 605818.576 ms 2024-03-28T18:00:38.921 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 491 sec latency 606818.705 ms 2024-03-28T18:00:39.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:38 smithi071 ceph-mon[36597]: pgmap v398: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 34 KiB/s wr, 17 op/s 2024-03-28T18:00:39.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:38 smithi151 ceph-mon[36481]: pgmap v398: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 34 KiB/s wr, 17 op/s 2024-03-28T18:00:39.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:38 smithi045 ceph-mon[31773]: pgmap v398: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 34 KiB/s wr, 17 op/s 2024-03-28T18:00:39.922 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 492 sec latency 607818.920 ms 2024-03-28T18:00:40.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:40 smithi045 ceph-mon[31773]: pgmap v399: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T18:00:40.922 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 493 sec latency 608819.015 ms 2024-03-28T18:00:41.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:40 smithi071 ceph-mon[36597]: pgmap v399: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T18:00:41.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:40 smithi151 ceph-mon[36481]: pgmap v399: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T18:00:41.922 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 494 sec latency 609819.275 ms 2024-03-28T18:00:42.922 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 495 sec latency 610819.429 ms 2024-03-28T18:00:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:42 smithi071 ceph-mon[36597]: pgmap v400: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 23 KiB/s wr, 14 op/s 2024-03-28T18:00:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:42 smithi071 ceph-mon[36597]: osdmap e133: 12 total, 12 up, 12 in 2024-03-28T18:00:43.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:42 smithi151 ceph-mon[36481]: pgmap v400: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 23 KiB/s wr, 14 op/s 2024-03-28T18:00:43.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:42 smithi151 ceph-mon[36481]: osdmap e133: 12 total, 12 up, 12 in 2024-03-28T18:00:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:42 smithi045 ceph-mon[31773]: pgmap v400: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 23 KiB/s wr, 14 op/s 2024-03-28T18:00:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:42 smithi045 ceph-mon[31773]: osdmap e133: 12 total, 12 up, 12 in 2024-03-28T18:00:43.922 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 496 sec latency 611819.590 ms 2024-03-28T18:00:44.922 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 497 sec latency 612819.720 ms 2024-03-28T18:00:45.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:44 smithi071 ceph-mon[36597]: pgmap v402: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-28T18:00:45.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:44 smithi151 ceph-mon[36481]: pgmap v402: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-28T18:00:45.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:44 smithi045 ceph-mon[31773]: pgmap v402: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-28T18:00:45.923 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 498 sec latency 613819.879 ms 2024-03-28T18:00:46.793 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:46 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 2. 2024-03-28T18:00:46.793 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:46 smithi045 systemd[1]: Stopped Ceph mds.g for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:00:46.793 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:46 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Consumed 1.585s CPU time. 2024-03-28T18:00:46.793 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:46 smithi045 systemd[1]: Starting Ceph mds.g for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T18:00:46.923 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 499 sec latency 614819.973 ms 2024-03-28T18:00:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:46 smithi071 ceph-mon[36597]: pgmap v403: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:00:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:46 smithi071 ceph-mon[36597]: osdmap e134: 12 total, 12 up, 12 in 2024-03-28T18:00:47.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:46 smithi151 ceph-mon[36481]: pgmap v403: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:00:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:46 smithi151 ceph-mon[36481]: osdmap e134: 12 total, 12 up, 12 in 2024-03-28T18:00:47.083 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:46 smithi045 podman[94382]: 2024-03-28 18:00:46.727985979 +0000 UTC m=+0.033741937 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:00:47.083 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:46 smithi045 podman[94382]: 2024-03-28 18:00:46.901768957 +0000 UTC m=+0.207524946 container create 6679fa2265c0175a484e989cec5c2d9361855a6799504668d96e87bf7c0a8c44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-28T18:00:47.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:46 smithi045 ceph-mon[31773]: pgmap v403: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:00:47.084 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:46 smithi045 ceph-mon[31773]: osdmap e134: 12 total, 12 up, 12 in 2024-03-28T18:00:47.480 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:47 smithi045 podman[94382]: 2024-03-28 18:00:47.196610498 +0000 UTC m=+0.502366527 container init 6679fa2265c0175a484e989cec5c2d9361855a6799504668d96e87bf7c0a8c44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5) 2024-03-28T18:00:47.480 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:47 smithi045 podman[94382]: 2024-03-28 18:00:47.207466799 +0000 UTC m=+0.513222795 container start 6679fa2265c0175a484e989cec5c2d9361855a6799504668d96e87bf7c0a8c44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:00:47.481 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:47 smithi045 ceph-mds[94401]: -- 172.21.15.45:0/3483787869 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55aa932a9a40 con 0x55aa9327d000 2024-03-28T18:00:47.481 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:47 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: starting mds.g at 2024-03-28T18:00:47.796 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:47 smithi045 bash[94382]: 6679fa2265c0175a484e989cec5c2d9361855a6799504668d96e87bf7c0a8c44 2024-03-28T18:00:47.796 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:00:47 smithi045 systemd[1]: Started Ceph mds.g for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:00:47.923 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 500 sec latency 615820.130 ms 2024-03-28T18:00:48.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:47 smithi071 ceph-mon[36597]: daemon mds.g restarted 2024-03-28T18:00:48.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:47 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:00:48.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:47 smithi071 ceph-mon[36597]: osdmap e135: 12 total, 12 up, 12 in 2024-03-28T18:00:48.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:47 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:00:48.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:47 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/1366698753,v1:172.21.15.45:6837/1366698753] up:boot 2024-03-28T18:00:48.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:47 smithi071 ceph-mon[36597]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-03-28T18:00:48.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:47 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:00:48.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:47 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:47 smithi151 ceph-mon[36481]: daemon mds.g restarted 2024-03-28T18:00:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:47 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:00:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:47 smithi151 ceph-mon[36481]: osdmap e135: 12 total, 12 up, 12 in 2024-03-28T18:00:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:47 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:00:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:47 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/1366698753,v1:172.21.15.45:6837/1366698753] up:boot 2024-03-28T18:00:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:47 smithi151 ceph-mon[36481]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-03-28T18:00:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:47 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:00:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:47 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:47 smithi045 ceph-mon[31773]: daemon mds.g restarted 2024-03-28T18:00:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:47 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:00:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:47 smithi045 ceph-mon[31773]: osdmap e135: 12 total, 12 up, 12 in 2024-03-28T18:00:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:47 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:00:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:47 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/1366698753,v1:172.21.15.45:6837/1366698753] up:boot 2024-03-28T18:00:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:47 smithi045 ceph-mon[31773]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-03-28T18:00:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:47 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:00:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:47 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:48.923 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 501 sec latency 616820.227 ms 2024-03-28T18:00:49.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:48 smithi071 ceph-mon[36597]: pgmap v406: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:00:49.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:48 smithi151 ceph-mon[36481]: pgmap v406: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:00:49.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:48 smithi045 ceph-mon[31773]: pgmap v406: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:00:49.923 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 502 sec latency 617820.345 ms 2024-03-28T18:00:50.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:49 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/3392572128,v1:172.21.15.45:6835/3392572128] up:resolve 2024-03-28T18:00:50.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:49 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:50.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:49 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/3392572128,v1:172.21.15.45:6835/3392572128] up:resolve 2024-03-28T18:00:50.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:49 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:50.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:49 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/3392572128,v1:172.21.15.45:6835/3392572128] up:resolve 2024-03-28T18:00:50.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:49 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:50.923 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 503 sec latency 618820.469 ms 2024-03-28T18:00:51.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:50 smithi071 ceph-mon[36597]: pgmap v407: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 132 B/s wr, 4 op/s 2024-03-28T18:00:51.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:50 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/3392572128,v1:172.21.15.45:6835/3392572128] up:reconnect 2024-03-28T18:00:51.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:50 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:51.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:50 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T18:00:51.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:50 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.0109998 2024-03-28T18:00:51.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:50 smithi045 ceph-mon[31773]: pgmap v407: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 132 B/s wr, 4 op/s 2024-03-28T18:00:51.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:50 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/3392572128,v1:172.21.15.45:6835/3392572128] up:reconnect 2024-03-28T18:00:51.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:50 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:51.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:50 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T18:00:51.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:50 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.0109998 2024-03-28T18:00:51.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:50 smithi151 ceph-mon[36481]: pgmap v407: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 132 B/s wr, 4 op/s 2024-03-28T18:00:51.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:50 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/3392572128,v1:172.21.15.45:6835/3392572128] up:reconnect 2024-03-28T18:00:51.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:50 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:51.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:50 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999985 2024-03-28T18:00:51.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:50 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.0109998 2024-03-28T18:00:51.836 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0efb600700 time 2024-03-28T18:00:51.501717+0000 2024-03-28T18:00:51.836 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:00:51.836 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:51.836 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0f06ee84a1] 2024-03-28T18:00:51.836 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f06ee8667] 2024-03-28T18:00:51.836 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5590b9d7f223] 2024-03-28T18:00:51.836 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5590b9bf95d6] 2024-03-28T18:00:51.836 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5590b9bf9da1] 2024-03-28T18:00:51.836 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5590b9bfc2d5] 2024-03-28T18:00:51.836 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5590b9c4c233] 2024-03-28T18:00:51.836 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5590b9c4e197] 2024-03-28T18:00:51.837 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5590b9b4d11b] 2024-03-28T18:00:51.837 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5590b9b4d6bc] 2024-03-28T18:00:51.837 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5590b9b371ff] 2024-03-28T18:00:51.837 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f0717a488] 2024-03-28T18:00:51.837 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 13: (DispatchQueue::entry()+0x51f) [0x7f0f0717760f] 2024-03-28T18:00:51.837 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f0724ae81] 2024-03-28T18:00:51.837 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0f05c651ca] 2024-03-28T18:00:51.837 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 16: clone() 2024-03-28T18:00:51.837 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 2024-03-28T18:00:51.502+0000 7f0efb600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0efb600700 time 2024-03-28T18:00:51.501717+0000 2024-03-28T18:00:51.838 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:00:51.838 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 2024-03-28T18:00:51.838 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:51.838 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0f06ee84a1] 2024-03-28T18:00:51.838 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f06ee8667] 2024-03-28T18:00:51.838 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5590b9d7f223] 2024-03-28T18:00:51.838 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5590b9bf95d6] 2024-03-28T18:00:51.838 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5590b9bf9da1] 2024-03-28T18:00:51.838 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5590b9bfc2d5] 2024-03-28T18:00:51.838 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5590b9c4c233] 2024-03-28T18:00:51.838 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5590b9c4e197] 2024-03-28T18:00:51.838 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5590b9b4d11b] 2024-03-28T18:00:51.838 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5590b9b4d6bc] 2024-03-28T18:00:51.839 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5590b9b371ff] 2024-03-28T18:00:51.839 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f0717a488] 2024-03-28T18:00:51.839 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 13: (DispatchQueue::entry()+0x51f) [0x7f0f0717760f] 2024-03-28T18:00:51.839 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f0724ae81] 2024-03-28T18:00:51.839 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0f05c651ca] 2024-03-28T18:00:51.839 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 16: clone() 2024-03-28T18:00:51.839 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 2024-03-28T18:00:51.839 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: *** Caught signal (Aborted) ** 2024-03-28T18:00:51.839 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: in thread 7f0efb600700 thread_name:ms_dispatch 2024-03-28T18:00:51.839 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:51.839 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0f05c6fd20] 2024-03-28T18:00:51.839 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/3392572128,v1:172.21.15.45:6835/3392572128] up:rejoin 2024-03-28T18:00:51.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:51.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-mon[31773]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-03-28T18:00:51.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:00:51.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/3392572128,v1:172.21.15.45:6835/3392572128] up:active 2024-03-28T18:00:51.840 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 3 up:standby-replay 2024-03-28T18:00:51.840 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 2: gsignal() 2024-03-28T18:00:51.840 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 3: abort() 2024-03-28T18:00:51.840 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0f06ee84fb] 2024-03-28T18:00:51.840 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f06ee8667] 2024-03-28T18:00:51.841 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5590b9d7f223] 2024-03-28T18:00:51.841 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5590b9bf95d6] 2024-03-28T18:00:51.841 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5590b9bf9da1] 2024-03-28T18:00:51.841 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5590b9bfc2d5] 2024-03-28T18:00:51.841 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5590b9c4c233] 2024-03-28T18:00:51.841 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5590b9c4e197] 2024-03-28T18:00:51.841 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5590b9b4d11b] 2024-03-28T18:00:51.841 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5590b9b4d6bc] 2024-03-28T18:00:51.841 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5590b9b371ff] 2024-03-28T18:00:51.841 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f0717a488] 2024-03-28T18:00:51.841 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 16: (DispatchQueue::entry()+0x51f) [0x7f0f0717760f] 2024-03-28T18:00:51.841 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f0724ae81] 2024-03-28T18:00:51.841 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0f05c651ca] 2024-03-28T18:00:51.841 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 19: clone() 2024-03-28T18:00:51.842 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 2024-03-28T18:00:51.505+0000 7f0efb600700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:00:51.842 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: in thread 7f0efb600700 thread_name:ms_dispatch 2024-03-28T18:00:51.842 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 2024-03-28T18:00:51.842 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:51.842 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0f05c6fd20] 2024-03-28T18:00:51.842 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 2: gsignal() 2024-03-28T18:00:51.842 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 3: abort() 2024-03-28T18:00:51.842 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0f06ee84fb] 2024-03-28T18:00:51.842 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f06ee8667] 2024-03-28T18:00:51.842 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5590b9d7f223] 2024-03-28T18:00:51.842 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5590b9bf95d6] 2024-03-28T18:00:51.842 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5590b9bf9da1] 2024-03-28T18:00:51.842 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5590b9bfc2d5] 2024-03-28T18:00:51.842 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5590b9c4c233] 2024-03-28T18:00:51.843 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5590b9c4e197] 2024-03-28T18:00:51.843 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5590b9b4d11b] 2024-03-28T18:00:51.843 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5590b9b4d6bc] 2024-03-28T18:00:51.843 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5590b9b371ff] 2024-03-28T18:00:51.843 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f0717a488] 2024-03-28T18:00:51.843 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 16: (DispatchQueue::entry()+0x51f) [0x7f0f0717760f] 2024-03-28T18:00:51.843 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f0724ae81] 2024-03-28T18:00:51.843 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0f05c651ca] 2024-03-28T18:00:51.843 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 19: clone() 2024-03-28T18:00:51.843 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:00:51.843 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 2024-03-28T18:00:51.843 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: -1> 2024-03-28T18:00:51.502+0000 7f0efb600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0efb600700 time 2024-03-28T18:00:51.501717+0000 2024-03-28T18:00:51.843 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:00:51.843 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 2024-03-28T18:00:51.843 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:51.844 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0f06ee84a1] 2024-03-28T18:00:51.844 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f06ee8667] 2024-03-28T18:00:51.844 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5590b9d7f223] 2024-03-28T18:00:51.844 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5590b9bf95d6] 2024-03-28T18:00:51.845 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5590b9bf9da1] 2024-03-28T18:00:51.845 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5590b9bfc2d5] 2024-03-28T18:00:51.845 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5590b9c4c233] 2024-03-28T18:00:51.845 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5590b9c4e197] 2024-03-28T18:00:51.845 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5590b9b4d11b] 2024-03-28T18:00:51.845 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5590b9b4d6bc] 2024-03-28T18:00:51.845 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5590b9b371ff] 2024-03-28T18:00:51.845 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f0717a488] 2024-03-28T18:00:51.845 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 13: (DispatchQueue::entry()+0x51f) [0x7f0f0717760f] 2024-03-28T18:00:51.845 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f0724ae81] 2024-03-28T18:00:51.845 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0f05c651ca] 2024-03-28T18:00:51.845 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 16: clone() 2024-03-28T18:00:51.845 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 2024-03-28T18:00:51.845 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 0> 2024-03-28T18:00:51.505+0000 7f0efb600700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:00:51.846 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: in thread 7f0efb600700 thread_name:ms_dispatch 2024-03-28T18:00:51.846 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 2024-03-28T18:00:51.846 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:51.846 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0f05c6fd20] 2024-03-28T18:00:51.846 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 2: gsignal() 2024-03-28T18:00:51.846 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 3: abort() 2024-03-28T18:00:51.846 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0f06ee84fb] 2024-03-28T18:00:51.846 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f06ee8667] 2024-03-28T18:00:51.846 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5590b9d7f223] 2024-03-28T18:00:51.846 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5590b9bf95d6] 2024-03-28T18:00:51.846 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5590b9bf9da1] 2024-03-28T18:00:51.846 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5590b9bfc2d5] 2024-03-28T18:00:51.846 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5590b9c4c233] 2024-03-28T18:00:51.846 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5590b9c4e197] 2024-03-28T18:00:51.847 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5590b9b4d11b] 2024-03-28T18:00:51.847 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5590b9b4d6bc] 2024-03-28T18:00:51.847 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5590b9b371ff] 2024-03-28T18:00:51.847 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f0717a488] 2024-03-28T18:00:51.847 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 16: (DispatchQueue::entry()+0x51f) [0x7f0f0717760f] 2024-03-28T18:00:51.847 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f0724ae81] 2024-03-28T18:00:51.847 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0f05c651ca] 2024-03-28T18:00:51.847 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 19: clone() 2024-03-28T18:00:51.847 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:00:51.847 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 2024-03-28T18:00:51.847 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: -1> 2024-03-28T18:00:51.502+0000 7f0efb600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0efb600700 time 2024-03-28T18:00:51.501717+0000 2024-03-28T18:00:51.847 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:00:51.847 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 2024-03-28T18:00:51.847 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:51.847 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0f06ee84a1] 2024-03-28T18:00:51.847 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f06ee8667] 2024-03-28T18:00:51.848 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5590b9d7f223] 2024-03-28T18:00:51.848 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5590b9bf95d6] 2024-03-28T18:00:51.848 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5590b9bf9da1] 2024-03-28T18:00:51.848 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5590b9bfc2d5] 2024-03-28T18:00:51.848 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5590b9c4c233] 2024-03-28T18:00:51.848 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5590b9c4e197] 2024-03-28T18:00:51.848 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5590b9b4d11b] 2024-03-28T18:00:51.848 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5590b9b4d6bc] 2024-03-28T18:00:51.848 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5590b9b371ff] 2024-03-28T18:00:51.848 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f0717a488] 2024-03-28T18:00:51.848 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 13: (DispatchQueue::entry()+0x51f) [0x7f0f0717760f] 2024-03-28T18:00:51.848 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f0724ae81] 2024-03-28T18:00:51.848 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0f05c651ca] 2024-03-28T18:00:51.848 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 16: clone() 2024-03-28T18:00:51.849 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 2024-03-28T18:00:51.849 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 0> 2024-03-28T18:00:51.505+0000 7f0efb600700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:00:51.849 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: in thread 7f0efb600700 thread_name:ms_dispatch 2024-03-28T18:00:51.849 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 2024-03-28T18:00:51.849 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:00:51.849 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0f05c6fd20] 2024-03-28T18:00:51.849 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 2: gsignal() 2024-03-28T18:00:51.849 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 3: abort() 2024-03-28T18:00:51.849 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0f06ee84fb] 2024-03-28T18:00:51.849 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f06ee8667] 2024-03-28T18:00:51.849 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5590b9d7f223] 2024-03-28T18:00:51.849 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5590b9bf95d6] 2024-03-28T18:00:51.849 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5590b9bf9da1] 2024-03-28T18:00:51.849 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5590b9bfc2d5] 2024-03-28T18:00:51.849 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5590b9c4c233] 2024-03-28T18:00:51.850 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5590b9c4e197] 2024-03-28T18:00:51.850 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5590b9b4d11b] 2024-03-28T18:00:51.850 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5590b9b4d6bc] 2024-03-28T18:00:51.850 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5590b9b371ff] 2024-03-28T18:00:51.850 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f0717a488] 2024-03-28T18:00:51.850 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 16: (DispatchQueue::entry()+0x51f) [0x7f0f0717760f] 2024-03-28T18:00:51.850 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f0724ae81] 2024-03-28T18:00:51.850 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0f05c651ca] 2024-03-28T18:00:51.850 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 19: clone() 2024-03-28T18:00:51.850 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:00:51.850 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94176]: 2024-03-28T18:00:51.850 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:51 smithi045 podman[94435]: 2024-03-28 18:00:51.604436076 +0000 UTC m=+0.034406841 container died af47547fe6999d32b3880a614e0e4bdf3ddfbad506059f9963821cec3cf1e373 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-28T18:00:51.923 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 504 sec latency 619820.561 ms 2024-03-28T18:00:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:51 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/3392572128,v1:172.21.15.45:6835/3392572128] up:rejoin 2024-03-28T18:00:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:51 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:51 smithi071 ceph-mon[36597]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-03-28T18:00:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:51 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:00:52.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:51 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/3392572128,v1:172.21.15.45:6835/3392572128] up:active 2024-03-28T18:00:52.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:51 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 3 up:standby-replay 2024-03-28T18:00:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:51 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/3392572128,v1:172.21.15.45:6835/3392572128] up:rejoin 2024-03-28T18:00:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:51 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:00:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:51 smithi151 ceph-mon[36481]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-03-28T18:00:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:51 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:00:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:51 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/3392572128,v1:172.21.15.45:6835/3392572128] up:active 2024-03-28T18:00:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:51 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 3 up:standby-replay 2024-03-28T18:00:52.837 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:52 smithi045 podman[94435]: 2024-03-28 18:00:52.429857046 +0000 UTC m=+0.859827851 container remove af47547fe6999d32b3880a614e0e4bdf3ddfbad506059f9963821cec3cf1e373 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-28T18:00:52.837 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:52 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:00:52.923 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 505 sec latency 620820.670 ms 2024-03-28T18:00:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:52 smithi045 ceph-mon[31773]: pgmap v408: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T18:00:53.166 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:52 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-28T18:00:53.166 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:00:52 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Consumed 1.579s CPU time. 2024-03-28T18:00:53.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:52 smithi071 ceph-mon[36597]: pgmap v408: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T18:00:53.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:52 smithi151 ceph-mon[36481]: pgmap v408: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T18:00:53.924 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 506 sec latency 621820.813 ms 2024-03-28T18:00:54.924 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 507 sec latency 622820.962 ms 2024-03-28T18:00:55.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:54 smithi151 ceph-mon[36481]: pgmap v409: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T18:00:55.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:54 smithi045 ceph-mon[31773]: pgmap v409: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T18:00:55.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:54 smithi071 ceph-mon[36597]: pgmap v409: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T18:00:55.924 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 508 sec latency 623821.133 ms 2024-03-28T18:00:56.924 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 509 sec latency 624821.304 ms 2024-03-28T18:00:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:56 smithi045 ceph-mon[31773]: pgmap v410: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 339 B/s wr, 12 op/s 2024-03-28T18:00:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:56 smithi071 ceph-mon[36597]: pgmap v410: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 339 B/s wr, 12 op/s 2024-03-28T18:00:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:56 smithi151 ceph-mon[36481]: pgmap v410: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 339 B/s wr, 12 op/s 2024-03-28T18:00:57.924 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 510 sec latency 625821.434 ms 2024-03-28T18:00:58.924 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 511 sec latency 626821.583 ms 2024-03-28T18:00:59.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:00:58 smithi045 ceph-mon[31773]: pgmap v411: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 299 B/s wr, 10 op/s 2024-03-28T18:00:59.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:00:58 smithi071 ceph-mon[36597]: pgmap v411: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 299 B/s wr, 10 op/s 2024-03-28T18:00:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:00:58 smithi151 ceph-mon[36481]: pgmap v411: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 299 B/s wr, 10 op/s 2024-03-28T18:00:59.924 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 512 sec latency 627821.714 ms 2024-03-28T18:01:00.925 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 513 sec latency 628821.926 ms 2024-03-28T18:01:01.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:00 smithi045 ceph-mon[31773]: pgmap v412: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T18:01:01.295 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:00 smithi071 ceph-mon[36597]: pgmap v412: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T18:01:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:00 smithi151 ceph-mon[36481]: pgmap v412: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T18:01:01.925 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 514 sec latency 629822.016 ms 2024-03-28T18:01:02.925 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 515 sec latency 630822.107 ms 2024-03-28T18:01:03.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:02 smithi045 ceph-mon[31773]: pgmap v413: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T18:01:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:02 smithi071 ceph-mon[36597]: pgmap v413: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T18:01:03.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:02 smithi151 ceph-mon[36481]: pgmap v413: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T18:01:03.573 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:03 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 2. 2024-03-28T18:01:03.573 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:03 smithi045 systemd[1]: Stopped Ceph mds.j for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:01:03.574 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:03 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Consumed 1.579s CPU time. 2024-03-28T18:01:03.574 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:03 smithi045 systemd[1]: Starting Ceph mds.j for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T18:01:03.855 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:03 smithi045 podman[94543]: 2024-03-28 18:01:03.483355381 +0000 UTC m=+0.043189556 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:01:03.855 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:03 smithi045 podman[94543]: 2024-03-28 18:01:03.85528761 +0000 UTC m=+0.415121725 container create aafd4ee1a9e67515a152ea7f8746364ca455b65622cbfaae37c8525818ff716a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-28T18:01:03.925 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 516 sec latency 631822.263 ms 2024-03-28T18:01:04.666 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:04 smithi045 podman[94543]: 2024-03-28 18:01:04.175752498 +0000 UTC m=+0.735586620 container init aafd4ee1a9e67515a152ea7f8746364ca455b65622cbfaae37c8525818ff716a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-28T18:01:04.666 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:04 smithi045 podman[94543]: 2024-03-28 18:01:04.186168973 +0000 UTC m=+0.746003094 container start aafd4ee1a9e67515a152ea7f8746364ca455b65622cbfaae37c8525818ff716a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:01:04.666 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:04 smithi045 ceph-mds[94562]: -- 172.21.15.45:0/4215437723 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ff48f35a40 con 0x55ff48f09000 2024-03-28T18:01:04.666 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:04 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: starting mds.j at 2024-03-28T18:01:04.666 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:04 smithi045 bash[94543]: aafd4ee1a9e67515a152ea7f8746364ca455b65622cbfaae37c8525818ff716a 2024-03-28T18:01:04.666 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:04 smithi045 systemd[1]: Started Ceph mds.j for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:01:04.925 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 517 sec latency 632822.407 ms 2024-03-28T18:01:05.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:04 smithi045 ceph-mon[31773]: pgmap v414: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T18:01:05.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:04 smithi045 ceph-mon[31773]: daemon mds.j restarted 2024-03-28T18:01:05.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:04 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:01:05.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:04 smithi045 ceph-mon[31773]: osdmap e136: 12 total, 12 up, 12 in 2024-03-28T18:01:05.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:04 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/1173454637,v1:172.21.15.45:6835/1173454637] up:boot 2024-03-28T18:01:05.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:04 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:01:05.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:04 smithi045 ceph-mon[31773]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-03-28T18:01:05.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:04 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:01:05.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:04 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:05.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:04 smithi071 ceph-mon[36597]: pgmap v414: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T18:01:05.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:04 smithi071 ceph-mon[36597]: daemon mds.j restarted 2024-03-28T18:01:05.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:04 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:01:05.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:04 smithi071 ceph-mon[36597]: osdmap e136: 12 total, 12 up, 12 in 2024-03-28T18:01:05.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:04 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/1173454637,v1:172.21.15.45:6835/1173454637] up:boot 2024-03-28T18:01:05.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:04 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:01:05.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:04 smithi071 ceph-mon[36597]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-03-28T18:01:05.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:04 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:01:05.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:04 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:05.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:04 smithi151 ceph-mon[36481]: pgmap v414: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T18:01:05.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:04 smithi151 ceph-mon[36481]: daemon mds.j restarted 2024-03-28T18:01:05.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:04 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:01:05.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:04 smithi151 ceph-mon[36481]: osdmap e136: 12 total, 12 up, 12 in 2024-03-28T18:01:05.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:04 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/1173454637,v1:172.21.15.45:6835/1173454637] up:boot 2024-03-28T18:01:05.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:04 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:01:05.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:04 smithi151 ceph-mon[36481]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-03-28T18:01:05.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:04 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:01:05.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:04 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:05.925 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 518 sec latency 633822.548 ms 2024-03-28T18:01:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:06 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/1366698753,v1:172.21.15.45:6837/1366698753] up:resolve 2024-03-28T18:01:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:06 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:06 smithi071 ceph-mon[36597]: pgmap v416: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:01:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:06 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/1366698753,v1:172.21.15.45:6837/1366698753] up:resolve 2024-03-28T18:01:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:06 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:06 smithi151 ceph-mon[36481]: pgmap v416: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:01:06.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:06 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/1366698753,v1:172.21.15.45:6837/1366698753] up:resolve 2024-03-28T18:01:06.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:06 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:06.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:06 smithi045 ceph-mon[31773]: pgmap v416: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:01:06.925 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 519 sec latency 634822.707 ms 2024-03-28T18:01:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:07 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:01:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:07 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/1366698753,v1:172.21.15.45:6837/1366698753] up:reconnect 2024-03-28T18:01:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:07 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:07 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T18:01:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:07 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:01:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:07 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/1366698753,v1:172.21.15.45:6837/1366698753] up:reconnect 2024-03-28T18:01:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:07 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:07 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T18:01:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:07 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:01:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:07 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/1366698753,v1:172.21.15.45:6837/1366698753] up:reconnect 2024-03-28T18:01:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:07 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:07 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T18:01:07.815 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:01:07.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:01:07.925 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 520 sec latency 635822.825 ms 2024-03-28T18:01:08.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.055+0000 7fc51f000640 1 -- 172.21.15.45:0/716398205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc518112520 msgr2=0x7fc518114910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:08.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.055+0000 7fc51f000640 1 --2- 172.21.15.45:0/716398205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc518112520 0x7fc518114910 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc510009ca0 tx=0x7fc51002c180 comp rx=0 tx=0).stop 2024-03-28T18:01:08.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.055+0000 7fc51f000640 1 -- 172.21.15.45:0/716398205 shutdown_connections 2024-03-28T18:01:08.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.055+0000 7fc51f000640 1 --2- 172.21.15.45:0/716398205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc518112520 0x7fc518114910 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.055+0000 7fc51f000640 1 --2- 172.21.15.45:0/716398205 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc518074360 0x7fc518111fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.055+0000 7fc51f000640 1 --2- 172.21.15.45:0/716398205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc518073a40 0x7fc518073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.055+0000 7fc51f000640 1 -- 172.21.15.45:0/716398205 >> 172.21.15.45:0/716398205 conn(0x7fc5180fd000 msgr2=0x7fc5180ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:08.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.055+0000 7fc51f000640 1 -- 172.21.15.45:0/716398205 shutdown_connections 2024-03-28T18:01:08.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.055+0000 7fc51f000640 1 -- 172.21.15.45:0/716398205 wait complete. 2024-03-28T18:01:08.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.056+0000 7fc51f000640 1 Processor -- start 2024-03-28T18:01:08.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.056+0000 7fc51f000640 1 -- start start 2024-03-28T18:01:08.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.057+0000 7fc51f000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc518073a40 0x7fc51819e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:08.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.057+0000 7fc51f000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc518074360 0x7fc51819e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:08.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.057+0000 7fc51f000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc518112520 0x7fc5181a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:08.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.057+0000 7fc51f000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc518117f20 con 0x7fc518074360 2024-03-28T18:01:08.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.057+0000 7fc51f000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc518117da0 con 0x7fc518073a40 2024-03-28T18:01:08.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.057+0000 7fc51f000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5181180a0 con 0x7fc518112520 2024-03-28T18:01:08.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.057+0000 7fc51dc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc518073a40 0x7fc51819e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:08.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.057+0000 7fc51d200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc518074360 0x7fc51819e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:08.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.057+0000 7fc51dc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc518073a40 0x7fc51819e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53744/0 (socket says 172.21.15.45:53744) 2024-03-28T18:01:08.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.057+0000 7fc51dc00640 1 -- 172.21.15.45:0/1617835037 learned_addr learned my addr 172.21.15.45:0/1617835037 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:01:08.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.057+0000 7fc51e600640 1 --2- 172.21.15.45:0/1617835037 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc518112520 0x7fc5181a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:08.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.057+0000 7fc51dc00640 1 -- 172.21.15.45:0/1617835037 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc518112520 msgr2=0x7fc5181a2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:08.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.057+0000 7fc51dc00640 1 --2- 172.21.15.45:0/1617835037 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc518112520 0x7fc5181a2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.057+0000 7fc51dc00640 1 -- 172.21.15.45:0/1617835037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc518074360 msgr2=0x7fc51819e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:08.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.057+0000 7fc51dc00640 1 --2- 172.21.15.45:0/1617835037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc518074360 0x7fc51819e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.057+0000 7fc51dc00640 1 -- 172.21.15.45:0/1617835037 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5181a3340 con 0x7fc518073a40 2024-03-28T18:01:08.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.057+0000 7fc51e600640 1 --2- 172.21.15.45:0/1617835037 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc518112520 0x7fc5181a2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:01:08.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.058+0000 7fc51dc00640 1 --2- 172.21.15.45:0/1617835037 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc518073a40 0x7fc51819e3a0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fc50c00c890 tx=0x7fc50c00cc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:08.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.058+0000 7fc516000640 1 -- 172.21.15.45:0/1617835037 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c004220 con 0x7fc518073a40 2024-03-28T18:01:08.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.058+0000 7fc516000640 1 -- 172.21.15.45:0/1617835037 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc50c0043c0 con 0x7fc518073a40 2024-03-28T18:01:08.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.058+0000 7fc516000640 1 -- 172.21.15.45:0/1617835037 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c004a00 con 0x7fc518073a40 2024-03-28T18:01:08.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.058+0000 7fc51f000640 1 -- 172.21.15.45:0/1617835037 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5181a3630 con 0x7fc518073a40 2024-03-28T18:01:08.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.058+0000 7fc51f000640 1 -- 172.21.15.45:0/1617835037 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5181aaec0 con 0x7fc518073a40 2024-03-28T18:01:08.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.061+0000 7fc523267740 1 Processor -- start 2024-03-28T18:01:08.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.061+0000 7fc516000640 1 -- 172.21.15.45:0/1617835037 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fc50c0079b0 con 0x7fc518073a40 2024-03-28T18:01:08.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.062+0000 7fc523267740 1 -- start start 2024-03-28T18:01:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.062+0000 7fc523267740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555755e88dd0 0x555755e87b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.062+0000 7fc523267740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555755e880f0 0x555755e88570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.062+0000 7fc523267740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e91460 0x555755e93850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.062+0000 7fc523267740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555755dec250 con 0x555755e880f0 2024-03-28T18:01:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.062+0000 7fc523267740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x555755d94a30 con 0x555755e91460 2024-03-28T18:01:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.062+0000 7fc523267740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x555755de82f0 con 0x555755e88dd0 2024-03-28T18:01:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.062+0000 7fc51d200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e91460 0x555755e93850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.062+0000 7fc51dc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555755e880f0 0x555755e88570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.062+0000 7fc51d200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e91460 0x555755e93850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53756/0 (socket says 172.21.15.45:53756) 2024-03-28T18:01:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.062+0000 7fc51d200640 1 -- 172.21.15.45:0/969051809 learned_addr learned my addr 172.21.15.45:0/969051809 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:01:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.063+0000 7fc51e600640 1 --2- 172.21.15.45:0/969051809 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555755e88dd0 0x555755e87b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:08.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.063+0000 7fc51d200640 1 -- 172.21.15.45:0/969051809 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555755e88dd0 msgr2=0x555755e87b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:01:08.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.063+0000 7fc51d200640 1 --2- 172.21.15.45:0/969051809 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555755e88dd0 0x555755e87b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.063+0000 7fc51d200640 1 -- 172.21.15.45:0/969051809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555755e880f0 msgr2=0x555755e88570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:01:08.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.063+0000 7fc51d200640 1 --2- 172.21.15.45:0/969051809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555755e880f0 0x555755e88570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.063+0000 7fc51d200640 1 -- 172.21.15.45:0/969051809 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555755d7f690 con 0x555755e91460 2024-03-28T18:01:08.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.063+0000 7fc51e600640 1 --2- 172.21.15.45:0/969051809 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555755e88dd0 0x555755e87b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:01:08.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.063+0000 7fc516000640 1 --2- 172.21.15.45:0/1617835037 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fc5000a29f0 0x7fc5000a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:08.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.063+0000 7fc51d200640 1 --2- 172.21.15.45:0/969051809 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e91460 0x555755e93850 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fc50400ac20 tx=0x7fc50400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c92fff87b7a864b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:08.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.063+0000 7fc516000640 1 -- 172.21.15.45:0/1617835037 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 11780+0+0 (secure 0 0 0) 0x7fc50c0cf7e0 con 0x7fc518073a40 2024-03-28T18:01:08.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.063+0000 7fc50aa00640 1 -- 172.21.15.45:0/969051809 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50401d070 con 0x555755e91460 2024-03-28T18:01:08.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.063+0000 7fc50aa00640 1 -- 172.21.15.45:0/969051809 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc504018b20 con 0x555755e91460 2024-03-28T18:01:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.063+0000 7fc50aa00640 1 -- 172.21.15.45:0/969051809 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc504010b20 con 0x555755e91460 2024-03-28T18:01:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.063+0000 7fc51e600640 1 --2- 172.21.15.45:0/1617835037 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fc5000a29f0 0x7fc5000a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.064+0000 7fc523267740 1 -- 172.21.15.45:0/969051809 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e91460 msgr2=0x555755e93850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.064+0000 7fc523267740 1 --2- 172.21.15.45:0/969051809 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e91460 0x555755e93850 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fc50400ac20 tx=0x7fc50400f790 comp rx=0 tx=0).stop 2024-03-28T18:01:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.064+0000 7fc51e600640 1 --2- 172.21.15.45:0/1617835037 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fc5000a29f0 0x7fc5000a4eb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc51002c6c0 tx=0x7fc51002e040 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.064+0000 7fc523267740 1 -- 172.21.15.45:0/969051809 shutdown_connections 2024-03-28T18:01:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.064+0000 7fc523267740 1 --2- 172.21.15.45:0/969051809 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e91460 0x555755e93850 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.064+0000 7fc523267740 1 --2- 172.21.15.45:0/969051809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555755e880f0 0x555755e88570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.064+0000 7fc523267740 1 --2- 172.21.15.45:0/969051809 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555755e88dd0 0x555755e87b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.064+0000 7fc523267740 1 -- 172.21.15.45:0/969051809 >> 172.21.15.45:0/969051809 conn(0x555755e4f200 msgr2=0x555755e82b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.065+0000 7fc523267740 1 -- 172.21.15.45:0/969051809 shutdown_connections 2024-03-28T18:01:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.065+0000 7fc523267740 1 -- 172.21.15.45:0/969051809 wait complete. 2024-03-28T18:01:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.065+0000 7fc523267740 1 -- start start 2024-03-28T18:01:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.065+0000 7fc523267740 1 Processor -- start 2024-03-28T18:01:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.066+0000 7fc523267740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555755e880f0 0x555755e90ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.066+0000 7fc523267740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555755e88dd0 0x555755e8de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.066+0000 7fc523267740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e91460 0x555755e8e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.066+0000 7fc523267740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555755e942d0 con 0x555755e880f0 2024-03-28T18:01:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.066+0000 7fc523267740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x555755da7e80 con 0x555755e91460 2024-03-28T18:01:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.066+0000 7fc523267740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x555755e94450 con 0x555755e88dd0 2024-03-28T18:01:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.066+0000 7fc523267740 10 client.0 _ll_register_callbacks cb 0x555755dba130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:01:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.066+0000 7fc51e600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e91460 0x555755e8e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.066+0000 7fc51dc00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555755e880f0 0x555755e90ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.066+0000 7fc51d200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555755e88dd0 0x555755e8de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.066+0000 7fc51e600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e91460 0x555755e8e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53768/0 (socket says 172.21.15.45:53768) 2024-03-28T18:01:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.066+0000 7fc51e600640 1 -- 172.21.15.45:0/1304638146 learned_addr learned my addr 172.21.15.45:0/1304638146 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:01:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.066+0000 7fc51d200640 1 -- 172.21.15.45:0/1304638146 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e91460 msgr2=0x555755e8e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:01:08.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.066+0000 7fc51d200640 1 --2- 172.21.15.45:0/1304638146 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e91460 0x555755e8e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.066+0000 7fc51d200640 1 -- 172.21.15.45:0/1304638146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555755e880f0 msgr2=0x555755e90ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:01:08.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.066+0000 7fc51d200640 1 --2- 172.21.15.45:0/1304638146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555755e880f0 0x555755e90ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.067+0000 7fc51d200640 1 -- 172.21.15.45:0/1304638146 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555755bca7d0 con 0x555755e88dd0 2024-03-28T18:01:08.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.067+0000 7fc51e600640 1 --2- 172.21.15.45:0/1304638146 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e91460 0x555755e8e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:01:08.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.067+0000 7fc51dc00640 1 --2- 172.21.15.45:0/1304638146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555755e880f0 0x555755e90ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:01:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.067+0000 7fc51d200640 1 --2- 172.21.15.45:0/1304638146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555755e88dd0 0x555755e8de70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc504005910 tx=0x7fc504018670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e9b4416ecba26d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.067+0000 7fc4ff400640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-28T18:01:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.067+0000 7fc4ff400640 1 -- 172.21.15.45:0/1304638146 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc504027020 con 0x555755e88dd0 2024-03-28T18:01:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.067+0000 7fc523267740 1 -- 172.21.15.45:0/1304638146 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc504018b20 con 0x555755e88dd0 2024-03-28T18:01:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.067+0000 7fc4ff400640 1 -- 172.21.15.45:0/1304638146 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(136..136 src has 1..136) v4 ==== 11780+0+0 (secure 0 0 0) 0x7fc504010860 con 0x555755e88dd0 2024-03-28T18:01:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.068+0000 7fc4ff400640 1 -- 172.21.15.45:0/1304638146 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=271) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc50401d040 con 0x555755e88dd0 2024-03-28T18:01:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.068+0000 7fc523267740 10 client.26383 fetch_fsmap learned FSMap version 271 2024-03-28T18:01:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.068+0000 7fc523267740 1 -- 172.21.15.45:0/1304638146 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=271,monmap=4+}) v3 -- 0x555755b003f0 con 0x555755e88dd0 2024-03-28T18:01:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.068+0000 7fc4ff400640 1 -- 172.21.15.45:0/1304638146 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc504018b20 con 0x555755e88dd0 2024-03-28T18:01:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.068+0000 7fc4ff400640 1 -- 172.21.15.45:0/1304638146 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 271) v1 ==== 4472+0+0 (secure 0 0 0) 0x7fc5040046c0 con 0x555755e88dd0 2024-03-28T18:01:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.068+0000 7fc523267740 10 client.26383 fetch_fsmap finished waiting for FSMap version 271 2024-03-28T18:01:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.068+0000 7fc523267740 10 client.26383 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:01:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.068+0000 7fc523267740 20 client.26383 populate_metadata read hostname 'smithi045' 2024-03-28T18:01:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.068+0000 7fc523267740 1 --2- 172.21.15.45:0/1304638146 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555755ea14a0 0x555755ea3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.068+0000 7fc523267740 4 client.26383 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:01:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.068+0000 7fc523267740 1 -- 172.21.15.45:0/1304638146 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555755df9790 con 0x555755ea14a0 2024-03-28T18:01:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.068+0000 7fc51dc00640 1 --2- 172.21.15.45:0/1304638146 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555755ea14a0 0x555755ea3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.069+0000 7fc51dc00640 1 --2- 172.21.15.45:0/1304638146 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555755ea14a0 0x555755ea3940 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=159945172ca9571a server_cookie=7e9d0eedf80be4ea in_seq=0 out_seq=0 2024-03-28T18:01:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.069+0000 7fc4ff400640 10 client.26383 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:01:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.070+0000 7fc4ff400640 1 -- 172.21.15.45:0/1304638146 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555755df9790 con 0x555755ea14a0 2024-03-28T18:01:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.070+0000 7fc4ff400640 10 client.26383 handle_command_reply: tid=0 2024-03-28T18:01:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.070+0000 7fc523267740 1 client.26383 shutdown 2024-03-28T18:01:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.070+0000 7fc523267740 1 -- 172.21.15.45:0/1304638146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555755e88dd0 msgr2=0x555755e8de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.070+0000 7fc523267740 1 --2- 172.21.15.45:0/1304638146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555755e88dd0 0x555755e8de70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc504005910 tx=0x7fc504018670 comp rx=0 tx=0).stop 2024-03-28T18:01:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.070+0000 7fc523267740 1 -- 172.21.15.45:0/1304638146 shutdown_connections 2024-03-28T18:01:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.070+0000 7fc523267740 1 --2- 172.21.15.45:0/1304638146 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555755ea14a0 0x555755ea3940 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.070+0000 7fc523267740 1 --2- 172.21.15.45:0/1304638146 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e91460 0x555755e8e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.070+0000 7fc523267740 1 --2- 172.21.15.45:0/1304638146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555755e88dd0 0x555755e8de70 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.070+0000 7fc4ff400640 0 client.26383 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:01:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.070+0000 7fc523267740 1 --2- 172.21.15.45:0/1304638146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555755e880f0 0x555755e90ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.070+0000 7fc523267740 1 -- 172.21.15.45:0/1304638146 >> 172.21.15.45:0/1304638146 conn(0x555755e4f200 msgr2=0x555755e84890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.070+0000 7fc523267740 1 -- 172.21.15.45:0/1304638146 shutdown_connections 2024-03-28T18:01:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.070+0000 7fc523267740 1 -- 172.21.15.45:0/1304638146 wait complete. 2024-03-28T18:01:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.070+0000 7fc523267740 20 client.26383 trim_cache size 0 max 16384 2024-03-28T18:01:08.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.088+0000 7fc523267740 1 Processor -- start 2024-03-28T18:01:08.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.088+0000 7fc523267740 1 -- start start 2024-03-28T18:01:08.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.088+0000 7fc523267740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555755e880f0 0x555755e905a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:08.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.088+0000 7fc523267740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e88dd0 0x555755e90ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:08.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.088+0000 7fc523267740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555755e91460 0x555755e92850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:08.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.088+0000 7fc523267740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5181b5ce0 con 0x555755e880f0 2024-03-28T18:01:08.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.088+0000 7fc523267740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5181b4360 con 0x555755e88dd0 2024-03-28T18:01:08.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.088+0000 7fc523267740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5181b5680 con 0x555755e91460 2024-03-28T18:01:08.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.089+0000 7fc51d200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e88dd0 0x555755e90ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:08.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.089+0000 7fc51d200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e88dd0 0x555755e90ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53774/0 (socket says 172.21.15.45:53774) 2024-03-28T18:01:08.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.089+0000 7fc51d200640 1 -- 172.21.15.45:0/836326422 learned_addr learned my addr 172.21.15.45:0/836326422 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:01:08.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.089+0000 7fc51e600640 1 --2- 172.21.15.45:0/836326422 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555755e91460 0x555755e92850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:08.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.089+0000 7fc51dc00640 1 --2- 172.21.15.45:0/836326422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555755e880f0 0x555755e905a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:08.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.089+0000 7fc51d200640 1 -- 172.21.15.45:0/836326422 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555755e91460 msgr2=0x555755e92850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:01:08.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.089+0000 7fc51d200640 1 --2- 172.21.15.45:0/836326422 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555755e91460 0x555755e92850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.089+0000 7fc51d200640 1 -- 172.21.15.45:0/836326422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555755e880f0 msgr2=0x555755e905a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:01:08.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.089+0000 7fc51d200640 1 --2- 172.21.15.45:0/836326422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555755e880f0 0x555755e905a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.089+0000 7fc51d200640 1 -- 172.21.15.45:0/836326422 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555755ac00f0 con 0x555755e88dd0 2024-03-28T18:01:08.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.089+0000 7fc51dc00640 1 --2- 172.21.15.45:0/836326422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555755e880f0 0x555755e905a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:01:08.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.089+0000 7fc51d200640 1 --2- 172.21.15.45:0/836326422 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e88dd0 0x555755e90ae0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc50400a590 tx=0x7fc50400fdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=364b3eceb2e4cfd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:08.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.090+0000 7fc50a000640 1 -- 172.21.15.45:0/836326422 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc504027040 con 0x555755e88dd0 2024-03-28T18:01:08.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.090+0000 7fc50a000640 1 -- 172.21.15.45:0/836326422 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc50400a630 con 0x555755e88dd0 2024-03-28T18:01:08.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.090+0000 7fc50a000640 1 -- 172.21.15.45:0/836326422 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5040107a0 con 0x555755e88dd0 2024-03-28T18:01:08.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.090+0000 7fc523267740 1 -- 172.21.15.45:0/836326422 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e88dd0 msgr2=0x555755e90ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:08.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.090+0000 7fc523267740 1 --2- 172.21.15.45:0/836326422 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e88dd0 0x555755e90ae0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc50400a590 tx=0x7fc50400fdc0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.090+0000 7fc523267740 1 -- 172.21.15.45:0/836326422 shutdown_connections 2024-03-28T18:01:08.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.090+0000 7fc523267740 1 --2- 172.21.15.45:0/836326422 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555755e91460 0x555755e92850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.090+0000 7fc523267740 1 --2- 172.21.15.45:0/836326422 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e88dd0 0x555755e90ae0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.090+0000 7fc523267740 1 --2- 172.21.15.45:0/836326422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555755e880f0 0x555755e905a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.090+0000 7fc523267740 1 -- 172.21.15.45:0/836326422 >> 172.21.15.45:0/836326422 conn(0x555755e4f200 msgr2=0x555755e8de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:08.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.090+0000 7fc523267740 1 -- 172.21.15.45:0/836326422 shutdown_connections 2024-03-28T18:01:08.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.090+0000 7fc523267740 1 -- 172.21.15.45:0/836326422 wait complete. 2024-03-28T18:01:08.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.091+0000 7fc523267740 1 -- start start 2024-03-28T18:01:08.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.091+0000 7fc523267740 1 Processor -- start 2024-03-28T18:01:08.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.091+0000 7fc523267740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e880f0 0x555755e946d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:08.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.091+0000 7fc523267740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555755e88dd0 0x555755e94c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:08.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.091+0000 7fc523267740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555755e91460 0x555755efae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:08.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.091+0000 7fc523267740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5181b5020 con 0x555755e88dd0 2024-03-28T18:01:08.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.091+0000 7fc523267740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5181b49c0 con 0x555755e880f0 2024-03-28T18:01:08.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.091+0000 7fc523267740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5181b33a0 con 0x555755e91460 2024-03-28T18:01:08.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.092+0000 7fc523267740 10 client.0 _ll_register_callbacks cb 0x7fc50401b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:01:08.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.092+0000 7fc51dc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e880f0 0x555755e946d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:08.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.092+0000 7fc51dc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e880f0 0x555755e946d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:53784/0 (socket says 172.21.15.45:53784) 2024-03-28T18:01:08.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.092+0000 7fc51dc00640 1 -- 172.21.15.45:0/2671192171 learned_addr learned my addr 172.21.15.45:0/2671192171 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:01:08.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.092+0000 7fc51d200640 1 --2- 172.21.15.45:0/2671192171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555755e88dd0 0x555755e94c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:08.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.092+0000 7fc51e600640 1 --2- 172.21.15.45:0/2671192171 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555755e91460 0x555755efae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:08.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.092+0000 7fc51dc00640 1 -- 172.21.15.45:0/2671192171 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555755e91460 msgr2=0x555755efae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:01:08.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.092+0000 7fc51dc00640 1 --2- 172.21.15.45:0/2671192171 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555755e91460 0x555755efae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.092+0000 7fc51dc00640 1 -- 172.21.15.45:0/2671192171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555755e88dd0 msgr2=0x555755e94c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:01:08.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.092+0000 7fc51dc00640 1 --2- 172.21.15.45:0/2671192171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555755e88dd0 0x555755e94c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.092+0000 7fc51dc00640 1 -- 172.21.15.45:0/2671192171 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555755efb340 con 0x555755e880f0 2024-03-28T18:01:08.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.092+0000 7fc51d200640 1 --2- 172.21.15.45:0/2671192171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555755e88dd0 0x555755e94c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:01:08.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.092+0000 7fc51dc00640 1 --2- 172.21.15.45:0/2671192171 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e880f0 0x555755e946d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc50c005840 tx=0x7fc50c0d1c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7df19f030c70f5a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:08.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.092+0000 7fc4ff400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T18:01:08.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.093+0000 7fc4ff400640 1 -- 172.21.15.45:0/2671192171 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c01d040 con 0x555755e880f0 2024-03-28T18:01:08.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.093+0000 7fc4ff400640 1 -- 172.21.15.45:0/2671192171 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(136..136 src has 1..136) v4 ==== 11780+0+0 (secure 0 0 0) 0x7fc50c0cfbd0 con 0x555755e880f0 2024-03-28T18:01:08.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.093+0000 7fc523267740 1 -- 172.21.15.45:0/2671192171 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc50400a630 con 0x555755e880f0 2024-03-28T18:01:08.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.093+0000 7fc4ff400640 1 -- 172.21.15.45:0/2671192171 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=271) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc50c09aef0 con 0x555755e880f0 2024-03-28T18:01:08.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.093+0000 7fc523267740 10 client.26471 fetch_fsmap learned FSMap version 271 2024-03-28T18:01:08.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.093+0000 7fc523267740 1 -- 172.21.15.45:0/2671192171 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=271,monmap=4+}) v3 -- 0x555755efc470 con 0x555755e880f0 2024-03-28T18:01:08.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.093+0000 7fc4ff400640 1 -- 172.21.15.45:0/2671192171 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc50400a630 con 0x555755e880f0 2024-03-28T18:01:08.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.093+0000 7fc4ff400640 1 -- 172.21.15.45:0/2671192171 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 271) v1 ==== 4472+0+0 (secure 0 0 0) 0x7fc504019ce0 con 0x555755e880f0 2024-03-28T18:01:08.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.093+0000 7fc523267740 10 client.26471 fetch_fsmap finished waiting for FSMap version 271 2024-03-28T18:01:08.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.093+0000 7fc523267740 10 client.26471 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:01:08.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.093+0000 7fc523267740 20 client.26471 populate_metadata read hostname 'smithi045' 2024-03-28T18:01:08.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.093+0000 7fc523267740 1 --2- 172.21.15.45:0/2671192171 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555755efcaf0 0x555755efeee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:08.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.093+0000 7fc523267740 4 client.26471 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:01:08.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.093+0000 7fc523267740 1 -- 172.21.15.45:0/2671192171 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555755df3ae0 con 0x555755efcaf0 2024-03-28T18:01:08.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.093+0000 7fc51d200640 1 --2- 172.21.15.45:0/2671192171 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555755efcaf0 0x555755efeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:08.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.094+0000 7fc51d200640 1 --2- 172.21.15.45:0/2671192171 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555755efcaf0 0x555755efeee0 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3b2564a13b88b16 server_cookie=96c23a91f8200a63 in_seq=0 out_seq=0 2024-03-28T18:01:08.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.094+0000 7fc4ff400640 10 client.26471 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:01:08.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.094+0000 7fc4ff400640 1 -- 172.21.15.45:0/2671192171 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x555755df3ae0 con 0x555755efcaf0 2024-03-28T18:01:08.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.094+0000 7fc4ff400640 10 client.26471 handle_command_reply: tid=0 2024-03-28T18:01:08.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.094+0000 7fc523267740 1 client.26471 shutdown 2024-03-28T18:01:08.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.094+0000 7fc523267740 1 -- 172.21.15.45:0/2671192171 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e880f0 msgr2=0x555755e946d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:08.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.094+0000 7fc523267740 1 --2- 172.21.15.45:0/2671192171 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e880f0 0x555755e946d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc50c005840 tx=0x7fc50c0d1c30 comp rx=0 tx=0).stop 2024-03-28T18:01:08.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.094+0000 7fc523267740 1 -- 172.21.15.45:0/2671192171 shutdown_connections 2024-03-28T18:01:08.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.094+0000 7fc523267740 1 --2- 172.21.15.45:0/2671192171 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x555755efcaf0 0x555755efeee0 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.094+0000 7fc523267740 1 --2- 172.21.15.45:0/2671192171 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x555755e91460 0x555755efae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.094+0000 7fc4ff400640 0 client.26471 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:01:08.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.094+0000 7fc523267740 1 --2- 172.21.15.45:0/2671192171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555755e88dd0 0x555755e94c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.094+0000 7fc523267740 1 --2- 172.21.15.45:0/2671192171 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555755e880f0 0x555755e946d0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.094+0000 7fc523267740 1 -- 172.21.15.45:0/2671192171 >> 172.21.15.45:0/2671192171 conn(0x555755e4f200 msgr2=0x555755e8c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:08.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.094+0000 7fc523267740 1 -- 172.21.15.45:0/2671192171 shutdown_connections 2024-03-28T18:01:08.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.095+0000 7fc523267740 1 -- 172.21.15.45:0/2671192171 wait complete. 2024-03-28T18:01:08.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.095+0000 7fc523267740 20 client.26471 trim_cache size 0 max 16384 2024-03-28T18:01:08.099 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T18:01:08.100 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:01:08.100 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T18:01:08.100 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T18:01:08.100 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T18:01:08.100 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T18:01:08.100 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T18:01:08.100 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:01:08.100 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:01:08.100 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T18:01:08.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.095+0000 7fc51f000640 1 -- 172.21.15.45:0/1617835037 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fc5000a29f0 msgr2=0x7fc5000a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:08.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.095+0000 7fc51f000640 1 --2- 172.21.15.45:0/1617835037 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fc5000a29f0 0x7fc5000a4eb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc51002c6c0 tx=0x7fc51002e040 comp rx=0 tx=0).stop 2024-03-28T18:01:08.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.095+0000 7fc51f000640 1 -- 172.21.15.45:0/1617835037 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc518073a40 msgr2=0x7fc51819e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:08.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.095+0000 7fc51f000640 1 --2- 172.21.15.45:0/1617835037 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc518073a40 0x7fc51819e3a0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fc50c00c890 tx=0x7fc50c00cc90 comp rx=0 tx=0).stop 2024-03-28T18:01:08.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.095+0000 7fc51f000640 1 -- 172.21.15.45:0/1617835037 shutdown_connections 2024-03-28T18:01:08.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.095+0000 7fc51f000640 1 --2- 172.21.15.45:0/1617835037 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fc5000a29f0 0x7fc5000a4eb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.095+0000 7fc51f000640 1 --2- 172.21.15.45:0/1617835037 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc518112520 0x7fc5181a2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.095+0000 7fc51f000640 1 --2- 172.21.15.45:0/1617835037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc518074360 0x7fc51819e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.095+0000 7fc51f000640 1 --2- 172.21.15.45:0/1617835037 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc518073a40 0x7fc51819e3a0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:08.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.095+0000 7fc51f000640 1 -- 172.21.15.45:0/1617835037 >> 172.21.15.45:0/1617835037 conn(0x7fc5180fd000 msgr2=0x7fc5180fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:08.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.096+0000 7fc51f000640 1 -- 172.21.15.45:0/1617835037 shutdown_connections 2024-03-28T18:01:08.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:08.096+0000 7fc51f000640 1 -- 172.21.15.45:0/1617835037 wait complete. 2024-03-28T18:01:08.113 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T18:01:08.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:08 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/1366698753,v1:172.21.15.45:6837/1366698753] up:rejoin 2024-03-28T18:01:08.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:08 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:08.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:08 smithi071 ceph-mon[36597]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-03-28T18:01:08.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:08 smithi071 ceph-mon[36597]: pgmap v417: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:01:08.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:08 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/1366698753,v1:172.21.15.45:6837/1366698753] up:rejoin 2024-03-28T18:01:08.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:08 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:08.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:08 smithi151 ceph-mon[36481]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-03-28T18:01:08.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:08 smithi151 ceph-mon[36481]: pgmap v417: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:01:08.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/1366698753,v1:172.21.15.45:6837/1366698753] up:rejoin 2024-03-28T18:01:08.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:08.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-mon[31773]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-03-28T18:01:08.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-mon[31773]: pgmap v417: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:01:08.666 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fdb61a00700 time 2024-03-28T18:01:08.313736+0000 2024-03-28T18:01:08.666 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:01:08.666 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:08.666 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fdb6d3e64a1] 2024-03-28T18:01:08.666 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdb6d3e6667] 2024-03-28T18:01:08.666 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55aa90ef9223] 2024-03-28T18:01:08.666 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55aa90d735d6] 2024-03-28T18:01:08.666 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55aa90d73da1] 2024-03-28T18:01:08.666 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55aa90d762d5] 2024-03-28T18:01:08.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55aa90dc6233] 2024-03-28T18:01:08.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55aa90dc8197] 2024-03-28T18:01:08.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55aa90cc711b] 2024-03-28T18:01:08.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55aa90cc76bc] 2024-03-28T18:01:08.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55aa90cb11ff] 2024-03-28T18:01:08.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdb6d678488] 2024-03-28T18:01:08.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 13: (DispatchQueue::entry()+0x51f) [0x7fdb6d67560f] 2024-03-28T18:01:08.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdb6d748e81] 2024-03-28T18:01:08.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fdb6c1631ca] 2024-03-28T18:01:08.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 16: clone() 2024-03-28T18:01:08.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: *** Caught signal (Aborted) ** 2024-03-28T18:01:08.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: in thread 7fdb61a00700 thread_name:ms_dispatch 2024-03-28T18:01:08.668 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 2024-03-28T18:01:08.316+0000 7fdb61a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fdb61a00700 time 2024-03-28T18:01:08.313736+0000 2024-03-28T18:01:08.668 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:01:08.668 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 2024-03-28T18:01:08.668 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:08.668 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fdb6d3e64a1] 2024-03-28T18:01:08.668 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdb6d3e6667] 2024-03-28T18:01:08.668 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55aa90ef9223] 2024-03-28T18:01:08.668 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55aa90d735d6] 2024-03-28T18:01:08.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55aa90d73da1] 2024-03-28T18:01:08.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55aa90d762d5] 2024-03-28T18:01:08.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55aa90dc6233] 2024-03-28T18:01:08.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55aa90dc8197] 2024-03-28T18:01:08.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55aa90cc711b] 2024-03-28T18:01:08.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55aa90cc76bc] 2024-03-28T18:01:08.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55aa90cb11ff] 2024-03-28T18:01:08.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdb6d678488] 2024-03-28T18:01:08.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 13: (DispatchQueue::entry()+0x51f) [0x7fdb6d67560f] 2024-03-28T18:01:08.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdb6d748e81] 2024-03-28T18:01:08.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fdb6c1631ca] 2024-03-28T18:01:08.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 16: clone() 2024-03-28T18:01:08.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 2024-03-28T18:01:08.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:08.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fdb6c16dd20] 2024-03-28T18:01:08.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 2: gsignal() 2024-03-28T18:01:08.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 3: abort() 2024-03-28T18:01:08.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fdb6d3e64fb] 2024-03-28T18:01:08.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdb6d3e6667] 2024-03-28T18:01:08.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55aa90ef9223] 2024-03-28T18:01:08.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55aa90d735d6] 2024-03-28T18:01:08.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55aa90d73da1] 2024-03-28T18:01:08.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55aa90d762d5] 2024-03-28T18:01:08.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55aa90dc6233] 2024-03-28T18:01:08.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55aa90dc8197] 2024-03-28T18:01:08.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55aa90cc711b] 2024-03-28T18:01:08.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55aa90cc76bc] 2024-03-28T18:01:08.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55aa90cb11ff] 2024-03-28T18:01:08.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdb6d678488] 2024-03-28T18:01:08.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 16: (DispatchQueue::entry()+0x51f) [0x7fdb6d67560f] 2024-03-28T18:01:08.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdb6d748e81] 2024-03-28T18:01:08.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fdb6c1631ca] 2024-03-28T18:01:08.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 19: clone() 2024-03-28T18:01:08.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 2024-03-28T18:01:08.319+0000 7fdb61a00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:01:08.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: in thread 7fdb61a00700 thread_name:ms_dispatch 2024-03-28T18:01:08.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 2024-03-28T18:01:08.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:08.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fdb6c16dd20] 2024-03-28T18:01:08.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 2: gsignal() 2024-03-28T18:01:08.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 3: abort() 2024-03-28T18:01:08.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fdb6d3e64fb] 2024-03-28T18:01:08.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdb6d3e6667] 2024-03-28T18:01:08.672 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55aa90ef9223] 2024-03-28T18:01:08.672 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55aa90d735d6] 2024-03-28T18:01:08.672 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55aa90d73da1] 2024-03-28T18:01:08.673 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55aa90d762d5] 2024-03-28T18:01:08.673 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55aa90dc6233] 2024-03-28T18:01:08.673 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55aa90dc8197] 2024-03-28T18:01:08.673 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55aa90cc711b] 2024-03-28T18:01:08.673 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55aa90cc76bc] 2024-03-28T18:01:08.673 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55aa90cb11ff] 2024-03-28T18:01:08.673 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdb6d678488] 2024-03-28T18:01:08.673 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 16: (DispatchQueue::entry()+0x51f) [0x7fdb6d67560f] 2024-03-28T18:01:08.673 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdb6d748e81] 2024-03-28T18:01:08.673 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fdb6c1631ca] 2024-03-28T18:01:08.673 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 19: clone() 2024-03-28T18:01:08.673 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:01:08.673 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 2024-03-28T18:01:08.673 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: -1> 2024-03-28T18:01:08.316+0000 7fdb61a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fdb61a00700 time 2024-03-28T18:01:08.313736+0000 2024-03-28T18:01:08.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:01:08.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 2024-03-28T18:01:08.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:08.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fdb6d3e64a1] 2024-03-28T18:01:08.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdb6d3e6667] 2024-03-28T18:01:08.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55aa90ef9223] 2024-03-28T18:01:08.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55aa90d735d6] 2024-03-28T18:01:08.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55aa90d73da1] 2024-03-28T18:01:08.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55aa90d762d5] 2024-03-28T18:01:08.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55aa90dc6233] 2024-03-28T18:01:08.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55aa90dc8197] 2024-03-28T18:01:08.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55aa90cc711b] 2024-03-28T18:01:08.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55aa90cc76bc] 2024-03-28T18:01:08.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55aa90cb11ff] 2024-03-28T18:01:08.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdb6d678488] 2024-03-28T18:01:08.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 13: (DispatchQueue::entry()+0x51f) [0x7fdb6d67560f] 2024-03-28T18:01:08.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdb6d748e81] 2024-03-28T18:01:08.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fdb6c1631ca] 2024-03-28T18:01:08.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 16: clone() 2024-03-28T18:01:08.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 2024-03-28T18:01:08.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 0> 2024-03-28T18:01:08.319+0000 7fdb61a00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:01:08.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: in thread 7fdb61a00700 thread_name:ms_dispatch 2024-03-28T18:01:08.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 2024-03-28T18:01:08.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:08.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fdb6c16dd20] 2024-03-28T18:01:08.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 2: gsignal() 2024-03-28T18:01:08.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 3: abort() 2024-03-28T18:01:08.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fdb6d3e64fb] 2024-03-28T18:01:08.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdb6d3e6667] 2024-03-28T18:01:08.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55aa90ef9223] 2024-03-28T18:01:08.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55aa90d735d6] 2024-03-28T18:01:08.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55aa90d73da1] 2024-03-28T18:01:08.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55aa90d762d5] 2024-03-28T18:01:08.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55aa90dc6233] 2024-03-28T18:01:08.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55aa90dc8197] 2024-03-28T18:01:08.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55aa90cc711b] 2024-03-28T18:01:08.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55aa90cc76bc] 2024-03-28T18:01:08.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55aa90cb11ff] 2024-03-28T18:01:08.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdb6d678488] 2024-03-28T18:01:08.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 16: (DispatchQueue::entry()+0x51f) [0x7fdb6d67560f] 2024-03-28T18:01:08.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdb6d748e81] 2024-03-28T18:01:08.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fdb6c1631ca] 2024-03-28T18:01:08.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 19: clone() 2024-03-28T18:01:08.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:01:08.677 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 2024-03-28T18:01:08.677 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: -1> 2024-03-28T18:01:08.316+0000 7fdb61a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fdb61a00700 time 2024-03-28T18:01:08.313736+0000 2024-03-28T18:01:08.677 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:01:08.677 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 2024-03-28T18:01:08.677 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:08.677 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fdb6d3e64a1] 2024-03-28T18:01:08.677 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdb6d3e6667] 2024-03-28T18:01:08.677 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55aa90ef9223] 2024-03-28T18:01:08.677 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55aa90d735d6] 2024-03-28T18:01:08.677 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55aa90d73da1] 2024-03-28T18:01:08.677 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55aa90d762d5] 2024-03-28T18:01:08.677 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55aa90dc6233] 2024-03-28T18:01:08.677 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55aa90dc8197] 2024-03-28T18:01:08.677 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55aa90cc711b] 2024-03-28T18:01:08.677 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55aa90cc76bc] 2024-03-28T18:01:08.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55aa90cb11ff] 2024-03-28T18:01:08.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdb6d678488] 2024-03-28T18:01:08.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 13: (DispatchQueue::entry()+0x51f) [0x7fdb6d67560f] 2024-03-28T18:01:08.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdb6d748e81] 2024-03-28T18:01:08.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fdb6c1631ca] 2024-03-28T18:01:08.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 16: clone() 2024-03-28T18:01:08.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 2024-03-28T18:01:08.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 0> 2024-03-28T18:01:08.319+0000 7fdb61a00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:01:08.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: in thread 7fdb61a00700 thread_name:ms_dispatch 2024-03-28T18:01:08.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 2024-03-28T18:01:08.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:08.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fdb6c16dd20] 2024-03-28T18:01:08.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 2: gsignal() 2024-03-28T18:01:08.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 3: abort() 2024-03-28T18:01:08.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fdb6d3e64fb] 2024-03-28T18:01:08.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdb6d3e6667] 2024-03-28T18:01:08.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55aa90ef9223] 2024-03-28T18:01:08.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55aa90d735d6] 2024-03-28T18:01:08.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55aa90d73da1] 2024-03-28T18:01:08.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55aa90d762d5] 2024-03-28T18:01:08.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55aa90dc6233] 2024-03-28T18:01:08.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55aa90dc8197] 2024-03-28T18:01:08.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55aa90cc711b] 2024-03-28T18:01:08.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55aa90cc76bc] 2024-03-28T18:01:08.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55aa90cb11ff] 2024-03-28T18:01:08.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdb6d678488] 2024-03-28T18:01:08.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 16: (DispatchQueue::entry()+0x51f) [0x7fdb6d67560f] 2024-03-28T18:01:08.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdb6d748e81] 2024-03-28T18:01:08.680 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fdb6c1631ca] 2024-03-28T18:01:08.680 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 19: clone() 2024-03-28T18:01:08.680 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:01:08.680 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94397]: 2024-03-28T18:01:08.680 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:08 smithi045 podman[94661]: 2024-03-28 18:01:08.42099009 +0000 UTC m=+0.033645352 container died 6679fa2265c0175a484e989cec5c2d9361855a6799504668d96e87bf7c0a8c44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-28T18:01:08.926 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 521 sec latency 636822.954 ms 2024-03-28T18:01:09.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:09 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:01:09.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:09 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/1366698753,v1:172.21.15.45:6837/1366698753] up:active 2024-03-28T18:01:09.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:09 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:09.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:09 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 3 up:standby-replay 2024-03-28T18:01:09.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:09 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:01:09.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:09 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/1366698753,v1:172.21.15.45:6837/1366698753] up:active 2024-03-28T18:01:09.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:09 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:09.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:09 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 3 up:standby-replay 2024-03-28T18:01:09.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:09 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:01:09.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:09 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/1366698753,v1:172.21.15.45:6837/1366698753] up:active 2024-03-28T18:01:09.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:09 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:09.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:09 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 3 up:standby-replay 2024-03-28T18:01:09.665 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:09 smithi045 podman[94661]: 2024-03-28 18:01:09.312608204 +0000 UTC m=+0.925263489 container remove 6679fa2265c0175a484e989cec5c2d9361855a6799504668d96e87bf7c0a8c44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:01:09.666 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:09 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:01:09.926 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 522 sec latency 637823.048 ms 2024-03-28T18:01:10.165 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:09 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-28T18:01:10.165 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:09 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Consumed 1.575s CPU time. 2024-03-28T18:01:10.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:10 smithi071 ceph-mon[36597]: pgmap v418: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:01:10.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:10 smithi151 ceph-mon[36481]: pgmap v418: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:01:10.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:10 smithi045 ceph-mon[31773]: pgmap v418: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:01:10.926 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 523 sec latency 638823.187 ms 2024-03-28T18:01:11.926 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 524 sec latency 639823.330 ms 2024-03-28T18:01:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:12 smithi045 ceph-mon[31773]: pgmap v419: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 29 op/s 2024-03-28T18:01:12.926 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 525 sec latency 640823.493 ms 2024-03-28T18:01:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:12 smithi071 ceph-mon[36597]: pgmap v419: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 29 op/s 2024-03-28T18:01:13.076 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:12 smithi151 ceph-mon[36481]: pgmap v419: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 29 op/s 2024-03-28T18:01:13.926 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 526 sec latency 641823.567 ms 2024-03-28T18:01:14.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:14 smithi045 ceph-mon[31773]: pgmap v420: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 29 op/s 2024-03-28T18:01:14.926 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 527 sec latency 642823.625 ms 2024-03-28T18:01:15.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:14 smithi071 ceph-mon[36597]: pgmap v420: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 29 op/s 2024-03-28T18:01:15.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:14 smithi151 ceph-mon[36481]: pgmap v420: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 29 op/s 2024-03-28T18:01:15.927 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 528 sec latency 643823.739 ms 2024-03-28T18:01:16.911 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:16 smithi045 ceph-mon[31773]: pgmap v421: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-03-28T18:01:16.927 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 529 sec latency 644823.921 ms 2024-03-28T18:01:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:16 smithi071 ceph-mon[36597]: pgmap v421: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-03-28T18:01:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:16 smithi151 ceph-mon[36481]: pgmap v421: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-03-28T18:01:17.927 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 530 sec latency 645824.015 ms 2024-03-28T18:01:18.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:18 smithi045 ceph-mon[31773]: pgmap v422: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T18:01:18.927 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 531 sec latency 646824.131 ms 2024-03-28T18:01:19.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:18 smithi071 ceph-mon[36597]: pgmap v422: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T18:01:19.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:18 smithi151 ceph-mon[36481]: pgmap v422: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T18:01:19.927 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 532 sec latency 647824.253 ms 2024-03-28T18:01:20.329 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:19 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 3. 2024-03-28T18:01:20.329 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:19 smithi045 systemd[1]: Stopped Ceph mds.g for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:01:20.330 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:19 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Consumed 1.575s CPU time. 2024-03-28T18:01:20.330 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:19 smithi045 systemd[1]: Starting Ceph mds.g for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T18:01:20.613 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:20 smithi045 podman[94762]: 2024-03-28 18:01:20.234384705 +0000 UTC m=+0.036220532 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:01:20.613 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:20 smithi045 podman[94762]: 2024-03-28 18:01:20.488155472 +0000 UTC m=+0.289991325 container create e8ad78db5928a4ade14c82b252e34bcd2c6301a07f542c91d6dd280701e231c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-28T18:01:20.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:20 smithi045 ceph-mon[31773]: pgmap v423: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T18:01:20.916 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:20 smithi045 podman[94762]: 2024-03-28 18:01:20.792365655 +0000 UTC m=+0.594201528 container init e8ad78db5928a4ade14c82b252e34bcd2c6301a07f542c91d6dd280701e231c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:01:20.916 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:20 smithi045 podman[94762]: 2024-03-28 18:01:20.803727037 +0000 UTC m=+0.605562862 container start e8ad78db5928a4ade14c82b252e34bcd2c6301a07f542c91d6dd280701e231c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-28T18:01:20.916 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:20 smithi045 ceph-mds[94780]: -- 172.21.15.45:0/1323152397 <== mon.0 v2:172.21.15.45:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b5f6263a40 con 0x55b5f6237000 2024-03-28T18:01:20.916 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:20 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: starting mds.g at 2024-03-28T18:01:20.927 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 533 sec latency 648824.377 ms 2024-03-28T18:01:21.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:20 smithi071 ceph-mon[36597]: pgmap v423: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T18:01:21.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:20 smithi151 ceph-mon[36481]: pgmap v423: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T18:01:21.416 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:20 smithi045 bash[94762]: e8ad78db5928a4ade14c82b252e34bcd2c6301a07f542c91d6dd280701e231c6 2024-03-28T18:01:21.416 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:20 smithi045 systemd[1]: Started Ceph mds.g for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:01:21.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:21 smithi045 ceph-mon[31773]: daemon mds.g restarted 2024-03-28T18:01:21.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:21 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:01:21.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:21 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:01:21.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:21 smithi045 ceph-mon[31773]: osdmap e137: 12 total, 12 up, 12 in 2024-03-28T18:01:21.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:21 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/1322964071,v1:172.21.15.45:6837/1322964071] up:boot 2024-03-28T18:01:21.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:21 smithi045 ceph-mon[31773]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-03-28T18:01:21.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:21 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:01:21.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:21 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:21.927 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 534 sec latency 649824.505 ms 2024-03-28T18:01:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:21 smithi071 ceph-mon[36597]: daemon mds.g restarted 2024-03-28T18:01:22.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:21 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:01:22.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:21 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:01:22.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:21 smithi071 ceph-mon[36597]: osdmap e137: 12 total, 12 up, 12 in 2024-03-28T18:01:22.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:21 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/1322964071,v1:172.21.15.45:6837/1322964071] up:boot 2024-03-28T18:01:22.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:21 smithi071 ceph-mon[36597]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-03-28T18:01:22.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:21 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:01:22.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:21 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:21 smithi151 ceph-mon[36481]: daemon mds.g restarted 2024-03-28T18:01:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:21 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:01:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:21 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:01:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:21 smithi151 ceph-mon[36481]: osdmap e137: 12 total, 12 up, 12 in 2024-03-28T18:01:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:21 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/1322964071,v1:172.21.15.45:6837/1322964071] up:boot 2024-03-28T18:01:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:21 smithi151 ceph-mon[36481]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-03-28T18:01:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:21 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:01:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:21 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:22.927 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 535 sec latency 650824.634 ms 2024-03-28T18:01:23.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:22 smithi045 ceph-mon[31773]: pgmap v425: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:01:23.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:22 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/1173454637,v1:172.21.15.45:6835/1173454637] up:resolve 2024-03-28T18:01:23.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:22 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:23.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:22 smithi071 ceph-mon[36597]: pgmap v425: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:01:23.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:22 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/1173454637,v1:172.21.15.45:6835/1173454637] up:resolve 2024-03-28T18:01:23.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:22 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:23.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:22 smithi151 ceph-mon[36481]: pgmap v425: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:01:23.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:22 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/1173454637,v1:172.21.15.45:6835/1173454637] up:resolve 2024-03-28T18:01:23.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:22 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:23.928 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 536 sec latency 651824.800 ms 2024-03-28T18:01:24.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:23 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:01:24.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:23 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/1173454637,v1:172.21.15.45:6835/1173454637] up:reconnect 2024-03-28T18:01:24.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:23 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:24.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:23 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T18:01:24.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:23 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:01:24.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:23 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/1173454637,v1:172.21.15.45:6835/1173454637] up:reconnect 2024-03-28T18:01:24.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:23 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:24.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:23 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T18:01:24.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:23 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:01:24.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:23 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/1173454637,v1:172.21.15.45:6835/1173454637] up:reconnect 2024-03-28T18:01:24.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:23 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:24.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:23 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T18:01:24.928 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 537 sec latency 652824.886 ms 2024-03-28T18:01:25.197 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-mon[31773]: pgmap v426: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:01:25.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/1173454637,v1:172.21.15.45:6835/1173454637] up:rejoin 2024-03-28T18:01:25.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:25.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-mon[31773]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-03-28T18:01:25.198 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f62a5800700 time 2024-03-28T18:01:24.932604+0000 2024-03-28T18:01:25.198 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:01:25.199 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:25.199 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f62b11af4a1] 2024-03-28T18:01:25.199 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f62b11af667] 2024-03-28T18:01:25.199 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ff4810c223] 2024-03-28T18:01:25.199 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ff47f865d6] 2024-03-28T18:01:25.199 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ff47f86da1] 2024-03-28T18:01:25.199 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ff47f892d5] 2024-03-28T18:01:25.199 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ff47fd9233] 2024-03-28T18:01:25.199 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff47fdb197] 2024-03-28T18:01:25.199 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ff47eda11b] 2024-03-28T18:01:25.199 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff47eda6bc] 2024-03-28T18:01:25.199 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ff47ec41ff] 2024-03-28T18:01:25.199 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f62b1441488] 2024-03-28T18:01:25.200 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 13: (DispatchQueue::entry()+0x51f) [0x7f62b143e60f] 2024-03-28T18:01:25.200 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f62b1511e81] 2024-03-28T18:01:25.200 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f62aff2c1ca] 2024-03-28T18:01:25.200 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 16: clone() 2024-03-28T18:01:25.200 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 2024-03-28T18:01:24.933+0000 7f62a5800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f62a5800700 time 2024-03-28T18:01:24.932604+0000 2024-03-28T18:01:25.200 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:01:25.200 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 2024-03-28T18:01:25.200 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:25.201 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f62b11af4a1] 2024-03-28T18:01:25.201 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f62b11af667] 2024-03-28T18:01:25.201 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ff4810c223] 2024-03-28T18:01:25.201 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ff47f865d6] 2024-03-28T18:01:25.201 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ff47f86da1] 2024-03-28T18:01:25.201 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ff47f892d5] 2024-03-28T18:01:25.201 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ff47fd9233] 2024-03-28T18:01:25.201 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff47fdb197] 2024-03-28T18:01:25.201 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ff47eda11b] 2024-03-28T18:01:25.201 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff47eda6bc] 2024-03-28T18:01:25.201 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ff47ec41ff] 2024-03-28T18:01:25.201 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f62b1441488] 2024-03-28T18:01:25.202 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 13: (DispatchQueue::entry()+0x51f) [0x7f62b143e60f] 2024-03-28T18:01:25.202 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f62b1511e81] 2024-03-28T18:01:25.202 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f62aff2c1ca] 2024-03-28T18:01:25.202 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 16: clone() 2024-03-28T18:01:25.202 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 2024-03-28T18:01:25.202 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: *** Caught signal (Aborted) ** 2024-03-28T18:01:25.202 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: in thread 7f62a5800700 thread_name:ms_dispatch 2024-03-28T18:01:25.202 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:25.202 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f62aff36d20] 2024-03-28T18:01:25.203 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 2: gsignal() 2024-03-28T18:01:25.203 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 3: abort() 2024-03-28T18:01:25.203 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f62b11af4fb] 2024-03-28T18:01:25.203 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f62b11af667] 2024-03-28T18:01:25.203 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ff4810c223] 2024-03-28T18:01:25.203 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ff47f865d6] 2024-03-28T18:01:25.203 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ff47f86da1] 2024-03-28T18:01:25.203 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ff47f892d5] 2024-03-28T18:01:25.203 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ff47fd9233] 2024-03-28T18:01:25.203 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff47fdb197] 2024-03-28T18:01:25.204 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ff47eda11b] 2024-03-28T18:01:25.204 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff47eda6bc] 2024-03-28T18:01:25.204 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ff47ec41ff] 2024-03-28T18:01:25.204 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f62b1441488] 2024-03-28T18:01:25.204 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 16: (DispatchQueue::entry()+0x51f) [0x7f62b143e60f] 2024-03-28T18:01:25.204 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f62b1511e81] 2024-03-28T18:01:25.204 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f62aff2c1ca] 2024-03-28T18:01:25.204 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 19: clone() 2024-03-28T18:01:25.204 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 2024-03-28T18:01:24.936+0000 7f62a5800700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:01:25.204 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: in thread 7f62a5800700 thread_name:ms_dispatch 2024-03-28T18:01:25.204 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 2024-03-28T18:01:25.204 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:25.205 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f62aff36d20] 2024-03-28T18:01:25.205 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 2: gsignal() 2024-03-28T18:01:25.205 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 3: abort() 2024-03-28T18:01:25.205 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f62b11af4fb] 2024-03-28T18:01:25.205 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f62b11af667] 2024-03-28T18:01:25.205 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ff4810c223] 2024-03-28T18:01:25.206 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ff47f865d6] 2024-03-28T18:01:25.206 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ff47f86da1] 2024-03-28T18:01:25.206 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ff47f892d5] 2024-03-28T18:01:25.206 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ff47fd9233] 2024-03-28T18:01:25.206 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff47fdb197] 2024-03-28T18:01:25.206 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ff47eda11b] 2024-03-28T18:01:25.206 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff47eda6bc] 2024-03-28T18:01:25.206 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ff47ec41ff] 2024-03-28T18:01:25.206 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f62b1441488] 2024-03-28T18:01:25.206 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 16: (DispatchQueue::entry()+0x51f) [0x7f62b143e60f] 2024-03-28T18:01:25.207 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f62b1511e81] 2024-03-28T18:01:25.207 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f62aff2c1ca] 2024-03-28T18:01:25.207 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 19: clone() 2024-03-28T18:01:25.207 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:01:25.207 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 2024-03-28T18:01:25.207 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: -1> 2024-03-28T18:01:24.933+0000 7f62a5800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f62a5800700 time 2024-03-28T18:01:24.932604+0000 2024-03-28T18:01:25.207 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:01:25.207 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 2024-03-28T18:01:25.207 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:25.207 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f62b11af4a1] 2024-03-28T18:01:25.207 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f62b11af667] 2024-03-28T18:01:25.208 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ff4810c223] 2024-03-28T18:01:25.208 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ff47f865d6] 2024-03-28T18:01:25.208 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ff47f86da1] 2024-03-28T18:01:25.208 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ff47f892d5] 2024-03-28T18:01:25.208 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ff47fd9233] 2024-03-28T18:01:25.208 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff47fdb197] 2024-03-28T18:01:25.208 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ff47eda11b] 2024-03-28T18:01:25.208 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff47eda6bc] 2024-03-28T18:01:25.208 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ff47ec41ff] 2024-03-28T18:01:25.208 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f62b1441488] 2024-03-28T18:01:25.208 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 13: (DispatchQueue::entry()+0x51f) [0x7f62b143e60f] 2024-03-28T18:01:25.209 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f62b1511e81] 2024-03-28T18:01:25.209 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f62aff2c1ca] 2024-03-28T18:01:25.209 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 16: clone() 2024-03-28T18:01:25.209 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 2024-03-28T18:01:25.209 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 0> 2024-03-28T18:01:24.936+0000 7f62a5800700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:01:25.209 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: in thread 7f62a5800700 thread_name:ms_dispatch 2024-03-28T18:01:25.209 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 2024-03-28T18:01:25.209 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:25.209 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f62aff36d20] 2024-03-28T18:01:25.209 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 2: gsignal() 2024-03-28T18:01:25.209 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 3: abort() 2024-03-28T18:01:25.209 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f62b11af4fb] 2024-03-28T18:01:25.209 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f62b11af667] 2024-03-28T18:01:25.210 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ff4810c223] 2024-03-28T18:01:25.210 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ff47f865d6] 2024-03-28T18:01:25.210 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ff47f86da1] 2024-03-28T18:01:25.210 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ff47f892d5] 2024-03-28T18:01:25.210 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ff47fd9233] 2024-03-28T18:01:25.211 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff47fdb197] 2024-03-28T18:01:25.211 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ff47eda11b] 2024-03-28T18:01:25.211 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff47eda6bc] 2024-03-28T18:01:25.211 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ff47ec41ff] 2024-03-28T18:01:25.211 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f62b1441488] 2024-03-28T18:01:25.211 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 16: (DispatchQueue::entry()+0x51f) [0x7f62b143e60f] 2024-03-28T18:01:25.211 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f62b1511e81] 2024-03-28T18:01:25.211 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f62aff2c1ca] 2024-03-28T18:01:25.211 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 19: clone() 2024-03-28T18:01:25.211 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:01:25.211 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 2024-03-28T18:01:25.211 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: -1> 2024-03-28T18:01:24.933+0000 7f62a5800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f62a5800700 time 2024-03-28T18:01:24.932604+0000 2024-03-28T18:01:25.211 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:01:25.212 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 2024-03-28T18:01:25.212 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:25.212 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f62b11af4a1] 2024-03-28T18:01:25.212 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f62b11af667] 2024-03-28T18:01:25.212 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ff4810c223] 2024-03-28T18:01:25.212 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ff47f865d6] 2024-03-28T18:01:25.212 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ff47f86da1] 2024-03-28T18:01:25.212 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ff47f892d5] 2024-03-28T18:01:25.212 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ff47fd9233] 2024-03-28T18:01:25.212 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff47fdb197] 2024-03-28T18:01:25.212 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ff47eda11b] 2024-03-28T18:01:25.212 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff47eda6bc] 2024-03-28T18:01:25.212 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ff47ec41ff] 2024-03-28T18:01:25.213 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f62b1441488] 2024-03-28T18:01:25.213 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 13: (DispatchQueue::entry()+0x51f) [0x7f62b143e60f] 2024-03-28T18:01:25.213 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f62b1511e81] 2024-03-28T18:01:25.213 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f62aff2c1ca] 2024-03-28T18:01:25.213 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 16: clone() 2024-03-28T18:01:25.213 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 2024-03-28T18:01:25.213 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 0> 2024-03-28T18:01:24.936+0000 7f62a5800700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:01:25.213 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: in thread 7f62a5800700 thread_name:ms_dispatch 2024-03-28T18:01:25.213 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 2024-03-28T18:01:25.213 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:25.213 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f62aff36d20] 2024-03-28T18:01:25.213 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 2: gsignal() 2024-03-28T18:01:25.213 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 3: abort() 2024-03-28T18:01:25.213 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f62b11af4fb] 2024-03-28T18:01:25.214 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f62b11af667] 2024-03-28T18:01:25.214 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ff4810c223] 2024-03-28T18:01:25.214 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ff47f865d6] 2024-03-28T18:01:25.214 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ff47f86da1] 2024-03-28T18:01:25.214 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ff47f892d5] 2024-03-28T18:01:25.214 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ff47fd9233] 2024-03-28T18:01:25.214 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff47fdb197] 2024-03-28T18:01:25.214 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ff47eda11b] 2024-03-28T18:01:25.214 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff47eda6bc] 2024-03-28T18:01:25.214 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ff47ec41ff] 2024-03-28T18:01:25.214 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f62b1441488] 2024-03-28T18:01:25.215 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 16: (DispatchQueue::entry()+0x51f) [0x7f62b143e60f] 2024-03-28T18:01:25.215 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f62b1511e81] 2024-03-28T18:01:25.215 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f62aff2c1ca] 2024-03-28T18:01:25.215 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 19: clone() 2024-03-28T18:01:25.215 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:01:25.215 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:24 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[94558]: 2024-03-28T18:01:25.215 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:25 smithi045 podman[94814]: 2024-03-28 18:01:25.041033748 +0000 UTC m=+0.035287096 container died aafd4ee1a9e67515a152ea7f8746364ca455b65622cbfaae37c8525818ff716a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-28T18:01:25.224 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:24 smithi151 ceph-mon[36481]: pgmap v426: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:01:25.224 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:24 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/1173454637,v1:172.21.15.45:6835/1173454637] up:rejoin 2024-03-28T18:01:25.224 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:24 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:25.224 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:24 smithi151 ceph-mon[36481]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-03-28T18:01:25.310 INFO:journalctl@ceph.mds.b.smithi071.stdout:Mar 28 18:01:24 smithi071 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-b[66900]: 2024-03-28T18:01:24.930+0000 7ff9f838b700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-28T18:01:25.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:24 smithi071 ceph-mon[36597]: pgmap v426: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:01:25.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:24 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/1173454637,v1:172.21.15.45:6835/1173454637] up:rejoin 2024-03-28T18:01:25.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:24 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:25.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:24 smithi071 ceph-mon[36597]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-03-28T18:01:25.928 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 538 sec latency 653824.965 ms 2024-03-28T18:01:26.165 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:25 smithi045 podman[94814]: 2024-03-28 18:01:25.853174149 +0000 UTC m=+0.847427556 container remove aafd4ee1a9e67515a152ea7f8746364ca455b65622cbfaae37c8525818ff716a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-28T18:01:26.165 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:25 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:01:26.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:25 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:01:26.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:25 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/1173454637,v1:172.21.15.45:6835/1173454637] up:active 2024-03-28T18:01:26.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:25 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:26.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:25 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 3 up:standby-replay 2024-03-28T18:01:26.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:25 smithi045 ceph-mon[31773]: pgmap v427: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-28T18:01:26.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:25 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:01:26.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:25 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/1173454637,v1:172.21.15.45:6835/1173454637] up:active 2024-03-28T18:01:26.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:25 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:26.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:25 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 3 up:standby-replay 2024-03-28T18:01:26.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:25 smithi071 ceph-mon[36597]: pgmap v427: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-28T18:01:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:25 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:01:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:25 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/1173454637,v1:172.21.15.45:6835/1173454637] up:active 2024-03-28T18:01:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:25 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:25 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 3 up:standby-replay 2024-03-28T18:01:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:25 smithi151 ceph-mon[36481]: pgmap v427: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-28T18:01:26.665 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:26 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-28T18:01:26.666 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:26 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Consumed 1.638s CPU time. 2024-03-28T18:01:26.928 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 539 sec latency 654825.051 ms 2024-03-28T18:01:27.928 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 540 sec latency 655825.156 ms 2024-03-28T18:01:28.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:28 smithi045 ceph-mon[31773]: pgmap v428: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-28T18:01:28.928 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 541 sec latency 656825.269 ms 2024-03-28T18:01:29.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:28 smithi071 ceph-mon[36597]: pgmap v428: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-28T18:01:29.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:28 smithi151 ceph-mon[36481]: pgmap v428: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-28T18:01:29.928 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 542 sec latency 657825.359 ms 2024-03-28T18:01:30.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:30 smithi045 ceph-mon[31773]: pgmap v429: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-28T18:01:30.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:30 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:01:30.928 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 543 sec latency 658825.434 ms 2024-03-28T18:01:31.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:30 smithi071 ceph-mon[36597]: pgmap v429: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-28T18:01:31.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:30 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:01:31.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:30 smithi151 ceph-mon[36481]: pgmap v429: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-28T18:01:31.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:30 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:01:31.888 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:31 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:31.888 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:31 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:01:31.888 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:31 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:01:31.928 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 544 sec latency 659825.552 ms 2024-03-28T18:01:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:31 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:31 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:01:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:31 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:01:32.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:31 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:01:32.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:31 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:01:32.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:31 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:01:32.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:32 smithi045 ceph-mon[31773]: pgmap v430: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 26 KiB/s wr, 14 op/s 2024-03-28T18:01:32.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:32 smithi045 ceph-mon[31773]: pgmap v431: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 28 KiB/s wr, 15 op/s 2024-03-28T18:01:32.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:32 smithi045 ceph-mon[31773]: osdmap e138: 12 total, 12 up, 12 in 2024-03-28T18:01:32.928 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 545 sec latency 660825.660 ms 2024-03-28T18:01:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:32 smithi071 ceph-mon[36597]: pgmap v430: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 26 KiB/s wr, 14 op/s 2024-03-28T18:01:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:32 smithi071 ceph-mon[36597]: pgmap v431: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 28 KiB/s wr, 15 op/s 2024-03-28T18:01:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:32 smithi071 ceph-mon[36597]: osdmap e138: 12 total, 12 up, 12 in 2024-03-28T18:01:33.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:32 smithi151 ceph-mon[36481]: pgmap v430: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 26 KiB/s wr, 14 op/s 2024-03-28T18:01:33.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:32 smithi151 ceph-mon[36481]: pgmap v431: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 28 KiB/s wr, 15 op/s 2024-03-28T18:01:33.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:32 smithi151 ceph-mon[36481]: osdmap e138: 12 total, 12 up, 12 in 2024-03-28T18:01:33.928 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 546 sec latency 661825.797 ms 2024-03-28T18:01:34.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:34 smithi045 ceph-mon[31773]: pgmap v433: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 133 B/s wr, 4 op/s 2024-03-28T18:01:34.928 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 547 sec latency 662825.961 ms 2024-03-28T18:01:35.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:34 smithi071 ceph-mon[36597]: pgmap v433: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 133 B/s wr, 4 op/s 2024-03-28T18:01:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:34 smithi151 ceph-mon[36481]: pgmap v433: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 133 B/s wr, 4 op/s 2024-03-28T18:01:35.929 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 548 sec latency 663826.122 ms 2024-03-28T18:01:36.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:36 smithi045 ceph-mon[31773]: pgmap v434: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 267 B/s wr, 8 op/s 2024-03-28T18:01:36.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:36 smithi045 ceph-mon[31773]: osdmap e139: 12 total, 12 up, 12 in 2024-03-28T18:01:36.815 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:36 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 3. 2024-03-28T18:01:36.815 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:36 smithi045 systemd[1]: Stopped Ceph mds.j for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:01:36.815 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:36 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Consumed 1.638s CPU time. 2024-03-28T18:01:36.815 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:36 smithi045 systemd[1]: Starting Ceph mds.j for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T18:01:36.929 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 549 sec latency 664826.270 ms 2024-03-28T18:01:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:36 smithi071 ceph-mon[36597]: pgmap v434: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 267 B/s wr, 8 op/s 2024-03-28T18:01:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:36 smithi071 ceph-mon[36597]: osdmap e139: 12 total, 12 up, 12 in 2024-03-28T18:01:37.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:36 smithi151 ceph-mon[36481]: pgmap v434: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 267 B/s wr, 8 op/s 2024-03-28T18:01:37.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:36 smithi151 ceph-mon[36481]: osdmap e139: 12 total, 12 up, 12 in 2024-03-28T18:01:37.165 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:36 smithi045 podman[95037]: 2024-03-28 18:01:36.720850398 +0000 UTC m=+0.034738759 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:01:37.166 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:36 smithi045 podman[95037]: 2024-03-28 18:01:36.845638787 +0000 UTC m=+0.159527187 container create a92c50311e28e3a30f961ff57ba85b45de302aad1d969d2dc21b465f7ef32efb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:01:37.635 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:37 smithi045 podman[95037]: 2024-03-28 18:01:37.168454419 +0000 UTC m=+0.482342814 container init a92c50311e28e3a30f961ff57ba85b45de302aad1d969d2dc21b465f7ef32efb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:01:37.635 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:37 smithi045 podman[95037]: 2024-03-28 18:01:37.178097844 +0000 UTC m=+0.491986239 container start a92c50311e28e3a30f961ff57ba85b45de302aad1d969d2dc21b465f7ef32efb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-28T18:01:37.635 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:37 smithi045 ceph-mds[95056]: -- 172.21.15.45:0/3331540159 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5604bc761a40 con 0x5604bc734c00 2024-03-28T18:01:37.635 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:37 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: starting mds.j at 2024-03-28T18:01:37.635 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:37 smithi045 bash[95037]: a92c50311e28e3a30f961ff57ba85b45de302aad1d969d2dc21b465f7ef32efb 2024-03-28T18:01:37.635 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:37 smithi045 systemd[1]: Started Ceph mds.j for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:01:37.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:37 smithi045 ceph-mon[31773]: daemon mds.j restarted 2024-03-28T18:01:37.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:37 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:01:37.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:37 smithi045 ceph-mon[31773]: osdmap e140: 12 total, 12 up, 12 in 2024-03-28T18:01:37.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:37 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/4155871960,v1:172.21.15.45:6835/4155871960] up:boot 2024-03-28T18:01:37.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:37 smithi045 ceph-mon[31773]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-03-28T18:01:37.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:37 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:01:37.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:37 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:01:37.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:37 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:37.929 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 550 sec latency 665826.416 ms 2024-03-28T18:01:38.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:37 smithi071 ceph-mon[36597]: daemon mds.j restarted 2024-03-28T18:01:38.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:37 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:01:38.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:37 smithi071 ceph-mon[36597]: osdmap e140: 12 total, 12 up, 12 in 2024-03-28T18:01:38.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:37 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/4155871960,v1:172.21.15.45:6835/4155871960] up:boot 2024-03-28T18:01:38.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:37 smithi071 ceph-mon[36597]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-03-28T18:01:38.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:37 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:01:38.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:37 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:01:38.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:37 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:38.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:37 smithi151 ceph-mon[36481]: daemon mds.j restarted 2024-03-28T18:01:38.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:37 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:01:38.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:37 smithi151 ceph-mon[36481]: osdmap e140: 12 total, 12 up, 12 in 2024-03-28T18:01:38.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:37 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/4155871960,v1:172.21.15.45:6835/4155871960] up:boot 2024-03-28T18:01:38.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:37 smithi151 ceph-mon[36481]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-03-28T18:01:38.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:37 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:01:38.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:37 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:01:38.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:37 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:38.114 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:01:38.114 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:01:38.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.354+0000 7f1f52200640 1 -- 172.21.15.45:0/1187214315 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f4c073a40 msgr2=0x7f1f4c073e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:38.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.354+0000 7f1f52200640 1 --2- 172.21.15.45:0/1187214315 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f4c073a40 0x7f1f4c073e20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f1f40009900 tx=0x7f1f4002bcc0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.354+0000 7f1f52200640 1 -- 172.21.15.45:0/1187214315 shutdown_connections 2024-03-28T18:01:38.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.354+0000 7f1f52200640 1 --2- 172.21.15.45:0/1187214315 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f4c112520 0x7f1f4c114910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.354+0000 7f1f52200640 1 --2- 172.21.15.45:0/1187214315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f4c074360 0x7f1f4c111fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.354+0000 7f1f52200640 1 --2- 172.21.15.45:0/1187214315 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f4c073a40 0x7f1f4c073e20 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.354+0000 7f1f52200640 1 -- 172.21.15.45:0/1187214315 >> 172.21.15.45:0/1187214315 conn(0x7f1f4c0fd000 msgr2=0x7f1f4c0ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:38.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.355+0000 7f1f52200640 1 -- 172.21.15.45:0/1187214315 shutdown_connections 2024-03-28T18:01:38.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.355+0000 7f1f52200640 1 -- 172.21.15.45:0/1187214315 wait complete. 2024-03-28T18:01:38.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.356+0000 7f1f52200640 1 Processor -- start 2024-03-28T18:01:38.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.356+0000 7f1f52200640 1 -- start start 2024-03-28T18:01:38.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.356+0000 7f1f52200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f4c073a40 0x7f1f4c19e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:38.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.356+0000 7f1f52200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f4c074360 0x7f1f4c19ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:38.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.356+0000 7f1f52200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f4c112520 0x7f1f4c1a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:38.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.356+0000 7f1f52200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c117f70 con 0x7f1f4c074360 2024-03-28T18:01:38.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.357+0000 7f1f52200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c117df0 con 0x7f1f4c073a40 2024-03-28T18:01:38.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.357+0000 7f1f52200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c1180f0 con 0x7f1f4c112520 2024-03-28T18:01:38.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.357+0000 7f1f50e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f4c073a40 0x7f1f4c19e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:38.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.357+0000 7f1f50e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f4c073a40 0x7f1f4c19e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:49250/0 (socket says 172.21.15.45:49250) 2024-03-28T18:01:38.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.357+0000 7f1f50e00640 1 -- 172.21.15.45:0/3722264986 learned_addr learned my addr 172.21.15.45:0/3722264986 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:01:38.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.357+0000 7f1f4be00640 1 --2- 172.21.15.45:0/3722264986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f4c074360 0x7f1f4c19ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:38.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.357+0000 7f1f51800640 1 --2- 172.21.15.45:0/3722264986 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f4c112520 0x7f1f4c1a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:38.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.357+0000 7f1f50e00640 1 -- 172.21.15.45:0/3722264986 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f4c112520 msgr2=0x7f1f4c1a2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:38.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.357+0000 7f1f50e00640 1 --2- 172.21.15.45:0/3722264986 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f4c112520 0x7f1f4c1a2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.357+0000 7f1f50e00640 1 -- 172.21.15.45:0/3722264986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f4c074360 msgr2=0x7f1f4c19ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:38.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.357+0000 7f1f50e00640 1 --2- 172.21.15.45:0/3722264986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f4c074360 0x7f1f4c19ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.357+0000 7f1f50e00640 1 -- 172.21.15.45:0/3722264986 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4c1a34c0 con 0x7f1f4c073a40 2024-03-28T18:01:38.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.357+0000 7f1f51800640 1 --2- 172.21.15.45:0/3722264986 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f4c112520 0x7f1f4c1a2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:01:38.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.358+0000 7f1f50e00640 1 --2- 172.21.15.45:0/3722264986 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f4c073a40 0x7f1f4c19e4d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f1f40000c00 tx=0x7f1f40004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:38.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.358+0000 7f1f49600640 1 -- 172.21.15.45:0/3722264986 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f40042020 con 0x7f1f4c073a40 2024-03-28T18:01:38.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.358+0000 7f1f49600640 1 -- 172.21.15.45:0/3722264986 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f40005000 con 0x7f1f4c073a40 2024-03-28T18:01:38.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.358+0000 7f1f49600640 1 -- 172.21.15.45:0/3722264986 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f40003b80 con 0x7f1f4c073a40 2024-03-28T18:01:38.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.358+0000 7f1f52200640 1 -- 172.21.15.45:0/3722264986 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f4c1a3750 con 0x7f1f4c073a40 2024-03-28T18:01:38.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.358+0000 7f1f52200640 1 -- 172.21.15.45:0/3722264986 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f4c1ab040 con 0x7f1f4c073a40 2024-03-28T18:01:38.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.361+0000 7f1f5643f740 1 Processor -- start 2024-03-28T18:01:38.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.361+0000 7f1f5643f740 1 -- start start 2024-03-28T18:01:38.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.361+0000 7f1f49600640 1 -- 172.21.15.45:0/3722264986 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f1f40045020 con 0x7f1f4c073a40 2024-03-28T18:01:38.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.362+0000 7f1f5643f740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b956330dd0 0x55b95632fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:38.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.362+0000 7f1f5643f740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b9563300f0 0x55b956330570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:38.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.362+0000 7f1f5643f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b956339460 0x55b95633b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:38.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.362+0000 7f1f5643f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b956294250 con 0x55b956339460 2024-03-28T18:01:38.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.362+0000 7f1f5643f740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b95623ca30 con 0x55b9563300f0 2024-03-28T18:01:38.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.362+0000 7f1f5643f740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9562902f0 con 0x55b956330dd0 2024-03-28T18:01:38.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.362+0000 7f1f50e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b9563300f0 0x55b956330570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:38.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.362+0000 7f1f50e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b9563300f0 0x55b956330570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:49256/0 (socket says 172.21.15.45:49256) 2024-03-28T18:01:38.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.362+0000 7f1f50e00640 1 -- 172.21.15.45:0/3313975530 learned_addr learned my addr 172.21.15.45:0/3313975530 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:01:38.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.362+0000 7f1f4be00640 1 --2- 172.21.15.45:0/3313975530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b956339460 0x55b95633b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:38.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.362+0000 7f1f51800640 1 --2- 172.21.15.45:0/3313975530 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b956330dd0 0x55b95632fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:38.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.363+0000 7f1f50e00640 1 -- 172.21.15.45:0/3313975530 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b956330dd0 msgr2=0x55b95632fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:01:38.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.363+0000 7f1f50e00640 1 --2- 172.21.15.45:0/3313975530 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b956330dd0 0x55b95632fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.363+0000 7f1f50e00640 1 -- 172.21.15.45:0/3313975530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b956339460 msgr2=0x55b95633b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:01:38.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.363+0000 7f1f50e00640 1 --2- 172.21.15.45:0/3313975530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b956339460 0x55b95633b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.363+0000 7f1f50e00640 1 -- 172.21.15.45:0/3313975530 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b956227690 con 0x55b9563300f0 2024-03-28T18:01:38.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.363+0000 7f1f50e00640 1 --2- 172.21.15.45:0/3313975530 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b9563300f0 0x55b956330570 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f1f400ee0f0 tx=0x7f1f400f7b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2b16c41f1b1d590 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:38.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.363+0000 7f1f3e000640 1 -- 172.21.15.45:0/3313975530 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f40100070 con 0x55b9563300f0 2024-03-28T18:01:38.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.363+0000 7f1f3e000640 1 -- 172.21.15.45:0/3313975530 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f400f9d60 con 0x55b9563300f0 2024-03-28T18:01:38.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.363+0000 7f1f3e000640 1 -- 172.21.15.45:0/3313975530 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f400f3890 con 0x55b9563300f0 2024-03-28T18:01:38.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.363+0000 7f1f49600640 1 --2- 172.21.15.45:0/3722264986 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f1f340a29f0 0x7f1f340a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:38.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.364+0000 7f1f49600640 1 -- 172.21.15.45:0/3722264986 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 11952+0+0 (secure 0 0 0) 0x7f1f400f16f0 con 0x7f1f4c073a40 2024-03-28T18:01:38.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.364+0000 7f1f51800640 1 --2- 172.21.15.45:0/3722264986 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f1f340a29f0 0x7f1f340a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:38.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.364+0000 7f1f5643f740 1 -- 172.21.15.45:0/3313975530 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b9563300f0 msgr2=0x55b956330570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:38.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.364+0000 7f1f5643f740 1 --2- 172.21.15.45:0/3313975530 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b9563300f0 0x55b956330570 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f1f400ee0f0 tx=0x7f1f400f7b20 comp rx=0 tx=0).stop 2024-03-28T18:01:38.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.364+0000 7f1f5643f740 1 -- 172.21.15.45:0/3313975530 shutdown_connections 2024-03-28T18:01:38.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.364+0000 7f1f5643f740 1 --2- 172.21.15.45:0/3313975530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b956339460 0x55b95633b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.364+0000 7f1f5643f740 1 --2- 172.21.15.45:0/3313975530 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b9563300f0 0x55b956330570 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.364+0000 7f1f5643f740 1 --2- 172.21.15.45:0/3313975530 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b956330dd0 0x55b95632fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.364+0000 7f1f5643f740 1 -- 172.21.15.45:0/3313975530 >> 172.21.15.45:0/3313975530 conn(0x55b9562f7200 msgr2=0x55b95632ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:38.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.364+0000 7f1f5643f740 1 -- 172.21.15.45:0/3313975530 shutdown_connections 2024-03-28T18:01:38.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.364+0000 7f1f5643f740 1 -- 172.21.15.45:0/3313975530 wait complete. 2024-03-28T18:01:38.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.365+0000 7f1f51800640 1 --2- 172.21.15.45:0/3722264986 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f1f340a29f0 0x7f1f340a4eb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c06ac50 tx=0x7f1f44009500 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:38.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.365+0000 7f1f5643f740 1 -- start start 2024-03-28T18:01:38.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.365+0000 7f1f5643f740 1 Processor -- start 2024-03-28T18:01:38.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.365+0000 7f1f5643f740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b9563300f0 0x55b956338af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:38.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.365+0000 7f1f5643f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b956330dd0 0x55b956335e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:38.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.365+0000 7f1f5643f740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b956339460 0x55b9563363c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:38.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.365+0000 7f1f5643f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b95633c090 con 0x55b956330dd0 2024-03-28T18:01:38.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.366+0000 7f1f5643f740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b95624fe80 con 0x55b9563300f0 2024-03-28T18:01:38.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.366+0000 7f1f5643f740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b95633c210 con 0x55b956339460 2024-03-28T18:01:38.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.366+0000 7f1f5643f740 10 client.0 _ll_register_callbacks cb 0x55b956262130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:01:38.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.366+0000 7f1f50e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b9563300f0 0x55b956338af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:38.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.366+0000 7f1f4be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b956330dd0 0x55b956335e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:38.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.366+0000 7f1f50e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b9563300f0 0x55b956338af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:49264/0 (socket says 172.21.15.45:49264) 2024-03-28T18:01:38.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.366+0000 7f1f50e00640 1 -- 172.21.15.45:0/2663435203 learned_addr learned my addr 172.21.15.45:0/2663435203 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:01:38.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.366+0000 7f1f51800640 1 --2- 172.21.15.45:0/2663435203 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b956339460 0x55b9563363c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:38.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.366+0000 7f1f50e00640 1 -- 172.21.15.45:0/2663435203 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b956339460 msgr2=0x55b9563363c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:01:38.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.366+0000 7f1f50e00640 1 --2- 172.21.15.45:0/2663435203 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b956339460 0x55b9563363c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.366+0000 7f1f50e00640 1 -- 172.21.15.45:0/2663435203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b956330dd0 msgr2=0x55b956335e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:01:38.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.366+0000 7f1f50e00640 1 --2- 172.21.15.45:0/2663435203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b956330dd0 0x55b956335e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.366+0000 7f1f50e00640 1 -- 172.21.15.45:0/2663435203 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b9560727d0 con 0x55b9563300f0 2024-03-28T18:01:38.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.366+0000 7f1f50e00640 1 --2- 172.21.15.45:0/2663435203 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b9563300f0 0x55b956338af0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1f400f9670 tx=0x7f1f400f3110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=994941dccbe3481 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:38.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.367+0000 7f1f32a00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T18:01:38.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.367+0000 7f1f32a00640 1 -- 172.21.15.45:0/2663435203 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f40100070 con 0x55b9563300f0 2024-03-28T18:01:38.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.367+0000 7f1f32a00640 1 -- 172.21.15.45:0/2663435203 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(140..140 src has 1..140) v4 ==== 11952+0+0 (secure 0 0 0) 0x7f1f40003d20 con 0x55b9563300f0 2024-03-28T18:01:38.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.367+0000 7f1f5643f740 1 -- 172.21.15.45:0/2663435203 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1f400f9d60 con 0x55b9563300f0 2024-03-28T18:01:38.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.367+0000 7f1f32a00640 1 -- 172.21.15.45:0/2663435203 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=282) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f400b9360 con 0x55b9563300f0 2024-03-28T18:01:38.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.367+0000 7f1f5643f740 10 client.26495 fetch_fsmap learned FSMap version 282 2024-03-28T18:01:38.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.367+0000 7f1f5643f740 1 -- 172.21.15.45:0/2663435203 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=282,monmap=4+}) v3 -- 0x55b955fa83f0 con 0x55b9563300f0 2024-03-28T18:01:38.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.367+0000 7f1f32a00640 1 -- 172.21.15.45:0/2663435203 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f400f9d60 con 0x55b9563300f0 2024-03-28T18:01:38.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.368+0000 7f1f32a00640 1 -- 172.21.15.45:0/2663435203 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 282) v1 ==== 4472+0+0 (secure 0 0 0) 0x7f1f400b9730 con 0x55b9563300f0 2024-03-28T18:01:38.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.368+0000 7f1f5643f740 10 client.26495 fetch_fsmap finished waiting for FSMap version 282 2024-03-28T18:01:38.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.368+0000 7f1f5643f740 10 client.26495 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:01:38.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.368+0000 7f1f5643f740 20 client.26495 populate_metadata read hostname 'smithi045' 2024-03-28T18:01:38.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.368+0000 7f1f5643f740 1 --2- 172.21.15.45:0/2663435203 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55b956348ef0 0x55b95634b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:38.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.368+0000 7f1f5643f740 4 client.26495 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:01:38.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.368+0000 7f1f5643f740 1 -- 172.21.15.45:0/2663435203 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b9562a1790 con 0x55b956348ef0 2024-03-28T18:01:38.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.368+0000 7f1f4be00640 1 --2- 172.21.15.45:0/2663435203 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55b956348ef0 0x55b95634b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:38.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.368+0000 7f1f4be00640 1 --2- 172.21.15.45:0/2663435203 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55b956348ef0 0x55b95634b2e0 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f8623a2c105670 server_cookie=d8e8335c856a9d38 in_seq=0 out_seq=0 2024-03-28T18:01:38.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.368+0000 7f1f32a00640 10 client.26495 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:01:38.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.369+0000 7f1f32a00640 1 -- 172.21.15.45:0/2663435203 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b9562a1790 con 0x55b956348ef0 2024-03-28T18:01:38.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.369+0000 7f1f32a00640 10 client.26495 handle_command_reply: tid=0 2024-03-28T18:01:38.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.369+0000 7f1f5643f740 1 client.26495 shutdown 2024-03-28T18:01:38.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.369+0000 7f1f5643f740 1 -- 172.21.15.45:0/2663435203 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b9563300f0 msgr2=0x55b956338af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:38.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.369+0000 7f1f5643f740 1 --2- 172.21.15.45:0/2663435203 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b9563300f0 0x55b956338af0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1f400f9670 tx=0x7f1f400f3110 comp rx=0 tx=0).stop 2024-03-28T18:01:38.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.370+0000 7f1f5643f740 1 -- 172.21.15.45:0/2663435203 shutdown_connections 2024-03-28T18:01:38.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.370+0000 7f1f5643f740 1 --2- 172.21.15.45:0/2663435203 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55b956348ef0 0x55b95634b2e0 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.370+0000 7f1f5643f740 1 --2- 172.21.15.45:0/2663435203 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b956339460 0x55b9563363c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.370+0000 7f1f32a00640 0 client.26495 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:01:38.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.370+0000 7f1f5643f740 1 --2- 172.21.15.45:0/2663435203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b956330dd0 0x55b956335e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.370+0000 7f1f5643f740 1 --2- 172.21.15.45:0/2663435203 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b9563300f0 0x55b956338af0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.370+0000 7f1f5643f740 1 -- 172.21.15.45:0/2663435203 >> 172.21.15.45:0/2663435203 conn(0x55b9562f7200 msgr2=0x55b95632c890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:38.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.370+0000 7f1f5643f740 1 -- 172.21.15.45:0/2663435203 shutdown_connections 2024-03-28T18:01:38.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.370+0000 7f1f5643f740 1 -- 172.21.15.45:0/2663435203 wait complete. 2024-03-28T18:01:38.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.370+0000 7f1f5643f740 20 client.26495 trim_cache size 0 max 16384 2024-03-28T18:01:38.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.387+0000 7f1f5643f740 1 Processor -- start 2024-03-28T18:01:38.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.388+0000 7f1f5643f740 1 -- start start 2024-03-28T18:01:38.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.388+0000 7f1f5643f740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b9563300f0 0x55b9563385b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:38.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.388+0000 7f1f5643f740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b956330dd0 0x55b956338af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:38.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.388+0000 7f1f5643f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b956339460 0x55b95633a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:38.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.388+0000 7f1f5643f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c1b5ec0 con 0x55b956339460 2024-03-28T18:01:38.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.388+0000 7f1f5643f740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c1b4540 con 0x55b956330dd0 2024-03-28T18:01:38.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.388+0000 7f1f5643f740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c1b5860 con 0x55b9563300f0 2024-03-28T18:01:38.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.388+0000 7f1f4be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b956330dd0 0x55b956338af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:38.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.388+0000 7f1f51800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b956339460 0x55b95633a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:38.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.388+0000 7f1f4be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b956330dd0 0x55b956338af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:49272/0 (socket says 172.21.15.45:49272) 2024-03-28T18:01:38.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.388+0000 7f1f4be00640 1 -- 172.21.15.45:0/279031614 learned_addr learned my addr 172.21.15.45:0/279031614 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:01:38.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.388+0000 7f1f50e00640 1 --2- 172.21.15.45:0/279031614 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b9563300f0 0x55b9563385b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:38.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.389+0000 7f1f4be00640 1 -- 172.21.15.45:0/279031614 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b9563300f0 msgr2=0x55b9563385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:01:38.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.389+0000 7f1f4be00640 1 --2- 172.21.15.45:0/279031614 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b9563300f0 0x55b9563385b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.389+0000 7f1f4be00640 1 -- 172.21.15.45:0/279031614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b956339460 msgr2=0x55b95633a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:01:38.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.389+0000 7f1f4be00640 1 --2- 172.21.15.45:0/279031614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b956339460 0x55b95633a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.389+0000 7f1f4be00640 1 -- 172.21.15.45:0/279031614 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b955f680f0 con 0x55b956330dd0 2024-03-28T18:01:38.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.389+0000 7f1f4be00640 1 --2- 172.21.15.45:0/279031614 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b956330dd0 0x55b956338af0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1f38009d10 tx=0x7f1f38005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c17fff2d6e9720fa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:38.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.389+0000 7f1f3d600640 1 -- 172.21.15.45:0/279031614 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3800f890 con 0x55b956330dd0 2024-03-28T18:01:38.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.389+0000 7f1f3d600640 1 -- 172.21.15.45:0/279031614 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f3801f700 con 0x55b956330dd0 2024-03-28T18:01:38.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.389+0000 7f1f3d600640 1 -- 172.21.15.45:0/279031614 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3801e5f0 con 0x55b956330dd0 2024-03-28T18:01:38.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.390+0000 7f1f5643f740 1 -- 172.21.15.45:0/279031614 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b956330dd0 msgr2=0x55b956338af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:38.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.390+0000 7f1f5643f740 1 --2- 172.21.15.45:0/279031614 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b956330dd0 0x55b956338af0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1f38009d10 tx=0x7f1f38005f90 comp rx=0 tx=0).stop 2024-03-28T18:01:38.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.390+0000 7f1f5643f740 1 -- 172.21.15.45:0/279031614 shutdown_connections 2024-03-28T18:01:38.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.390+0000 7f1f5643f740 1 --2- 172.21.15.45:0/279031614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b956339460 0x55b95633a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.390+0000 7f1f5643f740 1 --2- 172.21.15.45:0/279031614 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b956330dd0 0x55b956338af0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.390+0000 7f1f5643f740 1 --2- 172.21.15.45:0/279031614 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b9563300f0 0x55b9563385b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.390+0000 7f1f5643f740 1 -- 172.21.15.45:0/279031614 >> 172.21.15.45:0/279031614 conn(0x55b9562f7200 msgr2=0x55b956334750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.390+0000 7f1f5643f740 1 -- 172.21.15.45:0/279031614 shutdown_connections 2024-03-28T18:01:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.390+0000 7f1f5643f740 1 -- 172.21.15.45:0/279031614 wait complete. 2024-03-28T18:01:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.390+0000 7f1f5643f740 1 -- start start 2024-03-28T18:01:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.391+0000 7f1f5643f740 1 Processor -- start 2024-03-28T18:01:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.391+0000 7f1f5643f740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b9563300f0 0x55b956332ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.391+0000 7f1f5643f740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b956330dd0 0x55b9563331e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.391+0000 7f1f5643f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b956339460 0x55b9563a3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.391+0000 7f1f5643f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c1b5200 con 0x55b956339460 2024-03-28T18:01:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.391+0000 7f1f5643f740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c1b4ba0 con 0x55b9563300f0 2024-03-28T18:01:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.391+0000 7f1f5643f740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c1b3560 con 0x55b956330dd0 2024-03-28T18:01:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.391+0000 7f1f5643f740 10 client.0 _ll_register_callbacks cb 0x7f1f380097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:01:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.391+0000 7f1f50e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b9563300f0 0x55b956332ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.391+0000 7f1f51800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b956339460 0x55b9563a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.391+0000 7f1f50e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b9563300f0 0x55b956332ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:49280/0 (socket says 172.21.15.45:49280) 2024-03-28T18:01:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.391+0000 7f1f50e00640 1 -- 172.21.15.45:0/3910661223 learned_addr learned my addr 172.21.15.45:0/3910661223 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:01:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.392+0000 7f1f4be00640 1 --2- 172.21.15.45:0/3910661223 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b956330dd0 0x55b9563331e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.392+0000 7f1f50e00640 1 -- 172.21.15.45:0/3910661223 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b956330dd0 msgr2=0x55b9563331e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:01:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.392+0000 7f1f50e00640 1 --2- 172.21.15.45:0/3910661223 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b956330dd0 0x55b9563331e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.392+0000 7f1f50e00640 1 -- 172.21.15.45:0/3910661223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b956339460 msgr2=0x55b9563a3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:01:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.392+0000 7f1f50e00640 1 --2- 172.21.15.45:0/3910661223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b956339460 0x55b9563a3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.392+0000 7f1f50e00640 1 -- 172.21.15.45:0/3910661223 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b9563a3d20 con 0x55b9563300f0 2024-03-28T18:01:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.392+0000 7f1f50e00640 1 --2- 172.21.15.45:0/3910661223 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b9563300f0 0x55b956332ca0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f1f400bb370 tx=0x7f1f400f7b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c104cf53419a6f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:01:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.392+0000 7f1f32a00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T18:01:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.392+0000 7f1f32a00640 1 -- 172.21.15.45:0/3910661223 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f40100020 con 0x55b9563300f0 2024-03-28T18:01:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.392+0000 7f1f32a00640 1 -- 172.21.15.45:0/3910661223 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(140..140 src has 1..140) v4 ==== 11952+0+0 (secure 0 0 0) 0x7f1f40003d20 con 0x55b9563300f0 2024-03-28T18:01:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.392+0000 7f1f5643f740 1 -- 172.21.15.45:0/3910661223 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1f3801f700 con 0x55b9563300f0 2024-03-28T18:01:38.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.392+0000 7f1f32a00640 1 -- 172.21.15.45:0/3910661223 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=282) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f400bca20 con 0x55b9563300f0 2024-03-28T18:01:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.392+0000 7f1f5643f740 10 client.26501 fetch_fsmap learned FSMap version 282 2024-03-28T18:01:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.392+0000 7f1f5643f740 1 -- 172.21.15.45:0/3910661223 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=282,monmap=4+}) v3 -- 0x55b9563a3d20 con 0x55b9563300f0 2024-03-28T18:01:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.393+0000 7f1f32a00640 1 -- 172.21.15.45:0/3910661223 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f3801f700 con 0x55b9563300f0 2024-03-28T18:01:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.393+0000 7f1f32a00640 1 -- 172.21.15.45:0/3910661223 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 282) v1 ==== 4472+0+0 (secure 0 0 0) 0x7f1f400c53d0 con 0x55b9563300f0 2024-03-28T18:01:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.393+0000 7f1f5643f740 10 client.26501 fetch_fsmap finished waiting for FSMap version 282 2024-03-28T18:01:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.393+0000 7f1f5643f740 10 client.26501 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:01:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.393+0000 7f1f5643f740 20 client.26501 populate_metadata read hostname 'smithi045' 2024-03-28T18:01:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.393+0000 7f1f5643f740 1 --2- 172.21.15.45:0/3910661223 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55b956348ef0 0x55b9563a73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:01:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.393+0000 7f1f5643f740 4 client.26501 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:01:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.393+0000 7f1f5643f740 1 -- 172.21.15.45:0/3910661223 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b95629bae0 con 0x55b956348ef0 2024-03-28T18:01:38.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.393+0000 7f1f4be00640 1 --2- 172.21.15.45:0/3910661223 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55b956348ef0 0x55b9563a73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:01:38.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.394+0000 7f1f4be00640 1 --2- 172.21.15.45:0/3910661223 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55b956348ef0 0x55b9563a73d0 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a157d5be95ea9c7 server_cookie=11dde52b36699368 in_seq=0 out_seq=0 2024-03-28T18:01:38.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.394+0000 7f1f32a00640 10 client.26501 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:01:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.394+0000 7f1f32a00640 1 -- 172.21.15.45:0/3910661223 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55b95629bae0 con 0x55b956348ef0 2024-03-28T18:01:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.394+0000 7f1f32a00640 10 client.26501 handle_command_reply: tid=0 2024-03-28T18:01:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.394+0000 7f1f5643f740 1 client.26501 shutdown 2024-03-28T18:01:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.394+0000 7f1f5643f740 1 -- 172.21.15.45:0/3910661223 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b9563300f0 msgr2=0x55b956332ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.394+0000 7f1f5643f740 1 --2- 172.21.15.45:0/3910661223 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b9563300f0 0x55b956332ca0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f1f400bb370 tx=0x7f1f400f7b20 comp rx=0 tx=0).stop 2024-03-28T18:01:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.394+0000 7f1f5643f740 1 -- 172.21.15.45:0/3910661223 shutdown_connections 2024-03-28T18:01:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.394+0000 7f1f5643f740 1 --2- 172.21.15.45:0/3910661223 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55b956348ef0 0x55b9563a73d0 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.394+0000 7f1f5643f740 1 --2- 172.21.15.45:0/3910661223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b956339460 0x55b9563a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.394+0000 7f1f5643f740 1 --2- 172.21.15.45:0/3910661223 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55b956330dd0 0x55b9563331e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.394+0000 7f1f5643f740 1 --2- 172.21.15.45:0/3910661223 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b9563300f0 0x55b956332ca0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.394+0000 7f1f5643f740 1 -- 172.21.15.45:0/3910661223 >> 172.21.15.45:0/3910661223 conn(0x55b9562f7200 msgr2=0x55b956329330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.394+0000 7f1f32a00640 0 client.26501 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:01:38.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.394+0000 7f1f5643f740 1 -- 172.21.15.45:0/3910661223 shutdown_connections 2024-03-28T18:01:38.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.394+0000 7f1f5643f740 1 -- 172.21.15.45:0/3910661223 wait complete. 2024-03-28T18:01:38.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.394+0000 7f1f5643f740 20 client.26501 trim_cache size 0 max 16384 2024-03-28T18:01:38.399 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T18:01:38.399 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:01:38.399 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T18:01:38.399 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T18:01:38.399 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T18:01:38.399 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T18:01:38.399 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T18:01:38.399 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:01:38.399 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:01:38.400 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T18:01:38.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.395+0000 7f1f52200640 1 -- 172.21.15.45:0/3722264986 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f1f340a29f0 msgr2=0x7f1f340a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:38.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.395+0000 7f1f52200640 1 --2- 172.21.15.45:0/3722264986 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f1f340a29f0 0x7f1f340a4eb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c06ac50 tx=0x7f1f44009500 comp rx=0 tx=0).stop 2024-03-28T18:01:38.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.395+0000 7f1f52200640 1 -- 172.21.15.45:0/3722264986 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f4c073a40 msgr2=0x7f1f4c19e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:01:38.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.395+0000 7f1f52200640 1 --2- 172.21.15.45:0/3722264986 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f4c073a40 0x7f1f4c19e4d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f1f40000c00 tx=0x7f1f40004360 comp rx=0 tx=0).stop 2024-03-28T18:01:38.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.395+0000 7f1f52200640 1 -- 172.21.15.45:0/3722264986 shutdown_connections 2024-03-28T18:01:38.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.395+0000 7f1f52200640 1 --2- 172.21.15.45:0/3722264986 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f1f340a29f0 0x7f1f340a4eb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.395+0000 7f1f52200640 1 --2- 172.21.15.45:0/3722264986 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f4c112520 0x7f1f4c1a2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.395+0000 7f1f52200640 1 --2- 172.21.15.45:0/3722264986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f4c074360 0x7f1f4c19ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.395+0000 7f1f52200640 1 --2- 172.21.15.45:0/3722264986 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f4c073a40 0x7f1f4c19e4d0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:01:38.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.395+0000 7f1f52200640 1 -- 172.21.15.45:0/3722264986 >> 172.21.15.45:0/3722264986 conn(0x7f1f4c0fd000 msgr2=0x7f1f4c0fe7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:01:38.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.395+0000 7f1f52200640 1 -- 172.21.15.45:0/3722264986 shutdown_connections 2024-03-28T18:01:38.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:01:38.395+0000 7f1f52200640 1 -- 172.21.15.45:0/3722264986 wait complete. 2024-03-28T18:01:38.415 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T18:01:38.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:38 smithi045 ceph-mon[31773]: pgmap v436: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:01:38.929 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 551 sec latency 666826.516 ms 2024-03-28T18:01:38.961 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:38 smithi071 ceph-mon[36597]: pgmap v436: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:01:39.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:38 smithi151 ceph-mon[36481]: pgmap v436: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:01:39.929 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 552 sec latency 667826.642 ms 2024-03-28T18:01:40.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:39 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/1322964071,v1:172.21.15.45:6837/1322964071] up:resolve 2024-03-28T18:01:40.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:39 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:40.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:39 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/1322964071,v1:172.21.15.45:6837/1322964071] up:resolve 2024-03-28T18:01:40.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:39 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:40.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:39 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/1322964071,v1:172.21.15.45:6837/1322964071] up:resolve 2024-03-28T18:01:40.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:39 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:40.929 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 553 sec latency 668826.801 ms 2024-03-28T18:01:41.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:40 smithi071 ceph-mon[36597]: pgmap v438: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 135 B/s wr, 4 op/s 2024-03-28T18:01:41.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:40 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:01:41.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:40 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/1322964071,v1:172.21.15.45:6837/1322964071] up:reconnect 2024-03-28T18:01:41.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:40 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:41.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:40 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T18:01:41.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:40 smithi151 ceph-mon[36481]: pgmap v438: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 135 B/s wr, 4 op/s 2024-03-28T18:01:41.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:40 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:01:41.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:40 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/1322964071,v1:172.21.15.45:6837/1322964071] up:reconnect 2024-03-28T18:01:41.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:40 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:41.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:40 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T18:01:41.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:40 smithi045 ceph-mon[31773]: pgmap v438: 97 pgs: 97 active+clean; 27 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 135 B/s wr, 4 op/s 2024-03-28T18:01:41.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:40 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:01:41.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:40 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/1322964071,v1:172.21.15.45:6837/1322964071] up:reconnect 2024-03-28T18:01:41.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:40 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:41.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:40 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T18:01:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/1322964071,v1:172.21.15.45:6837/1322964071] up:rejoin 2024-03-28T18:01:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-mon[31773]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-03-28T18:01:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:01:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/1322964071,v1:172.21.15.45:6837/1322964071] up:active 2024-03-28T18:01:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 3 up:standby-replay 2024-03-28T18:01:41.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe8cd200700 time 2024-03-28T18:01:41.531954+0000 2024-03-28T18:01:41.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:01:41.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:41.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe8d8bea4a1] 2024-03-28T18:01:41.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe8d8bea667] 2024-03-28T18:01:41.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55b5f4158223] 2024-03-28T18:01:41.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55b5f3fd25d6] 2024-03-28T18:01:41.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55b5f3fd2da1] 2024-03-28T18:01:41.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55b5f3fd52d5] 2024-03-28T18:01:41.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55b5f4025233] 2024-03-28T18:01:41.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b5f4027197] 2024-03-28T18:01:41.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b5f3f2611b] 2024-03-28T18:01:41.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b5f3f266bc] 2024-03-28T18:01:41.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55b5f3f101ff] 2024-03-28T18:01:41.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe8d8e7c488] 2024-03-28T18:01:41.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 13: (DispatchQueue::entry()+0x51f) [0x7fe8d8e7960f] 2024-03-28T18:01:41.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe8d8f4ce81] 2024-03-28T18:01:41.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe8d79671ca] 2024-03-28T18:01:41.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 16: clone() 2024-03-28T18:01:41.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 2024-03-28T18:01:41.533+0000 7fe8cd200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe8cd200700 time 2024-03-28T18:01:41.531954+0000 2024-03-28T18:01:41.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:01:41.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 2024-03-28T18:01:41.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:41.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe8d8bea4a1] 2024-03-28T18:01:41.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe8d8bea667] 2024-03-28T18:01:41.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55b5f4158223] 2024-03-28T18:01:41.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55b5f3fd25d6] 2024-03-28T18:01:41.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55b5f3fd2da1] 2024-03-28T18:01:41.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55b5f3fd52d5] 2024-03-28T18:01:41.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55b5f4025233] 2024-03-28T18:01:41.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b5f4027197] 2024-03-28T18:01:41.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b5f3f2611b] 2024-03-28T18:01:41.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b5f3f266bc] 2024-03-28T18:01:41.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55b5f3f101ff] 2024-03-28T18:01:41.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe8d8e7c488] 2024-03-28T18:01:41.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 13: (DispatchQueue::entry()+0x51f) [0x7fe8d8e7960f] 2024-03-28T18:01:41.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe8d8f4ce81] 2024-03-28T18:01:41.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe8d79671ca] 2024-03-28T18:01:41.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 16: clone() 2024-03-28T18:01:41.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 2024-03-28T18:01:41.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: *** Caught signal (Aborted) ** 2024-03-28T18:01:41.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: in thread 7fe8cd200700 thread_name:ms_dispatch 2024-03-28T18:01:41.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:41.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe8d7971d20] 2024-03-28T18:01:41.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 2: gsignal() 2024-03-28T18:01:41.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 3: abort() 2024-03-28T18:01:41.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe8d8bea4fb] 2024-03-28T18:01:41.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe8d8bea667] 2024-03-28T18:01:41.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55b5f4158223] 2024-03-28T18:01:41.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55b5f3fd25d6] 2024-03-28T18:01:41.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55b5f3fd2da1] 2024-03-28T18:01:41.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55b5f3fd52d5] 2024-03-28T18:01:41.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55b5f4025233] 2024-03-28T18:01:41.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b5f4027197] 2024-03-28T18:01:41.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b5f3f2611b] 2024-03-28T18:01:41.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b5f3f266bc] 2024-03-28T18:01:41.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55b5f3f101ff] 2024-03-28T18:01:41.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe8d8e7c488] 2024-03-28T18:01:41.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 16: (DispatchQueue::entry()+0x51f) [0x7fe8d8e7960f] 2024-03-28T18:01:41.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe8d8f4ce81] 2024-03-28T18:01:41.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe8d79671ca] 2024-03-28T18:01:41.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 19: clone() 2024-03-28T18:01:41.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 2024-03-28T18:01:41.535+0000 7fe8cd200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:01:41.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: in thread 7fe8cd200700 thread_name:ms_dispatch 2024-03-28T18:01:41.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 2024-03-28T18:01:41.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:41.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe8d7971d20] 2024-03-28T18:01:41.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 2: gsignal() 2024-03-28T18:01:41.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 3: abort() 2024-03-28T18:01:41.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe8d8bea4fb] 2024-03-28T18:01:41.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe8d8bea667] 2024-03-28T18:01:41.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55b5f4158223] 2024-03-28T18:01:41.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55b5f3fd25d6] 2024-03-28T18:01:41.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55b5f3fd2da1] 2024-03-28T18:01:41.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55b5f3fd52d5] 2024-03-28T18:01:41.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55b5f4025233] 2024-03-28T18:01:41.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b5f4027197] 2024-03-28T18:01:41.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b5f3f2611b] 2024-03-28T18:01:41.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b5f3f266bc] 2024-03-28T18:01:41.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55b5f3f101ff] 2024-03-28T18:01:41.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe8d8e7c488] 2024-03-28T18:01:41.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 16: (DispatchQueue::entry()+0x51f) [0x7fe8d8e7960f] 2024-03-28T18:01:41.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe8d8f4ce81] 2024-03-28T18:01:41.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe8d79671ca] 2024-03-28T18:01:41.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 19: clone() 2024-03-28T18:01:41.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:01:41.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 2024-03-28T18:01:41.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: -1> 2024-03-28T18:01:41.533+0000 7fe8cd200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe8cd200700 time 2024-03-28T18:01:41.531954+0000 2024-03-28T18:01:41.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:01:41.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 2024-03-28T18:01:41.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:41.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe8d8bea4a1] 2024-03-28T18:01:41.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe8d8bea667] 2024-03-28T18:01:41.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55b5f4158223] 2024-03-28T18:01:41.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55b5f3fd25d6] 2024-03-28T18:01:41.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55b5f3fd2da1] 2024-03-28T18:01:41.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55b5f3fd52d5] 2024-03-28T18:01:41.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55b5f4025233] 2024-03-28T18:01:41.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b5f4027197] 2024-03-28T18:01:41.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b5f3f2611b] 2024-03-28T18:01:41.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b5f3f266bc] 2024-03-28T18:01:41.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55b5f3f101ff] 2024-03-28T18:01:41.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe8d8e7c488] 2024-03-28T18:01:41.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 13: (DispatchQueue::entry()+0x51f) [0x7fe8d8e7960f] 2024-03-28T18:01:41.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe8d8f4ce81] 2024-03-28T18:01:41.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe8d79671ca] 2024-03-28T18:01:41.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 16: clone() 2024-03-28T18:01:41.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 2024-03-28T18:01:41.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 0> 2024-03-28T18:01:41.535+0000 7fe8cd200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:01:41.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: in thread 7fe8cd200700 thread_name:ms_dispatch 2024-03-28T18:01:41.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 2024-03-28T18:01:41.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:41.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe8d7971d20] 2024-03-28T18:01:41.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 2: gsignal() 2024-03-28T18:01:41.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 3: abort() 2024-03-28T18:01:41.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe8d8bea4fb] 2024-03-28T18:01:41.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe8d8bea667] 2024-03-28T18:01:41.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55b5f4158223] 2024-03-28T18:01:41.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55b5f3fd25d6] 2024-03-28T18:01:41.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55b5f3fd2da1] 2024-03-28T18:01:41.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55b5f3fd52d5] 2024-03-28T18:01:41.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55b5f4025233] 2024-03-28T18:01:41.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b5f4027197] 2024-03-28T18:01:41.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b5f3f2611b] 2024-03-28T18:01:41.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b5f3f266bc] 2024-03-28T18:01:41.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55b5f3f101ff] 2024-03-28T18:01:41.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe8d8e7c488] 2024-03-28T18:01:41.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 16: (DispatchQueue::entry()+0x51f) [0x7fe8d8e7960f] 2024-03-28T18:01:41.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe8d8f4ce81] 2024-03-28T18:01:41.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe8d79671ca] 2024-03-28T18:01:41.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 19: clone() 2024-03-28T18:01:41.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:01:41.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 2024-03-28T18:01:41.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: -1> 2024-03-28T18:01:41.533+0000 7fe8cd200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe8cd200700 time 2024-03-28T18:01:41.531954+0000 2024-03-28T18:01:41.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:01:41.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 2024-03-28T18:01:41.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:41.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe8d8bea4a1] 2024-03-28T18:01:41.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe8d8bea667] 2024-03-28T18:01:41.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55b5f4158223] 2024-03-28T18:01:41.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55b5f3fd25d6] 2024-03-28T18:01:41.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55b5f3fd2da1] 2024-03-28T18:01:41.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55b5f3fd52d5] 2024-03-28T18:01:41.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55b5f4025233] 2024-03-28T18:01:41.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b5f4027197] 2024-03-28T18:01:41.929 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b5f3f2611b] 2024-03-28T18:01:41.929 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b5f3f266bc] 2024-03-28T18:01:41.929 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55b5f3f101ff] 2024-03-28T18:01:41.929 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe8d8e7c488] 2024-03-28T18:01:41.929 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 13: (DispatchQueue::entry()+0x51f) [0x7fe8d8e7960f] 2024-03-28T18:01:41.929 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe8d8f4ce81] 2024-03-28T18:01:41.929 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe8d79671ca] 2024-03-28T18:01:41.929 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 16: clone() 2024-03-28T18:01:41.929 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 2024-03-28T18:01:41.929 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 0> 2024-03-28T18:01:41.535+0000 7fe8cd200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:01:41.929 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: in thread 7fe8cd200700 thread_name:ms_dispatch 2024-03-28T18:01:41.929 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 2024-03-28T18:01:41.929 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:41.929 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe8d7971d20] 2024-03-28T18:01:41.929 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 2: gsignal() 2024-03-28T18:01:41.930 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 3: abort() 2024-03-28T18:01:41.930 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe8d8bea4fb] 2024-03-28T18:01:41.930 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe8d8bea667] 2024-03-28T18:01:41.930 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55b5f4158223] 2024-03-28T18:01:41.930 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55b5f3fd25d6] 2024-03-28T18:01:41.930 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55b5f3fd2da1] 2024-03-28T18:01:41.930 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55b5f3fd52d5] 2024-03-28T18:01:41.930 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55b5f4025233] 2024-03-28T18:01:41.930 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55b5f4027197] 2024-03-28T18:01:41.930 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55b5f3f2611b] 2024-03-28T18:01:41.930 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b5f3f266bc] 2024-03-28T18:01:41.930 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55b5f3f101ff] 2024-03-28T18:01:41.930 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe8d8e7c488] 2024-03-28T18:01:41.930 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 16: (DispatchQueue::entry()+0x51f) [0x7fe8d8e7960f] 2024-03-28T18:01:41.930 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe8d8f4ce81] 2024-03-28T18:01:41.931 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe8d79671ca] 2024-03-28T18:01:41.931 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 19: clone() 2024-03-28T18:01:41.931 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:01:41.931 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[94776]: 2024-03-28T18:01:41.931 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:41 smithi045 podman[95155]: 2024-03-28 18:01:41.649258896 +0000 UTC m=+0.044346577 container died e8ad78db5928a4ade14c82b252e34bcd2c6301a07f542c91d6dd280701e231c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-28T18:01:41.931 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 554 sec latency 669826.897 ms 2024-03-28T18:01:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:41 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/1322964071,v1:172.21.15.45:6837/1322964071] up:rejoin 2024-03-28T18:01:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:41 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:41 smithi071 ceph-mon[36597]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-03-28T18:01:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:41 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:01:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:41 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/1322964071,v1:172.21.15.45:6837/1322964071] up:active 2024-03-28T18:01:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:41 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 3 up:standby-replay 2024-03-28T18:01:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:41 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/1322964071,v1:172.21.15.45:6837/1322964071] up:rejoin 2024-03-28T18:01:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:41 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:41 smithi151 ceph-mon[36481]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-03-28T18:01:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:41 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:01:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:41 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/1322964071,v1:172.21.15.45:6837/1322964071] up:active 2024-03-28T18:01:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:41 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 3 up:standby-replay 2024-03-28T18:01:42.693 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:42 smithi045 podman[95155]: 2024-03-28 18:01:42.425895469 +0000 UTC m=+0.820983164 container remove e8ad78db5928a4ade14c82b252e34bcd2c6301a07f542c91d6dd280701e231c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-28T18:01:42.693 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:42 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:01:42.930 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 555 sec latency 670826.998 ms 2024-03-28T18:01:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:42 smithi071 ceph-mon[36597]: pgmap v439: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T18:01:43.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:42 smithi151 ceph-mon[36481]: pgmap v439: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T18:01:43.166 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:42 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-28T18:01:43.166 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:42 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Consumed 1.568s CPU time. 2024-03-28T18:01:43.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:42 smithi045 ceph-mon[31773]: pgmap v439: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T18:01:43.930 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 556 sec latency 671827.153 ms 2024-03-28T18:01:44.930 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 557 sec latency 672827.316 ms 2024-03-28T18:01:45.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:44 smithi071 ceph-mon[36597]: pgmap v440: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T18:01:45.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:44 smithi151 ceph-mon[36481]: pgmap v440: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T18:01:45.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:44 smithi045 ceph-mon[31773]: pgmap v440: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-28T18:01:45.930 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 558 sec latency 673827.457 ms 2024-03-28T18:01:46.930 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 559 sec latency 674827.602 ms 2024-03-28T18:01:46.968 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:46 smithi045 ceph-mon[31773]: pgmap v441: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 353 B/s wr, 13 op/s 2024-03-28T18:01:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:46 smithi071 ceph-mon[36597]: pgmap v441: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 353 B/s wr, 13 op/s 2024-03-28T18:01:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:46 smithi151 ceph-mon[36481]: pgmap v441: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 353 B/s wr, 13 op/s 2024-03-28T18:01:47.931 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 560 sec latency 675827.744 ms 2024-03-28T18:01:48.930 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 561 sec latency 676827.863 ms 2024-03-28T18:01:49.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:48 smithi071 ceph-mon[36597]: pgmap v442: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T18:01:49.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:48 smithi151 ceph-mon[36481]: pgmap v442: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T18:01:49.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:48 smithi045 ceph-mon[31773]: pgmap v442: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-28T18:01:49.931 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 562 sec latency 677827.936 ms 2024-03-28T18:01:50.931 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 563 sec latency 678828.500 ms 2024-03-28T18:01:51.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:50 smithi071 ceph-mon[36597]: pgmap v443: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 256 B/s wr, 9 op/s 2024-03-28T18:01:51.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:50 smithi151 ceph-mon[36481]: pgmap v443: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 256 B/s wr, 9 op/s 2024-03-28T18:01:51.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:50 smithi045 ceph-mon[31773]: pgmap v443: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 256 B/s wr, 9 op/s 2024-03-28T18:01:51.931 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 564 sec latency 679828.634 ms 2024-03-28T18:01:52.932 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 565 sec latency 680828.782 ms 2024-03-28T18:01:53.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:52 smithi071 ceph-mon[36597]: pgmap v444: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T18:01:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:52 smithi151 ceph-mon[36481]: pgmap v444: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T18:01:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:52 smithi045 ceph-mon[31773]: pgmap v444: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T18:01:53.601 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:53 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 4. 2024-03-28T18:01:53.601 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:53 smithi045 systemd[1]: Stopped Ceph mds.g for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:01:53.601 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:53 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Consumed 1.568s CPU time. 2024-03-28T18:01:53.601 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:53 smithi045 systemd[1]: Starting Ceph mds.g for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T18:01:53.916 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:53 smithi045 podman[95259]: 2024-03-28 18:01:53.506522094 +0000 UTC m=+0.039021723 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:01:53.916 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:53 smithi045 podman[95259]: 2024-03-28 18:01:53.848469197 +0000 UTC m=+0.380968911 container create 8c91cc183814082fd26233cc430ceb6a2696ec690c1ee9edcbc04080a05d87fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-28T18:01:53.931 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 566 sec latency 681828.924 ms 2024-03-28T18:01:54.416 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:54 smithi045 podman[95259]: 2024-03-28 18:01:54.157079612 +0000 UTC m=+0.689579300 container init 8c91cc183814082fd26233cc430ceb6a2696ec690c1ee9edcbc04080a05d87fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:01:54.416 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:54 smithi045 podman[95259]: 2024-03-28 18:01:54.169137183 +0000 UTC m=+0.701636857 container start 8c91cc183814082fd26233cc430ceb6a2696ec690c1ee9edcbc04080a05d87fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:01:54.416 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:54 smithi045 ceph-mds[95277]: -- 172.21.15.45:0/21805439 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558fed515a40 con 0x558fed4e8800 2024-03-28T18:01:54.416 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:54 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: starting mds.g at 2024-03-28T18:01:54.416 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:54 smithi045 bash[95259]: 8c91cc183814082fd26233cc430ceb6a2696ec690c1ee9edcbc04080a05d87fb 2024-03-28T18:01:54.416 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:01:54 smithi045 systemd[1]: Started Ceph mds.g for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:01:54.932 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 567 sec latency 682829.069 ms 2024-03-28T18:01:55.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:54 smithi071 ceph-mon[36597]: pgmap v445: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T18:01:55.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:54 smithi071 ceph-mon[36597]: daemon mds.g restarted 2024-03-28T18:01:55.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:54 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:01:55.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:54 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:01:55.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:54 smithi071 ceph-mon[36597]: osdmap e141: 12 total, 12 up, 12 in 2024-03-28T18:01:55.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:54 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2588031165,v1:172.21.15.45:6837/2588031165] up:boot 2024-03-28T18:01:55.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:54 smithi071 ceph-mon[36597]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-03-28T18:01:55.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:54 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:01:55.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:54 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:55.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:54 smithi151 ceph-mon[36481]: pgmap v445: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T18:01:55.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:54 smithi151 ceph-mon[36481]: daemon mds.g restarted 2024-03-28T18:01:55.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:54 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:01:55.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:54 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:01:55.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:54 smithi151 ceph-mon[36481]: osdmap e141: 12 total, 12 up, 12 in 2024-03-28T18:01:55.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:54 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2588031165,v1:172.21.15.45:6837/2588031165] up:boot 2024-03-28T18:01:55.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:54 smithi151 ceph-mon[36481]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-03-28T18:01:55.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:54 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:01:55.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:54 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:55.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:54 smithi045 ceph-mon[31773]: pgmap v445: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T18:01:55.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:54 smithi045 ceph-mon[31773]: daemon mds.g restarted 2024-03-28T18:01:55.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:54 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:01:55.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:54 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:01:55.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:54 smithi045 ceph-mon[31773]: osdmap e141: 12 total, 12 up, 12 in 2024-03-28T18:01:55.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:54 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2588031165,v1:172.21.15.45:6837/2588031165] up:boot 2024-03-28T18:01:55.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:54 smithi045 ceph-mon[31773]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-03-28T18:01:55.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:54 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:01:55.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:54 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:55.932 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 568 sec latency 683829.199 ms 2024-03-28T18:01:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:56 smithi071 ceph-mon[36597]: pgmap v447: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-28T18:01:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:56 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/4155871960,v1:172.21.15.45:6835/4155871960] up:resolve 2024-03-28T18:01:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:56 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:56 smithi151 ceph-mon[36481]: pgmap v447: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-28T18:01:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:56 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/4155871960,v1:172.21.15.45:6835/4155871960] up:resolve 2024-03-28T18:01:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:56 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:56.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:56 smithi045 ceph-mon[31773]: pgmap v447: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-28T18:01:56.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:56 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/4155871960,v1:172.21.15.45:6835/4155871960] up:resolve 2024-03-28T18:01:56.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:56 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:56.932 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 569 sec latency 684829.347 ms 2024-03-28T18:01:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:57 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:01:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:57 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/4155871960,v1:172.21.15.45:6835/4155871960] up:reconnect 2024-03-28T18:01:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:57 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:57 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999984 2024-03-28T18:01:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:57 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:01:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:57 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/4155871960,v1:172.21.15.45:6835/4155871960] up:reconnect 2024-03-28T18:01:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:57 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:57 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999984 2024-03-28T18:01:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:57 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:01:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:57 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/4155871960,v1:172.21.15.45:6835/4155871960] up:reconnect 2024-03-28T18:01:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:57 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:57 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999984 2024-03-28T18:01:57.932 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 570 sec latency 685829.505 ms 2024-03-28T18:01:58.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:58 smithi071 ceph-mon[36597]: pgmap v448: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-28T18:01:58.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:58 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/4155871960,v1:172.21.15.45:6835/4155871960] up:rejoin 2024-03-28T18:01:58.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:58 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:58.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:58 smithi071 ceph-mon[36597]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-03-28T18:01:58.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:58 smithi151 ceph-mon[36481]: pgmap v448: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-28T18:01:58.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:58 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/4155871960,v1:172.21.15.45:6835/4155871960] up:rejoin 2024-03-28T18:01:58.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:58 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:58.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:58 smithi151 ceph-mon[36481]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-03-28T18:01:58.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-mon[31773]: pgmap v448: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-28T18:01:58.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/4155871960,v1:172.21.15.45:6835/4155871960] up:rejoin 2024-03-28T18:01:58.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:58.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-mon[31773]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-03-28T18:01:58.667 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe262600700 time 2024-03-28T18:01:58.292197+0000 2024-03-28T18:01:58.667 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:01:58.667 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:58.667 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe26df124a1] 2024-03-28T18:01:58.667 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe26df12667] 2024-03-28T18:01:58.667 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5604ba004223] 2024-03-28T18:01:58.667 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5604b9e7e5d6] 2024-03-28T18:01:58.667 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5604b9e7eda1] 2024-03-28T18:01:58.667 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5604b9e812d5] 2024-03-28T18:01:58.667 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5604b9ed1233] 2024-03-28T18:01:58.668 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5604b9ed3197] 2024-03-28T18:01:58.668 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5604b9dd211b] 2024-03-28T18:01:58.668 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5604b9dd26bc] 2024-03-28T18:01:58.668 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5604b9dbc1ff] 2024-03-28T18:01:58.668 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe26e1a4488] 2024-03-28T18:01:58.668 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 13: (DispatchQueue::entry()+0x51f) [0x7fe26e1a160f] 2024-03-28T18:01:58.668 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe26e274e81] 2024-03-28T18:01:58.668 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe26cc8f1ca] 2024-03-28T18:01:58.668 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 16: clone() 2024-03-28T18:01:58.668 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 2024-03-28T18:01:58.293+0000 7fe262600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe262600700 time 2024-03-28T18:01:58.292197+0000 2024-03-28T18:01:58.669 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:01:58.669 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 2024-03-28T18:01:58.669 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:58.669 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe26df124a1] 2024-03-28T18:01:58.669 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe26df12667] 2024-03-28T18:01:58.669 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5604ba004223] 2024-03-28T18:01:58.669 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5604b9e7e5d6] 2024-03-28T18:01:58.669 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5604b9e7eda1] 2024-03-28T18:01:58.669 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5604b9e812d5] 2024-03-28T18:01:58.669 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5604b9ed1233] 2024-03-28T18:01:58.669 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5604b9ed3197] 2024-03-28T18:01:58.669 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5604b9dd211b] 2024-03-28T18:01:58.669 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5604b9dd26bc] 2024-03-28T18:01:58.669 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5604b9dbc1ff] 2024-03-28T18:01:58.669 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe26e1a4488] 2024-03-28T18:01:58.670 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 13: (DispatchQueue::entry()+0x51f) [0x7fe26e1a160f] 2024-03-28T18:01:58.670 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe26e274e81] 2024-03-28T18:01:58.670 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe26cc8f1ca] 2024-03-28T18:01:58.670 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 16: clone() 2024-03-28T18:01:58.670 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 2024-03-28T18:01:58.670 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: *** Caught signal (Aborted) ** 2024-03-28T18:01:58.670 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: in thread 7fe262600700 thread_name:ms_dispatch 2024-03-28T18:01:58.670 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:58.670 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe26cc99d20] 2024-03-28T18:01:58.670 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 2: gsignal() 2024-03-28T18:01:58.671 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 3: abort() 2024-03-28T18:01:58.671 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe26df124fb] 2024-03-28T18:01:58.671 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe26df12667] 2024-03-28T18:01:58.671 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5604ba004223] 2024-03-28T18:01:58.671 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5604b9e7e5d6] 2024-03-28T18:01:58.671 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5604b9e7eda1] 2024-03-28T18:01:58.671 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5604b9e812d5] 2024-03-28T18:01:58.671 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5604b9ed1233] 2024-03-28T18:01:58.671 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5604b9ed3197] 2024-03-28T18:01:58.671 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5604b9dd211b] 2024-03-28T18:01:58.671 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5604b9dd26bc] 2024-03-28T18:01:58.671 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5604b9dbc1ff] 2024-03-28T18:01:58.671 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe26e1a4488] 2024-03-28T18:01:58.671 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 16: (DispatchQueue::entry()+0x51f) [0x7fe26e1a160f] 2024-03-28T18:01:58.671 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe26e274e81] 2024-03-28T18:01:58.672 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe26cc8f1ca] 2024-03-28T18:01:58.672 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 19: clone() 2024-03-28T18:01:58.672 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 2024-03-28T18:01:58.296+0000 7fe262600700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:01:58.672 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: in thread 7fe262600700 thread_name:ms_dispatch 2024-03-28T18:01:58.672 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 2024-03-28T18:01:58.672 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:58.672 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe26cc99d20] 2024-03-28T18:01:58.672 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 2: gsignal() 2024-03-28T18:01:58.672 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 3: abort() 2024-03-28T18:01:58.672 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe26df124fb] 2024-03-28T18:01:58.672 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe26df12667] 2024-03-28T18:01:58.673 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5604ba004223] 2024-03-28T18:01:58.673 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5604b9e7e5d6] 2024-03-28T18:01:58.673 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5604b9e7eda1] 2024-03-28T18:01:58.673 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5604b9e812d5] 2024-03-28T18:01:58.673 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5604b9ed1233] 2024-03-28T18:01:58.673 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5604b9ed3197] 2024-03-28T18:01:58.673 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5604b9dd211b] 2024-03-28T18:01:58.673 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5604b9dd26bc] 2024-03-28T18:01:58.673 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5604b9dbc1ff] 2024-03-28T18:01:58.673 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe26e1a4488] 2024-03-28T18:01:58.673 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 16: (DispatchQueue::entry()+0x51f) [0x7fe26e1a160f] 2024-03-28T18:01:58.673 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe26e274e81] 2024-03-28T18:01:58.673 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe26cc8f1ca] 2024-03-28T18:01:58.673 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 19: clone() 2024-03-28T18:01:58.673 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:01:58.674 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 2024-03-28T18:01:58.674 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: -1> 2024-03-28T18:01:58.293+0000 7fe262600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe262600700 time 2024-03-28T18:01:58.292197+0000 2024-03-28T18:01:58.674 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:01:58.674 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 2024-03-28T18:01:58.674 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:58.674 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe26df124a1] 2024-03-28T18:01:58.674 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe26df12667] 2024-03-28T18:01:58.674 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5604ba004223] 2024-03-28T18:01:58.674 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5604b9e7e5d6] 2024-03-28T18:01:58.675 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5604b9e7eda1] 2024-03-28T18:01:58.675 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5604b9e812d5] 2024-03-28T18:01:58.675 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5604b9ed1233] 2024-03-28T18:01:58.675 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5604b9ed3197] 2024-03-28T18:01:58.675 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5604b9dd211b] 2024-03-28T18:01:58.675 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5604b9dd26bc] 2024-03-28T18:01:58.675 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5604b9dbc1ff] 2024-03-28T18:01:58.675 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe26e1a4488] 2024-03-28T18:01:58.675 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 13: (DispatchQueue::entry()+0x51f) [0x7fe26e1a160f] 2024-03-28T18:01:58.675 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe26e274e81] 2024-03-28T18:01:58.675 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe26cc8f1ca] 2024-03-28T18:01:58.675 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 16: clone() 2024-03-28T18:01:58.675 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 2024-03-28T18:01:58.675 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 0> 2024-03-28T18:01:58.296+0000 7fe262600700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:01:58.675 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: in thread 7fe262600700 thread_name:ms_dispatch 2024-03-28T18:01:58.676 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 2024-03-28T18:01:58.676 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:58.676 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe26cc99d20] 2024-03-28T18:01:58.676 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 2: gsignal() 2024-03-28T18:01:58.676 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 3: abort() 2024-03-28T18:01:58.676 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe26df124fb] 2024-03-28T18:01:58.676 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe26df12667] 2024-03-28T18:01:58.676 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5604ba004223] 2024-03-28T18:01:58.676 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5604b9e7e5d6] 2024-03-28T18:01:58.676 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5604b9e7eda1] 2024-03-28T18:01:58.676 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5604b9e812d5] 2024-03-28T18:01:58.676 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5604b9ed1233] 2024-03-28T18:01:58.677 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5604b9ed3197] 2024-03-28T18:01:58.677 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5604b9dd211b] 2024-03-28T18:01:58.677 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5604b9dd26bc] 2024-03-28T18:01:58.677 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5604b9dbc1ff] 2024-03-28T18:01:58.677 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe26e1a4488] 2024-03-28T18:01:58.677 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 16: (DispatchQueue::entry()+0x51f) [0x7fe26e1a160f] 2024-03-28T18:01:58.677 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe26e274e81] 2024-03-28T18:01:58.677 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe26cc8f1ca] 2024-03-28T18:01:58.677 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 19: clone() 2024-03-28T18:01:58.677 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:01:58.677 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 2024-03-28T18:01:58.677 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: -1> 2024-03-28T18:01:58.293+0000 7fe262600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe262600700 time 2024-03-28T18:01:58.292197+0000 2024-03-28T18:01:58.677 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:01:58.677 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 2024-03-28T18:01:58.678 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:58.678 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe26df124a1] 2024-03-28T18:01:58.678 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe26df12667] 2024-03-28T18:01:58.678 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5604ba004223] 2024-03-28T18:01:58.678 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5604b9e7e5d6] 2024-03-28T18:01:58.678 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5604b9e7eda1] 2024-03-28T18:01:58.678 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5604b9e812d5] 2024-03-28T18:01:58.679 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5604b9ed1233] 2024-03-28T18:01:58.679 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5604b9ed3197] 2024-03-28T18:01:58.679 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5604b9dd211b] 2024-03-28T18:01:58.679 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5604b9dd26bc] 2024-03-28T18:01:58.679 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5604b9dbc1ff] 2024-03-28T18:01:58.679 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe26e1a4488] 2024-03-28T18:01:58.679 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 13: (DispatchQueue::entry()+0x51f) [0x7fe26e1a160f] 2024-03-28T18:01:58.679 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe26e274e81] 2024-03-28T18:01:58.679 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe26cc8f1ca] 2024-03-28T18:01:58.679 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 16: clone() 2024-03-28T18:01:58.679 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 2024-03-28T18:01:58.679 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 0> 2024-03-28T18:01:58.296+0000 7fe262600700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:01:58.679 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: in thread 7fe262600700 thread_name:ms_dispatch 2024-03-28T18:01:58.679 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 2024-03-28T18:01:58.679 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:01:58.680 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe26cc99d20] 2024-03-28T18:01:58.680 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 2: gsignal() 2024-03-28T18:01:58.680 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 3: abort() 2024-03-28T18:01:58.680 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe26df124fb] 2024-03-28T18:01:58.680 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe26df12667] 2024-03-28T18:01:58.680 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5604ba004223] 2024-03-28T18:01:58.680 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5604b9e7e5d6] 2024-03-28T18:01:58.680 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5604b9e7eda1] 2024-03-28T18:01:58.680 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5604b9e812d5] 2024-03-28T18:01:58.680 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5604b9ed1233] 2024-03-28T18:01:58.680 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5604b9ed3197] 2024-03-28T18:01:58.680 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5604b9dd211b] 2024-03-28T18:01:58.680 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5604b9dd26bc] 2024-03-28T18:01:58.680 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5604b9dbc1ff] 2024-03-28T18:01:58.680 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe26e1a4488] 2024-03-28T18:01:58.681 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 16: (DispatchQueue::entry()+0x51f) [0x7fe26e1a160f] 2024-03-28T18:01:58.681 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe26e274e81] 2024-03-28T18:01:58.681 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe26cc8f1ca] 2024-03-28T18:01:58.681 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 19: clone() 2024-03-28T18:01:58.681 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:01:58.681 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95052]: 2024-03-28T18:01:58.681 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:58 smithi045 podman[95311]: 2024-03-28 18:01:58.413636234 +0000 UTC m=+0.037934365 container died a92c50311e28e3a30f961ff57ba85b45de302aad1d969d2dc21b465f7ef32efb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-28T18:01:58.932 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 571 sec latency 686829.637 ms 2024-03-28T18:01:59.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:59 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:01:59.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:59 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/4155871960,v1:172.21.15.45:6835/4155871960] up:active 2024-03-28T18:01:59.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:59 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:59.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:01:59 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 3 up:standby-replay 2024-03-28T18:01:59.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:59 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:01:59.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:59 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/4155871960,v1:172.21.15.45:6835/4155871960] up:active 2024-03-28T18:01:59.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:59 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:59.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:01:59 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 3 up:standby-replay 2024-03-28T18:01:59.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:59 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:01:59.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:59 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/4155871960,v1:172.21.15.45:6835/4155871960] up:active 2024-03-28T18:01:59.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:59 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:01:59.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:01:59 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 3 up:standby-replay 2024-03-28T18:01:59.666 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:59 smithi045 podman[95311]: 2024-03-28 18:01:59.257684577 +0000 UTC m=+0.881982736 container remove a92c50311e28e3a30f961ff57ba85b45de302aad1d969d2dc21b465f7ef32efb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-28T18:01:59.666 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:59 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:01:59.932 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 572 sec latency 687829.788 ms 2024-03-28T18:02:00.166 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:59 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-28T18:02:00.166 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:01:59 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Consumed 1.646s CPU time. 2024-03-28T18:02:00.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:00 smithi071 ceph-mon[36597]: pgmap v449: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-28T18:02:00.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:00 smithi151 ceph-mon[36481]: pgmap v449: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-28T18:02:00.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:00 smithi045 ceph-mon[31773]: pgmap v449: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-28T18:02:00.933 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 573 sec latency 688829.940 ms 2024-03-28T18:02:01.933 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 574 sec latency 689830.088 ms 2024-03-28T18:02:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:02 smithi071 ceph-mon[36597]: pgmap v450: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T18:02:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:02 smithi151 ceph-mon[36481]: pgmap v450: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T18:02:02.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:02 smithi045 ceph-mon[31773]: pgmap v450: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T18:02:02.933 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 575 sec latency 690830.224 ms 2024-03-28T18:02:03.933 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 576 sec latency 691830.382 ms 2024-03-28T18:02:04.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:04 smithi045 ceph-mon[31773]: pgmap v451: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T18:02:04.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:04 smithi071 ceph-mon[36597]: pgmap v451: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T18:02:04.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:04 smithi151 ceph-mon[36481]: pgmap v451: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T18:02:04.934 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 577 sec latency 692830.510 ms 2024-03-28T18:02:05.933 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 578 sec latency 693830.654 ms 2024-03-28T18:02:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:06 smithi071 ceph-mon[36597]: pgmap v452: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 186 B/s wr, 14 op/s 2024-03-28T18:02:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:06 smithi151 ceph-mon[36481]: pgmap v452: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 186 B/s wr, 14 op/s 2024-03-28T18:02:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:06 smithi045 ceph-mon[31773]: pgmap v452: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 186 B/s wr, 14 op/s 2024-03-28T18:02:06.933 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 579 sec latency 694830.791 ms 2024-03-28T18:02:07.934 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 580 sec latency 695830.951 ms 2024-03-28T18:02:08.416 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:02:08.416 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:02:08.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:08 smithi071 ceph-mon[36597]: pgmap v453: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T18:02:08.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:08 smithi151 ceph-mon[36481]: pgmap v453: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T18:02:08.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.659+0000 7fb1f9e00640 1 -- 172.21.15.45:0/2900323517 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1f4073a40 msgr2=0x7fb1f4073e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:08.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.659+0000 7fb1f9e00640 1 --2- 172.21.15.45:0/2900323517 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1f4073a40 0x7fb1f4073e20 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4009900 tx=0x7fb1e402bcd0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.659+0000 7fb1f9e00640 1 -- 172.21.15.45:0/2900323517 shutdown_connections 2024-03-28T18:02:08.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.659+0000 7fb1f9e00640 1 --2- 172.21.15.45:0/2900323517 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f4112640 0x7fb1f4114a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.659+0000 7fb1f9e00640 1 --2- 172.21.15.45:0/2900323517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1f4074360 0x7fb1f4112100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.659+0000 7fb1f9e00640 1 --2- 172.21.15.45:0/2900323517 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1f4073a40 0x7fb1f4073e20 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.659+0000 7fb1f9e00640 1 -- 172.21.15.45:0/2900323517 >> 172.21.15.45:0/2900323517 conn(0x7fb1f4078aa0 msgr2=0x7fb1f4078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:02:08.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.659+0000 7fb1f9e00640 1 -- 172.21.15.45:0/2900323517 shutdown_connections 2024-03-28T18:02:08.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.659+0000 7fb1f9e00640 1 -- 172.21.15.45:0/2900323517 wait complete. 2024-03-28T18:02:08.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.660+0000 7fb1f9e00640 1 Processor -- start 2024-03-28T18:02:08.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.660+0000 7fb1f9e00640 1 -- start start 2024-03-28T18:02:08.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.661+0000 7fb1f9e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f4073a40 0x7fb1f419e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:08.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.661+0000 7fb1f9e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1f4074360 0x7fb1f419ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:08.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.661+0000 7fb1f9e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1f4112640 0x7fb1f41a2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:08.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.661+0000 7fb1f9e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f4118090 con 0x7fb1f4112640 2024-03-28T18:02:08.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.661+0000 7fb1f9e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f4117f10 con 0x7fb1f4074360 2024-03-28T18:02:08.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.661+0000 7fb1f9e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f4118210 con 0x7fb1f4073a40 2024-03-28T18:02:08.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.661+0000 7fb1ebe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1f4074360 0x7fb1f419ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:08.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.661+0000 7fb1ebe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1f4074360 0x7fb1f419ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:56250/0 (socket says 172.21.15.45:56250) 2024-03-28T18:02:08.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.661+0000 7fb1f3400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1f4112640 0x7fb1f41a2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:08.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.661+0000 7fb1ebe00640 1 -- 172.21.15.45:0/474504533 learned_addr learned my addr 172.21.15.45:0/474504533 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:02:08.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.661+0000 7fb1f2a00640 1 --2- 172.21.15.45:0/474504533 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f4073a40 0x7fb1f419e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:08.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.662+0000 7fb1ebe00640 1 -- 172.21.15.45:0/474504533 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f4073a40 msgr2=0x7fb1f419e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:08.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.662+0000 7fb1ebe00640 1 --2- 172.21.15.45:0/474504533 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f4073a40 0x7fb1f419e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.662+0000 7fb1ebe00640 1 -- 172.21.15.45:0/474504533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1f4112640 msgr2=0x7fb1f41a2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:08.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.662+0000 7fb1ebe00640 1 --2- 172.21.15.45:0/474504533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1f4112640 0x7fb1f41a2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.662+0000 7fb1ebe00640 1 -- 172.21.15.45:0/474504533 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f41a3520 con 0x7fb1f4074360 2024-03-28T18:02:08.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.662+0000 7fb1ebe00640 1 --2- 172.21.15.45:0/474504533 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1f4074360 0x7fb1f419ea70 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc00e9f0 tx=0x7fb1dc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:02:08.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.662+0000 7fb1f8a00640 1 -- 172.21.15.45:0/474504533 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc00cd80 con 0x7fb1f4074360 2024-03-28T18:02:08.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.662+0000 7fb1f8a00640 1 -- 172.21.15.45:0/474504533 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb1dc010910 con 0x7fb1f4074360 2024-03-28T18:02:08.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.662+0000 7fb1f8a00640 1 -- 172.21.15.45:0/474504533 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc018990 con 0x7fb1f4074360 2024-03-28T18:02:08.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.662+0000 7fb1f9e00640 1 -- 172.21.15.45:0/474504533 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f41a3810 con 0x7fb1f4074360 2024-03-28T18:02:08.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.663+0000 7fb1f9e00640 1 -- 172.21.15.45:0/474504533 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f41ab0a0 con 0x7fb1f4074360 2024-03-28T18:02:08.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:08 smithi045 ceph-mon[31773]: pgmap v453: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T18:02:08.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.666+0000 7fb1fcb71740 1 Processor -- start 2024-03-28T18:02:08.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.666+0000 7fb1f8a00640 1 -- 172.21.15.45:0/474504533 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fb1dc010ac0 con 0x7fb1f4074360 2024-03-28T18:02:08.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.666+0000 7fb1fcb71740 1 -- start start 2024-03-28T18:02:08.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.666+0000 7fb1fcb71740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 0x557b7ab8c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:08.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.666+0000 7fb1fcb71740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557b7ab8cf20 0x557b7ab8d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:08.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.666+0000 7fb1fcb71740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab96290 0x557b7ab98680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:08.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.666+0000 7fb1fcb71740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b7aaf1250 con 0x557b7ab8cf20 2024-03-28T18:02:08.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.666+0000 7fb1fcb71740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b7aa99a30 con 0x557b7ab96290 2024-03-28T18:02:08.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.666+0000 7fb1fcb71740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b7aaed2f0 con 0x557b7ab8dc00 2024-03-28T18:02:08.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.667+0000 7fb1f3400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 0x557b7ab8c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:08.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.667+0000 7fb1ebe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab96290 0x557b7ab98680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:08.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.667+0000 7fb1f3400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 0x557b7ab8c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:34400/0 (socket says 172.21.15.45:34400) 2024-03-28T18:02:08.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.667+0000 7fb1f3400640 1 -- 172.21.15.45:0/3334397886 learned_addr learned my addr 172.21.15.45:0/3334397886 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:02:08.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.667+0000 7fb1f2a00640 1 --2- 172.21.15.45:0/3334397886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557b7ab8cf20 0x557b7ab8d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:08.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.667+0000 7fb1f3400640 1 -- 172.21.15.45:0/3334397886 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab96290 msgr2=0x557b7ab98680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:02:08.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.667+0000 7fb1f3400640 1 --2- 172.21.15.45:0/3334397886 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab96290 0x557b7ab98680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.667+0000 7fb1f3400640 1 -- 172.21.15.45:0/3334397886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557b7ab8cf20 msgr2=0x557b7ab8d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:02:08.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.667+0000 7fb1f3400640 1 --2- 172.21.15.45:0/3334397886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557b7ab8cf20 0x557b7ab8d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.667+0000 7fb1f3400640 1 -- 172.21.15.45:0/3334397886 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b7aa84690 con 0x557b7ab8dc00 2024-03-28T18:02:08.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.667+0000 7fb1ebe00640 1 --2- 172.21.15.45:0/3334397886 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab96290 0x557b7ab98680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:02:08.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.668+0000 7fb1f3400640 1 --2- 172.21.15.45:0/3334397886 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 0x557b7ab8c950 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec009f30 tx=0x7fb1ec0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76bb74133e974a2c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:02:08.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.668+0000 7fb1f8a00640 1 --2- 172.21.15.45:0/474504533 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fb1d80a29f0 0x7fb1d80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:08.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.668+0000 7fb1f8a00640 1 -- 172.21.15.45:0/474504533 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 12038+0+0 (secure 0 0 0) 0x7fb1dc0cff20 con 0x7fb1f4074360 2024-03-28T18:02:08.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.668+0000 7fb1e3e00640 1 -- 172.21.15.45:0/3334397886 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec004d20 con 0x557b7ab8dc00 2024-03-28T18:02:08.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.668+0000 7fb1e3e00640 1 -- 172.21.15.45:0/3334397886 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb1ec004050 con 0x557b7ab8dc00 2024-03-28T18:02:08.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.668+0000 7fb1e3e00640 1 -- 172.21.15.45:0/3334397886 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec005280 con 0x557b7ab8dc00 2024-03-28T18:02:08.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.668+0000 7fb1f2a00640 1 --2- 172.21.15.45:0/474504533 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fb1d80a29f0 0x7fb1d80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:08.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.669+0000 7fb1fcb71740 1 -- 172.21.15.45:0/3334397886 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 msgr2=0x557b7ab8c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:08.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.669+0000 7fb1fcb71740 1 --2- 172.21.15.45:0/3334397886 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 0x557b7ab8c950 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec009f30 tx=0x7fb1ec0118d0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.669+0000 7fb1f2a00640 1 --2- 172.21.15.45:0/474504533 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fb1d80a29f0 0x7fb1d80a4eb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4005ab0 tx=0x7fb1e402e040 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:02:08.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.669+0000 7fb1fcb71740 1 -- 172.21.15.45:0/3334397886 shutdown_connections 2024-03-28T18:02:08.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.669+0000 7fb1fcb71740 1 --2- 172.21.15.45:0/3334397886 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab96290 0x557b7ab98680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.669+0000 7fb1fcb71740 1 --2- 172.21.15.45:0/3334397886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557b7ab8cf20 0x557b7ab8d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.669+0000 7fb1fcb71740 1 --2- 172.21.15.45:0/3334397886 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 0x557b7ab8c950 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.669+0000 7fb1fcb71740 1 -- 172.21.15.45:0/3334397886 >> 172.21.15.45:0/3334397886 conn(0x557b7ab54200 msgr2=0x557b7ab87970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:02:08.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.669+0000 7fb1fcb71740 1 -- 172.21.15.45:0/3334397886 shutdown_connections 2024-03-28T18:02:08.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.669+0000 7fb1fcb71740 1 -- 172.21.15.45:0/3334397886 wait complete. 2024-03-28T18:02:08.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.669+0000 7fb1fcb71740 1 -- start start 2024-03-28T18:02:08.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.670+0000 7fb1fcb71740 1 Processor -- start 2024-03-28T18:02:08.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.670+0000 7fb1fcb71740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab8cf20 0x557b7ab958e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:08.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.670+0000 7fb1fcb71740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 0x557b7ab92c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:08.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.670+0000 7fb1fcb71740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557b7ab96290 0x557b7ab931b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:08.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.670+0000 7fb1fcb71740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b7ab99100 con 0x557b7ab96290 2024-03-28T18:02:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.670+0000 7fb1fcb71740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b7aaace80 con 0x557b7ab8cf20 2024-03-28T18:02:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.670+0000 7fb1fcb71740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b7ab99280 con 0x557b7ab8dc00 2024-03-28T18:02:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.670+0000 7fb1fcb71740 10 client.0 _ll_register_callbacks cb 0x557b7aabf130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:02:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.670+0000 7fb1f2a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab8cf20 0x557b7ab958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.670+0000 7fb1f2a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab8cf20 0x557b7ab958e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:56282/0 (socket says 172.21.15.45:56282) 2024-03-28T18:02:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.670+0000 7fb1f2a00640 1 -- 172.21.15.45:0/3577297877 learned_addr learned my addr 172.21.15.45:0/3577297877 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:02:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.670+0000 7fb1f3400640 1 --2- 172.21.15.45:0/3577297877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557b7ab96290 0x557b7ab931b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.671+0000 7fb1ebe00640 1 --2- 172.21.15.45:0/3577297877 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 0x557b7ab92c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.671+0000 7fb1f2a00640 1 -- 172.21.15.45:0/3577297877 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 msgr2=0x557b7ab92c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:02:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.671+0000 7fb1f2a00640 1 --2- 172.21.15.45:0/3577297877 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 0x557b7ab92c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.671+0000 7fb1f2a00640 1 -- 172.21.15.45:0/3577297877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557b7ab96290 msgr2=0x557b7ab931b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:02:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.671+0000 7fb1f2a00640 1 --2- 172.21.15.45:0/3577297877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557b7ab96290 0x557b7ab931b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.671+0000 7fb1f2a00640 1 -- 172.21.15.45:0/3577297877 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557b7a8cf7d0 con 0x557b7ab8cf20 2024-03-28T18:02:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.671+0000 7fb1f2a00640 1 --2- 172.21.15.45:0/3577297877 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab8cf20 0x557b7ab958e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fb1e40096c0 tx=0x7fb1e402bcd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fc3a5b12c58d094 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:02:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.671+0000 7fb1e0c00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T18:02:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.672+0000 7fb1e0c00640 1 -- 172.21.15.45:0/3577297877 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4051020 con 0x557b7ab8cf20 2024-03-28T18:02:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.672+0000 7fb1e0c00640 1 -- 172.21.15.45:0/3577297877 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(141..141 src has 1..141) v4 ==== 12038+0+0 (secure 0 0 0) 0x7fb1e40048d0 con 0x557b7ab8cf20 2024-03-28T18:02:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.672+0000 7fb1fcb71740 1 -- 172.21.15.45:0/3577297877 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb1ec004050 con 0x557b7ab8cf20 2024-03-28T18:02:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.672+0000 7fb1e0c00640 1 -- 172.21.15.45:0/3577297877 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=293) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb1e4004ad0 con 0x557b7ab8cf20 2024-03-28T18:02:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.672+0000 7fb1fcb71740 10 client.26519 fetch_fsmap learned FSMap version 293 2024-03-28T18:02:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.672+0000 7fb1fcb71740 1 -- 172.21.15.45:0/3577297877 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=293,monmap=4+}) v3 -- 0x557b7a8053f0 con 0x557b7ab8cf20 2024-03-28T18:02:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.672+0000 7fb1e0c00640 1 -- 172.21.15.45:0/3577297877 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb1ec004050 con 0x557b7ab8cf20 2024-03-28T18:02:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.672+0000 7fb1e0c00640 1 -- 172.21.15.45:0/3577297877 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 293) v1 ==== 4468+0+0 (secure 0 0 0) 0x7fb1e40037b0 con 0x557b7ab8cf20 2024-03-28T18:02:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.672+0000 7fb1fcb71740 10 client.26519 fetch_fsmap finished waiting for FSMap version 293 2024-03-28T18:02:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.672+0000 7fb1fcb71740 10 client.26519 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:02:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.672+0000 7fb1fcb71740 20 client.26519 populate_metadata read hostname 'smithi045' 2024-03-28T18:02:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.672+0000 7fb1fcb71740 1 --2- 172.21.15.45:0/3577297877 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x557b7aba6590 0x557b7aba89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.672+0000 7fb1fcb71740 4 client.26519 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:02:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.672+0000 7fb1fcb71740 1 -- 172.21.15.45:0/3577297877 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557b7aafe790 con 0x557b7aba6590 2024-03-28T18:02:08.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.673+0000 7fb1ebe00640 1 --2- 172.21.15.45:0/3577297877 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x557b7aba6590 0x557b7aba89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:08.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.673+0000 7fb1ebe00640 1 --2- 172.21.15.45:0/3577297877 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x557b7aba6590 0x557b7aba89d0 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b675fced9e8b6d80 server_cookie=f1bd0af69d4518da in_seq=0 out_seq=0 2024-03-28T18:02:08.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.673+0000 7fb1e0c00640 10 client.26519 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:02:08.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.674+0000 7fb1e0c00640 1 -- 172.21.15.45:0/3577297877 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557b7aafe790 con 0x557b7aba6590 2024-03-28T18:02:08.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.674+0000 7fb1e0c00640 10 client.26519 handle_command_reply: tid=0 2024-03-28T18:02:08.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.674+0000 7fb1fcb71740 1 client.26519 shutdown 2024-03-28T18:02:08.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.674+0000 7fb1fcb71740 1 -- 172.21.15.45:0/3577297877 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab8cf20 msgr2=0x557b7ab958e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:08.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.674+0000 7fb1fcb71740 1 --2- 172.21.15.45:0/3577297877 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab8cf20 0x557b7ab958e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fb1e40096c0 tx=0x7fb1e402bcd0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.674+0000 7fb1fcb71740 1 -- 172.21.15.45:0/3577297877 shutdown_connections 2024-03-28T18:02:08.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.674+0000 7fb1fcb71740 1 --2- 172.21.15.45:0/3577297877 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x557b7aba6590 0x557b7aba89d0 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.674+0000 7fb1fcb71740 1 --2- 172.21.15.45:0/3577297877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557b7ab96290 0x557b7ab931b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.674+0000 7fb1fcb71740 1 --2- 172.21.15.45:0/3577297877 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 0x557b7ab92c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.674+0000 7fb1fcb71740 1 --2- 172.21.15.45:0/3577297877 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab8cf20 0x557b7ab958e0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.674+0000 7fb1e0c00640 0 client.26519 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:02:08.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.674+0000 7fb1fcb71740 1 -- 172.21.15.45:0/3577297877 >> 172.21.15.45:0/3577297877 conn(0x557b7ab54200 msgr2=0x557b7ab896c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:02:08.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.675+0000 7fb1fcb71740 1 -- 172.21.15.45:0/3577297877 shutdown_connections 2024-03-28T18:02:08.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.675+0000 7fb1fcb71740 1 -- 172.21.15.45:0/3577297877 wait complete. 2024-03-28T18:02:08.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.675+0000 7fb1fcb71740 20 client.26519 trim_cache size 0 max 16384 2024-03-28T18:02:08.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.692+0000 7fb1fcb71740 1 Processor -- start 2024-03-28T18:02:08.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.693+0000 7fb1fcb71740 1 -- start start 2024-03-28T18:02:08.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.693+0000 7fb1fcb71740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557b7ab8cf20 0x557b7ab953a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:08.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.693+0000 7fb1fcb71740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 0x557b7ab958e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:08.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.693+0000 7fb1fcb71740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab96290 0x557b7ab97680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:08.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.693+0000 7fb1fcb71740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f41b5f20 con 0x557b7ab8cf20 2024-03-28T18:02:08.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.693+0000 7fb1fcb71740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f41b3f40 con 0x557b7ab96290 2024-03-28T18:02:08.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.693+0000 7fb1fcb71740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f41b58c0 con 0x557b7ab8dc00 2024-03-28T18:02:08.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.693+0000 7fb1f2a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557b7ab8cf20 0x557b7ab953a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:08.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.693+0000 7fb1f2a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557b7ab8cf20 0x557b7ab953a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43608/0 (socket says 172.21.15.45:43608) 2024-03-28T18:02:08.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.693+0000 7fb1f2a00640 1 -- 172.21.15.45:0/726277609 learned_addr learned my addr 172.21.15.45:0/726277609 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:02:08.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.693+0000 7fb1ebe00640 1 --2- 172.21.15.45:0/726277609 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 0x557b7ab958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:08.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.693+0000 7fb1f3400640 1 --2- 172.21.15.45:0/726277609 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab96290 0x557b7ab97680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:08.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.694+0000 7fb1ebe00640 1 -- 172.21.15.45:0/726277609 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab96290 msgr2=0x557b7ab97680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:02:08.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.694+0000 7fb1ebe00640 1 --2- 172.21.15.45:0/726277609 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab96290 0x557b7ab97680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.694+0000 7fb1ebe00640 1 -- 172.21.15.45:0/726277609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557b7ab8cf20 msgr2=0x557b7ab953a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:02:08.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.694+0000 7fb1ebe00640 1 --2- 172.21.15.45:0/726277609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557b7ab8cf20 0x557b7ab953a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.694+0000 7fb1ebe00640 1 -- 172.21.15.45:0/726277609 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b7a7c50f0 con 0x557b7ab8dc00 2024-03-28T18:02:08.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.694+0000 7fb1ebe00640 1 --2- 172.21.15.45:0/726277609 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 0x557b7ab958e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc0cc850 tx=0x7fb1dc086040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c52fd8607b914826 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:02:08.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.694+0000 7fb1e3400640 1 -- 172.21.15.45:0/726277609 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc09b170 con 0x557b7ab8dc00 2024-03-28T18:02:08.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.694+0000 7fb1e3400640 1 -- 172.21.15.45:0/726277609 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb1dc09b810 con 0x557b7ab8dc00 2024-03-28T18:02:08.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.694+0000 7fb1e3400640 1 -- 172.21.15.45:0/726277609 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc0ccad0 con 0x557b7ab8dc00 2024-03-28T18:02:08.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.695+0000 7fb1fcb71740 1 -- 172.21.15.45:0/726277609 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 msgr2=0x557b7ab958e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:08.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.695+0000 7fb1fcb71740 1 --2- 172.21.15.45:0/726277609 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 0x557b7ab958e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc0cc850 tx=0x7fb1dc086040 comp rx=0 tx=0).stop 2024-03-28T18:02:08.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.695+0000 7fb1fcb71740 1 -- 172.21.15.45:0/726277609 shutdown_connections 2024-03-28T18:02:08.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.695+0000 7fb1fcb71740 1 --2- 172.21.15.45:0/726277609 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab96290 0x557b7ab97680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.695+0000 7fb1fcb71740 1 --2- 172.21.15.45:0/726277609 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 0x557b7ab958e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.695+0000 7fb1fcb71740 1 --2- 172.21.15.45:0/726277609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557b7ab8cf20 0x557b7ab953a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.695+0000 7fb1fcb71740 1 -- 172.21.15.45:0/726277609 >> 172.21.15.45:0/726277609 conn(0x557b7ab54200 msgr2=0x557b7ab874a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:02:08.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.695+0000 7fb1fcb71740 1 -- 172.21.15.45:0/726277609 shutdown_connections 2024-03-28T18:02:08.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.695+0000 7fb1fcb71740 1 -- 172.21.15.45:0/726277609 wait complete. 2024-03-28T18:02:08.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.695+0000 7fb1fcb71740 1 -- start start 2024-03-28T18:02:08.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.695+0000 7fb1fcb71740 1 Processor -- start 2024-03-28T18:02:08.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.696+0000 7fb1fcb71740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557b7ab8cf20 0x557b7ab8f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:08.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.696+0000 7fb1fcb71740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 0x557b7ab8fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:08.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.696+0000 7fb1fcb71740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab96290 0x557b7ab90040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:08.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.696+0000 7fb1fcb71740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f41b4c00 con 0x557b7ab8cf20 2024-03-28T18:02:08.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.696+0000 7fb1fcb71740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f41b45a0 con 0x557b7ab96290 2024-03-28T18:02:08.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.696+0000 7fb1fcb71740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f41b5260 con 0x557b7ab8dc00 2024-03-28T18:02:08.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.696+0000 7fb1fcb71740 10 client.0 _ll_register_callbacks cb 0x7fb1e4044070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:02:08.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.696+0000 7fb1ebe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 0x557b7ab8fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:08.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.696+0000 7fb1f3400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab96290 0x557b7ab90040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:08.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.696+0000 7fb1ebe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 0x557b7ab8fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:34418/0 (socket says 172.21.15.45:34418) 2024-03-28T18:02:08.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.696+0000 7fb1ebe00640 1 -- 172.21.15.45:0/515981747 learned_addr learned my addr 172.21.15.45:0/515981747 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:02:08.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.696+0000 7fb1f2a00640 1 --2- 172.21.15.45:0/515981747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557b7ab8cf20 0x557b7ab8f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:08.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.697+0000 7fb1f3400640 1 -- 172.21.15.45:0/515981747 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 msgr2=0x557b7ab8fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:02:08.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.697+0000 7fb1f3400640 1 --2- 172.21.15.45:0/515981747 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 0x557b7ab8fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.697+0000 7fb1f3400640 1 -- 172.21.15.45:0/515981747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557b7ab8cf20 msgr2=0x557b7ab8f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:02:08.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.697+0000 7fb1f3400640 1 --2- 172.21.15.45:0/515981747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557b7ab8cf20 0x557b7ab8f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.697+0000 7fb1f3400640 1 -- 172.21.15.45:0/515981747 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557b7ac010f0 con 0x557b7ab96290 2024-03-28T18:02:08.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.697+0000 7fb1ebe00640 1 --2- 172.21.15.45:0/515981747 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 0x557b7ab8fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:02:08.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.697+0000 7fb1f3400640 1 --2- 172.21.15.45:0/515981747 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab96290 0x557b7ab90040 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec0047f0 tx=0x7fb1ec009d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c0107271efa026 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:02:08.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.697+0000 7fb1e0c00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T18:02:08.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.697+0000 7fb1e0c00640 1 -- 172.21.15.45:0/515981747 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec004250 con 0x557b7ab96290 2024-03-28T18:02:08.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.697+0000 7fb1fcb71740 1 -- 172.21.15.45:0/515981747 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb1dc09b810 con 0x557b7ab96290 2024-03-28T18:02:08.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.697+0000 7fb1e0c00640 1 -- 172.21.15.45:0/515981747 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(141..141 src has 1..141) v4 ==== 12038+0+0 (secure 0 0 0) 0x7fb1ec0043f0 con 0x557b7ab96290 2024-03-28T18:02:08.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.698+0000 7fb1e0c00640 1 -- 172.21.15.45:0/515981747 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=293) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb1ec036050 con 0x557b7ab96290 2024-03-28T18:02:08.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.698+0000 7fb1fcb71740 10 client.26525 fetch_fsmap learned FSMap version 293 2024-03-28T18:02:08.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.698+0000 7fb1fcb71740 1 -- 172.21.15.45:0/515981747 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=293,monmap=4+}) v3 -- 0x557b7ac02400 con 0x557b7ab96290 2024-03-28T18:02:08.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.698+0000 7fb1e0c00640 1 -- 172.21.15.45:0/515981747 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb1dc09b810 con 0x557b7ab96290 2024-03-28T18:02:08.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.698+0000 7fb1e0c00640 1 -- 172.21.15.45:0/515981747 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 293) v1 ==== 4468+0+0 (secure 0 0 0) 0x7fb1ec02e650 con 0x557b7ab96290 2024-03-28T18:02:08.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.698+0000 7fb1fcb71740 10 client.26525 fetch_fsmap finished waiting for FSMap version 293 2024-03-28T18:02:08.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.698+0000 7fb1fcb71740 10 client.26525 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:02:08.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.698+0000 7fb1fcb71740 20 client.26525 populate_metadata read hostname 'smithi045' 2024-03-28T18:02:08.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.698+0000 7fb1fcb71740 1 --2- 172.21.15.45:0/515981747 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x557b7ac02a00 0x557b7ac04e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:08.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.698+0000 7fb1fcb71740 4 client.26525 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:02:08.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.698+0000 7fb1fcb71740 1 -- 172.21.15.45:0/515981747 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557b7aaf8ae0 con 0x557b7ac02a00 2024-03-28T18:02:08.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.698+0000 7fb1f2a00640 1 --2- 172.21.15.45:0/515981747 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x557b7ac02a00 0x557b7ac04e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:08.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.699+0000 7fb1f2a00640 1 --2- 172.21.15.45:0/515981747 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x557b7ac02a00 0x557b7ac04e20 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cb8390ab272944c server_cookie=6fe00abb7c42c7b3 in_seq=0 out_seq=0 2024-03-28T18:02:08.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.699+0000 7fb1e0c00640 10 client.26525 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:02:08.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.699+0000 7fb1e0c00640 1 -- 172.21.15.45:0/515981747 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x557b7aaf8ae0 con 0x557b7ac02a00 2024-03-28T18:02:08.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.699+0000 7fb1e0c00640 10 client.26525 handle_command_reply: tid=0 2024-03-28T18:02:08.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.699+0000 7fb1fcb71740 1 client.26525 shutdown 2024-03-28T18:02:08.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.699+0000 7fb1fcb71740 1 -- 172.21.15.45:0/515981747 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab96290 msgr2=0x557b7ab90040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:08.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.699+0000 7fb1fcb71740 1 --2- 172.21.15.45:0/515981747 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab96290 0x557b7ab90040 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec0047f0 tx=0x7fb1ec009d00 comp rx=0 tx=0).stop 2024-03-28T18:02:08.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.699+0000 7fb1fcb71740 1 -- 172.21.15.45:0/515981747 shutdown_connections 2024-03-28T18:02:08.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.699+0000 7fb1fcb71740 1 --2- 172.21.15.45:0/515981747 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x557b7ac02a00 0x557b7ac04e20 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.699+0000 7fb1fcb71740 1 --2- 172.21.15.45:0/515981747 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b7ab96290 0x557b7ab90040 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.699+0000 7fb1fcb71740 1 --2- 172.21.15.45:0/515981747 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x557b7ab8dc00 0x557b7ab8fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.699+0000 7fb1fcb71740 1 --2- 172.21.15.45:0/515981747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557b7ab8cf20 0x557b7ab8f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.699+0000 7fb1e0c00640 0 client.26525 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:02:08.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.699+0000 7fb1fcb71740 1 -- 172.21.15.45:0/515981747 >> 172.21.15.45:0/515981747 conn(0x557b7ab54200 msgr2=0x557b7ab914c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:02:08.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.699+0000 7fb1fcb71740 1 -- 172.21.15.45:0/515981747 shutdown_connections 2024-03-28T18:02:08.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.699+0000 7fb1fcb71740 1 -- 172.21.15.45:0/515981747 wait complete. 2024-03-28T18:02:08.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.700+0000 7fb1fcb71740 20 client.26525 trim_cache size 0 max 16384 2024-03-28T18:02:08.704 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T18:02:08.704 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:02:08.704 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T18:02:08.704 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T18:02:08.704 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T18:02:08.704 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T18:02:08.705 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T18:02:08.705 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:02:08.705 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:02:08.705 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T18:02:08.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.700+0000 7fb1f9e00640 1 -- 172.21.15.45:0/474504533 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fb1d80a29f0 msgr2=0x7fb1d80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:08.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.700+0000 7fb1f9e00640 1 --2- 172.21.15.45:0/474504533 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fb1d80a29f0 0x7fb1d80a4eb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4005ab0 tx=0x7fb1e402e040 comp rx=0 tx=0).stop 2024-03-28T18:02:08.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.700+0000 7fb1f9e00640 1 -- 172.21.15.45:0/474504533 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1f4074360 msgr2=0x7fb1f419ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:08.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.700+0000 7fb1f9e00640 1 --2- 172.21.15.45:0/474504533 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1f4074360 0x7fb1f419ea70 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc00e9f0 tx=0x7fb1dc00eeb0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.700+0000 7fb1f9e00640 1 -- 172.21.15.45:0/474504533 shutdown_connections 2024-03-28T18:02:08.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.700+0000 7fb1f9e00640 1 --2- 172.21.15.45:0/474504533 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fb1d80a29f0 0x7fb1d80a4eb0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.700+0000 7fb1f9e00640 1 --2- 172.21.15.45:0/474504533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1f4112640 0x7fb1f41a2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.700+0000 7fb1f9e00640 1 --2- 172.21.15.45:0/474504533 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1f4074360 0x7fb1f419ea70 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.700+0000 7fb1f9e00640 1 --2- 172.21.15.45:0/474504533 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1f4073a40 0x7fb1f419e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:08.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.700+0000 7fb1f9e00640 1 -- 172.21.15.45:0/474504533 >> 172.21.15.45:0/474504533 conn(0x7fb1f4078aa0 msgr2=0x7fb1f410e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:02:08.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.700+0000 7fb1f9e00640 1 -- 172.21.15.45:0/474504533 shutdown_connections 2024-03-28T18:02:08.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:08.700+0000 7fb1f9e00640 1 -- 172.21.15.45:0/474504533 wait complete. 2024-03-28T18:02:08.718 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T18:02:08.934 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 581 sec latency 696831.110 ms 2024-03-28T18:02:09.934 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 582 sec latency 697831.245 ms 2024-03-28T18:02:10.312 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:09 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 4. 2024-03-28T18:02:10.312 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:09 smithi045 systemd[1]: Stopped Ceph mds.j for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:02:10.312 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:09 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Consumed 1.646s CPU time. 2024-03-28T18:02:10.312 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:09 smithi045 systemd[1]: Starting Ceph mds.j for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T18:02:10.571 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:10 smithi045 ceph-mon[31773]: pgmap v454: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T18:02:10.571 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:10 smithi045 podman[95467]: 2024-03-28 18:02:10.240045885 +0000 UTC m=+0.036790822 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:02:10.571 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:10 smithi045 podman[95467]: 2024-03-28 18:02:10.373083907 +0000 UTC m=+0.169828884 container create 1261d0c26a34c37e75cda7478c9ac75dabadc1cb43424d30002fb306d2c8c59a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-28T18:02:10.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:10 smithi151 ceph-mon[36481]: pgmap v454: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T18:02:10.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:10 smithi071 ceph-mon[36597]: pgmap v454: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T18:02:10.934 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 583 sec latency 698831.383 ms 2024-03-28T18:02:11.166 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:10 smithi045 podman[95467]: 2024-03-28 18:02:10.676387538 +0000 UTC m=+0.473132556 container init 1261d0c26a34c37e75cda7478c9ac75dabadc1cb43424d30002fb306d2c8c59a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:02:11.166 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:10 smithi045 podman[95467]: 2024-03-28 18:02:10.686667405 +0000 UTC m=+0.483412416 container start 1261d0c26a34c37e75cda7478c9ac75dabadc1cb43424d30002fb306d2c8c59a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-28T18:02:11.166 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:10 smithi045 ceph-mds[95486]: -- 172.21.15.45:0/395505149 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560d67383a40 con 0x560d67357000 2024-03-28T18:02:11.166 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:10 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: starting mds.j at 2024-03-28T18:02:11.166 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:10 smithi045 bash[95467]: 1261d0c26a34c37e75cda7478c9ac75dabadc1cb43424d30002fb306d2c8c59a 2024-03-28T18:02:11.167 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:10 smithi045 systemd[1]: Started Ceph mds.j for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:02:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:11 smithi151 ceph-mon[36481]: daemon mds.j restarted 2024-03-28T18:02:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:11 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:02:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:11 smithi151 ceph-mon[36481]: osdmap e142: 12 total, 12 up, 12 in 2024-03-28T18:02:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:11 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/1180007672,v1:172.21.15.45:6835/1180007672] up:boot 2024-03-28T18:02:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:11 smithi151 ceph-mon[36481]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-03-28T18:02:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:11 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:02:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:11 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:02:11.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:11 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:11 smithi045 ceph-mon[31773]: daemon mds.j restarted 2024-03-28T18:02:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:11 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:02:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:11 smithi045 ceph-mon[31773]: osdmap e142: 12 total, 12 up, 12 in 2024-03-28T18:02:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:11 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/1180007672,v1:172.21.15.45:6835/1180007672] up:boot 2024-03-28T18:02:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:11 smithi045 ceph-mon[31773]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-03-28T18:02:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:11 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:02:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:11 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:02:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:11 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:11 smithi071 ceph-mon[36597]: daemon mds.j restarted 2024-03-28T18:02:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:11 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:02:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:11 smithi071 ceph-mon[36597]: osdmap e142: 12 total, 12 up, 12 in 2024-03-28T18:02:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:11 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/1180007672,v1:172.21.15.45:6835/1180007672] up:boot 2024-03-28T18:02:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:11 smithi071 ceph-mon[36597]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-03-28T18:02:11.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:11 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:02:11.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:11 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:02:11.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:11 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:11.934 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 584 sec latency 699831.551 ms 2024-03-28T18:02:12.934 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 585 sec latency 700831.663 ms 2024-03-28T18:02:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:12 smithi071 ceph-mon[36597]: pgmap v456: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:02:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:12 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2588031165,v1:172.21.15.45:6837/2588031165] up:resolve 2024-03-28T18:02:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:12 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:13.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:12 smithi151 ceph-mon[36481]: pgmap v456: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:02:13.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:12 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2588031165,v1:172.21.15.45:6837/2588031165] up:resolve 2024-03-28T18:02:13.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:12 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:12 smithi045 ceph-mon[31773]: pgmap v456: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:02:13.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:12 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2588031165,v1:172.21.15.45:6837/2588031165] up:resolve 2024-03-28T18:02:13.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:12 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:13.934 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 586 sec latency 701831.827 ms 2024-03-28T18:02:14.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:14 smithi151 ceph-mon[36481]: pgmap v457: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:02:14.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:14 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:02:14.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:14 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2588031165,v1:172.21.15.45:6837/2588031165] up:reconnect 2024-03-28T18:02:14.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:14 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:14.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:14 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999984 2024-03-28T18:02:14.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:14 smithi045 ceph-mon[31773]: pgmap v457: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:02:14.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:14 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:02:14.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:14 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2588031165,v1:172.21.15.45:6837/2588031165] up:reconnect 2024-03-28T18:02:14.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:14 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:14.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:14 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999984 2024-03-28T18:02:14.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:14 smithi071 ceph-mon[36597]: pgmap v457: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:02:14.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:14 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:02:14.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:14 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2588031165,v1:172.21.15.45:6837/2588031165] up:reconnect 2024-03-28T18:02:14.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:14 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:14.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:14 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00999984 2024-03-28T18:02:14.935 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 587 sec latency 702831.984 ms 2024-03-28T18:02:15.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2588031165,v1:172.21.15.45:6837/2588031165] up:rejoin 2024-03-28T18:02:15.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:15.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-mon[31773]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-03-28T18:02:15.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4810a00700 time 2024-03-28T18:02:15.363689+0000 2024-03-28T18:02:15.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:02:15.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:15.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f481c3b84a1] 2024-03-28T18:02:15.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f481c3b8667] 2024-03-28T18:02:15.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558feb795223] 2024-03-28T18:02:15.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558feb60f5d6] 2024-03-28T18:02:15.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558feb60fda1] 2024-03-28T18:02:15.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558feb6122d5] 2024-03-28T18:02:15.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558feb662233] 2024-03-28T18:02:15.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558feb664197] 2024-03-28T18:02:15.667 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558feb56311b] 2024-03-28T18:02:15.668 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558feb5636bc] 2024-03-28T18:02:15.668 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558feb54d1ff] 2024-03-28T18:02:15.668 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f481c64a488] 2024-03-28T18:02:15.668 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 13: (DispatchQueue::entry()+0x51f) [0x7f481c64760f] 2024-03-28T18:02:15.668 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f481c71ae81] 2024-03-28T18:02:15.668 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f481b1351ca] 2024-03-28T18:02:15.668 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 16: clone() 2024-03-28T18:02:15.668 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: *** Caught signal (Aborted) ** 2024-03-28T18:02:15.668 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: in thread 7f4810a00700 thread_name:ms_dispatch 2024-03-28T18:02:15.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 2024-03-28T18:02:15.364+0000 7f4810a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4810a00700 time 2024-03-28T18:02:15.363689+0000 2024-03-28T18:02:15.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:02:15.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 2024-03-28T18:02:15.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:15.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f481c3b84a1] 2024-03-28T18:02:15.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f481c3b8667] 2024-03-28T18:02:15.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558feb795223] 2024-03-28T18:02:15.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558feb60f5d6] 2024-03-28T18:02:15.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558feb60fda1] 2024-03-28T18:02:15.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558feb6122d5] 2024-03-28T18:02:15.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558feb662233] 2024-03-28T18:02:15.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558feb664197] 2024-03-28T18:02:15.669 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558feb56311b] 2024-03-28T18:02:15.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558feb5636bc] 2024-03-28T18:02:15.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558feb54d1ff] 2024-03-28T18:02:15.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f481c64a488] 2024-03-28T18:02:15.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 13: (DispatchQueue::entry()+0x51f) [0x7f481c64760f] 2024-03-28T18:02:15.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f481c71ae81] 2024-03-28T18:02:15.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f481b1351ca] 2024-03-28T18:02:15.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 16: clone() 2024-03-28T18:02:15.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 2024-03-28T18:02:15.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:15.670 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f481b13fd20] 2024-03-28T18:02:15.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 2: gsignal() 2024-03-28T18:02:15.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 3: abort() 2024-03-28T18:02:15.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f481c3b84fb] 2024-03-28T18:02:15.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f481c3b8667] 2024-03-28T18:02:15.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558feb795223] 2024-03-28T18:02:15.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558feb60f5d6] 2024-03-28T18:02:15.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558feb60fda1] 2024-03-28T18:02:15.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558feb6122d5] 2024-03-28T18:02:15.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558feb662233] 2024-03-28T18:02:15.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558feb664197] 2024-03-28T18:02:15.671 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558feb56311b] 2024-03-28T18:02:15.672 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558feb5636bc] 2024-03-28T18:02:15.672 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558feb54d1ff] 2024-03-28T18:02:15.672 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f481c64a488] 2024-03-28T18:02:15.672 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 16: (DispatchQueue::entry()+0x51f) [0x7f481c64760f] 2024-03-28T18:02:15.672 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f481c71ae81] 2024-03-28T18:02:15.672 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f481b1351ca] 2024-03-28T18:02:15.672 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 19: clone() 2024-03-28T18:02:15.672 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 2024-03-28T18:02:15.367+0000 7f4810a00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:02:15.672 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: in thread 7f4810a00700 thread_name:ms_dispatch 2024-03-28T18:02:15.672 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 2024-03-28T18:02:15.672 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:15.672 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f481b13fd20] 2024-03-28T18:02:15.672 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 2: gsignal() 2024-03-28T18:02:15.672 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 3: abort() 2024-03-28T18:02:15.672 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f481c3b84fb] 2024-03-28T18:02:15.672 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f481c3b8667] 2024-03-28T18:02:15.673 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558feb795223] 2024-03-28T18:02:15.673 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558feb60f5d6] 2024-03-28T18:02:15.673 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558feb60fda1] 2024-03-28T18:02:15.673 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558feb6122d5] 2024-03-28T18:02:15.673 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558feb662233] 2024-03-28T18:02:15.673 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558feb664197] 2024-03-28T18:02:15.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558feb56311b] 2024-03-28T18:02:15.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558feb5636bc] 2024-03-28T18:02:15.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558feb54d1ff] 2024-03-28T18:02:15.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f481c64a488] 2024-03-28T18:02:15.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 16: (DispatchQueue::entry()+0x51f) [0x7f481c64760f] 2024-03-28T18:02:15.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f481c71ae81] 2024-03-28T18:02:15.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f481b1351ca] 2024-03-28T18:02:15.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 19: clone() 2024-03-28T18:02:15.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:02:15.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 2024-03-28T18:02:15.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: -1> 2024-03-28T18:02:15.364+0000 7f4810a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4810a00700 time 2024-03-28T18:02:15.363689+0000 2024-03-28T18:02:15.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:02:15.674 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 2024-03-28T18:02:15.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:15.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f481c3b84a1] 2024-03-28T18:02:15.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f481c3b8667] 2024-03-28T18:02:15.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558feb795223] 2024-03-28T18:02:15.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558feb60f5d6] 2024-03-28T18:02:15.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558feb60fda1] 2024-03-28T18:02:15.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558feb6122d5] 2024-03-28T18:02:15.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558feb662233] 2024-03-28T18:02:15.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558feb664197] 2024-03-28T18:02:15.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558feb56311b] 2024-03-28T18:02:15.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558feb5636bc] 2024-03-28T18:02:15.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558feb54d1ff] 2024-03-28T18:02:15.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f481c64a488] 2024-03-28T18:02:15.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 13: (DispatchQueue::entry()+0x51f) [0x7f481c64760f] 2024-03-28T18:02:15.675 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f481c71ae81] 2024-03-28T18:02:15.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f481b1351ca] 2024-03-28T18:02:15.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 16: clone() 2024-03-28T18:02:15.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 2024-03-28T18:02:15.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 0> 2024-03-28T18:02:15.367+0000 7f4810a00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:02:15.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: in thread 7f4810a00700 thread_name:ms_dispatch 2024-03-28T18:02:15.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 2024-03-28T18:02:15.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:15.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f481b13fd20] 2024-03-28T18:02:15.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 2: gsignal() 2024-03-28T18:02:15.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 3: abort() 2024-03-28T18:02:15.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f481c3b84fb] 2024-03-28T18:02:15.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f481c3b8667] 2024-03-28T18:02:15.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558feb795223] 2024-03-28T18:02:15.676 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558feb60f5d6] 2024-03-28T18:02:15.677 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558feb60fda1] 2024-03-28T18:02:15.677 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558feb6122d5] 2024-03-28T18:02:15.677 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558feb662233] 2024-03-28T18:02:15.677 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558feb664197] 2024-03-28T18:02:15.677 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558feb56311b] 2024-03-28T18:02:15.677 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558feb5636bc] 2024-03-28T18:02:15.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558feb54d1ff] 2024-03-28T18:02:15.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f481c64a488] 2024-03-28T18:02:15.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 16: (DispatchQueue::entry()+0x51f) [0x7f481c64760f] 2024-03-28T18:02:15.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f481c71ae81] 2024-03-28T18:02:15.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f481b1351ca] 2024-03-28T18:02:15.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 19: clone() 2024-03-28T18:02:15.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:02:15.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 2024-03-28T18:02:15.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: -1> 2024-03-28T18:02:15.364+0000 7f4810a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4810a00700 time 2024-03-28T18:02:15.363689+0000 2024-03-28T18:02:15.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:02:15.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 2024-03-28T18:02:15.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:15.678 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f481c3b84a1] 2024-03-28T18:02:15.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f481c3b8667] 2024-03-28T18:02:15.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558feb795223] 2024-03-28T18:02:15.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558feb60f5d6] 2024-03-28T18:02:15.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558feb60fda1] 2024-03-28T18:02:15.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558feb6122d5] 2024-03-28T18:02:15.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558feb662233] 2024-03-28T18:02:15.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558feb664197] 2024-03-28T18:02:15.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558feb56311b] 2024-03-28T18:02:15.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558feb5636bc] 2024-03-28T18:02:15.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558feb54d1ff] 2024-03-28T18:02:15.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f481c64a488] 2024-03-28T18:02:15.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 13: (DispatchQueue::entry()+0x51f) [0x7f481c64760f] 2024-03-28T18:02:15.679 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f481c71ae81] 2024-03-28T18:02:15.680 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f481b1351ca] 2024-03-28T18:02:15.680 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 16: clone() 2024-03-28T18:02:15.680 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 2024-03-28T18:02:15.680 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 0> 2024-03-28T18:02:15.367+0000 7f4810a00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:02:15.680 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: in thread 7f4810a00700 thread_name:ms_dispatch 2024-03-28T18:02:15.680 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 2024-03-28T18:02:15.680 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:15.680 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f481b13fd20] 2024-03-28T18:02:15.680 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 2: gsignal() 2024-03-28T18:02:15.680 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 3: abort() 2024-03-28T18:02:15.680 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f481c3b84fb] 2024-03-28T18:02:15.680 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f481c3b8667] 2024-03-28T18:02:15.680 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558feb795223] 2024-03-28T18:02:15.680 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558feb60f5d6] 2024-03-28T18:02:15.680 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558feb60fda1] 2024-03-28T18:02:15.681 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558feb6122d5] 2024-03-28T18:02:15.681 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558feb662233] 2024-03-28T18:02:15.681 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558feb664197] 2024-03-28T18:02:15.681 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558feb56311b] 2024-03-28T18:02:15.681 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558feb5636bc] 2024-03-28T18:02:15.681 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558feb54d1ff] 2024-03-28T18:02:15.681 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f481c64a488] 2024-03-28T18:02:15.681 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 16: (DispatchQueue::entry()+0x51f) [0x7f481c64760f] 2024-03-28T18:02:15.681 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f481c71ae81] 2024-03-28T18:02:15.681 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f481b1351ca] 2024-03-28T18:02:15.681 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 19: clone() 2024-03-28T18:02:15.681 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:02:15.681 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95273]: 2024-03-28T18:02:15.681 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:15 smithi045 podman[95520]: 2024-03-28 18:02:15.473997315 +0000 UTC m=+0.036702021 container died 8c91cc183814082fd26233cc430ceb6a2696ec690c1ee9edcbc04080a05d87fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-28T18:02:15.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:15 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2588031165,v1:172.21.15.45:6837/2588031165] up:rejoin 2024-03-28T18:02:15.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:15 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:15.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:15 smithi071 ceph-mon[36597]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-03-28T18:02:15.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:15 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2588031165,v1:172.21.15.45:6837/2588031165] up:rejoin 2024-03-28T18:02:15.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:15 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:15.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:15 smithi151 ceph-mon[36481]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-03-28T18:02:15.935 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 588 sec latency 703832.100 ms 2024-03-28T18:02:16.665 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:16 smithi045 podman[95520]: 2024-03-28 18:02:16.346390245 +0000 UTC m=+0.909094989 container remove 8c91cc183814082fd26233cc430ceb6a2696ec690c1ee9edcbc04080a05d87fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-28T18:02:16.665 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:16 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:02:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:16 smithi045 ceph-mon[31773]: pgmap v458: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:02:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:16 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:02:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:16 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/2588031165,v1:172.21.15.45:6837/2588031165] up:active 2024-03-28T18:02:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:16 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:16 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 3 up:standby-replay 2024-03-28T18:02:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:16 smithi071 ceph-mon[36597]: pgmap v458: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:02:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:16 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:02:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:16 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/2588031165,v1:172.21.15.45:6837/2588031165] up:active 2024-03-28T18:02:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:16 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:16 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 3 up:standby-replay 2024-03-28T18:02:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:16 smithi151 ceph-mon[36481]: pgmap v458: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:02:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:16 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:02:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:16 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/2588031165,v1:172.21.15.45:6837/2588031165] up:active 2024-03-28T18:02:16.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:16 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:16.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:16 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 3 up:standby-replay 2024-03-28T18:02:16.935 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 589 sec latency 704832.247 ms 2024-03-28T18:02:17.165 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:16 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-28T18:02:17.165 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:16 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Consumed 1.637s CPU time. 2024-03-28T18:02:17.935 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 590 sec latency 705832.377 ms 2024-03-28T18:02:18.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:18 smithi071 ceph-mon[36597]: pgmap v459: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:02:18.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:18 smithi151 ceph-mon[36481]: pgmap v459: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:02:18.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:18 smithi045 ceph-mon[31773]: pgmap v459: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:02:18.935 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 591 sec latency 706832.480 ms 2024-03-28T18:02:19.935 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 592 sec latency 707832.604 ms 2024-03-28T18:02:20.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:20 smithi071 ceph-mon[36597]: pgmap v460: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:02:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:20 smithi151 ceph-mon[36481]: pgmap v460: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:02:20.911 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:20 smithi045 ceph-mon[31773]: pgmap v460: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T18:02:20.935 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 593 sec latency 708832.811 ms 2024-03-28T18:02:21.936 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 594 sec latency 709832.912 ms 2024-03-28T18:02:22.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:22 smithi071 ceph-mon[36597]: pgmap v461: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 26 op/s 2024-03-28T18:02:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:22 smithi151 ceph-mon[36481]: pgmap v461: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 26 op/s 2024-03-28T18:02:22.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:22 smithi045 ceph-mon[31773]: pgmap v461: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 26 op/s 2024-03-28T18:02:22.936 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 595 sec latency 710833.067 ms 2024-03-28T18:02:23.936 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 596 sec latency 711833.227 ms 2024-03-28T18:02:24.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:24 smithi071 ceph-mon[36597]: pgmap v462: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T18:02:24.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:24 smithi151 ceph-mon[36481]: pgmap v462: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T18:02:24.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:24 smithi045 ceph-mon[31773]: pgmap v462: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T18:02:24.936 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 597 sec latency 712833.374 ms 2024-03-28T18:02:25.936 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 598 sec latency 713833.537 ms 2024-03-28T18:02:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:26 smithi151 ceph-mon[36481]: pgmap v463: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-28T18:02:26.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:26 smithi045 ceph-mon[31773]: pgmap v463: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-28T18:02:26.936 INFO:tasks.workunit.client.0.smithi045.stdout: 1 7632 0.00 MB/sec execute 599 sec latency 714833.683 ms 2024-03-28T18:02:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:26 smithi071 ceph-mon[36597]: pgmap v463: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-28T18:02:27.313 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:26 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 5. 2024-03-28T18:02:27.313 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:26 smithi045 systemd[1]: Stopped Ceph mds.g for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:02:27.313 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:26 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Consumed 1.637s CPU time. 2024-03-28T18:02:27.313 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:26 smithi045 systemd[1]: Starting Ceph mds.g for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T18:02:27.666 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:27 smithi045 podman[95625]: 2024-03-28 18:02:27.22084787 +0000 UTC m=+0.041394764 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:02:27.666 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:27 smithi045 podman[95625]: 2024-03-28 18:02:27.50954522 +0000 UTC m=+0.330092102 container create a3c2121e8f4ac803bdf37a7ee71ea6792f109f7608d3c736f054e852998c4c7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5) 2024-03-28T18:02:27.937 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 600 sec 2024-03-28T18:02:28.166 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:27 smithi045 podman[95625]: 2024-03-28 18:02:27.809388136 +0000 UTC m=+0.629935025 container init a3c2121e8f4ac803bdf37a7ee71ea6792f109f7608d3c736f054e852998c4c7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-28T18:02:28.166 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:27 smithi045 podman[95625]: 2024-03-28 18:02:27.822269834 +0000 UTC m=+0.642816686 container start a3c2121e8f4ac803bdf37a7ee71ea6792f109f7608d3c736f054e852998c4c7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-28T18:02:28.166 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:27 smithi045 ceph-mds[95644]: -- 172.21.15.45:0/3585558099 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560ea7625a40 con 0x560ea75f9000 2024-03-28T18:02:28.166 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:27 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: starting mds.g at 2024-03-28T18:02:28.166 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:27 smithi045 bash[95625]: a3c2121e8f4ac803bdf37a7ee71ea6792f109f7608d3c736f054e852998c4c7e 2024-03-28T18:02:28.166 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:27 smithi045 systemd[1]: Started Ceph mds.g for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:02:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:28 smithi151 ceph-mon[36481]: pgmap v464: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T18:02:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:28 smithi151 ceph-mon[36481]: daemon mds.g restarted 2024-03-28T18:02:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:28 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:02:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:28 smithi151 ceph-mon[36481]: osdmap e143: 12 total, 12 up, 12 in 2024-03-28T18:02:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:28 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/761645002,v1:172.21.15.45:6837/761645002] up:boot 2024-03-28T18:02:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:28 smithi151 ceph-mon[36481]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-03-28T18:02:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:28 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:02:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:28 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:02:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:28 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:28 smithi045 ceph-mon[31773]: pgmap v464: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T18:02:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:28 smithi045 ceph-mon[31773]: daemon mds.g restarted 2024-03-28T18:02:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:28 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:02:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:28 smithi045 ceph-mon[31773]: osdmap e143: 12 total, 12 up, 12 in 2024-03-28T18:02:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:28 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/761645002,v1:172.21.15.45:6837/761645002] up:boot 2024-03-28T18:02:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:28 smithi045 ceph-mon[31773]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-03-28T18:02:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:28 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:02:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:28 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:02:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:28 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:28.937 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 601 sec 2024-03-28T18:02:29.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:28 smithi071 ceph-mon[36597]: pgmap v464: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T18:02:29.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:28 smithi071 ceph-mon[36597]: daemon mds.g restarted 2024-03-28T18:02:29.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:28 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:02:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:28 smithi071 ceph-mon[36597]: osdmap e143: 12 total, 12 up, 12 in 2024-03-28T18:02:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:28 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/761645002,v1:172.21.15.45:6837/761645002] up:boot 2024-03-28T18:02:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:28 smithi071 ceph-mon[36597]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-03-28T18:02:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:28 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:02:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:28 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:02:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:28 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:29.937 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 602 sec 2024-03-28T18:02:30.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:29 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/1180007672,v1:172.21.15.45:6835/1180007672] up:resolve 2024-03-28T18:02:30.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:29 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:30.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:29 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/1180007672,v1:172.21.15.45:6835/1180007672] up:resolve 2024-03-28T18:02:30.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:29 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:30.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:29 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/1180007672,v1:172.21.15.45:6835/1180007672] up:resolve 2024-03-28T18:02:30.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:29 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:30.937 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 603 sec 2024-03-28T18:02:31.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:30 smithi045 ceph-mon[31773]: pgmap v466: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T18:02:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:30 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:02:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:30 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/1180007672,v1:172.21.15.45:6835/1180007672] up:reconnect 2024-03-28T18:02:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:30 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:30 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T18:02:31.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:30 smithi071 ceph-mon[36597]: pgmap v466: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T18:02:31.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:30 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:02:31.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:30 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/1180007672,v1:172.21.15.45:6835/1180007672] up:reconnect 2024-03-28T18:02:31.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:30 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:31.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:30 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T18:02:31.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:30 smithi151 ceph-mon[36481]: pgmap v466: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-28T18:02:31.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:30 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:02:31.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:30 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/1180007672,v1:172.21.15.45:6835/1180007672] up:reconnect 2024-03-28T18:02:31.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:30 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:31.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:30 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T18:02:31.918 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2bd2a00700 time 2024-03-28T18:02:31.576557+0000 2024-03-28T18:02:31.918 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:02:31.918 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:31.919 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2bde3d14a1] 2024-03-28T18:02:31.919 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2bde3d1667] 2024-03-28T18:02:31.919 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d65ab2223] 2024-03-28T18:02:31.919 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d6592c5d6] 2024-03-28T18:02:31.919 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d6592cda1] 2024-03-28T18:02:31.919 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d6592f2d5] 2024-03-28T18:02:31.919 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d6597f233] 2024-03-28T18:02:31.919 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d65981197] 2024-03-28T18:02:31.920 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d6588011b] 2024-03-28T18:02:31.920 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d658806bc] 2024-03-28T18:02:31.920 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d6586a1ff] 2024-03-28T18:02:31.920 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2bde663488] 2024-03-28T18:02:31.920 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 13: (DispatchQueue::entry()+0x51f) [0x7f2bde66060f] 2024-03-28T18:02:31.920 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2bde733e81] 2024-03-28T18:02:31.920 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2bdd14e1ca] 2024-03-28T18:02:31.920 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 16: clone() 2024-03-28T18:02:31.921 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 2024-03-28T18:02:31.578+0000 7f2bd2a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2bd2a00700 time 2024-03-28T18:02:31.576557+0000 2024-03-28T18:02:31.921 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:02:31.921 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 2024-03-28T18:02:31.921 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:31.921 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2bde3d14a1] 2024-03-28T18:02:31.921 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2bde3d1667] 2024-03-28T18:02:31.921 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d65ab2223] 2024-03-28T18:02:31.922 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d6592c5d6] 2024-03-28T18:02:31.922 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d6592cda1] 2024-03-28T18:02:31.922 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d6592f2d5] 2024-03-28T18:02:31.922 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d6597f233] 2024-03-28T18:02:31.922 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d65981197] 2024-03-28T18:02:31.922 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d6588011b] 2024-03-28T18:02:31.922 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d658806bc] 2024-03-28T18:02:31.922 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d6586a1ff] 2024-03-28T18:02:31.922 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2bde663488] 2024-03-28T18:02:31.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 13: (DispatchQueue::entry()+0x51f) [0x7f2bde66060f] 2024-03-28T18:02:31.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2bde733e81] 2024-03-28T18:02:31.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2bdd14e1ca] 2024-03-28T18:02:31.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 16: clone() 2024-03-28T18:02:31.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 2024-03-28T18:02:31.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: *** Caught signal (Aborted) ** 2024-03-28T18:02:31.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: in thread 7f2bd2a00700 thread_name:ms_dispatch 2024-03-28T18:02:31.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:31.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2bdd158d20] 2024-03-28T18:02:31.924 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 2: gsignal() 2024-03-28T18:02:31.924 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 3: abort() 2024-03-28T18:02:31.924 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2bde3d14fb] 2024-03-28T18:02:31.924 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2bde3d1667] 2024-03-28T18:02:31.924 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d65ab2223] 2024-03-28T18:02:31.924 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d6592c5d6] 2024-03-28T18:02:31.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d6592cda1] 2024-03-28T18:02:31.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d6592f2d5] 2024-03-28T18:02:31.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d6597f233] 2024-03-28T18:02:31.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d65981197] 2024-03-28T18:02:31.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d6588011b] 2024-03-28T18:02:31.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d658806bc] 2024-03-28T18:02:31.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d6586a1ff] 2024-03-28T18:02:31.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2bde663488] 2024-03-28T18:02:31.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 16: (DispatchQueue::entry()+0x51f) [0x7f2bde66060f] 2024-03-28T18:02:31.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2bde733e81] 2024-03-28T18:02:31.926 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2bdd14e1ca] 2024-03-28T18:02:31.926 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 19: clone() 2024-03-28T18:02:31.926 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 2024-03-28T18:02:31.582+0000 7f2bd2a00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:02:31.926 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: in thread 7f2bd2a00700 thread_name:ms_dispatch 2024-03-28T18:02:31.926 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 2024-03-28T18:02:31.926 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:31.926 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2bdd158d20] 2024-03-28T18:02:31.926 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 2: gsignal() 2024-03-28T18:02:31.926 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 3: abort() 2024-03-28T18:02:31.927 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2bde3d14fb] 2024-03-28T18:02:31.927 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2bde3d1667] 2024-03-28T18:02:31.927 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d65ab2223] 2024-03-28T18:02:31.927 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d6592c5d6] 2024-03-28T18:02:31.927 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d6592cda1] 2024-03-28T18:02:31.927 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d6592f2d5] 2024-03-28T18:02:31.928 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d6597f233] 2024-03-28T18:02:31.928 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d65981197] 2024-03-28T18:02:31.928 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d6588011b] 2024-03-28T18:02:31.928 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d658806bc] 2024-03-28T18:02:31.928 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d6586a1ff] 2024-03-28T18:02:31.928 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2bde663488] 2024-03-28T18:02:31.928 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 16: (DispatchQueue::entry()+0x51f) [0x7f2bde66060f] 2024-03-28T18:02:31.928 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2bde733e81] 2024-03-28T18:02:31.928 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2bdd14e1ca] 2024-03-28T18:02:31.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 19: clone() 2024-03-28T18:02:31.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:02:31.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 2024-03-28T18:02:31.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: -1> 2024-03-28T18:02:31.578+0000 7f2bd2a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2bd2a00700 time 2024-03-28T18:02:31.576557+0000 2024-03-28T18:02:31.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:02:31.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 2024-03-28T18:02:31.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:31.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2bde3d14a1] 2024-03-28T18:02:31.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2bde3d1667] 2024-03-28T18:02:31.930 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d65ab2223] 2024-03-28T18:02:31.930 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d6592c5d6] 2024-03-28T18:02:31.930 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d6592cda1] 2024-03-28T18:02:31.930 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d6592f2d5] 2024-03-28T18:02:31.930 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d6597f233] 2024-03-28T18:02:31.931 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d65981197] 2024-03-28T18:02:31.931 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d6588011b] 2024-03-28T18:02:31.931 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d658806bc] 2024-03-28T18:02:31.931 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d6586a1ff] 2024-03-28T18:02:31.931 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2bde663488] 2024-03-28T18:02:31.931 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 13: (DispatchQueue::entry()+0x51f) [0x7f2bde66060f] 2024-03-28T18:02:31.931 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2bde733e81] 2024-03-28T18:02:31.931 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2bdd14e1ca] 2024-03-28T18:02:31.931 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 16: clone() 2024-03-28T18:02:31.931 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 2024-03-28T18:02:31.932 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 0> 2024-03-28T18:02:31.582+0000 7f2bd2a00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:02:31.932 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: in thread 7f2bd2a00700 thread_name:ms_dispatch 2024-03-28T18:02:31.932 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 2024-03-28T18:02:31.932 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:31.932 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2bdd158d20] 2024-03-28T18:02:31.932 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 2: gsignal() 2024-03-28T18:02:31.932 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 3: abort() 2024-03-28T18:02:31.932 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2bde3d14fb] 2024-03-28T18:02:31.932 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2bde3d1667] 2024-03-28T18:02:31.932 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d65ab2223] 2024-03-28T18:02:31.932 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d6592c5d6] 2024-03-28T18:02:31.933 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d6592cda1] 2024-03-28T18:02:31.933 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d6592f2d5] 2024-03-28T18:02:31.933 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d6597f233] 2024-03-28T18:02:31.933 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d65981197] 2024-03-28T18:02:31.933 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d6588011b] 2024-03-28T18:02:31.933 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d658806bc] 2024-03-28T18:02:31.934 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d6586a1ff] 2024-03-28T18:02:31.934 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2bde663488] 2024-03-28T18:02:31.934 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 16: (DispatchQueue::entry()+0x51f) [0x7f2bde66060f] 2024-03-28T18:02:31.934 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2bde733e81] 2024-03-28T18:02:31.934 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2bdd14e1ca] 2024-03-28T18:02:31.934 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 19: clone() 2024-03-28T18:02:31.934 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:02:31.934 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 2024-03-28T18:02:31.934 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: -1> 2024-03-28T18:02:31.578+0000 7f2bd2a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2bd2a00700 time 2024-03-28T18:02:31.576557+0000 2024-03-28T18:02:31.935 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:02:31.935 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 2024-03-28T18:02:31.935 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:31.935 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2bde3d14a1] 2024-03-28T18:02:31.935 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2bde3d1667] 2024-03-28T18:02:31.935 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d65ab2223] 2024-03-28T18:02:31.935 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d6592c5d6] 2024-03-28T18:02:31.936 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d6592cda1] 2024-03-28T18:02:31.936 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d6592f2d5] 2024-03-28T18:02:31.936 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d6597f233] 2024-03-28T18:02:31.936 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d65981197] 2024-03-28T18:02:31.936 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d6588011b] 2024-03-28T18:02:31.936 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d658806bc] 2024-03-28T18:02:31.936 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d6586a1ff] 2024-03-28T18:02:31.936 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2bde663488] 2024-03-28T18:02:31.936 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 13: (DispatchQueue::entry()+0x51f) [0x7f2bde66060f] 2024-03-28T18:02:31.937 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2bde733e81] 2024-03-28T18:02:31.937 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2bdd14e1ca] 2024-03-28T18:02:31.937 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 16: clone() 2024-03-28T18:02:31.937 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 2024-03-28T18:02:31.937 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 0> 2024-03-28T18:02:31.582+0000 7f2bd2a00700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:02:31.937 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: in thread 7f2bd2a00700 thread_name:ms_dispatch 2024-03-28T18:02:31.937 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 2024-03-28T18:02:31.937 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:31.937 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2bdd158d20] 2024-03-28T18:02:31.937 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 2: gsignal() 2024-03-28T18:02:31.937 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 3: abort() 2024-03-28T18:02:31.938 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2bde3d14fb] 2024-03-28T18:02:31.938 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2bde3d1667] 2024-03-28T18:02:31.938 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d65ab2223] 2024-03-28T18:02:31.938 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d6592c5d6] 2024-03-28T18:02:31.938 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d6592cda1] 2024-03-28T18:02:31.938 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d6592f2d5] 2024-03-28T18:02:31.939 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d6597f233] 2024-03-28T18:02:31.939 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d65981197] 2024-03-28T18:02:31.939 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d6588011b] 2024-03-28T18:02:31.939 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d658806bc] 2024-03-28T18:02:31.939 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d6586a1ff] 2024-03-28T18:02:31.939 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2bde663488] 2024-03-28T18:02:31.939 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 16: (DispatchQueue::entry()+0x51f) [0x7f2bde66060f] 2024-03-28T18:02:31.939 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2bde733e81] 2024-03-28T18:02:31.940 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2bdd14e1ca] 2024-03-28T18:02:31.940 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 19: clone() 2024-03-28T18:02:31.940 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:02:31.940 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95482]: 2024-03-28T18:02:31.940 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:31 smithi045 podman[95716]: 2024-03-28 18:02:31.696604269 +0000 UTC m=+0.036002186 container died 1261d0c26a34c37e75cda7478c9ac75dabadc1cb43424d30002fb306d2c8c59a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-03-28T18:02:31.940 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 604 sec 2024-03-28T18:02:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:31 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/1180007672,v1:172.21.15.45:6835/1180007672] up:rejoin 2024-03-28T18:02:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:31 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:31 smithi071 ceph-mon[36597]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-03-28T18:02:32.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:31 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:02:32.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:31 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:02:32.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:31 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/1180007672,v1:172.21.15.45:6835/1180007672] up:active 2024-03-28T18:02:32.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:31 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 3 up:standby-replay 2024-03-28T18:02:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:31 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/1180007672,v1:172.21.15.45:6835/1180007672] up:rejoin 2024-03-28T18:02:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:31 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:31 smithi151 ceph-mon[36481]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-03-28T18:02:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:31 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:02:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:31 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:02:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:31 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/1180007672,v1:172.21.15.45:6835/1180007672] up:active 2024-03-28T18:02:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:31 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 3 up:standby-replay 2024-03-28T18:02:32.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/1180007672,v1:172.21.15.45:6835/1180007672] up:rejoin 2024-03-28T18:02:32.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:32.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-mon[31773]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-03-28T18:02:32.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:02:32.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:02:32.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/1180007672,v1:172.21.15.45:6835/1180007672] up:active 2024-03-28T18:02:32.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:31 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 3 up:standby-replay 2024-03-28T18:02:32.937 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 605 sec 2024-03-28T18:02:33.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:32 smithi045 ceph-mon[31773]: pgmap v467: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:02:33.118 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:32 smithi045 podman[95716]: 2024-03-28 18:02:32.711578958 +0000 UTC m=+1.050976903 container remove 1261d0c26a34c37e75cda7478c9ac75dabadc1cb43424d30002fb306d2c8c59a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:02:33.118 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:32 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:02:33.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:32 smithi071 ceph-mon[36597]: pgmap v467: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:02:33.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:32 smithi151 ceph-mon[36481]: pgmap v467: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:02:33.415 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:33 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-28T18:02:33.415 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:33 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Consumed 1.487s CPU time. 2024-03-28T18:02:33.937 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 606 sec 2024-03-28T18:02:34.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:33 smithi071 ceph-mon[36597]: pgmap v468: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:02:34.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:33 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:34.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:33 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:02:34.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:33 smithi071 ceph-mon[36597]: pgmap v469: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:02:34.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:33 smithi071 ceph-mon[36597]: pgmap v470: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-28T18:02:34.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:33 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:02:34.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:33 smithi151 ceph-mon[36481]: pgmap v468: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:02:34.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:33 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:34.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:33 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:02:34.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:33 smithi151 ceph-mon[36481]: pgmap v469: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:02:34.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:33 smithi151 ceph-mon[36481]: pgmap v470: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-28T18:02:34.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:33 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:02:34.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:33 smithi045 ceph-mon[31773]: pgmap v468: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T18:02:34.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:33 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:02:34.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:33 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:02:34.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:33 smithi045 ceph-mon[31773]: pgmap v469: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:02:34.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:33 smithi045 ceph-mon[31773]: pgmap v470: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-28T18:02:34.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:33 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:02:34.937 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 607 sec 2024-03-28T18:02:35.937 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 608 sec 2024-03-28T18:02:36.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:36 smithi045 ceph-mon[31773]: pgmap v471: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 36 KiB/s wr, 18 op/s 2024-03-28T18:02:36.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:36 smithi071 ceph-mon[36597]: pgmap v471: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 36 KiB/s wr, 18 op/s 2024-03-28T18:02:36.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:36 smithi151 ceph-mon[36481]: pgmap v471: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 36 KiB/s wr, 18 op/s 2024-03-28T18:02:36.938 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 609 sec 2024-03-28T18:02:37.938 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 610 sec 2024-03-28T18:02:38.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:38 smithi045 ceph-mon[31773]: pgmap v472: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 33 KiB/s wr, 17 op/s 2024-03-28T18:02:38.719 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:02:38.719 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:02:38.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:38 smithi071 ceph-mon[36597]: pgmap v472: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 33 KiB/s wr, 17 op/s 2024-03-28T18:02:38.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:38 smithi151 ceph-mon[36481]: pgmap v472: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 33 KiB/s wr, 17 op/s 2024-03-28T18:02:38.938 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 611 sec 2024-03-28T18:02:38.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.955+0000 7fd396e00640 1 -- 172.21.15.45:0/4219749407 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd390103840 msgr2=0x7fd390103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:38.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.955+0000 7fd396e00640 1 --2- 172.21.15.45:0/4219749407 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd390103840 0x7fd390103cc0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fd378009900 tx=0x7fd37802bc30 comp rx=0 tx=0).stop 2024-03-28T18:02:38.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.955+0000 7fd396e00640 1 -- 172.21.15.45:0/4219749407 shutdown_connections 2024-03-28T18:02:38.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.955+0000 7fd396e00640 1 --2- 172.21.15.45:0/4219749407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd390104200 0x7fd390110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.955+0000 7fd396e00640 1 --2- 172.21.15.45:0/4219749407 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd390103840 0x7fd390103cc0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.955+0000 7fd396e00640 1 --2- 172.21.15.45:0/4219749407 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd390109850 0x7fd390109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.955+0000 7fd396e00640 1 -- 172.21.15.45:0/4219749407 >> 172.21.15.45:0/4219749407 conn(0x7fd3900ff2f0 msgr2=0x7fd390101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:02:38.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.955+0000 7fd396e00640 1 -- 172.21.15.45:0/4219749407 shutdown_connections 2024-03-28T18:02:38.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.955+0000 7fd396e00640 1 -- 172.21.15.45:0/4219749407 wait complete. 2024-03-28T18:02:38.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.956+0000 7fd396e00640 1 Processor -- start 2024-03-28T18:02:38.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.956+0000 7fd396e00640 1 -- start start 2024-03-28T18:02:38.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.957+0000 7fd396e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd390103840 0x7fd39019a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:38.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.957+0000 7fd396e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd390104200 0x7fd39019a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:38.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.957+0000 7fd396e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd390109850 0x7fd39019ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:38.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.957+0000 7fd396e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd390113c60 con 0x7fd390103840 2024-03-28T18:02:38.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.957+0000 7fd396e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd390113ae0 con 0x7fd390104200 2024-03-28T18:02:38.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.957+0000 7fd396e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd390113de0 con 0x7fd390109850 2024-03-28T18:02:38.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.957+0000 7fd38fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd390103840 0x7fd39019a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:38.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.957+0000 7fd38fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd390103840 0x7fd39019a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52444/0 (socket says 172.21.15.45:52444) 2024-03-28T18:02:38.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.957+0000 7fd38f400640 1 --2- 172.21.15.45:0/2558236149 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd390104200 0x7fd39019a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:38.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.957+0000 7fd38fe00640 1 -- 172.21.15.45:0/2558236149 learned_addr learned my addr 172.21.15.45:0/2558236149 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:02:38.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.957+0000 7fd394e00640 1 --2- 172.21.15.45:0/2558236149 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd390109850 0x7fd39019ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:38.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.958+0000 7fd38f400640 1 -- 172.21.15.45:0/2558236149 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd390109850 msgr2=0x7fd39019ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:38.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.958+0000 7fd38f400640 1 --2- 172.21.15.45:0/2558236149 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd390109850 0x7fd39019ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.958+0000 7fd38f400640 1 -- 172.21.15.45:0/2558236149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd390103840 msgr2=0x7fd39019a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:38.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.958+0000 7fd38f400640 1 --2- 172.21.15.45:0/2558236149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd390103840 0x7fd39019a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.958+0000 7fd38f400640 1 -- 172.21.15.45:0/2558236149 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39019f130 con 0x7fd390104200 2024-03-28T18:02:38.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.958+0000 7fd38f400640 1 --2- 172.21.15.45:0/2558236149 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd390104200 0x7fd39019a680 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fd378000c00 tx=0x7fd3780042e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:02:38.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.958+0000 7fd38cc00640 1 -- 172.21.15.45:0/2558236149 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd378041070 con 0x7fd390104200 2024-03-28T18:02:38.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.958+0000 7fd38cc00640 1 -- 172.21.15.45:0/2558236149 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd37802ccc0 con 0x7fd390104200 2024-03-28T18:02:38.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.958+0000 7fd396e00640 1 -- 172.21.15.45:0/2558236149 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd39019f3c0 con 0x7fd390104200 2024-03-28T18:02:38.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.958+0000 7fd38cc00640 1 -- 172.21.15.45:0/2558236149 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd378003db0 con 0x7fd390104200 2024-03-28T18:02:38.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.959+0000 7fd396e00640 1 -- 172.21.15.45:0/2558236149 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3901a6cb0 con 0x7fd390104200 2024-03-28T18:02:38.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.963+0000 7fd399bae740 1 Processor -- start 2024-03-28T18:02:38.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.963+0000 7fd399bae740 1 -- start start 2024-03-28T18:02:38.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.964+0000 7fd38cc00640 1 -- 172.21.15.45:0/2558236149 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fd378044070 con 0x7fd390104200 2024-03-28T18:02:38.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.964+0000 7fd399bae740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ce06e58c00 0x55ce06e57950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:38.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.964+0000 7fd399bae740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e57f20 0x55ce06e583a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:38.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.964+0000 7fd399bae740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e61290 0x55ce06e63680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:38.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.964+0000 7fd399bae740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce06dbc250 con 0x55ce06e57f20 2024-03-28T18:02:38.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.964+0000 7fd399bae740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce06d64a30 con 0x55ce06e61290 2024-03-28T18:02:38.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.964+0000 7fd399bae740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce06db82f0 con 0x55ce06e58c00 2024-03-28T18:02:38.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.964+0000 7fd38fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e57f20 0x55ce06e583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:38.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.964+0000 7fd38f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e61290 0x55ce06e63680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:38.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.964+0000 7fd394e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ce06e58c00 0x55ce06e57950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:38.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.964+0000 7fd38fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e57f20 0x55ce06e583a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52450/0 (socket says 172.21.15.45:52450) 2024-03-28T18:02:38.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.964+0000 7fd394e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ce06e58c00 0x55ce06e57950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:32820/0 (socket says 172.21.15.45:32820) 2024-03-28T18:02:38.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.964+0000 7fd38f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e61290 0x55ce06e63680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:38752/0 (socket says 172.21.15.45:38752) 2024-03-28T18:02:38.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.964+0000 7fd38fe00640 1 -- 172.21.15.45:0/680229349 learned_addr learned my addr 172.21.15.45:0/680229349 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:02:38.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.964+0000 7fd38fe00640 1 -- 172.21.15.45:0/680229349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ce06e58c00 msgr2=0x55ce06e57950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:02:38.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.964+0000 7fd38fe00640 1 --2- 172.21.15.45:0/680229349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ce06e58c00 0x55ce06e57950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.964+0000 7fd38fe00640 1 -- 172.21.15.45:0/680229349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e61290 msgr2=0x55ce06e63680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:02:38.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.964+0000 7fd38fe00640 1 --2- 172.21.15.45:0/680229349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e61290 0x55ce06e63680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.965+0000 7fd38fe00640 1 -- 172.21.15.45:0/680229349 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce06d4f690 con 0x55ce06e57f20 2024-03-28T18:02:38.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.965+0000 7fd38f400640 1 --2- 172.21.15.45:0/680229349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e61290 0x55ce06e63680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:02:38.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.965+0000 7fd38fe00640 1 --2- 172.21.15.45:0/680229349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e57f20 0x55ce06e583a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd3800114f0 tx=0x7fd3800119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=427afbcca667b967 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:02:38.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.965+0000 7fd385600640 1 -- 172.21.15.45:0/680229349 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd380004b20 con 0x55ce06e57f20 2024-03-28T18:02:38.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.965+0000 7fd38cc00640 1 --2- 172.21.15.45:0/2558236149 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd3740a2970 0x7fd3740a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:38.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.965+0000 7fd385600640 1 -- 172.21.15.45:0/680229349 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd380004cc0 con 0x55ce06e57f20 2024-03-28T18:02:38.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.965+0000 7fd385600640 1 -- 172.21.15.45:0/680229349 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd380005410 con 0x55ce06e57f20 2024-03-28T18:02:38.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.965+0000 7fd38cc00640 1 -- 172.21.15.45:0/2558236149 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 12210+0+0 (secure 0 0 0) 0x7fd3780f18c0 con 0x7fd390104200 2024-03-28T18:02:38.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.966+0000 7fd394e00640 1 --2- 172.21.15.45:0/2558236149 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd3740a2970 0x7fd3740a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:38.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.966+0000 7fd399bae740 1 -- 172.21.15.45:0/680229349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e57f20 msgr2=0x55ce06e583a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:38.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.966+0000 7fd399bae740 1 --2- 172.21.15.45:0/680229349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e57f20 0x55ce06e583a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd3800114f0 tx=0x7fd3800119b0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.966+0000 7fd399bae740 1 -- 172.21.15.45:0/680229349 shutdown_connections 2024-03-28T18:02:38.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.966+0000 7fd399bae740 1 --2- 172.21.15.45:0/680229349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e61290 0x55ce06e63680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.966+0000 7fd399bae740 1 --2- 172.21.15.45:0/680229349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e57f20 0x55ce06e583a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.966+0000 7fd399bae740 1 --2- 172.21.15.45:0/680229349 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ce06e58c00 0x55ce06e57950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.966+0000 7fd399bae740 1 -- 172.21.15.45:0/680229349 >> 172.21.15.45:0/680229349 conn(0x55ce06e1f200 msgr2=0x55ce06e52970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:02:38.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.966+0000 7fd399bae740 1 -- 172.21.15.45:0/680229349 shutdown_connections 2024-03-28T18:02:38.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.966+0000 7fd394e00640 1 --2- 172.21.15.45:0/2558236149 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd3740a2970 0x7fd3740a4e30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd388009da0 tx=0x7fd388005f90 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:02:38.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.966+0000 7fd399bae740 1 -- 172.21.15.45:0/680229349 wait complete. 2024-03-28T18:02:38.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.967+0000 7fd399bae740 1 -- start start 2024-03-28T18:02:38.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.967+0000 7fd399bae740 1 Processor -- start 2024-03-28T18:02:38.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.967+0000 7fd399bae740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e57f20 0x55ce06e60800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:38.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.967+0000 7fd399bae740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ce06e58c00 0x55ce06e5db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:38.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.968+0000 7fd399bae740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e61290 0x55ce06e5e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:38.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.968+0000 7fd399bae740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce06e64100 con 0x55ce06e61290 2024-03-28T18:02:38.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.968+0000 7fd399bae740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce06d77e80 con 0x55ce06e57f20 2024-03-28T18:02:38.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.968+0000 7fd399bae740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce06e64280 con 0x55ce06e58c00 2024-03-28T18:02:38.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.968+0000 7fd399bae740 10 client.0 _ll_register_callbacks cb 0x55ce06d8a130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:02:38.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.968+0000 7fd38fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e57f20 0x55ce06e60800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:38.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.968+0000 7fd38fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e57f20 0x55ce06e60800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:38754/0 (socket says 172.21.15.45:38754) 2024-03-28T18:02:38.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.968+0000 7fd38fe00640 1 -- 172.21.15.45:0/1898441637 learned_addr learned my addr 172.21.15.45:0/1898441637 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:02:38.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.968+0000 7fd394e00640 1 --2- 172.21.15.45:0/1898441637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e61290 0x55ce06e5e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:38.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.968+0000 7fd38f400640 1 --2- 172.21.15.45:0/1898441637 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ce06e58c00 0x55ce06e5db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:38.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.968+0000 7fd38fe00640 1 -- 172.21.15.45:0/1898441637 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ce06e58c00 msgr2=0x55ce06e5db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:02:38.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.968+0000 7fd38fe00640 1 --2- 172.21.15.45:0/1898441637 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ce06e58c00 0x55ce06e5db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.968+0000 7fd38fe00640 1 -- 172.21.15.45:0/1898441637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e61290 msgr2=0x55ce06e5e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:02:38.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.968+0000 7fd38fe00640 1 --2- 172.21.15.45:0/1898441637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e61290 0x55ce06e5e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.968+0000 7fd38fe00640 1 -- 172.21.15.45:0/1898441637 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ce06b9a7d0 con 0x55ce06e57f20 2024-03-28T18:02:38.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.968+0000 7fd394e00640 1 --2- 172.21.15.45:0/1898441637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e61290 0x55ce06e5e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:02:38.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.969+0000 7fd38fe00640 1 --2- 172.21.15.45:0/1898441637 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e57f20 0x55ce06e60800 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd38001b040 tx=0x7fd3800092a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5e808c8d1942ce2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:02:38.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.969+0000 7fd37e000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T18:02:38.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.969+0000 7fd37e000640 1 -- 172.21.15.45:0/1898441637 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd380028070 con 0x55ce06e57f20 2024-03-28T18:02:38.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.969+0000 7fd399bae740 1 -- 172.21.15.45:0/1898441637 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd380004cc0 con 0x55ce06e57f20 2024-03-28T18:02:38.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.969+0000 7fd37e000640 1 -- 172.21.15.45:0/1898441637 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(143..143 src has 1..143) v4 ==== 12210+0+0 (secure 0 0 0) 0x7fd380005170 con 0x55ce06e57f20 2024-03-28T18:02:38.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.969+0000 7fd37e000640 1 -- 172.21.15.45:0/1898441637 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=306) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd380035050 con 0x55ce06e57f20 2024-03-28T18:02:38.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.969+0000 7fd399bae740 10 client.26549 fetch_fsmap learned FSMap version 306 2024-03-28T18:02:38.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.969+0000 7fd399bae740 1 -- 172.21.15.45:0/1898441637 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=306,monmap=4+}) v3 -- 0x55ce06ad03f0 con 0x55ce06e57f20 2024-03-28T18:02:38.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.969+0000 7fd37e000640 1 -- 172.21.15.45:0/1898441637 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd380004cc0 con 0x55ce06e57f20 2024-03-28T18:02:38.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.970+0000 7fd37e000640 1 -- 172.21.15.45:0/1898441637 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 306) v1 ==== 4468+0+0 (secure 0 0 0) 0x7fd38002e640 con 0x55ce06e57f20 2024-03-28T18:02:38.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.970+0000 7fd399bae740 10 client.26549 fetch_fsmap finished waiting for FSMap version 306 2024-03-28T18:02:38.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.970+0000 7fd399bae740 10 client.26549 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:02:38.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.970+0000 7fd399bae740 20 client.26549 populate_metadata read hostname 'smithi045' 2024-03-28T18:02:38.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.970+0000 7fd399bae740 1 --2- 172.21.15.45:0/1898441637 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ce06e71340 0x55ce06e73770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:38.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.970+0000 7fd399bae740 4 client.26549 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:02:38.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.970+0000 7fd399bae740 1 -- 172.21.15.45:0/1898441637 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ce06dc9790 con 0x55ce06e71340 2024-03-28T18:02:38.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.970+0000 7fd38f400640 1 --2- 172.21.15.45:0/1898441637 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ce06e71340 0x55ce06e73770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:38.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.970+0000 7fd38f400640 1 --2- 172.21.15.45:0/1898441637 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ce06e71340 0x55ce06e73770 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=186281e9f4f4a64e server_cookie=28a2b4cc818d9598 in_seq=0 out_seq=0 2024-03-28T18:02:38.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.970+0000 7fd37e000640 10 client.26549 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:02:38.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.971+0000 7fd37e000640 1 -- 172.21.15.45:0/1898441637 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ce06dc9790 con 0x55ce06e71340 2024-03-28T18:02:38.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.971+0000 7fd37e000640 10 client.26549 handle_command_reply: tid=0 2024-03-28T18:02:38.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.971+0000 7fd399bae740 1 client.26549 shutdown 2024-03-28T18:02:38.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.971+0000 7fd399bae740 1 -- 172.21.15.45:0/1898441637 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e57f20 msgr2=0x55ce06e60800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:38.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.971+0000 7fd399bae740 1 --2- 172.21.15.45:0/1898441637 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e57f20 0x55ce06e60800 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd38001b040 tx=0x7fd3800092a0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.971+0000 7fd399bae740 1 -- 172.21.15.45:0/1898441637 shutdown_connections 2024-03-28T18:02:38.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.971+0000 7fd399bae740 1 --2- 172.21.15.45:0/1898441637 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ce06e71340 0x55ce06e73770 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.971+0000 7fd399bae740 1 --2- 172.21.15.45:0/1898441637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e61290 0x55ce06e5e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.972+0000 7fd399bae740 1 --2- 172.21.15.45:0/1898441637 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ce06e58c00 0x55ce06e5db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.972+0000 7fd399bae740 1 --2- 172.21.15.45:0/1898441637 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e57f20 0x55ce06e60800 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.972+0000 7fd399bae740 1 -- 172.21.15.45:0/1898441637 >> 172.21.15.45:0/1898441637 conn(0x55ce06e1f200 msgr2=0x55ce06e546c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:02:38.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.972+0000 7fd37e000640 0 client.26549 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:02:38.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.972+0000 7fd399bae740 1 -- 172.21.15.45:0/1898441637 shutdown_connections 2024-03-28T18:02:38.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.972+0000 7fd399bae740 1 -- 172.21.15.45:0/1898441637 wait complete. 2024-03-28T18:02:38.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.972+0000 7fd399bae740 20 client.26549 trim_cache size 0 max 16384 2024-03-28T18:02:38.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.990+0000 7fd399bae740 1 Processor -- start 2024-03-28T18:02:38.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.990+0000 7fd399bae740 1 -- start start 2024-03-28T18:02:38.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.990+0000 7fd399bae740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ce06e1f200 0x55ce06e57560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:38.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.990+0000 7fd399bae740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e57f20 0x55ce06e62680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:38.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.990+0000 7fd399bae740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e58c00 0x55ce06e5c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:38.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.990+0000 7fd399bae740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3901b1d60 con 0x55ce06e57f20 2024-03-28T18:02:38.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.990+0000 7fd399bae740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3901afd80 con 0x55ce06e58c00 2024-03-28T18:02:38.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.990+0000 7fd399bae740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3901b1700 con 0x55ce06e1f200 2024-03-28T18:02:38.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.990+0000 7fd38fe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ce06e1f200 0x55ce06e57560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:38.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.990+0000 7fd38fe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ce06e1f200 0x55ce06e57560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:32842/0 (socket says 172.21.15.45:32842) 2024-03-28T18:02:38.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.991+0000 7fd38fe00640 1 -- 172.21.15.45:0/732759263 learned_addr learned my addr 172.21.15.45:0/732759263 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:02:38.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.991+0000 7fd394e00640 1 --2- 172.21.15.45:0/732759263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e58c00 0x55ce06e5c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:38.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.991+0000 7fd38f400640 1 --2- 172.21.15.45:0/732759263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e57f20 0x55ce06e62680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:38.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.991+0000 7fd38fe00640 1 -- 172.21.15.45:0/732759263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e58c00 msgr2=0x55ce06e5c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:02:38.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.991+0000 7fd38fe00640 1 --2- 172.21.15.45:0/732759263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e58c00 0x55ce06e5c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.991+0000 7fd38fe00640 1 -- 172.21.15.45:0/732759263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e57f20 msgr2=0x55ce06e62680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:02:38.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.991+0000 7fd38fe00640 1 --2- 172.21.15.45:0/732759263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e57f20 0x55ce06e62680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.991+0000 7fd38fe00640 1 -- 172.21.15.45:0/732759263 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce06a900f0 con 0x55ce06e1f200 2024-03-28T18:02:38.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.991+0000 7fd38f400640 1 --2- 172.21.15.45:0/732759263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e57f20 0x55ce06e62680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:02:38.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.991+0000 7fd38fe00640 1 --2- 172.21.15.45:0/732759263 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ce06e1f200 0x55ce06e57560 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd380030040 tx=0x7fd3800119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d67f1df70d20b64 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:02:38.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.992+0000 7fd384c00640 1 -- 172.21.15.45:0/732759263 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd380028070 con 0x55ce06e1f200 2024-03-28T18:02:38.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.992+0000 7fd384c00640 1 -- 172.21.15.45:0/732759263 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd380016070 con 0x55ce06e1f200 2024-03-28T18:02:38.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.992+0000 7fd384c00640 1 -- 172.21.15.45:0/732759263 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38002e7b0 con 0x55ce06e1f200 2024-03-28T18:02:38.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.992+0000 7fd399bae740 1 -- 172.21.15.45:0/732759263 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ce06e1f200 msgr2=0x55ce06e57560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:38.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.992+0000 7fd399bae740 1 --2- 172.21.15.45:0/732759263 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ce06e1f200 0x55ce06e57560 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd380030040 tx=0x7fd3800119b0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.992+0000 7fd399bae740 1 -- 172.21.15.45:0/732759263 shutdown_connections 2024-03-28T18:02:38.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.992+0000 7fd399bae740 1 --2- 172.21.15.45:0/732759263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e58c00 0x55ce06e5c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.992+0000 7fd399bae740 1 --2- 172.21.15.45:0/732759263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e57f20 0x55ce06e62680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.992+0000 7fd399bae740 1 --2- 172.21.15.45:0/732759263 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ce06e1f200 0x55ce06e57560 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:38.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.992+0000 7fd399bae740 1 -- 172.21.15.45:0/732759263 >> 172.21.15.45:0/732759263 conn(0x55ce06e5e230 msgr2=0x55ce06e524a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:02:38.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.993+0000 7fd399bae740 1 -- 172.21.15.45:0/732759263 shutdown_connections 2024-03-28T18:02:38.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.993+0000 7fd399bae740 1 -- 172.21.15.45:0/732759263 wait complete. 2024-03-28T18:02:38.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.993+0000 7fd399bae740 1 -- start start 2024-03-28T18:02:38.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.993+0000 7fd399bae740 1 Processor -- start 2024-03-28T18:02:38.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.994+0000 7fd399bae740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ce06e1f200 0x55ce06e694a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:38.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.994+0000 7fd399bae740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e57f20 0x55ce06e648b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:39.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.994+0000 7fd399bae740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e58c00 0x55ce06e64ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:39.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.994+0000 7fd399bae740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3901b0a40 con 0x55ce06e58c00 2024-03-28T18:02:39.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.994+0000 7fd399bae740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3901b03e0 con 0x55ce06e57f20 2024-03-28T18:02:39.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.994+0000 7fd399bae740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3901b10a0 con 0x55ce06e1f200 2024-03-28T18:02:39.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.994+0000 7fd399bae740 10 client.0 _ll_register_callbacks cb 0x7fd380009ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:02:39.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.994+0000 7fd38f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e57f20 0x55ce06e648b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:39.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.994+0000 7fd394e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e58c00 0x55ce06e64ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:39.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.994+0000 7fd38f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e57f20 0x55ce06e648b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:38776/0 (socket says 172.21.15.45:38776) 2024-03-28T18:02:39.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.994+0000 7fd38f400640 1 -- 172.21.15.45:0/451105132 learned_addr learned my addr 172.21.15.45:0/451105132 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:02:39.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.994+0000 7fd38fe00640 1 --2- 172.21.15.45:0/451105132 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ce06e1f200 0x55ce06e694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:39.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.994+0000 7fd38f400640 1 -- 172.21.15.45:0/451105132 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ce06e1f200 msgr2=0x55ce06e694a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:02:39.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.994+0000 7fd38f400640 1 --2- 172.21.15.45:0/451105132 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ce06e1f200 0x55ce06e694a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:39.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.994+0000 7fd38f400640 1 -- 172.21.15.45:0/451105132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e58c00 msgr2=0x55ce06e64ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:02:39.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.994+0000 7fd38f400640 1 --2- 172.21.15.45:0/451105132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e58c00 0x55ce06e64ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:39.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.994+0000 7fd38f400640 1 -- 172.21.15.45:0/451105132 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ce06ecbc00 con 0x55ce06e57f20 2024-03-28T18:02:39.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.995+0000 7fd38f400640 1 --2- 172.21.15.45:0/451105132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e57f20 0x55ce06e648b0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd3780ee7a0 tx=0x7fd37802c1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=266d0ae9e81b905b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:02:39.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.995+0000 7fd37e000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T18:02:39.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.995+0000 7fd37e000640 1 -- 172.21.15.45:0/451105132 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd37804b040 con 0x55ce06e57f20 2024-03-28T18:02:39.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.995+0000 7fd37e000640 1 -- 172.21.15.45:0/451105132 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(143..143 src has 1..143) v4 ==== 12210+0+0 (secure 0 0 0) 0x7fd3780bb570 con 0x55ce06e57f20 2024-03-28T18:02:39.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.995+0000 7fd399bae740 1 -- 172.21.15.45:0/451105132 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd380016070 con 0x55ce06e57f20 2024-03-28T18:02:39.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.995+0000 7fd37e000640 1 -- 172.21.15.45:0/451105132 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=306) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd3780bd040 con 0x55ce06e57f20 2024-03-28T18:02:39.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.995+0000 7fd399bae740 10 client.26555 fetch_fsmap learned FSMap version 306 2024-03-28T18:02:39.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.995+0000 7fd399bae740 1 -- 172.21.15.45:0/451105132 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=306,monmap=4+}) v3 -- 0x55ce06ecce30 con 0x55ce06e57f20 2024-03-28T18:02:39.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.995+0000 7fd37e000640 1 -- 172.21.15.45:0/451105132 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd380016070 con 0x55ce06e57f20 2024-03-28T18:02:39.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.995+0000 7fd37e000640 1 -- 172.21.15.45:0/451105132 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 306) v1 ==== 4468+0+0 (secure 0 0 0) 0x7fd3780ee420 con 0x55ce06e57f20 2024-03-28T18:02:39.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.995+0000 7fd399bae740 10 client.26555 fetch_fsmap finished waiting for FSMap version 306 2024-03-28T18:02:39.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.995+0000 7fd399bae740 10 client.26555 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:02:39.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.995+0000 7fd399bae740 20 client.26555 populate_metadata read hostname 'smithi045' 2024-03-28T18:02:39.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.995+0000 7fd399bae740 1 --2- 172.21.15.45:0/451105132 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ce06e61290 0x55ce06ecf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:02:39.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.995+0000 7fd399bae740 4 client.26555 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:02:39.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.995+0000 7fd399bae740 1 -- 172.21.15.45:0/451105132 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ce06dc3ae0 con 0x55ce06e61290 2024-03-28T18:02:39.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.996+0000 7fd38fe00640 1 --2- 172.21.15.45:0/451105132 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ce06e61290 0x55ce06ecf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:02:39.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.996+0000 7fd38fe00640 1 --2- 172.21.15.45:0/451105132 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ce06e61290 0x55ce06ecf5a0 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e32b215ebca02d server_cookie=b1002b444033520d in_seq=0 out_seq=0 2024-03-28T18:02:39.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.996+0000 7fd37e000640 10 client.26555 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:02:39.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.996+0000 7fd37e000640 1 -- 172.21.15.45:0/451105132 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55ce06dc3ae0 con 0x55ce06e61290 2024-03-28T18:02:39.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.996+0000 7fd37e000640 10 client.26555 handle_command_reply: tid=0 2024-03-28T18:02:39.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.996+0000 7fd399bae740 1 client.26555 shutdown 2024-03-28T18:02:39.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.997+0000 7fd399bae740 1 -- 172.21.15.45:0/451105132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e57f20 msgr2=0x55ce06e648b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:39.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.997+0000 7fd399bae740 1 --2- 172.21.15.45:0/451105132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e57f20 0x55ce06e648b0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd3780ee7a0 tx=0x7fd37802c1f0 comp rx=0 tx=0).stop 2024-03-28T18:02:39.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.997+0000 7fd399bae740 1 -- 172.21.15.45:0/451105132 shutdown_connections 2024-03-28T18:02:39.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.997+0000 7fd399bae740 1 --2- 172.21.15.45:0/451105132 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55ce06e61290 0x55ce06ecf5a0 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:39.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.997+0000 7fd399bae740 1 --2- 172.21.15.45:0/451105132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce06e58c00 0x55ce06e64ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:39.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.997+0000 7fd399bae740 1 --2- 172.21.15.45:0/451105132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ce06e57f20 0x55ce06e648b0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:39.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.997+0000 7fd399bae740 1 --2- 172.21.15.45:0/451105132 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55ce06e1f200 0x55ce06e694a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:39.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.997+0000 7fd37e000640 0 client.26555 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:02:39.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.997+0000 7fd399bae740 1 -- 172.21.15.45:0/451105132 >> 172.21.15.45:0/451105132 conn(0x55ce06e5e230 msgr2=0x55ce06e53d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:02:39.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.997+0000 7fd399bae740 1 -- 172.21.15.45:0/451105132 shutdown_connections 2024-03-28T18:02:39.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.997+0000 7fd399bae740 1 -- 172.21.15.45:0/451105132 wait complete. 2024-03-28T18:02:39.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.997+0000 7fd399bae740 20 client.26555 trim_cache size 0 max 16384 2024-03-28T18:02:39.004 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T18:02:39.004 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:02:39.004 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T18:02:39.005 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T18:02:39.005 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T18:02:39.005 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T18:02:39.005 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T18:02:39.005 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:02:39.005 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:02:39.005 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T18:02:39.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.998+0000 7fd396e00640 1 -- 172.21.15.45:0/2558236149 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd3740a2970 msgr2=0x7fd3740a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:39.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.998+0000 7fd396e00640 1 --2- 172.21.15.45:0/2558236149 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd3740a2970 0x7fd3740a4e30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd388009da0 tx=0x7fd388005f90 comp rx=0 tx=0).stop 2024-03-28T18:02:39.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.998+0000 7fd396e00640 1 -- 172.21.15.45:0/2558236149 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd390104200 msgr2=0x7fd39019a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:02:39.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.998+0000 7fd396e00640 1 --2- 172.21.15.45:0/2558236149 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd390104200 0x7fd39019a680 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fd378000c00 tx=0x7fd3780042e0 comp rx=0 tx=0).stop 2024-03-28T18:02:39.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.998+0000 7fd396e00640 1 -- 172.21.15.45:0/2558236149 shutdown_connections 2024-03-28T18:02:39.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.998+0000 7fd396e00640 1 --2- 172.21.15.45:0/2558236149 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fd3740a2970 0x7fd3740a4e30 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:39.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.998+0000 7fd396e00640 1 --2- 172.21.15.45:0/2558236149 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd390109850 0x7fd39019ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:39.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.998+0000 7fd396e00640 1 --2- 172.21.15.45:0/2558236149 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd390104200 0x7fd39019a680 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:39.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.998+0000 7fd396e00640 1 --2- 172.21.15.45:0/2558236149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd390103840 0x7fd39019a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:02:39.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.998+0000 7fd396e00640 1 -- 172.21.15.45:0/2558236149 >> 172.21.15.45:0/2558236149 conn(0x7fd3900ff2f0 msgr2=0x7fd3900ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:02:39.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.998+0000 7fd396e00640 1 -- 172.21.15.45:0/2558236149 shutdown_connections 2024-03-28T18:02:39.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:02:38.998+0000 7fd396e00640 1 -- 172.21.15.45:0/2558236149 wait complete. 2024-03-28T18:02:39.015 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T18:02:39.938 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 612 sec 2024-03-28T18:02:40.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:40 smithi045 ceph-mon[31773]: pgmap v473: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 33 KiB/s wr, 12 op/s 2024-03-28T18:02:40.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:40 smithi071 ceph-mon[36597]: pgmap v473: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 33 KiB/s wr, 12 op/s 2024-03-28T18:02:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:40 smithi151 ceph-mon[36481]: pgmap v473: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 33 KiB/s wr, 12 op/s 2024-03-28T18:02:40.938 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 613 sec 2024-03-28T18:02:41.938 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 614 sec 2024-03-28T18:02:42.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:42 smithi045 ceph-mon[31773]: pgmap v474: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 33 KiB/s wr, 16 op/s 2024-03-28T18:02:42.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:42 smithi045 ceph-mon[31773]: osdmap e144: 12 total, 12 up, 12 in 2024-03-28T18:02:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:42 smithi071 ceph-mon[36597]: pgmap v474: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 33 KiB/s wr, 16 op/s 2024-03-28T18:02:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:42 smithi071 ceph-mon[36597]: osdmap e144: 12 total, 12 up, 12 in 2024-03-28T18:02:42.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:42 smithi151 ceph-mon[36481]: pgmap v474: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 33 KiB/s wr, 16 op/s 2024-03-28T18:02:42.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:42 smithi151 ceph-mon[36481]: osdmap e144: 12 total, 12 up, 12 in 2024-03-28T18:02:42.938 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 615 sec 2024-03-28T18:02:43.561 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:43 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 5. 2024-03-28T18:02:43.561 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:43 smithi045 systemd[1]: Stopped Ceph mds.j for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:02:43.562 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:43 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Consumed 1.487s CPU time. 2024-03-28T18:02:43.562 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:43 smithi045 systemd[1]: Starting Ceph mds.j for 9f4888ec-ed29-11ee-95d0-87774f69a715... 2024-03-28T18:02:43.884 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:43 smithi045 podman[95967]: 2024-03-28 18:02:43.466431826 +0000 UTC m=+0.037443796 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:02:43.884 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:43 smithi045 podman[95967]: 2024-03-28 18:02:43.587957768 +0000 UTC m=+0.158969769 container create 5196cb77ce09818605e5e0bec9dfdd30f19da0a879a5ba3df5466ba6c5859735 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-28T18:02:43.884 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:43 smithi045 podman[95967]: 2024-03-28 18:02:43.883540216 +0000 UTC m=+0.454552237 container init 5196cb77ce09818605e5e0bec9dfdd30f19da0a879a5ba3df5466ba6c5859735 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:02:43.938 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 616 sec 2024-03-28T18:02:44.166 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:43 smithi045 podman[95967]: 2024-03-28 18:02:43.89458055 +0000 UTC m=+0.465592522 container start 5196cb77ce09818605e5e0bec9dfdd30f19da0a879a5ba3df5466ba6c5859735 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-03-28T18:02:44.166 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:43 smithi045 ceph-mds[95985]: -- 172.21.15.45:0/1381259492 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5571025dba40 con 0x5571025af000 2024-03-28T18:02:44.166 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:43 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: starting mds.j at 2024-03-28T18:02:44.166 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:43 smithi045 bash[95967]: 5196cb77ce09818605e5e0bec9dfdd30f19da0a879a5ba3df5466ba6c5859735 2024-03-28T18:02:44.166 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:02:43 smithi045 systemd[1]: Started Ceph mds.j for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:02:44.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:44 smithi045 ceph-mon[31773]: pgmap v476: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T18:02:44.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:44 smithi045 ceph-mon[31773]: daemon mds.j restarted 2024-03-28T18:02:44.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:44 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:02:44.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:44 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:02:44.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:44 smithi045 ceph-mon[31773]: osdmap e145: 12 total, 12 up, 12 in 2024-03-28T18:02:44.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:44 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/2048647835,v1:172.21.15.45:6835/2048647835] up:boot 2024-03-28T18:02:44.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:44 smithi045 ceph-mon[31773]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-03-28T18:02:44.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:44 smithi045 ceph-mon[31773]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:02:44.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:44 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:44.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:44 smithi071 ceph-mon[36597]: pgmap v476: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T18:02:44.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:44 smithi071 ceph-mon[36597]: daemon mds.j restarted 2024-03-28T18:02:44.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:44 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:02:44.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:44 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:02:44.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:44 smithi071 ceph-mon[36597]: osdmap e145: 12 total, 12 up, 12 in 2024-03-28T18:02:44.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:44 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/2048647835,v1:172.21.15.45:6835/2048647835] up:boot 2024-03-28T18:02:44.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:44 smithi071 ceph-mon[36597]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-03-28T18:02:44.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:44 smithi071 ceph-mon[36597]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:02:44.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:44 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:44 smithi151 ceph-mon[36481]: pgmap v476: 97 pgs: 97 active+clean; 27 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-03-28T18:02:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:44 smithi151 ceph-mon[36481]: daemon mds.j restarted 2024-03-28T18:02:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:44 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:02:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:44 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:02:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:44 smithi151 ceph-mon[36481]: osdmap e145: 12 total, 12 up, 12 in 2024-03-28T18:02:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:44 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/2048647835,v1:172.21.15.45:6835/2048647835] up:boot 2024-03-28T18:02:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:44 smithi151 ceph-mon[36481]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-03-28T18:02:44.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:44 smithi151 ceph-mon[36481]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-28T18:02:44.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:44 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:replay} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:44.938 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 617 sec 2024-03-28T18:02:45.938 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 618 sec 2024-03-28T18:02:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:46 smithi071 ceph-mon[36597]: pgmap v478: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T18:02:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:46 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/761645002,v1:172.21.15.45:6837/761645002] up:resolve 2024-03-28T18:02:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:46 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:46 smithi151 ceph-mon[36481]: pgmap v478: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T18:02:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:46 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/761645002,v1:172.21.15.45:6837/761645002] up:resolve 2024-03-28T18:02:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:46 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:46 smithi045 ceph-mon[31773]: pgmap v478: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T18:02:46.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:46 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/761645002,v1:172.21.15.45:6837/761645002] up:resolve 2024-03-28T18:02:46.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:46 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:resolve} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:46.938 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 619 sec 2024-03-28T18:02:47.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:47 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:02:47.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:47 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/761645002,v1:172.21.15.45:6837/761645002] up:reconnect 2024-03-28T18:02:47.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:47 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:47.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:47 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T18:02:47.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:47 smithi071 ceph-mon[36597]: osdmap e146: 12 total, 12 up, 12 in 2024-03-28T18:02:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:47 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:02:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:47 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/761645002,v1:172.21.15.45:6837/761645002] up:reconnect 2024-03-28T18:02:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:47 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:47 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T18:02:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:47 smithi151 ceph-mon[36481]: osdmap e146: 12 total, 12 up, 12 in 2024-03-28T18:02:47.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:47 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00299995 2024-03-28T18:02:47.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:47 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/761645002,v1:172.21.15.45:6837/761645002] up:reconnect 2024-03-28T18:02:47.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:47 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:reconnect} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:47.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:47 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T18:02:47.858 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:47 smithi045 ceph-mon[31773]: osdmap e146: 12 total, 12 up, 12 in 2024-03-28T18:02:47.939 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 620 sec 2024-03-28T18:02:48.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:48 smithi071 ceph-mon[36597]: pgmap v480: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T18:02:48.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:48 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/761645002,v1:172.21.15.45:6837/761645002] up:rejoin 2024-03-28T18:02:48.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:48 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:48.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:48 smithi071 ceph-mon[36597]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-03-28T18:02:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:48 smithi151 ceph-mon[36481]: pgmap v480: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T18:02:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:48 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/761645002,v1:172.21.15.45:6837/761645002] up:rejoin 2024-03-28T18:02:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:48 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:48 smithi151 ceph-mon[36481]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-03-28T18:02:48.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-mon[31773]: pgmap v480: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T18:02:48.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/761645002,v1:172.21.15.45:6837/761645002] up:rejoin 2024-03-28T18:02:48.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=g=up:rejoin} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:48.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-mon[31773]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-03-28T18:02:48.916 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3127200700 time 2024-03-28T18:02:48.457856+0000 2024-03-28T18:02:48.916 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:02:48.916 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:48.916 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3132bdd4a1] 2024-03-28T18:02:48.916 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3132bdd667] 2024-03-28T18:02:48.916 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560ea6943223] 2024-03-28T18:02:48.916 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560ea67bd5d6] 2024-03-28T18:02:48.916 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560ea67bdda1] 2024-03-28T18:02:48.916 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560ea67c02d5] 2024-03-28T18:02:48.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560ea6810233] 2024-03-28T18:02:48.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560ea6812197] 2024-03-28T18:02:48.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560ea671111b] 2024-03-28T18:02:48.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560ea67116bc] 2024-03-28T18:02:48.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560ea66fb1ff] 2024-03-28T18:02:48.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3132e6f488] 2024-03-28T18:02:48.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 13: (DispatchQueue::entry()+0x51f) [0x7f3132e6c60f] 2024-03-28T18:02:48.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3132f3fe81] 2024-03-28T18:02:48.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f313195a1ca] 2024-03-28T18:02:48.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 16: clone() 2024-03-28T18:02:48.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 2024-03-28T18:02:48.458+0000 7f3127200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3127200700 time 2024-03-28T18:02:48.457856+0000 2024-03-28T18:02:48.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:02:48.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 2024-03-28T18:02:48.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:48.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3132bdd4a1] 2024-03-28T18:02:48.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3132bdd667] 2024-03-28T18:02:48.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560ea6943223] 2024-03-28T18:02:48.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560ea67bd5d6] 2024-03-28T18:02:48.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560ea67bdda1] 2024-03-28T18:02:48.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560ea67c02d5] 2024-03-28T18:02:48.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560ea6810233] 2024-03-28T18:02:48.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560ea6812197] 2024-03-28T18:02:48.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560ea671111b] 2024-03-28T18:02:48.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560ea67116bc] 2024-03-28T18:02:48.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560ea66fb1ff] 2024-03-28T18:02:48.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3132e6f488] 2024-03-28T18:02:48.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 13: (DispatchQueue::entry()+0x51f) [0x7f3132e6c60f] 2024-03-28T18:02:48.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3132f3fe81] 2024-03-28T18:02:48.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f313195a1ca] 2024-03-28T18:02:48.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 16: clone() 2024-03-28T18:02:48.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 2024-03-28T18:02:48.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: *** Caught signal (Aborted) ** 2024-03-28T18:02:48.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: in thread 7f3127200700 thread_name:ms_dispatch 2024-03-28T18:02:48.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:48.919 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3131964d20] 2024-03-28T18:02:48.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 2: gsignal() 2024-03-28T18:02:48.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 3: abort() 2024-03-28T18:02:48.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3132bdd4fb] 2024-03-28T18:02:48.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3132bdd667] 2024-03-28T18:02:48.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560ea6943223] 2024-03-28T18:02:48.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560ea67bd5d6] 2024-03-28T18:02:48.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560ea67bdda1] 2024-03-28T18:02:48.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560ea67c02d5] 2024-03-28T18:02:48.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560ea6810233] 2024-03-28T18:02:48.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560ea6812197] 2024-03-28T18:02:48.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560ea671111b] 2024-03-28T18:02:48.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560ea67116bc] 2024-03-28T18:02:48.920 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560ea66fb1ff] 2024-03-28T18:02:48.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3132e6f488] 2024-03-28T18:02:48.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 16: (DispatchQueue::entry()+0x51f) [0x7f3132e6c60f] 2024-03-28T18:02:48.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3132f3fe81] 2024-03-28T18:02:48.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f313195a1ca] 2024-03-28T18:02:48.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 19: clone() 2024-03-28T18:02:48.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 2024-03-28T18:02:48.461+0000 7f3127200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:02:48.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: in thread 7f3127200700 thread_name:ms_dispatch 2024-03-28T18:02:48.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 2024-03-28T18:02:48.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:48.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3131964d20] 2024-03-28T18:02:48.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 2: gsignal() 2024-03-28T18:02:48.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 3: abort() 2024-03-28T18:02:48.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3132bdd4fb] 2024-03-28T18:02:48.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3132bdd667] 2024-03-28T18:02:48.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560ea6943223] 2024-03-28T18:02:48.921 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560ea67bd5d6] 2024-03-28T18:02:48.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560ea67bdda1] 2024-03-28T18:02:48.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560ea67c02d5] 2024-03-28T18:02:48.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560ea6810233] 2024-03-28T18:02:48.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560ea6812197] 2024-03-28T18:02:48.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560ea671111b] 2024-03-28T18:02:48.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560ea67116bc] 2024-03-28T18:02:48.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560ea66fb1ff] 2024-03-28T18:02:48.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3132e6f488] 2024-03-28T18:02:48.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 16: (DispatchQueue::entry()+0x51f) [0x7f3132e6c60f] 2024-03-28T18:02:48.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3132f3fe81] 2024-03-28T18:02:48.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f313195a1ca] 2024-03-28T18:02:48.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 19: clone() 2024-03-28T18:02:48.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:02:48.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 2024-03-28T18:02:48.922 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: -1> 2024-03-28T18:02:48.458+0000 7f3127200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3127200700 time 2024-03-28T18:02:48.457856+0000 2024-03-28T18:02:48.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:02:48.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 2024-03-28T18:02:48.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:48.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3132bdd4a1] 2024-03-28T18:02:48.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3132bdd667] 2024-03-28T18:02:48.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560ea6943223] 2024-03-28T18:02:48.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560ea67bd5d6] 2024-03-28T18:02:48.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560ea67bdda1] 2024-03-28T18:02:48.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560ea67c02d5] 2024-03-28T18:02:48.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560ea6810233] 2024-03-28T18:02:48.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560ea6812197] 2024-03-28T18:02:48.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560ea671111b] 2024-03-28T18:02:48.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560ea67116bc] 2024-03-28T18:02:48.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560ea66fb1ff] 2024-03-28T18:02:48.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3132e6f488] 2024-03-28T18:02:48.923 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 13: (DispatchQueue::entry()+0x51f) [0x7f3132e6c60f] 2024-03-28T18:02:48.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3132f3fe81] 2024-03-28T18:02:48.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f313195a1ca] 2024-03-28T18:02:48.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 16: clone() 2024-03-28T18:02:48.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 2024-03-28T18:02:48.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 0> 2024-03-28T18:02:48.461+0000 7f3127200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:02:48.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: in thread 7f3127200700 thread_name:ms_dispatch 2024-03-28T18:02:48.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 2024-03-28T18:02:48.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:48.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3131964d20] 2024-03-28T18:02:48.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 2: gsignal() 2024-03-28T18:02:48.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 3: abort() 2024-03-28T18:02:48.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3132bdd4fb] 2024-03-28T18:02:48.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3132bdd667] 2024-03-28T18:02:48.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560ea6943223] 2024-03-28T18:02:48.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560ea67bd5d6] 2024-03-28T18:02:48.924 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560ea67bdda1] 2024-03-28T18:02:48.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560ea67c02d5] 2024-03-28T18:02:48.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560ea6810233] 2024-03-28T18:02:48.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560ea6812197] 2024-03-28T18:02:48.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560ea671111b] 2024-03-28T18:02:48.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560ea67116bc] 2024-03-28T18:02:48.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560ea66fb1ff] 2024-03-28T18:02:48.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3132e6f488] 2024-03-28T18:02:48.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 16: (DispatchQueue::entry()+0x51f) [0x7f3132e6c60f] 2024-03-28T18:02:48.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3132f3fe81] 2024-03-28T18:02:48.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f313195a1ca] 2024-03-28T18:02:48.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 19: clone() 2024-03-28T18:02:48.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:02:48.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 2024-03-28T18:02:48.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: -1> 2024-03-28T18:02:48.458+0000 7f3127200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3127200700 time 2024-03-28T18:02:48.457856+0000 2024-03-28T18:02:48.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:02:48.925 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 2024-03-28T18:02:48.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:48.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3132bdd4a1] 2024-03-28T18:02:48.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3132bdd667] 2024-03-28T18:02:48.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560ea6943223] 2024-03-28T18:02:48.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560ea67bd5d6] 2024-03-28T18:02:48.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560ea67bdda1] 2024-03-28T18:02:48.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560ea67c02d5] 2024-03-28T18:02:48.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560ea6810233] 2024-03-28T18:02:48.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560ea6812197] 2024-03-28T18:02:48.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560ea671111b] 2024-03-28T18:02:48.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560ea67116bc] 2024-03-28T18:02:48.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560ea66fb1ff] 2024-03-28T18:02:48.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3132e6f488] 2024-03-28T18:02:48.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 13: (DispatchQueue::entry()+0x51f) [0x7f3132e6c60f] 2024-03-28T18:02:48.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3132f3fe81] 2024-03-28T18:02:48.926 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f313195a1ca] 2024-03-28T18:02:48.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 16: clone() 2024-03-28T18:02:48.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 2024-03-28T18:02:48.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 0> 2024-03-28T18:02:48.461+0000 7f3127200700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:02:48.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: in thread 7f3127200700 thread_name:ms_dispatch 2024-03-28T18:02:48.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 2024-03-28T18:02:48.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:02:48.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3131964d20] 2024-03-28T18:02:48.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 2: gsignal() 2024-03-28T18:02:48.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 3: abort() 2024-03-28T18:02:48.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3132bdd4fb] 2024-03-28T18:02:48.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3132bdd667] 2024-03-28T18:02:48.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560ea6943223] 2024-03-28T18:02:48.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560ea67bd5d6] 2024-03-28T18:02:48.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560ea67bdda1] 2024-03-28T18:02:48.927 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560ea67c02d5] 2024-03-28T18:02:48.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560ea6810233] 2024-03-28T18:02:48.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560ea6812197] 2024-03-28T18:02:48.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560ea671111b] 2024-03-28T18:02:48.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560ea67116bc] 2024-03-28T18:02:48.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560ea66fb1ff] 2024-03-28T18:02:48.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3132e6f488] 2024-03-28T18:02:48.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 16: (DispatchQueue::entry()+0x51f) [0x7f3132e6c60f] 2024-03-28T18:02:48.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3132f3fe81] 2024-03-28T18:02:48.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f313195a1ca] 2024-03-28T18:02:48.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 19: clone() 2024-03-28T18:02:48.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:02:48.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g[95640]: 2024-03-28T18:02:48.928 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:48 smithi045 podman[96028]: 2024-03-28 18:02:48.554004247 +0000 UTC m=+0.033528271 container died a3c2121e8f4ac803bdf37a7ee71ea6792f109f7608d3c736f054e852998c4c7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:02:48.939 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 621 sec 2024-03-28T18:02:49.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:49 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:02:49.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:49 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6836/761645002,v1:172.21.15.45:6837/761645002] up:active 2024-03-28T18:02:49.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:49 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:49.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:49 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 3 up:standby-replay 2024-03-28T18:02:49.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:49 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:02:49.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:49 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6836/761645002,v1:172.21.15.45:6837/761645002] up:active 2024-03-28T18:02:49.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:49 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:49.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:49 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 3 up:standby-replay 2024-03-28T18:02:49.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:49 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:02:49.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:49 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6836/761645002,v1:172.21.15.45:6837/761645002] up:active 2024-03-28T18:02:49.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:49 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 2 up:standby-replay 1 up:standby 2024-03-28T18:02:49.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:49 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=g=up:active} 3 up:standby-replay 2024-03-28T18:02:49.917 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:49 smithi045 podman[96028]: 2024-03-28 18:02:49.478490154 +0000 UTC m=+0.958014221 container remove a3c2121e8f4ac803bdf37a7ee71ea6792f109f7608d3c736f054e852998c4c7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:02:49.918 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:49 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:02:49.939 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 622 sec 2024-03-28T18:02:50.415 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:50 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-28T18:02:50.415 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:02:50 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Consumed 1.571s CPU time. 2024-03-28T18:02:50.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:50 smithi071 ceph-mon[36597]: pgmap v481: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 129 B/s wr, 4 op/s 2024-03-28T18:02:50.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:50 smithi151 ceph-mon[36481]: pgmap v481: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 129 B/s wr, 4 op/s 2024-03-28T18:02:50.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:50 smithi045 ceph-mon[31773]: pgmap v481: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 129 B/s wr, 4 op/s 2024-03-28T18:02:50.939 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 623 sec 2024-03-28T18:02:51.939 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 624 sec 2024-03-28T18:02:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:52 smithi071 ceph-mon[36597]: pgmap v482: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 511 B/s wr, 14 op/s 2024-03-28T18:02:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:52 smithi151 ceph-mon[36481]: pgmap v482: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 511 B/s wr, 14 op/s 2024-03-28T18:02:52.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:52 smithi045 ceph-mon[31773]: pgmap v482: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 511 B/s wr, 14 op/s 2024-03-28T18:02:52.939 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 625 sec 2024-03-28T18:02:53.939 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 626 sec 2024-03-28T18:02:54.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:54 smithi071 ceph-mon[36597]: pgmap v483: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 437 B/s wr, 12 op/s 2024-03-28T18:02:54.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:54 smithi151 ceph-mon[36481]: pgmap v483: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 437 B/s wr, 12 op/s 2024-03-28T18:02:54.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:54 smithi045 ceph-mon[31773]: pgmap v483: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 437 B/s wr, 12 op/s 2024-03-28T18:02:54.940 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 627 sec 2024-03-28T18:02:55.940 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 628 sec 2024-03-28T18:02:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:56 smithi071 ceph-mon[36597]: pgmap v484: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T18:02:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:56 smithi151 ceph-mon[36481]: pgmap v484: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T18:02:56.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:56 smithi045 ceph-mon[31773]: pgmap v484: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T18:02:56.940 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 629 sec 2024-03-28T18:02:57.940 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 630 sec 2024-03-28T18:02:58.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:02:58 smithi071 ceph-mon[36597]: pgmap v485: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 285 B/s wr, 9 op/s 2024-03-28T18:02:58.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:02:58 smithi151 ceph-mon[36481]: pgmap v485: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 285 B/s wr, 9 op/s 2024-03-28T18:02:58.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:02:58 smithi045 ceph-mon[31773]: pgmap v485: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 285 B/s wr, 9 op/s 2024-03-28T18:02:58.940 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 631 sec 2024-03-28T18:02:59.940 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 632 sec 2024-03-28T18:03:00.497 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:03:00 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 6. 2024-03-28T18:03:00.497 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:03:00 smithi045 systemd[1]: Stopped Ceph mds.g for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:03:00.497 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:03:00 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Consumed 1.571s CPU time. 2024-03-28T18:03:00.497 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:03:00 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Start request repeated too quickly. 2024-03-28T18:03:00.497 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:03:00 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-28T18:03:00.497 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 28 18:03:00 smithi045 systemd[1]: Failed to start Ceph mds.g for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:03:00.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:00 smithi071 ceph-mon[36597]: pgmap v486: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T18:03:00.845 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:00 smithi151 ceph-mon[36481]: pgmap v486: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T18:03:00.911 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:00 smithi045 ceph-mon[31773]: pgmap v486: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T18:03:00.940 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 633 sec 2024-03-28T18:03:01.941 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 634 sec 2024-03-28T18:03:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:02 smithi071 ceph-mon[36597]: pgmap v487: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T18:03:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:02 smithi151 ceph-mon[36481]: pgmap v487: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T18:03:02.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:02 smithi045 ceph-mon[31773]: pgmap v487: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T18:03:02.941 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 635 sec 2024-03-28T18:03:03.941 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 636 sec 2024-03-28T18:03:04.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:04 smithi071 ceph-mon[36597]: pgmap v488: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:03:04.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:04 smithi151 ceph-mon[36481]: pgmap v488: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:03:04.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:04 smithi045 ceph-mon[31773]: pgmap v488: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:03:04.941 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 637 sec 2024-03-28T18:03:05.941 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 638 sec 2024-03-28T18:03:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:06 smithi071 ceph-mon[36597]: pgmap v489: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-28T18:03:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:06 smithi151 ceph-mon[36481]: pgmap v489: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-28T18:03:06.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:06 smithi045 ceph-mon[31773]: pgmap v489: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-28T18:03:06.941 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 639 sec 2024-03-28T18:03:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:07 smithi071 ceph-mon[36597]: Replacing daemon mds.g as rank 2 with standby daemon mds.j 2024-03-28T18:03:07.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:07 smithi071 ceph-mon[36597]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-03-28T18:03:07.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:07 smithi071 ceph-mon[36597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:03:07.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:07 smithi071 ceph-mon[36597]: osdmap e147: 12 total, 12 up, 12 in 2024-03-28T18:03:07.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:07 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:replay} 2 up:standby-replay 2024-03-28T18:03:07.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:07 smithi151 ceph-mon[36481]: Replacing daemon mds.g as rank 2 with standby daemon mds.j 2024-03-28T18:03:07.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:07 smithi151 ceph-mon[36481]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-03-28T18:03:07.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:07 smithi151 ceph-mon[36481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:03:07.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:07 smithi151 ceph-mon[36481]: osdmap e147: 12 total, 12 up, 12 in 2024-03-28T18:03:07.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:07 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:replay} 2 up:standby-replay 2024-03-28T18:03:07.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:07 smithi045 ceph-mon[31773]: Replacing daemon mds.g as rank 2 with standby daemon mds.j 2024-03-28T18:03:07.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:07 smithi045 ceph-mon[31773]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-03-28T18:03:07.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:07 smithi045 ceph-mon[31773]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:03:07.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:07 smithi045 ceph-mon[31773]: osdmap e147: 12 total, 12 up, 12 in 2024-03-28T18:03:07.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:07 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:replay} 2 up:standby-replay 2024-03-28T18:03:07.941 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 640 sec 2024-03-28T18:03:08.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:08 smithi045 ceph-mon[31773]: pgmap v491: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:03:08.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:08 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/2048647835,v1:172.21.15.45:6835/2048647835] up:resolve 2024-03-28T18:03:08.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:08 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:resolve} 2 up:standby-replay 2024-03-28T18:03:08.942 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 641 sec 2024-03-28T18:03:09.016 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:03:09.016 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:03:09.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:08 smithi071 ceph-mon[36597]: pgmap v491: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:03:09.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:08 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/2048647835,v1:172.21.15.45:6835/2048647835] up:resolve 2024-03-28T18:03:09.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:08 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:resolve} 2 up:standby-replay 2024-03-28T18:03:09.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:08 smithi151 ceph-mon[36481]: pgmap v491: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:03:09.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:08 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/2048647835,v1:172.21.15.45:6835/2048647835] up:resolve 2024-03-28T18:03:09.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:08 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:resolve} 2 up:standby-replay 2024-03-28T18:03:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.260+0000 7fb829000640 1 -- 172.21.15.45:0/1238225051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb82410cf80 msgr2=0x7fb824114b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.260+0000 7fb829000640 1 --2- 172.21.15.45:0/1238225051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb82410cf80 0x7fb824114b50 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fb81c009900 tx=0x7fb81c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:03:09.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.261+0000 7fb829000640 1 -- 172.21.15.45:0/1238225051 shutdown_connections 2024-03-28T18:03:09.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.261+0000 7fb829000640 1 --2- 172.21.15.45:0/1238225051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb82410cf80 0x7fb824114b50 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.261+0000 7fb829000640 1 --2- 172.21.15.45:0/1238225051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb824108860 0x7fb82410c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.261+0000 7fb829000640 1 --2- 172.21.15.45:0/1238225051 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb824107eb0 0x7fb824108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.261+0000 7fb829000640 1 -- 172.21.15.45:0/1238225051 >> 172.21.15.45:0/1238225051 conn(0x7fb824075be0 msgr2=0x7fb824078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:03:09.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.261+0000 7fb829000640 1 -- 172.21.15.45:0/1238225051 shutdown_connections 2024-03-28T18:03:09.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.261+0000 7fb829000640 1 -- 172.21.15.45:0/1238225051 wait complete. 2024-03-28T18:03:09.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.262+0000 7fb829000640 1 Processor -- start 2024-03-28T18:03:09.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.262+0000 7fb829000640 1 -- start start 2024-03-28T18:03:09.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.262+0000 7fb829000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb824107eb0 0x7fb82419e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:09.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.262+0000 7fb829000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb82410cf80 0x7fb82419ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:09.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.262+0000 7fb829000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8241a2da0 0x7fb8241a3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:09.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.262+0000 7fb829000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb824118080 con 0x7fb8241a2da0 2024-03-28T18:03:09.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.262+0000 7fb829000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb824117f00 con 0x7fb824107eb0 2024-03-28T18:03:09.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.262+0000 7fb829000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb824118200 con 0x7fb82410cf80 2024-03-28T18:03:09.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.263+0000 7fb821e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb824107eb0 0x7fb82419e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:09.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.263+0000 7fb821e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb824107eb0 0x7fb82419e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:60232/0 (socket says 172.21.15.45:60232) 2024-03-28T18:03:09.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.263+0000 7fb821e00640 1 -- 172.21.15.45:0/2313236149 learned_addr learned my addr 172.21.15.45:0/2313236149 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:03:09.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.263+0000 7fb822800640 1 --2- 172.21.15.45:0/2313236149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8241a2da0 0x7fb8241a3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:09.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.263+0000 7fb819400640 1 --2- 172.21.15.45:0/2313236149 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb82410cf80 0x7fb82419ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:09.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.263+0000 7fb821e00640 1 -- 172.21.15.45:0/2313236149 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb82410cf80 msgr2=0x7fb82419ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:09.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.263+0000 7fb821e00640 1 --2- 172.21.15.45:0/2313236149 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb82410cf80 0x7fb82419ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.263+0000 7fb821e00640 1 -- 172.21.15.45:0/2313236149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8241a2da0 msgr2=0x7fb8241a3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:09.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.263+0000 7fb821e00640 1 --2- 172.21.15.45:0/2313236149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8241a2da0 0x7fb8241a3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.263+0000 7fb821e00640 1 -- 172.21.15.45:0/2313236149 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8241a3890 con 0x7fb824107eb0 2024-03-28T18:03:09.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.263+0000 7fb819400640 1 --2- 172.21.15.45:0/2313236149 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb82410cf80 0x7fb82419ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:03:09.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.263+0000 7fb821e00640 1 --2- 172.21.15.45:0/2313236149 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb824107eb0 0x7fb82419e4e0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fb81400ebc0 tx=0x7fb81400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:03:09.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.263+0000 7fb81b400640 1 -- 172.21.15.45:0/2313236149 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb814010630 con 0x7fb824107eb0 2024-03-28T18:03:09.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.263+0000 7fb829000640 1 -- 172.21.15.45:0/2313236149 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8241a3b80 con 0x7fb824107eb0 2024-03-28T18:03:09.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.263+0000 7fb81b400640 1 -- 172.21.15.45:0/2313236149 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb814010cb0 con 0x7fb824107eb0 2024-03-28T18:03:09.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.264+0000 7fb81b400640 1 -- 172.21.15.45:0/2313236149 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb814018bd0 con 0x7fb824107eb0 2024-03-28T18:03:09.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.264+0000 7fb829000640 1 -- 172.21.15.45:0/2313236149 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8241ab3c0 con 0x7fb824107eb0 2024-03-28T18:03:09.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.267+0000 7fb82bd97740 1 Processor -- start 2024-03-28T18:03:09.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.267+0000 7fb81b400640 1 -- 172.21.15.45:0/2313236149 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fb8140107d0 con 0x7fb824107eb0 2024-03-28T18:03:09.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.267+0000 7fb82bd97740 1 -- start start 2024-03-28T18:03:09.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.267+0000 7fb82bd97740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559153e12c00 0x559153e11950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:09.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.267+0000 7fb82bd97740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e11f20 0x559153e123a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:09.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.267+0000 7fb82bd97740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e1b290 0x559153e1d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:09.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.267+0000 7fb82bd97740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559153d76250 con 0x559153e11f20 2024-03-28T18:03:09.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.267+0000 7fb82bd97740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x559153d1ea30 con 0x559153e1b290 2024-03-28T18:03:09.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.267+0000 7fb82bd97740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x559153d722f0 con 0x559153e12c00 2024-03-28T18:03:09.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.268+0000 7fb819400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e1b290 0x559153e1d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:09.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.268+0000 7fb821e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e11f20 0x559153e123a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:09.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.268+0000 7fb819400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e1b290 0x559153e1d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:60242/0 (socket says 172.21.15.45:60242) 2024-03-28T18:03:09.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.268+0000 7fb819400640 1 -- 172.21.15.45:0/2892049079 learned_addr learned my addr 172.21.15.45:0/2892049079 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:03:09.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.268+0000 7fb822800640 1 --2- 172.21.15.45:0/2892049079 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559153e12c00 0x559153e11950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:09.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.268+0000 7fb819400640 1 -- 172.21.15.45:0/2892049079 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559153e12c00 msgr2=0x559153e11950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:03:09.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.268+0000 7fb819400640 1 --2- 172.21.15.45:0/2892049079 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559153e12c00 0x559153e11950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.268+0000 7fb819400640 1 -- 172.21.15.45:0/2892049079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e11f20 msgr2=0x559153e123a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:03:09.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.268+0000 7fb819400640 1 --2- 172.21.15.45:0/2892049079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e11f20 0x559153e123a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.268+0000 7fb819400640 1 -- 172.21.15.45:0/2892049079 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559153d09690 con 0x559153e1b290 2024-03-28T18:03:09.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.268+0000 7fb822800640 1 --2- 172.21.15.45:0/2892049079 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559153e12c00 0x559153e11950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:03:09.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.268+0000 7fb819400640 1 --2- 172.21.15.45:0/2892049079 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e1b290 0x559153e1d680 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb80c00bc60 tx=0x7fb80c00f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f95c5b67d77c75fb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:03:09.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.268+0000 7fb81b400640 1 --2- 172.21.15.45:0/2313236149 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fb8080a2970 0x7fb8080a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:09.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.269+0000 7fb813400640 1 -- 172.21.15.45:0/2892049079 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80c015070 con 0x559153e1b290 2024-03-28T18:03:09.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.269+0000 7fb813400640 1 -- 172.21.15.45:0/2892049079 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb80c01a940 con 0x559153e1b290 2024-03-28T18:03:09.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.269+0000 7fb813400640 1 -- 172.21.15.45:0/2892049079 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80c0119a0 con 0x559153e1b290 2024-03-28T18:03:09.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.269+0000 7fb81b400640 1 -- 172.21.15.45:0/2313236149 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 12382+0+0 (secure 0 0 0) 0x7fb8140d0410 con 0x7fb824107eb0 2024-03-28T18:03:09.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.269+0000 7fb822800640 1 --2- 172.21.15.45:0/2313236149 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fb8080a2970 0x7fb8080a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:09.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.269+0000 7fb82bd97740 1 -- 172.21.15.45:0/2892049079 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e1b290 msgr2=0x559153e1d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:09.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.269+0000 7fb82bd97740 1 --2- 172.21.15.45:0/2892049079 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e1b290 0x559153e1d680 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb80c00bc60 tx=0x7fb80c00f730 comp rx=0 tx=0).stop 2024-03-28T18:03:09.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.270+0000 7fb82bd97740 1 -- 172.21.15.45:0/2892049079 shutdown_connections 2024-03-28T18:03:09.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.270+0000 7fb82bd97740 1 --2- 172.21.15.45:0/2892049079 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e1b290 0x559153e1d680 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.270+0000 7fb82bd97740 1 --2- 172.21.15.45:0/2892049079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e11f20 0x559153e123a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.270+0000 7fb82bd97740 1 --2- 172.21.15.45:0/2892049079 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559153e12c00 0x559153e11950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.270+0000 7fb82bd97740 1 -- 172.21.15.45:0/2892049079 >> 172.21.15.45:0/2892049079 conn(0x559153dd9200 msgr2=0x559153e0c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:03:09.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.270+0000 7fb822800640 1 --2- 172.21.15.45:0/2313236149 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fb8080a2970 0x7fb8080a4e30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb81c006010 tx=0x7fb81c02df90 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:03:09.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.270+0000 7fb82bd97740 1 -- 172.21.15.45:0/2892049079 shutdown_connections 2024-03-28T18:03:09.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.270+0000 7fb82bd97740 1 -- 172.21.15.45:0/2892049079 wait complete. 2024-03-28T18:03:09.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.270+0000 7fb82bd97740 1 -- start start 2024-03-28T18:03:09.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.270+0000 7fb82bd97740 1 Processor -- start 2024-03-28T18:03:09.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.271+0000 7fb82bd97740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559153e11f20 0x559153e1a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:09.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.271+0000 7fb82bd97740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e12c00 0x559153e17ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:09.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.271+0000 7fb82bd97740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e1b290 0x559153e181e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:09.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.271+0000 7fb82bd97740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559153e1e100 con 0x559153e12c00 2024-03-28T18:03:09.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.271+0000 7fb82bd97740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x559153d31e80 con 0x559153e1b290 2024-03-28T18:03:09.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.271+0000 7fb82bd97740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x559153e1e280 con 0x559153e11f20 2024-03-28T18:03:09.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.271+0000 7fb82bd97740 10 client.0 _ll_register_callbacks cb 0x559153d44130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:03:09.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.271+0000 7fb819400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e12c00 0x559153e17ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:09.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.271+0000 7fb822800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e1b290 0x559153e181e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:09.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.271+0000 7fb819400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e12c00 0x559153e17ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55946/0 (socket says 172.21.15.45:55946) 2024-03-28T18:03:09.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.271+0000 7fb819400640 1 -- 172.21.15.45:0/722388683 learned_addr learned my addr 172.21.15.45:0/722388683 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:03:09.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.271+0000 7fb821e00640 1 --2- 172.21.15.45:0/722388683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559153e11f20 0x559153e1a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:09.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.272+0000 7fb819400640 1 -- 172.21.15.45:0/722388683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559153e11f20 msgr2=0x559153e1a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:03:09.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.272+0000 7fb819400640 1 --2- 172.21.15.45:0/722388683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559153e11f20 0x559153e1a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.272+0000 7fb819400640 1 -- 172.21.15.45:0/722388683 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e1b290 msgr2=0x559153e181e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:03:09.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.272+0000 7fb819400640 1 --2- 172.21.15.45:0/722388683 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e1b290 0x559153e181e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.272+0000 7fb819400640 1 -- 172.21.15.45:0/722388683 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559153b547d0 con 0x559153e12c00 2024-03-28T18:03:09.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.272+0000 7fb822800640 1 --2- 172.21.15.45:0/722388683 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e1b290 0x559153e181e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:03:09.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.272+0000 7fb819400640 1 --2- 172.21.15.45:0/722388683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e12c00 0x559153e17ca0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb80c005bc0 tx=0x7fb80c00bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d3b16e05026c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:03:09.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.272+0000 7fb807e00640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-28T18:03:09.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.272+0000 7fb807e00640 1 -- 172.21.15.45:0/722388683 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80c015040 con 0x559153e12c00 2024-03-28T18:03:09.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.272+0000 7fb807e00640 1 -- 172.21.15.45:0/722388683 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(147..147 src has 1..147) v4 ==== 12382+0+0 (secure 0 0 0) 0x7fb80c01aaf0 con 0x559153e12c00 2024-03-28T18:03:09.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.272+0000 7fb82bd97740 1 -- 172.21.15.45:0/722388683 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb80c01a940 con 0x559153e12c00 2024-03-28T18:03:09.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.273+0000 7fb807e00640 1 -- 172.21.15.45:0/722388683 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=315) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb80c01acf0 con 0x559153e12c00 2024-03-28T18:03:09.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.273+0000 7fb82bd97740 10 client.16278 fetch_fsmap learned FSMap version 315 2024-03-28T18:03:09.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.273+0000 7fb82bd97740 1 -- 172.21.15.45:0/722388683 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=315,monmap=4+}) v3 -- 0x559153a8a3f0 con 0x559153e12c00 2024-03-28T18:03:09.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.273+0000 7fb807e00640 1 -- 172.21.15.45:0/722388683 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb80c01a940 con 0x559153e12c00 2024-03-28T18:03:09.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.273+0000 7fb807e00640 1 -- 172.21.15.45:0/722388683 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 315) v1 ==== 3897+0+0 (secure 0 0 0) 0x7fb80c022030 con 0x559153e12c00 2024-03-28T18:03:09.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.273+0000 7fb82bd97740 10 client.16278 fetch_fsmap finished waiting for FSMap version 315 2024-03-28T18:03:09.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.273+0000 7fb82bd97740 10 client.16278 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:03:09.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.273+0000 7fb82bd97740 20 client.16278 populate_metadata read hostname 'smithi045' 2024-03-28T18:03:09.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.273+0000 7fb82bd97740 1 --2- 172.21.15.45:0/722388683 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x559153e2b390 0x559153e2d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:09.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.273+0000 7fb82bd97740 4 client.16278 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:03:09.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.273+0000 7fb82bd97740 1 -- 172.21.15.45:0/722388683 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559153d83790 con 0x559153e2b390 2024-03-28T18:03:09.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.273+0000 7fb821e00640 1 --2- 172.21.15.45:0/722388683 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x559153e2b390 0x559153e2d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:09.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.274+0000 7fb821e00640 1 --2- 172.21.15.45:0/722388683 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x559153e2b390 0x559153e2d7f0 crc :-1 s=READY pgs=405 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbc7d1b188133f9d server_cookie=4e690ea56cde435e in_seq=0 out_seq=0 2024-03-28T18:03:09.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.274+0000 7fb807e00640 10 client.16278 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:03:09.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.275+0000 7fb807e00640 1 -- 172.21.15.45:0/722388683 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559153d83790 con 0x559153e2b390 2024-03-28T18:03:09.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.275+0000 7fb807e00640 10 client.16278 handle_command_reply: tid=0 2024-03-28T18:03:09.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.275+0000 7fb82bd97740 1 client.16278 shutdown 2024-03-28T18:03:09.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.275+0000 7fb82bd97740 1 -- 172.21.15.45:0/722388683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e12c00 msgr2=0x559153e17ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:09.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.275+0000 7fb82bd97740 1 --2- 172.21.15.45:0/722388683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e12c00 0x559153e17ca0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb80c005bc0 tx=0x7fb80c00bf80 comp rx=0 tx=0).stop 2024-03-28T18:03:09.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.275+0000 7fb82bd97740 1 -- 172.21.15.45:0/722388683 shutdown_connections 2024-03-28T18:03:09.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.275+0000 7fb82bd97740 1 --2- 172.21.15.45:0/722388683 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x559153e2b390 0x559153e2d7f0 crc :-1 s=READY pgs=405 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.275+0000 7fb82bd97740 1 --2- 172.21.15.45:0/722388683 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e1b290 0x559153e181e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.275+0000 7fb82bd97740 1 --2- 172.21.15.45:0/722388683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e12c00 0x559153e17ca0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.275+0000 7fb82bd97740 1 --2- 172.21.15.45:0/722388683 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559153e11f20 0x559153e1a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.275+0000 7fb82bd97740 1 -- 172.21.15.45:0/722388683 >> 172.21.15.45:0/722388683 conn(0x559153dd9200 msgr2=0x559153e0e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:03:09.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.275+0000 7fb807e00640 0 client.16278 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:03:09.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.275+0000 7fb82bd97740 1 -- 172.21.15.45:0/722388683 shutdown_connections 2024-03-28T18:03:09.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.275+0000 7fb82bd97740 1 -- 172.21.15.45:0/722388683 wait complete. 2024-03-28T18:03:09.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.275+0000 7fb82bd97740 20 client.16278 trim_cache size 0 max 16384 2024-03-28T18:03:09.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.293+0000 7fb82bd97740 1 Processor -- start 2024-03-28T18:03:09.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.293+0000 7fb82bd97740 1 -- start start 2024-03-28T18:03:09.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.293+0000 7fb82bd97740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e11f20 0x559153e1a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:09.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.293+0000 7fb82bd97740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559153e12c00 0x559153e1a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:09.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.293+0000 7fb82bd97740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e1b290 0x559153e1c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:09.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.293+0000 7fb82bd97740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8241b5c40 con 0x559153e11f20 2024-03-28T18:03:09.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.293+0000 7fb82bd97740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8241b42c0 con 0x559153e1b290 2024-03-28T18:03:09.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.293+0000 7fb82bd97740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8241b55e0 con 0x559153e12c00 2024-03-28T18:03:09.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.294+0000 7fb822800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e1b290 0x559153e1c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:09.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.294+0000 7fb821e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e11f20 0x559153e1a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:09.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.294+0000 7fb822800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e1b290 0x559153e1c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:60266/0 (socket says 172.21.15.45:60266) 2024-03-28T18:03:09.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.294+0000 7fb822800640 1 -- 172.21.15.45:0/3796349138 learned_addr learned my addr 172.21.15.45:0/3796349138 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:03:09.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.294+0000 7fb819400640 1 --2- 172.21.15.45:0/3796349138 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559153e12c00 0x559153e1a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:09.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.294+0000 7fb822800640 1 -- 172.21.15.45:0/3796349138 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559153e12c00 msgr2=0x559153e1a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:03:09.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.294+0000 7fb822800640 1 --2- 172.21.15.45:0/3796349138 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559153e12c00 0x559153e1a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.294+0000 7fb822800640 1 -- 172.21.15.45:0/3796349138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e11f20 msgr2=0x559153e1a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:03:09.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.294+0000 7fb822800640 1 --2- 172.21.15.45:0/3796349138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e11f20 0x559153e1a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.294+0000 7fb822800640 1 -- 172.21.15.45:0/3796349138 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559153a4a0f0 con 0x559153e1b290 2024-03-28T18:03:09.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.294+0000 7fb822800640 1 --2- 172.21.15.45:0/3796349138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e1b290 0x559153e1c680 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb81c009900 tx=0x7fb81c041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cfcf10250745ecc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:03:09.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.295+0000 7fb812a00640 1 -- 172.21.15.45:0/3796349138 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c045070 con 0x559153e1b290 2024-03-28T18:03:09.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.295+0000 7fb812a00640 1 -- 172.21.15.45:0/3796349138 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb81c004ba0 con 0x559153e1b290 2024-03-28T18:03:09.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.295+0000 7fb812a00640 1 -- 172.21.15.45:0/3796349138 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c005390 con 0x559153e1b290 2024-03-28T18:03:09.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.295+0000 7fb82bd97740 1 -- 172.21.15.45:0/3796349138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e1b290 msgr2=0x559153e1c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:09.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.295+0000 7fb82bd97740 1 --2- 172.21.15.45:0/3796349138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e1b290 0x559153e1c680 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb81c009900 tx=0x7fb81c041000 comp rx=0 tx=0).stop 2024-03-28T18:03:09.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.295+0000 7fb82bd97740 1 -- 172.21.15.45:0/3796349138 shutdown_connections 2024-03-28T18:03:09.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.295+0000 7fb82bd97740 1 --2- 172.21.15.45:0/3796349138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e1b290 0x559153e1c680 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.295+0000 7fb82bd97740 1 --2- 172.21.15.45:0/3796349138 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559153e12c00 0x559153e1a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.295+0000 7fb82bd97740 1 --2- 172.21.15.45:0/3796349138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e11f20 0x559153e1a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.295+0000 7fb82bd97740 1 -- 172.21.15.45:0/3796349138 >> 172.21.15.45:0/3796349138 conn(0x559153dd9200 msgr2=0x559153e0c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:03:09.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.295+0000 7fb82bd97740 1 -- 172.21.15.45:0/3796349138 shutdown_connections 2024-03-28T18:03:09.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.296+0000 7fb82bd97740 1 -- 172.21.15.45:0/3796349138 wait complete. 2024-03-28T18:03:09.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.296+0000 7fb82bd97740 1 -- start start 2024-03-28T18:03:09.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.296+0000 7fb82bd97740 1 Processor -- start 2024-03-28T18:03:09.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.296+0000 7fb82bd97740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e11f20 0x559153e14550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:09.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.296+0000 7fb82bd97740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e12c00 0x559153e14a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:09.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.297+0000 7fb82bd97740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559153e1b290 0x559153e14fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:09.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.297+0000 7fb82bd97740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8241b4f80 con 0x559153e12c00 2024-03-28T18:03:09.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.297+0000 7fb82bd97740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8241b4920 con 0x559153e11f20 2024-03-28T18:03:09.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.297+0000 7fb82bd97740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x559153e16230 con 0x559153e1b290 2024-03-28T18:03:09.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.297+0000 7fb82bd97740 10 client.0 _ll_register_callbacks cb 0x7fb8140d3e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:03:09.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.297+0000 7fb819400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e12c00 0x559153e14a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:09.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.297+0000 7fb819400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e12c00 0x559153e14a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55964/0 (socket says 172.21.15.45:55964) 2024-03-28T18:03:09.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.297+0000 7fb819400640 1 -- 172.21.15.45:0/584245129 learned_addr learned my addr 172.21.15.45:0/584245129 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:03:09.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.297+0000 7fb821e00640 1 --2- 172.21.15.45:0/584245129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e11f20 0x559153e14550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:09.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.297+0000 7fb822800640 1 --2- 172.21.15.45:0/584245129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559153e1b290 0x559153e14fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:09.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.297+0000 7fb819400640 1 -- 172.21.15.45:0/584245129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559153e1b290 msgr2=0x559153e14fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:03:09.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.297+0000 7fb819400640 1 --2- 172.21.15.45:0/584245129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559153e1b290 0x559153e14fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.297+0000 7fb819400640 1 -- 172.21.15.45:0/584245129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e11f20 msgr2=0x559153e14550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:03:09.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.297+0000 7fb819400640 1 --2- 172.21.15.45:0/584245129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e11f20 0x559153e14550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.297+0000 7fb819400640 1 -- 172.21.15.45:0/584245129 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559153e85d10 con 0x559153e12c00 2024-03-28T18:03:09.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.297+0000 7fb819400640 1 --2- 172.21.15.45:0/584245129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e12c00 0x559153e14a90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb80c00c010 tx=0x7fb80c00b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76a7ab7f193b51aa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:03:09.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.297+0000 7fb807e00640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-28T18:03:09.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.298+0000 7fb807e00640 1 -- 172.21.15.45:0/584245129 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80c015040 con 0x559153e12c00 2024-03-28T18:03:09.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.298+0000 7fb807e00640 1 -- 172.21.15.45:0/584245129 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(147..147 src has 1..147) v4 ==== 12382+0+0 (secure 0 0 0) 0x7fb80c01aa80 con 0x559153e12c00 2024-03-28T18:03:09.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.298+0000 7fb82bd97740 1 -- 172.21.15.45:0/584245129 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb81c004ba0 con 0x559153e12c00 2024-03-28T18:03:09.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.298+0000 7fb807e00640 1 -- 172.21.15.45:0/584245129 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=315) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb80c03f050 con 0x559153e12c00 2024-03-28T18:03:09.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.298+0000 7fb82bd97740 10 client.16284 fetch_fsmap learned FSMap version 315 2024-03-28T18:03:09.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.298+0000 7fb82bd97740 1 -- 172.21.15.45:0/584245129 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=315,monmap=4+}) v3 -- 0x559153e86fe0 con 0x559153e12c00 2024-03-28T18:03:09.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.298+0000 7fb807e00640 1 -- 172.21.15.45:0/584245129 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb81c004ba0 con 0x559153e12c00 2024-03-28T18:03:09.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.298+0000 7fb807e00640 1 -- 172.21.15.45:0/584245129 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 315) v1 ==== 3897+0+0 (secure 0 0 0) 0x7fb80c022040 con 0x559153e12c00 2024-03-28T18:03:09.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.299+0000 7fb82bd97740 10 client.16284 fetch_fsmap finished waiting for FSMap version 315 2024-03-28T18:03:09.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.299+0000 7fb82bd97740 10 client.16284 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:03:09.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.299+0000 7fb82bd97740 20 client.16284 populate_metadata read hostname 'smithi045' 2024-03-28T18:03:09.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.299+0000 7fb82bd97740 1 --2- 172.21.15.45:0/584245129 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x559153e875e0 0x559153e89a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:09.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.299+0000 7fb82bd97740 4 client.16284 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:03:09.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.299+0000 7fb82bd97740 1 -- 172.21.15.45:0/584245129 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559153d7dae0 con 0x559153e875e0 2024-03-28T18:03:09.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.299+0000 7fb821e00640 1 --2- 172.21.15.45:0/584245129 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x559153e875e0 0x559153e89a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:09.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.299+0000 7fb821e00640 1 --2- 172.21.15.45:0/584245129 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x559153e875e0 0x559153e89a00 crc :-1 s=READY pgs=406 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dca3b88e0ec7b781 server_cookie=91e1c50a33197412 in_seq=0 out_seq=0 2024-03-28T18:03:09.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.299+0000 7fb807e00640 10 client.16284 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:03:09.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.299+0000 7fb807e00640 1 -- 172.21.15.45:0/584245129 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x559153d7dae0 con 0x559153e875e0 2024-03-28T18:03:09.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.299+0000 7fb807e00640 10 client.16284 handle_command_reply: tid=0 2024-03-28T18:03:09.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.299+0000 7fb82bd97740 1 client.16284 shutdown 2024-03-28T18:03:09.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.300+0000 7fb82bd97740 1 -- 172.21.15.45:0/584245129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e12c00 msgr2=0x559153e14a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:09.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.300+0000 7fb82bd97740 1 --2- 172.21.15.45:0/584245129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e12c00 0x559153e14a90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb80c00c010 tx=0x7fb80c00b9d0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.300+0000 7fb82bd97740 1 -- 172.21.15.45:0/584245129 shutdown_connections 2024-03-28T18:03:09.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.300+0000 7fb82bd97740 1 --2- 172.21.15.45:0/584245129 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x559153e875e0 0x559153e89a00 crc :-1 s=READY pgs=406 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.300+0000 7fb82bd97740 1 --2- 172.21.15.45:0/584245129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x559153e1b290 0x559153e14fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.300+0000 7fb82bd97740 1 --2- 172.21.15.45:0/584245129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559153e12c00 0x559153e14a90 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.300+0000 7fb807e00640 0 client.16284 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:03:09.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.300+0000 7fb82bd97740 1 --2- 172.21.15.45:0/584245129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559153e11f20 0x559153e14550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.300+0000 7fb82bd97740 1 -- 172.21.15.45:0/584245129 >> 172.21.15.45:0/584245129 conn(0x559153dd9200 msgr2=0x559153e16570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:03:09.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.300+0000 7fb82bd97740 1 -- 172.21.15.45:0/584245129 shutdown_connections 2024-03-28T18:03:09.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.300+0000 7fb82bd97740 1 -- 172.21.15.45:0/584245129 wait complete. 2024-03-28T18:03:09.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.300+0000 7fb82bd97740 20 client.16284 trim_cache size 0 max 16384 2024-03-28T18:03:09.308 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T18:03:09.308 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:03:09.308 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T18:03:09.308 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T18:03:09.308 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T18:03:09.308 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T18:03:09.308 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T18:03:09.308 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:03:09.308 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:03:09.308 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T18:03:09.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.301+0000 7fb829000640 1 -- 172.21.15.45:0/2313236149 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fb8080a2970 msgr2=0x7fb8080a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:09.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.301+0000 7fb829000640 1 --2- 172.21.15.45:0/2313236149 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fb8080a2970 0x7fb8080a4e30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb81c006010 tx=0x7fb81c02df90 comp rx=0 tx=0).stop 2024-03-28T18:03:09.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.301+0000 7fb829000640 1 -- 172.21.15.45:0/2313236149 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb824107eb0 msgr2=0x7fb82419e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:09.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.301+0000 7fb829000640 1 --2- 172.21.15.45:0/2313236149 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb824107eb0 0x7fb82419e4e0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fb81400ebc0 tx=0x7fb81400c6a0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.301+0000 7fb829000640 1 -- 172.21.15.45:0/2313236149 shutdown_connections 2024-03-28T18:03:09.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.301+0000 7fb829000640 1 --2- 172.21.15.45:0/2313236149 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fb8080a2970 0x7fb8080a4e30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.301+0000 7fb829000640 1 --2- 172.21.15.45:0/2313236149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8241a2da0 0x7fb8241a3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.301+0000 7fb829000640 1 --2- 172.21.15.45:0/2313236149 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb82410cf80 0x7fb82419ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.301+0000 7fb829000640 1 --2- 172.21.15.45:0/2313236149 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb824107eb0 0x7fb82419e4e0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:09.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.301+0000 7fb829000640 1 -- 172.21.15.45:0/2313236149 >> 172.21.15.45:0/2313236149 conn(0x7fb824075be0 msgr2=0x7fb8240766e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:03:09.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.301+0000 7fb829000640 1 -- 172.21.15.45:0/2313236149 shutdown_connections 2024-03-28T18:03:09.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:09.301+0000 7fb829000640 1 -- 172.21.15.45:0/2313236149 wait complete. 2024-03-28T18:03:09.319 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T18:03:09.942 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 642 sec 2024-03-28T18:03:10.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:10 smithi071 ceph-mon[36597]: pgmap v492: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:03:10.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:10 smithi071 ceph-mon[36597]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T18:03:10.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:10 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/2048647835,v1:172.21.15.45:6835/2048647835] up:reconnect 2024-03-28T18:03:10.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:10 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:reconnect} 2 up:standby-replay 2024-03-28T18:03:10.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:10 smithi071 ceph-mon[36597]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T18:03:10.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:10 smithi151 ceph-mon[36481]: pgmap v492: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:03:10.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:10 smithi151 ceph-mon[36481]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T18:03:10.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:10 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/2048647835,v1:172.21.15.45:6835/2048647835] up:reconnect 2024-03-28T18:03:10.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:10 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:reconnect} 2 up:standby-replay 2024-03-28T18:03:10.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:10 smithi151 ceph-mon[36481]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T18:03:10.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:10 smithi045 ceph-mon[31773]: pgmap v492: 97 pgs: 97 active+clean; 27 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:03:10.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:10 smithi045 ceph-mon[31773]: reconnect by client.25691 172.21.15.45:0/3500083598 after 0.00199997 2024-03-28T18:03:10.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:10 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/2048647835,v1:172.21.15.45:6835/2048647835] up:reconnect 2024-03-28T18:03:10.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:10 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:reconnect} 2 up:standby-replay 2024-03-28T18:03:10.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:10 smithi045 ceph-mon[31773]: reconnect by client.15069 v1:192.168.0.1:0/466750684 after 0.00899986 2024-03-28T18:03:10.942 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 643 sec 2024-03-28T18:03:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:11 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/2048647835,v1:172.21.15.45:6835/2048647835] up:rejoin 2024-03-28T18:03:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:11 smithi071 ceph-mon[36597]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:rejoin} 2 up:standby-replay 2024-03-28T18:03:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:11 smithi071 ceph-mon[36597]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-03-28T18:03:11.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:11 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/2048647835,v1:172.21.15.45:6835/2048647835] up:rejoin 2024-03-28T18:03:11.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:11 smithi151 ceph-mon[36481]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:rejoin} 2 up:standby-replay 2024-03-28T18:03:11.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:11 smithi151 ceph-mon[36481]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-03-28T18:03:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/2048647835,v1:172.21.15.45:6835/2048647835] up:rejoin 2024-03-28T18:03:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-mon[31773]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=j=up:rejoin} 2 up:standby-replay 2024-03-28T18:03:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-mon[31773]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-03-28T18:03:11.917 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7d98000700 time 2024-03-28T18:03:11.564031+0000 2024-03-28T18:03:11.917 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:03:11.917 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:03:11.917 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7da39734a1] 2024-03-28T18:03:11.917 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7da3973667] 2024-03-28T18:03:11.917 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55710112e223] 2024-03-28T18:03:11.917 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557100fa85d6] 2024-03-28T18:03:11.917 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557100fa8da1] 2024-03-28T18:03:11.917 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557100fab2d5] 2024-03-28T18:03:11.918 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557100ffb233] 2024-03-28T18:03:11.918 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557100ffd197] 2024-03-28T18:03:11.918 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557100efc11b] 2024-03-28T18:03:11.918 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557100efc6bc] 2024-03-28T18:03:11.918 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557100ee61ff] 2024-03-28T18:03:11.918 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7da3c05488] 2024-03-28T18:03:11.918 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 13: (DispatchQueue::entry()+0x51f) [0x7f7da3c0260f] 2024-03-28T18:03:11.918 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7da3cd5e81] 2024-03-28T18:03:11.918 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7da26f01ca] 2024-03-28T18:03:11.918 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 16: clone() 2024-03-28T18:03:11.919 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 2024-03-28T18:03:11.565+0000 7f7d98000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7d98000700 time 2024-03-28T18:03:11.564031+0000 2024-03-28T18:03:11.919 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:03:11.919 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 2024-03-28T18:03:11.919 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:03:11.919 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7da39734a1] 2024-03-28T18:03:11.919 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7da3973667] 2024-03-28T18:03:11.919 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55710112e223] 2024-03-28T18:03:11.919 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557100fa85d6] 2024-03-28T18:03:11.919 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557100fa8da1] 2024-03-28T18:03:11.919 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557100fab2d5] 2024-03-28T18:03:11.919 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557100ffb233] 2024-03-28T18:03:11.919 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557100ffd197] 2024-03-28T18:03:11.919 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557100efc11b] 2024-03-28T18:03:11.919 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557100efc6bc] 2024-03-28T18:03:11.920 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557100ee61ff] 2024-03-28T18:03:11.920 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7da3c05488] 2024-03-28T18:03:11.920 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 13: (DispatchQueue::entry()+0x51f) [0x7f7da3c0260f] 2024-03-28T18:03:11.920 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7da3cd5e81] 2024-03-28T18:03:11.920 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7da26f01ca] 2024-03-28T18:03:11.920 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 16: clone() 2024-03-28T18:03:11.920 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 2024-03-28T18:03:11.920 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: *** Caught signal (Aborted) ** 2024-03-28T18:03:11.920 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: in thread 7f7d98000700 thread_name:ms_dispatch 2024-03-28T18:03:11.920 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:03:11.920 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7da26fad20] 2024-03-28T18:03:11.920 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 2: gsignal() 2024-03-28T18:03:11.921 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 3: abort() 2024-03-28T18:03:11.921 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7da39734fb] 2024-03-28T18:03:11.921 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7da3973667] 2024-03-28T18:03:11.921 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55710112e223] 2024-03-28T18:03:11.921 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557100fa85d6] 2024-03-28T18:03:11.921 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557100fa8da1] 2024-03-28T18:03:11.921 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557100fab2d5] 2024-03-28T18:03:11.921 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557100ffb233] 2024-03-28T18:03:11.921 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557100ffd197] 2024-03-28T18:03:11.921 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557100efc11b] 2024-03-28T18:03:11.921 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557100efc6bc] 2024-03-28T18:03:11.921 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557100ee61ff] 2024-03-28T18:03:11.921 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7da3c05488] 2024-03-28T18:03:11.921 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 16: (DispatchQueue::entry()+0x51f) [0x7f7da3c0260f] 2024-03-28T18:03:11.921 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7da3cd5e81] 2024-03-28T18:03:11.921 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7da26f01ca] 2024-03-28T18:03:11.922 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 19: clone() 2024-03-28T18:03:11.922 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 2024-03-28T18:03:11.569+0000 7f7d98000700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:03:11.922 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: in thread 7f7d98000700 thread_name:ms_dispatch 2024-03-28T18:03:11.922 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 2024-03-28T18:03:11.922 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:03:11.922 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7da26fad20] 2024-03-28T18:03:11.922 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 2: gsignal() 2024-03-28T18:03:11.922 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 3: abort() 2024-03-28T18:03:11.922 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7da39734fb] 2024-03-28T18:03:11.922 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7da3973667] 2024-03-28T18:03:11.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55710112e223] 2024-03-28T18:03:11.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557100fa85d6] 2024-03-28T18:03:11.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557100fa8da1] 2024-03-28T18:03:11.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557100fab2d5] 2024-03-28T18:03:11.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557100ffb233] 2024-03-28T18:03:11.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557100ffd197] 2024-03-28T18:03:11.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557100efc11b] 2024-03-28T18:03:11.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557100efc6bc] 2024-03-28T18:03:11.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557100ee61ff] 2024-03-28T18:03:11.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7da3c05488] 2024-03-28T18:03:11.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 16: (DispatchQueue::entry()+0x51f) [0x7f7da3c0260f] 2024-03-28T18:03:11.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7da3cd5e81] 2024-03-28T18:03:11.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7da26f01ca] 2024-03-28T18:03:11.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 19: clone() 2024-03-28T18:03:11.923 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:03:11.924 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 2024-03-28T18:03:11.924 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: -1> 2024-03-28T18:03:11.565+0000 7f7d98000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7d98000700 time 2024-03-28T18:03:11.564031+0000 2024-03-28T18:03:11.924 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:03:11.924 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 2024-03-28T18:03:11.924 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:03:11.924 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7da39734a1] 2024-03-28T18:03:11.924 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7da3973667] 2024-03-28T18:03:11.924 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55710112e223] 2024-03-28T18:03:11.924 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557100fa85d6] 2024-03-28T18:03:11.924 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557100fa8da1] 2024-03-28T18:03:11.924 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557100fab2d5] 2024-03-28T18:03:11.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557100ffb233] 2024-03-28T18:03:11.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557100ffd197] 2024-03-28T18:03:11.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557100efc11b] 2024-03-28T18:03:11.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557100efc6bc] 2024-03-28T18:03:11.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557100ee61ff] 2024-03-28T18:03:11.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7da3c05488] 2024-03-28T18:03:11.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 13: (DispatchQueue::entry()+0x51f) [0x7f7da3c0260f] 2024-03-28T18:03:11.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7da3cd5e81] 2024-03-28T18:03:11.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7da26f01ca] 2024-03-28T18:03:11.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 16: clone() 2024-03-28T18:03:11.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 2024-03-28T18:03:11.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 0> 2024-03-28T18:03:11.569+0000 7f7d98000700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:03:11.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: in thread 7f7d98000700 thread_name:ms_dispatch 2024-03-28T18:03:11.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 2024-03-28T18:03:11.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:03:11.925 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7da26fad20] 2024-03-28T18:03:11.926 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 2: gsignal() 2024-03-28T18:03:11.926 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 3: abort() 2024-03-28T18:03:11.926 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7da39734fb] 2024-03-28T18:03:11.926 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7da3973667] 2024-03-28T18:03:11.926 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55710112e223] 2024-03-28T18:03:11.926 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557100fa85d6] 2024-03-28T18:03:11.926 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557100fa8da1] 2024-03-28T18:03:11.926 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557100fab2d5] 2024-03-28T18:03:11.926 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557100ffb233] 2024-03-28T18:03:11.926 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557100ffd197] 2024-03-28T18:03:11.926 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557100efc11b] 2024-03-28T18:03:11.927 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557100efc6bc] 2024-03-28T18:03:11.927 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557100ee61ff] 2024-03-28T18:03:11.927 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7da3c05488] 2024-03-28T18:03:11.927 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 16: (DispatchQueue::entry()+0x51f) [0x7f7da3c0260f] 2024-03-28T18:03:11.927 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7da3cd5e81] 2024-03-28T18:03:11.927 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7da26f01ca] 2024-03-28T18:03:11.927 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 19: clone() 2024-03-28T18:03:11.927 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:03:11.927 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 2024-03-28T18:03:11.927 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: -1> 2024-03-28T18:03:11.565+0000 7f7d98000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7d98000700 time 2024-03-28T18:03:11.564031+0000 2024-03-28T18:03:11.927 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-28T18:03:11.927 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 2024-03-28T18:03:11.927 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:03:11.927 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7da39734a1] 2024-03-28T18:03:11.927 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7da3973667] 2024-03-28T18:03:11.928 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55710112e223] 2024-03-28T18:03:11.928 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557100fa85d6] 2024-03-28T18:03:11.928 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557100fa8da1] 2024-03-28T18:03:11.928 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557100fab2d5] 2024-03-28T18:03:11.928 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557100ffb233] 2024-03-28T18:03:11.928 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557100ffd197] 2024-03-28T18:03:11.928 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557100efc11b] 2024-03-28T18:03:11.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557100efc6bc] 2024-03-28T18:03:11.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557100ee61ff] 2024-03-28T18:03:11.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7da3c05488] 2024-03-28T18:03:11.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 13: (DispatchQueue::entry()+0x51f) [0x7f7da3c0260f] 2024-03-28T18:03:11.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7da3cd5e81] 2024-03-28T18:03:11.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7da26f01ca] 2024-03-28T18:03:11.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 16: clone() 2024-03-28T18:03:11.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 2024-03-28T18:03:11.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 0> 2024-03-28T18:03:11.569+0000 7f7d98000700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:03:11.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: in thread 7f7d98000700 thread_name:ms_dispatch 2024-03-28T18:03:11.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 2024-03-28T18:03:11.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:03:11.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7da26fad20] 2024-03-28T18:03:11.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 2: gsignal() 2024-03-28T18:03:11.929 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 3: abort() 2024-03-28T18:03:11.930 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7da39734fb] 2024-03-28T18:03:11.930 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7da3973667] 2024-03-28T18:03:11.930 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55710112e223] 2024-03-28T18:03:11.930 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557100fa85d6] 2024-03-28T18:03:11.930 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557100fa8da1] 2024-03-28T18:03:11.930 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x557100fab2d5] 2024-03-28T18:03:11.930 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557100ffb233] 2024-03-28T18:03:11.930 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557100ffd197] 2024-03-28T18:03:11.930 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557100efc11b] 2024-03-28T18:03:11.930 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557100efc6bc] 2024-03-28T18:03:11.930 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557100ee61ff] 2024-03-28T18:03:11.930 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7da3c05488] 2024-03-28T18:03:11.930 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 16: (DispatchQueue::entry()+0x51f) [0x7f7da3c0260f] 2024-03-28T18:03:11.930 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7da3cd5e81] 2024-03-28T18:03:11.930 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7da26f01ca] 2024-03-28T18:03:11.930 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 19: clone() 2024-03-28T18:03:11.931 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:03:11.931 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j[95981]: 2024-03-28T18:03:11.931 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:11 smithi045 podman[96141]: 2024-03-28 18:03:11.687688343 +0000 UTC m=+0.032269391 container died 5196cb77ce09818605e5e0bec9dfdd30f19da0a879a5ba3df5466ba6c5859735 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-28T18:03:11.942 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 644 sec 2024-03-28T18:03:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:12 smithi071 ceph-mon[36597]: pgmap v493: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:03:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:12 smithi071 ceph-mon[36597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:03:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:12 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.45:6834/2048647835,v1:172.21.15.45:6835/2048647835] up:active 2024-03-28T18:03:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:12 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 2 up:standby-replay 2024-03-28T18:03:12.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:12 smithi151 ceph-mon[36481]: pgmap v493: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:03:12.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:12 smithi151 ceph-mon[36481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:03:12.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:12 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.45:6834/2048647835,v1:172.21.15.45:6835/2048647835] up:active 2024-03-28T18:03:12.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:12 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 2 up:standby-replay 2024-03-28T18:03:12.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:12 smithi045 ceph-mon[31773]: pgmap v493: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:03:12.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:12 smithi045 ceph-mon[31773]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:03:12.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:12 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.45:6834/2048647835,v1:172.21.15.45:6835/2048647835] up:active 2024-03-28T18:03:12.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:12 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active} 2 up:standby-replay 2024-03-28T18:03:12.916 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:12 smithi045 podman[96141]: 2024-03-28 18:03:12.68141051 +0000 UTC m=+1.025991602 container remove 5196cb77ce09818605e5e0bec9dfdd30f19da0a879a5ba3df5466ba6c5859735 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:03:12.916 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:12 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:03:12.942 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 645 sec 2024-03-28T18:03:13.665 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:13 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-28T18:03:13.666 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:13 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Consumed 1.625s CPU time. 2024-03-28T18:03:13.942 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 646 sec 2024-03-28T18:03:14.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:14 smithi071 ceph-mon[36597]: pgmap v494: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:03:14.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:14 smithi151 ceph-mon[36481]: pgmap v494: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:03:14.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:14 smithi045 ceph-mon[31773]: pgmap v494: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T18:03:14.942 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 647 sec 2024-03-28T18:03:15.942 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 648 sec 2024-03-28T18:03:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:16 smithi071 ceph-mon[36597]: pgmap v495: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 27 KiB/s wr, 25 op/s 2024-03-28T18:03:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:16 smithi151 ceph-mon[36481]: pgmap v495: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 27 KiB/s wr, 25 op/s 2024-03-28T18:03:16.911 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:16 smithi045 ceph-mon[31773]: pgmap v495: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 27 KiB/s wr, 25 op/s 2024-03-28T18:03:16.943 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 649 sec 2024-03-28T18:03:17.943 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 650 sec 2024-03-28T18:03:18.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:18 smithi071 ceph-mon[36597]: pgmap v496: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 24 op/s 2024-03-28T18:03:18.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:18 smithi151 ceph-mon[36481]: pgmap v496: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 24 op/s 2024-03-28T18:03:18.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:18 smithi045 ceph-mon[31773]: pgmap v496: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 24 op/s 2024-03-28T18:03:18.943 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 651 sec 2024-03-28T18:03:19.943 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 652 sec 2024-03-28T18:03:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:20 smithi151 ceph-mon[36481]: pgmap v497: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-03-28T18:03:20.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:20 smithi045 ceph-mon[31773]: pgmap v497: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-03-28T18:03:20.943 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 653 sec 2024-03-28T18:03:21.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:20 smithi071 ceph-mon[36597]: pgmap v497: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-03-28T18:03:21.943 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 654 sec 2024-03-28T18:03:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:22 smithi151 ceph-mon[36481]: pgmap v498: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:03:22.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:22 smithi045 ceph-mon[31773]: pgmap v498: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:03:22.943 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 655 sec 2024-03-28T18:03:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:22 smithi071 ceph-mon[36597]: pgmap v498: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:03:23.916 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:23 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 6. 2024-03-28T18:03:23.916 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:23 smithi045 systemd[1]: Stopped Ceph mds.j for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:03:23.916 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:23 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Consumed 1.625s CPU time. 2024-03-28T18:03:23.916 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:23 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Start request repeated too quickly. 2024-03-28T18:03:23.916 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:23 smithi045 systemd[1]: ceph-9f4888ec-ed29-11ee-95d0-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-28T18:03:23.916 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 28 18:03:23 smithi045 systemd[1]: Failed to start Ceph mds.j for 9f4888ec-ed29-11ee-95d0-87774f69a715. 2024-03-28T18:03:23.944 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 656 sec 2024-03-28T18:03:24.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:24 smithi045 ceph-mon[31773]: pgmap v499: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-28T18:03:24.944 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 657 sec 2024-03-28T18:03:25.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:24 smithi071 ceph-mon[36597]: pgmap v499: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-28T18:03:25.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:24 smithi151 ceph-mon[36481]: pgmap v499: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-28T18:03:25.944 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 658 sec 2024-03-28T18:03:26.875 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:26 smithi151 ceph-mon[36481]: pgmap v500: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T18:03:26.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:26 smithi045 ceph-mon[31773]: pgmap v500: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T18:03:26.944 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 659 sec 2024-03-28T18:03:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:26 smithi071 ceph-mon[36597]: pgmap v500: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T18:03:27.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:27 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T18:03:27.944 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 660 sec 2024-03-28T18:03:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:27 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T18:03:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:27 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T18:03:28.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:28 smithi045 ceph-mon[31773]: pgmap v501: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:03:28.944 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 661 sec 2024-03-28T18:03:29.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:28 smithi071 ceph-mon[36597]: pgmap v501: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:03:29.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:28 smithi151 ceph-mon[36481]: pgmap v501: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:03:29.945 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 662 sec 2024-03-28T18:03:30.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:30 smithi045 ceph-mon[31773]: pgmap v502: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:03:30.944 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 663 sec 2024-03-28T18:03:31.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:30 smithi071 ceph-mon[36597]: pgmap v502: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:03:31.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:30 smithi151 ceph-mon[36481]: pgmap v502: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:03:31.945 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 664 sec 2024-03-28T18:03:32.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:32 smithi045 ceph-mon[31773]: pgmap v503: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:03:32.945 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 665 sec 2024-03-28T18:03:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:32 smithi071 ceph-mon[36597]: pgmap v503: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:03:33.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:32 smithi151 ceph-mon[36481]: pgmap v503: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:03:33.945 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 666 sec 2024-03-28T18:03:34.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:34 smithi045 ceph-mon[31773]: pgmap v504: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-28T18:03:34.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:34 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:03:34.945 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 667 sec 2024-03-28T18:03:35.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:34 smithi071 ceph-mon[36597]: pgmap v504: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-28T18:03:35.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:34 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:03:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:34 smithi151 ceph-mon[36481]: pgmap v504: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-28T18:03:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:34 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:03:35.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:35 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:35.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:35 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:03:35.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:35 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:03:35.946 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 668 sec 2024-03-28T18:03:36.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:35 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:36.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:35 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:03:36.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:35 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:03:36.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:35 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:03:36.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:35 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:03:36.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:35 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:03:36.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:36 smithi045 ceph-mon[31773]: pgmap v505: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 87 B/s wr, 5 op/s 2024-03-28T18:03:36.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:36 smithi045 ceph-mon[31773]: pgmap v506: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:03:36.945 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 669 sec 2024-03-28T18:03:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:36 smithi071 ceph-mon[36597]: pgmap v505: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 87 B/s wr, 5 op/s 2024-03-28T18:03:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:36 smithi071 ceph-mon[36597]: pgmap v506: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:03:37.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:36 smithi151 ceph-mon[36481]: pgmap v505: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 87 B/s wr, 5 op/s 2024-03-28T18:03:37.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:36 smithi151 ceph-mon[36481]: pgmap v506: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:03:37.946 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 670 sec 2024-03-28T18:03:38.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:38 smithi045 ceph-mon[31773]: pgmap v507: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:03:38.946 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 671 sec 2024-03-28T18:03:39.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:38 smithi071 ceph-mon[36597]: pgmap v507: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:03:39.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:38 smithi151 ceph-mon[36481]: pgmap v507: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:03:39.320 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:03:39.320 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:03:39.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.557+0000 7f8e07400640 1 -- 172.21.15.45:0/384055520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e00067c00 msgr2=0x7f8e000ff880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:39.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.557+0000 7f8e07400640 1 --2- 172.21.15.45:0/384055520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e00067c00 0x7f8e000ff880 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f8df4009900 tx=0x7f8df402bc30 comp rx=0 tx=0).stop 2024-03-28T18:03:39.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.557+0000 7f8e07400640 1 -- 172.21.15.45:0/384055520 shutdown_connections 2024-03-28T18:03:39.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.557+0000 7f8e07400640 1 --2- 172.21.15.45:0/384055520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e0010a740 0x7f8e0010cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.557+0000 7f8e07400640 1 --2- 172.21.15.45:0/384055520 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e001061b0 0x7f8e000ffdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.557+0000 7f8e07400640 1 --2- 172.21.15.45:0/384055520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e00067c00 0x7f8e000ff880 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.557+0000 7f8e07400640 1 -- 172.21.15.45:0/384055520 >> 172.21.15.45:0/384055520 conn(0x7f8e000766f0 msgr2=0x7f8e000fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:03:39.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.557+0000 7f8e07400640 1 -- 172.21.15.45:0/384055520 shutdown_connections 2024-03-28T18:03:39.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.557+0000 7f8e07400640 1 -- 172.21.15.45:0/384055520 wait complete. 2024-03-28T18:03:39.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.558+0000 7f8e07400640 1 Processor -- start 2024-03-28T18:03:39.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.559+0000 7f8e07400640 1 -- start start 2024-03-28T18:03:39.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.559+0000 7f8e07400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e00067c00 0x7f8e0019ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:39.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.559+0000 7f8e07400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e001061b0 0x7f8e0019f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:39.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.559+0000 7f8e07400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e0010a740 0x7f8e001093b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:39.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.559+0000 7f8e07400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e00111ae0 con 0x7f8e00067c00 2024-03-28T18:03:39.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.559+0000 7f8e07400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e00111960 con 0x7f8e001061b0 2024-03-28T18:03:39.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.559+0000 7f8e07400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e00111c60 con 0x7f8e0010a740 2024-03-28T18:03:39.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.559+0000 7f8dffe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e001061b0 0x7f8e0019f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:39.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.559+0000 7f8dffe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e001061b0 0x7f8e0019f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:49750/0 (socket says 172.21.15.45:49750) 2024-03-28T18:03:39.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.559+0000 7f8dffe00640 1 -- 172.21.15.45:0/2999495853 learned_addr learned my addr 172.21.15.45:0/2999495853 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:03:39.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.559+0000 7f8e04a00640 1 --2- 172.21.15.45:0/2999495853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e00067c00 0x7f8e0019ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:39.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.559+0000 7f8e05400640 1 --2- 172.21.15.45:0/2999495853 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e0010a740 0x7f8e001093b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:39.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.560+0000 7f8dffe00640 1 -- 172.21.15.45:0/2999495853 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e0010a740 msgr2=0x7f8e001093b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:39.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.560+0000 7f8dffe00640 1 --2- 172.21.15.45:0/2999495853 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e0010a740 0x7f8e001093b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.560+0000 7f8dffe00640 1 -- 172.21.15.45:0/2999495853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e00067c00 msgr2=0x7f8e0019ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:39.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.560+0000 7f8dffe00640 1 --2- 172.21.15.45:0/2999495853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e00067c00 0x7f8e0019ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.560+0000 7f8dffe00640 1 -- 172.21.15.45:0/2999495853 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e00109b30 con 0x7f8e001061b0 2024-03-28T18:03:39.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.560+0000 7f8dffe00640 1 --2- 172.21.15.45:0/2999495853 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e001061b0 0x7f8e0019f320 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f8df000e9c0 tx=0x7f8df000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:03:39.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.560+0000 7f8dfd600640 1 -- 172.21.15.45:0/2999495853 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df000ccc0 con 0x7f8e001061b0 2024-03-28T18:03:39.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.560+0000 7f8dfd600640 1 -- 172.21.15.45:0/2999495853 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8df0005500 con 0x7f8e001061b0 2024-03-28T18:03:39.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.560+0000 7f8dfd600640 1 -- 172.21.15.45:0/2999495853 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df0010690 con 0x7f8e001061b0 2024-03-28T18:03:39.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.560+0000 7f8e07400640 1 -- 172.21.15.45:0/2999495853 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e00109e20 con 0x7f8e001061b0 2024-03-28T18:03:39.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.560+0000 7f8e07400640 1 -- 172.21.15.45:0/2999495853 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e001a6d00 con 0x7f8e001061b0 2024-03-28T18:03:39.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.565+0000 7f8e0a176740 1 Processor -- start 2024-03-28T18:03:39.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.565+0000 7f8e0a176740 1 -- start start 2024-03-28T18:03:39.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.565+0000 7f8e0a176740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560ac173dc00 0x560ac173c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:39.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.565+0000 7f8dfd600640 1 -- 172.21.15.45:0/2999495853 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f8df0004e80 con 0x7f8e001061b0 2024-03-28T18:03:39.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.565+0000 7f8e0a176740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560ac173cf20 0x560ac173d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:39.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.565+0000 7f8e0a176740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac1746290 0x560ac1748680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:39.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.565+0000 7f8e0a176740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac16a1250 con 0x560ac173cf20 2024-03-28T18:03:39.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.565+0000 7f8e0a176740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac1649a30 con 0x560ac1746290 2024-03-28T18:03:39.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.565+0000 7f8e0a176740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac169d2f0 con 0x560ac173dc00 2024-03-28T18:03:39.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.566+0000 7f8dffe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac1746290 0x560ac1748680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:39.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.566+0000 7f8e04a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560ac173cf20 0x560ac173d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:39.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.566+0000 7f8e05400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560ac173dc00 0x560ac173c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:39.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.566+0000 7f8dffe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac1746290 0x560ac1748680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:49758/0 (socket says 172.21.15.45:49758) 2024-03-28T18:03:39.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.566+0000 7f8dffe00640 1 -- 172.21.15.45:0/3741877265 learned_addr learned my addr 172.21.15.45:0/3741877265 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:03:39.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.566+0000 7f8dffe00640 1 -- 172.21.15.45:0/3741877265 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560ac173dc00 msgr2=0x560ac173c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:03:39.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.566+0000 7f8dffe00640 1 --2- 172.21.15.45:0/3741877265 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560ac173dc00 0x560ac173c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.566+0000 7f8dffe00640 1 -- 172.21.15.45:0/3741877265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560ac173cf20 msgr2=0x560ac173d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:03:39.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.566+0000 7f8dffe00640 1 --2- 172.21.15.45:0/3741877265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560ac173cf20 0x560ac173d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.566+0000 7f8dffe00640 1 -- 172.21.15.45:0/3741877265 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ac1634690 con 0x560ac1746290 2024-03-28T18:03:39.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.567+0000 7f8dffe00640 1 --2- 172.21.15.45:0/3741877265 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac1746290 0x560ac1748680 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f8df00cc8d0 tx=0x7f8df00d82a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6e8dc73682292e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:03:39.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.567+0000 7f8dfd600640 1 --2- 172.21.15.45:0/2999495853 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8de80a29f0 0x7f8de80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:39.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.567+0000 7f8dedc00640 1 -- 172.21.15.45:0/3741877265 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df00e3070 con 0x560ac1746290 2024-03-28T18:03:39.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.567+0000 7f8dedc00640 1 -- 172.21.15.45:0/3741877265 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8df00d05a0 con 0x560ac1746290 2024-03-28T18:03:39.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.567+0000 7f8dedc00640 1 -- 172.21.15.45:0/3741877265 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df00de660 con 0x560ac1746290 2024-03-28T18:03:39.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.567+0000 7f8dfd600640 1 -- 172.21.15.45:0/2999495853 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 12382+0+0 (secure 0 0 0) 0x7f8df00d01b0 con 0x7f8e001061b0 2024-03-28T18:03:39.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.567+0000 7f8e05400640 1 --2- 172.21.15.45:0/2999495853 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8de80a29f0 0x7f8de80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:39.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.568+0000 7f8e0a176740 1 -- 172.21.15.45:0/3741877265 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac1746290 msgr2=0x560ac1748680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:39.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.568+0000 7f8e0a176740 1 --2- 172.21.15.45:0/3741877265 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac1746290 0x560ac1748680 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f8df00cc8d0 tx=0x7f8df00d82a0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.568+0000 7f8e0a176740 1 -- 172.21.15.45:0/3741877265 shutdown_connections 2024-03-28T18:03:39.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.568+0000 7f8e0a176740 1 --2- 172.21.15.45:0/3741877265 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac1746290 0x560ac1748680 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.568+0000 7f8e0a176740 1 --2- 172.21.15.45:0/3741877265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560ac173cf20 0x560ac173d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.568+0000 7f8e0a176740 1 --2- 172.21.15.45:0/3741877265 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560ac173dc00 0x560ac173c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.568+0000 7f8e0a176740 1 -- 172.21.15.45:0/3741877265 >> 172.21.15.45:0/3741877265 conn(0x560ac1704200 msgr2=0x560ac1737970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:03:39.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.568+0000 7f8e0a176740 1 -- 172.21.15.45:0/3741877265 shutdown_connections 2024-03-28T18:03:39.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.568+0000 7f8e0a176740 1 -- 172.21.15.45:0/3741877265 wait complete. 2024-03-28T18:03:39.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.568+0000 7f8e05400640 1 --2- 172.21.15.45:0/2999495853 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8de80a29f0 0x7f8de80a4eb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8e0006af10 tx=0x7f8df8009500 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:03:39.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.568+0000 7f8e0a176740 1 -- start start 2024-03-28T18:03:39.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.569+0000 7f8e0a176740 1 Processor -- start 2024-03-28T18:03:39.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.569+0000 7f8e0a176740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560ac173cf20 0x560ac1745950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:39.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.569+0000 7f8e0a176740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560ac173dc00 0x560ac1742ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:39.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.569+0000 7f8e0a176740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac1746290 0x560ac1743220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:39.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.569+0000 7f8e0a176740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac1749100 con 0x560ac173dc00 2024-03-28T18:03:39.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.569+0000 7f8e0a176740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac165ce80 con 0x560ac1746290 2024-03-28T18:03:39.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.569+0000 7f8e0a176740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac1749280 con 0x560ac173cf20 2024-03-28T18:03:39.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.569+0000 7f8e0a176740 10 client.0 _ll_register_callbacks cb 0x560ac166f130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:03:39.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.569+0000 7f8dffe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560ac173dc00 0x560ac1742ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:39.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.569+0000 7f8e05400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac1746290 0x560ac1743220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:39.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.569+0000 7f8dffe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560ac173dc00 0x560ac1742ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48986/0 (socket says 172.21.15.45:48986) 2024-03-28T18:03:39.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.569+0000 7f8dffe00640 1 -- 172.21.15.45:0/264003460 learned_addr learned my addr 172.21.15.45:0/264003460 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:03:39.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.569+0000 7f8e04a00640 1 --2- 172.21.15.45:0/264003460 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560ac173cf20 0x560ac1745950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:39.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.570+0000 7f8e05400640 1 -- 172.21.15.45:0/264003460 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560ac173cf20 msgr2=0x560ac1745950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:03:39.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.570+0000 7f8e05400640 1 --2- 172.21.15.45:0/264003460 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560ac173cf20 0x560ac1745950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.570+0000 7f8e05400640 1 -- 172.21.15.45:0/264003460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560ac173dc00 msgr2=0x560ac1742ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:03:39.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.570+0000 7f8e05400640 1 --2- 172.21.15.45:0/264003460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560ac173dc00 0x560ac1742ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.570+0000 7f8e05400640 1 -- 172.21.15.45:0/264003460 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560ac147f7d0 con 0x560ac1746290 2024-03-28T18:03:39.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.570+0000 7f8e05400640 1 --2- 172.21.15.45:0/264003460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac1746290 0x560ac1743220 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8df801f650 tx=0x7f8df8004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45c5c471deb22b79 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:03:39.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.570+0000 7f8de2000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T18:03:39.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.570+0000 7f8de2000640 1 -- 172.21.15.45:0/264003460 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df80118d0 con 0x560ac1746290 2024-03-28T18:03:39.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.570+0000 7f8e0a176740 1 -- 172.21.15.45:0/264003460 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8df00d05a0 con 0x560ac1746290 2024-03-28T18:03:39.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.570+0000 7f8de2000640 1 -- 172.21.15.45:0/264003460 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(147..147 src has 1..147) v4 ==== 12382+0+0 (secure 0 0 0) 0x7f8df8017030 con 0x560ac1746290 2024-03-28T18:03:39.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.571+0000 7f8de2000640 1 -- 172.21.15.45:0/264003460 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=319) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8df8011a70 con 0x560ac1746290 2024-03-28T18:03:39.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.571+0000 7f8e0a176740 10 client.26591 fetch_fsmap learned FSMap version 319 2024-03-28T18:03:39.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.571+0000 7f8e0a176740 1 -- 172.21.15.45:0/264003460 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=319,monmap=4+}) v3 -- 0x560ac13b53f0 con 0x560ac1746290 2024-03-28T18:03:39.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.571+0000 7f8de2000640 1 -- 172.21.15.45:0/264003460 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8df00d05a0 con 0x560ac1746290 2024-03-28T18:03:39.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.571+0000 7f8de2000640 1 -- 172.21.15.45:0/264003460 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 319) v1 ==== 3905+0+0 (secure 0 0 0) 0x7f8df8032020 con 0x560ac1746290 2024-03-28T18:03:39.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.571+0000 7f8e0a176740 10 client.26591 fetch_fsmap finished waiting for FSMap version 319 2024-03-28T18:03:39.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.571+0000 7f8e0a176740 10 client.26591 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:03:39.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.571+0000 7f8e0a176740 20 client.26591 populate_metadata read hostname 'smithi045' 2024-03-28T18:03:39.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.571+0000 7f8e0a176740 1 --2- 172.21.15.45:0/264003460 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x560ac1756580 0x560ac17589a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:39.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.571+0000 7f8e0a176740 4 client.26591 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:03:39.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.571+0000 7f8e0a176740 1 -- 172.21.15.45:0/264003460 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560ac16ae790 con 0x560ac1756580 2024-03-28T18:03:39.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.572+0000 7f8e04a00640 1 --2- 172.21.15.45:0/264003460 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x560ac1756580 0x560ac17589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:39.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.572+0000 7f8e04a00640 1 --2- 172.21.15.45:0/264003460 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x560ac1756580 0x560ac17589a0 crc :-1 s=READY pgs=415 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43f17079b059fbe9 server_cookie=b3a643fd82dd253e in_seq=0 out_seq=0 2024-03-28T18:03:39.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.572+0000 7f8de2000640 10 client.26591 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:03:39.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.573+0000 7f8de2000640 1 -- 172.21.15.45:0/264003460 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560ac16ae790 con 0x560ac1756580 2024-03-28T18:03:39.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.573+0000 7f8de2000640 10 client.26591 handle_command_reply: tid=0 2024-03-28T18:03:39.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.573+0000 7f8e0a176740 1 client.26591 shutdown 2024-03-28T18:03:39.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.573+0000 7f8e0a176740 1 -- 172.21.15.45:0/264003460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac1746290 msgr2=0x560ac1743220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:39.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.573+0000 7f8e0a176740 1 --2- 172.21.15.45:0/264003460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac1746290 0x560ac1743220 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8df801f650 tx=0x7f8df8004ab0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.573+0000 7f8e0a176740 1 -- 172.21.15.45:0/264003460 shutdown_connections 2024-03-28T18:03:39.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.573+0000 7f8e0a176740 1 --2- 172.21.15.45:0/264003460 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x560ac1756580 0x560ac17589a0 crc :-1 s=READY pgs=415 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.573+0000 7f8e0a176740 1 --2- 172.21.15.45:0/264003460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac1746290 0x560ac1743220 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.573+0000 7f8e0a176740 1 --2- 172.21.15.45:0/264003460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560ac173dc00 0x560ac1742ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.573+0000 7f8e0a176740 1 --2- 172.21.15.45:0/264003460 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560ac173cf20 0x560ac1745950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.573+0000 7f8e0a176740 1 -- 172.21.15.45:0/264003460 >> 172.21.15.45:0/264003460 conn(0x560ac1704200 msgr2=0x560ac17396c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:03:39.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.573+0000 7f8de2000640 0 client.26591 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:03:39.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.573+0000 7f8e0a176740 1 -- 172.21.15.45:0/264003460 shutdown_connections 2024-03-28T18:03:39.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.573+0000 7f8e0a176740 1 -- 172.21.15.45:0/264003460 wait complete. 2024-03-28T18:03:39.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.573+0000 7f8e0a176740 20 client.26591 trim_cache size 0 max 16384 2024-03-28T18:03:39.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.591+0000 7f8e0a176740 1 Processor -- start 2024-03-28T18:03:39.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.591+0000 7f8e0a176740 1 -- start start 2024-03-28T18:03:39.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.591+0000 7f8e0a176740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac173cf20 0x560ac1745410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:39.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.591+0000 7f8e0a176740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560ac173dc00 0x560ac1745950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:39.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.591+0000 7f8e0a176740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560ac1746290 0x560ac1747680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:39.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.591+0000 7f8e0a176740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e001b1900 con 0x560ac173dc00 2024-03-28T18:03:39.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.591+0000 7f8e0a176740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e001aff80 con 0x560ac173cf20 2024-03-28T18:03:39.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.591+0000 7f8e0a176740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e001b12a0 con 0x560ac1746290 2024-03-28T18:03:39.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.592+0000 7f8e04a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac173cf20 0x560ac1745410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:39.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.592+0000 7f8e04a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac173cf20 0x560ac1745410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:49774/0 (socket says 172.21.15.45:49774) 2024-03-28T18:03:39.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.592+0000 7f8e04a00640 1 -- 172.21.15.45:0/1879969592 learned_addr learned my addr 172.21.15.45:0/1879969592 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:03:39.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.592+0000 7f8dffe00640 1 --2- 172.21.15.45:0/1879969592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560ac173dc00 0x560ac1745950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:39.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.592+0000 7f8e05400640 1 --2- 172.21.15.45:0/1879969592 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560ac1746290 0x560ac1747680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:39.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.592+0000 7f8e04a00640 1 -- 172.21.15.45:0/1879969592 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560ac1746290 msgr2=0x560ac1747680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:03:39.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.592+0000 7f8e04a00640 1 --2- 172.21.15.45:0/1879969592 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560ac1746290 0x560ac1747680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.592+0000 7f8e04a00640 1 -- 172.21.15.45:0/1879969592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560ac173dc00 msgr2=0x560ac1745950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:03:39.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.592+0000 7f8e04a00640 1 --2- 172.21.15.45:0/1879969592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560ac173dc00 0x560ac1745950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.592+0000 7f8e04a00640 1 -- 172.21.15.45:0/1879969592 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ac13750f0 con 0x560ac173cf20 2024-03-28T18:03:39.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.592+0000 7f8e05400640 1 --2- 172.21.15.45:0/1879969592 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560ac1746290 0x560ac1747680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:03:39.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.592+0000 7f8e04a00640 1 --2- 172.21.15.45:0/1879969592 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac173cf20 0x560ac1745410 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f8df4009230 tx=0x7f8df402df90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=622e7f86acb45d55 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:03:39.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.593+0000 7f8ded200640 1 -- 172.21.15.45:0/1879969592 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df4004040 con 0x560ac173cf20 2024-03-28T18:03:39.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.593+0000 7f8ded200640 1 -- 172.21.15.45:0/1879969592 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8df4051030 con 0x560ac173cf20 2024-03-28T18:03:39.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.593+0000 7f8ded200640 1 -- 172.21.15.45:0/1879969592 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df4004b20 con 0x560ac173cf20 2024-03-28T18:03:39.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.593+0000 7f8e0a176740 1 -- 172.21.15.45:0/1879969592 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac173cf20 msgr2=0x560ac1745410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:39.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.593+0000 7f8e0a176740 1 --2- 172.21.15.45:0/1879969592 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac173cf20 0x560ac1745410 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f8df4009230 tx=0x7f8df402df90 comp rx=0 tx=0).stop 2024-03-28T18:03:39.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.593+0000 7f8e0a176740 1 -- 172.21.15.45:0/1879969592 shutdown_connections 2024-03-28T18:03:39.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.593+0000 7f8e0a176740 1 --2- 172.21.15.45:0/1879969592 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560ac1746290 0x560ac1747680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.593+0000 7f8e0a176740 1 --2- 172.21.15.45:0/1879969592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560ac173dc00 0x560ac1745950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.593+0000 7f8e0a176740 1 --2- 172.21.15.45:0/1879969592 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac173cf20 0x560ac1745410 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.593+0000 7f8e0a176740 1 -- 172.21.15.45:0/1879969592 >> 172.21.15.45:0/1879969592 conn(0x560ac1704200 msgr2=0x560ac17374a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:03:39.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.593+0000 7f8e0a176740 1 -- 172.21.15.45:0/1879969592 shutdown_connections 2024-03-28T18:03:39.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.593+0000 7f8e0a176740 1 -- 172.21.15.45:0/1879969592 wait complete. 2024-03-28T18:03:39.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.594+0000 7f8e0a176740 1 -- start start 2024-03-28T18:03:39.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.594+0000 7f8e0a176740 1 Processor -- start 2024-03-28T18:03:39.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.594+0000 7f8e0a176740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac173cf20 0x560ac173f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:39.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.595+0000 7f8e0a176740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560ac173dc00 0x560ac173fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:39.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.595+0000 7f8e0a176740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560ac1746290 0x560ac173ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:39.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.595+0000 7f8e0a176740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e001b0c40 con 0x560ac1746290 2024-03-28T18:03:39.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.595+0000 7f8e0a176740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e001b05e0 con 0x560ac173cf20 2024-03-28T18:03:39.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.595+0000 7f8e0a176740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac1742ad0 con 0x560ac173dc00 2024-03-28T18:03:39.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.595+0000 7f8e0a176740 10 client.0 _ll_register_callbacks cb 0x7f8df8003f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:03:39.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.595+0000 7f8e04a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac173cf20 0x560ac173f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:39.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.595+0000 7f8e04a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac173cf20 0x560ac173f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:49784/0 (socket says 172.21.15.45:49784) 2024-03-28T18:03:39.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.595+0000 7f8e04a00640 1 -- 172.21.15.45:0/1554717910 learned_addr learned my addr 172.21.15.45:0/1554717910 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:03:39.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.595+0000 7f8e05400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560ac1746290 0x560ac173ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:39.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.595+0000 7f8dffe00640 1 --2- 172.21.15.45:0/1554717910 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560ac173dc00 0x560ac173fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:39.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.595+0000 7f8e04a00640 1 -- 172.21.15.45:0/1554717910 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560ac173dc00 msgr2=0x560ac173fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:03:39.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.595+0000 7f8e04a00640 1 --2- 172.21.15.45:0/1554717910 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560ac173dc00 0x560ac173fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.595+0000 7f8e04a00640 1 -- 172.21.15.45:0/1554717910 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560ac1746290 msgr2=0x560ac173ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:03:39.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.595+0000 7f8e04a00640 1 --2- 172.21.15.45:0/1554717910 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560ac1746290 0x560ac173ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.595+0000 7f8e04a00640 1 -- 172.21.15.45:0/1554717910 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560ac17b1020 con 0x560ac173cf20 2024-03-28T18:03:39.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.596+0000 7f8e04a00640 1 --2- 172.21.15.45:0/1554717910 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac173cf20 0x560ac173f530 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f8df4036040 tx=0x7f8df40048a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ba3bae7c5fb2a41 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:03:39.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.596+0000 7f8de2000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T18:03:39.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.596+0000 7f8de2000640 1 -- 172.21.15.45:0/1554717910 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df402c880 con 0x560ac173cf20 2024-03-28T18:03:39.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.596+0000 7f8e0a176740 1 -- 172.21.15.45:0/1554717910 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8df4051030 con 0x560ac173cf20 2024-03-28T18:03:39.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.596+0000 7f8de2000640 1 -- 172.21.15.45:0/1554717910 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(147..147 src has 1..147) v4 ==== 12382+0+0 (secure 0 0 0) 0x7f8df402ca20 con 0x560ac173cf20 2024-03-28T18:03:39.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.596+0000 7f8de2000640 1 -- 172.21.15.45:0/1554717910 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=319) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8df4069050 con 0x560ac173cf20 2024-03-28T18:03:39.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.596+0000 7f8e0a176740 10 client.26597 fetch_fsmap learned FSMap version 319 2024-03-28T18:03:39.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.596+0000 7f8e0a176740 1 -- 172.21.15.45:0/1554717910 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=319,monmap=4+}) v3 -- 0x560ac17b2260 con 0x560ac173cf20 2024-03-28T18:03:39.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.596+0000 7f8de2000640 1 -- 172.21.15.45:0/1554717910 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8df4051030 con 0x560ac173cf20 2024-03-28T18:03:39.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.596+0000 7f8de2000640 1 -- 172.21.15.45:0/1554717910 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 319) v1 ==== 3905+0+0 (secure 0 0 0) 0x7f8df40053a0 con 0x560ac173cf20 2024-03-28T18:03:39.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.596+0000 7f8e0a176740 10 client.26597 fetch_fsmap finished waiting for FSMap version 319 2024-03-28T18:03:39.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.596+0000 7f8e0a176740 10 client.26597 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:03:39.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.596+0000 7f8e0a176740 20 client.26597 populate_metadata read hostname 'smithi045' 2024-03-28T18:03:39.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.597+0000 7f8e0a176740 1 --2- 172.21.15.45:0/1554717910 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x560ac17b2870 0x560ac17b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:03:39.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.597+0000 7f8e0a176740 4 client.26597 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:03:39.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.597+0000 7f8e0a176740 1 -- 172.21.15.45:0/1554717910 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560ac16a8ae0 con 0x560ac17b2870 2024-03-28T18:03:39.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.597+0000 7f8dffe00640 1 --2- 172.21.15.45:0/1554717910 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x560ac17b2870 0x560ac17b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:03:39.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.597+0000 7f8dffe00640 1 --2- 172.21.15.45:0/1554717910 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x560ac17b2870 0x560ac17b4c90 crc :-1 s=READY pgs=416 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7389c044e336a1b4 server_cookie=7271a4913c11c432 in_seq=0 out_seq=0 2024-03-28T18:03:39.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.597+0000 7f8de2000640 10 client.26597 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:03:39.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.597+0000 7f8de2000640 1 -- 172.21.15.45:0/1554717910 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x560ac16a8ae0 con 0x560ac17b2870 2024-03-28T18:03:39.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.597+0000 7f8de2000640 10 client.26597 handle_command_reply: tid=0 2024-03-28T18:03:39.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.597+0000 7f8e0a176740 1 client.26597 shutdown 2024-03-28T18:03:39.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.598+0000 7f8e0a176740 1 -- 172.21.15.45:0/1554717910 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac173cf20 msgr2=0x560ac173f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:39.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.598+0000 7f8e0a176740 1 --2- 172.21.15.45:0/1554717910 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac173cf20 0x560ac173f530 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f8df4036040 tx=0x7f8df40048a0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.598+0000 7f8e0a176740 1 -- 172.21.15.45:0/1554717910 shutdown_connections 2024-03-28T18:03:39.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.598+0000 7f8e0a176740 1 --2- 172.21.15.45:0/1554717910 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x560ac17b2870 0x560ac17b4c90 crc :-1 s=READY pgs=416 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.598+0000 7f8e0a176740 1 --2- 172.21.15.45:0/1554717910 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560ac1746290 0x560ac173ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.598+0000 7f8e0a176740 1 --2- 172.21.15.45:0/1554717910 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560ac173dc00 0x560ac173fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.598+0000 7f8de2000640 0 client.26597 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:03:39.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.598+0000 7f8e0a176740 1 --2- 172.21.15.45:0/1554717910 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560ac173cf20 0x560ac173f530 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.598+0000 7f8e0a176740 1 -- 172.21.15.45:0/1554717910 >> 172.21.15.45:0/1554717910 conn(0x560ac1704200 msgr2=0x560ac17414f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:03:39.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.598+0000 7f8e0a176740 1 -- 172.21.15.45:0/1554717910 shutdown_connections 2024-03-28T18:03:39.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.598+0000 7f8e0a176740 1 -- 172.21.15.45:0/1554717910 wait complete. 2024-03-28T18:03:39.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.598+0000 7f8e0a176740 20 client.26597 trim_cache size 0 max 16384 2024-03-28T18:03:39.603 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T18:03:39.603 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:03:39.603 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T18:03:39.603 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T18:03:39.604 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T18:03:39.604 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T18:03:39.604 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T18:03:39.604 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:03:39.604 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:03:39.604 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T18:03:39.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.599+0000 7f8e07400640 1 -- 172.21.15.45:0/2999495853 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8de80a29f0 msgr2=0x7f8de80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:39.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.599+0000 7f8e07400640 1 --2- 172.21.15.45:0/2999495853 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8de80a29f0 0x7f8de80a4eb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8e0006af10 tx=0x7f8df8009500 comp rx=0 tx=0).stop 2024-03-28T18:03:39.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.599+0000 7f8e07400640 1 -- 172.21.15.45:0/2999495853 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e001061b0 msgr2=0x7f8e0019f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:03:39.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.599+0000 7f8e07400640 1 --2- 172.21.15.45:0/2999495853 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e001061b0 0x7f8e0019f320 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f8df000e9c0 tx=0x7f8df000ee80 comp rx=0 tx=0).stop 2024-03-28T18:03:39.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.599+0000 7f8e07400640 1 -- 172.21.15.45:0/2999495853 shutdown_connections 2024-03-28T18:03:39.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.599+0000 7f8e07400640 1 --2- 172.21.15.45:0/2999495853 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8de80a29f0 0x7f8de80a4eb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.599+0000 7f8e07400640 1 --2- 172.21.15.45:0/2999495853 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e0010a740 0x7f8e001093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.599+0000 7f8e07400640 1 --2- 172.21.15.45:0/2999495853 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e001061b0 0x7f8e0019f320 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.599+0000 7f8e07400640 1 --2- 172.21.15.45:0/2999495853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e00067c00 0x7f8e0019ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:03:39.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.599+0000 7f8e07400640 1 -- 172.21.15.45:0/2999495853 >> 172.21.15.45:0/2999495853 conn(0x7f8e000766f0 msgr2=0x7f8e0010e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:03:39.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.599+0000 7f8e07400640 1 -- 172.21.15.45:0/2999495853 shutdown_connections 2024-03-28T18:03:39.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:03:39.599+0000 7f8e07400640 1 -- 172.21.15.45:0/2999495853 wait complete. 2024-03-28T18:03:39.616 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T18:03:39.946 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 672 sec 2024-03-28T18:03:40.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:40 smithi045 ceph-mon[31773]: pgmap v508: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:03:40.946 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 673 sec 2024-03-28T18:03:41.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:40 smithi071 ceph-mon[36597]: pgmap v508: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:03:41.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:40 smithi151 ceph-mon[36481]: pgmap v508: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:03:41.947 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 674 sec 2024-03-28T18:03:42.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:42 smithi045 ceph-mon[31773]: pgmap v509: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:03:42.946 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 675 sec 2024-03-28T18:03:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:42 smithi071 ceph-mon[36597]: pgmap v509: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:03:43.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:42 smithi151 ceph-mon[36481]: pgmap v509: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:03:43.947 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 676 sec 2024-03-28T18:03:44.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:44 smithi045 ceph-mon[31773]: pgmap v510: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:03:44.947 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 677 sec 2024-03-28T18:03:45.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:44 smithi071 ceph-mon[36597]: pgmap v510: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:03:45.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:44 smithi151 ceph-mon[36481]: pgmap v510: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:03:45.947 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 678 sec 2024-03-28T18:03:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:46 smithi045 ceph-mon[31773]: pgmap v511: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:03:46.947 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 679 sec 2024-03-28T18:03:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:46 smithi071 ceph-mon[36597]: pgmap v511: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:03:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:46 smithi151 ceph-mon[36481]: pgmap v511: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:03:47.947 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 680 sec 2024-03-28T18:03:48.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:48 smithi045 ceph-mon[31773]: pgmap v512: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:03:48.947 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 681 sec 2024-03-28T18:03:49.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:48 smithi071 ceph-mon[36597]: pgmap v512: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:03:49.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:48 smithi151 ceph-mon[36481]: pgmap v512: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:03:49.948 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 682 sec 2024-03-28T18:03:50.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:50 smithi045 ceph-mon[31773]: pgmap v513: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:03:50.947 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 683 sec 2024-03-28T18:03:51.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:50 smithi071 ceph-mon[36597]: pgmap v513: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:03:51.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:50 smithi151 ceph-mon[36481]: pgmap v513: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:03:51.947 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 684 sec 2024-03-28T18:03:52.948 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 685 sec 2024-03-28T18:03:53.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:52 smithi071 ceph-mon[36597]: pgmap v514: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:03:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:52 smithi151 ceph-mon[36481]: pgmap v514: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:03:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:52 smithi045 ceph-mon[31773]: pgmap v514: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:03:53.948 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 686 sec 2024-03-28T18:03:54.948 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 687 sec 2024-03-28T18:03:55.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:54 smithi071 ceph-mon[36597]: pgmap v515: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:03:55.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:54 smithi151 ceph-mon[36481]: pgmap v515: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:03:55.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:54 smithi045 ceph-mon[31773]: pgmap v515: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:03:55.948 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 688 sec 2024-03-28T18:03:56.948 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 689 sec 2024-03-28T18:03:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:56 smithi071 ceph-mon[36597]: pgmap v516: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:03:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:56 smithi151 ceph-mon[36481]: pgmap v516: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:03:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:56 smithi045 ceph-mon[31773]: pgmap v516: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:03:57.948 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 690 sec 2024-03-28T18:03:58.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:03:58 smithi045 ceph-mon[31773]: pgmap v517: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:03:58.948 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 691 sec 2024-03-28T18:03:59.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:03:58 smithi071 ceph-mon[36597]: pgmap v517: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:03:59.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:03:58 smithi151 ceph-mon[36481]: pgmap v517: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:03:59.949 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 692 sec 2024-03-28T18:04:00.949 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 693 sec 2024-03-28T18:04:01.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:00 smithi071 ceph-mon[36597]: pgmap v518: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:01.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:00 smithi151 ceph-mon[36481]: pgmap v518: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:01.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:00 smithi045 ceph-mon[31773]: pgmap v518: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:01.165 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:04:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:04:00.674+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:04:01.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:04:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:04:00.674+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:04:01.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:04:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:04:00.674+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:04:01.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:04:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:04:00.674+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:04:01.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:04:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:04:00.674+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:04:01.949 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 694 sec 2024-03-28T18:04:02.949 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 695 sec 2024-03-28T18:04:03.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:02 smithi071 ceph-mon[36597]: pgmap v519: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:04:03.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:02 smithi151 ceph-mon[36481]: pgmap v519: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:04:03.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:02 smithi045 ceph-mon[31773]: pgmap v519: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:04:03.949 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 696 sec 2024-03-28T18:04:04.949 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 697 sec 2024-03-28T18:04:05.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:04 smithi071 ceph-mon[36597]: pgmap v520: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:05.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:04 smithi151 ceph-mon[36481]: pgmap v520: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:05.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:04 smithi045 ceph-mon[31773]: pgmap v520: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:05.949 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 698 sec 2024-03-28T18:04:06.950 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 699 sec 2024-03-28T18:04:07.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:06 smithi071 ceph-mon[36597]: pgmap v521: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:04:07.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:06 smithi151 ceph-mon[36481]: pgmap v521: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:04:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:06 smithi045 ceph-mon[31773]: pgmap v521: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:04:07.950 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 700 sec 2024-03-28T18:04:08.950 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 701 sec 2024-03-28T18:04:09.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:08 smithi071 ceph-mon[36597]: pgmap v522: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:04:09.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:08 smithi151 ceph-mon[36481]: pgmap v522: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:04:09.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:08 smithi045 ceph-mon[31773]: pgmap v522: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:04:09.617 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:04:09.618 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:04:09.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.852+0000 7f692b200640 1 -- 172.21.15.45:0/1497351990 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6924107eb0 msgr2=0x7f6924108290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:09.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.852+0000 7f692b200640 1 --2- 172.21.15.45:0/1497351990 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6924107eb0 0x7f6924108290 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f6914009900 tx=0x7f691402bc30 comp rx=0 tx=0).stop 2024-03-28T18:04:09.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.852+0000 7f692b200640 1 -- 172.21.15.45:0/1497351990 shutdown_connections 2024-03-28T18:04:09.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.852+0000 7f692b200640 1 --2- 172.21.15.45:0/1497351990 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f692410cf80 0x7f6924114b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.852+0000 7f692b200640 1 --2- 172.21.15.45:0/1497351990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6924108860 0x7f692410c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.852+0000 7f692b200640 1 --2- 172.21.15.45:0/1497351990 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6924107eb0 0x7f6924108290 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.852+0000 7f692b200640 1 -- 172.21.15.45:0/1497351990 >> 172.21.15.45:0/1497351990 conn(0x7f6924075be0 msgr2=0x7f6924078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:04:09.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.852+0000 7f692b200640 1 -- 172.21.15.45:0/1497351990 shutdown_connections 2024-03-28T18:04:09.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.852+0000 7f692b200640 1 -- 172.21.15.45:0/1497351990 wait complete. 2024-03-28T18:04:09.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.854+0000 7f692b200640 1 Processor -- start 2024-03-28T18:04:09.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.854+0000 7f692b200640 1 -- start start 2024-03-28T18:04:09.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.854+0000 7f692b200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6924107eb0 0x7f692419e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:09.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.854+0000 7f692b200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f692410cf80 0x7f692419eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:09.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.854+0000 7f692b200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69241a2e40 0x7f69241a32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:09.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.854+0000 7f692b200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6924118080 con 0x7f692410cf80 2024-03-28T18:04:09.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.854+0000 7f692b200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6924117f00 con 0x7f69241a2e40 2024-03-28T18:04:09.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.854+0000 7f692b200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6924118200 con 0x7f6924107eb0 2024-03-28T18:04:09.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.855+0000 7f6929200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69241a2e40 0x7f69241a32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:09.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.855+0000 7f6923e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6924107eb0 0x7f692419e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:09.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.855+0000 7f6929200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69241a2e40 0x7f69241a32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:44138/0 (socket says 172.21.15.45:44138) 2024-03-28T18:04:09.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.855+0000 7f6929200640 1 -- 172.21.15.45:0/3289226894 learned_addr learned my addr 172.21.15.45:0/3289226894 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:04:09.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.855+0000 7f6923e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6924107eb0 0x7f692419e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.45:59024/0 (socket says 172.21.15.45:59024) 2024-03-28T18:04:09.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.855+0000 7f691be00640 1 --2- 172.21.15.45:0/3289226894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f692410cf80 0x7f692419eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:09.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.855+0000 7f6929200640 1 -- 172.21.15.45:0/3289226894 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6924107eb0 msgr2=0x7f692419e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:09.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.855+0000 7f6929200640 1 --2- 172.21.15.45:0/3289226894 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6924107eb0 0x7f692419e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.855+0000 7f6929200640 1 -- 172.21.15.45:0/3289226894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f692410cf80 msgr2=0x7f692419eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:09.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.855+0000 7f6929200640 1 --2- 172.21.15.45:0/3289226894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f692410cf80 0x7f692419eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.855+0000 7f6929200640 1 -- 172.21.15.45:0/3289226894 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69241a3930 con 0x7f69241a2e40 2024-03-28T18:04:09.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.855+0000 7f6929200640 1 --2- 172.21.15.45:0/3289226894 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69241a2e40 0x7f69241a32f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f691c00eaa0 tx=0x7f691c00ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:04:09.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.856+0000 7f6921600640 1 -- 172.21.15.45:0/3289226894 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c00cdb0 con 0x7f69241a2e40 2024-03-28T18:04:09.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.856+0000 7f6921600640 1 -- 172.21.15.45:0/3289226894 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f691c005020 con 0x7f69241a2e40 2024-03-28T18:04:09.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.856+0000 7f6921600640 1 -- 172.21.15.45:0/3289226894 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c0116b0 con 0x7f69241a2e40 2024-03-28T18:04:09.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.856+0000 7f692b200640 1 -- 172.21.15.45:0/3289226894 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69241a3c20 con 0x7f69241a2e40 2024-03-28T18:04:09.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.856+0000 7f692b200640 1 -- 172.21.15.45:0/3289226894 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69241ab460 con 0x7f69241a2e40 2024-03-28T18:04:09.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.860+0000 7f6921600640 1 -- 172.21.15.45:0/3289226894 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f691c005560 con 0x7f69241a2e40 2024-03-28T18:04:09.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.860+0000 7f692dfe0740 1 Processor -- start 2024-03-28T18:04:09.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.861+0000 7f692dfe0740 1 -- start start 2024-03-28T18:04:09.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.861+0000 7f692dfe0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e9c00 0x55752a0e8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:09.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.861+0000 7f692dfe0740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0e8f20 0x55752a0e93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:09.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.861+0000 7f692dfe0740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55752a0f2290 0x55752a0f4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:09.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.861+0000 7f692dfe0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55752a04d250 con 0x55752a0e9c00 2024-03-28T18:04:09.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.861+0000 7f692dfe0740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x557529ff5a30 con 0x55752a0e8f20 2024-03-28T18:04:09.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.861+0000 7f692dfe0740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55752a0492f0 con 0x55752a0f2290 2024-03-28T18:04:09.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.861+0000 7f6923e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0e8f20 0x55752a0e93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:09.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.861+0000 7f6923e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0e8f20 0x55752a0e93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:44144/0 (socket says 172.21.15.45:44144) 2024-03-28T18:04:09.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.861+0000 7f6923e00640 1 -- 172.21.15.45:0/3865724596 learned_addr learned my addr 172.21.15.45:0/3865724596 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:04:09.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.861+0000 7f691be00640 1 --2- 172.21.15.45:0/3865724596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e9c00 0x55752a0e8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:09.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.862+0000 7f6923e00640 1 -- 172.21.15.45:0/3865724596 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55752a0f2290 msgr2=0x55752a0f4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:04:09.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.862+0000 7f6923e00640 1 --2- 172.21.15.45:0/3865724596 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55752a0f2290 0x55752a0f4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.862+0000 7f6923e00640 1 -- 172.21.15.45:0/3865724596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e9c00 msgr2=0x55752a0e8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:04:09.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.862+0000 7f6923e00640 1 --2- 172.21.15.45:0/3865724596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e9c00 0x55752a0e8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.862+0000 7f6923e00640 1 -- 172.21.15.45:0/3865724596 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557529fe0690 con 0x55752a0e8f20 2024-03-28T18:04:09.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.862+0000 7f6921600640 1 --2- 172.21.15.45:0/3289226894 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f69080a29f0 0x7f69080a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:09.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.862+0000 7f6923e00640 1 --2- 172.21.15.45:0/3865724596 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0e8f20 0x55752a0e93a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6914009900 tx=0x7f6914009610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29fec8680433d470 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:04:09.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.862+0000 7f6921600640 1 -- 172.21.15.45:0/3289226894 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 12382+0+0 (secure 0 0 0) 0x7f691c0cff00 con 0x7f69241a2e40 2024-03-28T18:04:09.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.862+0000 7f6919600640 1 -- 172.21.15.45:0/3865724596 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6914045070 con 0x55752a0e8f20 2024-03-28T18:04:09.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.862+0000 7f6919600640 1 -- 172.21.15.45:0/3865724596 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f691402c880 con 0x55752a0e8f20 2024-03-28T18:04:09.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.862+0000 7f6919600640 1 -- 172.21.15.45:0/3865724596 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6914004b90 con 0x55752a0e8f20 2024-03-28T18:04:09.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.862+0000 7f691be00640 1 --2- 172.21.15.45:0/3289226894 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f69080a29f0 0x7f69080a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:09.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.863+0000 7f692dfe0740 1 -- 172.21.15.45:0/3865724596 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0e8f20 msgr2=0x55752a0e93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:09.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.863+0000 7f692dfe0740 1 --2- 172.21.15.45:0/3865724596 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0e8f20 0x55752a0e93a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6914009900 tx=0x7f6914009610 comp rx=0 tx=0).stop 2024-03-28T18:04:09.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.863+0000 7f691be00640 1 --2- 172.21.15.45:0/3289226894 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f69080a29f0 0x7f69080a4eb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6924107010 tx=0x7f690c009820 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:04:09.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.863+0000 7f692dfe0740 1 -- 172.21.15.45:0/3865724596 shutdown_connections 2024-03-28T18:04:09.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.863+0000 7f692dfe0740 1 --2- 172.21.15.45:0/3865724596 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55752a0f2290 0x55752a0f4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.863+0000 7f692dfe0740 1 --2- 172.21.15.45:0/3865724596 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0e8f20 0x55752a0e93a0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.863+0000 7f692dfe0740 1 --2- 172.21.15.45:0/3865724596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e9c00 0x55752a0e8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.863+0000 7f692dfe0740 1 -- 172.21.15.45:0/3865724596 >> 172.21.15.45:0/3865724596 conn(0x55752a0b0200 msgr2=0x55752a0e3970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:04:09.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.863+0000 7f692dfe0740 1 -- 172.21.15.45:0/3865724596 shutdown_connections 2024-03-28T18:04:09.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.863+0000 7f692dfe0740 1 -- 172.21.15.45:0/3865724596 wait complete. 2024-03-28T18:04:09.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.864+0000 7f692dfe0740 1 -- start start 2024-03-28T18:04:09.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.864+0000 7f692dfe0740 1 Processor -- start 2024-03-28T18:04:09.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.864+0000 7f692dfe0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e8f20 0x55752a0f18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:09.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.864+0000 7f692dfe0740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0e9c00 0x55752a0eec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:09.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.864+0000 7f692dfe0740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55752a0f2290 0x55752a0ef1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:09.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.864+0000 7f692dfe0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55752a0f50a0 con 0x55752a0e8f20 2024-03-28T18:04:09.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.864+0000 7f692dfe0740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55752a008e80 con 0x55752a0e9c00 2024-03-28T18:04:09.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.864+0000 7f692dfe0740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55752a0f5220 con 0x55752a0f2290 2024-03-28T18:04:09.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.864+0000 7f692dfe0740 10 client.0 _ll_register_callbacks cb 0x55752a01b130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:04:09.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.865+0000 7f6923e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e8f20 0x55752a0f18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:09.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.865+0000 7f691be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0e9c00 0x55752a0eec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:09.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.865+0000 7f6923e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e8f20 0x55752a0f18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43352/0 (socket says 172.21.15.45:43352) 2024-03-28T18:04:09.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.865+0000 7f6923e00640 1 -- 172.21.15.45:0/178065370 learned_addr learned my addr 172.21.15.45:0/178065370 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:04:09.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.865+0000 7f6929200640 1 --2- 172.21.15.45:0/178065370 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55752a0f2290 0x55752a0ef1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:09.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.865+0000 7f691be00640 1 -- 172.21.15.45:0/178065370 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55752a0f2290 msgr2=0x55752a0ef1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:04:09.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.865+0000 7f691be00640 1 --2- 172.21.15.45:0/178065370 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55752a0f2290 0x55752a0ef1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.865+0000 7f691be00640 1 -- 172.21.15.45:0/178065370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e8f20 msgr2=0x55752a0f18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:04:09.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.865+0000 7f691be00640 1 --2- 172.21.15.45:0/178065370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e8f20 0x55752a0f18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.865+0000 7f691be00640 1 -- 172.21.15.45:0/178065370 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557529e2b7d0 con 0x55752a0e9c00 2024-03-28T18:04:09.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.865+0000 7f6923e00640 1 --2- 172.21.15.45:0/178065370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e8f20 0x55752a0f18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:04:09.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.865+0000 7f691be00640 1 --2- 172.21.15.45:0/178065370 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0e9c00 0x55752a0eec70 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f692406acf0 tx=0x7f690c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e9e2cd36aa9e2c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:04:09.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.865+0000 7f6912000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T18:04:09.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.866+0000 7f6912000640 1 -- 172.21.15.45:0/178065370 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690c0099b0 con 0x55752a0e9c00 2024-03-28T18:04:09.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.866+0000 7f6912000640 1 -- 172.21.15.45:0/178065370 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(147..147 src has 1..147) v4 ==== 12382+0+0 (secure 0 0 0) 0x7f690c01eb90 con 0x55752a0e9c00 2024-03-28T18:04:09.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.866+0000 7f692dfe0740 1 -- 172.21.15.45:0/178065370 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f691402c880 con 0x55752a0e9c00 2024-03-28T18:04:09.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.866+0000 7f6912000640 1 -- 172.21.15.45:0/178065370 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=319) v2 ==== 24+0+0 (secure 0 0 0) 0x7f690c01ed90 con 0x55752a0e9c00 2024-03-28T18:04:09.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.866+0000 7f692dfe0740 10 client.26615 fetch_fsmap learned FSMap version 319 2024-03-28T18:04:09.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.866+0000 7f692dfe0740 1 -- 172.21.15.45:0/178065370 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=319,monmap=4+}) v3 -- 0x557529d613f0 con 0x55752a0e9c00 2024-03-28T18:04:09.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.866+0000 7f6912000640 1 -- 172.21.15.45:0/178065370 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f691402c880 con 0x55752a0e9c00 2024-03-28T18:04:09.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.866+0000 7f6912000640 1 -- 172.21.15.45:0/178065370 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 319) v1 ==== 3905+0+0 (secure 0 0 0) 0x7f690c01a040 con 0x55752a0e9c00 2024-03-28T18:04:09.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.866+0000 7f692dfe0740 10 client.26615 fetch_fsmap finished waiting for FSMap version 319 2024-03-28T18:04:09.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.866+0000 7f692dfe0740 10 client.26615 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:04:09.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.867+0000 7f692dfe0740 20 client.26615 populate_metadata read hostname 'smithi045' 2024-03-28T18:04:09.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.867+0000 7f692dfe0740 1 --2- 172.21.15.45:0/178065370 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55752a102420 0x55752a1048c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:09.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.867+0000 7f692dfe0740 4 client.26615 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:04:09.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.867+0000 7f692dfe0740 1 -- 172.21.15.45:0/178065370 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55752a05a790 con 0x55752a102420 2024-03-28T18:04:09.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.867+0000 7f6923e00640 1 --2- 172.21.15.45:0/178065370 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55752a102420 0x55752a1048c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:09.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.867+0000 7f6923e00640 1 --2- 172.21.15.45:0/178065370 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55752a102420 0x55752a1048c0 crc :-1 s=READY pgs=420 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d5df79652dc5740 server_cookie=d29f2f89d75ac46d in_seq=0 out_seq=0 2024-03-28T18:04:09.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.867+0000 7f6912000640 10 client.26615 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:04:09.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.868+0000 7f6912000640 1 -- 172.21.15.45:0/178065370 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55752a05a790 con 0x55752a102420 2024-03-28T18:04:09.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.868+0000 7f6912000640 10 client.26615 handle_command_reply: tid=0 2024-03-28T18:04:09.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.868+0000 7f692dfe0740 1 client.26615 shutdown 2024-03-28T18:04:09.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.869+0000 7f692dfe0740 1 -- 172.21.15.45:0/178065370 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0e9c00 msgr2=0x55752a0eec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:09.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.869+0000 7f692dfe0740 1 --2- 172.21.15.45:0/178065370 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0e9c00 0x55752a0eec70 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f692406acf0 tx=0x7f690c0046c0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.869+0000 7f692dfe0740 1 -- 172.21.15.45:0/178065370 shutdown_connections 2024-03-28T18:04:09.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.869+0000 7f692dfe0740 1 --2- 172.21.15.45:0/178065370 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55752a102420 0x55752a1048c0 crc :-1 s=READY pgs=420 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.869+0000 7f692dfe0740 1 --2- 172.21.15.45:0/178065370 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55752a0f2290 0x55752a0ef1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.869+0000 7f692dfe0740 1 --2- 172.21.15.45:0/178065370 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0e9c00 0x55752a0eec70 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.869+0000 7f692dfe0740 1 --2- 172.21.15.45:0/178065370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e8f20 0x55752a0f18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.869+0000 7f6912000640 0 client.26615 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:04:09.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.869+0000 7f692dfe0740 1 -- 172.21.15.45:0/178065370 >> 172.21.15.45:0/178065370 conn(0x55752a0b0200 msgr2=0x55752a0e56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:04:09.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.869+0000 7f692dfe0740 1 -- 172.21.15.45:0/178065370 shutdown_connections 2024-03-28T18:04:09.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.869+0000 7f692dfe0740 1 -- 172.21.15.45:0/178065370 wait complete. 2024-03-28T18:04:09.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.869+0000 7f692dfe0740 20 client.26615 trim_cache size 0 max 16384 2024-03-28T18:04:09.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.887+0000 7f692dfe0740 1 Processor -- start 2024-03-28T18:04:09.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.887+0000 7f692dfe0740 1 -- start start 2024-03-28T18:04:09.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.887+0000 7f692dfe0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e8f20 0x55752a0f13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:09.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.887+0000 7f692dfe0740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55752a0e9c00 0x55752a0f18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:09.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.887+0000 7f692dfe0740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0f2290 0x55752a0f3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:09.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.887+0000 7f692dfe0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69241b5ce0 con 0x55752a0e8f20 2024-03-28T18:04:09.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.887+0000 7f692dfe0740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69241b4360 con 0x55752a0f2290 2024-03-28T18:04:09.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.887+0000 7f692dfe0740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69241b5680 con 0x55752a0e9c00 2024-03-28T18:04:09.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.887+0000 7f6929200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0f2290 0x55752a0f3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:09.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.887+0000 7f691be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55752a0e9c00 0x55752a0f18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:09.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.887+0000 7f6923e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e8f20 0x55752a0f13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:09.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.887+0000 7f6929200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0f2290 0x55752a0f3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:44156/0 (socket says 172.21.15.45:44156) 2024-03-28T18:04:09.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.887+0000 7f6929200640 1 -- 172.21.15.45:0/1057853016 learned_addr learned my addr 172.21.15.45:0/1057853016 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:04:09.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.888+0000 7f6929200640 1 -- 172.21.15.45:0/1057853016 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55752a0e9c00 msgr2=0x55752a0f18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:04:09.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.888+0000 7f6929200640 1 --2- 172.21.15.45:0/1057853016 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55752a0e9c00 0x55752a0f18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.888+0000 7f6929200640 1 -- 172.21.15.45:0/1057853016 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e8f20 msgr2=0x55752a0f13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:04:09.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.888+0000 7f6929200640 1 --2- 172.21.15.45:0/1057853016 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e8f20 0x55752a0f13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.888+0000 7f6929200640 1 -- 172.21.15.45:0/1057853016 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557529d210f0 con 0x55752a0f2290 2024-03-28T18:04:09.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.888+0000 7f6923e00640 1 --2- 172.21.15.45:0/1057853016 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e8f20 0x55752a0f13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:04:09.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.888+0000 7f691be00640 1 --2- 172.21.15.45:0/1057853016 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55752a0e9c00 0x55752a0f18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:04:09.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.888+0000 7f6929200640 1 --2- 172.21.15.45:0/1057853016 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0f2290 0x55752a0f3680 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f691c097a00 tx=0x7f691c0058d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53523e402ba21a90 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:04:09.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.888+0000 7f6918c00640 1 -- 172.21.15.45:0/1057853016 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c01f040 con 0x55752a0f2290 2024-03-28T18:04:09.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.888+0000 7f6918c00640 1 -- 172.21.15.45:0/1057853016 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f691c0a3d90 con 0x55752a0f2290 2024-03-28T18:04:09.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.888+0000 7f6918c00640 1 -- 172.21.15.45:0/1057853016 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c084b30 con 0x55752a0f2290 2024-03-28T18:04:09.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.889+0000 7f692dfe0740 1 -- 172.21.15.45:0/1057853016 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0f2290 msgr2=0x55752a0f3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:09.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.889+0000 7f692dfe0740 1 --2- 172.21.15.45:0/1057853016 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0f2290 0x55752a0f3680 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f691c097a00 tx=0x7f691c0058d0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.889+0000 7f692dfe0740 1 -- 172.21.15.45:0/1057853016 shutdown_connections 2024-03-28T18:04:09.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.889+0000 7f692dfe0740 1 --2- 172.21.15.45:0/1057853016 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0f2290 0x55752a0f3680 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.889+0000 7f692dfe0740 1 --2- 172.21.15.45:0/1057853016 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55752a0e9c00 0x55752a0f18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.889+0000 7f692dfe0740 1 --2- 172.21.15.45:0/1057853016 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e8f20 0x55752a0f13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.889+0000 7f692dfe0740 1 -- 172.21.15.45:0/1057853016 >> 172.21.15.45:0/1057853016 conn(0x55752a0b0200 msgr2=0x55752a0eec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:04:09.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.889+0000 7f692dfe0740 1 -- 172.21.15.45:0/1057853016 shutdown_connections 2024-03-28T18:04:09.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.889+0000 7f692dfe0740 1 -- 172.21.15.45:0/1057853016 wait complete. 2024-03-28T18:04:09.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.890+0000 7f692dfe0740 1 -- start start 2024-03-28T18:04:09.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.890+0000 7f692dfe0740 1 Processor -- start 2024-03-28T18:04:09.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.890+0000 7f692dfe0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e8f20 0x55752a0f9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:09.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.890+0000 7f692dfe0740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55752a0e9c00 0x55752a0fa400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:09.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.890+0000 7f692dfe0740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0f2290 0x55752a0f5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:09.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.890+0000 7f692dfe0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69241b5020 con 0x55752a0e8f20 2024-03-28T18:04:09.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.890+0000 7f692dfe0740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69241b49c0 con 0x55752a0f2290 2024-03-28T18:04:09.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.890+0000 7f692dfe0740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55752a0f4a50 con 0x55752a0e9c00 2024-03-28T18:04:09.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.890+0000 7f692dfe0740 10 client.0 _ll_register_callbacks cb 0x7f690c0118e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:04:09.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.891+0000 7f6923e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e8f20 0x55752a0f9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:09.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.891+0000 7f6929200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0f2290 0x55752a0f5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:09.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.891+0000 7f6929200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0f2290 0x55752a0f5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:44164/0 (socket says 172.21.15.45:44164) 2024-03-28T18:04:09.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.891+0000 7f6929200640 1 -- 172.21.15.45:0/3078663532 learned_addr learned my addr 172.21.15.45:0/3078663532 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:04:09.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.891+0000 7f6923e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e8f20 0x55752a0f9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43362/0 (socket says 172.21.15.45:43362) 2024-03-28T18:04:09.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.891+0000 7f691be00640 1 --2- 172.21.15.45:0/3078663532 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55752a0e9c00 0x55752a0fa400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:09.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.891+0000 7f6929200640 1 -- 172.21.15.45:0/3078663532 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55752a0e9c00 msgr2=0x55752a0fa400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:04:09.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.891+0000 7f6929200640 1 --2- 172.21.15.45:0/3078663532 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55752a0e9c00 0x55752a0fa400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.891+0000 7f6929200640 1 -- 172.21.15.45:0/3078663532 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e8f20 msgr2=0x55752a0f9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:04:09.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.891+0000 7f6929200640 1 --2- 172.21.15.45:0/3078663532 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e8f20 0x55752a0f9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.891+0000 7f6929200640 1 -- 172.21.15.45:0/3078663532 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55752a0f5e70 con 0x55752a0f2290 2024-03-28T18:04:09.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.891+0000 7f6923e00640 1 --2- 172.21.15.45:0/3078663532 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e8f20 0x55752a0f9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:04:09.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.891+0000 7f6929200640 1 --2- 172.21.15.45:0/3078663532 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0f2290 0x55752a0f5670 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f691c097a50 tx=0x7f691c099b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e56470ee34d62c3b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:04:09.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.891+0000 7f6912000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T18:04:09.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.891+0000 7f6912000640 1 -- 172.21.15.45:0/3078663532 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c01f070 con 0x55752a0f2290 2024-03-28T18:04:09.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.892+0000 7f6912000640 1 -- 172.21.15.45:0/3078663532 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(147..147 src has 1..147) v4 ==== 12382+0+0 (secure 0 0 0) 0x7f691c0b4240 con 0x55752a0f2290 2024-03-28T18:04:09.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.892+0000 7f692dfe0740 1 -- 172.21.15.45:0/3078663532 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f691c0a3d90 con 0x55752a0f2290 2024-03-28T18:04:09.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.892+0000 7f6912000640 1 -- 172.21.15.45:0/3078663532 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=319) v2 ==== 24+0+0 (secure 0 0 0) 0x7f691c0b4440 con 0x55752a0f2290 2024-03-28T18:04:09.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.892+0000 7f692dfe0740 10 client.26621 fetch_fsmap learned FSMap version 319 2024-03-28T18:04:09.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.892+0000 7f692dfe0740 1 -- 172.21.15.45:0/3078663532 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=319,monmap=4+}) v3 -- 0x55752a15d9d0 con 0x55752a0f2290 2024-03-28T18:04:09.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.892+0000 7f6912000640 1 -- 172.21.15.45:0/3078663532 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f691c0a3d90 con 0x55752a0f2290 2024-03-28T18:04:09.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.892+0000 7f6912000640 1 -- 172.21.15.45:0/3078663532 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 319) v1 ==== 3905+0+0 (secure 0 0 0) 0x7f691c084020 con 0x55752a0f2290 2024-03-28T18:04:09.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.892+0000 7f692dfe0740 10 client.26621 fetch_fsmap finished waiting for FSMap version 319 2024-03-28T18:04:09.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.892+0000 7f692dfe0740 10 client.26621 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:04:09.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.892+0000 7f692dfe0740 20 client.26621 populate_metadata read hostname 'smithi045' 2024-03-28T18:04:09.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.892+0000 7f692dfe0740 1 --2- 172.21.15.45:0/3078663532 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55752a15e090 0x55752a1604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:09.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.892+0000 7f692dfe0740 4 client.26621 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:04:09.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.892+0000 7f692dfe0740 1 -- 172.21.15.45:0/3078663532 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55752a054ae0 con 0x55752a15e090 2024-03-28T18:04:09.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.892+0000 7f6923e00640 1 --2- 172.21.15.45:0/3078663532 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55752a15e090 0x55752a1604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:09.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.893+0000 7f6923e00640 1 --2- 172.21.15.45:0/3078663532 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55752a15e090 0x55752a1604b0 crc :-1 s=READY pgs=421 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8dac404218e65bb server_cookie=bb4203dee91308bb in_seq=0 out_seq=0 2024-03-28T18:04:09.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.893+0000 7f6912000640 10 client.26621 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:04:09.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.893+0000 7f6912000640 1 -- 172.21.15.45:0/3078663532 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55752a054ae0 con 0x55752a15e090 2024-03-28T18:04:09.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.893+0000 7f6912000640 10 client.26621 handle_command_reply: tid=0 2024-03-28T18:04:09.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.893+0000 7f692dfe0740 1 client.26621 shutdown 2024-03-28T18:04:09.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.893+0000 7f692dfe0740 1 -- 172.21.15.45:0/3078663532 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0f2290 msgr2=0x55752a0f5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:09.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.893+0000 7f692dfe0740 1 --2- 172.21.15.45:0/3078663532 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0f2290 0x55752a0f5670 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f691c097a50 tx=0x7f691c099b00 comp rx=0 tx=0).stop 2024-03-28T18:04:09.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.893+0000 7f692dfe0740 1 -- 172.21.15.45:0/3078663532 shutdown_connections 2024-03-28T18:04:09.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.893+0000 7f692dfe0740 1 --2- 172.21.15.45:0/3078663532 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55752a15e090 0x55752a1604b0 crc :-1 s=READY pgs=421 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.893+0000 7f692dfe0740 1 --2- 172.21.15.45:0/3078663532 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55752a0f2290 0x55752a0f5670 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.893+0000 7f6912000640 0 client.26621 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:04:09.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.893+0000 7f692dfe0740 1 --2- 172.21.15.45:0/3078663532 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55752a0e9c00 0x55752a0fa400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.893+0000 7f692dfe0740 1 --2- 172.21.15.45:0/3078663532 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55752a0e8f20 0x55752a0f9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.893+0000 7f692dfe0740 1 -- 172.21.15.45:0/3078663532 >> 172.21.15.45:0/3078663532 conn(0x55752a0b0200 msgr2=0x55752a0ed150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:04:09.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.894+0000 7f692dfe0740 1 -- 172.21.15.45:0/3078663532 shutdown_connections 2024-03-28T18:04:09.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.894+0000 7f692dfe0740 1 -- 172.21.15.45:0/3078663532 wait complete. 2024-03-28T18:04:09.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.894+0000 7f692dfe0740 20 client.26621 trim_cache size 0 max 16384 2024-03-28T18:04:09.898 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T18:04:09.899 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:04:09.899 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T18:04:09.899 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T18:04:09.899 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T18:04:09.899 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T18:04:09.899 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T18:04:09.899 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:04:09.899 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:04:09.899 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T18:04:09.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.894+0000 7f692b200640 1 -- 172.21.15.45:0/3289226894 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f69080a29f0 msgr2=0x7f69080a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:09.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.894+0000 7f692b200640 1 --2- 172.21.15.45:0/3289226894 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f69080a29f0 0x7f69080a4eb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6924107010 tx=0x7f690c009820 comp rx=0 tx=0).stop 2024-03-28T18:04:09.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.894+0000 7f692b200640 1 -- 172.21.15.45:0/3289226894 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69241a2e40 msgr2=0x7f69241a32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:09.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.894+0000 7f692b200640 1 --2- 172.21.15.45:0/3289226894 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69241a2e40 0x7f69241a32f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f691c00eaa0 tx=0x7f691c00ef60 comp rx=0 tx=0).stop 2024-03-28T18:04:09.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.895+0000 7f692b200640 1 -- 172.21.15.45:0/3289226894 shutdown_connections 2024-03-28T18:04:09.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.895+0000 7f692b200640 1 --2- 172.21.15.45:0/3289226894 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f69080a29f0 0x7f69080a4eb0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.895+0000 7f692b200640 1 --2- 172.21.15.45:0/3289226894 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69241a2e40 0x7f69241a32f0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.895+0000 7f692b200640 1 --2- 172.21.15.45:0/3289226894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f692410cf80 0x7f692419eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.895+0000 7f692b200640 1 --2- 172.21.15.45:0/3289226894 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6924107eb0 0x7f692419e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:09.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.895+0000 7f692b200640 1 -- 172.21.15.45:0/3289226894 >> 172.21.15.45:0/3289226894 conn(0x7f6924075be0 msgr2=0x7f6924077ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:04:09.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.895+0000 7f692b200640 1 -- 172.21.15.45:0/3289226894 shutdown_connections 2024-03-28T18:04:09.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:09.895+0000 7f692b200640 1 -- 172.21.15.45:0/3289226894 wait complete. 2024-03-28T18:04:09.912 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T18:04:09.950 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 702 sec 2024-03-28T18:04:10.950 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 703 sec 2024-03-28T18:04:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:10 smithi071 ceph-mon[36597]: pgmap v523: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:04:11.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:10 smithi151 ceph-mon[36481]: pgmap v523: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:04:11.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:10 smithi045 ceph-mon[31773]: pgmap v523: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:04:11.950 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 704 sec 2024-03-28T18:04:12.951 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 705 sec 2024-03-28T18:04:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:12 smithi071 ceph-mon[36597]: pgmap v524: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T18:04:13.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:12 smithi151 ceph-mon[36481]: pgmap v524: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T18:04:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:12 smithi045 ceph-mon[31773]: pgmap v524: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T18:04:13.951 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 706 sec 2024-03-28T18:04:14.951 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 707 sec 2024-03-28T18:04:15.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:14 smithi071 ceph-mon[36597]: pgmap v525: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:04:15.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:14 smithi151 ceph-mon[36481]: pgmap v525: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:04:15.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:14 smithi045 ceph-mon[31773]: pgmap v525: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:04:15.951 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 708 sec 2024-03-28T18:04:16.951 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 709 sec 2024-03-28T18:04:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:16 smithi071 ceph-mon[36597]: pgmap v526: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T18:04:17.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:16 smithi151 ceph-mon[36481]: pgmap v526: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T18:04:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:16 smithi045 ceph-mon[31773]: pgmap v526: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T18:04:17.951 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 710 sec 2024-03-28T18:04:18.951 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 711 sec 2024-03-28T18:04:19.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:18 smithi071 ceph-mon[36597]: pgmap v527: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:04:19.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:18 smithi151 ceph-mon[36481]: pgmap v527: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:04:19.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:18 smithi045 ceph-mon[31773]: pgmap v527: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:04:19.952 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 712 sec 2024-03-28T18:04:20.953 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 713 sec 2024-03-28T18:04:21.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:20 smithi071 ceph-mon[36597]: pgmap v528: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:21.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:20 smithi151 ceph-mon[36481]: pgmap v528: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:21.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:20 smithi045 ceph-mon[31773]: pgmap v528: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:21.952 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 714 sec 2024-03-28T18:04:22.952 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 715 sec 2024-03-28T18:04:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:22 smithi071 ceph-mon[36597]: pgmap v529: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:04:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:22 smithi151 ceph-mon[36481]: pgmap v529: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:04:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:22 smithi045 ceph-mon[31773]: pgmap v529: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:04:23.952 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 716 sec 2024-03-28T18:04:24.952 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 717 sec 2024-03-28T18:04:25.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:24 smithi071 ceph-mon[36597]: pgmap v530: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:25.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:24 smithi151 ceph-mon[36481]: pgmap v530: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:25.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:24 smithi045 ceph-mon[31773]: pgmap v530: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:25.952 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 718 sec 2024-03-28T18:04:26.953 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 719 sec 2024-03-28T18:04:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:26 smithi071 ceph-mon[36597]: pgmap v531: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:04:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:26 smithi151 ceph-mon[36481]: pgmap v531: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:04:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:26 smithi045 ceph-mon[31773]: pgmap v531: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:04:27.953 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 720 sec 2024-03-28T18:04:28.953 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 721 sec 2024-03-28T18:04:29.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:28 smithi071 ceph-mon[36597]: pgmap v532: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:29.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:28 smithi151 ceph-mon[36481]: pgmap v532: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:29.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:28 smithi045 ceph-mon[31773]: pgmap v532: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:29.953 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 722 sec 2024-03-28T18:04:30.953 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 723 sec 2024-03-28T18:04:31.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:30 smithi071 ceph-mon[36597]: pgmap v533: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:31.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:30 smithi151 ceph-mon[36481]: pgmap v533: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:31.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:30 smithi045 ceph-mon[31773]: pgmap v533: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:31.953 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 724 sec 2024-03-28T18:04:32.954 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 725 sec 2024-03-28T18:04:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:32 smithi071 ceph-mon[36597]: pgmap v534: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:04:33.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:32 smithi151 ceph-mon[36481]: pgmap v534: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:04:33.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:32 smithi045 ceph-mon[31773]: pgmap v534: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:04:33.953 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 726 sec 2024-03-28T18:04:34.954 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 727 sec 2024-03-28T18:04:35.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:34 smithi071 ceph-mon[36597]: pgmap v535: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:34 smithi151 ceph-mon[36481]: pgmap v535: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:34 smithi045 ceph-mon[31773]: pgmap v535: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:35.954 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 728 sec 2024-03-28T18:04:36.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:35 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:04:36.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:35 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:04:36.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:35 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:04:36.954 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 729 sec 2024-03-28T18:04:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:36 smithi071 ceph-mon[36597]: pgmap v536: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:04:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:36 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:36 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:04:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:36 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:04:37.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:36 smithi151 ceph-mon[36481]: pgmap v536: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:04:37.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:36 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:37.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:36 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:04:37.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:36 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:04:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:36 smithi045 ceph-mon[31773]: pgmap v536: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:04:37.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:36 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:04:37.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:36 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:04:37.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:36 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:04:37.954 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 730 sec 2024-03-28T18:04:38.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:37 smithi071 ceph-mon[36597]: pgmap v537: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:38.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:37 smithi071 ceph-mon[36597]: pgmap v538: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:04:38.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:37 smithi151 ceph-mon[36481]: pgmap v537: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:38.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:37 smithi151 ceph-mon[36481]: pgmap v538: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:04:38.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:37 smithi045 ceph-mon[31773]: pgmap v537: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:38.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:37 smithi045 ceph-mon[31773]: pgmap v538: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:04:38.954 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 731 sec 2024-03-28T18:04:39.914 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:04:39.914 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:04:39.954 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 732 sec 2024-03-28T18:04:40.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:39 smithi071 ceph-mon[36597]: pgmap v539: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:04:40.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:39 smithi151 ceph-mon[36481]: pgmap v539: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:04:40.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.154+0000 7fcc25400640 1 -- 172.21.15.45:0/282615303 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc201005d0 msgr2=0x7fcc20104140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:40.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.154+0000 7fcc25400640 1 --2- 172.21.15.45:0/282615303 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc201005d0 0x7fcc20104140 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fcc18009ca0 tx=0x7fcc1802c100 comp rx=0 tx=0).stop 2024-03-28T18:04:40.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.154+0000 7fcc25400640 1 -- 172.21.15.45:0/282615303 shutdown_connections 2024-03-28T18:04:40.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.154+0000 7fcc25400640 1 --2- 172.21.15.45:0/282615303 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc201005d0 0x7fcc20104140 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.154+0000 7fcc25400640 1 --2- 172.21.15.45:0/282615303 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc200ffc10 0x7fcc20100090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.154+0000 7fcc25400640 1 --2- 172.21.15.45:0/282615303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc201066e0 0x7fcc20106ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.154+0000 7fcc25400640 1 -- 172.21.15.45:0/282615303 >> 172.21.15.45:0/282615303 conn(0x7fcc20076bd0 msgr2=0x7fcc200fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:04:40.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.154+0000 7fcc25400640 1 -- 172.21.15.45:0/282615303 shutdown_connections 2024-03-28T18:04:40.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.154+0000 7fcc25400640 1 -- 172.21.15.45:0/282615303 wait complete. 2024-03-28T18:04:40.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.155+0000 7fcc25400640 1 Processor -- start 2024-03-28T18:04:40.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.155+0000 7fcc25400640 1 -- start start 2024-03-28T18:04:40.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.156+0000 7fcc25400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc200ffc10 0x7fcc2019e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:40.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.156+0000 7fcc25400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc201005d0 0x7fcc2019ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:40.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.156+0000 7fcc25400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc201066e0 0x7fcc201a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:40.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.156+0000 7fcc25400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc20118090 con 0x7fcc201066e0 2024-03-28T18:04:40.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.156+0000 7fcc25400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc20117f10 con 0x7fcc200ffc10 2024-03-28T18:04:40.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.156+0000 7fcc25400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc20118210 con 0x7fcc201005d0 2024-03-28T18:04:40.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.156+0000 7fcc1e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc200ffc10 0x7fcc2019e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:40.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.156+0000 7fcc1f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc201066e0 0x7fcc201a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:40.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.156+0000 7fcc1e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc200ffc10 0x7fcc2019e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40722/0 (socket says 172.21.15.45:40722) 2024-03-28T18:04:40.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.156+0000 7fcc1e800640 1 -- 172.21.15.45:0/3273857559 learned_addr learned my addr 172.21.15.45:0/3273857559 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:04:40.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.156+0000 7fcc15e00640 1 --2- 172.21.15.45:0/3273857559 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc201005d0 0x7fcc2019ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:40.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.156+0000 7fcc1e800640 1 -- 172.21.15.45:0/3273857559 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc201005d0 msgr2=0x7fcc2019ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:40.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.156+0000 7fcc1e800640 1 --2- 172.21.15.45:0/3273857559 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc201005d0 0x7fcc2019ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.156+0000 7fcc1e800640 1 -- 172.21.15.45:0/3273857559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc201066e0 msgr2=0x7fcc201a2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:40.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.156+0000 7fcc1e800640 1 --2- 172.21.15.45:0/3273857559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc201066e0 0x7fcc201a2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.156+0000 7fcc1e800640 1 -- 172.21.15.45:0/3273857559 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc201a34c0 con 0x7fcc200ffc10 2024-03-28T18:04:40.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.157+0000 7fcc1e800640 1 --2- 172.21.15.45:0/3273857559 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc200ffc10 0x7fcc2019e4d0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000d770 tx=0x7fcc1000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:04:40.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.157+0000 7fcc17e00640 1 -- 172.21.15.45:0/3273857559 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc10017070 con 0x7fcc200ffc10 2024-03-28T18:04:40.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.157+0000 7fcc17e00640 1 -- 172.21.15.45:0/3273857559 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc10009c30 con 0x7fcc200ffc10 2024-03-28T18:04:40.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.157+0000 7fcc17e00640 1 -- 172.21.15.45:0/3273857559 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc10003dd0 con 0x7fcc200ffc10 2024-03-28T18:04:40.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.157+0000 7fcc25400640 1 -- 172.21.15.45:0/3273857559 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc201a37b0 con 0x7fcc200ffc10 2024-03-28T18:04:40.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.157+0000 7fcc25400640 1 -- 172.21.15.45:0/3273857559 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc201ab040 con 0x7fcc200ffc10 2024-03-28T18:04:40.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.161+0000 7fcc2803c740 1 Processor -- start 2024-03-28T18:04:40.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.162+0000 7fcc2803c740 1 -- start start 2024-03-28T18:04:40.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.162+0000 7fcc17e00640 1 -- 172.21.15.45:0/3273857559 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fcc10003f70 con 0x7fcc200ffc10 2024-03-28T18:04:40.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.162+0000 7fcc2803c740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f5f0812c00 0x55f5f0811950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:40.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.162+0000 7fcc2803c740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f0811f20 0x55f5f08123a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:40.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.162+0000 7fcc2803c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5f081b290 0x55f5f081d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:40.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.162+0000 7fcc2803c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f0776250 con 0x55f5f081b290 2024-03-28T18:04:40.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.162+0000 7fcc2803c740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f071ea30 con 0x55f5f0811f20 2024-03-28T18:04:40.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.162+0000 7fcc2803c740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f07722f0 con 0x55f5f0812c00 2024-03-28T18:04:40.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.163+0000 7fcc1e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f0811f20 0x55f5f08123a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:40.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.163+0000 7fcc15e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5f081b290 0x55f5f081d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:40.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.163+0000 7fcc1e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f0811f20 0x55f5f08123a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40724/0 (socket says 172.21.15.45:40724) 2024-03-28T18:04:40.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.163+0000 7fcc1e800640 1 -- 172.21.15.45:0/168817658 learned_addr learned my addr 172.21.15.45:0/168817658 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:04:40.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.163+0000 7fcc1f200640 1 --2- 172.21.15.45:0/168817658 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f5f0812c00 0x55f5f0811950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:40.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.163+0000 7fcc1e800640 1 -- 172.21.15.45:0/168817658 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f5f0812c00 msgr2=0x55f5f0811950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:04:40.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.163+0000 7fcc1e800640 1 --2- 172.21.15.45:0/168817658 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f5f0812c00 0x55f5f0811950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.163+0000 7fcc1e800640 1 -- 172.21.15.45:0/168817658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5f081b290 msgr2=0x55f5f081d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:04:40.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.163+0000 7fcc1e800640 1 --2- 172.21.15.45:0/168817658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5f081b290 0x55f5f081d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.163+0000 7fcc1e800640 1 -- 172.21.15.45:0/168817658 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5f0709690 con 0x55f5f0811f20 2024-03-28T18:04:40.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.163+0000 7fcc15e00640 1 --2- 172.21.15.45:0/168817658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5f081b290 0x55f5f081d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:04:40.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.163+0000 7fcc1e800640 1 --2- 172.21.15.45:0/168817658 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f0811f20 0x55f5f08123a0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fcc100cbd20 tx=0x7fcc100d5bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d926270fcb07e9db server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:04:40.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.164+0000 7fcc0f400640 1 -- 172.21.15.45:0/168817658 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc100e2070 con 0x55f5f0811f20 2024-03-28T18:04:40.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.164+0000 7fcc0f400640 1 -- 172.21.15.45:0/168817658 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc100d5e20 con 0x55f5f0811f20 2024-03-28T18:04:40.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.164+0000 7fcc17e00640 1 --2- 172.21.15.45:0/3273857559 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fcc040a29f0 0x7fcc040a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:40.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.164+0000 7fcc0f400640 1 -- 172.21.15.45:0/168817658 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc100d15c0 con 0x55f5f0811f20 2024-03-28T18:04:40.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.164+0000 7fcc17e00640 1 -- 172.21.15.45:0/3273857559 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 12382+0+0 (secure 0 0 0) 0x7fcc100cf600 con 0x7fcc200ffc10 2024-03-28T18:04:40.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.164+0000 7fcc1f200640 1 --2- 172.21.15.45:0/3273857559 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fcc040a29f0 0x7fcc040a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:40.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.164+0000 7fcc2803c740 1 -- 172.21.15.45:0/168817658 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f0811f20 msgr2=0x55f5f08123a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:40.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.164+0000 7fcc2803c740 1 --2- 172.21.15.45:0/168817658 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f0811f20 0x55f5f08123a0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fcc100cbd20 tx=0x7fcc100d5bd0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.165+0000 7fcc2803c740 1 -- 172.21.15.45:0/168817658 shutdown_connections 2024-03-28T18:04:40.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.165+0000 7fcc2803c740 1 --2- 172.21.15.45:0/168817658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5f081b290 0x55f5f081d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.165+0000 7fcc2803c740 1 --2- 172.21.15.45:0/168817658 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f0811f20 0x55f5f08123a0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.165+0000 7fcc2803c740 1 --2- 172.21.15.45:0/168817658 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f5f0812c00 0x55f5f0811950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.165+0000 7fcc2803c740 1 -- 172.21.15.45:0/168817658 >> 172.21.15.45:0/168817658 conn(0x55f5f07d9200 msgr2=0x55f5f080c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:04:40.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.165+0000 7fcc2803c740 1 -- 172.21.15.45:0/168817658 shutdown_connections 2024-03-28T18:04:40.168 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:39 smithi045 ceph-mon[31773]: pgmap v539: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:04:40.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.165+0000 7fcc2803c740 1 -- 172.21.15.45:0/168817658 wait complete. 2024-03-28T18:04:40.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.165+0000 7fcc1f200640 1 --2- 172.21.15.45:0/3273857559 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fcc040a29f0 0x7fcc040a4eb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcc18005e30 tx=0x7fcc18005d80 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:04:40.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.165+0000 7fcc2803c740 1 -- start start 2024-03-28T18:04:40.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.166+0000 7fcc2803c740 1 Processor -- start 2024-03-28T18:04:40.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.166+0000 7fcc2803c740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f5f0811f20 0x55f5f081a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:40.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.166+0000 7fcc2803c740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f0812c00 0x55f5f0817ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:40.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.166+0000 7fcc2803c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5f081b290 0x55f5f0818220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:40.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.166+0000 7fcc2803c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f081e100 con 0x55f5f081b290 2024-03-28T18:04:40.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.166+0000 7fcc2803c740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f0731e80 con 0x55f5f0812c00 2024-03-28T18:04:40.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.166+0000 7fcc2803c740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f081e280 con 0x55f5f0811f20 2024-03-28T18:04:40.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.166+0000 7fcc2803c740 10 client.0 _ll_register_callbacks cb 0x55f5f0744130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:04:40.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.166+0000 7fcc15e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f0812c00 0x55f5f0817ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:40.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.166+0000 7fcc15e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f0812c00 0x55f5f0817ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40740/0 (socket says 172.21.15.45:40740) 2024-03-28T18:04:40.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.166+0000 7fcc15e00640 1 -- 172.21.15.45:0/1761604598 learned_addr learned my addr 172.21.15.45:0/1761604598 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:04:40.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.166+0000 7fcc1f200640 1 --2- 172.21.15.45:0/1761604598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5f081b290 0x55f5f0818220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:40.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.167+0000 7fcc1e800640 1 --2- 172.21.15.45:0/1761604598 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f5f0811f20 0x55f5f081a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:40.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.167+0000 7fcc15e00640 1 -- 172.21.15.45:0/1761604598 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f5f0811f20 msgr2=0x55f5f081a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:04:40.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.167+0000 7fcc15e00640 1 --2- 172.21.15.45:0/1761604598 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f5f0811f20 0x55f5f081a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.167+0000 7fcc15e00640 1 -- 172.21.15.45:0/1761604598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5f081b290 msgr2=0x55f5f0818220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:04:40.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.167+0000 7fcc15e00640 1 --2- 172.21.15.45:0/1761604598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5f081b290 0x55f5f0818220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.167+0000 7fcc15e00640 1 -- 172.21.15.45:0/1761604598 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f5f05547d0 con 0x55f5f0812c00 2024-03-28T18:04:40.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.167+0000 7fcc1f200640 1 --2- 172.21.15.45:0/1761604598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5f081b290 0x55f5f0818220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:04:40.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.167+0000 7fcc15e00640 1 --2- 172.21.15.45:0/1761604598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f0812c00 0x55f5f0817ce0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fcc08012770 tx=0x7fcc08012c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a2f1bedacf234 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:04:40.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.167+0000 7fcc03e00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T18:04:40.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.167+0000 7fcc03e00640 1 -- 172.21.15.45:0/1761604598 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0800f920 con 0x55f5f0812c00 2024-03-28T18:04:40.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.167+0000 7fcc2803c740 1 -- 172.21.15.45:0/1761604598 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcc100d5e20 con 0x55f5f0812c00 2024-03-28T18:04:40.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.167+0000 7fcc03e00640 1 -- 172.21.15.45:0/1761604598 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(147..147 src has 1..147) v4 ==== 12382+0+0 (secure 0 0 0) 0x7fcc0800fac0 con 0x55f5f0812c00 2024-03-28T18:04:40.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.168+0000 7fcc03e00640 1 -- 172.21.15.45:0/1761604598 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=319) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc0800fcc0 con 0x55f5f0812c00 2024-03-28T18:04:40.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.168+0000 7fcc2803c740 10 client.26633 fetch_fsmap learned FSMap version 319 2024-03-28T18:04:40.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.168+0000 7fcc2803c740 1 -- 172.21.15.45:0/1761604598 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=319,monmap=4+}) v3 -- 0x55f5f048a3f0 con 0x55f5f0812c00 2024-03-28T18:04:40.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.168+0000 7fcc03e00640 1 -- 172.21.15.45:0/1761604598 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc100d5e20 con 0x55f5f0812c00 2024-03-28T18:04:40.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.168+0000 7fcc03e00640 1 -- 172.21.15.45:0/1761604598 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 319) v1 ==== 3905+0+0 (secure 0 0 0) 0x7fcc0801c420 con 0x55f5f0812c00 2024-03-28T18:04:40.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.168+0000 7fcc2803c740 10 client.26633 fetch_fsmap finished waiting for FSMap version 319 2024-03-28T18:04:40.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.168+0000 7fcc2803c740 10 client.26633 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:04:40.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.168+0000 7fcc2803c740 20 client.26633 populate_metadata read hostname 'smithi045' 2024-03-28T18:04:40.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.168+0000 7fcc2803c740 1 --2- 172.21.15.45:0/1761604598 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f5f082b630 0x55f5f082da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:40.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.168+0000 7fcc2803c740 4 client.26633 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:04:40.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.168+0000 7fcc2803c740 1 -- 172.21.15.45:0/1761604598 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f5f0783790 con 0x55f5f082b630 2024-03-28T18:04:40.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.168+0000 7fcc1e800640 1 --2- 172.21.15.45:0/1761604598 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f5f082b630 0x55f5f082da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:40.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.169+0000 7fcc1e800640 1 --2- 172.21.15.45:0/1761604598 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f5f082b630 0x55f5f082da50 crc :-1 s=READY pgs=424 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=492063e4b67c1f28 server_cookie=a5507957f75fdf7 in_seq=0 out_seq=0 2024-03-28T18:04:40.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.169+0000 7fcc03e00640 10 client.26633 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:04:40.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.169+0000 7fcc03e00640 1 -- 172.21.15.45:0/1761604598 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f5f0783790 con 0x55f5f082b630 2024-03-28T18:04:40.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.169+0000 7fcc03e00640 10 client.26633 handle_command_reply: tid=0 2024-03-28T18:04:40.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.169+0000 7fcc2803c740 1 client.26633 shutdown 2024-03-28T18:04:40.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.170+0000 7fcc2803c740 1 -- 172.21.15.45:0/1761604598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f0812c00 msgr2=0x55f5f0817ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:40.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.170+0000 7fcc2803c740 1 --2- 172.21.15.45:0/1761604598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f0812c00 0x55f5f0817ce0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fcc08012770 tx=0x7fcc08012c30 comp rx=0 tx=0).stop 2024-03-28T18:04:40.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.170+0000 7fcc2803c740 1 -- 172.21.15.45:0/1761604598 shutdown_connections 2024-03-28T18:04:40.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.170+0000 7fcc2803c740 1 --2- 172.21.15.45:0/1761604598 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f5f082b630 0x55f5f082da50 crc :-1 s=READY pgs=424 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.170+0000 7fcc2803c740 1 --2- 172.21.15.45:0/1761604598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5f081b290 0x55f5f0818220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.170+0000 7fcc2803c740 1 --2- 172.21.15.45:0/1761604598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f0812c00 0x55f5f0817ce0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.170+0000 7fcc2803c740 1 --2- 172.21.15.45:0/1761604598 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f5f0811f20 0x55f5f081a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.170+0000 7fcc03e00640 0 client.26633 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:04:40.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.170+0000 7fcc2803c740 1 -- 172.21.15.45:0/1761604598 >> 172.21.15.45:0/1761604598 conn(0x55f5f07d9200 msgr2=0x55f5f080e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:04:40.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.170+0000 7fcc2803c740 1 -- 172.21.15.45:0/1761604598 shutdown_connections 2024-03-28T18:04:40.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.170+0000 7fcc2803c740 1 -- 172.21.15.45:0/1761604598 wait complete. 2024-03-28T18:04:40.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.170+0000 7fcc2803c740 20 client.26633 trim_cache size 0 max 16384 2024-03-28T18:04:40.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.188+0000 7fcc2803c740 1 Processor -- start 2024-03-28T18:04:40.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.188+0000 7fcc2803c740 1 -- start start 2024-03-28T18:04:40.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.188+0000 7fcc2803c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5f0811f20 0x55f5f081a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:40.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.188+0000 7fcc2803c740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f5f0812c00 0x55f5f081a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:40.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.188+0000 7fcc2803c740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f081b290 0x55f5f081c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:40.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.188+0000 7fcc2803c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc201b5c40 con 0x55f5f0811f20 2024-03-28T18:04:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.188+0000 7fcc2803c740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc201b42c0 con 0x55f5f081b290 2024-03-28T18:04:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.188+0000 7fcc2803c740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc201b55e0 con 0x55f5f0812c00 2024-03-28T18:04:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.188+0000 7fcc1f200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f081b290 0x55f5f081c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.188+0000 7fcc1f200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f081b290 0x55f5f081c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40744/0 (socket says 172.21.15.45:40744) 2024-03-28T18:04:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.188+0000 7fcc1f200640 1 -- 172.21.15.45:0/402738005 learned_addr learned my addr 172.21.15.45:0/402738005 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:04:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.188+0000 7fcc1e800640 1 --2- 172.21.15.45:0/402738005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5f0811f20 0x55f5f081a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.189+0000 7fcc15e00640 1 --2- 172.21.15.45:0/402738005 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f5f0812c00 0x55f5f081a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.189+0000 7fcc1f200640 1 -- 172.21.15.45:0/402738005 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f5f0812c00 msgr2=0x55f5f081a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:04:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.189+0000 7fcc1f200640 1 --2- 172.21.15.45:0/402738005 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f5f0812c00 0x55f5f081a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.189+0000 7fcc1f200640 1 -- 172.21.15.45:0/402738005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5f0811f20 msgr2=0x55f5f081a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:04:40.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.189+0000 7fcc1f200640 1 --2- 172.21.15.45:0/402738005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5f0811f20 0x55f5f081a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.189+0000 7fcc1f200640 1 -- 172.21.15.45:0/402738005 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5f044a0f0 con 0x55f5f081b290 2024-03-28T18:04:40.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.189+0000 7fcc1f200640 1 --2- 172.21.15.45:0/402738005 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f081b290 0x55f5f081c680 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fcc18009ca0 tx=0x7fcc18041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64b72e0b59849fbb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:04:40.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.189+0000 7fcc0ea00640 1 -- 172.21.15.45:0/402738005 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18039040 con 0x55f5f081b290 2024-03-28T18:04:40.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.189+0000 7fcc0ea00640 1 -- 172.21.15.45:0/402738005 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc18003b70 con 0x55f5f081b290 2024-03-28T18:04:40.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.189+0000 7fcc0ea00640 1 -- 172.21.15.45:0/402738005 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1804e880 con 0x55f5f081b290 2024-03-28T18:04:40.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.190+0000 7fcc2803c740 1 -- 172.21.15.45:0/402738005 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f081b290 msgr2=0x55f5f081c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:40.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.190+0000 7fcc2803c740 1 --2- 172.21.15.45:0/402738005 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f081b290 0x55f5f081c680 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fcc18009ca0 tx=0x7fcc18041000 comp rx=0 tx=0).stop 2024-03-28T18:04:40.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.190+0000 7fcc2803c740 1 -- 172.21.15.45:0/402738005 shutdown_connections 2024-03-28T18:04:40.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.190+0000 7fcc2803c740 1 --2- 172.21.15.45:0/402738005 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f081b290 0x55f5f081c680 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.190+0000 7fcc2803c740 1 --2- 172.21.15.45:0/402738005 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f5f0812c00 0x55f5f081a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.190+0000 7fcc2803c740 1 --2- 172.21.15.45:0/402738005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5f0811f20 0x55f5f081a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.190+0000 7fcc2803c740 1 -- 172.21.15.45:0/402738005 >> 172.21.15.45:0/402738005 conn(0x55f5f07d9200 msgr2=0x55f5f08165b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:04:40.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.190+0000 7fcc2803c740 1 -- 172.21.15.45:0/402738005 shutdown_connections 2024-03-28T18:04:40.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.190+0000 7fcc2803c740 1 -- 172.21.15.45:0/402738005 wait complete. 2024-03-28T18:04:40.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.190+0000 7fcc2803c740 1 -- start start 2024-03-28T18:04:40.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.191+0000 7fcc2803c740 1 Processor -- start 2024-03-28T18:04:40.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.191+0000 7fcc2803c740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f5f0811f20 0x55f5f0814b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:40.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.191+0000 7fcc2803c740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f0812c00 0x55f5f08150a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:40.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.191+0000 7fcc2803c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5f081b290 0x55f5f0885f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:40.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.191+0000 7fcc2803c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc201b4f80 con 0x55f5f081b290 2024-03-28T18:04:40.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.191+0000 7fcc2803c740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc201b4920 con 0x55f5f0812c00 2024-03-28T18:04:40.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.191+0000 7fcc2803c740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f0808030 con 0x55f5f0811f20 2024-03-28T18:04:40.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.191+0000 7fcc2803c740 10 client.0 _ll_register_callbacks cb 0x7fcc08009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:04:40.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.192+0000 7fcc15e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f0812c00 0x55f5f08150a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:40.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.192+0000 7fcc1f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5f081b290 0x55f5f0885f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:40.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.192+0000 7fcc15e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f0812c00 0x55f5f08150a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:40752/0 (socket says 172.21.15.45:40752) 2024-03-28T18:04:40.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.192+0000 7fcc15e00640 1 -- 172.21.15.45:0/585413881 learned_addr learned my addr 172.21.15.45:0/585413881 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:04:40.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.192+0000 7fcc1e800640 1 --2- 172.21.15.45:0/585413881 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f5f0811f20 0x55f5f0814b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:40.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.192+0000 7fcc15e00640 1 -- 172.21.15.45:0/585413881 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f5f0811f20 msgr2=0x55f5f0814b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:04:40.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.192+0000 7fcc15e00640 1 --2- 172.21.15.45:0/585413881 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f5f0811f20 0x55f5f0814b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.192+0000 7fcc15e00640 1 -- 172.21.15.45:0/585413881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5f081b290 msgr2=0x55f5f0885f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:04:40.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.192+0000 7fcc15e00640 1 --2- 172.21.15.45:0/585413881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5f081b290 0x55f5f0885f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.192+0000 7fcc15e00640 1 -- 172.21.15.45:0/585413881 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f5f08864b0 con 0x55f5f0812c00 2024-03-28T18:04:40.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.192+0000 7fcc15e00640 1 --2- 172.21.15.45:0/585413881 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f0812c00 0x55f5f08150a0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fcc08012740 tx=0x7fcc0801c6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1c08042a34a825d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:04:40.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.192+0000 7fcc03e00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T18:04:40.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.192+0000 7fcc03e00640 1 -- 172.21.15.45:0/585413881 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0800fe60 con 0x55f5f0812c00 2024-03-28T18:04:40.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.192+0000 7fcc03e00640 1 -- 172.21.15.45:0/585413881 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(147..147 src has 1..147) v4 ==== 12382+0+0 (secure 0 0 0) 0x7fcc0801cc90 con 0x55f5f0812c00 2024-03-28T18:04:40.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.192+0000 7fcc2803c740 1 -- 172.21.15.45:0/585413881 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcc18003b70 con 0x55f5f0812c00 2024-03-28T18:04:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.193+0000 7fcc03e00640 1 -- 172.21.15.45:0/585413881 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=319) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc08005040 con 0x55f5f0812c00 2024-03-28T18:04:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.193+0000 7fcc2803c740 10 client.26639 fetch_fsmap learned FSMap version 319 2024-03-28T18:04:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.193+0000 7fcc2803c740 1 -- 172.21.15.45:0/585413881 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=319,monmap=4+}) v3 -- 0x55f5f08877c0 con 0x55f5f0812c00 2024-03-28T18:04:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.193+0000 7fcc03e00640 1 -- 172.21.15.45:0/585413881 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc18003b70 con 0x55f5f0812c00 2024-03-28T18:04:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.193+0000 7fcc03e00640 1 -- 172.21.15.45:0/585413881 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 319) v1 ==== 3905+0+0 (secure 0 0 0) 0x7fcc08003a00 con 0x55f5f0812c00 2024-03-28T18:04:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.193+0000 7fcc2803c740 10 client.26639 fetch_fsmap finished waiting for FSMap version 319 2024-03-28T18:04:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.193+0000 7fcc2803c740 10 client.26639 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:04:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.193+0000 7fcc2803c740 20 client.26639 populate_metadata read hostname 'smithi045' 2024-03-28T18:04:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.193+0000 7fcc2803c740 1 --2- 172.21.15.45:0/585413881 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f5f0887eb0 0x55f5f088a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:04:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.193+0000 7fcc2803c740 4 client.26639 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:04:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.193+0000 7fcc2803c740 1 -- 172.21.15.45:0/585413881 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f5f077dae0 con 0x55f5f0887eb0 2024-03-28T18:04:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.193+0000 7fcc1e800640 1 --2- 172.21.15.45:0/585413881 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f5f0887eb0 0x55f5f088a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:04:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.194+0000 7fcc1e800640 1 --2- 172.21.15.45:0/585413881 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f5f0887eb0 0x55f5f088a2d0 crc :-1 s=READY pgs=425 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13ca556fc2355854 server_cookie=18d1438fc21092bc in_seq=0 out_seq=0 2024-03-28T18:04:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.194+0000 7fcc03e00640 10 client.26639 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:04:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.194+0000 7fcc03e00640 1 -- 172.21.15.45:0/585413881 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55f5f077dae0 con 0x55f5f0887eb0 2024-03-28T18:04:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.194+0000 7fcc03e00640 10 client.26639 handle_command_reply: tid=0 2024-03-28T18:04:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.194+0000 7fcc2803c740 1 client.26639 shutdown 2024-03-28T18:04:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.194+0000 7fcc2803c740 1 -- 172.21.15.45:0/585413881 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f0812c00 msgr2=0x55f5f08150a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.194+0000 7fcc2803c740 1 --2- 172.21.15.45:0/585413881 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f0812c00 0x55f5f08150a0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fcc08012740 tx=0x7fcc0801c6b0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.194+0000 7fcc2803c740 1 -- 172.21.15.45:0/585413881 shutdown_connections 2024-03-28T18:04:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.194+0000 7fcc2803c740 1 --2- 172.21.15.45:0/585413881 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x55f5f0887eb0 0x55f5f088a2d0 crc :-1 s=READY pgs=425 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.194+0000 7fcc2803c740 1 --2- 172.21.15.45:0/585413881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5f081b290 0x55f5f0885f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.194+0000 7fcc03e00640 0 client.26639 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:04:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.194+0000 7fcc2803c740 1 --2- 172.21.15.45:0/585413881 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f5f0812c00 0x55f5f08150a0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.194+0000 7fcc2803c740 1 --2- 172.21.15.45:0/585413881 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55f5f0811f20 0x55f5f0814b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.194+0000 7fcc2803c740 1 -- 172.21.15.45:0/585413881 >> 172.21.15.45:0/585413881 conn(0x55f5f07d9200 msgr2=0x55f5f080b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:04:40.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.194+0000 7fcc2803c740 1 -- 172.21.15.45:0/585413881 shutdown_connections 2024-03-28T18:04:40.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.194+0000 7fcc2803c740 1 -- 172.21.15.45:0/585413881 wait complete. 2024-03-28T18:04:40.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.194+0000 7fcc2803c740 20 client.26639 trim_cache size 0 max 16384 2024-03-28T18:04:40.201 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T18:04:40.201 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:04:40.201 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T18:04:40.201 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T18:04:40.202 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T18:04:40.202 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T18:04:40.202 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T18:04:40.202 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:04:40.202 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:04:40.202 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T18:04:40.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.195+0000 7fcc25400640 1 -- 172.21.15.45:0/3273857559 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fcc040a29f0 msgr2=0x7fcc040a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:40.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.195+0000 7fcc25400640 1 --2- 172.21.15.45:0/3273857559 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fcc040a29f0 0x7fcc040a4eb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcc18005e30 tx=0x7fcc18005d80 comp rx=0 tx=0).stop 2024-03-28T18:04:40.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.195+0000 7fcc25400640 1 -- 172.21.15.45:0/3273857559 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc200ffc10 msgr2=0x7fcc2019e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:04:40.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.195+0000 7fcc25400640 1 --2- 172.21.15.45:0/3273857559 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc200ffc10 0x7fcc2019e4d0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000d770 tx=0x7fcc1000dc30 comp rx=0 tx=0).stop 2024-03-28T18:04:40.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.195+0000 7fcc25400640 1 -- 172.21.15.45:0/3273857559 shutdown_connections 2024-03-28T18:04:40.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.195+0000 7fcc25400640 1 --2- 172.21.15.45:0/3273857559 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7fcc040a29f0 0x7fcc040a4eb0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.195+0000 7fcc25400640 1 --2- 172.21.15.45:0/3273857559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc201066e0 0x7fcc201a2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.195+0000 7fcc25400640 1 --2- 172.21.15.45:0/3273857559 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc201005d0 0x7fcc2019ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.195+0000 7fcc25400640 1 --2- 172.21.15.45:0/3273857559 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc200ffc10 0x7fcc2019e4d0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:04:40.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.195+0000 7fcc25400640 1 -- 172.21.15.45:0/3273857559 >> 172.21.15.45:0/3273857559 conn(0x7fcc20076bd0 msgr2=0x7fcc20101fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:04:40.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.195+0000 7fcc25400640 1 -- 172.21.15.45:0/3273857559 shutdown_connections 2024-03-28T18:04:40.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:04:40.195+0000 7fcc25400640 1 -- 172.21.15.45:0/3273857559 wait complete. 2024-03-28T18:04:40.216 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T18:04:40.954 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 733 sec 2024-03-28T18:04:41.954 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 734 sec 2024-03-28T18:04:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:41 smithi071 ceph-mon[36597]: pgmap v540: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:04:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:41 smithi151 ceph-mon[36481]: pgmap v540: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:04:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:41 smithi045 ceph-mon[31773]: pgmap v540: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:04:42.955 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 735 sec 2024-03-28T18:04:43.955 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 736 sec 2024-03-28T18:04:44.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:43 smithi071 ceph-mon[36597]: pgmap v541: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:04:44.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:43 smithi151 ceph-mon[36481]: pgmap v541: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:04:44.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:43 smithi045 ceph-mon[31773]: pgmap v541: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:04:44.955 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 737 sec 2024-03-28T18:04:45.955 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 738 sec 2024-03-28T18:04:46.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:45 smithi151 ceph-mon[36481]: pgmap v542: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:04:46.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:45 smithi045 ceph-mon[31773]: pgmap v542: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:04:46.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:45 smithi071 ceph-mon[36597]: pgmap v542: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:04:46.955 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 739 sec 2024-03-28T18:04:47.955 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 740 sec 2024-03-28T18:04:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:47 smithi151 ceph-mon[36481]: pgmap v543: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:04:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:47 smithi045 ceph-mon[31773]: pgmap v543: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:04:48.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:47 smithi071 ceph-mon[36597]: pgmap v543: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:04:48.955 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 741 sec 2024-03-28T18:04:49.955 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 742 sec 2024-03-28T18:04:50.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:49 smithi151 ceph-mon[36481]: pgmap v544: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:50.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:49 smithi045 ceph-mon[31773]: pgmap v544: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:50.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:49 smithi071 ceph-mon[36597]: pgmap v544: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:50.956 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 743 sec 2024-03-28T18:04:51.956 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 744 sec 2024-03-28T18:04:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:51 smithi045 ceph-mon[31773]: pgmap v545: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:04:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:51 smithi071 ceph-mon[36597]: pgmap v545: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:04:52.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:51 smithi151 ceph-mon[36481]: pgmap v545: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:04:52.956 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 745 sec 2024-03-28T18:04:53.956 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 746 sec 2024-03-28T18:04:54.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:53 smithi045 ceph-mon[31773]: pgmap v546: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:54.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:53 smithi071 ceph-mon[36597]: pgmap v546: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:54.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:53 smithi151 ceph-mon[36481]: pgmap v546: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:54.956 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 747 sec 2024-03-28T18:04:55.956 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 748 sec 2024-03-28T18:04:56.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:55 smithi151 ceph-mon[36481]: pgmap v547: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:56.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:55 smithi045 ceph-mon[31773]: pgmap v547: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:56.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:55 smithi071 ceph-mon[36597]: pgmap v547: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:04:56.956 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 749 sec 2024-03-28T18:04:57.957 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 750 sec 2024-03-28T18:04:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:57 smithi045 ceph-mon[31773]: pgmap v548: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:04:58.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:57 smithi071 ceph-mon[36597]: pgmap v548: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:04:58.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:57 smithi151 ceph-mon[36481]: pgmap v548: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:04:58.957 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 751 sec 2024-03-28T18:04:59.957 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 752 sec 2024-03-28T18:05:00.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:04:59 smithi045 ceph-mon[31773]: pgmap v549: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:00.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:04:59 smithi071 ceph-mon[36597]: pgmap v549: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:00.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:04:59 smithi151 ceph-mon[36481]: pgmap v549: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:00.957 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 753 sec 2024-03-28T18:05:01.957 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 754 sec 2024-03-28T18:05:02.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:01 smithi045 ceph-mon[31773]: pgmap v550: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:01 smithi071 ceph-mon[36597]: pgmap v550: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:01 smithi151 ceph-mon[36481]: pgmap v550: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:02.957 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 755 sec 2024-03-28T18:05:03.957 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 756 sec 2024-03-28T18:05:04.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:03 smithi045 ceph-mon[31773]: pgmap v551: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:04.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:03 smithi071 ceph-mon[36597]: pgmap v551: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:04.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:03 smithi151 ceph-mon[36481]: pgmap v551: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:04.958 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 757 sec 2024-03-28T18:05:05.958 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 758 sec 2024-03-28T18:05:06.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:05 smithi045 ceph-mon[31773]: pgmap v552: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:06.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:05 smithi071 ceph-mon[36597]: pgmap v552: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:06.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:05 smithi151 ceph-mon[36481]: pgmap v552: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:06.958 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 759 sec 2024-03-28T18:05:07.958 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 760 sec 2024-03-28T18:05:08.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:07 smithi045 ceph-mon[31773]: pgmap v553: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:08.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:07 smithi071 ceph-mon[36597]: pgmap v553: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:08.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:07 smithi151 ceph-mon[36481]: pgmap v553: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:08.958 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 761 sec 2024-03-28T18:05:09.958 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 762 sec 2024-03-28T18:05:10.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:09 smithi045 ceph-mon[31773]: pgmap v554: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:10.217 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:05:10.217 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:05:10.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:09 smithi071 ceph-mon[36597]: pgmap v554: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:10.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:09 smithi151 ceph-mon[36481]: pgmap v554: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:10.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.449+0000 7f0e0d200640 1 -- 172.21.15.45:0/4191398526 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e08108860 msgr2=0x7f0e0810c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:10.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.449+0000 7f0e0d200640 1 --2- 172.21.15.45:0/4191398526 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e08108860 0x7f0e0810c8a0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f0df4009900 tx=0x7f0df402bc30 comp rx=0 tx=0).stop 2024-03-28T18:05:10.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.449+0000 7f0e0d200640 1 -- 172.21.15.45:0/4191398526 shutdown_connections 2024-03-28T18:05:10.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.449+0000 7f0e0d200640 1 --2- 172.21.15.45:0/4191398526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e0810cf80 0x7f0e08114b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.449+0000 7f0e0d200640 1 --2- 172.21.15.45:0/4191398526 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e08108860 0x7f0e0810c8a0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.449+0000 7f0e0d200640 1 --2- 172.21.15.45:0/4191398526 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e08107eb0 0x7f0e08108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.449+0000 7f0e0d200640 1 -- 172.21.15.45:0/4191398526 >> 172.21.15.45:0/4191398526 conn(0x7f0e08075be0 msgr2=0x7f0e08078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:05:10.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.449+0000 7f0e0d200640 1 -- 172.21.15.45:0/4191398526 shutdown_connections 2024-03-28T18:05:10.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.449+0000 7f0e0d200640 1 -- 172.21.15.45:0/4191398526 wait complete. 2024-03-28T18:05:10.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.450+0000 7f0e0d200640 1 Processor -- start 2024-03-28T18:05:10.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.450+0000 7f0e0d200640 1 -- start start 2024-03-28T18:05:10.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.450+0000 7f0e0d200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e08107eb0 0x7f0e0819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:10.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.450+0000 7f0e0d200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e0810cf80 0x7f0e0819eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:10.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.450+0000 7f0e0d200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e081a2ed0 0x7f0e081a3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:10.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.450+0000 7f0e0d200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e08118080 con 0x7f0e0810cf80 2024-03-28T18:05:10.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.450+0000 7f0e0d200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e08117f00 con 0x7f0e08107eb0 2024-03-28T18:05:10.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.450+0000 7f0e0d200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e08118200 con 0x7f0e081a2ed0 2024-03-28T18:05:10.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.451+0000 7f0e05e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e08107eb0 0x7f0e0819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:10.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.451+0000 7f0e05e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e08107eb0 0x7f0e0819e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:60714/0 (socket says 172.21.15.45:60714) 2024-03-28T18:05:10.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.451+0000 7f0e05e00640 1 -- 172.21.15.45:0/2903263731 learned_addr learned my addr 172.21.15.45:0/2903263731 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:05:10.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.451+0000 7f0e05400640 1 --2- 172.21.15.45:0/2903263731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e0810cf80 0x7f0e0819eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:10.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.451+0000 7f0e06800640 1 --2- 172.21.15.45:0/2903263731 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e081a2ed0 0x7f0e081a3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:10.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.451+0000 7f0e05e00640 1 -- 172.21.15.45:0/2903263731 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e081a2ed0 msgr2=0x7f0e081a3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:10.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.451+0000 7f0e05e00640 1 --2- 172.21.15.45:0/2903263731 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e081a2ed0 0x7f0e081a3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.451+0000 7f0e05e00640 1 -- 172.21.15.45:0/2903263731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e0810cf80 msgr2=0x7f0e0819eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:10.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.451+0000 7f0e05e00640 1 --2- 172.21.15.45:0/2903263731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e0810cf80 0x7f0e0819eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.451+0000 7f0e05e00640 1 -- 172.21.15.45:0/2903263731 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e081a39c0 con 0x7f0e08107eb0 2024-03-28T18:05:10.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.451+0000 7f0e06800640 1 --2- 172.21.15.45:0/2903263731 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e081a2ed0 0x7f0e081a3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:05:10.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.451+0000 7f0e05e00640 1 --2- 172.21.15.45:0/2903263731 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e08107eb0 0x7f0e0819e660 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f0df800e9f0 tx=0x7f0df800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:05:10.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.451+0000 7f0dfea00640 1 -- 172.21.15.45:0/2903263731 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df800cd80 con 0x7f0e08107eb0 2024-03-28T18:05:10.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.452+0000 7f0dfea00640 1 -- 172.21.15.45:0/2903263731 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0df8005500 con 0x7f0e08107eb0 2024-03-28T18:05:10.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.452+0000 7f0e0d200640 1 -- 172.21.15.45:0/2903263731 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e081a3cb0 con 0x7f0e08107eb0 2024-03-28T18:05:10.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.452+0000 7f0dfea00640 1 -- 172.21.15.45:0/2903263731 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df8010690 con 0x7f0e08107eb0 2024-03-28T18:05:10.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.452+0000 7f0e0d200640 1 -- 172.21.15.45:0/2903263731 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e081ab420 con 0x7f0e08107eb0 2024-03-28T18:05:10.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.455+0000 7f0dfea00640 1 -- 172.21.15.45:0/2903263731 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f0df80046c0 con 0x7f0e08107eb0 2024-03-28T18:05:10.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.455+0000 7f0e0fff0740 1 Processor -- start 2024-03-28T18:05:10.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.455+0000 7f0e0fff0740 1 -- start start 2024-03-28T18:05:10.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.455+0000 7f0e0fff0740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5646fca78c00 0x5646fca77950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:10.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.455+0000 7f0e0fff0740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca77f20 0x5646fca783a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:10.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.455+0000 7f0e0fff0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5646fca81290 0x5646fca83680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:10.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.455+0000 7f0e0fff0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646fc9dc250 con 0x5646fca81290 2024-03-28T18:05:10.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.455+0000 7f0e0fff0740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646fc984a30 con 0x5646fca77f20 2024-03-28T18:05:10.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.455+0000 7f0e0fff0740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646fc9d82f0 con 0x5646fca78c00 2024-03-28T18:05:10.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.456+0000 7f0e05e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca77f20 0x5646fca783a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:10.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.456+0000 7f0e05e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca77f20 0x5646fca783a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:60716/0 (socket says 172.21.15.45:60716) 2024-03-28T18:05:10.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.456+0000 7f0e05e00640 1 -- 172.21.15.45:0/2949441763 learned_addr learned my addr 172.21.15.45:0/2949441763 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:05:10.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.456+0000 7f0e06800640 1 --2- 172.21.15.45:0/2949441763 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5646fca78c00 0x5646fca77950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:10.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.456+0000 7f0e05400640 1 --2- 172.21.15.45:0/2949441763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5646fca81290 0x5646fca83680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:10.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.456+0000 7f0e05e00640 1 -- 172.21.15.45:0/2949441763 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5646fca78c00 msgr2=0x5646fca77950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:05:10.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.456+0000 7f0e05e00640 1 --2- 172.21.15.45:0/2949441763 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5646fca78c00 0x5646fca77950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.456+0000 7f0e05e00640 1 -- 172.21.15.45:0/2949441763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5646fca81290 msgr2=0x5646fca83680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:05:10.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.456+0000 7f0e05e00640 1 --2- 172.21.15.45:0/2949441763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5646fca81290 0x5646fca83680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.456+0000 7f0e05e00640 1 -- 172.21.15.45:0/2949441763 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5646fc96f690 con 0x5646fca77f20 2024-03-28T18:05:10.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.456+0000 7f0e05e00640 1 --2- 172.21.15.45:0/2949441763 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca77f20 0x5646fca783a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f0df80cc9a0 tx=0x7f0df80d83b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d007f92436d47a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:05:10.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.456+0000 7f0dfea00640 1 --2- 172.21.15.45:0/2903263731 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f0dec0a2970 0x7f0dec0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:10.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.457+0000 7f0dfea00640 1 -- 172.21.15.45:0/2903263731 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 12382+0+0 (secure 0 0 0) 0x7f0df80d0280 con 0x7f0e08107eb0 2024-03-28T18:05:10.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.457+0000 7f0e06800640 1 --2- 172.21.15.45:0/2903263731 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f0dec0a2970 0x7f0dec0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:10.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.457+0000 7f0df3400640 1 -- 172.21.15.45:0/2949441763 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df80e3070 con 0x5646fca77f20 2024-03-28T18:05:10.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.457+0000 7f0df3400640 1 -- 172.21.15.45:0/2949441763 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0df809c070 con 0x5646fca77f20 2024-03-28T18:05:10.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.457+0000 7f0df3400640 1 -- 172.21.15.45:0/2949441763 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df809b4b0 con 0x5646fca77f20 2024-03-28T18:05:10.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.458+0000 7f0e0fff0740 1 -- 172.21.15.45:0/2949441763 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca77f20 msgr2=0x5646fca783a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:10.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.458+0000 7f0e0fff0740 1 --2- 172.21.15.45:0/2949441763 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca77f20 0x5646fca783a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f0df80cc9a0 tx=0x7f0df80d83b0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.458+0000 7f0e06800640 1 --2- 172.21.15.45:0/2903263731 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f0dec0a2970 0x7f0dec0a4e30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0e00005fd0 tx=0x7f0e00009500 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:05:10.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.458+0000 7f0e0fff0740 1 -- 172.21.15.45:0/2949441763 shutdown_connections 2024-03-28T18:05:10.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.458+0000 7f0e0fff0740 1 --2- 172.21.15.45:0/2949441763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5646fca81290 0x5646fca83680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.458+0000 7f0e0fff0740 1 --2- 172.21.15.45:0/2949441763 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca77f20 0x5646fca783a0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.458+0000 7f0e0fff0740 1 --2- 172.21.15.45:0/2949441763 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5646fca78c00 0x5646fca77950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.458+0000 7f0e0fff0740 1 -- 172.21.15.45:0/2949441763 >> 172.21.15.45:0/2949441763 conn(0x5646fca3f200 msgr2=0x5646fca72970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:05:10.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.458+0000 7f0e0fff0740 1 -- 172.21.15.45:0/2949441763 shutdown_connections 2024-03-28T18:05:10.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.458+0000 7f0e0fff0740 1 -- 172.21.15.45:0/2949441763 wait complete. 2024-03-28T18:05:10.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.458+0000 7f0e0fff0740 1 -- start start 2024-03-28T18:05:10.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.458+0000 7f0e0fff0740 1 Processor -- start 2024-03-28T18:05:10.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.459+0000 7f0e0fff0740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca77f20 0x5646fca80960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:10.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.459+0000 7f0e0fff0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5646fca78c00 0x5646fca7dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:10.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.459+0000 7f0e0fff0740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5646fca81290 0x5646fca7e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:10.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.459+0000 7f0e0fff0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646fca84100 con 0x5646fca78c00 2024-03-28T18:05:10.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.459+0000 7f0e0fff0740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646fc997e80 con 0x5646fca77f20 2024-03-28T18:05:10.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.459+0000 7f0e0fff0740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646fca84280 con 0x5646fca81290 2024-03-28T18:05:10.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.459+0000 7f0e0fff0740 10 client.0 _ll_register_callbacks cb 0x5646fc9aa130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:05:10.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.459+0000 7f0e05e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca77f20 0x5646fca80960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:10.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.459+0000 7f0e05e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca77f20 0x5646fca80960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:60730/0 (socket says 172.21.15.45:60730) 2024-03-28T18:05:10.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.459+0000 7f0e05e00640 1 -- 172.21.15.45:0/1202242889 learned_addr learned my addr 172.21.15.45:0/1202242889 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:05:10.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.459+0000 7f0e05400640 1 --2- 172.21.15.45:0/1202242889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5646fca78c00 0x5646fca7dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:10.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.459+0000 7f0e06800640 1 --2- 172.21.15.45:0/1202242889 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5646fca81290 0x5646fca7e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:10.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.459+0000 7f0e05e00640 1 -- 172.21.15.45:0/1202242889 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5646fca81290 msgr2=0x5646fca7e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:05:10.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.459+0000 7f0e05e00640 1 --2- 172.21.15.45:0/1202242889 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5646fca81290 0x5646fca7e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.460+0000 7f0e05e00640 1 -- 172.21.15.45:0/1202242889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5646fca78c00 msgr2=0x5646fca7dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:05:10.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.460+0000 7f0e05e00640 1 --2- 172.21.15.45:0/1202242889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5646fca78c00 0x5646fca7dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.460+0000 7f0e05e00640 1 -- 172.21.15.45:0/1202242889 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5646fc7ba7d0 con 0x5646fca77f20 2024-03-28T18:05:10.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.460+0000 7f0e06800640 1 --2- 172.21.15.45:0/1202242889 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5646fca81290 0x5646fca7e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:05:10.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.460+0000 7f0e05e00640 1 --2- 172.21.15.45:0/1202242889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca77f20 0x5646fca80960 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0df80049d0 tx=0x7f0df80d0500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30535db0210ad9be server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:05:10.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.460+0000 7f0de7e00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T18:05:10.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.460+0000 7f0de7e00640 1 -- 172.21.15.45:0/1202242889 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df809b2d0 con 0x5646fca77f20 2024-03-28T18:05:10.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.460+0000 7f0de7e00640 1 -- 172.21.15.45:0/1202242889 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(147..147 src has 1..147) v4 ==== 12382+0+0 (secure 0 0 0) 0x7f0df80ccce0 con 0x5646fca77f20 2024-03-28T18:05:10.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.460+0000 7f0e0fff0740 1 -- 172.21.15.45:0/1202242889 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0df809c070 con 0x5646fca77f20 2024-03-28T18:05:10.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.460+0000 7f0de7e00640 1 -- 172.21.15.45:0/1202242889 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=319) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0df809b4b0 con 0x5646fca77f20 2024-03-28T18:05:10.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.460+0000 7f0e0fff0740 10 client.26651 fetch_fsmap learned FSMap version 319 2024-03-28T18:05:10.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.460+0000 7f0e0fff0740 1 -- 172.21.15.45:0/1202242889 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=319,monmap=4+}) v3 -- 0x5646fc6f03f0 con 0x5646fca77f20 2024-03-28T18:05:10.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.461+0000 7f0de7e00640 1 -- 172.21.15.45:0/1202242889 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0df809c070 con 0x5646fca77f20 2024-03-28T18:05:10.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.461+0000 7f0de7e00640 1 -- 172.21.15.45:0/1202242889 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 319) v1 ==== 3905+0+0 (secure 0 0 0) 0x7f0df809bbd0 con 0x5646fca77f20 2024-03-28T18:05:10.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.461+0000 7f0e0fff0740 10 client.26651 fetch_fsmap finished waiting for FSMap version 319 2024-03-28T18:05:10.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.461+0000 7f0e0fff0740 10 client.26651 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:05:10.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.461+0000 7f0e0fff0740 20 client.26651 populate_metadata read hostname 'smithi045' 2024-03-28T18:05:10.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.461+0000 7f0e0fff0740 1 --2- 172.21.15.45:0/1202242889 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5646fca91050 0x5646fca93440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:10.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.461+0000 7f0e0fff0740 4 client.26651 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:05:10.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.461+0000 7f0e0fff0740 1 -- 172.21.15.45:0/1202242889 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5646fc9e9790 con 0x5646fca91050 2024-03-28T18:05:10.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.461+0000 7f0e05400640 1 --2- 172.21.15.45:0/1202242889 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5646fca91050 0x5646fca93440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:10.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.462+0000 7f0e05400640 1 --2- 172.21.15.45:0/1202242889 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5646fca91050 0x5646fca93440 crc :-1 s=READY pgs=428 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2101d36df9b8cb4e server_cookie=31d36ad996cae5c7 in_seq=0 out_seq=0 2024-03-28T18:05:10.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.462+0000 7f0de7e00640 10 client.26651 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:05:10.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.463+0000 7f0de7e00640 1 -- 172.21.15.45:0/1202242889 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5646fc9e9790 con 0x5646fca91050 2024-03-28T18:05:10.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.463+0000 7f0de7e00640 10 client.26651 handle_command_reply: tid=0 2024-03-28T18:05:10.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.463+0000 7f0e0fff0740 1 client.26651 shutdown 2024-03-28T18:05:10.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.463+0000 7f0e0fff0740 1 -- 172.21.15.45:0/1202242889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca77f20 msgr2=0x5646fca80960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:10.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.463+0000 7f0e0fff0740 1 --2- 172.21.15.45:0/1202242889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca77f20 0x5646fca80960 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0df80049d0 tx=0x7f0df80d0500 comp rx=0 tx=0).stop 2024-03-28T18:05:10.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.463+0000 7f0e0fff0740 1 -- 172.21.15.45:0/1202242889 shutdown_connections 2024-03-28T18:05:10.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.463+0000 7f0e0fff0740 1 --2- 172.21.15.45:0/1202242889 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5646fca91050 0x5646fca93440 crc :-1 s=READY pgs=428 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.463+0000 7f0e0fff0740 1 --2- 172.21.15.45:0/1202242889 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5646fca81290 0x5646fca7e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.463+0000 7f0e0fff0740 1 --2- 172.21.15.45:0/1202242889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5646fca78c00 0x5646fca7dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.463+0000 7f0de7e00640 0 client.26651 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:05:10.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.463+0000 7f0e0fff0740 1 --2- 172.21.15.45:0/1202242889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca77f20 0x5646fca80960 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.463+0000 7f0e0fff0740 1 -- 172.21.15.45:0/1202242889 >> 172.21.15.45:0/1202242889 conn(0x5646fca3f200 msgr2=0x5646fca746c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:05:10.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.463+0000 7f0e0fff0740 1 -- 172.21.15.45:0/1202242889 shutdown_connections 2024-03-28T18:05:10.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.463+0000 7f0e0fff0740 1 -- 172.21.15.45:0/1202242889 wait complete. 2024-03-28T18:05:10.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.463+0000 7f0e0fff0740 20 client.26651 trim_cache size 0 max 16384 2024-03-28T18:05:10.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.481+0000 7f0e0fff0740 1 Processor -- start 2024-03-28T18:05:10.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.481+0000 7f0e0fff0740 1 -- start start 2024-03-28T18:05:10.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.481+0000 7f0e0fff0740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5646fca77f20 0x5646fca80420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:10.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.481+0000 7f0e0fff0740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca78c00 0x5646fca80960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:10.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.481+0000 7f0e0fff0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5646fca81290 0x5646fca82680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:10.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.481+0000 7f0e0fff0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e081b5ce0 con 0x5646fca81290 2024-03-28T18:05:10.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.481+0000 7f0e0fff0740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e081b4360 con 0x5646fca78c00 2024-03-28T18:05:10.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.481+0000 7f0e0fff0740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e081b5680 con 0x5646fca77f20 2024-03-28T18:05:10.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.481+0000 7f0e05400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca78c00 0x5646fca80960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:10.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.481+0000 7f0e05400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca78c00 0x5646fca80960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:60738/0 (socket says 172.21.15.45:60738) 2024-03-28T18:05:10.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.482+0000 7f0e05400640 1 -- 172.21.15.45:0/1977957120 learned_addr learned my addr 172.21.15.45:0/1977957120 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:05:10.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.482+0000 7f0e06800640 1 --2- 172.21.15.45:0/1977957120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5646fca81290 0x5646fca82680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:10.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.482+0000 7f0e05400640 1 -- 172.21.15.45:0/1977957120 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5646fca77f20 msgr2=0x5646fca80420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:05:10.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.482+0000 7f0e05e00640 1 --2- 172.21.15.45:0/1977957120 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5646fca77f20 0x5646fca80420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:10.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.482+0000 7f0e05400640 1 --2- 172.21.15.45:0/1977957120 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5646fca77f20 0x5646fca80420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.482+0000 7f0e05400640 1 -- 172.21.15.45:0/1977957120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5646fca81290 msgr2=0x5646fca82680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:05:10.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.482+0000 7f0e05400640 1 --2- 172.21.15.45:0/1977957120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5646fca81290 0x5646fca82680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.482+0000 7f0e05400640 1 -- 172.21.15.45:0/1977957120 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5646fc6b00f0 con 0x5646fca78c00 2024-03-28T18:05:10.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.482+0000 7f0e06800640 1 --2- 172.21.15.45:0/1977957120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5646fca81290 0x5646fca82680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:05:10.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.482+0000 7f0e05e00640 1 --2- 172.21.15.45:0/1977957120 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5646fca77f20 0x5646fca80420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:05:10.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.482+0000 7f0e05400640 1 --2- 172.21.15.45:0/1977957120 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca78c00 0x5646fca80960 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f0df40096c0 tx=0x7f0df402df90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=677f740623384e24 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:05:10.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.482+0000 7f0df2a00640 1 -- 172.21.15.45:0/1977957120 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df4004040 con 0x5646fca78c00 2024-03-28T18:05:10.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.482+0000 7f0df2a00640 1 -- 172.21.15.45:0/1977957120 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0df4051030 con 0x5646fca78c00 2024-03-28T18:05:10.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.482+0000 7f0df2a00640 1 -- 172.21.15.45:0/1977957120 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df4004ae0 con 0x5646fca78c00 2024-03-28T18:05:10.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.483+0000 7f0e0fff0740 1 -- 172.21.15.45:0/1977957120 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca78c00 msgr2=0x5646fca80960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:10.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.483+0000 7f0e0fff0740 1 --2- 172.21.15.45:0/1977957120 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca78c00 0x5646fca80960 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f0df40096c0 tx=0x7f0df402df90 comp rx=0 tx=0).stop 2024-03-28T18:05:10.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.483+0000 7f0e0fff0740 1 -- 172.21.15.45:0/1977957120 shutdown_connections 2024-03-28T18:05:10.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.483+0000 7f0e0fff0740 1 --2- 172.21.15.45:0/1977957120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5646fca81290 0x5646fca82680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.483+0000 7f0e0fff0740 1 --2- 172.21.15.45:0/1977957120 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca78c00 0x5646fca80960 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.483+0000 7f0e0fff0740 1 --2- 172.21.15.45:0/1977957120 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5646fca77f20 0x5646fca80420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.483+0000 7f0e0fff0740 1 -- 172.21.15.45:0/1977957120 >> 172.21.15.45:0/1977957120 conn(0x5646fca3f200 msgr2=0x5646fca7c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:05:10.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.483+0000 7f0e0fff0740 1 -- 172.21.15.45:0/1977957120 shutdown_connections 2024-03-28T18:05:10.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.483+0000 7f0e0fff0740 1 -- 172.21.15.45:0/1977957120 wait complete. 2024-03-28T18:05:10.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.483+0000 7f0e0fff0740 1 -- start start 2024-03-28T18:05:10.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.484+0000 7f0e0fff0740 1 Processor -- start 2024-03-28T18:05:10.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.484+0000 7f0e0fff0740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5646fca77f20 0x5646fca7a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:10.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.484+0000 7f0e0fff0740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca78c00 0x5646fca7ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:10.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.484+0000 7f0e0fff0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5646fca81290 0x5646fca7b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:10.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.484+0000 7f0e0fff0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e081b5020 con 0x5646fca81290 2024-03-28T18:05:10.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.484+0000 7f0e0fff0740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e081b49c0 con 0x5646fca78c00 2024-03-28T18:05:10.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.484+0000 7f0e0fff0740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646fca93a70 con 0x5646fca77f20 2024-03-28T18:05:10.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.484+0000 7f0e0fff0740 10 client.0 _ll_register_callbacks cb 0x7f0df80e1070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:05:10.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.484+0000 7f0e05400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca78c00 0x5646fca7ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:10.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.484+0000 7f0e06800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5646fca81290 0x5646fca7b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:10.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.484+0000 7f0e05e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5646fca77f20 0x5646fca7a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:10.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.484+0000 7f0e05400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca78c00 0x5646fca7ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:60740/0 (socket says 172.21.15.45:60740) 2024-03-28T18:05:10.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.484+0000 7f0e05400640 1 -- 172.21.15.45:0/1457591720 learned_addr learned my addr 172.21.15.45:0/1457591720 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:05:10.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.485+0000 7f0e05400640 1 -- 172.21.15.45:0/1457591720 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5646fca77f20 msgr2=0x5646fca7a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:05:10.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.485+0000 7f0e05400640 1 --2- 172.21.15.45:0/1457591720 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5646fca77f20 0x5646fca7a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.485+0000 7f0e05400640 1 -- 172.21.15.45:0/1457591720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5646fca81290 msgr2=0x5646fca7b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:05:10.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.485+0000 7f0e05400640 1 --2- 172.21.15.45:0/1457591720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5646fca81290 0x5646fca7b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.485+0000 7f0e05400640 1 -- 172.21.15.45:0/1457591720 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5646fcaebb00 con 0x5646fca78c00 2024-03-28T18:05:10.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.485+0000 7f0e05e00640 1 --2- 172.21.15.45:0/1457591720 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5646fca77f20 0x5646fca7a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:05:10.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.485+0000 7f0e05400640 1 --2- 172.21.15.45:0/1457591720 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca78c00 0x5646fca7ab60 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f0df4005ec0 tx=0x7f0df4004950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=613e4fe2fee5ec9c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:05:10.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.485+0000 7f0de7e00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T18:05:10.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.485+0000 7f0de7e00640 1 -- 172.21.15.45:0/1457591720 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df402c880 con 0x5646fca78c00 2024-03-28T18:05:10.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.486+0000 7f0de7e00640 1 -- 172.21.15.45:0/1457591720 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(147..147 src has 1..147) v4 ==== 12382+0+0 (secure 0 0 0) 0x7f0df402ca20 con 0x5646fca78c00 2024-03-28T18:05:10.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.486+0000 7f0e0fff0740 1 -- 172.21.15.45:0/1457591720 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0df4051030 con 0x5646fca78c00 2024-03-28T18:05:10.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.486+0000 7f0de7e00640 1 -- 172.21.15.45:0/1457591720 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=319) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0df402cc20 con 0x5646fca78c00 2024-03-28T18:05:10.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.486+0000 7f0e0fff0740 10 client.26657 fetch_fsmap learned FSMap version 319 2024-03-28T18:05:10.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.486+0000 7f0e0fff0740 1 -- 172.21.15.45:0/1457591720 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=319,monmap=4+}) v3 -- 0x5646fcaecd60 con 0x5646fca78c00 2024-03-28T18:05:10.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.486+0000 7f0de7e00640 1 -- 172.21.15.45:0/1457591720 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0df4051030 con 0x5646fca78c00 2024-03-28T18:05:10.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.486+0000 7f0de7e00640 1 -- 172.21.15.45:0/1457591720 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 319) v1 ==== 3905+0+0 (secure 0 0 0) 0x7f0df4068020 con 0x5646fca78c00 2024-03-28T18:05:10.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.486+0000 7f0e0fff0740 10 client.26657 fetch_fsmap finished waiting for FSMap version 319 2024-03-28T18:05:10.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.486+0000 7f0e0fff0740 10 client.26657 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:05:10.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.486+0000 7f0e0fff0740 20 client.26657 populate_metadata read hostname 'smithi045' 2024-03-28T18:05:10.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.486+0000 7f0e0fff0740 1 --2- 172.21.15.45:0/1457591720 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5646fcaed460 0x5646fcaef880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:10.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.486+0000 7f0e0fff0740 4 client.26657 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:05:10.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.486+0000 7f0e0fff0740 1 -- 172.21.15.45:0/1457591720 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5646fc9e3ae0 con 0x5646fcaed460 2024-03-28T18:05:10.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.486+0000 7f0e05e00640 1 --2- 172.21.15.45:0/1457591720 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5646fcaed460 0x5646fcaef880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:10.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.487+0000 7f0e05e00640 1 --2- 172.21.15.45:0/1457591720 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5646fcaed460 0x5646fcaef880 crc :-1 s=READY pgs=429 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7a4bf41a3817cc4 server_cookie=7fd7f0e274b1545a in_seq=0 out_seq=0 2024-03-28T18:05:10.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.487+0000 7f0de7e00640 10 client.26657 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:05:10.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.487+0000 7f0de7e00640 1 -- 172.21.15.45:0/1457591720 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5646fc9e3ae0 con 0x5646fcaed460 2024-03-28T18:05:10.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.487+0000 7f0de7e00640 10 client.26657 handle_command_reply: tid=0 2024-03-28T18:05:10.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.487+0000 7f0e0fff0740 1 client.26657 shutdown 2024-03-28T18:05:10.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.487+0000 7f0e0fff0740 1 -- 172.21.15.45:0/1457591720 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca78c00 msgr2=0x5646fca7ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:10.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.487+0000 7f0e0fff0740 1 --2- 172.21.15.45:0/1457591720 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca78c00 0x5646fca7ab60 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f0df4005ec0 tx=0x7f0df4004950 comp rx=0 tx=0).stop 2024-03-28T18:05:10.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.487+0000 7f0e0fff0740 1 -- 172.21.15.45:0/1457591720 shutdown_connections 2024-03-28T18:05:10.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.487+0000 7f0e0fff0740 1 --2- 172.21.15.45:0/1457591720 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5646fcaed460 0x5646fcaef880 crc :-1 s=READY pgs=429 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.487+0000 7f0e0fff0740 1 --2- 172.21.15.45:0/1457591720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5646fca81290 0x5646fca7b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.487+0000 7f0e0fff0740 1 --2- 172.21.15.45:0/1457591720 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5646fca78c00 0x5646fca7ab60 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.487+0000 7f0e0fff0740 1 --2- 172.21.15.45:0/1457591720 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5646fca77f20 0x5646fca7a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.487+0000 7f0de7e00640 0 client.26657 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:05:10.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.487+0000 7f0e0fff0740 1 -- 172.21.15.45:0/1457591720 >> 172.21.15.45:0/1457591720 conn(0x5646fca3f200 msgr2=0x5646fca71200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:05:10.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.487+0000 7f0e0fff0740 1 -- 172.21.15.45:0/1457591720 shutdown_connections 2024-03-28T18:05:10.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.487+0000 7f0e0fff0740 1 -- 172.21.15.45:0/1457591720 wait complete. 2024-03-28T18:05:10.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.488+0000 7f0e0fff0740 20 client.26657 trim_cache size 0 max 16384 2024-03-28T18:05:10.492 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T18:05:10.492 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:05:10.493 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T18:05:10.493 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T18:05:10.493 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T18:05:10.493 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T18:05:10.493 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T18:05:10.493 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:05:10.493 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:05:10.493 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T18:05:10.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.488+0000 7f0e0d200640 1 -- 172.21.15.45:0/2903263731 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f0dec0a2970 msgr2=0x7f0dec0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:10.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.488+0000 7f0e0d200640 1 --2- 172.21.15.45:0/2903263731 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f0dec0a2970 0x7f0dec0a4e30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0e00005fd0 tx=0x7f0e00009500 comp rx=0 tx=0).stop 2024-03-28T18:05:10.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.488+0000 7f0e0d200640 1 -- 172.21.15.45:0/2903263731 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e08107eb0 msgr2=0x7f0e0819e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:10.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.488+0000 7f0e0d200640 1 --2- 172.21.15.45:0/2903263731 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e08107eb0 0x7f0e0819e660 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f0df800e9f0 tx=0x7f0df800edf0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.488+0000 7f0e0d200640 1 -- 172.21.15.45:0/2903263731 shutdown_connections 2024-03-28T18:05:10.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.488+0000 7f0e0d200640 1 --2- 172.21.15.45:0/2903263731 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f0dec0a2970 0x7f0dec0a4e30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.488+0000 7f0e0d200640 1 --2- 172.21.15.45:0/2903263731 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e081a2ed0 0x7f0e081a3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.488+0000 7f0e0d200640 1 --2- 172.21.15.45:0/2903263731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e0810cf80 0x7f0e0819eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.488+0000 7f0e0d200640 1 --2- 172.21.15.45:0/2903263731 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e08107eb0 0x7f0e0819e660 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:10.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.488+0000 7f0e0d200640 1 -- 172.21.15.45:0/2903263731 >> 172.21.15.45:0/2903263731 conn(0x7f0e08075be0 msgr2=0x7f0e08077f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:05:10.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.488+0000 7f0e0d200640 1 -- 172.21.15.45:0/2903263731 shutdown_connections 2024-03-28T18:05:10.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:10.488+0000 7f0e0d200640 1 -- 172.21.15.45:0/2903263731 wait complete. 2024-03-28T18:05:10.505 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T18:05:10.958 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 763 sec 2024-03-28T18:05:11.958 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 764 sec 2024-03-28T18:05:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:11 smithi045 ceph-mon[31773]: pgmap v555: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:11 smithi071 ceph-mon[36597]: pgmap v555: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:11 smithi151 ceph-mon[36481]: pgmap v555: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:12.959 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 765 sec 2024-03-28T18:05:13.959 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 766 sec 2024-03-28T18:05:14.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:13 smithi045 ceph-mon[31773]: pgmap v556: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:14.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:13 smithi071 ceph-mon[36597]: pgmap v556: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:14.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:13 smithi151 ceph-mon[36481]: pgmap v556: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:14.959 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 767 sec 2024-03-28T18:05:15.959 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 768 sec 2024-03-28T18:05:16.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:15 smithi045 ceph-mon[31773]: pgmap v557: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:16.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:15 smithi071 ceph-mon[36597]: pgmap v557: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:16.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:15 smithi151 ceph-mon[36481]: pgmap v557: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:16.959 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 769 sec 2024-03-28T18:05:17.959 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 770 sec 2024-03-28T18:05:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:17 smithi045 ceph-mon[31773]: pgmap v558: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:18.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:17 smithi071 ceph-mon[36597]: pgmap v558: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:18.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:17 smithi151 ceph-mon[36481]: pgmap v558: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:18.960 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 771 sec 2024-03-28T18:05:19.959 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 772 sec 2024-03-28T18:05:20.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:19 smithi045 ceph-mon[31773]: pgmap v559: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:20.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:19 smithi071 ceph-mon[36597]: pgmap v559: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:20.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:19 smithi151 ceph-mon[36481]: pgmap v559: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:20.960 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 773 sec 2024-03-28T18:05:21.960 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 774 sec 2024-03-28T18:05:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:21 smithi045 ceph-mon[31773]: pgmap v560: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:21 smithi071 ceph-mon[36597]: pgmap v560: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:21 smithi151 ceph-mon[36481]: pgmap v560: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:22.960 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 775 sec 2024-03-28T18:05:23.960 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 776 sec 2024-03-28T18:05:24.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:23 smithi045 ceph-mon[31773]: pgmap v561: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:24.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:23 smithi071 ceph-mon[36597]: pgmap v561: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:24.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:23 smithi151 ceph-mon[36481]: pgmap v561: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:24.960 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 777 sec 2024-03-28T18:05:25.960 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 778 sec 2024-03-28T18:05:26.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:25 smithi045 ceph-mon[31773]: pgmap v562: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:26.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:25 smithi071 ceph-mon[36597]: pgmap v562: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:25 smithi151 ceph-mon[36481]: pgmap v562: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:26.961 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 779 sec 2024-03-28T18:05:27.961 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 780 sec 2024-03-28T18:05:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:27 smithi045 ceph-mon[31773]: pgmap v563: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:28.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:27 smithi071 ceph-mon[36597]: pgmap v563: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:28.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:27 smithi151 ceph-mon[36481]: pgmap v563: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:28.961 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 781 sec 2024-03-28T18:05:29.961 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 782 sec 2024-03-28T18:05:30.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:29 smithi045 ceph-mon[31773]: pgmap v564: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:30.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:29 smithi071 ceph-mon[36597]: pgmap v564: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:30.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:29 smithi151 ceph-mon[36481]: pgmap v564: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:30.961 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 783 sec 2024-03-28T18:05:31.961 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 784 sec 2024-03-28T18:05:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:31 smithi071 ceph-mon[36597]: pgmap v565: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:32.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:31 smithi151 ceph-mon[36481]: pgmap v565: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:32.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:31 smithi045 ceph-mon[31773]: pgmap v565: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:32.961 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 785 sec 2024-03-28T18:05:33.961 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 786 sec 2024-03-28T18:05:34.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:33 smithi071 ceph-mon[36597]: pgmap v566: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:34.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:33 smithi151 ceph-mon[36481]: pgmap v566: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:34.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:33 smithi045 ceph-mon[31773]: pgmap v566: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:34.962 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 787 sec 2024-03-28T18:05:35.962 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 788 sec 2024-03-28T18:05:36.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:35 smithi071 ceph-mon[36597]: pgmap v567: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:36.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:35 smithi151 ceph-mon[36481]: pgmap v567: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:36.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:35 smithi045 ceph-mon[31773]: pgmap v567: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:36.962 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 789 sec 2024-03-28T18:05:37.962 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 790 sec 2024-03-28T18:05:38.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:37 smithi071 ceph-mon[36597]: pgmap v568: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:38.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:37 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:05:38.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:37 smithi151 ceph-mon[36481]: pgmap v568: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:38.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:37 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:05:38.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:37 smithi045 ceph-mon[31773]: pgmap v568: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:38.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:37 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:05:38.962 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 791 sec 2024-03-28T18:05:39.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:38 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:39.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:38 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:05:39.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:38 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:05:39.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:38 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:39.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:38 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:05:39.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:38 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:05:39.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:38 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:05:39.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:38 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:05:39.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:38 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:05:39.962 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 792 sec 2024-03-28T18:05:40.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:39 smithi071 ceph-mon[36597]: pgmap v569: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:40.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:39 smithi071 ceph-mon[36597]: pgmap v570: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:05:40.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:39 smithi151 ceph-mon[36481]: pgmap v569: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:40.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:39 smithi151 ceph-mon[36481]: pgmap v570: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:05:40.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:39 smithi045 ceph-mon[31773]: pgmap v569: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:40.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:39 smithi045 ceph-mon[31773]: pgmap v570: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:05:40.506 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:05:40.506 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:05:40.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.737+0000 7f8e5ca00640 1 -- 172.21.15.45:0/1548419952 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e581040e0 msgr2=0x7f8e581105f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:40.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.737+0000 7f8e5ca00640 1 --2- 172.21.15.45:0/1548419952 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e581040e0 0x7f8e581105f0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8e50009ca0 tx=0x7f8e5002c100 comp rx=0 tx=0).stop 2024-03-28T18:05:40.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.737+0000 7f8e5ca00640 1 -- 172.21.15.45:0/1548419952 shutdown_connections 2024-03-28T18:05:40.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.737+0000 7f8e5ca00640 1 --2- 172.21.15.45:0/1548419952 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e581040e0 0x7f8e581105f0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.737+0000 7f8e5ca00640 1 --2- 172.21.15.45:0/1548419952 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e58103720 0x7f8e58103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.737+0000 7f8e5ca00640 1 --2- 172.21.15.45:0/1548419952 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e58109730 0x7f8e58109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.737+0000 7f8e5ca00640 1 -- 172.21.15.45:0/1548419952 >> 172.21.15.45:0/1548419952 conn(0x7f8e580ff1b0 msgr2=0x7f8e581015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:05:40.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.737+0000 7f8e5ca00640 1 -- 172.21.15.45:0/1548419952 shutdown_connections 2024-03-28T18:05:40.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.737+0000 7f8e5ca00640 1 -- 172.21.15.45:0/1548419952 wait complete. 2024-03-28T18:05:40.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.738+0000 7f8e5ca00640 1 Processor -- start 2024-03-28T18:05:40.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.738+0000 7f8e5ca00640 1 -- start start 2024-03-28T18:05:40.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.739+0000 7f8e5ca00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e58103720 0x7f8e5819a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:40.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.739+0000 7f8e5ca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e581040e0 0x7f8e5819a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:40.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.739+0000 7f8e5ca00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e58109730 0x7f8e5819eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:40.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.739+0000 7f8e5ca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e58113b40 con 0x7f8e581040e0 2024-03-28T18:05:40.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.739+0000 7f8e5ca00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e581139c0 con 0x7f8e58109730 2024-03-28T18:05:40.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.739+0000 7f8e5ca00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e58113cc0 con 0x7f8e58103720 2024-03-28T18:05:40.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.739+0000 7f8e56a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e581040e0 0x7f8e5819a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:40.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.739+0000 7f8e57e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e58109730 0x7f8e5819eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:40.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.739+0000 7f8e56a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e581040e0 0x7f8e5819a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52364/0 (socket says 172.21.15.45:52364) 2024-03-28T18:05:40.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.739+0000 7f8e56a00640 1 -- 172.21.15.45:0/1832343213 learned_addr learned my addr 172.21.15.45:0/1832343213 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:05:40.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.739+0000 7f8e57400640 1 --2- 172.21.15.45:0/1832343213 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e58103720 0x7f8e5819a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:40.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.740+0000 7f8e57e00640 1 -- 172.21.15.45:0/1832343213 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e58103720 msgr2=0x7f8e5819a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:40.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.740+0000 7f8e57e00640 1 --2- 172.21.15.45:0/1832343213 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e58103720 0x7f8e5819a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.740+0000 7f8e57e00640 1 -- 172.21.15.45:0/1832343213 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e581040e0 msgr2=0x7f8e5819a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:40.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.740+0000 7f8e57e00640 1 --2- 172.21.15.45:0/1832343213 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e581040e0 0x7f8e5819a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.740+0000 7f8e57e00640 1 -- 172.21.15.45:0/1832343213 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5819f260 con 0x7f8e58109730 2024-03-28T18:05:40.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.740+0000 7f8e57e00640 1 --2- 172.21.15.45:0/1832343213 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e58109730 0x7f8e5819eb30 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8e50009bf0 tx=0x7f8e50034670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:05:40.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.740+0000 7f8e4fe00640 1 -- 172.21.15.45:0/1832343213 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e50039050 con 0x7f8e58109730 2024-03-28T18:05:40.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.740+0000 7f8e4fe00640 1 -- 172.21.15.45:0/1832343213 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e50045070 con 0x7f8e58109730 2024-03-28T18:05:40.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.740+0000 7f8e5ca00640 1 -- 172.21.15.45:0/1832343213 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e5819f4f0 con 0x7f8e58109730 2024-03-28T18:05:40.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.740+0000 7f8e4fe00640 1 -- 172.21.15.45:0/1832343213 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e500045e0 con 0x7f8e58109730 2024-03-28T18:05:40.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.741+0000 7f8e5ca00640 1 -- 172.21.15.45:0/1832343213 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e581a6de0 con 0x7f8e58109730 2024-03-28T18:05:40.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.744+0000 7f8e60d63740 1 Processor -- start 2024-03-28T18:05:40.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.744+0000 7f8e4fe00640 1 -- 172.21.15.45:0/1832343213 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 28) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f8e50003970 con 0x7f8e58109730 2024-03-28T18:05:40.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.744+0000 7f8e60d63740 1 -- start start 2024-03-28T18:05:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.744+0000 7f8e60d63740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aade1dd0 0x5595aade0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.744+0000 7f8e60d63740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5595aade10f0 0x5595aade1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.744+0000 7f8e60d63740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595aadea460 0x5595aadec850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.744+0000 7f8e60d63740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595aad45250 con 0x5595aadea460 2024-03-28T18:05:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.744+0000 7f8e60d63740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595aaceda30 con 0x5595aade1dd0 2024-03-28T18:05:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.744+0000 7f8e60d63740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595aad412f0 con 0x5595aade10f0 2024-03-28T18:05:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.745+0000 7f8e56a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aade1dd0 0x5595aade0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.745+0000 7f8e56a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aade1dd0 0x5595aade0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:52932/0 (socket says 172.21.15.45:52932) 2024-03-28T18:05:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.745+0000 7f8e56a00640 1 -- 172.21.15.45:0/977317339 learned_addr learned my addr 172.21.15.45:0/977317339 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:05:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.745+0000 7f8e56a00640 1 --2- 172.21.15.45:0/977317339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595aadea460 0x5595aadec850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.745+0000 7f8e57400640 1 --2- 172.21.15.45:0/977317339 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5595aade10f0 0x5595aade1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.745+0000 7f8e56a00640 1 -- 172.21.15.45:0/977317339 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5595aade10f0 msgr2=0x5595aade1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:05:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.745+0000 7f8e56a00640 1 --2- 172.21.15.45:0/977317339 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5595aade10f0 0x5595aade1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.745+0000 7f8e56a00640 1 -- 172.21.15.45:0/977317339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595aadea460 msgr2=0x5595aadec850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:05:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.745+0000 7f8e56a00640 1 --2- 172.21.15.45:0/977317339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595aadea460 0x5595aadec850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.745+0000 7f8e56a00640 1 -- 172.21.15.45:0/977317339 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595aacd8690 con 0x5595aade1dd0 2024-03-28T18:05:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.746+0000 7f8e4fe00640 1 --2- 172.21.15.45:0/1832343213 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8e3c0a29f0 0x7f8e3c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.746+0000 7f8e56a00640 1 --2- 172.21.15.45:0/977317339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aade1dd0 0x5595aade0b20 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f8e440118f0 tx=0x7f8e440098d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b35fbe74d825b201 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:05:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.746+0000 7f8e4fe00640 1 -- 172.21.15.45:0/1832343213 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 12382+0+0 (secure 0 0 0) 0x7f8e500f22d0 con 0x7f8e58109730 2024-03-28T18:05:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.746+0000 7f8e43e00640 1 -- 172.21.15.45:0/977317339 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4400f7c0 con 0x5595aade1dd0 2024-03-28T18:05:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.746+0000 7f8e43e00640 1 -- 172.21.15.45:0/977317339 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e4400fe40 con 0x5595aade1dd0 2024-03-28T18:05:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.746+0000 7f8e43e00640 1 -- 172.21.15.45:0/977317339 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4401fda0 con 0x5595aade1dd0 2024-03-28T18:05:40.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.746+0000 7f8e57400640 1 --2- 172.21.15.45:0/1832343213 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8e3c0a29f0 0x7f8e3c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:40.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.746+0000 7f8e60d63740 1 -- 172.21.15.45:0/977317339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aade1dd0 msgr2=0x5595aade0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:40.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.746+0000 7f8e60d63740 1 --2- 172.21.15.45:0/977317339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aade1dd0 0x5595aade0b20 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f8e440118f0 tx=0x7f8e440098d0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.747+0000 7f8e60d63740 1 -- 172.21.15.45:0/977317339 shutdown_connections 2024-03-28T18:05:40.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.747+0000 7f8e60d63740 1 --2- 172.21.15.45:0/977317339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595aadea460 0x5595aadec850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.747+0000 7f8e60d63740 1 --2- 172.21.15.45:0/977317339 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5595aade10f0 0x5595aade1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.747+0000 7f8e60d63740 1 --2- 172.21.15.45:0/977317339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aade1dd0 0x5595aade0b20 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.747+0000 7f8e60d63740 1 -- 172.21.15.45:0/977317339 >> 172.21.15.45:0/977317339 conn(0x5595aada8200 msgr2=0x5595aaddbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:05:40.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.747+0000 7f8e60d63740 1 -- 172.21.15.45:0/977317339 shutdown_connections 2024-03-28T18:05:40.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.747+0000 7f8e57400640 1 --2- 172.21.15.45:0/1832343213 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8e3c0a29f0 0x7f8e3c0a4eb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x5595aade2320 tx=0x7f8e4800b390 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:05:40.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.747+0000 7f8e60d63740 1 -- 172.21.15.45:0/977317339 wait complete. 2024-03-28T18:05:40.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.747+0000 7f8e60d63740 1 -- start start 2024-03-28T18:05:40.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.747+0000 7f8e60d63740 1 Processor -- start 2024-03-28T18:05:40.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.748+0000 7f8e60d63740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5595aade10f0 0x5595aade9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:40.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.748+0000 7f8e60d63740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595aade1dd0 0x5595aade6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:40.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.748+0000 7f8e60d63740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aadea460 0x5595aade74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:40.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.748+0000 7f8e60d63740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595aaded2d0 con 0x5595aade1dd0 2024-03-28T18:05:40.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.748+0000 7f8e60d63740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595aad00e80 con 0x5595aadea460 2024-03-28T18:05:40.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.748+0000 7f8e60d63740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595aaded450 con 0x5595aade10f0 2024-03-28T18:05:40.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.748+0000 7f8e60d63740 10 client.0 _ll_register_callbacks cb 0x5595aad13130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:05:40.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.748+0000 7f8e57e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aadea460 0x5595aade74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:40.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.748+0000 7f8e57e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aadea460 0x5595aade74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:52934/0 (socket says 172.21.15.45:52934) 2024-03-28T18:05:40.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.748+0000 7f8e57e00640 1 -- 172.21.15.45:0/2670209103 learned_addr learned my addr 172.21.15.45:0/2670209103 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:05:40.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.748+0000 7f8e56a00640 1 --2- 172.21.15.45:0/2670209103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595aade1dd0 0x5595aade6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:40.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.748+0000 7f8e57400640 1 --2- 172.21.15.45:0/2670209103 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5595aade10f0 0x5595aade9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:40.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.748+0000 7f8e57e00640 1 -- 172.21.15.45:0/2670209103 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5595aade10f0 msgr2=0x5595aade9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:05:40.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.748+0000 7f8e57e00640 1 --2- 172.21.15.45:0/2670209103 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5595aade10f0 0x5595aade9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.748+0000 7f8e57e00640 1 -- 172.21.15.45:0/2670209103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595aade1dd0 msgr2=0x5595aade6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:05:40.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.748+0000 7f8e57e00640 1 --2- 172.21.15.45:0/2670209103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595aade1dd0 0x5595aade6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.748+0000 7f8e57e00640 1 -- 172.21.15.45:0/2670209103 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5595aab237d0 con 0x5595aadea460 2024-03-28T18:05:40.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.748+0000 7f8e57400640 1 --2- 172.21.15.45:0/2670209103 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5595aade10f0 0x5595aade9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:05:40.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.749+0000 7f8e57e00640 1 --2- 172.21.15.45:0/2670209103 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aadea460 0x5595aade74a0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f8e50037040 tx=0x7f8e50034920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=209896b30139cd7a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:05:40.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.749+0000 7f8e40c00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T18:05:40.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.749+0000 7f8e40c00640 1 -- 172.21.15.45:0/2670209103 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e500c1c20 con 0x5595aadea460 2024-03-28T18:05:40.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.749+0000 7f8e40c00640 1 -- 172.21.15.45:0/2670209103 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(147..147 src has 1..147) v4 ==== 12382+0+0 (secure 0 0 0) 0x7f8e500bbdd0 con 0x5595aadea460 2024-03-28T18:05:40.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.749+0000 7f8e60d63740 1 -- 172.21.15.45:0/2670209103 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8e4400fe40 con 0x5595aadea460 2024-03-28T18:05:40.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.749+0000 7f8e40c00640 1 -- 172.21.15.45:0/2670209103 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=319) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e500c6270 con 0x5595aadea460 2024-03-28T18:05:40.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.749+0000 7f8e60d63740 10 client.26669 fetch_fsmap learned FSMap version 319 2024-03-28T18:05:40.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.749+0000 7f8e60d63740 1 -- 172.21.15.45:0/2670209103 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=319,monmap=4+}) v3 -- 0x5595aaa593f0 con 0x5595aadea460 2024-03-28T18:05:40.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.749+0000 7f8e40c00640 1 -- 172.21.15.45:0/2670209103 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e4400fe40 con 0x5595aadea460 2024-03-28T18:05:40.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.750+0000 7f8e40c00640 1 -- 172.21.15.45:0/2670209103 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 319) v1 ==== 3905+0+0 (secure 0 0 0) 0x7f8e500dc140 con 0x5595aadea460 2024-03-28T18:05:40.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.750+0000 7f8e60d63740 10 client.26669 fetch_fsmap finished waiting for FSMap version 319 2024-03-28T18:05:40.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.750+0000 7f8e60d63740 10 client.26669 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:05:40.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.750+0000 7f8e60d63740 20 client.26669 populate_metadata read hostname 'smithi045' 2024-03-28T18:05:40.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.750+0000 7f8e60d63740 1 --2- 172.21.15.45:0/2670209103 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5595aadfa410 0x5595aadfc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:40.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.750+0000 7f8e60d63740 4 client.26669 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:05:40.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.750+0000 7f8e60d63740 1 -- 172.21.15.45:0/2670209103 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5595aad52790 con 0x5595aadfa410 2024-03-28T18:05:40.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.750+0000 7f8e57400640 1 --2- 172.21.15.45:0/2670209103 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5595aadfa410 0x5595aadfc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:40.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.750+0000 7f8e40c00640 10 client.26669 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:05:40.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.750+0000 7f8e57400640 1 --2- 172.21.15.45:0/2670209103 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5595aadfa410 0x5595aadfc840 crc :-1 s=READY pgs=432 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edec08ee0ed024fb server_cookie=283176256dc203de in_seq=0 out_seq=0 2024-03-28T18:05:40.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.751+0000 7f8e40c00640 1 -- 172.21.15.45:0/2670209103 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5595aad52790 con 0x5595aadfa410 2024-03-28T18:05:40.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.751+0000 7f8e40c00640 10 client.26669 handle_command_reply: tid=0 2024-03-28T18:05:40.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.751+0000 7f8e60d63740 1 client.26669 shutdown 2024-03-28T18:05:40.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.751+0000 7f8e60d63740 1 -- 172.21.15.45:0/2670209103 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aadea460 msgr2=0x5595aade74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:40.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.751+0000 7f8e60d63740 1 --2- 172.21.15.45:0/2670209103 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aadea460 0x5595aade74a0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f8e50037040 tx=0x7f8e50034920 comp rx=0 tx=0).stop 2024-03-28T18:05:40.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.751+0000 7f8e60d63740 1 -- 172.21.15.45:0/2670209103 shutdown_connections 2024-03-28T18:05:40.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.752+0000 7f8e60d63740 1 --2- 172.21.15.45:0/2670209103 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5595aadfa410 0x5595aadfc840 crc :-1 s=READY pgs=432 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.752+0000 7f8e60d63740 1 --2- 172.21.15.45:0/2670209103 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aadea460 0x5595aade74a0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.752+0000 7f8e60d63740 1 --2- 172.21.15.45:0/2670209103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595aade1dd0 0x5595aade6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.752+0000 7f8e40c00640 0 client.26669 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:05:40.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.752+0000 7f8e60d63740 1 --2- 172.21.15.45:0/2670209103 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5595aade10f0 0x5595aade9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.752+0000 7f8e60d63740 1 -- 172.21.15.45:0/2670209103 >> 172.21.15.45:0/2670209103 conn(0x5595aada8200 msgr2=0x5595aaddd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:05:40.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.752+0000 7f8e60d63740 1 -- 172.21.15.45:0/2670209103 shutdown_connections 2024-03-28T18:05:40.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.752+0000 7f8e60d63740 1 -- 172.21.15.45:0/2670209103 wait complete. 2024-03-28T18:05:40.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.752+0000 7f8e60d63740 20 client.26669 trim_cache size 0 max 16384 2024-03-28T18:05:40.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.769+0000 7f8e60d63740 1 Processor -- start 2024-03-28T18:05:40.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.770+0000 7f8e60d63740 1 -- start start 2024-03-28T18:05:40.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.770+0000 7f8e60d63740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595aade10f0 0x5595aade5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:40.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.770+0000 7f8e60d63740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aade1dd0 0x5595aade9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:40.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.770+0000 7f8e60d63740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5595aadea460 0x5595aade9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:40.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.770+0000 7f8e60d63740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e581b1490 con 0x5595aade10f0 2024-03-28T18:05:40.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.770+0000 7f8e60d63740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e581afb10 con 0x5595aade1dd0 2024-03-28T18:05:40.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.770+0000 7f8e60d63740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e581b0e30 con 0x5595aadea460 2024-03-28T18:05:40.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.770+0000 7f8e56a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aade1dd0 0x5595aade9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:40.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.770+0000 7f8e57400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595aade10f0 0x5595aade5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:40.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.770+0000 7f8e56a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aade1dd0 0x5595aade9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.45:52938/0 (socket says 172.21.15.45:52938) 2024-03-28T18:05:40.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.770+0000 7f8e56a00640 1 -- 172.21.15.45:0/2719456968 learned_addr learned my addr 172.21.15.45:0/2719456968 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:05:40.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.770+0000 7f8e57e00640 1 --2- 172.21.15.45:0/2719456968 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5595aadea460 0x5595aade9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:40.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.771+0000 7f8e56a00640 1 -- 172.21.15.45:0/2719456968 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5595aadea460 msgr2=0x5595aade9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:05:40.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.771+0000 7f8e56a00640 1 --2- 172.21.15.45:0/2719456968 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5595aadea460 0x5595aade9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.771+0000 7f8e56a00640 1 -- 172.21.15.45:0/2719456968 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595aade10f0 msgr2=0x5595aade5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:05:40.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.771+0000 7f8e56a00640 1 --2- 172.21.15.45:0/2719456968 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595aade10f0 0x5595aade5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.771+0000 7f8e56a00640 1 -- 172.21.15.45:0/2719456968 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595aaa190f0 con 0x5595aade1dd0 2024-03-28T18:05:40.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.771+0000 7f8e57400640 1 --2- 172.21.15.45:0/2719456968 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595aade10f0 0x5595aade5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:05:40.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.771+0000 7f8e56a00640 1 --2- 172.21.15.45:0/2719456968 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aade1dd0 0x5595aade9690 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f8e440098a0 tx=0x7f8e44009530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72bb2ea072f959d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:05:40.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.771+0000 7f8e43400640 1 -- 172.21.15.45:0/2719456968 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e44009d60 con 0x5595aade1dd0 2024-03-28T18:05:40.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.771+0000 7f8e43400640 1 -- 172.21.15.45:0/2719456968 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e4401f440 con 0x5595aade1dd0 2024-03-28T18:05:40.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.771+0000 7f8e43400640 1 -- 172.21.15.45:0/2719456968 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e44005440 con 0x5595aade1dd0 2024-03-28T18:05:40.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.772+0000 7f8e60d63740 1 -- 172.21.15.45:0/2719456968 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aade1dd0 msgr2=0x5595aade9690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:40.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.772+0000 7f8e60d63740 1 --2- 172.21.15.45:0/2719456968 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aade1dd0 0x5595aade9690 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f8e440098a0 tx=0x7f8e44009530 comp rx=0 tx=0).stop 2024-03-28T18:05:40.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.772+0000 7f8e60d63740 1 -- 172.21.15.45:0/2719456968 shutdown_connections 2024-03-28T18:05:40.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.772+0000 7f8e60d63740 1 --2- 172.21.15.45:0/2719456968 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5595aadea460 0x5595aade9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.772+0000 7f8e60d63740 1 --2- 172.21.15.45:0/2719456968 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aade1dd0 0x5595aade9690 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.772+0000 7f8e60d63740 1 --2- 172.21.15.45:0/2719456968 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595aade10f0 0x5595aade5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.772+0000 7f8e60d63740 1 -- 172.21.15.45:0/2719456968 >> 172.21.15.45:0/2719456968 conn(0x5595aada8200 msgr2=0x5595aade5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:05:40.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.772+0000 7f8e60d63740 1 -- 172.21.15.45:0/2719456968 shutdown_connections 2024-03-28T18:05:40.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.772+0000 7f8e60d63740 1 -- 172.21.15.45:0/2719456968 wait complete. 2024-03-28T18:05:40.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.772+0000 7f8e60d63740 1 -- start start 2024-03-28T18:05:40.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.773+0000 7f8e60d63740 1 Processor -- start 2024-03-28T18:05:40.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.773+0000 7f8e60d63740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595aade10f0 0x5595aade44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:40.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.773+0000 7f8e60d63740 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5595aade1dd0 0x5595aade49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:40.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.773+0000 7f8e60d63740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aadea460 0x5595aadeaf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:40.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.773+0000 7f8e60d63740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e581b07d0 con 0x5595aade10f0 2024-03-28T18:05:40.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.773+0000 7f8e60d63740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e581b0170 con 0x5595aadea460 2024-03-28T18:05:40.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.773+0000 7f8e60d63740 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595aade6d50 con 0x5595aade1dd0 2024-03-28T18:05:40.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.773+0000 7f8e60d63740 10 client.0 _ll_register_callbacks cb 0x7f8e50005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:05:40.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.773+0000 7f8e57400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595aade10f0 0x5595aade44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:40.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.773+0000 7f8e57e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aadea460 0x5595aadeaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:40.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.773+0000 7f8e57400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595aade10f0 0x5595aade44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52398/0 (socket says 172.21.15.45:52398) 2024-03-28T18:05:40.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.773+0000 7f8e57400640 1 -- 172.21.15.45:0/503267527 learned_addr learned my addr 172.21.15.45:0/503267527 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-28T18:05:40.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.773+0000 7f8e56a00640 1 --2- 172.21.15.45:0/503267527 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5595aade1dd0 0x5595aade49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:40.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.774+0000 7f8e57e00640 1 -- 172.21.15.45:0/503267527 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5595aade1dd0 msgr2=0x5595aade49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:05:40.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.774+0000 7f8e57e00640 1 --2- 172.21.15.45:0/503267527 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5595aade1dd0 0x5595aade49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.774+0000 7f8e57e00640 1 -- 172.21.15.45:0/503267527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595aade10f0 msgr2=0x5595aade44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:05:40.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.774+0000 7f8e57e00640 1 --2- 172.21.15.45:0/503267527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595aade10f0 0x5595aade44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.774+0000 7f8e57e00640 1 -- 172.21.15.45:0/503267527 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5595aadeb670 con 0x5595aadea460 2024-03-28T18:05:40.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.774+0000 7f8e57400640 1 --2- 172.21.15.45:0/503267527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595aade10f0 0x5595aade44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:05:40.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.774+0000 7f8e56a00640 1 --2- 172.21.15.45:0/503267527 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5595aade1dd0 0x5595aade49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:05:40.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.774+0000 7f8e57e00640 1 --2- 172.21.15.45:0/503267527 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aadea460 0x5595aadeaf80 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8e500bbd80 tx=0x7f8e5002c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=669af010f1a3c968 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:05:40.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.774+0000 7f8e40c00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-03-28T18:05:40.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.774+0000 7f8e40c00640 1 -- 172.21.15.45:0/503267527 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e500c1c20 con 0x5595aadea460 2024-03-28T18:05:40.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.774+0000 7f8e40c00640 1 -- 172.21.15.45:0/503267527 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(147..147 src has 1..147) v4 ==== 12382+0+0 (secure 0 0 0) 0x7f8e50009800 con 0x5595aadea460 2024-03-28T18:05:40.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.774+0000 7f8e60d63740 1 -- 172.21.15.45:0/503267527 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8e4401f440 con 0x5595aadea460 2024-03-28T18:05:40.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.775+0000 7f8e40c00640 1 -- 172.21.15.45:0/503267527 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=319) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e500c5de0 con 0x5595aadea460 2024-03-28T18:05:40.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.775+0000 7f8e60d63740 10 client.26675 fetch_fsmap learned FSMap version 319 2024-03-28T18:05:40.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.775+0000 7f8e60d63740 1 -- 172.21.15.45:0/503267527 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=319,monmap=4+}) v3 -- 0x5595aadeb670 con 0x5595aadea460 2024-03-28T18:05:40.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.775+0000 7f8e40c00640 1 -- 172.21.15.45:0/503267527 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e4401f440 con 0x5595aadea460 2024-03-28T18:05:40.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.775+0000 7f8e40c00640 1 -- 172.21.15.45:0/503267527 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 319) v1 ==== 3905+0+0 (secure 0 0 0) 0x7f8e500d6b90 con 0x5595aadea460 2024-03-28T18:05:40.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.775+0000 7f8e60d63740 10 client.26675 fetch_fsmap finished waiting for FSMap version 319 2024-03-28T18:05:40.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.775+0000 7f8e60d63740 10 client.26675 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:05:40.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.775+0000 7f8e60d63740 20 client.26675 populate_metadata read hostname 'smithi045' 2024-03-28T18:05:40.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.775+0000 7f8e60d63740 1 --2- 172.21.15.45:0/503267527 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5595aae56aa0 0x5595aae58e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:05:40.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.775+0000 7f8e60d63740 4 client.26675 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:05:40.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.775+0000 7f8e60d63740 1 -- 172.21.15.45:0/503267527 --> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5595aad4cae0 con 0x5595aae56aa0 2024-03-28T18:05:40.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.775+0000 7f8e57400640 1 --2- 172.21.15.45:0/503267527 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5595aae56aa0 0x5595aae58e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:05:40.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.776+0000 7f8e57400640 1 --2- 172.21.15.45:0/503267527 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5595aae56aa0 0x5595aae58e90 crc :-1 s=READY pgs=433 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=482c282ef554fb37 server_cookie=10e99bce6e420831 in_seq=0 out_seq=0 2024-03-28T18:05:40.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.776+0000 7f8e40c00640 10 client.26675 ms_handle_connect on v2:172.21.15.71:6838/2709938800 2024-03-28T18:05:40.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.776+0000 7f8e40c00640 1 -- 172.21.15.45:0/503267527 <== mds.0 v2:172.21.15.71:6838/2709938800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5595aad4cae0 con 0x5595aae56aa0 2024-03-28T18:05:40.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.776+0000 7f8e40c00640 10 client.26675 handle_command_reply: tid=0 2024-03-28T18:05:40.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.776+0000 7f8e60d63740 1 client.26675 shutdown 2024-03-28T18:05:40.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.776+0000 7f8e60d63740 1 -- 172.21.15.45:0/503267527 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aadea460 msgr2=0x5595aadeaf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:40.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.776+0000 7f8e60d63740 1 --2- 172.21.15.45:0/503267527 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aadea460 0x5595aadeaf80 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8e500bbd80 tx=0x7f8e5002c920 comp rx=0 tx=0).stop 2024-03-28T18:05:40.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.776+0000 7f8e60d63740 1 -- 172.21.15.45:0/503267527 shutdown_connections 2024-03-28T18:05:40.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.776+0000 7f8e60d63740 1 --2- 172.21.15.45:0/503267527 >> [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] conn(0x5595aae56aa0 0x5595aae58e90 crc :-1 s=READY pgs=433 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.776+0000 7f8e60d63740 1 --2- 172.21.15.45:0/503267527 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5595aadea460 0x5595aadeaf80 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.776+0000 7f8e40c00640 0 client.26675 ms_handle_reset on v2:172.21.15.71:6838/2709938800 2024-03-28T18:05:40.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.776+0000 7f8e60d63740 1 --2- 172.21.15.45:0/503267527 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x5595aade1dd0 0x5595aade49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.776+0000 7f8e60d63740 1 --2- 172.21.15.45:0/503267527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595aade10f0 0x5595aade44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.776+0000 7f8e60d63740 1 -- 172.21.15.45:0/503267527 >> 172.21.15.45:0/503267527 conn(0x5595aada8200 msgr2=0x5595aade60c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:05:40.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.776+0000 7f8e60d63740 1 -- 172.21.15.45:0/503267527 shutdown_connections 2024-03-28T18:05:40.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.776+0000 7f8e60d63740 1 -- 172.21.15.45:0/503267527 wait complete. 2024-03-28T18:05:40.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.776+0000 7f8e60d63740 20 client.26675 trim_cache size 0 max 16384 2024-03-28T18:05:40.782 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-28T18:05:40.782 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:05:40.782 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-28T18:05:40.782 INFO:teuthology.orchestra.run.smithi045.stdout: "380421fa-046c-4088-b9a6-edb16e6bb32a": { 2024-03-28T18:05:40.783 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-28T18:05:40.783 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "380421fa-046c-4088-b9a6-edb16e6bb32a", 2024-03-28T18:05:40.783 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-28T18:05:40.783 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:05:40.783 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-28T18:05:40.783 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-28T18:05:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.777+0000 7f8e5ca00640 1 -- 172.21.15.45:0/1832343213 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8e3c0a29f0 msgr2=0x7f8e3c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.777+0000 7f8e5ca00640 1 --2- 172.21.15.45:0/1832343213 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8e3c0a29f0 0x7f8e3c0a4eb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x5595aade2320 tx=0x7f8e4800b390 comp rx=0 tx=0).stop 2024-03-28T18:05:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.777+0000 7f8e5ca00640 1 -- 172.21.15.45:0/1832343213 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e58109730 msgr2=0x7f8e5819eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:05:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.777+0000 7f8e5ca00640 1 --2- 172.21.15.45:0/1832343213 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e58109730 0x7f8e5819eb30 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8e50009bf0 tx=0x7f8e50034670 comp rx=0 tx=0).stop 2024-03-28T18:05:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.777+0000 7f8e5ca00640 1 -- 172.21.15.45:0/1832343213 shutdown_connections 2024-03-28T18:05:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.777+0000 7f8e5ca00640 1 --2- 172.21.15.45:0/1832343213 >> [v2:172.21.15.45:6800/1309597278,v1:172.21.15.45:6801/1309597278] conn(0x7f8e3c0a29f0 0x7f8e3c0a4eb0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.777+0000 7f8e5ca00640 1 --2- 172.21.15.45:0/1832343213 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e58109730 0x7f8e5819eb30 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.777+0000 7f8e5ca00640 1 --2- 172.21.15.45:0/1832343213 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e581040e0 0x7f8e5819a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.777+0000 7f8e5ca00640 1 --2- 172.21.15.45:0/1832343213 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e58103720 0x7f8e5819a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:05:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.777+0000 7f8e5ca00640 1 -- 172.21.15.45:0/1832343213 >> 172.21.15.45:0/1832343213 conn(0x7f8e580ff1b0 msgr2=0x7f8e580ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:05:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.777+0000 7f8e5ca00640 1 -- 172.21.15.45:0/1832343213 shutdown_connections 2024-03-28T18:05:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-28T18:05:40.777+0000 7f8e5ca00640 1 -- 172.21.15.45:0/1832343213 wait complete. 2024-03-28T18:05:40.795 INFO:tasks.cephfs.filesystem:scrub status for tag:380421fa-046c-4088-b9a6-edb16e6bb32a - {'path': '/', 'tag': '380421fa-046c-4088-b9a6-edb16e6bb32a', 'options': 'recursive,force'} 2024-03-28T18:05:40.795 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/fwd_scrub.py", line 38, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/fwd_scrub.py", line 55, in do_scrub self._scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/fwd_scrub.py", line 76, in _scrub done = self.fs.wait_until_scrub_complete(tag=tag, sleep=30, timeout=self.scrub_timeout) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 1749, in wait_until_scrub_complete while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (31) after waiting for 900 seconds 2024-03-28T18:05:40.962 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 793 sec 2024-03-28T18:05:41.962 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 794 sec 2024-03-28T18:05:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:41 smithi071 ceph-mon[36597]: pgmap v571: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:05:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:41 smithi151 ceph-mon[36481]: pgmap v571: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:05:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:41 smithi045 ceph-mon[31773]: pgmap v571: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:05:42.963 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 795 sec 2024-03-28T18:05:43.963 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 796 sec 2024-03-28T18:05:44.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:43 smithi071 ceph-mon[36597]: pgmap v572: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:05:44.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:43 smithi151 ceph-mon[36481]: pgmap v572: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:05:44.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:43 smithi045 ceph-mon[31773]: pgmap v572: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:05:44.963 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 797 sec 2024-03-28T18:05:45.963 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 798 sec 2024-03-28T18:05:46.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:45 smithi071 ceph-mon[36597]: pgmap v573: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:05:46.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:45 smithi151 ceph-mon[36481]: pgmap v573: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:05:46.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:45 smithi045 ceph-mon[31773]: pgmap v573: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:05:46.963 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 799 sec 2024-03-28T18:05:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:46 smithi071 ceph-mon[36597]: pgmap v574: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:05:47.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:46 smithi045 ceph-mon[31773]: pgmap v574: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:05:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:46 smithi151 ceph-mon[36481]: pgmap v574: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:05:47.963 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 800 sec 2024-03-28T18:05:48.963 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 801 sec 2024-03-28T18:05:49.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:49 smithi045 ceph-mon[31773]: pgmap v575: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:05:49.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:49 smithi071 ceph-mon[36597]: pgmap v575: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:05:49.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:49 smithi151 ceph-mon[36481]: pgmap v575: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:05:49.963 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 802 sec 2024-03-28T18:05:50.964 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 803 sec 2024-03-28T18:05:51.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:51 smithi045 ceph-mon[31773]: pgmap v576: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:51 smithi071 ceph-mon[36597]: pgmap v576: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:51 smithi151 ceph-mon[36481]: pgmap v576: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:51.964 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 804 sec 2024-03-28T18:05:52.964 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 805 sec 2024-03-28T18:05:53.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:53 smithi045 ceph-mon[31773]: pgmap v577: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:53.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:53 smithi071 ceph-mon[36597]: pgmap v577: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:53.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:53 smithi151 ceph-mon[36481]: pgmap v577: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:53.964 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 806 sec 2024-03-28T18:05:54.964 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 807 sec 2024-03-28T18:05:55.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:55 smithi045 ceph-mon[31773]: pgmap v578: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:55.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:55 smithi071 ceph-mon[36597]: pgmap v578: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:55.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:55 smithi151 ceph-mon[36481]: pgmap v578: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:55.964 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 808 sec 2024-03-28T18:05:56.964 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 809 sec 2024-03-28T18:05:57.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:57 smithi045 ceph-mon[31773]: pgmap v579: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:57 smithi071 ceph-mon[36597]: pgmap v579: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:57.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:57 smithi151 ceph-mon[36481]: pgmap v579: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:05:57.964 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 810 sec 2024-03-28T18:05:58.965 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 811 sec 2024-03-28T18:05:59.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:05:59 smithi045 ceph-mon[31773]: pgmap v580: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:59.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:05:59 smithi071 ceph-mon[36597]: pgmap v580: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:59.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:05:59 smithi151 ceph-mon[36481]: pgmap v580: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:05:59.965 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 812 sec 2024-03-28T18:06:00.965 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 813 sec 2024-03-28T18:06:01.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:01 smithi045 ceph-mon[31773]: pgmap v581: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:01 smithi071 ceph-mon[36597]: pgmap v581: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:01 smithi151 ceph-mon[36481]: pgmap v581: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:01.965 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 814 sec 2024-03-28T18:06:02.965 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 815 sec 2024-03-28T18:06:03.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:03 smithi045 ceph-mon[31773]: pgmap v582: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:03.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:03 smithi071 ceph-mon[36597]: pgmap v582: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:03.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:03 smithi151 ceph-mon[36481]: pgmap v582: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:03.965 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 816 sec 2024-03-28T18:06:04.965 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 817 sec 2024-03-28T18:06:05.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:05 smithi045 ceph-mon[31773]: pgmap v583: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:05.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:05 smithi071 ceph-mon[36597]: pgmap v583: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:05 smithi151 ceph-mon[36481]: pgmap v583: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:05.966 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 818 sec 2024-03-28T18:06:06.966 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 819 sec 2024-03-28T18:06:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:07 smithi045 ceph-mon[31773]: pgmap v584: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:07 smithi071 ceph-mon[36597]: pgmap v584: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:07.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:07 smithi151 ceph-mon[36481]: pgmap v584: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:07.966 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 820 sec 2024-03-28T18:06:08.966 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 821 sec 2024-03-28T18:06:09.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:09 smithi045 ceph-mon[31773]: pgmap v585: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:09.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:09 smithi071 ceph-mon[36597]: pgmap v585: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:09.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:09 smithi151 ceph-mon[36481]: pgmap v585: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:09.966 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 822 sec 2024-03-28T18:06:10.966 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 823 sec 2024-03-28T18:06:11.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:11 smithi071 ceph-mon[36597]: pgmap v586: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:11.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:11 smithi151 ceph-mon[36481]: pgmap v586: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:11 smithi045 ceph-mon[31773]: pgmap v586: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:11.966 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 824 sec 2024-03-28T18:06:12.967 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 825 sec 2024-03-28T18:06:13.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:13 smithi071 ceph-mon[36597]: pgmap v587: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:13.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:13 smithi151 ceph-mon[36481]: pgmap v587: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:13.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:13 smithi045 ceph-mon[31773]: pgmap v587: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:13.967 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 826 sec 2024-03-28T18:06:14.967 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 827 sec 2024-03-28T18:06:15.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:15 smithi071 ceph-mon[36597]: pgmap v588: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:15.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:15 smithi151 ceph-mon[36481]: pgmap v588: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:15.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:15 smithi045 ceph-mon[31773]: pgmap v588: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:15.967 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 828 sec 2024-03-28T18:06:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:16 smithi071 ceph-mon[36597]: 2 slow requests, 2 included below; oldest blocked for > 184.253601 secs 2024-03-28T18:06:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:16 smithi071 ceph-mon[36597]: slow request 184.253601 seconds old, received at 2024-03-28T18:03:11.560545+0000: peer_request:client.15069:1629 currently acquired locks 2024-03-28T18:06:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:16 smithi071 ceph-mon[36597]: slow request 184.157476 seconds old, received at 2024-03-28T18:03:11.656670+0000: client_request(client.25691:164 rmsnap #0x1//scheduled-2024-03-28-17_57_00_UTC 2024-03-28T18:03:11.656797+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:06:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:16 smithi151 ceph-mon[36481]: 2 slow requests, 2 included below; oldest blocked for > 184.253601 secs 2024-03-28T18:06:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:16 smithi151 ceph-mon[36481]: slow request 184.253601 seconds old, received at 2024-03-28T18:03:11.560545+0000: peer_request:client.15069:1629 currently acquired locks 2024-03-28T18:06:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:16 smithi151 ceph-mon[36481]: slow request 184.157476 seconds old, received at 2024-03-28T18:03:11.656670+0000: client_request(client.25691:164 rmsnap #0x1//scheduled-2024-03-28-17_57_00_UTC 2024-03-28T18:03:11.656797+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:06:16.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:16 smithi045 ceph-mon[31773]: 2 slow requests, 2 included below; oldest blocked for > 184.253601 secs 2024-03-28T18:06:16.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:16 smithi045 ceph-mon[31773]: slow request 184.253601 seconds old, received at 2024-03-28T18:03:11.560545+0000: peer_request:client.15069:1629 currently acquired locks 2024-03-28T18:06:16.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:16 smithi045 ceph-mon[31773]: slow request 184.157476 seconds old, received at 2024-03-28T18:03:11.656670+0000: client_request(client.25691:164 rmsnap #0x1//scheduled-2024-03-28-17_57_00_UTC 2024-03-28T18:03:11.656797+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:06:16.967 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 829 sec 2024-03-28T18:06:17.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:17 smithi071 ceph-mon[36597]: pgmap v589: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:17.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:17 smithi045 ceph-mon[31773]: pgmap v589: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:17 smithi151 ceph-mon[36481]: pgmap v589: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:17.967 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 830 sec 2024-03-28T18:06:18.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:18 smithi071 ceph-mon[36597]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-28T18:06:18.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:18 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] up:active 2024-03-28T18:06:18.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:18 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T18:06:18.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:18 smithi151 ceph-mon[36481]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-28T18:06:18.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:18 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] up:active 2024-03-28T18:06:18.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:18 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T18:06:18.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:18 smithi045 ceph-mon[31773]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-28T18:06:18.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:18 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] up:active 2024-03-28T18:06:18.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:18 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T18:06:18.967 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 831 sec 2024-03-28T18:06:19.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:19 smithi071 ceph-mon[36597]: pgmap v590: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:19.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:19 smithi151 ceph-mon[36481]: pgmap v590: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:19.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:19 smithi045 ceph-mon[31773]: pgmap v590: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:19.967 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 832 sec 2024-03-28T18:06:20.968 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 833 sec 2024-03-28T18:06:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:21 smithi071 ceph-mon[36597]: pgmap v591: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 189.253664 secs 2024-03-28T18:06:21.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:21 smithi151 ceph-mon[36481]: pgmap v591: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 189.253664 secs 2024-03-28T18:06:21.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:21 smithi045 ceph-mon[31773]: pgmap v591: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:21.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 189.253664 secs 2024-03-28T18:06:21.968 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 834 sec 2024-03-28T18:06:22.968 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 835 sec 2024-03-28T18:06:23.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:23 smithi071 ceph-mon[36597]: pgmap v592: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:23.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:23 smithi151 ceph-mon[36481]: pgmap v592: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:23.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:23 smithi045 ceph-mon[31773]: pgmap v592: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:23.968 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 836 sec 2024-03-28T18:06:24.968 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 837 sec 2024-03-28T18:06:25.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:25 smithi071 ceph-mon[36597]: pgmap v593: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:25.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:25 smithi151 ceph-mon[36481]: pgmap v593: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:25.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:25 smithi045 ceph-mon[31773]: pgmap v593: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:25.968 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 838 sec 2024-03-28T18:06:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 194.253723 secs 2024-03-28T18:06:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 194.253723 secs 2024-03-28T18:06:26.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 194.253723 secs 2024-03-28T18:06:26.968 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 839 sec 2024-03-28T18:06:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:27 smithi071 ceph-mon[36597]: pgmap v594: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:27 smithi151 ceph-mon[36481]: pgmap v594: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:27.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:27 smithi045 ceph-mon[31773]: pgmap v594: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:27.969 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 840 sec 2024-03-28T18:06:28.969 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 841 sec 2024-03-28T18:06:29.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:29 smithi071 ceph-mon[36597]: pgmap v595: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:29.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:29 smithi151 ceph-mon[36481]: pgmap v595: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:29.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:29 smithi045 ceph-mon[31773]: pgmap v595: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:29.969 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 842 sec 2024-03-28T18:06:30.969 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 843 sec 2024-03-28T18:06:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:31 smithi071 ceph-mon[36597]: pgmap v596: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 199.253782 secs 2024-03-28T18:06:31.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:31 smithi151 ceph-mon[36481]: pgmap v596: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:31.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 199.253782 secs 2024-03-28T18:06:31.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:31 smithi045 ceph-mon[31773]: pgmap v596: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:31.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 199.253782 secs 2024-03-28T18:06:31.969 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 844 sec 2024-03-28T18:06:32.969 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 845 sec 2024-03-28T18:06:33.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:33 smithi071 ceph-mon[36597]: pgmap v597: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:33.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:33 smithi151 ceph-mon[36481]: pgmap v597: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:33.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:33 smithi045 ceph-mon[31773]: pgmap v597: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:33.969 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 846 sec 2024-03-28T18:06:34.970 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 847 sec 2024-03-28T18:06:35.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:35 smithi071 ceph-mon[36597]: pgmap v598: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:35.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:35 smithi151 ceph-mon[36481]: pgmap v598: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:35.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:35 smithi045 ceph-mon[31773]: pgmap v598: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:35.970 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 848 sec 2024-03-28T18:06:36.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 204.253797 secs 2024-03-28T18:06:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 204.253797 secs 2024-03-28T18:06:36.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 204.253797 secs 2024-03-28T18:06:36.970 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 849 sec 2024-03-28T18:06:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:37 smithi071 ceph-mon[36597]: pgmap v599: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:37 smithi151 ceph-mon[36481]: pgmap v599: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:37.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:37 smithi045 ceph-mon[31773]: pgmap v599: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:37.970 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 850 sec 2024-03-28T18:06:38.970 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 851 sec 2024-03-28T18:06:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:39 smithi071 ceph-mon[36597]: pgmap v600: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:39 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:06:39.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:39 smithi151 ceph-mon[36481]: pgmap v600: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:39 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:06:39.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:39 smithi045 ceph-mon[31773]: pgmap v600: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:39.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:39 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:06:39.970 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 852 sec 2024-03-28T18:06:40.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:40 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:40.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:40 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:06:40.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:40 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:06:40.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:40 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:40 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:06:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:40 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:06:40.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:40 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:06:40.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:40 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:06:40.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:40 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:06:40.970 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 853 sec 2024-03-28T18:06:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:41 smithi071 ceph-mon[36597]: pgmap v601: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:41 smithi071 ceph-mon[36597]: pgmap v602: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:06:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 209.253860 secs 2024-03-28T18:06:41.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:41 smithi151 ceph-mon[36481]: pgmap v601: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:41 smithi151 ceph-mon[36481]: pgmap v602: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:06:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 209.253860 secs 2024-03-28T18:06:41.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:41 smithi045 ceph-mon[31773]: pgmap v601: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:41 smithi045 ceph-mon[31773]: pgmap v602: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:06:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 209.253860 secs 2024-03-28T18:06:41.970 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 854 sec 2024-03-28T18:06:42.971 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 855 sec 2024-03-28T18:06:43.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:43 smithi071 ceph-mon[36597]: pgmap v603: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:06:43.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:43 smithi151 ceph-mon[36481]: pgmap v603: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:06:43.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:43 smithi045 ceph-mon[31773]: pgmap v603: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:06:43.971 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 856 sec 2024-03-28T18:06:44.972 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 857 sec 2024-03-28T18:06:45.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:45 smithi071 ceph-mon[36597]: pgmap v604: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:06:45.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:45 smithi151 ceph-mon[36481]: pgmap v604: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:06:45.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:45 smithi045 ceph-mon[31773]: pgmap v604: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:06:45.971 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 858 sec 2024-03-28T18:06:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 214.253921 secs 2024-03-28T18:06:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 214.253921 secs 2024-03-28T18:06:46.971 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 859 sec 2024-03-28T18:06:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 214.253921 secs 2024-03-28T18:06:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:47 smithi151 ceph-mon[36481]: pgmap v605: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:06:47.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:47 smithi045 ceph-mon[31773]: pgmap v605: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:06:47.971 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 860 sec 2024-03-28T18:06:48.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:47 smithi071 ceph-mon[36597]: pgmap v605: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:06:48.971 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 861 sec 2024-03-28T18:06:49.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:49 smithi045 ceph-mon[31773]: pgmap v606: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:06:49.971 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 862 sec 2024-03-28T18:06:50.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:49 smithi071 ceph-mon[36597]: pgmap v606: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:06:50.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:49 smithi151 ceph-mon[36481]: pgmap v606: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:06:50.972 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 863 sec 2024-03-28T18:06:51.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:51 smithi045 ceph-mon[31773]: pgmap v607: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:06:51.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 219.253933 secs 2024-03-28T18:06:51.972 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 864 sec 2024-03-28T18:06:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:51 smithi071 ceph-mon[36597]: pgmap v607: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:06:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 219.253933 secs 2024-03-28T18:06:52.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:51 smithi151 ceph-mon[36481]: pgmap v607: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:06:52.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 219.253933 secs 2024-03-28T18:06:52.972 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 865 sec 2024-03-28T18:06:53.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:53 smithi045 ceph-mon[31773]: pgmap v608: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:53.972 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 866 sec 2024-03-28T18:06:54.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:53 smithi071 ceph-mon[36597]: pgmap v608: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:54.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:53 smithi151 ceph-mon[36481]: pgmap v608: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:54.972 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 867 sec 2024-03-28T18:06:55.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:55 smithi045 ceph-mon[31773]: pgmap v609: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:55.972 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 868 sec 2024-03-28T18:06:56.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:55 smithi071 ceph-mon[36597]: pgmap v609: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:56.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:55 smithi151 ceph-mon[36481]: pgmap v609: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 224.253994 secs 2024-03-28T18:06:56.972 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 869 sec 2024-03-28T18:06:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 224.253994 secs 2024-03-28T18:06:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 224.253994 secs 2024-03-28T18:06:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:57 smithi045 ceph-mon[31773]: pgmap v610: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:57.972 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 870 sec 2024-03-28T18:06:57.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:57 smithi151 ceph-mon[36481]: pgmap v610: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:57 smithi071 ceph-mon[36597]: pgmap v610: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:06:58.972 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 871 sec 2024-03-28T18:06:59.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:06:59 smithi045 ceph-mon[31773]: pgmap v611: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:06:59.973 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 872 sec 2024-03-28T18:07:00.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:06:59 smithi071 ceph-mon[36597]: pgmap v611: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:00.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:06:59 smithi151 ceph-mon[36481]: pgmap v611: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:00.973 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 873 sec 2024-03-28T18:07:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:01 smithi045 ceph-mon[31773]: pgmap v612: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 229.254054 secs 2024-03-28T18:07:01.973 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 874 sec 2024-03-28T18:07:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:01 smithi071 ceph-mon[36597]: pgmap v612: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 229.254054 secs 2024-03-28T18:07:02.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:01 smithi151 ceph-mon[36481]: pgmap v612: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:02.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 229.254054 secs 2024-03-28T18:07:02.973 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 875 sec 2024-03-28T18:07:03.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:03 smithi045 ceph-mon[31773]: pgmap v613: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:03.973 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 876 sec 2024-03-28T18:07:04.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:03 smithi071 ceph-mon[36597]: pgmap v613: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:04.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:03 smithi151 ceph-mon[36481]: pgmap v613: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:04.973 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 877 sec 2024-03-28T18:07:05.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:05 smithi045 ceph-mon[31773]: pgmap v614: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:05.973 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 878 sec 2024-03-28T18:07:06.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:05 smithi071 ceph-mon[36597]: pgmap v614: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:06.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:05 smithi151 ceph-mon[36481]: pgmap v614: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:06.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 234.254118 secs 2024-03-28T18:07:06.974 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 879 sec 2024-03-28T18:07:07.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 234.254118 secs 2024-03-28T18:07:07.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 234.254118 secs 2024-03-28T18:07:07.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:07 smithi045 ceph-mon[31773]: pgmap v615: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:07.974 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 880 sec 2024-03-28T18:07:08.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:07 smithi071 ceph-mon[36597]: pgmap v615: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:08.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:07 smithi151 ceph-mon[36481]: pgmap v615: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:08.974 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 881 sec 2024-03-28T18:07:09.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:09 smithi045 ceph-mon[31773]: pgmap v616: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:09.974 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 882 sec 2024-03-28T18:07:10.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:09 smithi071 ceph-mon[36597]: pgmap v616: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:09 smithi151 ceph-mon[36481]: pgmap v616: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:10.974 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 883 sec 2024-03-28T18:07:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:11 smithi045 ceph-mon[31773]: pgmap v617: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:11.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 239.254129 secs 2024-03-28T18:07:11.974 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 884 sec 2024-03-28T18:07:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:11 smithi071 ceph-mon[36597]: pgmap v617: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 239.254129 secs 2024-03-28T18:07:12.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:11 smithi151 ceph-mon[36481]: pgmap v617: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:12.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 239.254129 secs 2024-03-28T18:07:12.975 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 885 sec 2024-03-28T18:07:13.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:13 smithi045 ceph-mon[31773]: pgmap v618: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:13.975 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 886 sec 2024-03-28T18:07:14.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:13 smithi071 ceph-mon[36597]: pgmap v618: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:13 smithi151 ceph-mon[36481]: pgmap v618: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:14.975 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 887 sec 2024-03-28T18:07:15.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:15 smithi045 ceph-mon[31773]: pgmap v619: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:15.975 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 888 sec 2024-03-28T18:07:16.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:15 smithi071 ceph-mon[36597]: pgmap v619: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:16.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:15 smithi151 ceph-mon[36481]: pgmap v619: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:16.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 244.254190 secs 2024-03-28T18:07:16.975 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 889 sec 2024-03-28T18:07:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 244.254190 secs 2024-03-28T18:07:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 244.254190 secs 2024-03-28T18:07:17.975 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 890 sec 2024-03-28T18:07:18.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:17 smithi071 ceph-mon[36597]: pgmap v620: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:17 smithi151 ceph-mon[36481]: pgmap v620: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:17 smithi045 ceph-mon[31773]: pgmap v620: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:18.976 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 891 sec 2024-03-28T18:07:19.976 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 892 sec 2024-03-28T18:07:20.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:19 smithi071 ceph-mon[36597]: pgmap v621: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:20.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:19 smithi151 ceph-mon[36481]: pgmap v621: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:20.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:19 smithi045 ceph-mon[31773]: pgmap v621: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:20.976 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 893 sec 2024-03-28T18:07:21.976 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 894 sec 2024-03-28T18:07:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:21 smithi071 ceph-mon[36597]: pgmap v622: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 249.254229 secs 2024-03-28T18:07:22.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:21 smithi151 ceph-mon[36481]: pgmap v622: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 249.254229 secs 2024-03-28T18:07:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:21 smithi045 ceph-mon[31773]: pgmap v622: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 249.254229 secs 2024-03-28T18:07:22.976 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 895 sec 2024-03-28T18:07:23.977 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 896 sec 2024-03-28T18:07:24.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:23 smithi071 ceph-mon[36597]: pgmap v623: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:24.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:23 smithi151 ceph-mon[36481]: pgmap v623: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:24.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:23 smithi045 ceph-mon[31773]: pgmap v623: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:24.976 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 897 sec 2024-03-28T18:07:25.976 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 898 sec 2024-03-28T18:07:26.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:25 smithi071 ceph-mon[36597]: pgmap v624: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:26.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:25 smithi151 ceph-mon[36481]: pgmap v624: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:26.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:25 smithi045 ceph-mon[31773]: pgmap v624: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:26.977 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 899 sec 2024-03-28T18:07:27.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 254.254316 secs 2024-03-28T18:07:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 254.254316 secs 2024-03-28T18:07:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 254.254316 secs 2024-03-28T18:07:27.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:27 smithi151 ceph-mon[36481]: pgmap v625: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:27.977 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 900 sec 2024-03-28T18:07:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:27 smithi071 ceph-mon[36597]: pgmap v625: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:27 smithi045 ceph-mon[31773]: pgmap v625: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:28.977 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 901 sec 2024-03-28T18:07:29.977 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 902 sec 2024-03-28T18:07:30.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:29 smithi071 ceph-mon[36597]: pgmap v626: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:30.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:29 smithi151 ceph-mon[36481]: pgmap v626: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:30.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:29 smithi045 ceph-mon[31773]: pgmap v626: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:30.977 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 903 sec 2024-03-28T18:07:31.977 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 904 sec 2024-03-28T18:07:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:31 smithi071 ceph-mon[36597]: pgmap v627: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 259.254404 secs 2024-03-28T18:07:32.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:31 smithi151 ceph-mon[36481]: pgmap v627: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:32.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 259.254404 secs 2024-03-28T18:07:32.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:31 smithi045 ceph-mon[31773]: pgmap v627: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:32.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 259.254404 secs 2024-03-28T18:07:32.978 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 905 sec 2024-03-28T18:07:33.978 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 906 sec 2024-03-28T18:07:34.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:33 smithi071 ceph-mon[36597]: pgmap v628: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:34.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:33 smithi151 ceph-mon[36481]: pgmap v628: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:34.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:33 smithi045 ceph-mon[31773]: pgmap v628: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:34.978 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 907 sec 2024-03-28T18:07:35.978 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 908 sec 2024-03-28T18:07:36.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:35 smithi071 ceph-mon[36597]: pgmap v629: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:36.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:35 smithi151 ceph-mon[36481]: pgmap v629: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:36.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:35 smithi045 ceph-mon[31773]: pgmap v629: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:36.978 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 909 sec 2024-03-28T18:07:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 264.254492 secs 2024-03-28T18:07:37.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 264.254492 secs 2024-03-28T18:07:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 264.254492 secs 2024-03-28T18:07:37.978 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 910 sec 2024-03-28T18:07:38.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:37 smithi071 ceph-mon[36597]: pgmap v630: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:38.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:37 smithi151 ceph-mon[36481]: pgmap v630: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:38.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:37 smithi045 ceph-mon[31773]: pgmap v630: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:38.978 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 911 sec 2024-03-28T18:07:39.978 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 912 sec 2024-03-28T18:07:40.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:39 smithi071 ceph-mon[36597]: pgmap v631: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:40.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:39 smithi151 ceph-mon[36481]: pgmap v631: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:40.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:39 smithi045 ceph-mon[31773]: pgmap v631: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:40.978 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 913 sec 2024-03-28T18:07:41.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:40 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:07:41.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:40 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:07:41.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:40 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:07:41.979 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 914 sec 2024-03-28T18:07:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:41 smithi071 ceph-mon[36597]: pgmap v632: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 269.254554 secs 2024-03-28T18:07:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:41 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:41 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:07:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:41 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:07:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:41 smithi151 ceph-mon[36481]: pgmap v632: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 269.254554 secs 2024-03-28T18:07:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:41 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:41 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:07:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:41 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:07:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:41 smithi045 ceph-mon[31773]: pgmap v632: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:42.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 269.254554 secs 2024-03-28T18:07:42.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:41 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:07:42.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:41 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:07:42.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:41 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:07:42.979 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 915 sec 2024-03-28T18:07:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:42 smithi071 ceph-mon[36597]: pgmap v633: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:42 smithi071 ceph-mon[36597]: pgmap v634: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:07:43.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:42 smithi151 ceph-mon[36481]: pgmap v633: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:43.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:42 smithi151 ceph-mon[36481]: pgmap v634: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:07:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:42 smithi045 ceph-mon[31773]: pgmap v633: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:42 smithi045 ceph-mon[31773]: pgmap v634: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:07:43.979 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 916 sec 2024-03-28T18:07:44.979 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 917 sec 2024-03-28T18:07:45.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:44 smithi071 ceph-mon[36597]: pgmap v635: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:07:45.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:44 smithi151 ceph-mon[36481]: pgmap v635: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:07:45.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:44 smithi045 ceph-mon[31773]: pgmap v635: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:07:45.979 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 918 sec 2024-03-28T18:07:46.979 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 919 sec 2024-03-28T18:07:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:46 smithi071 ceph-mon[36597]: pgmap v636: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:07:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 274.254639 secs 2024-03-28T18:07:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:46 smithi151 ceph-mon[36481]: pgmap v636: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:07:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 274.254639 secs 2024-03-28T18:07:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:46 smithi045 ceph-mon[31773]: pgmap v636: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:07:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 274.254639 secs 2024-03-28T18:07:47.979 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 920 sec 2024-03-28T18:07:48.980 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 921 sec 2024-03-28T18:07:49.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:48 smithi071 ceph-mon[36597]: pgmap v637: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:07:49.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:48 smithi151 ceph-mon[36481]: pgmap v637: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:07:49.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:48 smithi045 ceph-mon[31773]: pgmap v637: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:07:49.980 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 922 sec 2024-03-28T18:07:50.980 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 923 sec 2024-03-28T18:07:51.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:50 smithi071 ceph-mon[36597]: pgmap v638: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:07:51.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:50 smithi151 ceph-mon[36481]: pgmap v638: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:07:51.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:50 smithi045 ceph-mon[31773]: pgmap v638: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:07:51.980 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 924 sec 2024-03-28T18:07:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 279.254728 secs 2024-03-28T18:07:52.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 279.254728 secs 2024-03-28T18:07:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 279.254728 secs 2024-03-28T18:07:52.980 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 925 sec 2024-03-28T18:07:53.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:52 smithi071 ceph-mon[36597]: pgmap v639: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:07:53.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:52 smithi151 ceph-mon[36481]: pgmap v639: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:07:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:52 smithi045 ceph-mon[31773]: pgmap v639: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:07:53.980 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 926 sec 2024-03-28T18:07:54.980 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 927 sec 2024-03-28T18:07:55.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:54 smithi071 ceph-mon[36597]: pgmap v640: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:55.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:54 smithi151 ceph-mon[36481]: pgmap v640: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:55.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:54 smithi045 ceph-mon[31773]: pgmap v640: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:55.981 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 928 sec 2024-03-28T18:07:56.981 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 929 sec 2024-03-28T18:07:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:56 smithi071 ceph-mon[36597]: pgmap v641: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 284.254813 secs 2024-03-28T18:07:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:56 smithi151 ceph-mon[36481]: pgmap v641: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 284.254813 secs 2024-03-28T18:07:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:56 smithi045 ceph-mon[31773]: pgmap v641: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:07:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 284.254813 secs 2024-03-28T18:07:57.981 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 930 sec 2024-03-28T18:07:58.981 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 931 sec 2024-03-28T18:07:59.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:07:58 smithi071 ceph-mon[36597]: pgmap v642: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:59.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:07:58 smithi151 ceph-mon[36481]: pgmap v642: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:59.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:07:58 smithi045 ceph-mon[31773]: pgmap v642: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:07:59.981 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 932 sec 2024-03-28T18:08:00.981 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 933 sec 2024-03-28T18:08:01.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:00 smithi071 ceph-mon[36597]: pgmap v643: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:01.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:00 smithi151 ceph-mon[36481]: pgmap v643: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:01.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:00 smithi045 ceph-mon[31773]: pgmap v643: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:01.981 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 934 sec 2024-03-28T18:08:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 289.254874 secs 2024-03-28T18:08:02.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 289.254874 secs 2024-03-28T18:08:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 289.254874 secs 2024-03-28T18:08:02.982 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 935 sec 2024-03-28T18:08:03.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:02 smithi151 ceph-mon[36481]: pgmap v644: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:03.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:02 smithi045 ceph-mon[31773]: pgmap v644: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:02 smithi071 ceph-mon[36597]: pgmap v644: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:03.982 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 936 sec 2024-03-28T18:08:04.982 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 937 sec 2024-03-28T18:08:05.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:04 smithi151 ceph-mon[36481]: pgmap v645: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:05.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:04 smithi045 ceph-mon[31773]: pgmap v645: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:05.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:04 smithi071 ceph-mon[36597]: pgmap v645: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:05.982 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 938 sec 2024-03-28T18:08:06.982 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 939 sec 2024-03-28T18:08:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:06 smithi045 ceph-mon[31773]: pgmap v646: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 294.254945 secs 2024-03-28T18:08:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:06 smithi071 ceph-mon[36597]: pgmap v646: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 294.254945 secs 2024-03-28T18:08:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:06 smithi151 ceph-mon[36481]: pgmap v646: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 294.254945 secs 2024-03-28T18:08:07.982 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 940 sec 2024-03-28T18:08:08.982 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 941 sec 2024-03-28T18:08:09.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:08 smithi045 ceph-mon[31773]: pgmap v647: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:09.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:08 smithi071 ceph-mon[36597]: pgmap v647: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:09.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:08 smithi151 ceph-mon[36481]: pgmap v647: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:09.983 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 942 sec 2024-03-28T18:08:10.983 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 943 sec 2024-03-28T18:08:11.112 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:10 smithi071 ceph-mon[36597]: pgmap v648: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:11.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:10 smithi045 ceph-mon[31773]: pgmap v648: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:11.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:10 smithi151 ceph-mon[36481]: pgmap v648: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:11.983 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 944 sec 2024-03-28T18:08:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 299.255035 secs 2024-03-28T18:08:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 299.255035 secs 2024-03-28T18:08:12.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 299.255035 secs 2024-03-28T18:08:12.983 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 945 sec 2024-03-28T18:08:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:12 smithi045 ceph-mon[31773]: pgmap v649: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:13.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:12 smithi071 ceph-mon[36597]: pgmap v649: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:13.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:12 smithi151 ceph-mon[36481]: pgmap v649: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:13.983 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 946 sec 2024-03-28T18:08:14.983 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 947 sec 2024-03-28T18:08:15.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:14 smithi045 ceph-mon[31773]: pgmap v650: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:15.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:14 smithi071 ceph-mon[36597]: pgmap v650: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:15.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:14 smithi151 ceph-mon[36481]: pgmap v650: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:15.983 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 948 sec 2024-03-28T18:08:16.984 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 949 sec 2024-03-28T18:08:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:16 smithi045 ceph-mon[31773]: pgmap v651: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 304.255125 secs 2024-03-28T18:08:17.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:16 smithi071 ceph-mon[36597]: pgmap v651: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:17.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 304.255125 secs 2024-03-28T18:08:17.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:16 smithi151 ceph-mon[36481]: pgmap v651: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 304.255125 secs 2024-03-28T18:08:17.984 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 950 sec 2024-03-28T18:08:18.984 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 951 sec 2024-03-28T18:08:19.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:18 smithi045 ceph-mon[31773]: pgmap v652: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:19.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:18 smithi071 ceph-mon[36597]: pgmap v652: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:18 smithi151 ceph-mon[36481]: pgmap v652: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:19.984 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 952 sec 2024-03-28T18:08:20.984 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 953 sec 2024-03-28T18:08:21.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:20 smithi045 ceph-mon[31773]: pgmap v653: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:21.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:20 smithi071 ceph-mon[36597]: pgmap v653: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:21.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:20 smithi151 ceph-mon[36481]: pgmap v653: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:21.984 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 954 sec 2024-03-28T18:08:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 309.255186 secs 2024-03-28T18:08:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 309.255186 secs 2024-03-28T18:08:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 309.255186 secs 2024-03-28T18:08:22.984 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 955 sec 2024-03-28T18:08:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:22 smithi045 ceph-mon[31773]: pgmap v654: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:23.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:22 smithi071 ceph-mon[36597]: pgmap v654: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:23.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:22 smithi151 ceph-mon[36481]: pgmap v654: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:23.985 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 956 sec 2024-03-28T18:08:24.985 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 957 sec 2024-03-28T18:08:25.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:24 smithi045 ceph-mon[31773]: pgmap v655: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:25.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:24 smithi071 ceph-mon[36597]: pgmap v655: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:25.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:24 smithi151 ceph-mon[36481]: pgmap v655: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:25.985 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 958 sec 2024-03-28T18:08:26.985 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 959 sec 2024-03-28T18:08:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:26 smithi045 ceph-mon[31773]: pgmap v656: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 314.255247 secs 2024-03-28T18:08:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:26 smithi071 ceph-mon[36597]: pgmap v656: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 314.255247 secs 2024-03-28T18:08:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:26 smithi151 ceph-mon[36481]: pgmap v656: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 314.255247 secs 2024-03-28T18:08:27.985 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 960 sec 2024-03-28T18:08:28.985 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 961 sec 2024-03-28T18:08:29.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:28 smithi045 ceph-mon[31773]: pgmap v657: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:29.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:28 smithi071 ceph-mon[36597]: pgmap v657: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:29.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:28 smithi151 ceph-mon[36481]: pgmap v657: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:29.986 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 962 sec 2024-03-28T18:08:30.986 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 963 sec 2024-03-28T18:08:31.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:30 smithi045 ceph-mon[31773]: pgmap v658: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:31.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:30 smithi071 ceph-mon[36597]: pgmap v658: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:31.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:30 smithi151 ceph-mon[36481]: pgmap v658: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:31.986 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 964 sec 2024-03-28T18:08:32.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 319.255337 secs 2024-03-28T18:08:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 319.255337 secs 2024-03-28T18:08:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 319.255337 secs 2024-03-28T18:08:32.986 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 965 sec 2024-03-28T18:08:33.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:32 smithi045 ceph-mon[31773]: pgmap v659: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:33.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:32 smithi071 ceph-mon[36597]: pgmap v659: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:33.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:32 smithi151 ceph-mon[36481]: pgmap v659: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:33.986 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 966 sec 2024-03-28T18:08:34.987 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 967 sec 2024-03-28T18:08:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:34 smithi045 ceph-mon[31773]: pgmap v660: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:35.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:34 smithi071 ceph-mon[36597]: pgmap v660: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:35.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:34 smithi151 ceph-mon[36481]: pgmap v660: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:35.986 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 968 sec 2024-03-28T18:08:36.987 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 969 sec 2024-03-28T18:08:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:36 smithi071 ceph-mon[36597]: pgmap v661: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 324.255419 secs 2024-03-28T18:08:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:36 smithi151 ceph-mon[36481]: pgmap v661: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 324.255419 secs 2024-03-28T18:08:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:36 smithi045 ceph-mon[31773]: pgmap v661: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 324.255419 secs 2024-03-28T18:08:37.987 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 970 sec 2024-03-28T18:08:38.987 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 971 sec 2024-03-28T18:08:39.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:38 smithi071 ceph-mon[36597]: pgmap v662: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:39.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:38 smithi151 ceph-mon[36481]: pgmap v662: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:39.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:38 smithi045 ceph-mon[31773]: pgmap v662: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:39.987 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 972 sec 2024-03-28T18:08:40.987 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 973 sec 2024-03-28T18:08:41.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:40 smithi071 ceph-mon[36597]: pgmap v663: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:41.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:40 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 329.255506 secs 2024-03-28T18:08:41.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:40 smithi151 ceph-mon[36481]: pgmap v663: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:41.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:40 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 329.255506 secs 2024-03-28T18:08:41.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:40 smithi045 ceph-mon[31773]: pgmap v663: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:41.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:40 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 329.255506 secs 2024-03-28T18:08:41.989 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 974 sec 2024-03-28T18:08:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:41 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:08:42.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:41 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:08:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:41 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:08:42.987 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 975 sec 2024-03-28T18:08:43.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:42 smithi071 ceph-mon[36597]: pgmap v664: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:43.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:42 smithi151 ceph-mon[36481]: pgmap v664: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:43.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:42 smithi045 ceph-mon[31773]: pgmap v664: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:43.988 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 976 sec 2024-03-28T18:08:44.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:43 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:08:44.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:43 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:08:44.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:43 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:08:44.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:43 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:08:44.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:43 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:08:44.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:43 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:08:44.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:43 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:08:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:43 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:08:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:43 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:08:44.988 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 977 sec 2024-03-28T18:08:45.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:44 smithi071 ceph-mon[36597]: pgmap v665: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:45.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:44 smithi071 ceph-mon[36597]: pgmap v666: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:08:45.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:44 smithi151 ceph-mon[36481]: pgmap v665: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:45.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:44 smithi151 ceph-mon[36481]: pgmap v666: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:08:45.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:44 smithi045 ceph-mon[31773]: pgmap v665: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:45.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:44 smithi045 ceph-mon[31773]: pgmap v666: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:08:45.988 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 978 sec 2024-03-28T18:08:46.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:45 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 334.255589 secs 2024-03-28T18:08:46.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:45 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 334.255589 secs 2024-03-28T18:08:46.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:45 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 334.255589 secs 2024-03-28T18:08:46.988 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 979 sec 2024-03-28T18:08:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:46 smithi071 ceph-mon[36597]: pgmap v667: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:08:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:46 smithi151 ceph-mon[36481]: pgmap v667: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:08:47.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:46 smithi045 ceph-mon[31773]: pgmap v667: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:08:47.988 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 980 sec 2024-03-28T18:08:48.988 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 981 sec 2024-03-28T18:08:49.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:48 smithi071 ceph-mon[36597]: pgmap v668: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:08:49.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:48 smithi151 ceph-mon[36481]: pgmap v668: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:08:49.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:48 smithi045 ceph-mon[31773]: pgmap v668: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:08:49.988 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 982 sec 2024-03-28T18:08:50.988 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 983 sec 2024-03-28T18:08:51.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:50 smithi071 ceph-mon[36597]: pgmap v669: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:08:51.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:50 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 339.255649 secs 2024-03-28T18:08:51.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:50 smithi151 ceph-mon[36481]: pgmap v669: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:08:51.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:50 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 339.255649 secs 2024-03-28T18:08:51.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:50 smithi045 ceph-mon[31773]: pgmap v669: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:08:51.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:50 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 339.255649 secs 2024-03-28T18:08:51.989 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 984 sec 2024-03-28T18:08:52.989 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 985 sec 2024-03-28T18:08:53.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:52 smithi071 ceph-mon[36597]: pgmap v670: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:08:53.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:52 smithi151 ceph-mon[36481]: pgmap v670: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:08:53.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:52 smithi045 ceph-mon[31773]: pgmap v670: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:08:53.989 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 986 sec 2024-03-28T18:08:54.989 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 987 sec 2024-03-28T18:08:55.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:54 smithi071 ceph-mon[36597]: pgmap v671: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:08:55.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:54 smithi151 ceph-mon[36481]: pgmap v671: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:08:55.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:54 smithi045 ceph-mon[31773]: pgmap v671: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:08:55.989 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 988 sec 2024-03-28T18:08:56.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:55 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 344.255735 secs 2024-03-28T18:08:56.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:55 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 344.255735 secs 2024-03-28T18:08:56.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:55 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 344.255735 secs 2024-03-28T18:08:56.989 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 989 sec 2024-03-28T18:08:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:56 smithi071 ceph-mon[36597]: pgmap v672: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:57.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:56 smithi151 ceph-mon[36481]: pgmap v672: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:57.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:56 smithi045 ceph-mon[31773]: pgmap v672: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:08:57.989 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 990 sec 2024-03-28T18:08:58.990 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 991 sec 2024-03-28T18:08:59.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:08:58 smithi071 ceph-mon[36597]: pgmap v673: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:59.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:08:58 smithi151 ceph-mon[36481]: pgmap v673: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:59.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:08:58 smithi045 ceph-mon[31773]: pgmap v673: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:08:59.990 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 992 sec 2024-03-28T18:09:00.990 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 993 sec 2024-03-28T18:09:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:01 smithi071 ceph-mon[36597]: pgmap v674: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 349.255798 secs 2024-03-28T18:09:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:01 smithi151 ceph-mon[36481]: pgmap v674: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 349.255798 secs 2024-03-28T18:09:01.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:01 smithi045 ceph-mon[31773]: pgmap v674: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:01.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 349.255798 secs 2024-03-28T18:09:01.990 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 994 sec 2024-03-28T18:09:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:02 smithi071 ceph-mon[36597]: pgmap v675: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:09:02.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:02 smithi151 ceph-mon[36481]: pgmap v675: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:09:02.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:02 smithi045 ceph-mon[31773]: pgmap v675: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:09:02.990 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 995 sec 2024-03-28T18:09:03.990 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 996 sec 2024-03-28T18:09:04.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:04 smithi045 ceph-mon[31773]: pgmap v676: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:04.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:04 smithi071 ceph-mon[36597]: pgmap v676: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:04.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:04 smithi151 ceph-mon[36481]: pgmap v676: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:04.990 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 997 sec 2024-03-28T18:09:05.990 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 998 sec 2024-03-28T18:09:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:06 smithi045 ceph-mon[31773]: pgmap v677: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:09:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 354.255886 secs 2024-03-28T18:09:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:06 smithi071 ceph-mon[36597]: pgmap v677: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:09:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 354.255886 secs 2024-03-28T18:09:06.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:06 smithi151 ceph-mon[36481]: pgmap v677: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:09:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 354.255886 secs 2024-03-28T18:09:06.990 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 999 sec 2024-03-28T18:09:07.991 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1000 sec 2024-03-28T18:09:08.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:08 smithi045 ceph-mon[31773]: pgmap v678: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T18:09:08.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:08 smithi071 ceph-mon[36597]: pgmap v678: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T18:09:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:08 smithi151 ceph-mon[36481]: pgmap v678: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T18:09:08.991 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1001 sec 2024-03-28T18:09:09.991 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1002 sec 2024-03-28T18:09:10.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:10 smithi045 ceph-mon[31773]: pgmap v679: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T18:09:10.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:10 smithi071 ceph-mon[36597]: pgmap v679: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T18:09:10.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:10 smithi151 ceph-mon[36481]: pgmap v679: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T18:09:10.991 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1003 sec 2024-03-28T18:09:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 359.255964 secs 2024-03-28T18:09:11.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 359.255964 secs 2024-03-28T18:09:11.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 359.255964 secs 2024-03-28T18:09:11.991 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1004 sec 2024-03-28T18:09:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:12 smithi045 ceph-mon[31773]: pgmap v680: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:09:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:12 smithi071 ceph-mon[36597]: pgmap v680: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:09:12.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:12 smithi151 ceph-mon[36481]: pgmap v680: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:09:12.991 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1005 sec 2024-03-28T18:09:13.992 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1006 sec 2024-03-28T18:09:14.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:14 smithi045 ceph-mon[31773]: pgmap v681: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T18:09:14.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:14 smithi071 ceph-mon[36597]: pgmap v681: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T18:09:14.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:14 smithi151 ceph-mon[36481]: pgmap v681: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T18:09:14.992 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1007 sec 2024-03-28T18:09:15.992 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1008 sec 2024-03-28T18:09:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:16 smithi045 ceph-mon[31773]: pgmap v682: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:09:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:16 smithi045 ceph-mon[31773]: 2 slow requests, 2 included below; oldest blocked for > 364.256004 secs 2024-03-28T18:09:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:16 smithi045 ceph-mon[31773]: slow request 364.256003 seconds old, received at 2024-03-28T18:03:11.560545+0000: peer_request:client.15069:1629 currently acquired locks 2024-03-28T18:09:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:16 smithi045 ceph-mon[31773]: slow request 364.159878 seconds old, received at 2024-03-28T18:03:11.656670+0000: client_request(client.25691:164 rmsnap #0x1//scheduled-2024-03-28-17_57_00_UTC 2024-03-28T18:03:11.656797+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:09:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:16 smithi071 ceph-mon[36597]: pgmap v682: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:09:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:16 smithi071 ceph-mon[36597]: 2 slow requests, 2 included below; oldest blocked for > 364.256004 secs 2024-03-28T18:09:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:16 smithi071 ceph-mon[36597]: slow request 364.256003 seconds old, received at 2024-03-28T18:03:11.560545+0000: peer_request:client.15069:1629 currently acquired locks 2024-03-28T18:09:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:16 smithi071 ceph-mon[36597]: slow request 364.159878 seconds old, received at 2024-03-28T18:03:11.656670+0000: client_request(client.25691:164 rmsnap #0x1//scheduled-2024-03-28-17_57_00_UTC 2024-03-28T18:03:11.656797+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:09:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:16 smithi151 ceph-mon[36481]: pgmap v682: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T18:09:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:16 smithi151 ceph-mon[36481]: 2 slow requests, 2 included below; oldest blocked for > 364.256004 secs 2024-03-28T18:09:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:16 smithi151 ceph-mon[36481]: slow request 364.256003 seconds old, received at 2024-03-28T18:03:11.560545+0000: peer_request:client.15069:1629 currently acquired locks 2024-03-28T18:09:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:16 smithi151 ceph-mon[36481]: slow request 364.159878 seconds old, received at 2024-03-28T18:03:11.656670+0000: client_request(client.25691:164 rmsnap #0x1//scheduled-2024-03-28-17_57_00_UTC 2024-03-28T18:03:11.656797+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:09:16.992 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1009 sec 2024-03-28T18:09:17.992 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1010 sec 2024-03-28T18:09:18.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:18 smithi045 ceph-mon[31773]: pgmap v683: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:18.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:18 smithi071 ceph-mon[36597]: pgmap v683: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:18.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:18 smithi151 ceph-mon[36481]: pgmap v683: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:18.992 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1011 sec 2024-03-28T18:09:19.993 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1012 sec 2024-03-28T18:09:20.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:20 smithi045 ceph-mon[31773]: pgmap v684: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:20.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:20 smithi071 ceph-mon[36597]: pgmap v684: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:20.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:20 smithi151 ceph-mon[36481]: pgmap v684: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:20.992 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1013 sec 2024-03-28T18:09:21.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 369.256066 secs 2024-03-28T18:09:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 369.256066 secs 2024-03-28T18:09:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 369.256066 secs 2024-03-28T18:09:21.993 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1014 sec 2024-03-28T18:09:22.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:22 smithi045 ceph-mon[31773]: pgmap v685: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:09:22.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:22 smithi071 ceph-mon[36597]: pgmap v685: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:09:22.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:22 smithi151 ceph-mon[36481]: pgmap v685: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:09:22.993 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1015 sec 2024-03-28T18:09:23.993 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1016 sec 2024-03-28T18:09:24.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:24 smithi045 ceph-mon[31773]: pgmap v686: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:24.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:24 smithi071 ceph-mon[36597]: pgmap v686: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:24.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:24 smithi151 ceph-mon[36481]: pgmap v686: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:24.993 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1017 sec 2024-03-28T18:09:25.993 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1018 sec 2024-03-28T18:09:26.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:26 smithi045 ceph-mon[31773]: pgmap v687: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:09:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 374.256107 secs 2024-03-28T18:09:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:26 smithi071 ceph-mon[36597]: pgmap v687: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:09:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 374.256107 secs 2024-03-28T18:09:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:26 smithi151 ceph-mon[36481]: pgmap v687: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:09:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 374.256107 secs 2024-03-28T18:09:26.993 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1019 sec 2024-03-28T18:09:27.994 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1020 sec 2024-03-28T18:09:28.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:28 smithi045 ceph-mon[31773]: pgmap v688: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:28.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:28 smithi071 ceph-mon[36597]: pgmap v688: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:28.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:28 smithi151 ceph-mon[36481]: pgmap v688: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:28.994 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1021 sec 2024-03-28T18:09:29.994 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1022 sec 2024-03-28T18:09:30.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:30 smithi045 ceph-mon[31773]: pgmap v689: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:30.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:30 smithi071 ceph-mon[36597]: pgmap v689: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:30.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:30 smithi151 ceph-mon[36481]: pgmap v689: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:30.994 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1023 sec 2024-03-28T18:09:31.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:09:30 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:09:30.878+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:09:31.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:09:30 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:09:30.878+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:09:31.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:09:30 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:09:30.878+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:09:31.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:09:30 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:09:30.878+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:09:31.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:09:30 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:09:30.878+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:09:31.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 379.256167 secs 2024-03-28T18:09:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 379.256167 secs 2024-03-28T18:09:31.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 379.256167 secs 2024-03-28T18:09:31.994 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1024 sec 2024-03-28T18:09:32.994 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1025 sec 2024-03-28T18:09:33.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:32 smithi151 ceph-mon[36481]: pgmap v690: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:09:33.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:32 smithi045 ceph-mon[31773]: pgmap v690: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:09:33.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:32 smithi071 ceph-mon[36597]: pgmap v690: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:09:33.994 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1026 sec 2024-03-28T18:09:34.995 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1027 sec 2024-03-28T18:09:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:34 smithi151 ceph-mon[36481]: pgmap v691: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:34 smithi045 ceph-mon[31773]: pgmap v691: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:35.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:34 smithi071 ceph-mon[36597]: pgmap v691: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:35.995 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1028 sec 2024-03-28T18:09:36.995 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1029 sec 2024-03-28T18:09:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:36 smithi045 ceph-mon[31773]: pgmap v692: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:09:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 384.256256 secs 2024-03-28T18:09:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:36 smithi071 ceph-mon[36597]: pgmap v692: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:09:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 384.256256 secs 2024-03-28T18:09:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:36 smithi151 ceph-mon[36481]: pgmap v692: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:09:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 384.256256 secs 2024-03-28T18:09:37.995 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1030 sec 2024-03-28T18:09:38.995 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1031 sec 2024-03-28T18:09:39.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:38 smithi045 ceph-mon[31773]: pgmap v693: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:09:39.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:38 smithi071 ceph-mon[36597]: pgmap v693: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:09:39.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:38 smithi151 ceph-mon[36481]: pgmap v693: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:09:39.995 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1032 sec 2024-03-28T18:09:40.995 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1033 sec 2024-03-28T18:09:41.112 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:40 smithi071 ceph-mon[36597]: pgmap v694: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:09:41.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:40 smithi045 ceph-mon[31773]: pgmap v694: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:09:41.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:40 smithi151 ceph-mon[36481]: pgmap v694: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:09:41.996 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1034 sec 2024-03-28T18:09:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 389.256339 secs 2024-03-28T18:09:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 389.256339 secs 2024-03-28T18:09:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 389.256339 secs 2024-03-28T18:09:42.996 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1035 sec 2024-03-28T18:09:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:42 smithi045 ceph-mon[31773]: pgmap v695: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:09:43.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:42 smithi071 ceph-mon[36597]: pgmap v695: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:09:43.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:42 smithi151 ceph-mon[36481]: pgmap v695: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:09:43.996 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1036 sec 2024-03-28T18:09:44.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:43 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:09:44.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:43 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:09:44.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:43 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:09:44.996 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1037 sec 2024-03-28T18:09:45.135 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:44 smithi045 ceph-mon[31773]: pgmap v696: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:09:45.135 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:44 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:45.135 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:44 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:45.135 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:44 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:45.135 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:44 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:45.136 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:44 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:45.136 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:44 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:45.208 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:44 smithi071 ceph-mon[36597]: pgmap v696: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:09:45.208 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:44 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:45.209 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:44 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:45.209 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:44 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:45.209 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:44 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:45.209 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:44 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:45.209 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:44 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:45.216 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:44 smithi151 ceph-mon[36481]: pgmap v696: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:09:45.216 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:44 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:45.216 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:44 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:45.216 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:44 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:45.216 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:44 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:45.216 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:44 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:45.217 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:44 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:45.996 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1038 sec 2024-03-28T18:09:46.996 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1039 sec 2024-03-28T18:09:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:46 smithi045 ceph-mon[31773]: pgmap v697: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:09:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 394.256430 secs 2024-03-28T18:09:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:46 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:46 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:47.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:46 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:47.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:46 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:46 smithi071 ceph-mon[36597]: pgmap v697: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:09:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 394.256430 secs 2024-03-28T18:09:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:46 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:46 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:47.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:46 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:47.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:46 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:46 smithi151 ceph-mon[36481]: pgmap v697: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:09:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 394.256430 secs 2024-03-28T18:09:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:46 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:46 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:46 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:46 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:47.996 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1040 sec 2024-03-28T18:09:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:47 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:47 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:47 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:47 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:47 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:47 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:47 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:47 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:47 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:47 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:47 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:47 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:47 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:47 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:47 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:47 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:47 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:47 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:47 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:47 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:47 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:47 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:47 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:47 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:47 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:47 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:47 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:47 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:47 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:47 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:47 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:47 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:47 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:47 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:47 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:47 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:47 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:47 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:47 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:47 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:47 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:47 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:47 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:47 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:47 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:47 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:47 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:47 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:48.996 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1041 sec 2024-03-28T18:09:49.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:48 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T18:09:49.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:48 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T18:09:49.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:48 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T18:09:49.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:48 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T18:09:49.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:48 smithi045 ceph-mon[31773]: pgmap v698: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:49.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:48 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T18:09:49.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:48 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T18:09:49.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:48 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T18:09:49.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:48 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T18:09:49.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:48 smithi071 ceph-mon[36597]: pgmap v698: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:49.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:48 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T18:09:49.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:48 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T18:09:49.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:48 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T18:09:49.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:48 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T18:09:49.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:48 smithi151 ceph-mon[36481]: pgmap v698: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:49.996 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1042 sec 2024-03-28T18:09:50.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:50 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:50.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:50 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:50.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:50 smithi045 ceph-mon[31773]: pgmap v699: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:50.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:50 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:50.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:50 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:50.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:50 smithi071 ceph-mon[36597]: pgmap v699: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:50.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:50 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:50.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:50 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:50.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:50 smithi151 ceph-mon[36481]: pgmap v699: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:09:50.997 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1043 sec 2024-03-28T18:09:51.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:51 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:51 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:51 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:51 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:51 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:51 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:51 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:51 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:51 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T18:09:51.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:51 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:09:51.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:51 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:09:51.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:51 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:09:51.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:51 smithi045 ceph-mon[31773]: pgmap v700: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T18:09:51.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:51 smithi045 ceph-mon[31773]: pgmap v701: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:09:51.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:51 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:51.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 399.256521 secs 2024-03-28T18:09:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:51 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:51 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:51 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:51 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:51 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:51 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:51 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:51 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:51 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T18:09:51.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:51 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:09:51.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:51 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:09:51.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:51 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:09:51.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:51 smithi071 ceph-mon[36597]: pgmap v700: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T18:09:51.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:51 smithi071 ceph-mon[36597]: pgmap v701: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:09:51.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:51 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:51.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 399.256521 secs 2024-03-28T18:09:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:51 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:51 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:51 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:51 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:51 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:51 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:51 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:51 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:09:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:51 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T18:09:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:51 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:09:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:51 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:09:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:51 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:09:51.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:51 smithi151 ceph-mon[36481]: pgmap v700: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T18:09:51.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:51 smithi151 ceph-mon[36481]: pgmap v701: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:09:51.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:51 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:09:51.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 399.256521 secs 2024-03-28T18:09:51.997 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1044 sec 2024-03-28T18:09:52.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:52 smithi045 ceph-mon[31773]: Health check update: 8 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-28T18:09:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:52 smithi071 ceph-mon[36597]: Health check update: 8 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-28T18:09:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:52 smithi151 ceph-mon[36481]: Health check update: 8 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-28T18:09:52.997 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1045 sec 2024-03-28T18:09:53.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:53 smithi045 ceph-mon[31773]: pgmap v702: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:09:53.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:53 smithi071 ceph-mon[36597]: pgmap v702: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:09:53.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:53 smithi151 ceph-mon[36481]: pgmap v702: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:09:53.997 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1046 sec 2024-03-28T18:09:54.997 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1047 sec 2024-03-28T18:09:55.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:55 smithi045 ceph-mon[31773]: pgmap v703: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:09:55.997 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1048 sec 2024-03-28T18:09:56.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:55 smithi071 ceph-mon[36597]: pgmap v703: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:09:56.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:55 smithi151 ceph-mon[36481]: pgmap v703: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:09:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 404.256582 secs 2024-03-28T18:09:56.997 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1049 sec 2024-03-28T18:09:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 404.256582 secs 2024-03-28T18:09:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 404.256582 secs 2024-03-28T18:09:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:57 smithi045 ceph-mon[31773]: pgmap v704: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:09:57.997 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1050 sec 2024-03-28T18:09:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:57 smithi071 ceph-mon[36597]: pgmap v704: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:09:58.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:57 smithi151 ceph-mon[36481]: pgmap v704: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:09:58.998 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1051 sec 2024-03-28T18:09:59.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:09:59 smithi045 ceph-mon[31773]: pgmap v705: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:09:59.998 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1052 sec 2024-03-28T18:10:00.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:09:59 smithi071 ceph-mon[36597]: pgmap v705: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:10:00.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:09:59 smithi151 ceph-mon[36481]: pgmap v705: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:10:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:00 smithi045 ceph-mon[31773]: Health detail: HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-28T18:10:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:00 smithi045 ceph-mon[31773]: [WRN] CEPHADM_FAILED_DAEMON: 8 failed cephadm daemon(s) 2024-03-28T18:10:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:00 smithi045 ceph-mon[31773]: daemon mds.j on smithi045 is in error state 2024-03-28T18:10:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:00 smithi045 ceph-mon[31773]: daemon mds.d on smithi045 is in error state 2024-03-28T18:10:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:00 smithi045 ceph-mon[31773]: daemon mds.g on smithi045 is in error state 2024-03-28T18:10:00.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:00 smithi045 ceph-mon[31773]: daemon mds.a on smithi045 is in error state 2024-03-28T18:10:00.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:00 smithi045 ceph-mon[31773]: daemon mds.k on smithi071 is in error state 2024-03-28T18:10:00.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:00 smithi045 ceph-mon[31773]: daemon mds.l on smithi151 is in error state 2024-03-28T18:10:00.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:00 smithi045 ceph-mon[31773]: daemon mds.f on smithi151 is in error state 2024-03-28T18:10:00.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:00 smithi045 ceph-mon[31773]: daemon mds.i on smithi151 is in error state 2024-03-28T18:10:00.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:00 smithi045 ceph-mon[31773]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-03-28T18:10:00.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:00 smithi045 ceph-mon[31773]: mds.b(mds.0): 2 slow requests are blocked > 180 secs 2024-03-28T18:10:00.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:00 smithi151 ceph-mon[36481]: Health detail: HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-28T18:10:00.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:00 smithi151 ceph-mon[36481]: [WRN] CEPHADM_FAILED_DAEMON: 8 failed cephadm daemon(s) 2024-03-28T18:10:00.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:00 smithi151 ceph-mon[36481]: daemon mds.j on smithi045 is in error state 2024-03-28T18:10:00.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:00 smithi151 ceph-mon[36481]: daemon mds.d on smithi045 is in error state 2024-03-28T18:10:00.975 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:00 smithi151 ceph-mon[36481]: daemon mds.g on smithi045 is in error state 2024-03-28T18:10:00.975 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:00 smithi151 ceph-mon[36481]: daemon mds.a on smithi045 is in error state 2024-03-28T18:10:00.975 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:00 smithi151 ceph-mon[36481]: daemon mds.k on smithi071 is in error state 2024-03-28T18:10:00.975 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:00 smithi151 ceph-mon[36481]: daemon mds.l on smithi151 is in error state 2024-03-28T18:10:00.975 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:00 smithi151 ceph-mon[36481]: daemon mds.f on smithi151 is in error state 2024-03-28T18:10:00.975 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:00 smithi151 ceph-mon[36481]: daemon mds.i on smithi151 is in error state 2024-03-28T18:10:00.975 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:00 smithi151 ceph-mon[36481]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-03-28T18:10:00.975 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:00 smithi151 ceph-mon[36481]: mds.b(mds.0): 2 slow requests are blocked > 180 secs 2024-03-28T18:10:00.998 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1053 sec 2024-03-28T18:10:01.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:00 smithi071 ceph-mon[36597]: Health detail: HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-28T18:10:01.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:00 smithi071 ceph-mon[36597]: [WRN] CEPHADM_FAILED_DAEMON: 8 failed cephadm daemon(s) 2024-03-28T18:10:01.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:00 smithi071 ceph-mon[36597]: daemon mds.j on smithi045 is in error state 2024-03-28T18:10:01.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:00 smithi071 ceph-mon[36597]: daemon mds.d on smithi045 is in error state 2024-03-28T18:10:01.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:00 smithi071 ceph-mon[36597]: daemon mds.g on smithi045 is in error state 2024-03-28T18:10:01.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:00 smithi071 ceph-mon[36597]: daemon mds.a on smithi045 is in error state 2024-03-28T18:10:01.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:00 smithi071 ceph-mon[36597]: daemon mds.k on smithi071 is in error state 2024-03-28T18:10:01.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:00 smithi071 ceph-mon[36597]: daemon mds.l on smithi151 is in error state 2024-03-28T18:10:01.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:00 smithi071 ceph-mon[36597]: daemon mds.f on smithi151 is in error state 2024-03-28T18:10:01.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:00 smithi071 ceph-mon[36597]: daemon mds.i on smithi151 is in error state 2024-03-28T18:10:01.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:00 smithi071 ceph-mon[36597]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-03-28T18:10:01.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:00 smithi071 ceph-mon[36597]: mds.b(mds.0): 2 slow requests are blocked > 180 secs 2024-03-28T18:10:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:01 smithi045 ceph-mon[31773]: pgmap v706: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:10:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 409.256643 secs 2024-03-28T18:10:01.998 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1054 sec 2024-03-28T18:10:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:01 smithi071 ceph-mon[36597]: pgmap v706: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:10:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 409.256643 secs 2024-03-28T18:10:02.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:01 smithi151 ceph-mon[36481]: pgmap v706: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:10:02.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 409.256643 secs 2024-03-28T18:10:02.998 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1055 sec 2024-03-28T18:10:03.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:03 smithi045 ceph-mon[31773]: pgmap v707: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:03.998 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1056 sec 2024-03-28T18:10:04.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:03 smithi071 ceph-mon[36597]: pgmap v707: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:04.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:03 smithi151 ceph-mon[36481]: pgmap v707: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:04.998 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1057 sec 2024-03-28T18:10:05.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:05 smithi045 ceph-mon[31773]: pgmap v708: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:05.998 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1058 sec 2024-03-28T18:10:06.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:05 smithi071 ceph-mon[36597]: pgmap v708: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:06.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:05 smithi151 ceph-mon[36481]: pgmap v708: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:06.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 414.256704 secs 2024-03-28T18:10:06.998 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1059 sec 2024-03-28T18:10:07.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 414.256704 secs 2024-03-28T18:10:07.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 414.256704 secs 2024-03-28T18:10:07.999 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1060 sec 2024-03-28T18:10:08.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:07 smithi071 ceph-mon[36597]: pgmap v709: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:10:08.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:07 smithi151 ceph-mon[36481]: pgmap v709: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:10:08.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:07 smithi045 ceph-mon[31773]: pgmap v709: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:10:08.999 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1061 sec 2024-03-28T18:10:09.999 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1062 sec 2024-03-28T18:10:10.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:09 smithi071 ceph-mon[36597]: pgmap v710: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:10:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:09 smithi151 ceph-mon[36481]: pgmap v710: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:10:10.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:09 smithi045 ceph-mon[31773]: pgmap v710: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:10:11.002 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1063 sec 2024-03-28T18:10:11.999 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1064 sec 2024-03-28T18:10:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:11 smithi071 ceph-mon[36597]: pgmap v711: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:10:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 419.256722 secs 2024-03-28T18:10:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:11 smithi151 ceph-mon[36481]: pgmap v711: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:10:12.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 419.256722 secs 2024-03-28T18:10:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:11 smithi045 ceph-mon[31773]: pgmap v711: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:10:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 419.256722 secs 2024-03-28T18:10:13.000 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1065 sec 2024-03-28T18:10:14.000 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1066 sec 2024-03-28T18:10:14.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:13 smithi071 ceph-mon[36597]: pgmap v712: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:10:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:13 smithi151 ceph-mon[36481]: pgmap v712: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:10:14.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:13 smithi045 ceph-mon[31773]: pgmap v712: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:10:15.000 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1067 sec 2024-03-28T18:10:16.000 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1068 sec 2024-03-28T18:10:16.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:15 smithi071 ceph-mon[36597]: pgmap v713: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:10:16.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:15 smithi151 ceph-mon[36481]: pgmap v713: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:10:16.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:15 smithi045 ceph-mon[31773]: pgmap v713: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:10:17.000 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1069 sec 2024-03-28T18:10:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 424.256735 secs 2024-03-28T18:10:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 424.256735 secs 2024-03-28T18:10:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 424.256735 secs 2024-03-28T18:10:18.001 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1070 sec 2024-03-28T18:10:18.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:17 smithi071 ceph-mon[36597]: pgmap v714: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:10:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:17 smithi151 ceph-mon[36481]: pgmap v714: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:10:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:17 smithi045 ceph-mon[31773]: pgmap v714: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:10:19.000 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1071 sec 2024-03-28T18:10:20.000 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1072 sec 2024-03-28T18:10:20.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:19 smithi071 ceph-mon[36597]: pgmap v715: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:20.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:19 smithi151 ceph-mon[36481]: pgmap v715: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:20.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:19 smithi045 ceph-mon[31773]: pgmap v715: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:21.001 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1073 sec 2024-03-28T18:10:21.968 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:21 smithi045 ceph-mon[31773]: pgmap v716: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:21.969 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 429.256797 secs 2024-03-28T18:10:22.001 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1074 sec 2024-03-28T18:10:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:21 smithi071 ceph-mon[36597]: pgmap v716: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 429.256797 secs 2024-03-28T18:10:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:21 smithi151 ceph-mon[36481]: pgmap v716: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 429.256797 secs 2024-03-28T18:10:23.001 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1075 sec 2024-03-28T18:10:24.001 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1076 sec 2024-03-28T18:10:24.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:23 smithi071 ceph-mon[36597]: pgmap v717: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:24.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:23 smithi151 ceph-mon[36481]: pgmap v717: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:24.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:23 smithi045 ceph-mon[31773]: pgmap v717: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:25.001 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1077 sec 2024-03-28T18:10:26.001 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1078 sec 2024-03-28T18:10:26.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:25 smithi071 ceph-mon[36597]: pgmap v718: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:26.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:25 smithi151 ceph-mon[36481]: pgmap v718: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:26.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:25 smithi045 ceph-mon[31773]: pgmap v718: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:27.002 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1079 sec 2024-03-28T18:10:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 434.256884 secs 2024-03-28T18:10:27.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 434.256884 secs 2024-03-28T18:10:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 434.256884 secs 2024-03-28T18:10:28.002 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1080 sec 2024-03-28T18:10:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:27 smithi071 ceph-mon[36597]: pgmap v719: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:27 smithi151 ceph-mon[36481]: pgmap v719: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:27 smithi045 ceph-mon[31773]: pgmap v719: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:29.002 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1081 sec 2024-03-28T18:10:30.002 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1082 sec 2024-03-28T18:10:30.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:29 smithi071 ceph-mon[36597]: pgmap v720: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:30.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:29 smithi151 ceph-mon[36481]: pgmap v720: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:30.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:29 smithi045 ceph-mon[31773]: pgmap v720: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:31.002 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1083 sec 2024-03-28T18:10:32.002 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1084 sec 2024-03-28T18:10:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:31 smithi071 ceph-mon[36597]: pgmap v721: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 439.256973 secs 2024-03-28T18:10:32.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:31 smithi151 ceph-mon[36481]: pgmap v721: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:32.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 439.256973 secs 2024-03-28T18:10:32.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:31 smithi045 ceph-mon[31773]: pgmap v721: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:32.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 439.256973 secs 2024-03-28T18:10:33.002 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1085 sec 2024-03-28T18:10:34.002 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1086 sec 2024-03-28T18:10:34.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:33 smithi071 ceph-mon[36597]: pgmap v722: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:34.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:33 smithi151 ceph-mon[36481]: pgmap v722: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:34.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:33 smithi045 ceph-mon[31773]: pgmap v722: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:35.003 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1087 sec 2024-03-28T18:10:36.003 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1088 sec 2024-03-28T18:10:36.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:35 smithi071 ceph-mon[36597]: pgmap v723: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:36.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:35 smithi151 ceph-mon[36481]: pgmap v723: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:36.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:35 smithi045 ceph-mon[31773]: pgmap v723: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:37.003 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1089 sec 2024-03-28T18:10:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 444.257012 secs 2024-03-28T18:10:37.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 444.257012 secs 2024-03-28T18:10:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 444.257012 secs 2024-03-28T18:10:38.003 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1090 sec 2024-03-28T18:10:38.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:37 smithi071 ceph-mon[36597]: pgmap v724: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:38.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:37 smithi151 ceph-mon[36481]: pgmap v724: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:38.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:37 smithi045 ceph-mon[31773]: pgmap v724: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:39.003 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1091 sec 2024-03-28T18:10:40.003 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1092 sec 2024-03-28T18:10:40.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:39 smithi071 ceph-mon[36597]: pgmap v725: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:40.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:39 smithi151 ceph-mon[36481]: pgmap v725: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:40.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:39 smithi045 ceph-mon[31773]: pgmap v725: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:41.004 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1093 sec 2024-03-28T18:10:42.004 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1094 sec 2024-03-28T18:10:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:41 smithi071 ceph-mon[36597]: pgmap v726: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 449.257100 secs 2024-03-28T18:10:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:41 smithi151 ceph-mon[36481]: pgmap v726: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 449.257100 secs 2024-03-28T18:10:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:41 smithi045 ceph-mon[31773]: pgmap v726: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:42.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 449.257100 secs 2024-03-28T18:10:43.004 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1095 sec 2024-03-28T18:10:44.004 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1096 sec 2024-03-28T18:10:44.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:43 smithi071 ceph-mon[36597]: pgmap v727: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:44.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:43 smithi151 ceph-mon[36481]: pgmap v727: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:44.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:43 smithi045 ceph-mon[31773]: pgmap v727: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:45.004 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1097 sec 2024-03-28T18:10:46.004 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1098 sec 2024-03-28T18:10:46.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:45 smithi071 ceph-mon[36597]: pgmap v728: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:46.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:45 smithi151 ceph-mon[36481]: pgmap v728: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:46.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:45 smithi045 ceph-mon[31773]: pgmap v728: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:47.004 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1099 sec 2024-03-28T18:10:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 454.257189 secs 2024-03-28T18:10:47.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 454.257189 secs 2024-03-28T18:10:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 454.257189 secs 2024-03-28T18:10:48.004 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1100 sec 2024-03-28T18:10:48.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:47 smithi071 ceph-mon[36597]: pgmap v729: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:48.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:47 smithi151 ceph-mon[36481]: pgmap v729: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:48.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:47 smithi045 ceph-mon[31773]: pgmap v729: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:49.004 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1101 sec 2024-03-28T18:10:50.005 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1102 sec 2024-03-28T18:10:50.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:49 smithi071 ceph-mon[36597]: pgmap v730: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:50.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:49 smithi151 ceph-mon[36481]: pgmap v730: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:50.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:49 smithi045 ceph-mon[31773]: pgmap v730: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:51.005 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1103 sec 2024-03-28T18:10:52.005 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1104 sec 2024-03-28T18:10:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:51 smithi071 ceph-mon[36597]: pgmap v731: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:51 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:10:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 459.257278 secs 2024-03-28T18:10:52.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:51 smithi151 ceph-mon[36481]: pgmap v731: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:52.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:51 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:10:52.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 459.257278 secs 2024-03-28T18:10:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:51 smithi045 ceph-mon[31773]: pgmap v731: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:10:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:51 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:10:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 459.257278 secs 2024-03-28T18:10:53.005 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1105 sec 2024-03-28T18:10:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:52 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:10:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:52 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:10:53.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:52 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:10:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:52 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:10:53.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:52 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:10:53.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:52 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:10:53.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:52 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:10:53.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:52 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:10:53.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:52 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:10:54.005 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1106 sec 2024-03-28T18:10:54.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:53 smithi151 ceph-mon[36481]: pgmap v732: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:54.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:53 smithi151 ceph-mon[36481]: pgmap v733: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:10:54.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:53 smithi045 ceph-mon[31773]: pgmap v732: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:54.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:53 smithi045 ceph-mon[31773]: pgmap v733: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:10:54.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:53 smithi071 ceph-mon[36597]: pgmap v732: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:10:54.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:53 smithi071 ceph-mon[36597]: pgmap v733: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:10:55.005 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1107 sec 2024-03-28T18:10:56.005 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1108 sec 2024-03-28T18:10:56.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:55 smithi151 ceph-mon[36481]: pgmap v734: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:10:56.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:55 smithi045 ceph-mon[31773]: pgmap v734: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:10:56.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:55 smithi071 ceph-mon[36597]: pgmap v734: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:10:57.005 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1109 sec 2024-03-28T18:10:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 464.257339 secs 2024-03-28T18:10:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 464.257339 secs 2024-03-28T18:10:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 464.257339 secs 2024-03-28T18:10:58.006 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1110 sec 2024-03-28T18:10:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:57 smithi045 ceph-mon[31773]: pgmap v735: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:10:58.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:57 smithi071 ceph-mon[36597]: pgmap v735: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:10:58.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:57 smithi151 ceph-mon[36481]: pgmap v735: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:10:59.006 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1111 sec 2024-03-28T18:11:00.006 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1112 sec 2024-03-28T18:11:00.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:10:59 smithi045 ceph-mon[31773]: pgmap v736: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:11:00.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:10:59 smithi071 ceph-mon[36597]: pgmap v736: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:11:00.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:10:59 smithi151 ceph-mon[36481]: pgmap v736: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:11:01.006 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1113 sec 2024-03-28T18:11:02.006 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1114 sec 2024-03-28T18:11:02.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:01 smithi151 ceph-mon[36481]: pgmap v737: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:11:02.126 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 469.257428 secs 2024-03-28T18:11:02.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:01 smithi045 ceph-mon[31773]: pgmap v737: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:11:02.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 469.257428 secs 2024-03-28T18:11:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:01 smithi071 ceph-mon[36597]: pgmap v737: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:11:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 469.257428 secs 2024-03-28T18:11:03.007 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1115 sec 2024-03-28T18:11:04.007 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1116 sec 2024-03-28T18:11:04.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:03 smithi045 ceph-mon[31773]: pgmap v738: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:11:04.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:03 smithi071 ceph-mon[36597]: pgmap v738: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:11:04.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:03 smithi151 ceph-mon[36481]: pgmap v738: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:11:05.007 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1117 sec 2024-03-28T18:11:06.007 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1118 sec 2024-03-28T18:11:06.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:05 smithi045 ceph-mon[31773]: pgmap v739: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:06.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:05 smithi071 ceph-mon[36597]: pgmap v739: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:06.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:05 smithi151 ceph-mon[36481]: pgmap v739: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:07.007 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1119 sec 2024-03-28T18:11:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 474.257439 secs 2024-03-28T18:11:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 474.257439 secs 2024-03-28T18:11:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 474.257439 secs 2024-03-28T18:11:08.007 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1120 sec 2024-03-28T18:11:08.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:07 smithi045 ceph-mon[31773]: pgmap v740: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:08.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:07 smithi071 ceph-mon[36597]: pgmap v740: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:08.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:07 smithi151 ceph-mon[36481]: pgmap v740: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:09.007 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1121 sec 2024-03-28T18:11:10.007 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1122 sec 2024-03-28T18:11:10.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:09 smithi045 ceph-mon[31773]: pgmap v741: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:10.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:09 smithi071 ceph-mon[36597]: pgmap v741: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:10.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:09 smithi151 ceph-mon[36481]: pgmap v741: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:11.008 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1123 sec 2024-03-28T18:11:12.008 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1124 sec 2024-03-28T18:11:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:11 smithi045 ceph-mon[31773]: pgmap v742: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:11:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 479.257492 secs 2024-03-28T18:11:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:11 smithi071 ceph-mon[36597]: pgmap v742: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:11:12.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 479.257492 secs 2024-03-28T18:11:12.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:11 smithi151 ceph-mon[36481]: pgmap v742: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:11:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 479.257492 secs 2024-03-28T18:11:13.008 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1125 sec 2024-03-28T18:11:14.008 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1126 sec 2024-03-28T18:11:14.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:13 smithi045 ceph-mon[31773]: pgmap v743: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:11:14.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:13 smithi071 ceph-mon[36597]: pgmap v743: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:11:14.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:13 smithi151 ceph-mon[36481]: pgmap v743: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:11:15.008 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1127 sec 2024-03-28T18:11:16.008 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1128 sec 2024-03-28T18:11:16.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:15 smithi045 ceph-mon[31773]: pgmap v744: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:11:16.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:15 smithi071 ceph-mon[36597]: pgmap v744: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:11:16.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:15 smithi151 ceph-mon[36481]: pgmap v744: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:11:17.008 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1129 sec 2024-03-28T18:11:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 484.257553 secs 2024-03-28T18:11:17.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 484.257553 secs 2024-03-28T18:11:17.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 484.257553 secs 2024-03-28T18:11:18.009 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1130 sec 2024-03-28T18:11:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:17 smithi045 ceph-mon[31773]: pgmap v745: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:11:18.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:17 smithi071 ceph-mon[36597]: pgmap v745: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:11:18.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:17 smithi151 ceph-mon[36481]: pgmap v745: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:11:19.009 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1131 sec 2024-03-28T18:11:20.009 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1132 sec 2024-03-28T18:11:20.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:19 smithi045 ceph-mon[31773]: pgmap v746: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:11:20.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:19 smithi071 ceph-mon[36597]: pgmap v746: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:11:20.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:19 smithi151 ceph-mon[36481]: pgmap v746: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:11:21.009 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1133 sec 2024-03-28T18:11:22.009 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1134 sec 2024-03-28T18:11:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:21 smithi045 ceph-mon[31773]: pgmap v747: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:11:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 489.257637 secs 2024-03-28T18:11:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:21 smithi071 ceph-mon[36597]: pgmap v747: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:11:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 489.257637 secs 2024-03-28T18:11:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:21 smithi151 ceph-mon[36481]: pgmap v747: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:11:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 489.257637 secs 2024-03-28T18:11:23.009 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1135 sec 2024-03-28T18:11:24.009 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1136 sec 2024-03-28T18:11:24.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:23 smithi045 ceph-mon[31773]: pgmap v748: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:24.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:23 smithi071 ceph-mon[36597]: pgmap v748: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:24.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:23 smithi151 ceph-mon[36481]: pgmap v748: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:25.009 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1137 sec 2024-03-28T18:11:26.010 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1138 sec 2024-03-28T18:11:26.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:25 smithi071 ceph-mon[36597]: pgmap v749: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:25 smithi151 ceph-mon[36481]: pgmap v749: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:26.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:25 smithi045 ceph-mon[31773]: pgmap v749: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:27.010 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1139 sec 2024-03-28T18:11:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 494.257725 secs 2024-03-28T18:11:27.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 494.257725 secs 2024-03-28T18:11:27.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 494.257725 secs 2024-03-28T18:11:28.010 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1140 sec 2024-03-28T18:11:28.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:27 smithi071 ceph-mon[36597]: pgmap v750: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:28.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:27 smithi151 ceph-mon[36481]: pgmap v750: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:28.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:27 smithi045 ceph-mon[31773]: pgmap v750: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:29.010 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1141 sec 2024-03-28T18:11:30.010 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1142 sec 2024-03-28T18:11:30.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:29 smithi071 ceph-mon[36597]: pgmap v751: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:30.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:29 smithi151 ceph-mon[36481]: pgmap v751: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:30.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:29 smithi045 ceph-mon[31773]: pgmap v751: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:31.010 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1143 sec 2024-03-28T18:11:31.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:30 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 499.257786 secs 2024-03-28T18:11:31.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:30 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 499.257786 secs 2024-03-28T18:11:31.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:30 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 499.257786 secs 2024-03-28T18:11:32.010 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1144 sec 2024-03-28T18:11:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:31 smithi071 ceph-mon[36597]: pgmap v752: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:31 smithi151 ceph-mon[36481]: pgmap v752: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:32.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:31 smithi045 ceph-mon[31773]: pgmap v752: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:33.011 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1145 sec 2024-03-28T18:11:34.011 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1146 sec 2024-03-28T18:11:34.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:33 smithi071 ceph-mon[36597]: pgmap v753: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:34.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:33 smithi151 ceph-mon[36481]: pgmap v753: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:34.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:33 smithi045 ceph-mon[31773]: pgmap v753: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:35.011 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1147 sec 2024-03-28T18:11:36.011 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1148 sec 2024-03-28T18:11:36.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:36 smithi071 ceph-mon[36597]: pgmap v754: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:36.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 504.257847 secs 2024-03-28T18:11:36.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:36 smithi151 ceph-mon[36481]: pgmap v754: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:36.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 504.257847 secs 2024-03-28T18:11:36.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:36 smithi045 ceph-mon[31773]: pgmap v754: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:36.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 504.257847 secs 2024-03-28T18:11:37.011 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1149 sec 2024-03-28T18:11:38.011 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1150 sec 2024-03-28T18:11:38.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:38 smithi071 ceph-mon[36597]: pgmap v755: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:38.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:38 smithi151 ceph-mon[36481]: pgmap v755: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:38.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:38 smithi045 ceph-mon[31773]: pgmap v755: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:39.011 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1151 sec 2024-03-28T18:11:40.011 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1152 sec 2024-03-28T18:11:40.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:40 smithi071 ceph-mon[36597]: pgmap v756: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:40.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:40 smithi151 ceph-mon[36481]: pgmap v756: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:40.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:40 smithi045 ceph-mon[31773]: pgmap v756: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:41.012 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1153 sec 2024-03-28T18:11:41.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 509.257908 secs 2024-03-28T18:11:41.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 509.257908 secs 2024-03-28T18:11:41.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 509.257908 secs 2024-03-28T18:11:42.012 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1154 sec 2024-03-28T18:11:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:42 smithi071 ceph-mon[36597]: pgmap v757: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:42 smithi151 ceph-mon[36481]: pgmap v757: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:42 smithi045 ceph-mon[31773]: pgmap v757: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:43.012 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1155 sec 2024-03-28T18:11:44.012 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1156 sec 2024-03-28T18:11:44.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:44 smithi071 ceph-mon[36597]: pgmap v758: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:44.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:44 smithi151 ceph-mon[36481]: pgmap v758: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:44.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:44 smithi045 ceph-mon[31773]: pgmap v758: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:45.012 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1157 sec 2024-03-28T18:11:46.012 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1158 sec 2024-03-28T18:11:46.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:46 smithi071 ceph-mon[36597]: pgmap v759: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:46.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 514.257970 secs 2024-03-28T18:11:46.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:46 smithi151 ceph-mon[36481]: pgmap v759: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:46.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 514.257970 secs 2024-03-28T18:11:46.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:46 smithi045 ceph-mon[31773]: pgmap v759: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:46.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 514.257970 secs 2024-03-28T18:11:47.012 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1159 sec 2024-03-28T18:11:48.013 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1160 sec 2024-03-28T18:11:48.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:48 smithi071 ceph-mon[36597]: pgmap v760: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:48 smithi151 ceph-mon[36481]: pgmap v760: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:48.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:48 smithi045 ceph-mon[31773]: pgmap v760: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:49.013 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1161 sec 2024-03-28T18:11:49.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:49 smithi045 ceph-mon[31773]: pgmap v761: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:49.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:49 smithi071 ceph-mon[36597]: pgmap v761: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:49.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:49 smithi151 ceph-mon[36481]: pgmap v761: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:50.013 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1162 sec 2024-03-28T18:11:51.013 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1163 sec 2024-03-28T18:11:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:51 smithi071 ceph-mon[36597]: pgmap v762: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 519.258008 secs 2024-03-28T18:11:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:51 smithi151 ceph-mon[36481]: pgmap v762: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 519.258008 secs 2024-03-28T18:11:51.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:51 smithi045 ceph-mon[31773]: pgmap v762: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:11:51.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 519.258008 secs 2024-03-28T18:11:52.013 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1164 sec 2024-03-28T18:11:53.013 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1165 sec 2024-03-28T18:11:53.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:53 smithi071 ceph-mon[36597]: pgmap v763: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:53.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:53 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:11:53.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:53 smithi151 ceph-mon[36481]: pgmap v763: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:53.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:53 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:11:53.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:53 smithi045 ceph-mon[31773]: pgmap v763: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:53.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:53 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:11:54.013 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1166 sec 2024-03-28T18:11:54.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:54 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:11:54.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:54 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:11:54.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:54 smithi071 ceph-mon[36597]: pgmap v764: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:54.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:54 smithi071 ceph-mon[36597]: pgmap v765: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:11:54.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:54 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:11:54.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:54 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:11:54.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:54 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:11:54.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:54 smithi151 ceph-mon[36481]: pgmap v764: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:54.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:54 smithi151 ceph-mon[36481]: pgmap v765: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:11:54.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:54 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:11:54.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:54 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:11:54.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:54 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:11:54.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:54 smithi045 ceph-mon[31773]: pgmap v764: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:11:54.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:54 smithi045 ceph-mon[31773]: pgmap v765: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:11:54.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:54 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:11:55.014 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1167 sec 2024-03-28T18:11:56.014 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1168 sec 2024-03-28T18:11:57.014 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1169 sec 2024-03-28T18:11:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:56 smithi045 ceph-mon[31773]: pgmap v766: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:11:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 524.258093 secs 2024-03-28T18:11:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:56 smithi071 ceph-mon[36597]: pgmap v766: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:11:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 524.258093 secs 2024-03-28T18:11:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:56 smithi151 ceph-mon[36481]: pgmap v766: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:11:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 524.258093 secs 2024-03-28T18:11:58.014 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1170 sec 2024-03-28T18:11:59.014 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1171 sec 2024-03-28T18:11:59.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:11:58 smithi045 ceph-mon[31773]: pgmap v767: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:11:59.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:11:58 smithi071 ceph-mon[36597]: pgmap v767: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:11:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:11:58 smithi151 ceph-mon[36481]: pgmap v767: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:12:00.014 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1172 sec 2024-03-28T18:12:01.014 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1173 sec 2024-03-28T18:12:01.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:00 smithi045 ceph-mon[31773]: pgmap v768: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:12:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:00 smithi071 ceph-mon[36597]: pgmap v768: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:12:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:00 smithi151 ceph-mon[36481]: pgmap v768: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:12:02.015 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1174 sec 2024-03-28T18:12:02.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 529.258181 secs 2024-03-28T18:12:02.224 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 529.258181 secs 2024-03-28T18:12:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 529.258181 secs 2024-03-28T18:12:03.015 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1175 sec 2024-03-28T18:12:03.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:02 smithi045 ceph-mon[31773]: pgmap v769: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:12:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:02 smithi071 ceph-mon[36597]: pgmap v769: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:12:03.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:02 smithi151 ceph-mon[36481]: pgmap v769: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:12:04.015 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1176 sec 2024-03-28T18:12:05.015 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1177 sec 2024-03-28T18:12:05.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:04 smithi045 ceph-mon[31773]: pgmap v770: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:12:05.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:04 smithi071 ceph-mon[36597]: pgmap v770: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:12:05.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:04 smithi151 ceph-mon[36481]: pgmap v770: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:12:06.015 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1178 sec 2024-03-28T18:12:07.015 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1179 sec 2024-03-28T18:12:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:06 smithi045 ceph-mon[31773]: pgmap v771: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 534.258243 secs 2024-03-28T18:12:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:06 smithi071 ceph-mon[36597]: pgmap v771: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 534.258243 secs 2024-03-28T18:12:07.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:06 smithi151 ceph-mon[36481]: pgmap v771: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 534.258243 secs 2024-03-28T18:12:08.015 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1180 sec 2024-03-28T18:12:09.015 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1181 sec 2024-03-28T18:12:09.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:08 smithi071 ceph-mon[36597]: pgmap v772: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:09.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:08 smithi151 ceph-mon[36481]: pgmap v772: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:09.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:08 smithi045 ceph-mon[31773]: pgmap v772: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:10.016 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1182 sec 2024-03-28T18:12:11.016 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1183 sec 2024-03-28T18:12:11.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:10 smithi071 ceph-mon[36597]: pgmap v773: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:11.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:10 smithi151 ceph-mon[36481]: pgmap v773: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:11.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:10 smithi045 ceph-mon[31773]: pgmap v773: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:12.016 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1184 sec 2024-03-28T18:12:12.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 539.258331 secs 2024-03-28T18:12:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 539.258331 secs 2024-03-28T18:12:12.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 539.258331 secs 2024-03-28T18:12:13.016 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1185 sec 2024-03-28T18:12:13.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:12 smithi071 ceph-mon[36597]: pgmap v774: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:13.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:12 smithi151 ceph-mon[36481]: pgmap v774: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:13.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:12 smithi045 ceph-mon[31773]: pgmap v774: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:14.016 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1186 sec 2024-03-28T18:12:15.016 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1187 sec 2024-03-28T18:12:15.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:14 smithi071 ceph-mon[36597]: pgmap v775: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:15.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:14 smithi151 ceph-mon[36481]: pgmap v775: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:15.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:14 smithi045 ceph-mon[31773]: pgmap v775: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:16.016 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1188 sec 2024-03-28T18:12:16.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:15 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 544.258418 secs 2024-03-28T18:12:16.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:15 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 544.258418 secs 2024-03-28T18:12:16.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:15 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 544.258418 secs 2024-03-28T18:12:17.016 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1189 sec 2024-03-28T18:12:17.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:16 smithi071 ceph-mon[36597]: pgmap v776: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:16 smithi151 ceph-mon[36481]: pgmap v776: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:16 smithi045 ceph-mon[31773]: pgmap v776: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:18.017 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1190 sec 2024-03-28T18:12:19.017 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1191 sec 2024-03-28T18:12:19.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:18 smithi071 ceph-mon[36597]: pgmap v777: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:18 smithi151 ceph-mon[36481]: pgmap v777: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:19.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:18 smithi045 ceph-mon[31773]: pgmap v777: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:20.017 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1192 sec 2024-03-28T18:12:21.017 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1193 sec 2024-03-28T18:12:21.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:21 smithi071 ceph-mon[36597]: pgmap v778: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:21.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 549.258461 secs 2024-03-28T18:12:21.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:21 smithi151 ceph-mon[36481]: pgmap v778: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:21.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 549.258461 secs 2024-03-28T18:12:21.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:21 smithi045 ceph-mon[31773]: pgmap v778: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:21.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 549.258461 secs 2024-03-28T18:12:22.017 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1194 sec 2024-03-28T18:12:23.018 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1195 sec 2024-03-28T18:12:23.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:23 smithi071 ceph-mon[36597]: pgmap v779: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:23.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:23 smithi151 ceph-mon[36481]: pgmap v779: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:23.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:23 smithi045 ceph-mon[31773]: pgmap v779: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:24.018 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1196 sec 2024-03-28T18:12:25.018 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1197 sec 2024-03-28T18:12:25.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:25 smithi071 ceph-mon[36597]: pgmap v780: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:25.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:25 smithi151 ceph-mon[36481]: pgmap v780: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:25.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:25 smithi045 ceph-mon[31773]: pgmap v780: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:26.018 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1198 sec 2024-03-28T18:12:26.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 554.258495 secs 2024-03-28T18:12:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 554.258495 secs 2024-03-28T18:12:26.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 554.258495 secs 2024-03-28T18:12:27.018 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1199 sec 2024-03-28T18:12:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:27 smithi071 ceph-mon[36597]: pgmap v781: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:27.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:27 smithi151 ceph-mon[36481]: pgmap v781: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:27.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:27 smithi045 ceph-mon[31773]: pgmap v781: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:28.018 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1200 sec 2024-03-28T18:12:29.018 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1201 sec 2024-03-28T18:12:29.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:29 smithi071 ceph-mon[36597]: pgmap v782: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:29.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:29 smithi151 ceph-mon[36481]: pgmap v782: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:29.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:29 smithi045 ceph-mon[31773]: pgmap v782: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:30.018 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1202 sec 2024-03-28T18:12:31.018 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1203 sec 2024-03-28T18:12:31.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:31 smithi071 ceph-mon[36597]: pgmap v783: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:31.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 559.258581 secs 2024-03-28T18:12:31.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:31 smithi151 ceph-mon[36481]: pgmap v783: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:31.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 559.258581 secs 2024-03-28T18:12:31.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:31 smithi045 ceph-mon[31773]: pgmap v783: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:31.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 559.258581 secs 2024-03-28T18:12:32.018 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1204 sec 2024-03-28T18:12:33.019 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1205 sec 2024-03-28T18:12:33.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:33 smithi071 ceph-mon[36597]: pgmap v784: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:33.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:33 smithi151 ceph-mon[36481]: pgmap v784: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:33.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:33 smithi045 ceph-mon[31773]: pgmap v784: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:34.019 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1206 sec 2024-03-28T18:12:35.019 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1207 sec 2024-03-28T18:12:35.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:35 smithi071 ceph-mon[36597]: pgmap v785: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:35.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:35 smithi151 ceph-mon[36481]: pgmap v785: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:35.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:35 smithi045 ceph-mon[31773]: pgmap v785: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:36.019 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1208 sec 2024-03-28T18:12:36.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 564.258643 secs 2024-03-28T18:12:36.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 564.258643 secs 2024-03-28T18:12:36.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 564.258643 secs 2024-03-28T18:12:37.019 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1209 sec 2024-03-28T18:12:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:37 smithi071 ceph-mon[36597]: pgmap v786: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:37 smithi151 ceph-mon[36481]: pgmap v786: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:37 smithi045 ceph-mon[31773]: pgmap v786: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:38.019 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1210 sec 2024-03-28T18:12:38.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:38 smithi151 ceph-mon[36481]: pgmap v787: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:38.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:38 smithi045 ceph-mon[31773]: pgmap v787: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:38.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:38 smithi071 ceph-mon[36597]: pgmap v787: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:39.020 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1211 sec 2024-03-28T18:12:40.020 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1212 sec 2024-03-28T18:12:41.020 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1213 sec 2024-03-28T18:12:41.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:40 smithi045 ceph-mon[31773]: pgmap v788: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:41.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:40 smithi071 ceph-mon[36597]: pgmap v788: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:41.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:40 smithi151 ceph-mon[36481]: pgmap v788: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:42.020 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1214 sec 2024-03-28T18:12:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 569.258655 secs 2024-03-28T18:12:42.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 569.258655 secs 2024-03-28T18:12:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 569.258655 secs 2024-03-28T18:12:43.020 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1215 sec 2024-03-28T18:12:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:42 smithi045 ceph-mon[31773]: pgmap v789: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:43.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:42 smithi071 ceph-mon[36597]: pgmap v789: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:43.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:42 smithi151 ceph-mon[36481]: pgmap v789: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:44.020 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1216 sec 2024-03-28T18:12:45.020 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1217 sec 2024-03-28T18:12:45.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:44 smithi045 ceph-mon[31773]: pgmap v790: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:45.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:44 smithi071 ceph-mon[36597]: pgmap v790: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:45.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:44 smithi151 ceph-mon[36481]: pgmap v790: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:46.020 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1218 sec 2024-03-28T18:12:47.021 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1219 sec 2024-03-28T18:12:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:46 smithi045 ceph-mon[31773]: pgmap v791: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 574.258760 secs 2024-03-28T18:12:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:46 smithi071 ceph-mon[36597]: pgmap v791: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 574.258760 secs 2024-03-28T18:12:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:46 smithi151 ceph-mon[36481]: pgmap v791: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 574.258760 secs 2024-03-28T18:12:48.021 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1220 sec 2024-03-28T18:12:49.021 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1221 sec 2024-03-28T18:12:49.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:48 smithi045 ceph-mon[31773]: pgmap v792: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:49.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:48 smithi071 ceph-mon[36597]: pgmap v792: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:49.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:48 smithi151 ceph-mon[36481]: pgmap v792: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:50.021 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1222 sec 2024-03-28T18:12:51.021 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1223 sec 2024-03-28T18:12:51.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:50 smithi045 ceph-mon[31773]: pgmap v793: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:51.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:50 smithi071 ceph-mon[36597]: pgmap v793: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:51.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:50 smithi151 ceph-mon[36481]: pgmap v793: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:52.021 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1224 sec 2024-03-28T18:12:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 579.258848 secs 2024-03-28T18:12:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 579.258848 secs 2024-03-28T18:12:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 579.258848 secs 2024-03-28T18:12:53.021 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1225 sec 2024-03-28T18:12:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:52 smithi045 ceph-mon[31773]: pgmap v794: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:53.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:52 smithi071 ceph-mon[36597]: pgmap v794: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:53.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:52 smithi151 ceph-mon[36481]: pgmap v794: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:12:54.021 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1226 sec 2024-03-28T18:12:55.022 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1227 sec 2024-03-28T18:12:55.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:54 smithi045 ceph-mon[31773]: pgmap v795: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:55.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:54 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:12:55.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:54 smithi071 ceph-mon[36597]: pgmap v795: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:55.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:54 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:12:55.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:54 smithi151 ceph-mon[36481]: pgmap v795: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:12:55.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:54 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:12:56.022 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1228 sec 2024-03-28T18:12:56.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:55 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:12:56.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:55 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:12:56.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:55 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:12:56.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:55 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:12:56.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:55 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:12:56.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:55 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:12:56.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:55 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:12:56.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:55 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:12:56.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:55 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:12:57.022 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1229 sec 2024-03-28T18:12:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:56 smithi071 ceph-mon[36597]: pgmap v796: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T18:12:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:56 smithi071 ceph-mon[36597]: pgmap v797: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:12:57.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 584.258933 secs 2024-03-28T18:12:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:56 smithi151 ceph-mon[36481]: pgmap v796: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T18:12:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:56 smithi151 ceph-mon[36481]: pgmap v797: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:12:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 584.258933 secs 2024-03-28T18:12:57.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:56 smithi045 ceph-mon[31773]: pgmap v796: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T18:12:57.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:56 smithi045 ceph-mon[31773]: pgmap v797: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:12:57.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 584.258933 secs 2024-03-28T18:12:58.022 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1230 sec 2024-03-28T18:12:59.022 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1231 sec 2024-03-28T18:12:59.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:12:58 smithi071 ceph-mon[36597]: pgmap v798: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:12:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:12:58 smithi151 ceph-mon[36481]: pgmap v798: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:12:59.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:12:58 smithi045 ceph-mon[31773]: pgmap v798: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:13:00.022 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1232 sec 2024-03-28T18:13:01.022 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1233 sec 2024-03-28T18:13:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:00 smithi071 ceph-mon[36597]: pgmap v799: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:13:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:00 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 589.258993 secs 2024-03-28T18:13:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:00 smithi151 ceph-mon[36481]: pgmap v799: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:13:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:00 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 589.258993 secs 2024-03-28T18:13:01.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:00 smithi045 ceph-mon[31773]: pgmap v799: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:13:01.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:00 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 589.258993 secs 2024-03-28T18:13:02.022 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1234 sec 2024-03-28T18:13:03.023 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1235 sec 2024-03-28T18:13:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:02 smithi071 ceph-mon[36597]: pgmap v800: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:13:03.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:02 smithi151 ceph-mon[36481]: pgmap v800: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:13:03.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:02 smithi045 ceph-mon[31773]: pgmap v800: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:13:04.023 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1236 sec 2024-03-28T18:13:05.023 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1237 sec 2024-03-28T18:13:05.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:04 smithi071 ceph-mon[36597]: pgmap v801: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:13:05.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:04 smithi151 ceph-mon[36481]: pgmap v801: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:13:05.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:04 smithi045 ceph-mon[31773]: pgmap v801: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-28T18:13:06.023 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1238 sec 2024-03-28T18:13:06.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:05 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 594.259082 secs 2024-03-28T18:13:06.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:05 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 594.259082 secs 2024-03-28T18:13:06.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:05 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 594.259082 secs 2024-03-28T18:13:07.023 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1239 sec 2024-03-28T18:13:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:06 smithi071 ceph-mon[36597]: pgmap v802: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:13:07.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:06 smithi151 ceph-mon[36481]: pgmap v802: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:13:07.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:06 smithi045 ceph-mon[31773]: pgmap v802: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:13:08.023 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1240 sec 2024-03-28T18:13:09.024 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1241 sec 2024-03-28T18:13:09.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:08 smithi071 ceph-mon[36597]: pgmap v803: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:09.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:08 smithi151 ceph-mon[36481]: pgmap v803: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:09.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:08 smithi045 ceph-mon[31773]: pgmap v803: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:10.024 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1242 sec 2024-03-28T18:13:11.024 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1243 sec 2024-03-28T18:13:11.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:10 smithi071 ceph-mon[36597]: pgmap v804: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:11.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:10 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 599.259181 secs 2024-03-28T18:13:11.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:10 smithi151 ceph-mon[36481]: pgmap v804: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:11.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:10 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 599.259181 secs 2024-03-28T18:13:11.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:10 smithi045 ceph-mon[31773]: pgmap v804: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:11.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:10 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 599.259181 secs 2024-03-28T18:13:12.024 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1244 sec 2024-03-28T18:13:13.024 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1245 sec 2024-03-28T18:13:13.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:12 smithi071 ceph-mon[36597]: pgmap v805: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:13.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:12 smithi151 ceph-mon[36481]: pgmap v805: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:13.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:12 smithi045 ceph-mon[31773]: pgmap v805: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:14.024 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1246 sec 2024-03-28T18:13:15.024 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1247 sec 2024-03-28T18:13:15.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:14 smithi071 ceph-mon[36597]: pgmap v806: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:15.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:14 smithi151 ceph-mon[36481]: pgmap v806: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:15.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:14 smithi045 ceph-mon[31773]: pgmap v806: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:16.025 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1248 sec 2024-03-28T18:13:16.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 604.259270 secs 2024-03-28T18:13:16.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 604.259270 secs 2024-03-28T18:13:16.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 604.259270 secs 2024-03-28T18:13:17.025 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1249 sec 2024-03-28T18:13:17.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:17 smithi071 ceph-mon[36597]: pgmap v807: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:17 smithi151 ceph-mon[36481]: pgmap v807: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:17 smithi045 ceph-mon[31773]: pgmap v807: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:18.025 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1250 sec 2024-03-28T18:13:19.025 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1251 sec 2024-03-28T18:13:19.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:19 smithi071 ceph-mon[36597]: pgmap v808: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:19.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:19 smithi151 ceph-mon[36481]: pgmap v808: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:19.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:19 smithi045 ceph-mon[31773]: pgmap v808: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:20.025 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1252 sec 2024-03-28T18:13:21.025 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1253 sec 2024-03-28T18:13:21.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:21 smithi071 ceph-mon[36597]: pgmap v809: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:21.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 609.259363 secs 2024-03-28T18:13:21.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:21 smithi151 ceph-mon[36481]: pgmap v809: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:21.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 609.259363 secs 2024-03-28T18:13:21.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:21 smithi045 ceph-mon[31773]: pgmap v809: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:21.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 609.259363 secs 2024-03-28T18:13:22.025 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1254 sec 2024-03-28T18:13:23.026 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1255 sec 2024-03-28T18:13:23.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:23 smithi071 ceph-mon[36597]: pgmap v810: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:23.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:23 smithi151 ceph-mon[36481]: pgmap v810: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:23.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:23 smithi045 ceph-mon[31773]: pgmap v810: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:24.025 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1256 sec 2024-03-28T18:13:25.026 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1257 sec 2024-03-28T18:13:25.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:25 smithi071 ceph-mon[36597]: pgmap v811: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:25.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:25 smithi151 ceph-mon[36481]: pgmap v811: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:25.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:25 smithi045 ceph-mon[31773]: pgmap v811: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:26.026 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1258 sec 2024-03-28T18:13:26.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 614.259424 secs 2024-03-28T18:13:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 614.259424 secs 2024-03-28T18:13:26.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 614.259424 secs 2024-03-28T18:13:27.026 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1259 sec 2024-03-28T18:13:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:27 smithi071 ceph-mon[36597]: pgmap v812: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:27 smithi151 ceph-mon[36481]: pgmap v812: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:27.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:27 smithi045 ceph-mon[31773]: pgmap v812: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:28.026 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1260 sec 2024-03-28T18:13:29.026 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1261 sec 2024-03-28T18:13:29.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:29 smithi071 ceph-mon[36597]: pgmap v813: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:29.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:29 smithi151 ceph-mon[36481]: pgmap v813: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:29.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:29 smithi045 ceph-mon[31773]: pgmap v813: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:30.026 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1262 sec 2024-03-28T18:13:31.026 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1263 sec 2024-03-28T18:13:31.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:31 smithi071 ceph-mon[36597]: pgmap v814: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:31.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 619.259512 secs 2024-03-28T18:13:31.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:31 smithi151 ceph-mon[36481]: pgmap v814: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:31.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 619.259512 secs 2024-03-28T18:13:31.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:31 smithi045 ceph-mon[31773]: pgmap v814: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:31.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 619.259512 secs 2024-03-28T18:13:32.027 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1264 sec 2024-03-28T18:13:33.027 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1265 sec 2024-03-28T18:13:33.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:33 smithi071 ceph-mon[36597]: pgmap v815: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:33.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:33 smithi151 ceph-mon[36481]: pgmap v815: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:33.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:33 smithi045 ceph-mon[31773]: pgmap v815: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:34.027 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1266 sec 2024-03-28T18:13:35.027 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1267 sec 2024-03-28T18:13:35.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:35 smithi071 ceph-mon[36597]: pgmap v816: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:35.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:35 smithi151 ceph-mon[36481]: pgmap v816: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:35.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:35 smithi045 ceph-mon[31773]: pgmap v816: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:36.027 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1268 sec 2024-03-28T18:13:36.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 624.259576 secs 2024-03-28T18:13:36.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 624.259576 secs 2024-03-28T18:13:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 624.259576 secs 2024-03-28T18:13:37.027 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1269 sec 2024-03-28T18:13:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:37 smithi151 ceph-mon[36481]: pgmap v817: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:37 smithi045 ceph-mon[31773]: pgmap v817: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:37 smithi071 ceph-mon[36597]: pgmap v817: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:38.028 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1270 sec 2024-03-28T18:13:38.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:38 smithi045 ceph-mon[31773]: pgmap v818: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:38.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:38 smithi071 ceph-mon[36597]: pgmap v818: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:38.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:38 smithi151 ceph-mon[36481]: pgmap v818: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:39.028 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1271 sec 2024-03-28T18:13:40.028 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1272 sec 2024-03-28T18:13:40.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:40 smithi071 ceph-mon[36597]: pgmap v819: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:40 smithi151 ceph-mon[36481]: pgmap v819: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:40.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:40 smithi045 ceph-mon[31773]: pgmap v819: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:41.028 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1273 sec 2024-03-28T18:13:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 629.259639 secs 2024-03-28T18:13:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 629.259639 secs 2024-03-28T18:13:41.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 629.259639 secs 2024-03-28T18:13:42.028 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1274 sec 2024-03-28T18:13:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:42 smithi071 ceph-mon[36597]: pgmap v820: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:42.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:42 smithi151 ceph-mon[36481]: pgmap v820: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:42.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:42 smithi045 ceph-mon[31773]: pgmap v820: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:43.028 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1275 sec 2024-03-28T18:13:44.028 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1276 sec 2024-03-28T18:13:44.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:44 smithi071 ceph-mon[36597]: pgmap v821: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:44.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:44 smithi151 ceph-mon[36481]: pgmap v821: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:44.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:44 smithi045 ceph-mon[31773]: pgmap v821: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:45.029 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1277 sec 2024-03-28T18:13:46.029 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1278 sec 2024-03-28T18:13:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:46 smithi151 ceph-mon[36481]: pgmap v822: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 634.259677 secs 2024-03-28T18:13:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:46 smithi045 ceph-mon[31773]: pgmap v822: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:46.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 634.259677 secs 2024-03-28T18:13:47.029 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1279 sec 2024-03-28T18:13:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:46 smithi071 ceph-mon[36597]: pgmap v822: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 634.259677 secs 2024-03-28T18:13:48.029 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1280 sec 2024-03-28T18:13:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:48 smithi151 ceph-mon[36481]: pgmap v823: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:48.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:48 smithi045 ceph-mon[31773]: pgmap v823: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:49.029 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1281 sec 2024-03-28T18:13:49.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:48 smithi071 ceph-mon[36597]: pgmap v823: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:50.029 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1282 sec 2024-03-28T18:13:50.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:50 smithi045 ceph-mon[31773]: pgmap v824: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:51.029 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1283 sec 2024-03-28T18:13:51.094 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:50 smithi151 ceph-mon[36481]: pgmap v824: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:51.094 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:50 smithi071 ceph-mon[36597]: pgmap v824: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:51.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 639.259764 secs 2024-03-28T18:13:52.030 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1284 sec 2024-03-28T18:13:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 639.259764 secs 2024-03-28T18:13:52.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 639.259764 secs 2024-03-28T18:13:52.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:52 smithi045 ceph-mon[31773]: pgmap v825: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:53.030 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1285 sec 2024-03-28T18:13:53.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:52 smithi071 ceph-mon[36597]: pgmap v825: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:52 smithi151 ceph-mon[36481]: pgmap v825: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:54.030 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1286 sec 2024-03-28T18:13:54.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:54 smithi045 ceph-mon[31773]: pgmap v826: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:55.030 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1287 sec 2024-03-28T18:13:55.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:54 smithi071 ceph-mon[36597]: pgmap v826: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:55.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:54 smithi151 ceph-mon[36481]: pgmap v826: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:56.030 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1288 sec 2024-03-28T18:13:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:56 smithi045 ceph-mon[31773]: pgmap v827: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:56.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:56 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:13:56.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 644.259831 secs 2024-03-28T18:13:57.030 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1289 sec 2024-03-28T18:13:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:56 smithi071 ceph-mon[36597]: pgmap v827: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:56 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:13:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 644.259831 secs 2024-03-28T18:13:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:56 smithi151 ceph-mon[36481]: pgmap v827: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:13:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:56 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:13:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 644.259831 secs 2024-03-28T18:13:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:57 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:13:57.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:57 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:13:57.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:57 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:13:58.030 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1290 sec 2024-03-28T18:13:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:57 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:13:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:57 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:13:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:57 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:13:58.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:57 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:13:58.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:57 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:13:58.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:57 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:13:58.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:58 smithi045 ceph-mon[31773]: pgmap v828: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:58.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:13:58 smithi045 ceph-mon[31773]: pgmap v829: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:13:59.030 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1291 sec 2024-03-28T18:13:59.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:58 smithi071 ceph-mon[36597]: pgmap v828: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:59.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:13:58 smithi071 ceph-mon[36597]: pgmap v829: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:13:59.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:58 smithi151 ceph-mon[36481]: pgmap v828: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:13:59.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:13:58 smithi151 ceph-mon[36481]: pgmap v829: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:14:00.031 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1292 sec 2024-03-28T18:14:00.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:00 smithi045 ceph-mon[31773]: pgmap v830: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:14:01.030 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1293 sec 2024-03-28T18:14:01.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:00 smithi071 ceph-mon[36597]: pgmap v830: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:14:01.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:00 smithi151 ceph-mon[36481]: pgmap v830: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:14:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 649.259921 secs 2024-03-28T18:14:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 649.259921 secs 2024-03-28T18:14:02.031 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1294 sec 2024-03-28T18:14:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 649.259921 secs 2024-03-28T18:14:02.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:02 smithi045 ceph-mon[31773]: pgmap v831: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:14:03.031 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1295 sec 2024-03-28T18:14:03.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:02 smithi071 ceph-mon[36597]: pgmap v831: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:14:03.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:02 smithi151 ceph-mon[36481]: pgmap v831: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:14:04.031 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1296 sec 2024-03-28T18:14:04.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:04 smithi045 ceph-mon[31773]: pgmap v832: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:14:05.031 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1297 sec 2024-03-28T18:14:05.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:04 smithi071 ceph-mon[36597]: pgmap v832: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:14:05.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:04 smithi151 ceph-mon[36481]: pgmap v832: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:14:06.031 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1298 sec 2024-03-28T18:14:06.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:06 smithi045 ceph-mon[31773]: pgmap v833: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:14:06.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 654.260009 secs 2024-03-28T18:14:07.031 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1299 sec 2024-03-28T18:14:07.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:06 smithi071 ceph-mon[36597]: pgmap v833: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:14:07.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 654.260009 secs 2024-03-28T18:14:07.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:06 smithi151 ceph-mon[36481]: pgmap v833: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:14:07.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 654.260009 secs 2024-03-28T18:14:08.032 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1300 sec 2024-03-28T18:14:08.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:08 smithi045 ceph-mon[31773]: pgmap v834: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:14:09.032 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1301 sec 2024-03-28T18:14:09.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:08 smithi071 ceph-mon[36597]: pgmap v834: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:14:09.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:08 smithi151 ceph-mon[36481]: pgmap v834: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:14:10.032 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1302 sec 2024-03-28T18:14:10.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:10 smithi045 ceph-mon[31773]: pgmap v835: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:11.032 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1303 sec 2024-03-28T18:14:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:10 smithi071 ceph-mon[36597]: pgmap v835: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:11.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:10 smithi151 ceph-mon[36481]: pgmap v835: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 659.260079 secs 2024-03-28T18:14:12.032 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1304 sec 2024-03-28T18:14:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 659.260079 secs 2024-03-28T18:14:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 659.260079 secs 2024-03-28T18:14:12.919 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:12 smithi045 ceph-mon[31773]: pgmap v836: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:14:13.032 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1305 sec 2024-03-28T18:14:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:12 smithi071 ceph-mon[36597]: pgmap v836: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:14:13.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:12 smithi151 ceph-mon[36481]: pgmap v836: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:14:14.033 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1306 sec 2024-03-28T18:14:14.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:14 smithi045 ceph-mon[31773]: pgmap v837: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:15.033 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1307 sec 2024-03-28T18:14:15.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:14 smithi071 ceph-mon[36597]: pgmap v837: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:15.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:14 smithi151 ceph-mon[36481]: pgmap v837: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:16.033 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1308 sec 2024-03-28T18:14:17.033 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1309 sec 2024-03-28T18:14:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:16 smithi071 ceph-mon[36597]: pgmap v838: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:14:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 664.260167 secs 2024-03-28T18:14:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:16 smithi151 ceph-mon[36481]: pgmap v838: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:14:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 664.260167 secs 2024-03-28T18:14:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:16 smithi045 ceph-mon[31773]: pgmap v838: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:14:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 664.260167 secs 2024-03-28T18:14:18.033 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1310 sec 2024-03-28T18:14:19.033 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1311 sec 2024-03-28T18:14:19.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:18 smithi071 ceph-mon[36597]: pgmap v839: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:19.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:18 smithi151 ceph-mon[36481]: pgmap v839: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:19.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:18 smithi045 ceph-mon[31773]: pgmap v839: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:20.034 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1312 sec 2024-03-28T18:14:21.033 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1313 sec 2024-03-28T18:14:21.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:20 smithi071 ceph-mon[36597]: pgmap v840: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:21.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:20 smithi151 ceph-mon[36481]: pgmap v840: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:21.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:20 smithi045 ceph-mon[31773]: pgmap v840: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:22.034 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1314 sec 2024-03-28T18:14:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 669.260268 secs 2024-03-28T18:14:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 669.260268 secs 2024-03-28T18:14:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 669.260268 secs 2024-03-28T18:14:23.034 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1315 sec 2024-03-28T18:14:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:22 smithi071 ceph-mon[36597]: pgmap v841: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:14:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:22 smithi151 ceph-mon[36481]: pgmap v841: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:14:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:22 smithi045 ceph-mon[31773]: pgmap v841: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:14:24.034 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1316 sec 2024-03-28T18:14:24.970 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:24 smithi045 ceph-mon[31773]: pgmap v842: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:25.034 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1317 sec 2024-03-28T18:14:25.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:24 smithi071 ceph-mon[36597]: pgmap v842: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:25.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:24 smithi151 ceph-mon[36481]: pgmap v842: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:26.034 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1318 sec 2024-03-28T18:14:27.035 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1319 sec 2024-03-28T18:14:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:26 smithi071 ceph-mon[36597]: pgmap v843: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:14:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 674.260353 secs 2024-03-28T18:14:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:26 smithi151 ceph-mon[36481]: pgmap v843: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:14:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 674.260353 secs 2024-03-28T18:14:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:26 smithi045 ceph-mon[31773]: pgmap v843: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:14:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 674.260353 secs 2024-03-28T18:14:28.034 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1320 sec 2024-03-28T18:14:29.035 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1321 sec 2024-03-28T18:14:29.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:28 smithi071 ceph-mon[36597]: pgmap v844: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:29.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:28 smithi151 ceph-mon[36481]: pgmap v844: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:29.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:28 smithi045 ceph-mon[31773]: pgmap v844: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:30.035 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1322 sec 2024-03-28T18:14:31.035 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1323 sec 2024-03-28T18:14:31.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:30 smithi071 ceph-mon[36597]: pgmap v845: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:31.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:30 smithi151 ceph-mon[36481]: pgmap v845: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:31.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:30 smithi045 ceph-mon[31773]: pgmap v845: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:32.035 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1324 sec 2024-03-28T18:14:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 679.260425 secs 2024-03-28T18:14:32.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 679.260425 secs 2024-03-28T18:14:32.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 679.260425 secs 2024-03-28T18:14:33.035 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1325 sec 2024-03-28T18:14:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:32 smithi071 ceph-mon[36597]: pgmap v846: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:14:33.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:32 smithi151 ceph-mon[36481]: pgmap v846: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:14:33.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:32 smithi045 ceph-mon[31773]: pgmap v846: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:14:34.035 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1326 sec 2024-03-28T18:14:35.036 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1327 sec 2024-03-28T18:14:35.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:34 smithi071 ceph-mon[36597]: pgmap v847: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:34 smithi151 ceph-mon[36481]: pgmap v847: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:34 smithi045 ceph-mon[31773]: pgmap v847: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:36.036 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1328 sec 2024-03-28T18:14:37.036 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1329 sec 2024-03-28T18:14:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:36 smithi071 ceph-mon[36597]: pgmap v848: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:14:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 684.260484 secs 2024-03-28T18:14:37.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:36 smithi151 ceph-mon[36481]: pgmap v848: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:14:37.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 684.260484 secs 2024-03-28T18:14:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:36 smithi045 ceph-mon[31773]: pgmap v848: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:14:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 684.260484 secs 2024-03-28T18:14:38.036 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1330 sec 2024-03-28T18:14:39.036 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1331 sec 2024-03-28T18:14:39.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:38 smithi071 ceph-mon[36597]: pgmap v849: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:14:39.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:38 smithi151 ceph-mon[36481]: pgmap v849: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:14:39.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:38 smithi045 ceph-mon[31773]: pgmap v849: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:14:40.036 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1332 sec 2024-03-28T18:14:41.036 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1333 sec 2024-03-28T18:14:41.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:40 smithi071 ceph-mon[36597]: pgmap v850: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:14:41.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:40 smithi045 ceph-mon[31773]: pgmap v850: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:14:41.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:40 smithi151 ceph-mon[36481]: pgmap v850: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:14:42.036 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1334 sec 2024-03-28T18:14:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 689.260568 secs 2024-03-28T18:14:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 689.260568 secs 2024-03-28T18:14:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 689.260568 secs 2024-03-28T18:14:43.036 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1335 sec 2024-03-28T18:14:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:42 smithi071 ceph-mon[36597]: pgmap v851: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:14:43.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:42 smithi151 ceph-mon[36481]: pgmap v851: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:14:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:42 smithi045 ceph-mon[31773]: pgmap v851: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:14:44.037 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1336 sec 2024-03-28T18:14:45.037 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1337 sec 2024-03-28T18:14:45.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:44 smithi071 ceph-mon[36597]: pgmap v852: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:14:45.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:44 smithi151 ceph-mon[36481]: pgmap v852: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:14:45.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:44 smithi045 ceph-mon[31773]: pgmap v852: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:14:46.037 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1338 sec 2024-03-28T18:14:47.037 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1339 sec 2024-03-28T18:14:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:46 smithi071 ceph-mon[36597]: pgmap v853: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:14:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 694.260652 secs 2024-03-28T18:14:47.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:46 smithi151 ceph-mon[36481]: pgmap v853: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:14:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 694.260652 secs 2024-03-28T18:14:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:46 smithi045 ceph-mon[31773]: pgmap v853: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:14:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 694.260652 secs 2024-03-28T18:14:48.037 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1340 sec 2024-03-28T18:14:49.038 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1341 sec 2024-03-28T18:14:49.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:48 smithi071 ceph-mon[36597]: pgmap v854: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:49.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:48 smithi151 ceph-mon[36481]: pgmap v854: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:49.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:48 smithi045 ceph-mon[31773]: pgmap v854: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:50.038 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1342 sec 2024-03-28T18:14:51.038 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1343 sec 2024-03-28T18:14:51.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:50 smithi071 ceph-mon[36597]: pgmap v855: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:51.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:50 smithi151 ceph-mon[36481]: pgmap v855: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:51.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:50 smithi045 ceph-mon[31773]: pgmap v855: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:52.038 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1344 sec 2024-03-28T18:14:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 699.260715 secs 2024-03-28T18:14:52.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 699.260715 secs 2024-03-28T18:14:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 699.260715 secs 2024-03-28T18:14:53.038 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1345 sec 2024-03-28T18:14:53.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:52 smithi071 ceph-mon[36597]: pgmap v856: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:14:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:52 smithi151 ceph-mon[36481]: pgmap v856: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:14:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:52 smithi045 ceph-mon[31773]: pgmap v856: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:14:54.038 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1346 sec 2024-03-28T18:14:55.038 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1347 sec 2024-03-28T18:14:55.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:54 smithi071 ceph-mon[36597]: pgmap v857: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:55.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:54 smithi151 ceph-mon[36481]: pgmap v857: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:55.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:54 smithi045 ceph-mon[31773]: pgmap v857: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:56.038 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1348 sec 2024-03-28T18:14:57.038 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1349 sec 2024-03-28T18:14:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:56 smithi071 ceph-mon[36597]: pgmap v858: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:14:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 704.260800 secs 2024-03-28T18:14:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:56 smithi151 ceph-mon[36481]: pgmap v858: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:14:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 704.260800 secs 2024-03-28T18:14:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:56 smithi045 ceph-mon[31773]: pgmap v858: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:14:57.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 704.260800 secs 2024-03-28T18:14:58.038 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1350 sec 2024-03-28T18:14:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:57 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:14:58.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:57 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:14:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:57 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:14:59.039 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1351 sec 2024-03-28T18:14:59.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:14:58 smithi071 ceph-mon[36597]: pgmap v859: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:59.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:14:58 smithi151 ceph-mon[36481]: pgmap v859: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:14:59.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:14:58 smithi045 ceph-mon[31773]: pgmap v859: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:00.039 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1352 sec 2024-03-28T18:15:01.039 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1353 sec 2024-03-28T18:15:01.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:00 smithi071 ceph-mon[36597]: pgmap v860: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:01.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:00 smithi151 ceph-mon[36481]: pgmap v860: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:01.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:00 smithi045 ceph-mon[31773]: pgmap v860: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:01.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:15:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:15:00.939+0000 7f52f6e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:15:01.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:15:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:15:00.939+0000 7f52f6e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:15:01.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:15:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:15:00.939+0000 7f52f6e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:15:01.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:15:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:15:00.939+0000 7f52f6e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:15:01.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:15:00 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:15:00.939+0000 7f52f6e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:15:02.013 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 709.260863 secs 2024-03-28T18:15:02.039 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1354 sec 2024-03-28T18:15:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 709.260863 secs 2024-03-28T18:15:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 709.260863 secs 2024-03-28T18:15:03.039 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1355 sec 2024-03-28T18:15:03.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:02 smithi045 ceph-mon[31773]: pgmap v861: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:03.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:02 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:15:03.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:02 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:15:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:02 smithi071 ceph-mon[36597]: pgmap v861: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:02 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:15:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:02 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:15:03.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:02 smithi151 ceph-mon[36481]: pgmap v861: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:03.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:02 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:15:03.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:02 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:15:04.039 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1356 sec 2024-03-28T18:15:04.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:04 smithi045 ceph-mon[31773]: pgmap v862: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:05.039 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1357 sec 2024-03-28T18:15:05.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:04 smithi071 ceph-mon[36597]: pgmap v862: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:05.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:04 smithi151 ceph-mon[36481]: pgmap v862: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:06.040 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1358 sec 2024-03-28T18:15:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:06 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:15:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:06 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:15:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:06 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:15:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:06 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:15:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:06 smithi045 ceph-mon[31773]: pgmap v863: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:15:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:06 smithi045 ceph-mon[31773]: pgmap v864: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-28T18:15:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:06 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:15:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 714.260925 secs 2024-03-28T18:15:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:06 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:15:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:06 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:15:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:06 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:15:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:06 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:15:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:06 smithi071 ceph-mon[36597]: pgmap v863: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:15:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:06 smithi071 ceph-mon[36597]: pgmap v864: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-28T18:15:06.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:06 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:15:06.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 714.260925 secs 2024-03-28T18:15:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:06 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:15:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:06 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:15:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:06 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:15:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:06 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:15:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:06 smithi151 ceph-mon[36481]: pgmap v863: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:15:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:06 smithi151 ceph-mon[36481]: pgmap v864: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-28T18:15:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:06 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:15:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 714.260925 secs 2024-03-28T18:15:07.040 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1359 sec 2024-03-28T18:15:08.040 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1360 sec 2024-03-28T18:15:08.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:08 smithi071 ceph-mon[36597]: pgmap v865: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-28T18:15:08.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:08 smithi151 ceph-mon[36481]: pgmap v865: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-28T18:15:08.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:08 smithi045 ceph-mon[31773]: pgmap v865: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-28T18:15:09.040 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1361 sec 2024-03-28T18:15:10.040 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1362 sec 2024-03-28T18:15:10.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:10 smithi071 ceph-mon[36597]: pgmap v866: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-28T18:15:10.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:10 smithi151 ceph-mon[36481]: pgmap v866: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-28T18:15:10.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:10 smithi045 ceph-mon[31773]: pgmap v866: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-28T18:15:11.040 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1363 sec 2024-03-28T18:15:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 719.261011 secs 2024-03-28T18:15:11.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 719.261011 secs 2024-03-28T18:15:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 719.261011 secs 2024-03-28T18:15:12.040 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1364 sec 2024-03-28T18:15:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:12 smithi071 ceph-mon[36597]: pgmap v867: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-28T18:15:12.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:12 smithi151 ceph-mon[36481]: pgmap v867: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-28T18:15:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:12 smithi045 ceph-mon[31773]: pgmap v867: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-28T18:15:13.040 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1365 sec 2024-03-28T18:15:14.041 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1366 sec 2024-03-28T18:15:14.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:14 smithi071 ceph-mon[36597]: pgmap v868: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-28T18:15:14.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:14 smithi151 ceph-mon[36481]: pgmap v868: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-28T18:15:14.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:14 smithi045 ceph-mon[31773]: pgmap v868: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-28T18:15:15.041 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1367 sec 2024-03-28T18:15:16.041 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1368 sec 2024-03-28T18:15:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:16 smithi071 ceph-mon[36597]: pgmap v869: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:15:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:16 smithi071 ceph-mon[36597]: 2 slow requests, 2 included below; oldest blocked for > 724.261100 secs 2024-03-28T18:15:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:16 smithi071 ceph-mon[36597]: slow request 724.261100 seconds old, received at 2024-03-28T18:03:11.560545+0000: peer_request:client.15069:1629 currently acquired locks 2024-03-28T18:15:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:16 smithi071 ceph-mon[36597]: slow request 724.164975 seconds old, received at 2024-03-28T18:03:11.656670+0000: client_request(client.25691:164 rmsnap #0x1//scheduled-2024-03-28-17_57_00_UTC 2024-03-28T18:03:11.656797+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:15:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:16 smithi151 ceph-mon[36481]: pgmap v869: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:15:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:16 smithi151 ceph-mon[36481]: 2 slow requests, 2 included below; oldest blocked for > 724.261100 secs 2024-03-28T18:15:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:16 smithi151 ceph-mon[36481]: slow request 724.261100 seconds old, received at 2024-03-28T18:03:11.560545+0000: peer_request:client.15069:1629 currently acquired locks 2024-03-28T18:15:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:16 smithi151 ceph-mon[36481]: slow request 724.164975 seconds old, received at 2024-03-28T18:03:11.656670+0000: client_request(client.25691:164 rmsnap #0x1//scheduled-2024-03-28-17_57_00_UTC 2024-03-28T18:03:11.656797+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:15:16.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:16 smithi045 ceph-mon[31773]: pgmap v869: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:15:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:16 smithi045 ceph-mon[31773]: 2 slow requests, 2 included below; oldest blocked for > 724.261100 secs 2024-03-28T18:15:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:16 smithi045 ceph-mon[31773]: slow request 724.261100 seconds old, received at 2024-03-28T18:03:11.560545+0000: peer_request:client.15069:1629 currently acquired locks 2024-03-28T18:15:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:16 smithi045 ceph-mon[31773]: slow request 724.164975 seconds old, received at 2024-03-28T18:03:11.656670+0000: client_request(client.25691:164 rmsnap #0x1//scheduled-2024-03-28-17_57_00_UTC 2024-03-28T18:03:11.656797+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:15:17.041 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1369 sec 2024-03-28T18:15:18.041 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1370 sec 2024-03-28T18:15:18.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:18 smithi071 ceph-mon[36597]: pgmap v870: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:18.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:18 smithi151 ceph-mon[36481]: pgmap v870: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:18.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:18 smithi045 ceph-mon[31773]: pgmap v870: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:19.041 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1371 sec 2024-03-28T18:15:20.041 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1372 sec 2024-03-28T18:15:20.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:20 smithi071 ceph-mon[36597]: pgmap v871: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:20.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:20 smithi151 ceph-mon[36481]: pgmap v871: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:20.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:20 smithi045 ceph-mon[31773]: pgmap v871: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:21.042 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1373 sec 2024-03-28T18:15:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 729.261142 secs 2024-03-28T18:15:21.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 729.261142 secs 2024-03-28T18:15:21.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 729.261142 secs 2024-03-28T18:15:22.042 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1374 sec 2024-03-28T18:15:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:22 smithi071 ceph-mon[36597]: pgmap v872: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:22 smithi151 ceph-mon[36481]: pgmap v872: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:22.659 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:22 smithi045 ceph-mon[31773]: pgmap v872: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:23.042 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1375 sec 2024-03-28T18:15:24.043 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1376 sec 2024-03-28T18:15:24.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:24 smithi071 ceph-mon[36597]: pgmap v873: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:24.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:24 smithi045 ceph-mon[31773]: pgmap v873: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:24.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:24 smithi151 ceph-mon[36481]: pgmap v873: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:25.042 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1377 sec 2024-03-28T18:15:26.042 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1378 sec 2024-03-28T18:15:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:26 smithi071 ceph-mon[36597]: pgmap v874: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 734.261230 secs 2024-03-28T18:15:26.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:26 smithi151 ceph-mon[36481]: pgmap v874: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:26.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 734.261230 secs 2024-03-28T18:15:26.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:26 smithi045 ceph-mon[31773]: pgmap v874: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:26.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 734.261230 secs 2024-03-28T18:15:27.042 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1379 sec 2024-03-28T18:15:28.043 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1380 sec 2024-03-28T18:15:28.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:28 smithi071 ceph-mon[36597]: pgmap v875: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:28.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:28 smithi151 ceph-mon[36481]: pgmap v875: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:28.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:28 smithi045 ceph-mon[31773]: pgmap v875: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:29.042 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1381 sec 2024-03-28T18:15:30.043 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1382 sec 2024-03-28T18:15:30.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:30 smithi071 ceph-mon[36597]: pgmap v876: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:30.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:30 smithi151 ceph-mon[36481]: pgmap v876: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:30.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:30 smithi045 ceph-mon[31773]: pgmap v876: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:31.043 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1383 sec 2024-03-28T18:15:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 739.261289 secs 2024-03-28T18:15:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 739.261289 secs 2024-03-28T18:15:31.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 739.261289 secs 2024-03-28T18:15:32.043 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1384 sec 2024-03-28T18:15:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:32 smithi071 ceph-mon[36597]: pgmap v877: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:32 smithi151 ceph-mon[36481]: pgmap v877: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:32.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:32 smithi045 ceph-mon[31773]: pgmap v877: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:33.043 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1385 sec 2024-03-28T18:15:34.043 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1386 sec 2024-03-28T18:15:34.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:34 smithi071 ceph-mon[36597]: pgmap v878: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:34.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:34 smithi151 ceph-mon[36481]: pgmap v878: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:34.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:34 smithi045 ceph-mon[31773]: pgmap v878: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:35.043 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1387 sec 2024-03-28T18:15:36.044 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1388 sec 2024-03-28T18:15:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:36 smithi071 ceph-mon[36597]: pgmap v879: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 744.261377 secs 2024-03-28T18:15:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:36 smithi151 ceph-mon[36481]: pgmap v879: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 744.261377 secs 2024-03-28T18:15:36.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:36 smithi045 ceph-mon[31773]: pgmap v879: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:36.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 744.261377 secs 2024-03-28T18:15:37.044 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1389 sec 2024-03-28T18:15:38.044 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1390 sec 2024-03-28T18:15:38.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:38 smithi071 ceph-mon[36597]: pgmap v880: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:38.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:38 smithi151 ceph-mon[36481]: pgmap v880: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:38.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:38 smithi045 ceph-mon[31773]: pgmap v880: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:39.044 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1391 sec 2024-03-28T18:15:40.044 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1392 sec 2024-03-28T18:15:40.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:40 smithi071 ceph-mon[36597]: pgmap v881: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:40.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:40 smithi151 ceph-mon[36481]: pgmap v881: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:40.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:40 smithi045 ceph-mon[31773]: pgmap v881: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:41.044 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1393 sec 2024-03-28T18:15:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 749.261437 secs 2024-03-28T18:15:41.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 749.261437 secs 2024-03-28T18:15:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 749.261437 secs 2024-03-28T18:15:42.044 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1394 sec 2024-03-28T18:15:42.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:42 smithi151 ceph-mon[36481]: pgmap v882: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:42.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:42 smithi045 ceph-mon[31773]: pgmap v882: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:42 smithi071 ceph-mon[36597]: pgmap v882: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:43.044 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1395 sec 2024-03-28T18:15:44.045 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1396 sec 2024-03-28T18:15:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:44 smithi151 ceph-mon[36481]: pgmap v883: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:44.611 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:44 smithi071 ceph-mon[36597]: pgmap v883: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:44.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:44 smithi045 ceph-mon[31773]: pgmap v883: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:45.045 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1397 sec 2024-03-28T18:15:46.045 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1398 sec 2024-03-28T18:15:46.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:46 smithi045 ceph-mon[31773]: pgmap v884: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:46.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 754.261484 secs 2024-03-28T18:15:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:46 smithi071 ceph-mon[36597]: pgmap v884: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 754.261484 secs 2024-03-28T18:15:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:46 smithi151 ceph-mon[36481]: pgmap v884: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 754.261484 secs 2024-03-28T18:15:47.045 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1399 sec 2024-03-28T18:15:48.045 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1400 sec 2024-03-28T18:15:48.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:48 smithi045 ceph-mon[31773]: pgmap v885: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:48.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:48 smithi071 ceph-mon[36597]: pgmap v885: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:48 smithi151 ceph-mon[36481]: pgmap v885: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:49.045 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1401 sec 2024-03-28T18:15:50.045 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1402 sec 2024-03-28T18:15:50.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:50 smithi045 ceph-mon[31773]: pgmap v886: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:50.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:50 smithi071 ceph-mon[36597]: pgmap v886: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:50.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:50 smithi151 ceph-mon[36481]: pgmap v886: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:51.046 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1403 sec 2024-03-28T18:15:51.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 759.261546 secs 2024-03-28T18:15:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 759.261546 secs 2024-03-28T18:15:51.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 759.261546 secs 2024-03-28T18:15:52.046 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1404 sec 2024-03-28T18:15:52.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:52 smithi045 ceph-mon[31773]: pgmap v887: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:52 smithi071 ceph-mon[36597]: pgmap v887: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:52 smithi151 ceph-mon[36481]: pgmap v887: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:53.046 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1405 sec 2024-03-28T18:15:54.046 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1406 sec 2024-03-28T18:15:54.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:54 smithi045 ceph-mon[31773]: pgmap v888: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:54.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:54 smithi071 ceph-mon[36597]: pgmap v888: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:54.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:54 smithi151 ceph-mon[36481]: pgmap v888: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:55.046 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1407 sec 2024-03-28T18:15:56.046 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1408 sec 2024-03-28T18:15:56.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:56 smithi045 ceph-mon[31773]: pgmap v889: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:56.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 764.261604 secs 2024-03-28T18:15:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:56 smithi071 ceph-mon[36597]: pgmap v889: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 764.261604 secs 2024-03-28T18:15:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:56 smithi151 ceph-mon[36481]: pgmap v889: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:15:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 764.261604 secs 2024-03-28T18:15:57.046 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1409 sec 2024-03-28T18:15:58.047 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1410 sec 2024-03-28T18:15:58.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:15:58 smithi045 ceph-mon[31773]: pgmap v890: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:58.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:15:58 smithi071 ceph-mon[36597]: pgmap v890: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:58.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:15:58 smithi151 ceph-mon[36481]: pgmap v890: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:15:59.047 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1411 sec 2024-03-28T18:16:00.047 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1412 sec 2024-03-28T18:16:00.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:00 smithi045 ceph-mon[31773]: pgmap v891: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:00.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:00 smithi071 ceph-mon[36597]: pgmap v891: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:00 smithi151 ceph-mon[36481]: pgmap v891: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:01.047 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1413 sec 2024-03-28T18:16:01.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 769.261614 secs 2024-03-28T18:16:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 769.261614 secs 2024-03-28T18:16:01.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 769.261614 secs 2024-03-28T18:16:02.047 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1414 sec 2024-03-28T18:16:02.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:02 smithi045 ceph-mon[31773]: pgmap v892: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:02 smithi071 ceph-mon[36597]: pgmap v892: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:02 smithi151 ceph-mon[36481]: pgmap v892: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:03.047 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1415 sec 2024-03-28T18:16:04.047 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1416 sec 2024-03-28T18:16:04.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:04 smithi045 ceph-mon[31773]: pgmap v893: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:04.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:04 smithi071 ceph-mon[36597]: pgmap v893: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:04.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:04 smithi151 ceph-mon[36481]: pgmap v893: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:05.047 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1417 sec 2024-03-28T18:16:06.048 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1418 sec 2024-03-28T18:16:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:06 smithi045 ceph-mon[31773]: pgmap v894: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:06.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:06 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:16:06.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 774.261680 secs 2024-03-28T18:16:06.804 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:06 smithi151 ceph-mon[36481]: pgmap v894: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:06.804 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:06 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:16:06.804 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 774.261680 secs 2024-03-28T18:16:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:06 smithi071 ceph-mon[36597]: pgmap v894: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:06 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:16:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 774.261680 secs 2024-03-28T18:16:07.048 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1419 sec 2024-03-28T18:16:08.048 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1420 sec 2024-03-28T18:16:08.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:08 smithi151 ceph-mon[36481]: pgmap v895: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:08.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:08 smithi045 ceph-mon[31773]: pgmap v895: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:08.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:08 smithi071 ceph-mon[36597]: pgmap v895: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:09.048 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1421 sec 2024-03-28T18:16:10.048 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1422 sec 2024-03-28T18:16:11.048 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1423 sec 2024-03-28T18:16:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:10 smithi071 ceph-mon[36597]: pgmap v896: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:10 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:16:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:10 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:16:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:10 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:16:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:10 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:16:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:10 smithi071 ceph-mon[36597]: pgmap v897: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:16:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:10 smithi071 ceph-mon[36597]: pgmap v898: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:16:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:10 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:16:11.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:10 smithi151 ceph-mon[36481]: pgmap v896: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:11.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:10 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:16:11.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:10 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:16:11.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:10 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:16:11.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:10 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:16:11.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:10 smithi151 ceph-mon[36481]: pgmap v897: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:16:11.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:10 smithi151 ceph-mon[36481]: pgmap v898: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:16:11.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:10 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:16:11.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:10 smithi045 ceph-mon[31773]: pgmap v896: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:11.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:10 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:16:11.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:10 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:16:11.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:10 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:16:11.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:10 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:16:11.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:10 smithi045 ceph-mon[31773]: pgmap v897: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:16:11.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:10 smithi045 ceph-mon[31773]: pgmap v898: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:16:11.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:10 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:16:12.049 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1424 sec 2024-03-28T18:16:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 779.261743 secs 2024-03-28T18:16:12.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 779.261743 secs 2024-03-28T18:16:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 779.261743 secs 2024-03-28T18:16:13.049 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1425 sec 2024-03-28T18:16:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:12 smithi071 ceph-mon[36597]: pgmap v899: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:16:13.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:12 smithi151 ceph-mon[36481]: pgmap v899: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:16:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:12 smithi045 ceph-mon[31773]: pgmap v899: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:16:14.050 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1426 sec 2024-03-28T18:16:15.049 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1427 sec 2024-03-28T18:16:15.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:14 smithi045 ceph-mon[31773]: pgmap v900: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:16:15.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:14 smithi071 ceph-mon[36597]: pgmap v900: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:16:15.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:14 smithi151 ceph-mon[36481]: pgmap v900: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:16:16.049 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1428 sec 2024-03-28T18:16:17.049 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1429 sec 2024-03-28T18:16:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:16 smithi045 ceph-mon[31773]: pgmap v901: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:16:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 784.261801 secs 2024-03-28T18:16:17.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:16 smithi071 ceph-mon[36597]: pgmap v901: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:16:17.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 784.261801 secs 2024-03-28T18:16:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:16 smithi151 ceph-mon[36481]: pgmap v901: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:16:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 784.261801 secs 2024-03-28T18:16:18.049 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1430 sec 2024-03-28T18:16:19.050 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1431 sec 2024-03-28T18:16:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:19 smithi151 ceph-mon[36481]: pgmap v902: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:16:19.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:19 smithi045 ceph-mon[31773]: pgmap v902: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:16:19.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:19 smithi071 ceph-mon[36597]: pgmap v902: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:16:20.049 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1432 sec 2024-03-28T18:16:21.050 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1433 sec 2024-03-28T18:16:21.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:21 smithi151 ceph-mon[36481]: pgmap v903: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:16:21.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 789.261812 secs 2024-03-28T18:16:21.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:21 smithi045 ceph-mon[31773]: pgmap v903: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:16:21.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 789.261812 secs 2024-03-28T18:16:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:21 smithi071 ceph-mon[36597]: pgmap v903: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:16:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 789.261812 secs 2024-03-28T18:16:22.050 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1434 sec 2024-03-28T18:16:23.050 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1435 sec 2024-03-28T18:16:23.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:23 smithi045 ceph-mon[31773]: pgmap v904: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:23.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:23 smithi071 ceph-mon[36597]: pgmap v904: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:23.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:23 smithi151 ceph-mon[36481]: pgmap v904: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:24.050 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1436 sec 2024-03-28T18:16:25.050 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1437 sec 2024-03-28T18:16:25.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:25 smithi045 ceph-mon[31773]: pgmap v905: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:25.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:25 smithi071 ceph-mon[36597]: pgmap v905: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:25.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:25 smithi151 ceph-mon[36481]: pgmap v905: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:26.050 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1438 sec 2024-03-28T18:16:26.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 794.261900 secs 2024-03-28T18:16:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 794.261900 secs 2024-03-28T18:16:26.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 794.261900 secs 2024-03-28T18:16:27.050 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1439 sec 2024-03-28T18:16:27.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:27 smithi045 ceph-mon[31773]: pgmap v906: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:27 smithi071 ceph-mon[36597]: pgmap v906: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:27 smithi151 ceph-mon[36481]: pgmap v906: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:28.051 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1440 sec 2024-03-28T18:16:29.051 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1441 sec 2024-03-28T18:16:29.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:29 smithi045 ceph-mon[31773]: pgmap v907: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:29.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:29 smithi071 ceph-mon[36597]: pgmap v907: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:29.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:29 smithi151 ceph-mon[36481]: pgmap v907: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:30.051 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1442 sec 2024-03-28T18:16:30.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:30 smithi045 ceph-mon[31773]: pgmap v908: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:30.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:30 smithi071 ceph-mon[36597]: pgmap v908: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:30.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:30 smithi151 ceph-mon[36481]: pgmap v908: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:31.051 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1443 sec 2024-03-28T18:16:31.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 799.261990 secs 2024-03-28T18:16:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 799.261990 secs 2024-03-28T18:16:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 799.261990 secs 2024-03-28T18:16:32.051 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1444 sec 2024-03-28T18:16:32.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:32 smithi045 ceph-mon[31773]: pgmap v909: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:32 smithi071 ceph-mon[36597]: pgmap v909: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:32.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:32 smithi151 ceph-mon[36481]: pgmap v909: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:33.051 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1445 sec 2024-03-28T18:16:34.051 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1446 sec 2024-03-28T18:16:35.052 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1447 sec 2024-03-28T18:16:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:34 smithi045 ceph-mon[31773]: pgmap v910: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:35.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:34 smithi071 ceph-mon[36597]: pgmap v910: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:35.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:34 smithi151 ceph-mon[36481]: pgmap v910: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:36.052 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1448 sec 2024-03-28T18:16:37.052 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1449 sec 2024-03-28T18:16:37.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:36 smithi151 ceph-mon[36481]: pgmap v911: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:37.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 804.262078 secs 2024-03-28T18:16:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:36 smithi045 ceph-mon[31773]: pgmap v911: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 804.262078 secs 2024-03-28T18:16:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:36 smithi071 ceph-mon[36597]: pgmap v911: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 804.262078 secs 2024-03-28T18:16:38.052 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1450 sec 2024-03-28T18:16:39.052 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1451 sec 2024-03-28T18:16:39.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:38 smithi045 ceph-mon[31773]: pgmap v912: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:39.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:38 smithi071 ceph-mon[36597]: pgmap v912: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:39.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:38 smithi151 ceph-mon[36481]: pgmap v912: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:40.052 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1452 sec 2024-03-28T18:16:41.053 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1453 sec 2024-03-28T18:16:41.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:40 smithi045 ceph-mon[31773]: pgmap v913: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:41.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:40 smithi071 ceph-mon[36597]: pgmap v913: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:41.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:40 smithi151 ceph-mon[36481]: pgmap v913: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:42.053 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1454 sec 2024-03-28T18:16:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 809.262162 secs 2024-03-28T18:16:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 809.262162 secs 2024-03-28T18:16:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 809.262162 secs 2024-03-28T18:16:43.053 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1455 sec 2024-03-28T18:16:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:42 smithi045 ceph-mon[31773]: pgmap v914: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:43.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:42 smithi071 ceph-mon[36597]: pgmap v914: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:43.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:42 smithi151 ceph-mon[36481]: pgmap v914: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:44.053 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1456 sec 2024-03-28T18:16:45.053 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1457 sec 2024-03-28T18:16:45.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:44 smithi045 ceph-mon[31773]: pgmap v915: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:45.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:44 smithi071 ceph-mon[36597]: pgmap v915: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:45.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:44 smithi151 ceph-mon[36481]: pgmap v915: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:46.053 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1458 sec 2024-03-28T18:16:47.053 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1459 sec 2024-03-28T18:16:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:46 smithi045 ceph-mon[31773]: pgmap v916: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 814.262247 secs 2024-03-28T18:16:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:46 smithi071 ceph-mon[36597]: pgmap v916: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 814.262247 secs 2024-03-28T18:16:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:46 smithi151 ceph-mon[36481]: pgmap v916: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 814.262247 secs 2024-03-28T18:16:48.053 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1460 sec 2024-03-28T18:16:49.054 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1461 sec 2024-03-28T18:16:49.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:48 smithi045 ceph-mon[31773]: pgmap v917: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:49.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:48 smithi071 ceph-mon[36597]: pgmap v917: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:49.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:48 smithi151 ceph-mon[36481]: pgmap v917: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:50.054 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1462 sec 2024-03-28T18:16:51.054 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1463 sec 2024-03-28T18:16:51.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:50 smithi045 ceph-mon[31773]: pgmap v918: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:51.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:50 smithi071 ceph-mon[36597]: pgmap v918: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:51.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:50 smithi151 ceph-mon[36481]: pgmap v918: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:52.054 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1464 sec 2024-03-28T18:16:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 819.262334 secs 2024-03-28T18:16:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 819.262334 secs 2024-03-28T18:16:52.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 819.262334 secs 2024-03-28T18:16:53.054 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1465 sec 2024-03-28T18:16:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:52 smithi045 ceph-mon[31773]: pgmap v919: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:53.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:52 smithi071 ceph-mon[36597]: pgmap v919: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:53.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:52 smithi151 ceph-mon[36481]: pgmap v919: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:54.054 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1466 sec 2024-03-28T18:16:55.055 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1467 sec 2024-03-28T18:16:55.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:54 smithi045 ceph-mon[31773]: pgmap v920: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:55.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:54 smithi071 ceph-mon[36597]: pgmap v920: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:55.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:54 smithi151 ceph-mon[36481]: pgmap v920: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:56.055 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1468 sec 2024-03-28T18:16:57.055 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1469 sec 2024-03-28T18:16:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:56 smithi045 ceph-mon[31773]: pgmap v921: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 824.262392 secs 2024-03-28T18:16:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:56 smithi071 ceph-mon[36597]: pgmap v921: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 824.262392 secs 2024-03-28T18:16:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:56 smithi151 ceph-mon[36481]: pgmap v921: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:16:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 824.262392 secs 2024-03-28T18:16:58.055 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1470 sec 2024-03-28T18:16:59.055 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1471 sec 2024-03-28T18:16:59.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:16:59 smithi045 ceph-mon[31773]: pgmap v922: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:59.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:16:59 smithi071 ceph-mon[36597]: pgmap v922: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:16:59.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:16:59 smithi151 ceph-mon[36481]: pgmap v922: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:00.055 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1472 sec 2024-03-28T18:17:01.055 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1473 sec 2024-03-28T18:17:01.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:01 smithi045 ceph-mon[31773]: pgmap v923: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:01.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 829.262405 secs 2024-03-28T18:17:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:01 smithi071 ceph-mon[36597]: pgmap v923: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 829.262405 secs 2024-03-28T18:17:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:01 smithi151 ceph-mon[36481]: pgmap v923: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 829.262405 secs 2024-03-28T18:17:02.055 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1474 sec 2024-03-28T18:17:03.056 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1475 sec 2024-03-28T18:17:03.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:03 smithi045 ceph-mon[31773]: pgmap v924: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:03.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:03 smithi071 ceph-mon[36597]: pgmap v924: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:03.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:03 smithi151 ceph-mon[36481]: pgmap v924: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:04.056 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1476 sec 2024-03-28T18:17:05.056 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1477 sec 2024-03-28T18:17:05.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:05 smithi045 ceph-mon[31773]: pgmap v925: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:05.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:05 smithi071 ceph-mon[36597]: pgmap v925: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:05.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:05 smithi151 ceph-mon[36481]: pgmap v925: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:06.056 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1478 sec 2024-03-28T18:17:06.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:06 smithi045 ceph-mon[31773]: pgmap v926: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:06.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 834.262503 secs 2024-03-28T18:17:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:06 smithi071 ceph-mon[36597]: pgmap v926: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 834.262503 secs 2024-03-28T18:17:06.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:06 smithi151 ceph-mon[36481]: pgmap v926: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 834.262503 secs 2024-03-28T18:17:07.056 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1479 sec 2024-03-28T18:17:08.056 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1480 sec 2024-03-28T18:17:09.056 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1481 sec 2024-03-28T18:17:09.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:08 smithi045 ceph-mon[31773]: pgmap v927: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:09.276 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:08 smithi151 ceph-mon[36481]: pgmap v927: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:09.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:08 smithi071 ceph-mon[36597]: pgmap v927: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:10.057 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1482 sec 2024-03-28T18:17:11.057 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1483 sec 2024-03-28T18:17:11.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:10 smithi045 ceph-mon[31773]: pgmap v928: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:11.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:10 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:17:11.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:10 smithi071 ceph-mon[36597]: pgmap v928: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:11.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:10 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:17:11.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:10 smithi151 ceph-mon[36481]: pgmap v928: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:11.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:10 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:17:12.057 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1484 sec 2024-03-28T18:17:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 839.262524 secs 2024-03-28T18:17:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:11 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:17:12.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:11 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:17:12.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:11 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:17:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 839.262524 secs 2024-03-28T18:17:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:11 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:17:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:11 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:17:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:11 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:17:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 839.262524 secs 2024-03-28T18:17:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:11 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:17:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:11 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:17:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:11 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:17:13.057 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1485 sec 2024-03-28T18:17:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:12 smithi045 ceph-mon[31773]: pgmap v929: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:17:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:12 smithi045 ceph-mon[31773]: pgmap v930: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:17:13.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:12 smithi071 ceph-mon[36597]: pgmap v929: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:17:13.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:12 smithi071 ceph-mon[36597]: pgmap v930: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:17:13.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:12 smithi151 ceph-mon[36481]: pgmap v929: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:17:13.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:12 smithi151 ceph-mon[36481]: pgmap v930: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:17:14.057 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1486 sec 2024-03-28T18:17:15.057 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1487 sec 2024-03-28T18:17:15.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:14 smithi045 ceph-mon[31773]: pgmap v931: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:17:15.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:14 smithi071 ceph-mon[36597]: pgmap v931: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:17:15.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:14 smithi151 ceph-mon[36481]: pgmap v931: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:17:16.058 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1488 sec 2024-03-28T18:17:17.058 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1489 sec 2024-03-28T18:17:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:16 smithi045 ceph-mon[31773]: pgmap v932: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:17:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 844.262586 secs 2024-03-28T18:17:17.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:16 smithi071 ceph-mon[36597]: pgmap v932: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:17:17.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 844.262586 secs 2024-03-28T18:17:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:16 smithi151 ceph-mon[36481]: pgmap v932: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:17:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 844.262586 secs 2024-03-28T18:17:18.058 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1490 sec 2024-03-28T18:17:19.058 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1491 sec 2024-03-28T18:17:19.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:18 smithi045 ceph-mon[31773]: pgmap v933: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:17:19.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:18 smithi071 ceph-mon[36597]: pgmap v933: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:17:19.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:18 smithi151 ceph-mon[36481]: pgmap v933: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:17:20.058 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1492 sec 2024-03-28T18:17:21.058 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1493 sec 2024-03-28T18:17:21.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:20 smithi045 ceph-mon[31773]: pgmap v934: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:17:21.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:20 smithi071 ceph-mon[36597]: pgmap v934: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:17:21.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:20 smithi151 ceph-mon[36481]: pgmap v934: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:17:22.058 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1494 sec 2024-03-28T18:17:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 849.262602 secs 2024-03-28T18:17:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 849.262602 secs 2024-03-28T18:17:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 849.262602 secs 2024-03-28T18:17:23.059 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1495 sec 2024-03-28T18:17:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:22 smithi045 ceph-mon[31773]: pgmap v935: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:17:23.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:22 smithi071 ceph-mon[36597]: pgmap v935: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:17:23.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:22 smithi151 ceph-mon[36481]: pgmap v935: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:17:24.059 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1496 sec 2024-03-28T18:17:25.059 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1497 sec 2024-03-28T18:17:25.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:24 smithi071 ceph-mon[36597]: pgmap v936: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:25.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:24 smithi151 ceph-mon[36481]: pgmap v936: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:25.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:24 smithi045 ceph-mon[31773]: pgmap v936: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:26.059 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1498 sec 2024-03-28T18:17:27.059 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1499 sec 2024-03-28T18:17:27.219 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:26 smithi045 ceph-mon[31773]: pgmap v937: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:27.219 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 854.262685 secs 2024-03-28T18:17:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:26 smithi071 ceph-mon[36597]: pgmap v937: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 854.262685 secs 2024-03-28T18:17:27.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:26 smithi151 ceph-mon[36481]: pgmap v937: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 854.262685 secs 2024-03-28T18:17:28.059 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1500 sec 2024-03-28T18:17:29.059 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1501 sec 2024-03-28T18:17:29.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:28 smithi071 ceph-mon[36597]: pgmap v938: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:29.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:28 smithi151 ceph-mon[36481]: pgmap v938: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:29.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:28 smithi045 ceph-mon[31773]: pgmap v938: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:30.060 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1502 sec 2024-03-28T18:17:31.060 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1503 sec 2024-03-28T18:17:31.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:30 smithi071 ceph-mon[36597]: pgmap v939: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:31.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:30 smithi151 ceph-mon[36481]: pgmap v939: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:31.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:30 smithi045 ceph-mon[31773]: pgmap v939: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:32.060 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1504 sec 2024-03-28T18:17:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 859.262697 secs 2024-03-28T18:17:32.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 859.262697 secs 2024-03-28T18:17:32.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 859.262697 secs 2024-03-28T18:17:33.060 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1505 sec 2024-03-28T18:17:33.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:32 smithi071 ceph-mon[36597]: pgmap v940: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:33.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:32 smithi151 ceph-mon[36481]: pgmap v940: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:33.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:32 smithi045 ceph-mon[31773]: pgmap v940: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:34.060 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1506 sec 2024-03-28T18:17:35.060 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1507 sec 2024-03-28T18:17:35.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:34 smithi071 ceph-mon[36597]: pgmap v941: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:35.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:34 smithi151 ceph-mon[36481]: pgmap v941: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:35.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:34 smithi045 ceph-mon[31773]: pgmap v941: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:36.060 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1508 sec 2024-03-28T18:17:37.061 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1509 sec 2024-03-28T18:17:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:36 smithi071 ceph-mon[36597]: pgmap v942: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 864.262784 secs 2024-03-28T18:17:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:36 smithi151 ceph-mon[36481]: pgmap v942: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 864.262784 secs 2024-03-28T18:17:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:36 smithi045 ceph-mon[31773]: pgmap v942: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 864.262784 secs 2024-03-28T18:17:38.061 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1510 sec 2024-03-28T18:17:39.061 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1511 sec 2024-03-28T18:17:39.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:38 smithi071 ceph-mon[36597]: pgmap v943: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:39.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:38 smithi151 ceph-mon[36481]: pgmap v943: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:39.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:38 smithi045 ceph-mon[31773]: pgmap v943: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:40.061 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1512 sec 2024-03-28T18:17:41.061 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1513 sec 2024-03-28T18:17:41.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:40 smithi071 ceph-mon[36597]: pgmap v944: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:41.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:40 smithi151 ceph-mon[36481]: pgmap v944: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:41.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:40 smithi045 ceph-mon[31773]: pgmap v944: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:42.061 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1514 sec 2024-03-28T18:17:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 869.262863 secs 2024-03-28T18:17:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 869.262863 secs 2024-03-28T18:17:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 869.262863 secs 2024-03-28T18:17:43.061 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1515 sec 2024-03-28T18:17:43.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:42 smithi071 ceph-mon[36597]: pgmap v945: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:43.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:42 smithi151 ceph-mon[36481]: pgmap v945: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:43.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:42 smithi045 ceph-mon[31773]: pgmap v945: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:44.062 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1516 sec 2024-03-28T18:17:45.062 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1517 sec 2024-03-28T18:17:45.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:44 smithi071 ceph-mon[36597]: pgmap v946: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:45.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:44 smithi151 ceph-mon[36481]: pgmap v946: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:45.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:44 smithi045 ceph-mon[31773]: pgmap v946: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:46.062 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1518 sec 2024-03-28T18:17:46.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:45 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 874.262951 secs 2024-03-28T18:17:46.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:45 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 874.262951 secs 2024-03-28T18:17:46.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:45 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 874.262951 secs 2024-03-28T18:17:47.062 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1519 sec 2024-03-28T18:17:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:46 smithi071 ceph-mon[36597]: pgmap v947: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:47.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:46 smithi045 ceph-mon[31773]: pgmap v947: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:46 smithi151 ceph-mon[36481]: pgmap v947: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:48.062 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1520 sec 2024-03-28T18:17:49.063 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1521 sec 2024-03-28T18:17:49.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:48 smithi071 ceph-mon[36597]: pgmap v948: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:49.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:48 smithi151 ceph-mon[36481]: pgmap v948: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:49.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:48 smithi045 ceph-mon[31773]: pgmap v948: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:50.062 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1522 sec 2024-03-28T18:17:51.063 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1523 sec 2024-03-28T18:17:51.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:51 smithi071 ceph-mon[36597]: pgmap v949: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:51.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 879.263038 secs 2024-03-28T18:17:51.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:51 smithi151 ceph-mon[36481]: pgmap v949: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:51.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 879.263038 secs 2024-03-28T18:17:51.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:51 smithi045 ceph-mon[31773]: pgmap v949: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:51.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 879.263038 secs 2024-03-28T18:17:52.063 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1524 sec 2024-03-28T18:17:53.063 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1525 sec 2024-03-28T18:17:53.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:53 smithi071 ceph-mon[36597]: pgmap v950: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:53.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:53 smithi151 ceph-mon[36481]: pgmap v950: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:53.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:53 smithi045 ceph-mon[31773]: pgmap v950: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:54.063 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1526 sec 2024-03-28T18:17:55.063 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1527 sec 2024-03-28T18:17:55.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:55 smithi071 ceph-mon[36597]: pgmap v951: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:55.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:55 smithi151 ceph-mon[36481]: pgmap v951: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:55.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:55 smithi045 ceph-mon[31773]: pgmap v951: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:56.063 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1528 sec 2024-03-28T18:17:56.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 884.263127 secs 2024-03-28T18:17:56.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 884.263127 secs 2024-03-28T18:17:56.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 884.263127 secs 2024-03-28T18:17:57.063 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1529 sec 2024-03-28T18:17:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:57 smithi071 ceph-mon[36597]: pgmap v952: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:57.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:57 smithi045 ceph-mon[31773]: pgmap v952: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:57 smithi151 ceph-mon[36481]: pgmap v952: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:17:58.063 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1530 sec 2024-03-28T18:17:59.064 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1531 sec 2024-03-28T18:17:59.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:17:59 smithi071 ceph-mon[36597]: pgmap v953: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:17:59 smithi151 ceph-mon[36481]: pgmap v953: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:17:59.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:17:59 smithi045 ceph-mon[31773]: pgmap v953: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:00.064 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1532 sec 2024-03-28T18:18:01.064 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1533 sec 2024-03-28T18:18:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:01 smithi071 ceph-mon[36597]: pgmap v954: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 889.263188 secs 2024-03-28T18:18:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:01 smithi151 ceph-mon[36481]: pgmap v954: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 889.263188 secs 2024-03-28T18:18:01.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:01 smithi045 ceph-mon[31773]: pgmap v954: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:01.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 889.263188 secs 2024-03-28T18:18:02.064 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1534 sec 2024-03-28T18:18:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:02 smithi071 ceph-mon[36597]: 1 slow requests, 1 included below; oldest blocked for > 180.686877 secs 2024-03-28T18:18:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:02 smithi071 ceph-mon[36597]: slow request 180.686876 seconds old, received at 2024-03-28T18:15:00.947128+0000: client_request(client.26687:1 getattr pAsLsXsFs #0x1 2024-03-28T18:15:00.943510+0000 caller_uid=0, caller_gid=0{}) currently dispatched 2024-03-28T18:18:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:02 smithi151 ceph-mon[36481]: 1 slow requests, 1 included below; oldest blocked for > 180.686877 secs 2024-03-28T18:18:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:02 smithi151 ceph-mon[36481]: slow request 180.686876 seconds old, received at 2024-03-28T18:15:00.947128+0000: client_request(client.26687:1 getattr pAsLsXsFs #0x1 2024-03-28T18:15:00.943510+0000 caller_uid=0, caller_gid=0{}) currently dispatched 2024-03-28T18:18:02.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:02 smithi045 ceph-mon[31773]: 1 slow requests, 1 included below; oldest blocked for > 180.686877 secs 2024-03-28T18:18:02.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:02 smithi045 ceph-mon[31773]: slow request 180.686876 seconds old, received at 2024-03-28T18:15:00.947128+0000: client_request(client.26687:1 getattr pAsLsXsFs #0x1 2024-03-28T18:15:00.943510+0000 caller_uid=0, caller_gid=0{}) currently dispatched 2024-03-28T18:18:03.064 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1535 sec 2024-03-28T18:18:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:03 smithi071 ceph-mon[36597]: pgmap v955: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:03.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:03 smithi151 ceph-mon[36481]: pgmap v955: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:03.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:03 smithi045 ceph-mon[31773]: pgmap v955: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:04.064 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1536 sec 2024-03-28T18:18:04.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:04 smithi045 ceph-mon[31773]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-28T18:18:04.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:04 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] up:active 2024-03-28T18:18:04.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:04 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T18:18:04.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:04 smithi071 ceph-mon[36597]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-28T18:18:04.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:04 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] up:active 2024-03-28T18:18:04.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:04 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T18:18:04.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:04 smithi151 ceph-mon[36481]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-28T18:18:04.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:04 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] up:active 2024-03-28T18:18:04.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:04 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T18:18:05.064 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1537 sec 2024-03-28T18:18:05.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:05 smithi045 ceph-mon[31773]: pgmap v956: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:05.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:05 smithi071 ceph-mon[36597]: pgmap v956: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:05.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:05 smithi151 ceph-mon[36481]: pgmap v956: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:06.065 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1538 sec 2024-03-28T18:18:06.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 894.263277 secs 2024-03-28T18:18:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 894.263277 secs 2024-03-28T18:18:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 894.263277 secs 2024-03-28T18:18:07.065 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1539 sec 2024-03-28T18:18:07.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:07 smithi045 ceph-mon[31773]: pgmap v957: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:07.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:07 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 185.686957 secs 2024-03-28T18:18:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:07 smithi071 ceph-mon[36597]: pgmap v957: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:07 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 185.686957 secs 2024-03-28T18:18:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:07 smithi151 ceph-mon[36481]: pgmap v957: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:07 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 185.686957 secs 2024-03-28T18:18:08.065 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1540 sec 2024-03-28T18:18:08.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:08 smithi045 ceph-mon[31773]: pgmap v958: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:08.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:08 smithi071 ceph-mon[36597]: pgmap v958: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:08.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:08 smithi151 ceph-mon[36481]: pgmap v958: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:09.065 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1541 sec 2024-03-28T18:18:10.065 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1542 sec 2024-03-28T18:18:10.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:10 smithi071 ceph-mon[36597]: pgmap v959: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:10.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:10 smithi151 ceph-mon[36481]: pgmap v959: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:10.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:10 smithi045 ceph-mon[31773]: pgmap v959: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:11.065 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1543 sec 2024-03-28T18:18:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 899.263318 secs 2024-03-28T18:18:11.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 899.263318 secs 2024-03-28T18:18:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 899.263318 secs 2024-03-28T18:18:12.065 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1544 sec 2024-03-28T18:18:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:12 smithi071 ceph-mon[36597]: pgmap v960: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:12 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 190.686971 secs 2024-03-28T18:18:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:12 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:18:12.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:12 smithi151 ceph-mon[36481]: pgmap v960: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:12.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:12 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 190.686971 secs 2024-03-28T18:18:12.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:12 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:18:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:12 smithi045 ceph-mon[31773]: pgmap v960: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:12.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:12 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 190.686971 secs 2024-03-28T18:18:12.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:12 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:18:13.065 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1545 sec 2024-03-28T18:18:13.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:13 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:18:13.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:13 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:18:13.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:13 smithi071 ceph-mon[36597]: pgmap v961: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:13.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:13 smithi071 ceph-mon[36597]: pgmap v962: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:18:13.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:13 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:18:13.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:13 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:18:13.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:13 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:18:13.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:13 smithi151 ceph-mon[36481]: pgmap v961: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:13.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:13 smithi151 ceph-mon[36481]: pgmap v962: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:18:13.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:13 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:18:13.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:13 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:18:13.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:13 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:18:13.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:13 smithi045 ceph-mon[31773]: pgmap v961: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:13.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:13 smithi045 ceph-mon[31773]: pgmap v962: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:18:13.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:13 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:18:14.065 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1546 sec 2024-03-28T18:18:15.066 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1547 sec 2024-03-28T18:18:16.066 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1548 sec 2024-03-28T18:18:16.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:16 smithi045 ceph-mon[31773]: pgmap v963: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:18:16.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 904.263358 secs 2024-03-28T18:18:16.513 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:16 smithi071 ceph-mon[36597]: pgmap v963: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:18:16.513 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 904.263358 secs 2024-03-28T18:18:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:16 smithi151 ceph-mon[36481]: pgmap v963: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:18:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 904.263358 secs 2024-03-28T18:18:17.066 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1549 sec 2024-03-28T18:18:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:17 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 195.687055 secs 2024-03-28T18:18:17.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:17 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 195.687055 secs 2024-03-28T18:18:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:17 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 195.687055 secs 2024-03-28T18:18:18.066 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1550 sec 2024-03-28T18:18:18.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:18 smithi045 ceph-mon[31773]: pgmap v964: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:18:18.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:18 smithi071 ceph-mon[36597]: pgmap v964: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:18:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:18 smithi151 ceph-mon[36481]: pgmap v964: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:18:19.066 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1551 sec 2024-03-28T18:18:19.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:19 smithi045 ceph-mon[31773]: pgmap v965: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:18:19.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:19 smithi071 ceph-mon[36597]: pgmap v965: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:18:19.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:19 smithi151 ceph-mon[36481]: pgmap v965: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:18:20.066 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1552 sec 2024-03-28T18:18:21.067 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1553 sec 2024-03-28T18:18:21.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 909.263420 secs 2024-03-28T18:18:21.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 909.263420 secs 2024-03-28T18:18:21.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 909.263420 secs 2024-03-28T18:18:22.067 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1554 sec 2024-03-28T18:18:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:22 smithi071 ceph-mon[36597]: pgmap v966: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:18:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:22 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 200.687114 secs 2024-03-28T18:18:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:22 smithi151 ceph-mon[36481]: pgmap v966: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:18:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:22 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 200.687114 secs 2024-03-28T18:18:22.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:22 smithi045 ceph-mon[31773]: pgmap v966: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:18:22.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:22 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 200.687114 secs 2024-03-28T18:18:23.067 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1555 sec 2024-03-28T18:18:24.067 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1556 sec 2024-03-28T18:18:24.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:24 smithi045 ceph-mon[31773]: pgmap v967: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:18:24.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:24 smithi071 ceph-mon[36597]: pgmap v967: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:18:24.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:24 smithi151 ceph-mon[36481]: pgmap v967: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:18:25.067 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1557 sec 2024-03-28T18:18:25.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:25 smithi045 ceph-mon[31773]: pgmap v968: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:25.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:25 smithi071 ceph-mon[36597]: pgmap v968: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:25.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:25 smithi151 ceph-mon[36481]: pgmap v968: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:26.067 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1558 sec 2024-03-28T18:18:26.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 914.263488 secs 2024-03-28T18:18:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 914.263488 secs 2024-03-28T18:18:26.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 914.263488 secs 2024-03-28T18:18:27.067 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1559 sec 2024-03-28T18:18:27.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:27 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 205.687126 secs 2024-03-28T18:18:27.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:27 smithi045 ceph-mon[31773]: pgmap v969: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:27 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 205.687126 secs 2024-03-28T18:18:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:27 smithi071 ceph-mon[36597]: pgmap v969: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:27 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 205.687126 secs 2024-03-28T18:18:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:27 smithi151 ceph-mon[36481]: pgmap v969: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:28.068 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1560 sec 2024-03-28T18:18:29.068 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1561 sec 2024-03-28T18:18:29.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:29 smithi071 ceph-mon[36597]: pgmap v970: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:29.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:29 smithi151 ceph-mon[36481]: pgmap v970: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:29.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:29 smithi045 ceph-mon[31773]: pgmap v970: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:30.068 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1562 sec 2024-03-28T18:18:31.068 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1563 sec 2024-03-28T18:18:31.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 919.263577 secs 2024-03-28T18:18:31.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 919.263577 secs 2024-03-28T18:18:31.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 919.263577 secs 2024-03-28T18:18:32.069 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1564 sec 2024-03-28T18:18:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:32 smithi071 ceph-mon[36597]: pgmap v971: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:32 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 210.687215 secs 2024-03-28T18:18:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:32 smithi151 ceph-mon[36481]: pgmap v971: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:32 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 210.687215 secs 2024-03-28T18:18:32.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:32 smithi045 ceph-mon[31773]: pgmap v971: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:32.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:32 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 210.687215 secs 2024-03-28T18:18:33.068 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1565 sec 2024-03-28T18:18:33.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:33 smithi071 ceph-mon[36597]: pgmap v972: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:33.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:33 smithi151 ceph-mon[36481]: pgmap v972: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:33.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:33 smithi045 ceph-mon[31773]: pgmap v972: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:34.068 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1566 sec 2024-03-28T18:18:35.069 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1567 sec 2024-03-28T18:18:35.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:35 smithi071 ceph-mon[36597]: pgmap v973: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:35.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:35 smithi151 ceph-mon[36481]: pgmap v973: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:35.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:35 smithi045 ceph-mon[31773]: pgmap v973: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:36.069 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1568 sec 2024-03-28T18:18:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 924.263638 secs 2024-03-28T18:18:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 924.263638 secs 2024-03-28T18:18:36.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 924.263638 secs 2024-03-28T18:18:37.069 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1569 sec 2024-03-28T18:18:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:37 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 215.687303 secs 2024-03-28T18:18:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:37 smithi071 ceph-mon[36597]: pgmap v974: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:37 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 215.687303 secs 2024-03-28T18:18:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:37 smithi151 ceph-mon[36481]: pgmap v974: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:37.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:37 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 215.687303 secs 2024-03-28T18:18:37.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:37 smithi045 ceph-mon[31773]: pgmap v974: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:38.069 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1570 sec 2024-03-28T18:18:39.069 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1571 sec 2024-03-28T18:18:39.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:39 smithi071 ceph-mon[36597]: pgmap v975: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:39.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:39 smithi151 ceph-mon[36481]: pgmap v975: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:39.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:39 smithi045 ceph-mon[31773]: pgmap v975: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:40.069 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1572 sec 2024-03-28T18:18:41.069 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1573 sec 2024-03-28T18:18:41.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 929.263722 secs 2024-03-28T18:18:41.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 929.263722 secs 2024-03-28T18:18:41.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 929.263722 secs 2024-03-28T18:18:42.070 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1574 sec 2024-03-28T18:18:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:42 smithi071 ceph-mon[36597]: pgmap v976: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:42 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 220.687394 secs 2024-03-28T18:18:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:42 smithi151 ceph-mon[36481]: pgmap v976: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:42 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 220.687394 secs 2024-03-28T18:18:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:42 smithi045 ceph-mon[31773]: pgmap v976: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:42 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 220.687394 secs 2024-03-28T18:18:43.070 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1575 sec 2024-03-28T18:18:43.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:43 smithi071 ceph-mon[36597]: pgmap v977: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:43.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:43 smithi151 ceph-mon[36481]: pgmap v977: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:43.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:43 smithi045 ceph-mon[31773]: pgmap v977: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:44.070 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1576 sec 2024-03-28T18:18:45.070 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1577 sec 2024-03-28T18:18:45.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:45 smithi071 ceph-mon[36597]: pgmap v978: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:45.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:45 smithi151 ceph-mon[36481]: pgmap v978: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:45.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:45 smithi045 ceph-mon[31773]: pgmap v978: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:46.070 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1578 sec 2024-03-28T18:18:46.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 934.263781 secs 2024-03-28T18:18:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 934.263781 secs 2024-03-28T18:18:46.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 934.263781 secs 2024-03-28T18:18:47.070 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1579 sec 2024-03-28T18:18:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:47 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 225.687478 secs 2024-03-28T18:18:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:47 smithi071 ceph-mon[36597]: pgmap v979: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:47 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 225.687478 secs 2024-03-28T18:18:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:47 smithi151 ceph-mon[36481]: pgmap v979: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:47 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 225.687478 secs 2024-03-28T18:18:47.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:47 smithi045 ceph-mon[31773]: pgmap v979: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:48.070 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1580 sec 2024-03-28T18:18:49.071 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1581 sec 2024-03-28T18:18:49.468 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:49 smithi045 ceph-mon[31773]: pgmap v980: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:49.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:49 smithi071 ceph-mon[36597]: pgmap v980: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:49 smithi151 ceph-mon[36481]: pgmap v980: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:50.071 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1582 sec 2024-03-28T18:18:51.071 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1583 sec 2024-03-28T18:18:51.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 939.263820 secs 2024-03-28T18:18:51.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 939.263820 secs 2024-03-28T18:18:51.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 939.263820 secs 2024-03-28T18:18:52.071 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1584 sec 2024-03-28T18:18:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:52 smithi071 ceph-mon[36597]: pgmap v981: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:52 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 230.687518 secs 2024-03-28T18:18:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:52 smithi151 ceph-mon[36481]: pgmap v981: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:52 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 230.687518 secs 2024-03-28T18:18:52.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:52 smithi045 ceph-mon[31773]: pgmap v981: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:52.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:52 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 230.687518 secs 2024-03-28T18:18:53.071 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1585 sec 2024-03-28T18:18:53.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:53 smithi071 ceph-mon[36597]: pgmap v982: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:53.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:53 smithi151 ceph-mon[36481]: pgmap v982: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:53.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:53 smithi045 ceph-mon[31773]: pgmap v982: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:54.071 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1586 sec 2024-03-28T18:18:55.071 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1587 sec 2024-03-28T18:18:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:55 smithi071 ceph-mon[36597]: pgmap v983: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:55.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:55 smithi151 ceph-mon[36481]: pgmap v983: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:55.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:55 smithi045 ceph-mon[31773]: pgmap v983: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:18:56.072 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1588 sec 2024-03-28T18:18:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 944.263903 secs 2024-03-28T18:18:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 944.263903 secs 2024-03-28T18:18:56.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 944.263903 secs 2024-03-28T18:18:57.072 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1589 sec 2024-03-28T18:18:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:57 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 235.687607 secs 2024-03-28T18:18:57.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:57 smithi071 ceph-mon[36597]: pgmap v984: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:57.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:57 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 235.687607 secs 2024-03-28T18:18:57.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:57 smithi045 ceph-mon[31773]: pgmap v984: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:57 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 235.687607 secs 2024-03-28T18:18:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:57 smithi151 ceph-mon[36481]: pgmap v984: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:58.071 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1590 sec 2024-03-28T18:18:59.072 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1591 sec 2024-03-28T18:18:59.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:18:59 smithi071 ceph-mon[36597]: pgmap v985: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:59.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:18:59 smithi151 ceph-mon[36481]: pgmap v985: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:18:59.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:18:59 smithi045 ceph-mon[31773]: pgmap v985: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:00.072 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1592 sec 2024-03-28T18:19:01.072 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1593 sec 2024-03-28T18:19:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 949.263963 secs 2024-03-28T18:19:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 949.263963 secs 2024-03-28T18:19:01.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 949.263963 secs 2024-03-28T18:19:02.072 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1594 sec 2024-03-28T18:19:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:02 smithi071 ceph-mon[36597]: pgmap v986: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:02 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 240.687666 secs 2024-03-28T18:19:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:02 smithi151 ceph-mon[36481]: pgmap v986: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:02 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 240.687666 secs 2024-03-28T18:19:02.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:02 smithi045 ceph-mon[31773]: pgmap v986: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:02.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:02 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 240.687666 secs 2024-03-28T18:19:03.072 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1595 sec 2024-03-28T18:19:03.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:03 smithi071 ceph-mon[36597]: pgmap v987: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:03 smithi151 ceph-mon[36481]: pgmap v987: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:03.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:03 smithi045 ceph-mon[31773]: pgmap v987: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:04.072 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1596 sec 2024-03-28T18:19:05.073 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1597 sec 2024-03-28T18:19:05.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:05 smithi071 ceph-mon[36597]: pgmap v988: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:05.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:05 smithi151 ceph-mon[36481]: pgmap v988: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:05.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:05 smithi045 ceph-mon[31773]: pgmap v988: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:06.073 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1598 sec 2024-03-28T18:19:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 954.264025 secs 2024-03-28T18:19:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 954.264025 secs 2024-03-28T18:19:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 954.264025 secs 2024-03-28T18:19:07.073 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1599 sec 2024-03-28T18:19:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:07 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 245.687740 secs 2024-03-28T18:19:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:07 smithi071 ceph-mon[36597]: pgmap v989: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:07 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 245.687740 secs 2024-03-28T18:19:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:07 smithi151 ceph-mon[36481]: pgmap v989: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:07 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 245.687740 secs 2024-03-28T18:19:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:07 smithi045 ceph-mon[31773]: pgmap v989: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:08.073 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1600 sec 2024-03-28T18:19:09.073 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1601 sec 2024-03-28T18:19:09.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:09 smithi071 ceph-mon[36597]: pgmap v990: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:09.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:09 smithi151 ceph-mon[36481]: pgmap v990: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:09.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:09 smithi045 ceph-mon[31773]: pgmap v990: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:10.073 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1602 sec 2024-03-28T18:19:11.074 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1603 sec 2024-03-28T18:19:11.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 959.264116 secs 2024-03-28T18:19:11.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 959.264116 secs 2024-03-28T18:19:11.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 959.264116 secs 2024-03-28T18:19:12.074 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1604 sec 2024-03-28T18:19:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:12 smithi071 ceph-mon[36597]: pgmap v991: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:12 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 250.687827 secs 2024-03-28T18:19:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:12 smithi151 ceph-mon[36481]: pgmap v991: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:12 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 250.687827 secs 2024-03-28T18:19:12.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:12 smithi045 ceph-mon[31773]: pgmap v991: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:12.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:12 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 250.687827 secs 2024-03-28T18:19:13.074 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1605 sec 2024-03-28T18:19:13.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:13 smithi071 ceph-mon[36597]: pgmap v992: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:13.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:13 smithi151 ceph-mon[36481]: pgmap v992: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:13.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:13 smithi045 ceph-mon[31773]: pgmap v992: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:14.074 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1606 sec 2024-03-28T18:19:14.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:14 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:19:14.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:14 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:19:14.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:14 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:19:15.074 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1607 sec 2024-03-28T18:19:15.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:15 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:19:15.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:15 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:19:15.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:15 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:19:15.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:15 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:19:15.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:15 smithi045 ceph-mon[31773]: pgmap v993: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:15.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:15 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:19:15.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:15 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:19:15.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:15 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:19:15.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:15 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:19:15.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:15 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:19:16.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:15 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:19:16.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:15 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:19:16.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:15 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:19:16.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:15 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:19:16.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:15 smithi071 ceph-mon[36597]: pgmap v993: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:16.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:15 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:19:16.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:15 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:19:16.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:15 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:19:16.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:15 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:19:16.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:15 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:19:16.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:15 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:19:16.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:15 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:19:16.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:15 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:19:16.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:15 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:19:16.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:15 smithi151 ceph-mon[36481]: pgmap v993: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:16.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:15 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:19:16.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:15 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:19:16.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:15 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:19:16.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:15 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:19:16.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:15 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:19:16.074 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1608 sec 2024-03-28T18:19:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:16 smithi071 ceph-mon[36597]: pgmap v994: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:19:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:16 smithi071 ceph-mon[36597]: pgmap v995: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:19:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 964.264189 secs 2024-03-28T18:19:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:16 smithi151 ceph-mon[36481]: pgmap v994: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:19:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:16 smithi151 ceph-mon[36481]: pgmap v995: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:19:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 964.264189 secs 2024-03-28T18:19:17.074 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1609 sec 2024-03-28T18:19:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:16 smithi045 ceph-mon[31773]: pgmap v994: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:19:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:16 smithi045 ceph-mon[31773]: pgmap v995: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:19:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 964.264189 secs 2024-03-28T18:19:18.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:17 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 255.687900 secs 2024-03-28T18:19:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:17 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 255.687900 secs 2024-03-28T18:19:18.074 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1610 sec 2024-03-28T18:19:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:17 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 255.687900 secs 2024-03-28T18:19:19.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:18 smithi071 ceph-mon[36597]: pgmap v996: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:19:19.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:18 smithi151 ceph-mon[36481]: pgmap v996: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:19:19.074 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1611 sec 2024-03-28T18:19:19.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:18 smithi045 ceph-mon[31773]: pgmap v996: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:19:20.075 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1612 sec 2024-03-28T18:19:21.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:20 smithi071 ceph-mon[36597]: pgmap v997: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:19:21.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:20 smithi151 ceph-mon[36481]: pgmap v997: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:19:21.075 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1613 sec 2024-03-28T18:19:21.135 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:20 smithi045 ceph-mon[31773]: pgmap v997: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:19:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 969.264227 secs 2024-03-28T18:19:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 969.264227 secs 2024-03-28T18:19:22.075 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1614 sec 2024-03-28T18:19:22.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 969.264227 secs 2024-03-28T18:19:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:22 smithi071 ceph-mon[36597]: pgmap v998: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:19:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:22 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 260.687949 secs 2024-03-28T18:19:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:22 smithi151 ceph-mon[36481]: pgmap v998: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:19:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:22 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 260.687949 secs 2024-03-28T18:19:23.075 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1615 sec 2024-03-28T18:19:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:22 smithi045 ceph-mon[31773]: pgmap v998: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:19:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:22 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 260.687949 secs 2024-03-28T18:19:24.075 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1616 sec 2024-03-28T18:19:25.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:24 smithi071 ceph-mon[36597]: pgmap v999: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:19:25.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:24 smithi151 ceph-mon[36481]: pgmap v999: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:19:25.076 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1617 sec 2024-03-28T18:19:25.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:24 smithi045 ceph-mon[31773]: pgmap v999: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:19:26.075 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1618 sec 2024-03-28T18:19:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:26 smithi071 ceph-mon[36597]: pgmap v1000: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:19:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 974.264288 secs 2024-03-28T18:19:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:26 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 265.688010 secs 2024-03-28T18:19:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:26 smithi151 ceph-mon[36481]: pgmap v1000: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:19:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 974.264288 secs 2024-03-28T18:19:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:26 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 265.688010 secs 2024-03-28T18:19:27.075 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1619 sec 2024-03-28T18:19:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:26 smithi045 ceph-mon[31773]: pgmap v1000: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:19:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 974.264288 secs 2024-03-28T18:19:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:26 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 265.688010 secs 2024-03-28T18:19:28.076 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1620 sec 2024-03-28T18:19:29.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:28 smithi071 ceph-mon[36597]: pgmap v1001: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:29.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:28 smithi151 ceph-mon[36481]: pgmap v1001: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:29.076 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1621 sec 2024-03-28T18:19:29.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:28 smithi045 ceph-mon[31773]: pgmap v1001: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:30.076 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1622 sec 2024-03-28T18:19:31.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:30 smithi071 ceph-mon[36597]: pgmap v1002: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:31.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:30 smithi151 ceph-mon[36481]: pgmap v1002: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:31.076 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1623 sec 2024-03-28T18:19:31.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:30 smithi045 ceph-mon[31773]: pgmap v1002: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:32.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 979.264378 secs 2024-03-28T18:19:32.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:31 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 270.688094 secs 2024-03-28T18:19:32.076 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1624 sec 2024-03-28T18:19:32.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 979.264378 secs 2024-03-28T18:19:32.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:31 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 270.688094 secs 2024-03-28T18:19:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 979.264378 secs 2024-03-28T18:19:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:31 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 270.688094 secs 2024-03-28T18:19:33.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:32 smithi151 ceph-mon[36481]: pgmap v1003: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:33.076 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1625 sec 2024-03-28T18:19:33.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:32 smithi045 ceph-mon[31773]: pgmap v1003: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:33.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:32 smithi071 ceph-mon[36597]: pgmap v1003: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:34.077 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1626 sec 2024-03-28T18:19:35.077 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1627 sec 2024-03-28T18:19:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:34 smithi045 ceph-mon[31773]: pgmap v1004: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:35.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:34 smithi071 ceph-mon[36597]: pgmap v1004: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:35.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:34 smithi151 ceph-mon[36481]: pgmap v1004: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:36.077 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1628 sec 2024-03-28T18:19:37.077 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1629 sec 2024-03-28T18:19:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:36 smithi045 ceph-mon[31773]: pgmap v1005: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 984.264439 secs 2024-03-28T18:19:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:36 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 275.688155 secs 2024-03-28T18:19:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:36 smithi071 ceph-mon[36597]: pgmap v1005: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 984.264439 secs 2024-03-28T18:19:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:36 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 275.688155 secs 2024-03-28T18:19:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:36 smithi151 ceph-mon[36481]: pgmap v1005: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 984.264439 secs 2024-03-28T18:19:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:36 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 275.688155 secs 2024-03-28T18:19:38.077 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1630 sec 2024-03-28T18:19:39.077 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1631 sec 2024-03-28T18:19:39.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:38 smithi045 ceph-mon[31773]: pgmap v1006: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:39.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:38 smithi071 ceph-mon[36597]: pgmap v1006: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:39.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:38 smithi151 ceph-mon[36481]: pgmap v1006: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:40.077 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1632 sec 2024-03-28T18:19:41.077 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1633 sec 2024-03-28T18:19:41.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:40 smithi151 ceph-mon[36481]: pgmap v1007: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:41.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:40 smithi045 ceph-mon[31773]: pgmap v1007: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:41.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:40 smithi071 ceph-mon[36597]: pgmap v1007: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:42.078 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1634 sec 2024-03-28T18:19:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 989.264462 secs 2024-03-28T18:19:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:41 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 280.688217 secs 2024-03-28T18:19:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 989.264462 secs 2024-03-28T18:19:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:41 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 280.688217 secs 2024-03-28T18:19:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 989.264462 secs 2024-03-28T18:19:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:41 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 280.688217 secs 2024-03-28T18:19:43.078 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1635 sec 2024-03-28T18:19:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:42 smithi045 ceph-mon[31773]: pgmap v1008: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:43.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:42 smithi071 ceph-mon[36597]: pgmap v1008: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:43.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:42 smithi151 ceph-mon[36481]: pgmap v1008: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:44.078 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1636 sec 2024-03-28T18:19:45.078 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1637 sec 2024-03-28T18:19:45.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:44 smithi045 ceph-mon[31773]: pgmap v1009: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:45.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:44 smithi071 ceph-mon[36597]: pgmap v1009: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:45.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:44 smithi151 ceph-mon[36481]: pgmap v1009: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:46.078 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1638 sec 2024-03-28T18:19:47.078 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1639 sec 2024-03-28T18:19:47.112 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:46 smithi071 ceph-mon[36597]: pgmap v1010: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:47.112 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 994.264523 secs 2024-03-28T18:19:47.112 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:46 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 285.688232 secs 2024-03-28T18:19:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:46 smithi045 ceph-mon[31773]: pgmap v1010: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 994.264523 secs 2024-03-28T18:19:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:46 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 285.688232 secs 2024-03-28T18:19:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:46 smithi151 ceph-mon[36481]: pgmap v1010: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 994.264523 secs 2024-03-28T18:19:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:46 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 285.688232 secs 2024-03-28T18:19:48.078 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1640 sec 2024-03-28T18:19:49.079 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1641 sec 2024-03-28T18:19:49.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:48 smithi045 ceph-mon[31773]: pgmap v1011: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:49.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:48 smithi071 ceph-mon[36597]: pgmap v1011: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:49.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:48 smithi151 ceph-mon[36481]: pgmap v1011: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:50.079 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1642 sec 2024-03-28T18:19:51.079 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1643 sec 2024-03-28T18:19:51.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:50 smithi045 ceph-mon[31773]: pgmap v1012: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:51.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:50 smithi071 ceph-mon[36597]: pgmap v1012: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:51.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:50 smithi151 ceph-mon[36481]: pgmap v1012: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:52.079 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1644 sec 2024-03-28T18:19:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 999.264585 secs 2024-03-28T18:19:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:51 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 290.688290 secs 2024-03-28T18:19:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 999.264585 secs 2024-03-28T18:19:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:51 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 290.688290 secs 2024-03-28T18:19:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 999.264585 secs 2024-03-28T18:19:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:51 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 290.688290 secs 2024-03-28T18:19:53.079 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1645 sec 2024-03-28T18:19:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:52 smithi045 ceph-mon[31773]: pgmap v1013: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:53.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:52 smithi071 ceph-mon[36597]: pgmap v1013: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:53.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:52 smithi151 ceph-mon[36481]: pgmap v1013: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:54.079 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1646 sec 2024-03-28T18:19:55.080 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1647 sec 2024-03-28T18:19:55.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:54 smithi045 ceph-mon[31773]: pgmap v1014: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:55.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:54 smithi071 ceph-mon[36597]: pgmap v1014: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:55.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:54 smithi151 ceph-mon[36481]: pgmap v1014: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:56.080 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1648 sec 2024-03-28T18:19:57.080 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1649 sec 2024-03-28T18:19:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:56 smithi045 ceph-mon[31773]: pgmap v1015: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:57.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1004.264646 secs 2024-03-28T18:19:57.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:56 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 295.688351 secs 2024-03-28T18:19:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:56 smithi071 ceph-mon[36597]: pgmap v1015: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1004.264646 secs 2024-03-28T18:19:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:56 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 295.688351 secs 2024-03-28T18:19:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:56 smithi151 ceph-mon[36481]: pgmap v1015: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:19:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1004.264646 secs 2024-03-28T18:19:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:56 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 295.688351 secs 2024-03-28T18:19:58.080 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1650 sec 2024-03-28T18:19:59.080 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1651 sec 2024-03-28T18:19:59.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:19:58 smithi045 ceph-mon[31773]: pgmap v1016: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:59.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:19:58 smithi071 ceph-mon[36597]: pgmap v1016: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:19:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:19:58 smithi151 ceph-mon[36481]: pgmap v1016: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:00.080 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1652 sec 2024-03-28T18:20:01.080 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1653 sec 2024-03-28T18:20:01.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:20:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:20:01.127+0000 7f52f6e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:20:01.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:20:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:20:01.127+0000 7f52f6e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:20:01.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:20:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:20:01.127+0000 7f52f6e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:20:01.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:20:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:20:01.127+0000 7f52f6e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:20:01.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:20:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:20:01.127+0000 7f52f6e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:20:01.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:00 smithi045 ceph-mon[31773]: pgmap v1017: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:01.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:00 smithi045 ceph-mon[31773]: Health detail: HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-03-28T18:20:01.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:00 smithi045 ceph-mon[31773]: [WRN] CEPHADM_FAILED_DAEMON: 8 failed cephadm daemon(s) 2024-03-28T18:20:01.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:00 smithi045 ceph-mon[31773]: daemon mds.j on smithi045 is in error state 2024-03-28T18:20:01.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:00 smithi045 ceph-mon[31773]: daemon mds.d on smithi045 is in error state 2024-03-28T18:20:01.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:00 smithi045 ceph-mon[31773]: daemon mds.g on smithi045 is in error state 2024-03-28T18:20:01.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:00 smithi045 ceph-mon[31773]: daemon mds.a on smithi045 is in error state 2024-03-28T18:20:01.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:00 smithi045 ceph-mon[31773]: daemon mds.k on smithi071 is in error state 2024-03-28T18:20:01.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:00 smithi045 ceph-mon[31773]: daemon mds.l on smithi151 is in error state 2024-03-28T18:20:01.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:00 smithi045 ceph-mon[31773]: daemon mds.f on smithi151 is in error state 2024-03-28T18:20:01.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:00 smithi045 ceph-mon[31773]: daemon mds.i on smithi151 is in error state 2024-03-28T18:20:01.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:00 smithi045 ceph-mon[31773]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2024-03-28T18:20:01.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:00 smithi045 ceph-mon[31773]: mds.h(mds.1): 1 slow requests are blocked > 180 secs 2024-03-28T18:20:01.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:00 smithi045 ceph-mon[31773]: mds.b(mds.0): 2 slow requests are blocked > 180 secs 2024-03-28T18:20:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:00 smithi071 ceph-mon[36597]: pgmap v1017: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:00 smithi071 ceph-mon[36597]: Health detail: HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-03-28T18:20:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:00 smithi071 ceph-mon[36597]: [WRN] CEPHADM_FAILED_DAEMON: 8 failed cephadm daemon(s) 2024-03-28T18:20:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:00 smithi071 ceph-mon[36597]: daemon mds.j on smithi045 is in error state 2024-03-28T18:20:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:00 smithi071 ceph-mon[36597]: daemon mds.d on smithi045 is in error state 2024-03-28T18:20:01.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:00 smithi071 ceph-mon[36597]: daemon mds.g on smithi045 is in error state 2024-03-28T18:20:01.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:00 smithi071 ceph-mon[36597]: daemon mds.a on smithi045 is in error state 2024-03-28T18:20:01.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:00 smithi071 ceph-mon[36597]: daemon mds.k on smithi071 is in error state 2024-03-28T18:20:01.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:00 smithi071 ceph-mon[36597]: daemon mds.l on smithi151 is in error state 2024-03-28T18:20:01.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:00 smithi071 ceph-mon[36597]: daemon mds.f on smithi151 is in error state 2024-03-28T18:20:01.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:00 smithi071 ceph-mon[36597]: daemon mds.i on smithi151 is in error state 2024-03-28T18:20:01.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:00 smithi071 ceph-mon[36597]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2024-03-28T18:20:01.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:00 smithi071 ceph-mon[36597]: mds.h(mds.1): 1 slow requests are blocked > 180 secs 2024-03-28T18:20:01.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:00 smithi071 ceph-mon[36597]: mds.b(mds.0): 2 slow requests are blocked > 180 secs 2024-03-28T18:20:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:00 smithi151 ceph-mon[36481]: pgmap v1017: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:00 smithi151 ceph-mon[36481]: Health detail: HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-03-28T18:20:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:00 smithi151 ceph-mon[36481]: [WRN] CEPHADM_FAILED_DAEMON: 8 failed cephadm daemon(s) 2024-03-28T18:20:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:00 smithi151 ceph-mon[36481]: daemon mds.j on smithi045 is in error state 2024-03-28T18:20:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:00 smithi151 ceph-mon[36481]: daemon mds.d on smithi045 is in error state 2024-03-28T18:20:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:00 smithi151 ceph-mon[36481]: daemon mds.g on smithi045 is in error state 2024-03-28T18:20:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:00 smithi151 ceph-mon[36481]: daemon mds.a on smithi045 is in error state 2024-03-28T18:20:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:00 smithi151 ceph-mon[36481]: daemon mds.k on smithi071 is in error state 2024-03-28T18:20:01.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:00 smithi151 ceph-mon[36481]: daemon mds.l on smithi151 is in error state 2024-03-28T18:20:01.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:00 smithi151 ceph-mon[36481]: daemon mds.f on smithi151 is in error state 2024-03-28T18:20:01.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:00 smithi151 ceph-mon[36481]: daemon mds.i on smithi151 is in error state 2024-03-28T18:20:01.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:00 smithi151 ceph-mon[36481]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2024-03-28T18:20:01.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:00 smithi151 ceph-mon[36481]: mds.h(mds.1): 1 slow requests are blocked > 180 secs 2024-03-28T18:20:01.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:00 smithi151 ceph-mon[36481]: mds.b(mds.0): 2 slow requests are blocked > 180 secs 2024-03-28T18:20:02.080 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1654 sec 2024-03-28T18:20:02.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1009.264707 secs 2024-03-28T18:20:02.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:01 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 300.688439 secs 2024-03-28T18:20:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1009.264707 secs 2024-03-28T18:20:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:01 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 300.688439 secs 2024-03-28T18:20:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1009.264707 secs 2024-03-28T18:20:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:01 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 300.688439 secs 2024-03-28T18:20:03.081 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1655 sec 2024-03-28T18:20:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:02 smithi071 ceph-mon[36597]: pgmap v1018: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:20:03.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:02 smithi151 ceph-mon[36481]: pgmap v1018: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:20:03.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:02 smithi045 ceph-mon[31773]: pgmap v1018: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:20:04.081 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1656 sec 2024-03-28T18:20:05.081 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1657 sec 2024-03-28T18:20:05.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:04 smithi071 ceph-mon[36597]: pgmap v1019: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:05.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:04 smithi151 ceph-mon[36481]: pgmap v1019: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:05.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:04 smithi045 ceph-mon[31773]: pgmap v1019: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:06.081 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1658 sec 2024-03-28T18:20:07.081 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1659 sec 2024-03-28T18:20:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:06 smithi071 ceph-mon[36597]: pgmap v1020: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 255 B/s wr, 16 op/s 2024-03-28T18:20:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1014.264767 secs 2024-03-28T18:20:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:06 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 305.688499 secs 2024-03-28T18:20:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:06 smithi151 ceph-mon[36481]: pgmap v1020: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 255 B/s wr, 16 op/s 2024-03-28T18:20:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1014.264767 secs 2024-03-28T18:20:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:06 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 305.688499 secs 2024-03-28T18:20:07.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:06 smithi045 ceph-mon[31773]: pgmap v1020: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 255 B/s wr, 16 op/s 2024-03-28T18:20:07.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1014.264767 secs 2024-03-28T18:20:07.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:06 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 305.688499 secs 2024-03-28T18:20:08.081 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1660 sec 2024-03-28T18:20:09.081 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1661 sec 2024-03-28T18:20:09.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:08 smithi071 ceph-mon[36597]: pgmap v1021: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-28T18:20:09.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:08 smithi151 ceph-mon[36481]: pgmap v1021: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-28T18:20:09.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:08 smithi045 ceph-mon[31773]: pgmap v1021: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-28T18:20:10.081 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1662 sec 2024-03-28T18:20:11.081 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1663 sec 2024-03-28T18:20:11.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:10 smithi071 ceph-mon[36597]: pgmap v1022: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-28T18:20:11.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:10 smithi151 ceph-mon[36481]: pgmap v1022: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-28T18:20:11.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:10 smithi045 ceph-mon[31773]: pgmap v1022: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-28T18:20:12.082 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1664 sec 2024-03-28T18:20:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1019.264830 secs 2024-03-28T18:20:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:11 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 310.688560 secs 2024-03-28T18:20:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1019.264830 secs 2024-03-28T18:20:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:11 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 310.688560 secs 2024-03-28T18:20:12.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1019.264830 secs 2024-03-28T18:20:12.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:11 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 310.688560 secs 2024-03-28T18:20:13.082 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1665 sec 2024-03-28T18:20:13.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:12 smithi071 ceph-mon[36597]: pgmap v1023: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 255 B/s wr, 16 op/s 2024-03-28T18:20:13.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:12 smithi151 ceph-mon[36481]: pgmap v1023: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 255 B/s wr, 16 op/s 2024-03-28T18:20:13.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:12 smithi045 ceph-mon[31773]: pgmap v1023: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 255 B/s wr, 16 op/s 2024-03-28T18:20:14.082 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1666 sec 2024-03-28T18:20:15.082 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1667 sec 2024-03-28T18:20:15.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:14 smithi071 ceph-mon[36597]: pgmap v1024: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-28T18:20:15.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:14 smithi151 ceph-mon[36481]: pgmap v1024: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-28T18:20:15.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:14 smithi045 ceph-mon[31773]: pgmap v1024: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-28T18:20:16.082 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1668 sec 2024-03-28T18:20:16.290 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:15 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:20:16.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:15 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:20:16.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:15 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:20:17.082 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1669 sec 2024-03-28T18:20:17.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:16 smithi071 ceph-mon[36597]: pgmap v1025: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 255 B/s wr, 16 op/s 2024-03-28T18:20:17.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1024.264893 secs 2024-03-28T18:20:17.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:16 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:17.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:16 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:17.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:16 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:17.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:16 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:17.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:16 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 315.688607 secs 2024-03-28T18:20:17.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:16 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:17.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:16 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:16 smithi151 ceph-mon[36481]: pgmap v1025: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 255 B/s wr, 16 op/s 2024-03-28T18:20:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1024.264893 secs 2024-03-28T18:20:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:16 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:16 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:16 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:16 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:17.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:16 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 315.688607 secs 2024-03-28T18:20:17.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:16 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:17.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:16 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:16 smithi045 ceph-mon[31773]: pgmap v1025: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 255 B/s wr, 16 op/s 2024-03-28T18:20:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1024.264893 secs 2024-03-28T18:20:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:16 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:16 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:16 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:17.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:16 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:17.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:16 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 315.688607 secs 2024-03-28T18:20:17.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:16 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:17.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:16 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:18.083 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1670 sec 2024-03-28T18:20:19.083 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1671 sec 2024-03-28T18:20:19.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:18 smithi071 ceph-mon[36597]: pgmap v1026: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:19.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:18 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:19.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:18 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:19.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:18 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:19.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:18 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:18 smithi151 ceph-mon[36481]: pgmap v1026: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:18 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:18 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:18 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:18 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:19.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:18 smithi045 ceph-mon[31773]: pgmap v1026: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:19.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:18 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:19.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:18 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:19.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:18 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:19.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:18 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:20.083 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1672 sec 2024-03-28T18:20:20.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:19 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:19 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:19 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:19 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:19 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:19 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:19 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:19 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:19 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:19 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:19 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:19 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:19 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:19 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:19 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:19 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:19 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:19 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:19 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:19 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:19 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:19 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:19 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:19 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:19 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:19 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:19 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:19 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:19 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:19 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:19 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:19 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:19 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:19 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:19 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:19 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:19 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:19 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:19 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:19 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:19 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:19 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:19 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.418 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:19 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.418 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:19 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.418 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:19 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.418 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:19 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:20.418 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:19 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:21.083 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1673 sec 2024-03-28T18:20:21.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:20 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T18:20:21.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:20 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T18:20:21.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:20 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T18:20:21.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:20 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T18:20:21.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:20 smithi071 ceph-mon[36597]: pgmap v1027: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:21.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:20 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T18:20:21.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:20 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T18:20:21.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:20 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T18:20:21.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:20 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T18:20:21.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:20 smithi151 ceph-mon[36481]: pgmap v1027: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:20 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T18:20:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:20 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T18:20:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:20 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T18:20:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:20 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T18:20:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:20 smithi045 ceph-mon[31773]: pgmap v1027: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:22.083 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1674 sec 2024-03-28T18:20:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1029.264955 secs 2024-03-28T18:20:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:21 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:21 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:22.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:21 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 320.688696 secs 2024-03-28T18:20:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1029.264955 secs 2024-03-28T18:20:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:21 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:21 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:21 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 320.688696 secs 2024-03-28T18:20:22.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1029.264955 secs 2024-03-28T18:20:22.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:21 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:22.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:21 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:22.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:21 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 320.688696 secs 2024-03-28T18:20:23.083 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1675 sec 2024-03-28T18:20:23.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:22 smithi071 ceph-mon[36597]: pgmap v1028: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:20:23.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:22 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:22 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:22 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:22 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:22 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:22 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:22 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:22 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:22 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:20:23.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:22 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:20:23.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:22 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:23.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:22 smithi151 ceph-mon[36481]: pgmap v1028: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:20:23.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:22 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:22 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:22 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:22 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:22 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:22 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:22 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:22 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:22 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:20:23.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:22 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:20:23.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:22 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:23.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:22 smithi045 ceph-mon[31773]: pgmap v1028: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:20:23.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:22 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:22 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:22 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:22 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:22 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:22 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:22 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:22 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:20:23.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:22 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:20:23.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:22 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:20:23.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:22 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:20:24.083 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1676 sec 2024-03-28T18:20:24.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:23 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T18:20:24.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:23 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:20:24.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:23 smithi071 ceph-mon[36597]: pgmap v1029: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T18:20:24.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:23 smithi071 ceph-mon[36597]: pgmap v1030: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:20:24.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:23 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T18:20:24.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:23 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:20:24.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:23 smithi045 ceph-mon[31773]: pgmap v1029: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T18:20:24.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:23 smithi045 ceph-mon[31773]: pgmap v1030: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:20:24.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:23 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T18:20:24.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:23 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:20:24.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:23 smithi151 ceph-mon[36481]: pgmap v1029: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T18:20:24.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:23 smithi151 ceph-mon[36481]: pgmap v1030: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:20:25.083 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1677 sec 2024-03-28T18:20:26.084 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1678 sec 2024-03-28T18:20:26.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:25 smithi071 ceph-mon[36597]: pgmap v1031: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:20:26.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:25 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1034.265015 secs 2024-03-28T18:20:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:25 smithi151 ceph-mon[36481]: pgmap v1031: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:20:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:25 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1034.265015 secs 2024-03-28T18:20:26.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:25 smithi045 ceph-mon[31773]: pgmap v1031: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:20:26.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:25 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1034.265015 secs 2024-03-28T18:20:27.084 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1679 sec 2024-03-28T18:20:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:27 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 325.688783 secs 2024-03-28T18:20:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:27 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 325.688783 secs 2024-03-28T18:20:27.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:27 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 325.688783 secs 2024-03-28T18:20:28.084 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1680 sec 2024-03-28T18:20:28.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:28 smithi071 ceph-mon[36597]: pgmap v1032: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:20:28.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:28 smithi045 ceph-mon[31773]: pgmap v1032: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:20:28.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:28 smithi151 ceph-mon[36481]: pgmap v1032: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:20:29.084 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1681 sec 2024-03-28T18:20:30.084 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1682 sec 2024-03-28T18:20:30.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:30 smithi071 ceph-mon[36597]: pgmap v1033: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:20:30.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:30 smithi151 ceph-mon[36481]: pgmap v1033: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:20:30.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:30 smithi045 ceph-mon[31773]: pgmap v1033: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:20:31.084 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1683 sec 2024-03-28T18:20:31.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1039.265077 secs 2024-03-28T18:20:31.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1039.265077 secs 2024-03-28T18:20:31.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1039.265077 secs 2024-03-28T18:20:32.084 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1684 sec 2024-03-28T18:20:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:32 smithi071 ceph-mon[36597]: pgmap v1034: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:20:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:32 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 330.688826 secs 2024-03-28T18:20:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:32 smithi151 ceph-mon[36481]: pgmap v1034: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:20:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:32 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 330.688826 secs 2024-03-28T18:20:32.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:32 smithi045 ceph-mon[31773]: pgmap v1034: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:20:32.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:32 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 330.688826 secs 2024-03-28T18:20:33.084 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1685 sec 2024-03-28T18:20:34.084 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1686 sec 2024-03-28T18:20:34.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:34 smithi071 ceph-mon[36597]: pgmap v1035: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:20:34.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:34 smithi151 ceph-mon[36481]: pgmap v1035: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:20:34.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:34 smithi045 ceph-mon[31773]: pgmap v1035: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:20:35.085 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1687 sec 2024-03-28T18:20:36.085 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1688 sec 2024-03-28T18:20:36.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:36 smithi071 ceph-mon[36597]: pgmap v1036: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:36.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1044.265162 secs 2024-03-28T18:20:36.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:36 smithi151 ceph-mon[36481]: pgmap v1036: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:36.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1044.265162 secs 2024-03-28T18:20:36.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:36 smithi045 ceph-mon[31773]: pgmap v1036: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:36.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1044.265162 secs 2024-03-28T18:20:37.085 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1689 sec 2024-03-28T18:20:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:37 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 335.688919 secs 2024-03-28T18:20:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:37 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 335.688919 secs 2024-03-28T18:20:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:37 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 335.688919 secs 2024-03-28T18:20:38.085 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1690 sec 2024-03-28T18:20:38.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:38 smithi071 ceph-mon[36597]: pgmap v1037: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:20:38.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:38 smithi151 ceph-mon[36481]: pgmap v1037: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:20:38.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:38 smithi045 ceph-mon[31773]: pgmap v1037: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:20:39.085 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1691 sec 2024-03-28T18:20:40.085 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1692 sec 2024-03-28T18:20:40.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:40 smithi071 ceph-mon[36597]: pgmap v1038: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:40.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:40 smithi151 ceph-mon[36481]: pgmap v1038: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:40.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:40 smithi045 ceph-mon[31773]: pgmap v1038: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:41.085 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1693 sec 2024-03-28T18:20:41.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1049.265225 secs 2024-03-28T18:20:41.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1049.265225 secs 2024-03-28T18:20:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1049.265225 secs 2024-03-28T18:20:42.086 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1694 sec 2024-03-28T18:20:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:42 smithi151 ceph-mon[36481]: pgmap v1039: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:20:42.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:42 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 340.689003 secs 2024-03-28T18:20:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:42 smithi045 ceph-mon[31773]: pgmap v1039: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:20:42.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:42 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 340.689003 secs 2024-03-28T18:20:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:42 smithi071 ceph-mon[36597]: pgmap v1039: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:20:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:42 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 340.689003 secs 2024-03-28T18:20:43.086 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1695 sec 2024-03-28T18:20:44.086 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1696 sec 2024-03-28T18:20:44.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:44 smithi151 ceph-mon[36481]: pgmap v1040: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:44.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:44 smithi045 ceph-mon[31773]: pgmap v1040: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:44.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:44 smithi071 ceph-mon[36597]: pgmap v1040: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:45.086 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1697 sec 2024-03-28T18:20:46.086 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1698 sec 2024-03-28T18:20:46.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:46 smithi045 ceph-mon[31773]: pgmap v1041: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:46.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1054.265312 secs 2024-03-28T18:20:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:46 smithi071 ceph-mon[36597]: pgmap v1041: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1054.265312 secs 2024-03-28T18:20:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:46 smithi151 ceph-mon[36481]: pgmap v1041: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1054.265312 secs 2024-03-28T18:20:47.086 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1699 sec 2024-03-28T18:20:47.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:47 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 345.689038 secs 2024-03-28T18:20:47.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:47 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 345.689038 secs 2024-03-28T18:20:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:47 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 345.689038 secs 2024-03-28T18:20:48.086 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1700 sec 2024-03-28T18:20:48.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:48 smithi045 ceph-mon[31773]: pgmap v1042: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:20:48.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:48 smithi071 ceph-mon[36597]: pgmap v1042: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:20:48.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:48 smithi151 ceph-mon[36481]: pgmap v1042: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:20:49.087 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1701 sec 2024-03-28T18:20:50.087 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1702 sec 2024-03-28T18:20:50.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:50 smithi045 ceph-mon[31773]: pgmap v1043: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:50.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:50 smithi071 ceph-mon[36597]: pgmap v1043: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:50.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:50 smithi151 ceph-mon[36481]: pgmap v1043: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:51.087 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1703 sec 2024-03-28T18:20:51.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1059.265327 secs 2024-03-28T18:20:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1059.265327 secs 2024-03-28T18:20:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1059.265327 secs 2024-03-28T18:20:52.087 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1704 sec 2024-03-28T18:20:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:52 smithi045 ceph-mon[31773]: pgmap v1044: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:20:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:52 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 350.689128 secs 2024-03-28T18:20:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:52 smithi071 ceph-mon[36597]: pgmap v1044: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:20:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:52 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 350.689128 secs 2024-03-28T18:20:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:52 smithi151 ceph-mon[36481]: pgmap v1044: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:20:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:52 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 350.689128 secs 2024-03-28T18:20:53.087 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1705 sec 2024-03-28T18:20:54.087 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1706 sec 2024-03-28T18:20:54.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:54 smithi045 ceph-mon[31773]: pgmap v1045: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:54.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:54 smithi071 ceph-mon[36597]: pgmap v1045: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:54.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:54 smithi151 ceph-mon[36481]: pgmap v1045: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:55.087 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1707 sec 2024-03-28T18:20:56.088 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1708 sec 2024-03-28T18:20:56.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:56 smithi045 ceph-mon[31773]: pgmap v1046: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:56.418 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1064.265387 secs 2024-03-28T18:20:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:56 smithi071 ceph-mon[36597]: pgmap v1046: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1064.265387 secs 2024-03-28T18:20:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:56 smithi151 ceph-mon[36481]: pgmap v1046: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:20:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1064.265387 secs 2024-03-28T18:20:57.088 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1709 sec 2024-03-28T18:20:57.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:57 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 355.689188 secs 2024-03-28T18:20:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:57 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 355.689188 secs 2024-03-28T18:20:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:57 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 355.689188 secs 2024-03-28T18:20:58.088 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1710 sec 2024-03-28T18:20:58.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:20:58 smithi045 ceph-mon[31773]: pgmap v1047: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:20:58.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:20:58 smithi071 ceph-mon[36597]: pgmap v1047: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:20:58.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:20:58 smithi151 ceph-mon[36481]: pgmap v1047: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:20:59.088 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1711 sec 2024-03-28T18:21:00.088 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1712 sec 2024-03-28T18:21:00.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:00 smithi045 ceph-mon[31773]: pgmap v1048: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:00.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:00 smithi071 ceph-mon[36597]: pgmap v1048: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:00.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:00 smithi151 ceph-mon[36481]: pgmap v1048: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:01.088 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1713 sec 2024-03-28T18:21:01.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1069.265455 secs 2024-03-28T18:21:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1069.265455 secs 2024-03-28T18:21:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1069.265455 secs 2024-03-28T18:21:02.089 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1714 sec 2024-03-28T18:21:02.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:02 smithi045 ceph-mon[31773]: pgmap v1049: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:21:02.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:02 smithi045 ceph-mon[31773]: 1 slow requests, 1 included below; oldest blocked for > 360.689279 secs 2024-03-28T18:21:02.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:02 smithi045 ceph-mon[31773]: slow request 360.689279 seconds old, received at 2024-03-28T18:15:00.947128+0000: client_request(client.26687:1 getattr pAsLsXsFs #0x1 2024-03-28T18:15:00.943510+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:21:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:02 smithi071 ceph-mon[36597]: pgmap v1049: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:21:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:02 smithi071 ceph-mon[36597]: 1 slow requests, 1 included below; oldest blocked for > 360.689279 secs 2024-03-28T18:21:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:02 smithi071 ceph-mon[36597]: slow request 360.689279 seconds old, received at 2024-03-28T18:15:00.947128+0000: client_request(client.26687:1 getattr pAsLsXsFs #0x1 2024-03-28T18:15:00.943510+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:21:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:02 smithi151 ceph-mon[36481]: pgmap v1049: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:21:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:02 smithi151 ceph-mon[36481]: 1 slow requests, 1 included below; oldest blocked for > 360.689279 secs 2024-03-28T18:21:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:02 smithi151 ceph-mon[36481]: slow request 360.689279 seconds old, received at 2024-03-28T18:15:00.947128+0000: client_request(client.26687:1 getattr pAsLsXsFs #0x1 2024-03-28T18:15:00.943510+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:21:03.089 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1715 sec 2024-03-28T18:21:04.089 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1716 sec 2024-03-28T18:21:04.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:04 smithi045 ceph-mon[31773]: pgmap v1050: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:04.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:04 smithi071 ceph-mon[36597]: pgmap v1050: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:04.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:04 smithi151 ceph-mon[36481]: pgmap v1050: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:05.089 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1717 sec 2024-03-28T18:21:06.089 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1718 sec 2024-03-28T18:21:06.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:06 smithi045 ceph-mon[31773]: pgmap v1051: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1074.265538 secs 2024-03-28T18:21:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:06 smithi071 ceph-mon[36597]: pgmap v1051: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1074.265538 secs 2024-03-28T18:21:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:06 smithi151 ceph-mon[36481]: pgmap v1051: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1074.265538 secs 2024-03-28T18:21:07.089 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1719 sec 2024-03-28T18:21:07.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:07 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 365.689367 secs 2024-03-28T18:21:07.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:07 smithi045 ceph-mon[31773]: pgmap v1052: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:21:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:07 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 365.689367 secs 2024-03-28T18:21:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:07 smithi071 ceph-mon[36597]: pgmap v1052: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:21:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:07 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 365.689367 secs 2024-03-28T18:21:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:07 smithi151 ceph-mon[36481]: pgmap v1052: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:21:08.089 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1720 sec 2024-03-28T18:21:09.090 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1721 sec 2024-03-28T18:21:10.090 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1722 sec 2024-03-28T18:21:10.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:09 smithi071 ceph-mon[36597]: pgmap v1053: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:21:10.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:09 smithi151 ceph-mon[36481]: pgmap v1053: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:21:10.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:09 smithi045 ceph-mon[31773]: pgmap v1053: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:21:11.090 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1723 sec 2024-03-28T18:21:12.090 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1724 sec 2024-03-28T18:21:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1079.265599 secs 2024-03-28T18:21:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:11 smithi071 ceph-mon[36597]: pgmap v1054: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:21:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:11 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 370.689428 secs 2024-03-28T18:21:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1079.265599 secs 2024-03-28T18:21:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:11 smithi151 ceph-mon[36481]: pgmap v1054: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:21:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:11 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 370.689428 secs 2024-03-28T18:21:12.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1079.265599 secs 2024-03-28T18:21:12.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:11 smithi045 ceph-mon[31773]: pgmap v1054: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:21:12.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:11 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 370.689428 secs 2024-03-28T18:21:13.090 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1725 sec 2024-03-28T18:21:14.090 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1726 sec 2024-03-28T18:21:14.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:13 smithi071 ceph-mon[36597]: pgmap v1055: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:21:14.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:13 smithi151 ceph-mon[36481]: pgmap v1055: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:21:14.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:13 smithi045 ceph-mon[31773]: pgmap v1055: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:21:15.090 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1727 sec 2024-03-28T18:21:16.091 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1728 sec 2024-03-28T18:21:16.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:15 smithi071 ceph-mon[36597]: pgmap v1056: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:21:16.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:15 smithi151 ceph-mon[36481]: pgmap v1056: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:21:16.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:15 smithi045 ceph-mon[31773]: pgmap v1056: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:21:17.091 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1729 sec 2024-03-28T18:21:17.212 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1084.265660 secs 2024-03-28T18:21:17.212 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:16 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 375.689515 secs 2024-03-28T18:21:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1084.265660 secs 2024-03-28T18:21:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:16 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 375.689515 secs 2024-03-28T18:21:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1084.265660 secs 2024-03-28T18:21:17.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:16 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 375.689515 secs 2024-03-28T18:21:18.091 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1730 sec 2024-03-28T18:21:18.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:17 smithi071 ceph-mon[36597]: pgmap v1057: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:21:18.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:17 smithi151 ceph-mon[36481]: pgmap v1057: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:21:18.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:17 smithi045 ceph-mon[31773]: pgmap v1057: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:21:19.091 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1731 sec 2024-03-28T18:21:20.092 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1732 sec 2024-03-28T18:21:20.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:19 smithi071 ceph-mon[36597]: pgmap v1058: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:20.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:19 smithi151 ceph-mon[36481]: pgmap v1058: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:20.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:19 smithi045 ceph-mon[31773]: pgmap v1058: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:21.091 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1733 sec 2024-03-28T18:21:22.092 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1734 sec 2024-03-28T18:21:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1089.265724 secs 2024-03-28T18:21:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:21 smithi071 ceph-mon[36597]: pgmap v1059: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:21:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:21 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 380.689554 secs 2024-03-28T18:21:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1089.265724 secs 2024-03-28T18:21:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:21 smithi151 ceph-mon[36481]: pgmap v1059: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:21:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:21 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 380.689554 secs 2024-03-28T18:21:22.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1089.265724 secs 2024-03-28T18:21:22.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:21 smithi045 ceph-mon[31773]: pgmap v1059: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:21:22.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:21 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 380.689554 secs 2024-03-28T18:21:23.092 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1735 sec 2024-03-28T18:21:24.092 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1736 sec 2024-03-28T18:21:24.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:23 smithi071 ceph-mon[36597]: pgmap v1060: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:24.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:23 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:21:24.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:23 smithi151 ceph-mon[36481]: pgmap v1060: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:24.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:23 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:21:24.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:23 smithi045 ceph-mon[31773]: pgmap v1060: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:24.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:23 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:21:25.092 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1737 sec 2024-03-28T18:21:25.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:25 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:21:25.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:25 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:21:25.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:25 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:21:25.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:24 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:21:25.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:24 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:21:25.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:24 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:21:25.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:24 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:21:25.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:24 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:21:25.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:25 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:21:26.093 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1738 sec 2024-03-28T18:21:26.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:26 smithi071 ceph-mon[36597]: pgmap v1061: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:26.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:26 smithi071 ceph-mon[36597]: pgmap v1062: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:21:26.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1094.265812 secs 2024-03-28T18:21:26.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:26 smithi045 ceph-mon[31773]: pgmap v1061: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:26.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:26 smithi045 ceph-mon[31773]: pgmap v1062: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:21:26.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1094.265812 secs 2024-03-28T18:21:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:26 smithi151 ceph-mon[36481]: pgmap v1061: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:26 smithi151 ceph-mon[36481]: pgmap v1062: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:21:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1094.265812 secs 2024-03-28T18:21:27.092 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1739 sec 2024-03-28T18:21:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:27 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 385.689640 secs 2024-03-28T18:21:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:27 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 385.689640 secs 2024-03-28T18:21:27.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:27 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 385.689640 secs 2024-03-28T18:21:28.092 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1740 sec 2024-03-28T18:21:28.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:28 smithi071 ceph-mon[36597]: pgmap v1063: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:21:28.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:28 smithi045 ceph-mon[31773]: pgmap v1063: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:21:28.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:28 smithi151 ceph-mon[36481]: pgmap v1063: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:21:29.093 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1741 sec 2024-03-28T18:21:30.093 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1742 sec 2024-03-28T18:21:30.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:30 smithi071 ceph-mon[36597]: pgmap v1064: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:21:30.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:30 smithi151 ceph-mon[36481]: pgmap v1064: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:21:30.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:30 smithi045 ceph-mon[31773]: pgmap v1064: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:21:31.093 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1743 sec 2024-03-28T18:21:31.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1099.265901 secs 2024-03-28T18:21:31.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1099.265901 secs 2024-03-28T18:21:31.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1099.265901 secs 2024-03-28T18:21:32.093 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1744 sec 2024-03-28T18:21:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:32 smithi071 ceph-mon[36597]: pgmap v1065: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:21:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:32 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 390.689727 secs 2024-03-28T18:21:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:32 smithi151 ceph-mon[36481]: pgmap v1065: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:21:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:32 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 390.689727 secs 2024-03-28T18:21:32.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:32 smithi045 ceph-mon[31773]: pgmap v1065: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:21:32.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:32 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 390.689727 secs 2024-03-28T18:21:33.093 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1745 sec 2024-03-28T18:21:34.093 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1746 sec 2024-03-28T18:21:34.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:34 smithi071 ceph-mon[36597]: pgmap v1066: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:21:34.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:34 smithi151 ceph-mon[36481]: pgmap v1066: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:21:34.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:34 smithi045 ceph-mon[31773]: pgmap v1066: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:21:35.094 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1747 sec 2024-03-28T18:21:36.094 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1748 sec 2024-03-28T18:21:36.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:36 smithi071 ceph-mon[36597]: pgmap v1067: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:21:36.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1104.265961 secs 2024-03-28T18:21:36.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:36 smithi151 ceph-mon[36481]: pgmap v1067: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:21:36.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1104.265961 secs 2024-03-28T18:21:36.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:36 smithi045 ceph-mon[31773]: pgmap v1067: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:21:36.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1104.265961 secs 2024-03-28T18:21:37.094 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1749 sec 2024-03-28T18:21:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:37 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 395.689817 secs 2024-03-28T18:21:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:37 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 395.689817 secs 2024-03-28T18:21:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:37 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 395.689817 secs 2024-03-28T18:21:38.094 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1750 sec 2024-03-28T18:21:38.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:38 smithi071 ceph-mon[36597]: pgmap v1068: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:21:38.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:38 smithi151 ceph-mon[36481]: pgmap v1068: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:21:38.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:38 smithi045 ceph-mon[31773]: pgmap v1068: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:21:39.094 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1751 sec 2024-03-28T18:21:40.094 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1752 sec 2024-03-28T18:21:40.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:40 smithi071 ceph-mon[36597]: pgmap v1069: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:40.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:40 smithi151 ceph-mon[36481]: pgmap v1069: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:40.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:40 smithi045 ceph-mon[31773]: pgmap v1069: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:41.094 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1753 sec 2024-03-28T18:21:41.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1109.266039 secs 2024-03-28T18:21:41.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1109.266039 secs 2024-03-28T18:21:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1109.266039 secs 2024-03-28T18:21:42.095 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1754 sec 2024-03-28T18:21:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:42 smithi151 ceph-mon[36481]: pgmap v1070: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:21:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:42 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 400.689918 secs 2024-03-28T18:21:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:42 smithi045 ceph-mon[31773]: pgmap v1070: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:21:42.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:42 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 400.689918 secs 2024-03-28T18:21:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:42 smithi071 ceph-mon[36597]: pgmap v1070: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:21:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:42 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 400.689918 secs 2024-03-28T18:21:43.095 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1755 sec 2024-03-28T18:21:44.095 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1756 sec 2024-03-28T18:21:44.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:44 smithi045 ceph-mon[31773]: pgmap v1071: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:44.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:44 smithi071 ceph-mon[36597]: pgmap v1071: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:44 smithi151 ceph-mon[36481]: pgmap v1071: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:45.095 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1757 sec 2024-03-28T18:21:46.095 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1758 sec 2024-03-28T18:21:46.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:46 smithi045 ceph-mon[31773]: pgmap v1072: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:46.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1114.266126 secs 2024-03-28T18:21:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:46 smithi071 ceph-mon[36597]: pgmap v1072: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1114.266126 secs 2024-03-28T18:21:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:46 smithi151 ceph-mon[36481]: pgmap v1072: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1114.266126 secs 2024-03-28T18:21:47.095 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1759 sec 2024-03-28T18:21:47.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:47 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 405.689963 secs 2024-03-28T18:21:47.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:47 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 405.689963 secs 2024-03-28T18:21:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:47 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 405.689963 secs 2024-03-28T18:21:48.095 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1760 sec 2024-03-28T18:21:48.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:48 smithi045 ceph-mon[31773]: pgmap v1073: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:21:48.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:48 smithi071 ceph-mon[36597]: pgmap v1073: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:21:48.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:48 smithi151 ceph-mon[36481]: pgmap v1073: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:21:49.095 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1761 sec 2024-03-28T18:21:50.096 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1762 sec 2024-03-28T18:21:50.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:50 smithi045 ceph-mon[31773]: pgmap v1074: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:50.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:50 smithi071 ceph-mon[36597]: pgmap v1074: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:50.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:50 smithi151 ceph-mon[36481]: pgmap v1074: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:51.096 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1763 sec 2024-03-28T18:21:51.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1119.266214 secs 2024-03-28T18:21:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1119.266214 secs 2024-03-28T18:21:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1119.266214 secs 2024-03-28T18:21:52.096 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1764 sec 2024-03-28T18:21:52.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:52 smithi045 ceph-mon[31773]: pgmap v1075: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:21:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:52 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 410.689978 secs 2024-03-28T18:21:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:52 smithi071 ceph-mon[36597]: pgmap v1075: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:21:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:52 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 410.689978 secs 2024-03-28T18:21:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:52 smithi151 ceph-mon[36481]: pgmap v1075: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:21:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:52 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 410.689978 secs 2024-03-28T18:21:53.096 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1765 sec 2024-03-28T18:21:54.096 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1766 sec 2024-03-28T18:21:54.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:54 smithi045 ceph-mon[31773]: pgmap v1076: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:54.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:54 smithi071 ceph-mon[36597]: pgmap v1076: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:54.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:54 smithi151 ceph-mon[36481]: pgmap v1076: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:55.096 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1767 sec 2024-03-28T18:21:56.096 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1768 sec 2024-03-28T18:21:56.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:56 smithi045 ceph-mon[31773]: pgmap v1077: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:56.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1124.266303 secs 2024-03-28T18:21:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:56 smithi071 ceph-mon[36597]: pgmap v1077: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1124.266303 secs 2024-03-28T18:21:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:56 smithi151 ceph-mon[36481]: pgmap v1077: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:21:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1124.266303 secs 2024-03-28T18:21:57.097 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1769 sec 2024-03-28T18:21:57.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:57 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 415.690041 secs 2024-03-28T18:21:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:57 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 415.690041 secs 2024-03-28T18:21:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:57 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 415.690041 secs 2024-03-28T18:21:58.097 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1770 sec 2024-03-28T18:21:58.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:21:58 smithi045 ceph-mon[31773]: pgmap v1078: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:21:58.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:21:58 smithi071 ceph-mon[36597]: pgmap v1078: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:21:58.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:21:58 smithi151 ceph-mon[36481]: pgmap v1078: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:21:59.097 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1771 sec 2024-03-28T18:22:00.097 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1772 sec 2024-03-28T18:22:00.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:00 smithi045 ceph-mon[31773]: pgmap v1079: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:00.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:00 smithi071 ceph-mon[36597]: pgmap v1079: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:00.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:00 smithi151 ceph-mon[36481]: pgmap v1079: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:01.097 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1773 sec 2024-03-28T18:22:01.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1129.266393 secs 2024-03-28T18:22:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1129.266393 secs 2024-03-28T18:22:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1129.266393 secs 2024-03-28T18:22:02.097 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1774 sec 2024-03-28T18:22:02.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:02 smithi045 ceph-mon[31773]: pgmap v1080: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:02.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:02 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 420.690104 secs 2024-03-28T18:22:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:02 smithi071 ceph-mon[36597]: pgmap v1080: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:02 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 420.690104 secs 2024-03-28T18:22:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:02 smithi151 ceph-mon[36481]: pgmap v1080: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:02 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 420.690104 secs 2024-03-28T18:22:03.097 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1775 sec 2024-03-28T18:22:04.098 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1776 sec 2024-03-28T18:22:04.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:04 smithi045 ceph-mon[31773]: pgmap v1081: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:04.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:04 smithi071 ceph-mon[36597]: pgmap v1081: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:04.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:04 smithi151 ceph-mon[36481]: pgmap v1081: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:05.098 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1777 sec 2024-03-28T18:22:06.098 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1778 sec 2024-03-28T18:22:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:06 smithi045 ceph-mon[31773]: pgmap v1082: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:06.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1134.266479 secs 2024-03-28T18:22:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:06 smithi071 ceph-mon[36597]: pgmap v1082: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1134.266479 secs 2024-03-28T18:22:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:06 smithi151 ceph-mon[36481]: pgmap v1082: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:06.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1134.266479 secs 2024-03-28T18:22:07.098 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1779 sec 2024-03-28T18:22:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:07 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 425.690201 secs 2024-03-28T18:22:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:07 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 425.690201 secs 2024-03-28T18:22:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:07 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 425.690201 secs 2024-03-28T18:22:08.098 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1780 sec 2024-03-28T18:22:08.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:08 smithi071 ceph-mon[36597]: pgmap v1083: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:08.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:08 smithi151 ceph-mon[36481]: pgmap v1083: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:08.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:08 smithi045 ceph-mon[31773]: pgmap v1083: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:09.098 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1781 sec 2024-03-28T18:22:09.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:09 smithi071 ceph-mon[36597]: pgmap v1084: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:09.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:09 smithi151 ceph-mon[36481]: pgmap v1084: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:09.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:09 smithi045 ceph-mon[31773]: pgmap v1084: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:10.098 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1782 sec 2024-03-28T18:22:11.099 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1783 sec 2024-03-28T18:22:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:11 smithi045 ceph-mon[31773]: pgmap v1085: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:11.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1139.266541 secs 2024-03-28T18:22:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:11 smithi071 ceph-mon[36597]: pgmap v1085: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1139.266541 secs 2024-03-28T18:22:12.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:11 smithi151 ceph-mon[36481]: pgmap v1085: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:12.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1139.266541 secs 2024-03-28T18:22:12.099 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1784 sec 2024-03-28T18:22:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:12 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 430.690291 secs 2024-03-28T18:22:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:12 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 430.690291 secs 2024-03-28T18:22:13.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:12 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 430.690291 secs 2024-03-28T18:22:13.099 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1785 sec 2024-03-28T18:22:13.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:13 smithi045 ceph-mon[31773]: pgmap v1086: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:14.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:13 smithi071 ceph-mon[36597]: pgmap v1086: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:13 smithi151 ceph-mon[36481]: pgmap v1086: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:14.099 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1786 sec 2024-03-28T18:22:15.099 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1787 sec 2024-03-28T18:22:15.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:15 smithi045 ceph-mon[31773]: pgmap v1087: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:16.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:15 smithi071 ceph-mon[36597]: pgmap v1087: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:16.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:15 smithi151 ceph-mon[36481]: pgmap v1087: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:16.099 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1788 sec 2024-03-28T18:22:16.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1144.266628 secs 2024-03-28T18:22:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1144.266628 secs 2024-03-28T18:22:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1144.266628 secs 2024-03-28T18:22:17.099 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1789 sec 2024-03-28T18:22:17.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:17 smithi045 ceph-mon[31773]: pgmap v1088: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:17.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:17 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 435.690352 secs 2024-03-28T18:22:17.961 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:17 smithi071 ceph-mon[36597]: pgmap v1088: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:17.961 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:17 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 435.690352 secs 2024-03-28T18:22:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:17 smithi151 ceph-mon[36481]: pgmap v1088: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:17 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 435.690352 secs 2024-03-28T18:22:18.100 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1790 sec 2024-03-28T18:22:19.100 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1791 sec 2024-03-28T18:22:19.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:19 smithi045 ceph-mon[31773]: pgmap v1089: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:20.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:19 smithi071 ceph-mon[36597]: pgmap v1089: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:20.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:19 smithi151 ceph-mon[36481]: pgmap v1089: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:20.100 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1792 sec 2024-03-28T18:22:21.100 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1793 sec 2024-03-28T18:22:21.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:21 smithi045 ceph-mon[31773]: pgmap v1090: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:21.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1149.266692 secs 2024-03-28T18:22:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:21 smithi071 ceph-mon[36597]: pgmap v1090: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1149.266692 secs 2024-03-28T18:22:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:21 smithi151 ceph-mon[36481]: pgmap v1090: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1149.266692 secs 2024-03-28T18:22:22.100 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1794 sec 2024-03-28T18:22:22.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:22 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 440.690415 secs 2024-03-28T18:22:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:22 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 440.690415 secs 2024-03-28T18:22:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:22 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 440.690415 secs 2024-03-28T18:22:23.100 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1795 sec 2024-03-28T18:22:23.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:23 smithi045 ceph-mon[31773]: pgmap v1091: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:24.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:23 smithi071 ceph-mon[36597]: pgmap v1091: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:24.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:23 smithi151 ceph-mon[36481]: pgmap v1091: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:24.100 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1796 sec 2024-03-28T18:22:25.101 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1797 sec 2024-03-28T18:22:26.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:25 smithi071 ceph-mon[36597]: pgmap v1092: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:26.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:25 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:22:26.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:25 smithi151 ceph-mon[36481]: pgmap v1092: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:26.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:25 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:22:26.101 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1798 sec 2024-03-28T18:22:26.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:25 smithi045 ceph-mon[31773]: pgmap v1092: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:26.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:25 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:22:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1154.266755 secs 2024-03-28T18:22:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:26 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:22:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:26 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:22:27.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:26 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:22:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1154.266755 secs 2024-03-28T18:22:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:26 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:22:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:26 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:22:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:26 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:22:27.101 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1799 sec 2024-03-28T18:22:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1154.266755 secs 2024-03-28T18:22:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:26 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:22:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:26 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:22:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:26 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:22:28.016 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:27 smithi045 ceph-mon[31773]: pgmap v1093: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:22:28.016 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:27 smithi045 ceph-mon[31773]: pgmap v1094: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:22:28.016 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:27 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 445.690477 secs 2024-03-28T18:22:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:27 smithi071 ceph-mon[36597]: pgmap v1093: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:22:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:27 smithi071 ceph-mon[36597]: pgmap v1094: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:22:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:27 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 445.690477 secs 2024-03-28T18:22:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:27 smithi151 ceph-mon[36481]: pgmap v1093: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:22:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:27 smithi151 ceph-mon[36481]: pgmap v1094: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:22:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:27 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 445.690477 secs 2024-03-28T18:22:28.101 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1800 sec 2024-03-28T18:22:29.101 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1801 sec 2024-03-28T18:22:30.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:29 smithi071 ceph-mon[36597]: pgmap v1095: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:22:30.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:29 smithi151 ceph-mon[36481]: pgmap v1095: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:22:30.101 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1802 sec 2024-03-28T18:22:30.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:29 smithi045 ceph-mon[31773]: pgmap v1095: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:22:31.102 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1803 sec 2024-03-28T18:22:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:31 smithi071 ceph-mon[36597]: pgmap v1096: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:22:32.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1159.266817 secs 2024-03-28T18:22:32.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:31 smithi151 ceph-mon[36481]: pgmap v1096: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:22:32.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1159.266817 secs 2024-03-28T18:22:32.102 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1804 sec 2024-03-28T18:22:32.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:31 smithi045 ceph-mon[31773]: pgmap v1096: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:22:32.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1159.266817 secs 2024-03-28T18:22:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:32 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 450.690566 secs 2024-03-28T18:22:33.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:32 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 450.690566 secs 2024-03-28T18:22:33.102 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1805 sec 2024-03-28T18:22:33.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:32 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 450.690566 secs 2024-03-28T18:22:34.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:33 smithi071 ceph-mon[36597]: pgmap v1097: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:22:34.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:33 smithi151 ceph-mon[36481]: pgmap v1097: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:22:34.102 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1806 sec 2024-03-28T18:22:34.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:33 smithi045 ceph-mon[31773]: pgmap v1097: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:22:35.102 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1807 sec 2024-03-28T18:22:36.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:35 smithi071 ceph-mon[36597]: pgmap v1098: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:22:36.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:35 smithi151 ceph-mon[36481]: pgmap v1098: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:22:36.102 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1808 sec 2024-03-28T18:22:36.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:35 smithi045 ceph-mon[31773]: pgmap v1098: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:22:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1164.266878 secs 2024-03-28T18:22:37.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1164.266878 secs 2024-03-28T18:22:37.102 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1809 sec 2024-03-28T18:22:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1164.266878 secs 2024-03-28T18:22:38.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:37 smithi071 ceph-mon[36597]: pgmap v1099: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:22:38.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:37 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 455.690648 secs 2024-03-28T18:22:38.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:37 smithi151 ceph-mon[36481]: pgmap v1099: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:22:38.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:37 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 455.690648 secs 2024-03-28T18:22:38.102 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1810 sec 2024-03-28T18:22:38.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:37 smithi045 ceph-mon[31773]: pgmap v1099: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:22:38.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:37 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 455.690648 secs 2024-03-28T18:22:39.102 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1811 sec 2024-03-28T18:22:40.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:39 smithi071 ceph-mon[36597]: pgmap v1100: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:40.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:39 smithi151 ceph-mon[36481]: pgmap v1100: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:40.103 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1812 sec 2024-03-28T18:22:40.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:39 smithi045 ceph-mon[31773]: pgmap v1100: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:41.103 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1813 sec 2024-03-28T18:22:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:41 smithi071 ceph-mon[36597]: pgmap v1101: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:42.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1169.266939 secs 2024-03-28T18:22:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:41 smithi151 ceph-mon[36481]: pgmap v1101: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1169.266939 secs 2024-03-28T18:22:42.103 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1814 sec 2024-03-28T18:22:42.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:41 smithi045 ceph-mon[31773]: pgmap v1101: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:42.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1169.266939 secs 2024-03-28T18:22:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:42 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 460.690736 secs 2024-03-28T18:22:43.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:42 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 460.690736 secs 2024-03-28T18:22:43.103 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1815 sec 2024-03-28T18:22:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:42 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 460.690736 secs 2024-03-28T18:22:44.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:43 smithi071 ceph-mon[36597]: pgmap v1102: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:44.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:43 smithi151 ceph-mon[36481]: pgmap v1102: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:44.103 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1816 sec 2024-03-28T18:22:44.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:43 smithi045 ceph-mon[31773]: pgmap v1102: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:45.103 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1817 sec 2024-03-28T18:22:46.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:45 smithi071 ceph-mon[36597]: pgmap v1103: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:46.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:45 smithi151 ceph-mon[36481]: pgmap v1103: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:46.104 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1818 sec 2024-03-28T18:22:46.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:45 smithi045 ceph-mon[31773]: pgmap v1103: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1174.266999 secs 2024-03-28T18:22:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1174.266999 secs 2024-03-28T18:22:47.104 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1819 sec 2024-03-28T18:22:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1174.266999 secs 2024-03-28T18:22:48.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:47 smithi071 ceph-mon[36597]: pgmap v1104: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:48.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:47 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 465.690823 secs 2024-03-28T18:22:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:47 smithi151 ceph-mon[36481]: pgmap v1104: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:47 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 465.690823 secs 2024-03-28T18:22:48.104 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1820 sec 2024-03-28T18:22:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:47 smithi045 ceph-mon[31773]: pgmap v1104: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:47 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 465.690823 secs 2024-03-28T18:22:49.104 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1821 sec 2024-03-28T18:22:50.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:49 smithi071 ceph-mon[36597]: pgmap v1105: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:50.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:49 smithi151 ceph-mon[36481]: pgmap v1105: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:50.104 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1822 sec 2024-03-28T18:22:50.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:49 smithi045 ceph-mon[31773]: pgmap v1105: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:51.104 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1823 sec 2024-03-28T18:22:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:51 smithi071 ceph-mon[36597]: pgmap v1106: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1179.267087 secs 2024-03-28T18:22:52.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:51 smithi151 ceph-mon[36481]: pgmap v1106: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:52.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1179.267087 secs 2024-03-28T18:22:52.104 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1824 sec 2024-03-28T18:22:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:51 smithi045 ceph-mon[31773]: pgmap v1106: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1179.267087 secs 2024-03-28T18:22:53.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:52 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 470.690877 secs 2024-03-28T18:22:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:52 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 470.690877 secs 2024-03-28T18:22:53.105 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1825 sec 2024-03-28T18:22:53.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:52 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 470.690877 secs 2024-03-28T18:22:54.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:53 smithi071 ceph-mon[36597]: pgmap v1107: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:54.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:53 smithi151 ceph-mon[36481]: pgmap v1107: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:54.105 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1826 sec 2024-03-28T18:22:54.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:53 smithi045 ceph-mon[31773]: pgmap v1107: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:55.105 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1827 sec 2024-03-28T18:22:56.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:55 smithi071 ceph-mon[36597]: pgmap v1108: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:56.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:55 smithi151 ceph-mon[36481]: pgmap v1108: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:56.105 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1828 sec 2024-03-28T18:22:56.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:55 smithi045 ceph-mon[31773]: pgmap v1108: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:22:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1184.267176 secs 2024-03-28T18:22:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1184.267176 secs 2024-03-28T18:22:57.105 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1829 sec 2024-03-28T18:22:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1184.267176 secs 2024-03-28T18:22:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:57 smithi071 ceph-mon[36597]: pgmap v1109: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:57 smithi071 ceph-mon[36597]: 1 slow requests, 0 included below; oldest blocked for > 475.690906 secs 2024-03-28T18:22:58.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:57 smithi045 ceph-mon[31773]: pgmap v1109: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:58.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:57 smithi045 ceph-mon[31773]: 1 slow requests, 0 included below; oldest blocked for > 475.690906 secs 2024-03-28T18:22:58.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:57 smithi151 ceph-mon[36481]: pgmap v1109: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:22:58.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:57 smithi151 ceph-mon[36481]: 1 slow requests, 0 included below; oldest blocked for > 475.690906 secs 2024-03-28T18:22:58.105 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1830 sec 2024-03-28T18:22:59.105 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1831 sec 2024-03-28T18:23:00.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:22:59 smithi071 ceph-mon[36597]: pgmap v1110: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:00.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:22:59 smithi151 ceph-mon[36481]: pgmap v1110: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:00.105 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1832 sec 2024-03-28T18:23:00.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:22:59 smithi045 ceph-mon[31773]: pgmap v1110: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:01.106 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1833 sec 2024-03-28T18:23:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:01 smithi071 ceph-mon[36597]: pgmap v1111: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1189.267238 secs 2024-03-28T18:23:02.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:01 smithi151 ceph-mon[36481]: pgmap v1111: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:02.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1189.267238 secs 2024-03-28T18:23:02.106 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1834 sec 2024-03-28T18:23:02.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:01 smithi045 ceph-mon[31773]: pgmap v1111: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:02.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1189.267238 secs 2024-03-28T18:23:03.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:02 smithi151 ceph-mon[36481]: 2 slow requests, 1 included below; oldest blocked for > 480.690968 secs 2024-03-28T18:23:03.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:02 smithi151 ceph-mon[36481]: slow request 180.503324 seconds old, received at 2024-03-28T18:20:01.134771+0000: client_request(client.16398:1 getattr pAsLsXsFs #0x1 2024-03-28T18:20:01.130880+0000 caller_uid=0, caller_gid=0{}) currently joining batch getattr 2024-03-28T18:23:03.106 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1835 sec 2024-03-28T18:23:03.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:02 smithi045 ceph-mon[31773]: 2 slow requests, 1 included below; oldest blocked for > 480.690968 secs 2024-03-28T18:23:03.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:02 smithi045 ceph-mon[31773]: slow request 180.503324 seconds old, received at 2024-03-28T18:20:01.134771+0000: client_request(client.16398:1 getattr pAsLsXsFs #0x1 2024-03-28T18:20:01.130880+0000 caller_uid=0, caller_gid=0{}) currently joining batch getattr 2024-03-28T18:23:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:02 smithi071 ceph-mon[36597]: 2 slow requests, 1 included below; oldest blocked for > 480.690968 secs 2024-03-28T18:23:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:02 smithi071 ceph-mon[36597]: slow request 180.503324 seconds old, received at 2024-03-28T18:20:01.134771+0000: client_request(client.16398:1 getattr pAsLsXsFs #0x1 2024-03-28T18:20:01.130880+0000 caller_uid=0, caller_gid=0{}) currently joining batch getattr 2024-03-28T18:23:04.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:03 smithi151 ceph-mon[36481]: pgmap v1112: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:04.106 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1836 sec 2024-03-28T18:23:04.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:03 smithi045 ceph-mon[31773]: pgmap v1112: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:04.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:03 smithi071 ceph-mon[36597]: pgmap v1112: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:05.106 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1837 sec 2024-03-28T18:23:06.106 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1838 sec 2024-03-28T18:23:06.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:05 smithi045 ceph-mon[31773]: pgmap v1113: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:06.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:05 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] up:active 2024-03-28T18:23:06.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:05 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T18:23:06.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:05 smithi071 ceph-mon[36597]: pgmap v1113: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:06.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:05 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] up:active 2024-03-28T18:23:06.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:05 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T18:23:06.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:05 smithi151 ceph-mon[36481]: pgmap v1113: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:06.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:05 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] up:active 2024-03-28T18:23:06.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:05 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T18:23:07.106 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1839 sec 2024-03-28T18:23:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1194.267298 secs 2024-03-28T18:23:07.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 485.691029 secs 2024-03-28T18:23:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1194.267298 secs 2024-03-28T18:23:07.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 485.691029 secs 2024-03-28T18:23:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1194.267298 secs 2024-03-28T18:23:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 485.691029 secs 2024-03-28T18:23:08.107 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1840 sec 2024-03-28T18:23:08.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:07 smithi045 ceph-mon[31773]: pgmap v1114: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:08.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:07 smithi071 ceph-mon[36597]: pgmap v1114: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:08.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:07 smithi151 ceph-mon[36481]: pgmap v1114: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:09.107 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1841 sec 2024-03-28T18:23:10.107 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1842 sec 2024-03-28T18:23:10.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:09 smithi045 ceph-mon[31773]: pgmap v1115: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:10.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:09 smithi071 ceph-mon[36597]: pgmap v1115: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:10.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:09 smithi151 ceph-mon[36481]: pgmap v1115: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:11.107 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1843 sec 2024-03-28T18:23:12.107 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1844 sec 2024-03-28T18:23:12.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:11 smithi151 ceph-mon[36481]: pgmap v1116: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:12.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1199.267358 secs 2024-03-28T18:23:12.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 490.691090 secs 2024-03-28T18:23:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:11 smithi045 ceph-mon[31773]: pgmap v1116: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:12.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1199.267358 secs 2024-03-28T18:23:12.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 490.691090 secs 2024-03-28T18:23:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:11 smithi071 ceph-mon[36597]: pgmap v1116: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1199.267358 secs 2024-03-28T18:23:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 490.691090 secs 2024-03-28T18:23:13.107 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1845 sec 2024-03-28T18:23:14.108 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1846 sec 2024-03-28T18:23:14.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:13 smithi045 ceph-mon[31773]: pgmap v1117: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:14.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:13 smithi071 ceph-mon[36597]: pgmap v1117: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:14.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:13 smithi151 ceph-mon[36481]: pgmap v1117: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:15.108 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1847 sec 2024-03-28T18:23:16.108 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1848 sec 2024-03-28T18:23:16.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:15 smithi045 ceph-mon[31773]: pgmap v1118: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:16.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:15 smithi071 ceph-mon[36597]: pgmap v1118: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:16.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:15 smithi151 ceph-mon[36481]: pgmap v1118: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:17.108 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1849 sec 2024-03-28T18:23:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1204.267421 secs 2024-03-28T18:23:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 495.691106 secs 2024-03-28T18:23:17.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1204.267421 secs 2024-03-28T18:23:17.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 495.691106 secs 2024-03-28T18:23:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1204.267421 secs 2024-03-28T18:23:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 495.691106 secs 2024-03-28T18:23:18.108 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1850 sec 2024-03-28T18:23:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:17 smithi045 ceph-mon[31773]: pgmap v1119: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:18.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:17 smithi071 ceph-mon[36597]: pgmap v1119: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:18.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:17 smithi151 ceph-mon[36481]: pgmap v1119: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:19.108 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1851 sec 2024-03-28T18:23:20.109 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1852 sec 2024-03-28T18:23:20.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:19 smithi045 ceph-mon[31773]: pgmap v1120: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:20.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:19 smithi071 ceph-mon[36597]: pgmap v1120: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:20.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:19 smithi151 ceph-mon[36481]: pgmap v1120: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:21.109 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1853 sec 2024-03-28T18:23:22.109 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1854 sec 2024-03-28T18:23:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:21 smithi045 ceph-mon[31773]: pgmap v1121: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:22.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1209.267485 secs 2024-03-28T18:23:22.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 500.691121 secs 2024-03-28T18:23:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:21 smithi071 ceph-mon[36597]: pgmap v1121: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1209.267485 secs 2024-03-28T18:23:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 500.691121 secs 2024-03-28T18:23:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:21 smithi151 ceph-mon[36481]: pgmap v1121: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1209.267485 secs 2024-03-28T18:23:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 500.691121 secs 2024-03-28T18:23:23.109 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1855 sec 2024-03-28T18:23:24.109 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1856 sec 2024-03-28T18:23:24.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:23 smithi045 ceph-mon[31773]: pgmap v1122: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:24.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:23 smithi071 ceph-mon[36597]: pgmap v1122: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:24.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:23 smithi151 ceph-mon[36481]: pgmap v1122: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:25.109 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1857 sec 2024-03-28T18:23:26.110 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1858 sec 2024-03-28T18:23:26.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:25 smithi045 ceph-mon[31773]: pgmap v1123: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:26.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:25 smithi071 ceph-mon[36597]: pgmap v1123: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:25 smithi151 ceph-mon[36481]: pgmap v1123: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:27.110 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1859 sec 2024-03-28T18:23:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1214.267548 secs 2024-03-28T18:23:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:26 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:23:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 505.691141 secs 2024-03-28T18:23:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1214.267548 secs 2024-03-28T18:23:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:26 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:23:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 505.691141 secs 2024-03-28T18:23:27.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1214.267548 secs 2024-03-28T18:23:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:26 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:23:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 505.691141 secs 2024-03-28T18:23:28.110 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1860 sec 2024-03-28T18:23:28.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:27 smithi045 ceph-mon[31773]: pgmap v1124: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:23:28.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:27 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:23:28.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:27 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:23:28.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:27 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:23:28.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:27 smithi071 ceph-mon[36597]: pgmap v1124: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:23:28.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:27 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:23:28.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:27 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:23:28.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:27 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:23:28.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:27 smithi151 ceph-mon[36481]: pgmap v1124: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:23:28.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:27 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:23:28.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:27 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:23:28.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:27 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:23:29.110 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1861 sec 2024-03-28T18:23:29.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:28 smithi045 ceph-mon[31773]: pgmap v1125: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:23:29.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:28 smithi045 ceph-mon[31773]: pgmap v1126: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:23:29.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:28 smithi071 ceph-mon[36597]: pgmap v1125: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:23:29.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:28 smithi071 ceph-mon[36597]: pgmap v1126: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:23:29.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:28 smithi151 ceph-mon[36481]: pgmap v1125: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:23:29.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:28 smithi151 ceph-mon[36481]: pgmap v1126: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:23:30.110 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1862 sec 2024-03-28T18:23:31.110 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1863 sec 2024-03-28T18:23:31.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:30 smithi045 ceph-mon[31773]: pgmap v1127: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 955 B/s rd, 1 op/s 2024-03-28T18:23:31.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:30 smithi071 ceph-mon[36597]: pgmap v1127: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 955 B/s rd, 1 op/s 2024-03-28T18:23:31.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:30 smithi151 ceph-mon[36481]: pgmap v1127: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 955 B/s rd, 1 op/s 2024-03-28T18:23:32.110 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1864 sec 2024-03-28T18:23:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1219.267611 secs 2024-03-28T18:23:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 510.691202 secs 2024-03-28T18:23:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1219.267611 secs 2024-03-28T18:23:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 510.691202 secs 2024-03-28T18:23:32.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1219.267611 secs 2024-03-28T18:23:32.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 510.691202 secs 2024-03-28T18:23:33.110 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1865 sec 2024-03-28T18:23:33.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:32 smithi071 ceph-mon[36597]: pgmap v1128: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:23:33.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:32 smithi151 ceph-mon[36481]: pgmap v1128: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:23:33.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:32 smithi045 ceph-mon[31773]: pgmap v1128: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:23:34.110 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1866 sec 2024-03-28T18:23:35.111 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1867 sec 2024-03-28T18:23:35.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:34 smithi071 ceph-mon[36597]: pgmap v1129: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:23:35.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:34 smithi151 ceph-mon[36481]: pgmap v1129: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:23:35.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:34 smithi045 ceph-mon[31773]: pgmap v1129: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:23:36.111 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1868 sec 2024-03-28T18:23:37.111 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1869 sec 2024-03-28T18:23:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1224.267673 secs 2024-03-28T18:23:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:36 smithi071 ceph-mon[36597]: pgmap v1130: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:23:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 515.691290 secs 2024-03-28T18:23:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1224.267673 secs 2024-03-28T18:23:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:36 smithi151 ceph-mon[36481]: pgmap v1130: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:23:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 515.691290 secs 2024-03-28T18:23:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1224.267673 secs 2024-03-28T18:23:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:36 smithi045 ceph-mon[31773]: pgmap v1130: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:23:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 515.691290 secs 2024-03-28T18:23:38.111 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1870 sec 2024-03-28T18:23:39.111 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1871 sec 2024-03-28T18:23:39.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:38 smithi071 ceph-mon[36597]: pgmap v1131: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:23:39.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:38 smithi151 ceph-mon[36481]: pgmap v1131: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:23:39.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:38 smithi045 ceph-mon[31773]: pgmap v1131: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:23:40.111 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1872 sec 2024-03-28T18:23:41.112 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1873 sec 2024-03-28T18:23:41.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:40 smithi071 ceph-mon[36597]: pgmap v1132: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:41.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:40 smithi151 ceph-mon[36481]: pgmap v1132: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:41.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:40 smithi045 ceph-mon[31773]: pgmap v1132: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:42.112 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1874 sec 2024-03-28T18:23:42.224 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1229.267736 secs 2024-03-28T18:23:42.225 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 520.691378 secs 2024-03-28T18:23:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1229.267736 secs 2024-03-28T18:23:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 520.691378 secs 2024-03-28T18:23:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1229.267736 secs 2024-03-28T18:23:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 520.691378 secs 2024-03-28T18:23:43.112 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1875 sec 2024-03-28T18:23:43.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:42 smithi071 ceph-mon[36597]: pgmap v1133: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:43.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:42 smithi151 ceph-mon[36481]: pgmap v1133: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:43.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:42 smithi045 ceph-mon[31773]: pgmap v1133: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:44.112 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1876 sec 2024-03-28T18:23:45.112 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1877 sec 2024-03-28T18:23:45.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:44 smithi071 ceph-mon[36597]: pgmap v1134: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:45.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:44 smithi151 ceph-mon[36481]: pgmap v1134: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:45.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:44 smithi045 ceph-mon[31773]: pgmap v1134: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:46.112 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1878 sec 2024-03-28T18:23:47.112 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1879 sec 2024-03-28T18:23:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1234.267799 secs 2024-03-28T18:23:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:46 smithi071 ceph-mon[36597]: pgmap v1135: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 525.691465 secs 2024-03-28T18:23:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1234.267799 secs 2024-03-28T18:23:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:46 smithi151 ceph-mon[36481]: pgmap v1135: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 525.691465 secs 2024-03-28T18:23:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1234.267799 secs 2024-03-28T18:23:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:46 smithi045 ceph-mon[31773]: pgmap v1135: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 525.691465 secs 2024-03-28T18:23:48.112 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1880 sec 2024-03-28T18:23:49.113 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1881 sec 2024-03-28T18:23:49.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:48 smithi071 ceph-mon[36597]: pgmap v1136: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:49.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:48 smithi151 ceph-mon[36481]: pgmap v1136: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:49.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:48 smithi045 ceph-mon[31773]: pgmap v1136: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:50.113 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1882 sec 2024-03-28T18:23:51.113 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1883 sec 2024-03-28T18:23:51.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:50 smithi071 ceph-mon[36597]: pgmap v1137: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:51.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:50 smithi151 ceph-mon[36481]: pgmap v1137: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:51.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:50 smithi045 ceph-mon[31773]: pgmap v1137: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:52.113 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1884 sec 2024-03-28T18:23:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1239.267886 secs 2024-03-28T18:23:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 530.691508 secs 2024-03-28T18:23:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1239.267886 secs 2024-03-28T18:23:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 530.691508 secs 2024-03-28T18:23:52.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1239.267886 secs 2024-03-28T18:23:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 530.691508 secs 2024-03-28T18:23:53.113 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1885 sec 2024-03-28T18:23:53.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:52 smithi071 ceph-mon[36597]: pgmap v1138: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:53.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:52 smithi151 ceph-mon[36481]: pgmap v1138: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:53.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:52 smithi045 ceph-mon[31773]: pgmap v1138: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:54.113 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1886 sec 2024-03-28T18:23:55.113 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1887 sec 2024-03-28T18:23:55.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:54 smithi071 ceph-mon[36597]: pgmap v1139: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:55.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:54 smithi151 ceph-mon[36481]: pgmap v1139: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:55.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:54 smithi045 ceph-mon[31773]: pgmap v1139: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:56.113 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1888 sec 2024-03-28T18:23:56.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1244.267975 secs 2024-03-28T18:23:56.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1244.267975 secs 2024-03-28T18:23:56.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1244.267975 secs 2024-03-28T18:23:57.114 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1889 sec 2024-03-28T18:23:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:57 smithi071 ceph-mon[36597]: pgmap v1140: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 535.691598 secs 2024-03-28T18:23:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:57 smithi151 ceph-mon[36481]: pgmap v1140: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 535.691598 secs 2024-03-28T18:23:57.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:57 smithi045 ceph-mon[31773]: pgmap v1140: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:23:57.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 535.691598 secs 2024-03-28T18:23:58.114 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1890 sec 2024-03-28T18:23:59.114 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1891 sec 2024-03-28T18:23:59.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:23:59 smithi071 ceph-mon[36597]: pgmap v1141: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:23:59 smithi151 ceph-mon[36481]: pgmap v1141: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:23:59.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:23:59 smithi045 ceph-mon[31773]: pgmap v1141: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:00.114 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1892 sec 2024-03-28T18:24:01.114 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1893 sec 2024-03-28T18:24:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:01 smithi071 ceph-mon[36597]: pgmap v1142: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1249.268037 secs 2024-03-28T18:24:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:01 smithi151 ceph-mon[36481]: pgmap v1142: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1249.268037 secs 2024-03-28T18:24:01.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:01 smithi045 ceph-mon[31773]: pgmap v1142: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:01.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1249.268037 secs 2024-03-28T18:24:02.114 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1894 sec 2024-03-28T18:24:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 540.691684 secs 2024-03-28T18:24:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 540.691684 secs 2024-03-28T18:24:02.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 540.691684 secs 2024-03-28T18:24:03.115 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1895 sec 2024-03-28T18:24:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:03 smithi071 ceph-mon[36597]: pgmap v1143: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:24:03.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:03 smithi151 ceph-mon[36481]: pgmap v1143: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:24:03.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:03 smithi045 ceph-mon[31773]: pgmap v1143: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:24:04.115 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1896 sec 2024-03-28T18:24:05.115 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1897 sec 2024-03-28T18:24:05.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:05 smithi071 ceph-mon[36597]: pgmap v1144: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:05.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:05 smithi151 ceph-mon[36481]: pgmap v1144: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:05.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:05 smithi045 ceph-mon[31773]: pgmap v1144: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:06.115 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1898 sec 2024-03-28T18:24:06.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1254.268126 secs 2024-03-28T18:24:06.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1254.268126 secs 2024-03-28T18:24:06.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1254.268126 secs 2024-03-28T18:24:07.115 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1899 sec 2024-03-28T18:24:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:07 smithi071 ceph-mon[36597]: pgmap v1145: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:24:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 545.691772 secs 2024-03-28T18:24:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:07 smithi151 ceph-mon[36481]: pgmap v1145: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:24:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 545.691772 secs 2024-03-28T18:24:07.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:07 smithi045 ceph-mon[31773]: pgmap v1145: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:24:07.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 545.691772 secs 2024-03-28T18:24:08.115 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1900 sec 2024-03-28T18:24:09.115 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1901 sec 2024-03-28T18:24:09.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:09 smithi071 ceph-mon[36597]: pgmap v1146: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:09.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:09 smithi151 ceph-mon[36481]: pgmap v1146: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:09.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:09 smithi045 ceph-mon[31773]: pgmap v1146: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:10.116 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1902 sec 2024-03-28T18:24:11.116 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1903 sec 2024-03-28T18:24:11.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:11 smithi151 ceph-mon[36481]: pgmap v1147: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:11.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1259.268204 secs 2024-03-28T18:24:11.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:11 smithi045 ceph-mon[31773]: pgmap v1147: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:11.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1259.268204 secs 2024-03-28T18:24:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:11 smithi071 ceph-mon[36597]: pgmap v1147: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1259.268204 secs 2024-03-28T18:24:12.116 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1904 sec 2024-03-28T18:24:12.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 550.691863 secs 2024-03-28T18:24:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 550.691863 secs 2024-03-28T18:24:12.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 550.691863 secs 2024-03-28T18:24:13.116 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1905 sec 2024-03-28T18:24:13.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:13 smithi045 ceph-mon[31773]: pgmap v1148: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:24:13.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:13 smithi071 ceph-mon[36597]: pgmap v1148: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:24:13.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:13 smithi151 ceph-mon[36481]: pgmap v1148: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:24:14.116 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1906 sec 2024-03-28T18:24:15.116 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1907 sec 2024-03-28T18:24:15.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:15 smithi045 ceph-mon[31773]: pgmap v1149: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:15.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:15 smithi071 ceph-mon[36597]: pgmap v1149: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:15 smithi151 ceph-mon[36481]: pgmap v1149: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:16.116 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1908 sec 2024-03-28T18:24:16.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1264.268227 secs 2024-03-28T18:24:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1264.268227 secs 2024-03-28T18:24:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1264.268227 secs 2024-03-28T18:24:17.116 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1909 sec 2024-03-28T18:24:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:17 smithi045 ceph-mon[31773]: pgmap v1150: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:24:17.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 555.691948 secs 2024-03-28T18:24:17.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:17 smithi071 ceph-mon[36597]: pgmap v1150: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:24:17.462 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 555.691948 secs 2024-03-28T18:24:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:17 smithi151 ceph-mon[36481]: pgmap v1150: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:24:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 555.691948 secs 2024-03-28T18:24:18.117 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1910 sec 2024-03-28T18:24:19.117 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1911 sec 2024-03-28T18:24:19.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:19 smithi045 ceph-mon[31773]: pgmap v1151: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:19.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:19 smithi071 ceph-mon[36597]: pgmap v1151: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:19.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:19 smithi151 ceph-mon[36481]: pgmap v1151: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:20.117 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1912 sec 2024-03-28T18:24:21.117 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1913 sec 2024-03-28T18:24:21.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:21 smithi045 ceph-mon[31773]: pgmap v1152: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1269.268288 secs 2024-03-28T18:24:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:21 smithi071 ceph-mon[36597]: pgmap v1152: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1269.268288 secs 2024-03-28T18:24:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:21 smithi151 ceph-mon[36481]: pgmap v1152: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1269.268288 secs 2024-03-28T18:24:22.117 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1914 sec 2024-03-28T18:24:22.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 560.692033 secs 2024-03-28T18:24:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 560.692033 secs 2024-03-28T18:24:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 560.692033 secs 2024-03-28T18:24:23.117 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1915 sec 2024-03-28T18:24:23.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:23 smithi045 ceph-mon[31773]: pgmap v1153: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:24:23.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:23 smithi071 ceph-mon[36597]: pgmap v1153: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:24:23.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:23 smithi151 ceph-mon[36481]: pgmap v1153: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:24:24.117 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1916 sec 2024-03-28T18:24:25.117 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1917 sec 2024-03-28T18:24:25.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:25 smithi045 ceph-mon[31773]: pgmap v1154: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:25.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:25 smithi071 ceph-mon[36597]: pgmap v1154: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:25.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:25 smithi151 ceph-mon[36481]: pgmap v1154: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:26.118 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1918 sec 2024-03-28T18:24:26.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1274.268349 secs 2024-03-28T18:24:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1274.268349 secs 2024-03-28T18:24:26.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1274.268349 secs 2024-03-28T18:24:27.118 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1919 sec 2024-03-28T18:24:27.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:27 smithi045 ceph-mon[31773]: pgmap v1155: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:24:27.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 565.692121 secs 2024-03-28T18:24:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:27 smithi071 ceph-mon[36597]: pgmap v1155: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:24:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 565.692121 secs 2024-03-28T18:24:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:27 smithi151 ceph-mon[36481]: pgmap v1155: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:24:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 565.692121 secs 2024-03-28T18:24:28.118 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1920 sec 2024-03-28T18:24:29.118 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1921 sec 2024-03-28T18:24:29.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:29 smithi045 ceph-mon[31773]: pgmap v1156: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:29.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:29 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:24:29.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:29 smithi071 ceph-mon[36597]: pgmap v1156: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:29.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:29 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:24:29.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:29 smithi151 ceph-mon[36481]: pgmap v1156: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:29.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:29 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:24:30.118 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1922 sec 2024-03-28T18:24:30.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:30 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:24:30.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:30 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:24:30.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:30 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:24:30.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:30 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:24:30.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:30 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:24:30.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:30 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:24:30.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:30 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:24:30.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:30 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:24:30.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:30 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:24:31.118 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1923 sec 2024-03-28T18:24:31.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:31 smithi045 ceph-mon[31773]: pgmap v1157: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:31.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:31 smithi045 ceph-mon[31773]: pgmap v1158: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:24:31.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1279.268439 secs 2024-03-28T18:24:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:31 smithi071 ceph-mon[36597]: pgmap v1157: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:31 smithi071 ceph-mon[36597]: pgmap v1158: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:24:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1279.268439 secs 2024-03-28T18:24:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:31 smithi151 ceph-mon[36481]: pgmap v1157: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:31 smithi151 ceph-mon[36481]: pgmap v1158: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:24:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1279.268439 secs 2024-03-28T18:24:32.118 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1924 sec 2024-03-28T18:24:32.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 570.692184 secs 2024-03-28T18:24:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 570.692184 secs 2024-03-28T18:24:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 570.692184 secs 2024-03-28T18:24:33.119 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1925 sec 2024-03-28T18:24:33.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:33 smithi045 ceph-mon[31773]: pgmap v1159: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:24:33.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:33 smithi071 ceph-mon[36597]: pgmap v1159: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:24:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:33 smithi151 ceph-mon[36481]: pgmap v1159: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:24:34.119 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1926 sec 2024-03-28T18:24:35.119 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1927 sec 2024-03-28T18:24:35.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:35 smithi071 ceph-mon[36597]: pgmap v1160: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:24:35.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:35 smithi151 ceph-mon[36481]: pgmap v1160: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:24:35.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:35 smithi045 ceph-mon[31773]: pgmap v1160: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:24:36.119 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1928 sec 2024-03-28T18:24:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1284.268476 secs 2024-03-28T18:24:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1284.268476 secs 2024-03-28T18:24:36.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1284.268476 secs 2024-03-28T18:24:37.119 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1929 sec 2024-03-28T18:24:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:37 smithi071 ceph-mon[36597]: pgmap v1161: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:24:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 575.692225 secs 2024-03-28T18:24:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:37 smithi151 ceph-mon[36481]: pgmap v1161: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:24:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 575.692225 secs 2024-03-28T18:24:37.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:37 smithi045 ceph-mon[31773]: pgmap v1161: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:24:37.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 575.692225 secs 2024-03-28T18:24:38.119 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1930 sec 2024-03-28T18:24:39.120 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1931 sec 2024-03-28T18:24:39.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:39 smithi071 ceph-mon[36597]: pgmap v1162: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:24:39.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:39 smithi151 ceph-mon[36481]: pgmap v1162: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:24:39.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:39 smithi045 ceph-mon[31773]: pgmap v1162: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:24:40.120 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1932 sec 2024-03-28T18:24:41.120 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1933 sec 2024-03-28T18:24:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:41 smithi071 ceph-mon[36597]: pgmap v1163: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:24:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1289.268564 secs 2024-03-28T18:24:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:41 smithi151 ceph-mon[36481]: pgmap v1163: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:24:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1289.268564 secs 2024-03-28T18:24:41.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:41 smithi045 ceph-mon[31773]: pgmap v1163: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:24:41.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1289.268564 secs 2024-03-28T18:24:42.120 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1934 sec 2024-03-28T18:24:42.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 580.692287 secs 2024-03-28T18:24:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 580.692287 secs 2024-03-28T18:24:42.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 580.692287 secs 2024-03-28T18:24:43.120 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1935 sec 2024-03-28T18:24:43.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:43 smithi071 ceph-mon[36597]: pgmap v1164: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:24:43.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:43 smithi151 ceph-mon[36481]: pgmap v1164: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:24:43.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:43 smithi045 ceph-mon[31773]: pgmap v1164: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:24:44.120 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1936 sec 2024-03-28T18:24:44.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:44 smithi071 ceph-mon[36597]: pgmap v1165: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:44 smithi151 ceph-mon[36481]: pgmap v1165: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:44.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:44 smithi045 ceph-mon[31773]: pgmap v1165: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:45.120 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1937 sec 2024-03-28T18:24:46.121 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1938 sec 2024-03-28T18:24:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:46 smithi045 ceph-mon[31773]: pgmap v1166: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:24:46.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1294.268606 secs 2024-03-28T18:24:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:46 smithi071 ceph-mon[36597]: pgmap v1166: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:24:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1294.268606 secs 2024-03-28T18:24:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:46 smithi151 ceph-mon[36481]: pgmap v1166: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:24:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1294.268606 secs 2024-03-28T18:24:47.120 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1939 sec 2024-03-28T18:24:47.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 585.692349 secs 2024-03-28T18:24:48.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 585.692349 secs 2024-03-28T18:24:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 585.692349 secs 2024-03-28T18:24:48.120 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1940 sec 2024-03-28T18:24:48.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:48 smithi045 ceph-mon[31773]: pgmap v1167: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:24:49.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:48 smithi071 ceph-mon[36597]: pgmap v1167: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:24:49.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:48 smithi151 ceph-mon[36481]: pgmap v1167: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:24:49.121 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1941 sec 2024-03-28T18:24:50.121 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1942 sec 2024-03-28T18:24:50.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:50 smithi045 ceph-mon[31773]: pgmap v1168: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:24:51.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:50 smithi071 ceph-mon[36597]: pgmap v1168: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:24:51.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:50 smithi151 ceph-mon[36481]: pgmap v1168: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:24:51.121 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1943 sec 2024-03-28T18:24:51.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1299.268692 secs 2024-03-28T18:24:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1299.268692 secs 2024-03-28T18:24:52.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1299.268692 secs 2024-03-28T18:24:52.121 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1944 sec 2024-03-28T18:24:52.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:52 smithi045 ceph-mon[31773]: pgmap v1169: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:24:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 590.692410 secs 2024-03-28T18:24:53.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:52 smithi071 ceph-mon[36597]: pgmap v1169: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:24:53.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 590.692410 secs 2024-03-28T18:24:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:52 smithi151 ceph-mon[36481]: pgmap v1169: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:24:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 590.692410 secs 2024-03-28T18:24:53.121 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1945 sec 2024-03-28T18:24:54.121 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1946 sec 2024-03-28T18:24:54.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:54 smithi045 ceph-mon[31773]: pgmap v1170: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:24:55.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:54 smithi071 ceph-mon[36597]: pgmap v1170: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:24:55.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:54 smithi151 ceph-mon[36481]: pgmap v1170: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:24:55.121 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1947 sec 2024-03-28T18:24:56.122 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1948 sec 2024-03-28T18:24:56.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:56 smithi045 ceph-mon[31773]: pgmap v1171: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:24:56.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1304.268709 secs 2024-03-28T18:24:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:56 smithi071 ceph-mon[36597]: pgmap v1171: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:24:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1304.268709 secs 2024-03-28T18:24:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:56 smithi151 ceph-mon[36481]: pgmap v1171: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:24:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1304.268709 secs 2024-03-28T18:24:57.122 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1949 sec 2024-03-28T18:24:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 595.692467 secs 2024-03-28T18:24:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 595.692467 secs 2024-03-28T18:24:58.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 595.692467 secs 2024-03-28T18:24:58.122 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1950 sec 2024-03-28T18:24:58.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:24:58 smithi045 ceph-mon[31773]: pgmap v1172: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:59.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:24:58 smithi071 ceph-mon[36597]: pgmap v1172: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:59.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:24:58 smithi151 ceph-mon[36481]: pgmap v1172: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:24:59.122 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1951 sec 2024-03-28T18:25:00.122 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1952 sec 2024-03-28T18:25:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:00 smithi045 ceph-mon[31773]: pgmap v1173: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:01.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:00 smithi071 ceph-mon[36597]: pgmap v1173: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:01.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:00 smithi151 ceph-mon[36481]: pgmap v1173: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:01.122 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1953 sec 2024-03-28T18:25:01.662 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:25:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:25:01.193+0000 7f52eec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:25:01.662 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:25:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:25:01.193+0000 7f52eec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:25:01.662 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:25:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:25:01.193+0000 7f52eec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:25:01.662 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:25:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:25:01.193+0000 7f52eec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:25:01.662 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:25:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:25:01.193+0000 7f52eec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:25:01.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1309.268773 secs 2024-03-28T18:25:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1309.268773 secs 2024-03-28T18:25:02.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1309.268773 secs 2024-03-28T18:25:02.122 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1954 sec 2024-03-28T18:25:02.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:02 smithi045 ceph-mon[31773]: pgmap v1174: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:25:02.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 600.692551 secs 2024-03-28T18:25:03.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:02 smithi071 ceph-mon[36597]: pgmap v1174: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:25:03.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 600.692551 secs 2024-03-28T18:25:03.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:02 smithi151 ceph-mon[36481]: pgmap v1174: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:25:03.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 600.692551 secs 2024-03-28T18:25:03.123 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1955 sec 2024-03-28T18:25:04.123 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1956 sec 2024-03-28T18:25:05.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:04 smithi071 ceph-mon[36597]: pgmap v1175: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:05.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:04 smithi151 ceph-mon[36481]: pgmap v1175: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:05.123 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1957 sec 2024-03-28T18:25:05.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:04 smithi045 ceph-mon[31773]: pgmap v1175: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:06.123 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1958 sec 2024-03-28T18:25:07.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:06 smithi071 ceph-mon[36597]: pgmap v1176: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:25:07.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1314.268834 secs 2024-03-28T18:25:07.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:06 smithi151 ceph-mon[36481]: pgmap v1176: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:25:07.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1314.268834 secs 2024-03-28T18:25:07.123 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1959 sec 2024-03-28T18:25:07.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:06 smithi045 ceph-mon[31773]: pgmap v1176: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:25:07.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1314.268834 secs 2024-03-28T18:25:08.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 605.692591 secs 2024-03-28T18:25:08.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 605.692591 secs 2024-03-28T18:25:08.124 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1960 sec 2024-03-28T18:25:08.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 605.692591 secs 2024-03-28T18:25:09.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:08 smithi071 ceph-mon[36597]: pgmap v1177: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:25:09.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:08 smithi151 ceph-mon[36481]: pgmap v1177: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:25:09.124 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1961 sec 2024-03-28T18:25:09.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:08 smithi045 ceph-mon[31773]: pgmap v1177: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:25:10.124 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1962 sec 2024-03-28T18:25:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:10 smithi071 ceph-mon[36597]: pgmap v1178: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:25:11.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:10 smithi151 ceph-mon[36481]: pgmap v1178: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:25:11.124 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1963 sec 2024-03-28T18:25:11.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:10 smithi045 ceph-mon[31773]: pgmap v1178: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:25:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1319.268896 secs 2024-03-28T18:25:12.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1319.268896 secs 2024-03-28T18:25:12.124 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1964 sec 2024-03-28T18:25:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1319.268896 secs 2024-03-28T18:25:12.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:12 smithi151 ceph-mon[36481]: pgmap v1179: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:25:12.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 610.692679 secs 2024-03-28T18:25:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:12 smithi071 ceph-mon[36597]: pgmap v1179: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:25:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 610.692679 secs 2024-03-28T18:25:13.124 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1965 sec 2024-03-28T18:25:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:12 smithi045 ceph-mon[31773]: pgmap v1179: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:25:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 610.692679 secs 2024-03-28T18:25:14.124 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1966 sec 2024-03-28T18:25:15.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:14 smithi071 ceph-mon[36597]: pgmap v1180: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:25:15.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:14 smithi151 ceph-mon[36481]: pgmap v1180: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:25:15.125 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1967 sec 2024-03-28T18:25:15.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:14 smithi045 ceph-mon[31773]: pgmap v1180: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:25:16.125 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1968 sec 2024-03-28T18:25:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:16 smithi071 ceph-mon[36597]: pgmap v1181: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:25:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1324.268986 secs 2024-03-28T18:25:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:16 smithi151 ceph-mon[36481]: pgmap v1181: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:25:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1324.268986 secs 2024-03-28T18:25:17.125 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1969 sec 2024-03-28T18:25:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:16 smithi045 ceph-mon[31773]: pgmap v1181: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:25:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1324.268986 secs 2024-03-28T18:25:18.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 615.692740 secs 2024-03-28T18:25:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 615.692740 secs 2024-03-28T18:25:18.125 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1970 sec 2024-03-28T18:25:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 615.692740 secs 2024-03-28T18:25:19.023 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:18 smithi045 ceph-mon[31773]: pgmap v1182: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:19.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:18 smithi071 ceph-mon[36597]: pgmap v1182: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:19.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:18 smithi151 ceph-mon[36481]: pgmap v1182: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:19.125 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1971 sec 2024-03-28T18:25:20.125 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1972 sec 2024-03-28T18:25:21.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:20 smithi071 ceph-mon[36597]: pgmap v1183: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:21.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:20 smithi151 ceph-mon[36481]: pgmap v1183: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:21.125 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1973 sec 2024-03-28T18:25:21.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:20 smithi045 ceph-mon[31773]: pgmap v1183: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1329.269077 secs 2024-03-28T18:25:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1329.269077 secs 2024-03-28T18:25:22.126 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1974 sec 2024-03-28T18:25:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1329.269077 secs 2024-03-28T18:25:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:22 smithi071 ceph-mon[36597]: pgmap v1184: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:25:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 620.692828 secs 2024-03-28T18:25:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:22 smithi151 ceph-mon[36481]: pgmap v1184: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:25:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 620.692828 secs 2024-03-28T18:25:23.126 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1975 sec 2024-03-28T18:25:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:22 smithi045 ceph-mon[31773]: pgmap v1184: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:25:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 620.692828 secs 2024-03-28T18:25:24.126 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1976 sec 2024-03-28T18:25:25.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:24 smithi071 ceph-mon[36597]: pgmap v1185: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:25.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:24 smithi151 ceph-mon[36481]: pgmap v1185: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:25.126 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1977 sec 2024-03-28T18:25:25.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:24 smithi045 ceph-mon[31773]: pgmap v1185: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:26.126 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1978 sec 2024-03-28T18:25:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:26 smithi071 ceph-mon[36597]: pgmap v1186: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:25:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1334.269166 secs 2024-03-28T18:25:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:26 smithi151 ceph-mon[36481]: pgmap v1186: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:25:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1334.269166 secs 2024-03-28T18:25:27.126 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1979 sec 2024-03-28T18:25:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:26 smithi045 ceph-mon[31773]: pgmap v1186: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:25:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1334.269166 secs 2024-03-28T18:25:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 625.692919 secs 2024-03-28T18:25:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 625.692919 secs 2024-03-28T18:25:28.126 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1980 sec 2024-03-28T18:25:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 625.692919 secs 2024-03-28T18:25:29.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:28 smithi071 ceph-mon[36597]: pgmap v1187: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:29.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:28 smithi151 ceph-mon[36481]: pgmap v1187: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:29.126 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1981 sec 2024-03-28T18:25:29.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:28 smithi045 ceph-mon[31773]: pgmap v1187: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:30.127 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1982 sec 2024-03-28T18:25:31.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:30 smithi071 ceph-mon[36597]: pgmap v1188: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:31.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:30 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:25:31.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:30 smithi151 ceph-mon[36481]: pgmap v1188: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:31.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:30 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:25:31.127 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1983 sec 2024-03-28T18:25:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:30 smithi045 ceph-mon[31773]: pgmap v1188: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:30 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:25:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1339.269186 secs 2024-03-28T18:25:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:31 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:25:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:31 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:25:32.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:31 smithi071 ceph-mon[36597]: pgmap v1189: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:25:32.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:31 smithi071 ceph-mon[36597]: pgmap v1190: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:25:32.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:31 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:25:32.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1339.269186 secs 2024-03-28T18:25:32.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:31 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:25:32.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:31 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:25:32.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:31 smithi151 ceph-mon[36481]: pgmap v1189: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:25:32.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:31 smithi151 ceph-mon[36481]: pgmap v1190: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:25:32.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:31 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:25:32.127 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1984 sec 2024-03-28T18:25:32.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1339.269186 secs 2024-03-28T18:25:32.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:31 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:25:32.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:31 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:25:32.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:31 smithi045 ceph-mon[31773]: pgmap v1189: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:25:32.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:31 smithi045 ceph-mon[31773]: pgmap v1190: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:25:32.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:31 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:25:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 630.692981 secs 2024-03-28T18:25:33.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 630.692981 secs 2024-03-28T18:25:33.127 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1985 sec 2024-03-28T18:25:33.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 630.692981 secs 2024-03-28T18:25:34.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:33 smithi071 ceph-mon[36597]: pgmap v1191: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:25:34.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:33 smithi151 ceph-mon[36481]: pgmap v1191: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:25:34.127 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1986 sec 2024-03-28T18:25:34.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:33 smithi045 ceph-mon[31773]: pgmap v1191: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:25:35.128 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1987 sec 2024-03-28T18:25:36.127 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1988 sec 2024-03-28T18:25:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:36 smithi071 ceph-mon[36597]: pgmap v1192: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:25:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1344.269275 secs 2024-03-28T18:25:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:36 smithi151 ceph-mon[36481]: pgmap v1192: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:25:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1344.269275 secs 2024-03-28T18:25:36.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:36 smithi045 ceph-mon[31773]: pgmap v1192: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:25:36.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1344.269275 secs 2024-03-28T18:25:37.128 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1989 sec 2024-03-28T18:25:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 635.693026 secs 2024-03-28T18:25:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:37 smithi071 ceph-mon[36597]: pgmap v1193: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:25:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 635.693026 secs 2024-03-28T18:25:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:37 smithi151 ceph-mon[36481]: pgmap v1193: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:25:37.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 635.693026 secs 2024-03-28T18:25:37.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:37 smithi045 ceph-mon[31773]: pgmap v1193: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:25:38.128 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1990 sec 2024-03-28T18:25:39.128 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1991 sec 2024-03-28T18:25:40.128 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1992 sec 2024-03-28T18:25:40.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:40 smithi071 ceph-mon[36597]: pgmap v1194: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:25:40.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:40 smithi151 ceph-mon[36481]: pgmap v1194: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:25:40.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:40 smithi045 ceph-mon[31773]: pgmap v1194: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:25:41.128 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1993 sec 2024-03-28T18:25:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1349.269289 secs 2024-03-28T18:25:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:41 smithi071 ceph-mon[36597]: pgmap v1195: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:25:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1349.269289 secs 2024-03-28T18:25:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:41 smithi151 ceph-mon[36481]: pgmap v1195: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:25:41.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1349.269289 secs 2024-03-28T18:25:41.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:41 smithi045 ceph-mon[31773]: pgmap v1195: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:25:42.128 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1994 sec 2024-03-28T18:25:42.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 640.693117 secs 2024-03-28T18:25:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 640.693117 secs 2024-03-28T18:25:42.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 640.693117 secs 2024-03-28T18:25:43.129 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1995 sec 2024-03-28T18:25:43.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:43 smithi071 ceph-mon[36597]: pgmap v1196: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:43.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:43 smithi151 ceph-mon[36481]: pgmap v1196: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:43.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:43 smithi045 ceph-mon[31773]: pgmap v1196: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:44.129 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1996 sec 2024-03-28T18:25:45.129 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1997 sec 2024-03-28T18:25:45.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:45 smithi071 ceph-mon[36597]: pgmap v1197: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:25:45.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:45 smithi151 ceph-mon[36481]: pgmap v1197: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:25:45.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:45 smithi045 ceph-mon[31773]: pgmap v1197: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:25:46.129 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1998 sec 2024-03-28T18:25:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1354.269377 secs 2024-03-28T18:25:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1354.269377 secs 2024-03-28T18:25:46.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1354.269377 secs 2024-03-28T18:25:47.129 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 1999 sec 2024-03-28T18:25:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 645.693204 secs 2024-03-28T18:25:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:47 smithi071 ceph-mon[36597]: pgmap v1198: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 645.693204 secs 2024-03-28T18:25:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:47 smithi151 ceph-mon[36481]: pgmap v1198: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:47.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 645.693204 secs 2024-03-28T18:25:47.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:47 smithi045 ceph-mon[31773]: pgmap v1198: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:48.129 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2000 sec 2024-03-28T18:25:49.129 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2001 sec 2024-03-28T18:25:49.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:49 smithi071 ceph-mon[36597]: pgmap v1199: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:49 smithi151 ceph-mon[36481]: pgmap v1199: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:49.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:49 smithi045 ceph-mon[31773]: pgmap v1199: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:50.129 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2002 sec 2024-03-28T18:25:51.130 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2003 sec 2024-03-28T18:25:51.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1359.269439 secs 2024-03-28T18:25:51.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1359.269439 secs 2024-03-28T18:25:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1359.269439 secs 2024-03-28T18:25:52.130 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2004 sec 2024-03-28T18:25:52.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:52 smithi045 ceph-mon[31773]: pgmap v1200: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:25:52.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 650.693264 secs 2024-03-28T18:25:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:52 smithi071 ceph-mon[36597]: pgmap v1200: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:25:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 650.693264 secs 2024-03-28T18:25:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:52 smithi151 ceph-mon[36481]: pgmap v1200: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:25:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 650.693264 secs 2024-03-28T18:25:53.130 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2005 sec 2024-03-28T18:25:53.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:53 smithi071 ceph-mon[36597]: pgmap v1201: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:53.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:53 smithi151 ceph-mon[36481]: pgmap v1201: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:53.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:53 smithi045 ceph-mon[31773]: pgmap v1201: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:54.130 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2006 sec 2024-03-28T18:25:55.130 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2007 sec 2024-03-28T18:25:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:55 smithi071 ceph-mon[36597]: pgmap v1202: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:25:55.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:55 smithi151 ceph-mon[36481]: pgmap v1202: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:25:55.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:55 smithi045 ceph-mon[31773]: pgmap v1202: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:25:56.130 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2008 sec 2024-03-28T18:25:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1364.269501 secs 2024-03-28T18:25:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1364.269501 secs 2024-03-28T18:25:56.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1364.269501 secs 2024-03-28T18:25:57.130 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2009 sec 2024-03-28T18:25:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 655.693326 secs 2024-03-28T18:25:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:57 smithi071 ceph-mon[36597]: pgmap v1203: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 655.693326 secs 2024-03-28T18:25:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:57 smithi151 ceph-mon[36481]: pgmap v1203: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:57.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 655.693326 secs 2024-03-28T18:25:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:57 smithi045 ceph-mon[31773]: pgmap v1203: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:58.131 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2010 sec 2024-03-28T18:25:59.131 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2011 sec 2024-03-28T18:25:59.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:25:59 smithi071 ceph-mon[36597]: pgmap v1204: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:59.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:25:59 smithi151 ceph-mon[36481]: pgmap v1204: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:25:59.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:25:59 smithi045 ceph-mon[31773]: pgmap v1204: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:00.131 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2012 sec 2024-03-28T18:26:01.131 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2013 sec 2024-03-28T18:26:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1369.269513 secs 2024-03-28T18:26:01.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1369.269513 secs 2024-03-28T18:26:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1369.269513 secs 2024-03-28T18:26:02.131 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2014 sec 2024-03-28T18:26:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:02 smithi151 ceph-mon[36481]: pgmap v1205: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:02 smithi151 ceph-mon[36481]: 2 slow requests, 1 included below; oldest blocked for > 660.693366 secs 2024-03-28T18:26:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:02 smithi151 ceph-mon[36481]: slow request 360.505723 seconds old, received at 2024-03-28T18:20:01.134771+0000: client_request(client.16398:1 getattr pAsLsXsFs #0x1 2024-03-28T18:20:01.130880+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:26:02.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:02 smithi045 ceph-mon[31773]: pgmap v1205: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:02.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:02 smithi045 ceph-mon[31773]: 2 slow requests, 1 included below; oldest blocked for > 660.693366 secs 2024-03-28T18:26:02.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:02 smithi045 ceph-mon[31773]: slow request 360.505723 seconds old, received at 2024-03-28T18:20:01.134771+0000: client_request(client.16398:1 getattr pAsLsXsFs #0x1 2024-03-28T18:20:01.130880+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:26:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:02 smithi071 ceph-mon[36597]: pgmap v1205: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:02 smithi071 ceph-mon[36597]: 2 slow requests, 1 included below; oldest blocked for > 660.693366 secs 2024-03-28T18:26:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:02 smithi071 ceph-mon[36597]: slow request 360.505723 seconds old, received at 2024-03-28T18:20:01.134771+0000: client_request(client.16398:1 getattr pAsLsXsFs #0x1 2024-03-28T18:20:01.130880+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:26:03.131 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2015 sec 2024-03-28T18:26:03.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:03 smithi071 ceph-mon[36597]: pgmap v1206: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:03 smithi151 ceph-mon[36481]: pgmap v1206: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:03.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:03 smithi045 ceph-mon[31773]: pgmap v1206: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:04.131 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2016 sec 2024-03-28T18:26:05.132 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2017 sec 2024-03-28T18:26:05.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:05 smithi071 ceph-mon[36597]: pgmap v1207: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:05.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:05 smithi151 ceph-mon[36481]: pgmap v1207: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:05.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:05 smithi045 ceph-mon[31773]: pgmap v1207: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:06.132 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2018 sec 2024-03-28T18:26:06.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1374.269574 secs 2024-03-28T18:26:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1374.269574 secs 2024-03-28T18:26:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1374.269574 secs 2024-03-28T18:26:07.132 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2019 sec 2024-03-28T18:26:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 665.693455 secs 2024-03-28T18:26:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:07 smithi071 ceph-mon[36597]: pgmap v1208: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 665.693455 secs 2024-03-28T18:26:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:07 smithi151 ceph-mon[36481]: pgmap v1208: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 665.693455 secs 2024-03-28T18:26:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:07 smithi045 ceph-mon[31773]: pgmap v1208: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:08.132 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2020 sec 2024-03-28T18:26:09.132 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2021 sec 2024-03-28T18:26:09.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:09 smithi071 ceph-mon[36597]: pgmap v1209: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:09.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:09 smithi151 ceph-mon[36481]: pgmap v1209: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:09.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:09 smithi045 ceph-mon[31773]: pgmap v1209: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:10.132 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2022 sec 2024-03-28T18:26:11.133 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2023 sec 2024-03-28T18:26:11.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1379.269636 secs 2024-03-28T18:26:11.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1379.269636 secs 2024-03-28T18:26:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1379.269636 secs 2024-03-28T18:26:12.132 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2024 sec 2024-03-28T18:26:12.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:12 smithi045 ceph-mon[31773]: pgmap v1210: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:12.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 670.693470 secs 2024-03-28T18:26:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:12 smithi071 ceph-mon[36597]: pgmap v1210: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 670.693470 secs 2024-03-28T18:26:12.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:12 smithi151 ceph-mon[36481]: pgmap v1210: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:12.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 670.693470 secs 2024-03-28T18:26:13.133 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2025 sec 2024-03-28T18:26:13.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:13 smithi071 ceph-mon[36597]: pgmap v1211: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:13.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:13 smithi151 ceph-mon[36481]: pgmap v1211: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:13.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:13 smithi045 ceph-mon[31773]: pgmap v1211: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:14.133 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2026 sec 2024-03-28T18:26:15.133 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2027 sec 2024-03-28T18:26:15.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:15 smithi071 ceph-mon[36597]: pgmap v1212: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:15 smithi151 ceph-mon[36481]: pgmap v1212: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:15.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:15 smithi045 ceph-mon[31773]: pgmap v1212: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:16.133 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2028 sec 2024-03-28T18:26:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1384.269698 secs 2024-03-28T18:26:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1384.269698 secs 2024-03-28T18:26:16.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1384.269698 secs 2024-03-28T18:26:17.133 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2029 sec 2024-03-28T18:26:17.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 675.693562 secs 2024-03-28T18:26:17.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:17 smithi071 ceph-mon[36597]: pgmap v1213: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 675.693562 secs 2024-03-28T18:26:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:17 smithi151 ceph-mon[36481]: pgmap v1213: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:17.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 675.693562 secs 2024-03-28T18:26:17.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:17 smithi045 ceph-mon[31773]: pgmap v1213: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:18.133 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2030 sec 2024-03-28T18:26:19.134 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2031 sec 2024-03-28T18:26:19.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:19 smithi071 ceph-mon[36597]: pgmap v1214: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:19.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:19 smithi151 ceph-mon[36481]: pgmap v1214: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:19 smithi045 ceph-mon[31773]: pgmap v1214: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:20.134 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2032 sec 2024-03-28T18:26:21.134 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2033 sec 2024-03-28T18:26:21.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1389.269758 secs 2024-03-28T18:26:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1389.269758 secs 2024-03-28T18:26:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1389.269758 secs 2024-03-28T18:26:22.134 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2034 sec 2024-03-28T18:26:22.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:22 smithi045 ceph-mon[31773]: pgmap v1215: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:22.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 680.693645 secs 2024-03-28T18:26:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:22 smithi071 ceph-mon[36597]: pgmap v1215: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 680.693645 secs 2024-03-28T18:26:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:22 smithi151 ceph-mon[36481]: pgmap v1215: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 680.693645 secs 2024-03-28T18:26:23.134 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2035 sec 2024-03-28T18:26:23.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:23 smithi071 ceph-mon[36597]: pgmap v1216: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:23.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:23 smithi151 ceph-mon[36481]: pgmap v1216: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:23.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:23 smithi045 ceph-mon[31773]: pgmap v1216: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:24.134 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2036 sec 2024-03-28T18:26:25.135 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2037 sec 2024-03-28T18:26:25.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:25 smithi071 ceph-mon[36597]: pgmap v1217: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:25.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:25 smithi151 ceph-mon[36481]: pgmap v1217: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:25.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:25 smithi045 ceph-mon[31773]: pgmap v1217: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:26.134 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2038 sec 2024-03-28T18:26:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1394.269819 secs 2024-03-28T18:26:26.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1394.269819 secs 2024-03-28T18:26:26.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1394.269819 secs 2024-03-28T18:26:27.135 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2039 sec 2024-03-28T18:26:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 685.693736 secs 2024-03-28T18:26:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:27 smithi071 ceph-mon[36597]: pgmap v1218: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 685.693736 secs 2024-03-28T18:26:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:27 smithi151 ceph-mon[36481]: pgmap v1218: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:27.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 685.693736 secs 2024-03-28T18:26:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:27 smithi045 ceph-mon[31773]: pgmap v1218: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:28.135 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2040 sec 2024-03-28T18:26:29.135 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2041 sec 2024-03-28T18:26:29.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:29 smithi071 ceph-mon[36597]: pgmap v1219: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:29.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:29 smithi151 ceph-mon[36481]: pgmap v1219: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:29.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:29 smithi045 ceph-mon[31773]: pgmap v1219: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:30.135 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2042 sec 2024-03-28T18:26:31.135 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2043 sec 2024-03-28T18:26:31.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1399.269881 secs 2024-03-28T18:26:31.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1399.269881 secs 2024-03-28T18:26:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1399.269881 secs 2024-03-28T18:26:32.135 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2044 sec 2024-03-28T18:26:32.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:32 smithi045 ceph-mon[31773]: pgmap v1220: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:32.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:32 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:26:32.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 690.693829 secs 2024-03-28T18:26:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:32 smithi071 ceph-mon[36597]: pgmap v1220: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:32 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:26:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 690.693829 secs 2024-03-28T18:26:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:32 smithi151 ceph-mon[36481]: pgmap v1220: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:32 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:26:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 690.693829 secs 2024-03-28T18:26:33.135 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2045 sec 2024-03-28T18:26:33.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:33 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:26:33.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:33 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:26:33.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:33 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:26:33.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:33 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:26:33.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:33 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:26:33.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:33 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:26:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:33 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:26:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:33 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:26:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:33 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:26:34.136 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2046 sec 2024-03-28T18:26:34.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:34 smithi045 ceph-mon[31773]: pgmap v1221: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:34.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:34 smithi045 ceph-mon[31773]: pgmap v1222: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:26:34.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:34 smithi071 ceph-mon[36597]: pgmap v1221: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:34.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:34 smithi071 ceph-mon[36597]: pgmap v1222: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:26:34.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:34 smithi151 ceph-mon[36481]: pgmap v1221: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:34.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:34 smithi151 ceph-mon[36481]: pgmap v1222: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:26:35.136 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2047 sec 2024-03-28T18:26:36.136 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2048 sec 2024-03-28T18:26:36.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:36 smithi045 ceph-mon[31773]: pgmap v1223: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:26:36.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1404.269944 secs 2024-03-28T18:26:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:36 smithi071 ceph-mon[36597]: pgmap v1223: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:26:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1404.269944 secs 2024-03-28T18:26:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:36 smithi151 ceph-mon[36481]: pgmap v1223: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:26:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1404.269944 secs 2024-03-28T18:26:37.136 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2049 sec 2024-03-28T18:26:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 695.693900 secs 2024-03-28T18:26:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 695.693900 secs 2024-03-28T18:26:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 695.693900 secs 2024-03-28T18:26:38.136 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2050 sec 2024-03-28T18:26:38.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:38 smithi045 ceph-mon[31773]: pgmap v1224: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:26:38.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:38 smithi071 ceph-mon[36597]: pgmap v1224: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:26:38.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:38 smithi151 ceph-mon[36481]: pgmap v1224: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:26:39.136 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2051 sec 2024-03-28T18:26:40.136 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2052 sec 2024-03-28T18:26:40.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:40 smithi045 ceph-mon[31773]: pgmap v1225: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:26:40.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:40 smithi071 ceph-mon[36597]: pgmap v1225: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:26:40.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:40 smithi151 ceph-mon[36481]: pgmap v1225: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:26:41.136 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2053 sec 2024-03-28T18:26:41.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1409.270035 secs 2024-03-28T18:26:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1409.270035 secs 2024-03-28T18:26:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1409.270035 secs 2024-03-28T18:26:42.136 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2054 sec 2024-03-28T18:26:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:42 smithi045 ceph-mon[31773]: pgmap v1226: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:26:42.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 700.693963 secs 2024-03-28T18:26:42.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:42 smithi151 ceph-mon[36481]: pgmap v1226: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:26:42.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 700.693963 secs 2024-03-28T18:26:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:42 smithi071 ceph-mon[36597]: pgmap v1226: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:26:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 700.693963 secs 2024-03-28T18:26:43.137 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2055 sec 2024-03-28T18:26:44.137 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2056 sec 2024-03-28T18:26:44.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:44 smithi045 ceph-mon[31773]: pgmap v1227: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:26:44.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:44 smithi071 ceph-mon[36597]: pgmap v1227: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:26:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:44 smithi151 ceph-mon[36481]: pgmap v1227: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:26:45.137 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2057 sec 2024-03-28T18:26:46.137 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2058 sec 2024-03-28T18:26:46.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:46 smithi045 ceph-mon[31773]: pgmap v1228: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:46.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1414.270096 secs 2024-03-28T18:26:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:46 smithi071 ceph-mon[36597]: pgmap v1228: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1414.270096 secs 2024-03-28T18:26:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:46 smithi151 ceph-mon[36481]: pgmap v1228: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1414.270096 secs 2024-03-28T18:26:47.137 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2059 sec 2024-03-28T18:26:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 705.694022 secs 2024-03-28T18:26:47.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 705.694022 secs 2024-03-28T18:26:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 705.694022 secs 2024-03-28T18:26:48.137 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2060 sec 2024-03-28T18:26:48.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:48 smithi045 ceph-mon[31773]: pgmap v1229: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:48.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:48 smithi071 ceph-mon[36597]: pgmap v1229: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:48.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:48 smithi151 ceph-mon[36481]: pgmap v1229: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:49.137 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2061 sec 2024-03-28T18:26:50.137 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2062 sec 2024-03-28T18:26:50.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:50 smithi045 ceph-mon[31773]: pgmap v1230: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:50.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:50 smithi071 ceph-mon[36597]: pgmap v1230: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:50.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:50 smithi151 ceph-mon[36481]: pgmap v1230: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:51.138 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2063 sec 2024-03-28T18:26:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1419.270154 secs 2024-03-28T18:26:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1419.270154 secs 2024-03-28T18:26:51.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1419.270154 secs 2024-03-28T18:26:52.138 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2064 sec 2024-03-28T18:26:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:52 smithi071 ceph-mon[36597]: pgmap v1231: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 710.694112 secs 2024-03-28T18:26:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:52 smithi151 ceph-mon[36481]: pgmap v1231: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 710.694112 secs 2024-03-28T18:26:52.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:52 smithi045 ceph-mon[31773]: pgmap v1231: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:52.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 710.694112 secs 2024-03-28T18:26:53.138 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2065 sec 2024-03-28T18:26:54.138 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2066 sec 2024-03-28T18:26:54.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:54 smithi071 ceph-mon[36597]: pgmap v1232: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:54.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:54 smithi151 ceph-mon[36481]: pgmap v1232: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:54.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:54 smithi045 ceph-mon[31773]: pgmap v1232: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:55.138 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2067 sec 2024-03-28T18:26:56.138 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2068 sec 2024-03-28T18:26:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:56 smithi071 ceph-mon[36597]: pgmap v1233: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1424.270243 secs 2024-03-28T18:26:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:56 smithi151 ceph-mon[36481]: pgmap v1233: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1424.270243 secs 2024-03-28T18:26:56.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:56 smithi045 ceph-mon[31773]: pgmap v1233: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:26:56.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1424.270243 secs 2024-03-28T18:26:57.138 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2069 sec 2024-03-28T18:26:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 715.694174 secs 2024-03-28T18:26:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 715.694174 secs 2024-03-28T18:26:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 715.694174 secs 2024-03-28T18:26:58.138 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2070 sec 2024-03-28T18:26:58.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:26:58 smithi071 ceph-mon[36597]: pgmap v1234: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:58.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:26:58 smithi151 ceph-mon[36481]: pgmap v1234: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:58.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:26:58 smithi045 ceph-mon[31773]: pgmap v1234: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:26:59.139 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2071 sec 2024-03-28T18:27:00.139 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2072 sec 2024-03-28T18:27:00.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:00 smithi071 ceph-mon[36597]: pgmap v1235: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:00.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:00 smithi151 ceph-mon[36481]: pgmap v1235: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:00.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:00 smithi045 ceph-mon[31773]: pgmap v1235: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:01.139 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2073 sec 2024-03-28T18:27:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1429.270302 secs 2024-03-28T18:27:01.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1429.270302 secs 2024-03-28T18:27:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1429.270302 secs 2024-03-28T18:27:02.139 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2074 sec 2024-03-28T18:27:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:02 smithi071 ceph-mon[36597]: pgmap v1236: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:02 smithi071 ceph-mon[36597]: 2 slow requests, 1 included below; oldest blocked for > 720.694236 secs 2024-03-28T18:27:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:02 smithi071 ceph-mon[36597]: slow request 720.694236 seconds old, received at 2024-03-28T18:15:00.947128+0000: client_request(client.26687:1 getattr pAsLsXsFs #0x1 2024-03-28T18:15:00.943510+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:27:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:02 smithi151 ceph-mon[36481]: pgmap v1236: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:02 smithi151 ceph-mon[36481]: 2 slow requests, 1 included below; oldest blocked for > 720.694236 secs 2024-03-28T18:27:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:02 smithi151 ceph-mon[36481]: slow request 720.694236 seconds old, received at 2024-03-28T18:15:00.947128+0000: client_request(client.26687:1 getattr pAsLsXsFs #0x1 2024-03-28T18:15:00.943510+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:27:02.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:02 smithi045 ceph-mon[31773]: pgmap v1236: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:02.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:02 smithi045 ceph-mon[31773]: 2 slow requests, 1 included below; oldest blocked for > 720.694236 secs 2024-03-28T18:27:02.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:02 smithi045 ceph-mon[31773]: slow request 720.694236 seconds old, received at 2024-03-28T18:15:00.947128+0000: client_request(client.26687:1 getattr pAsLsXsFs #0x1 2024-03-28T18:15:00.943510+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:27:03.139 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2075 sec 2024-03-28T18:27:04.139 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2076 sec 2024-03-28T18:27:04.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:04 smithi071 ceph-mon[36597]: pgmap v1237: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:04.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:04 smithi151 ceph-mon[36481]: pgmap v1237: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:04.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:04 smithi045 ceph-mon[31773]: pgmap v1237: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:05.139 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2077 sec 2024-03-28T18:27:06.140 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2078 sec 2024-03-28T18:27:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:06 smithi071 ceph-mon[36597]: pgmap v1238: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1434.270340 secs 2024-03-28T18:27:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:06 smithi151 ceph-mon[36481]: pgmap v1238: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1434.270340 secs 2024-03-28T18:27:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:06 smithi045 ceph-mon[31773]: pgmap v1238: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1434.270340 secs 2024-03-28T18:27:07.140 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2079 sec 2024-03-28T18:27:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 725.694297 secs 2024-03-28T18:27:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 725.694297 secs 2024-03-28T18:27:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 725.694297 secs 2024-03-28T18:27:08.140 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2080 sec 2024-03-28T18:27:08.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:08 smithi045 ceph-mon[31773]: pgmap v1239: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:08.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:08 smithi071 ceph-mon[36597]: pgmap v1239: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:08.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:08 smithi151 ceph-mon[36481]: pgmap v1239: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:09.140 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2081 sec 2024-03-28T18:27:09.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:09 smithi071 ceph-mon[36597]: pgmap v1240: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:09.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:09 smithi151 ceph-mon[36481]: pgmap v1240: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:09.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:09 smithi045 ceph-mon[31773]: pgmap v1240: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:10.140 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2082 sec 2024-03-28T18:27:11.140 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2083 sec 2024-03-28T18:27:12.140 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2084 sec 2024-03-28T18:27:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:11 smithi045 ceph-mon[31773]: pgmap v1241: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1439.270401 secs 2024-03-28T18:27:12.276 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:11 smithi151 ceph-mon[36481]: pgmap v1241: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:12.276 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1439.270401 secs 2024-03-28T18:27:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:11 smithi071 ceph-mon[36597]: pgmap v1241: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1439.270401 secs 2024-03-28T18:27:13.141 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2085 sec 2024-03-28T18:27:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 730.694357 secs 2024-03-28T18:27:13.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 730.694357 secs 2024-03-28T18:27:13.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 730.694357 secs 2024-03-28T18:27:14.141 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2086 sec 2024-03-28T18:27:14.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:13 smithi045 ceph-mon[31773]: pgmap v1242: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:14.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:13 smithi071 ceph-mon[36597]: pgmap v1242: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:14.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:13 smithi151 ceph-mon[36481]: pgmap v1242: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:15.141 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2087 sec 2024-03-28T18:27:16.141 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2088 sec 2024-03-28T18:27:16.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:15 smithi045 ceph-mon[31773]: pgmap v1243: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:16.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:15 smithi071 ceph-mon[36597]: pgmap v1243: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:16.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:15 smithi151 ceph-mon[36481]: pgmap v1243: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:17.141 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2089 sec 2024-03-28T18:27:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:16 smithi045 ceph-mon[31773]: 2 slow requests, 2 included below; oldest blocked for > 1444.270504 secs 2024-03-28T18:27:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:16 smithi045 ceph-mon[31773]: slow request 1444.270504 seconds old, received at 2024-03-28T18:03:11.560545+0000: peer_request:client.15069:1629 currently acquired locks 2024-03-28T18:27:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:16 smithi045 ceph-mon[31773]: slow request 1444.174379 seconds old, received at 2024-03-28T18:03:11.656670+0000: client_request(client.25691:164 rmsnap #0x1//scheduled-2024-03-28-17_57_00_UTC 2024-03-28T18:03:11.656797+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:27:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 735.694416 secs 2024-03-28T18:27:17.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:16 smithi071 ceph-mon[36597]: 2 slow requests, 2 included below; oldest blocked for > 1444.270504 secs 2024-03-28T18:27:17.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:16 smithi071 ceph-mon[36597]: slow request 1444.270504 seconds old, received at 2024-03-28T18:03:11.560545+0000: peer_request:client.15069:1629 currently acquired locks 2024-03-28T18:27:17.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:16 smithi071 ceph-mon[36597]: slow request 1444.174379 seconds old, received at 2024-03-28T18:03:11.656670+0000: client_request(client.25691:164 rmsnap #0x1//scheduled-2024-03-28-17_57_00_UTC 2024-03-28T18:03:11.656797+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:27:17.212 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 735.694416 secs 2024-03-28T18:27:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:16 smithi151 ceph-mon[36481]: 2 slow requests, 2 included below; oldest blocked for > 1444.270504 secs 2024-03-28T18:27:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:16 smithi151 ceph-mon[36481]: slow request 1444.270504 seconds old, received at 2024-03-28T18:03:11.560545+0000: peer_request:client.15069:1629 currently acquired locks 2024-03-28T18:27:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:16 smithi151 ceph-mon[36481]: slow request 1444.174379 seconds old, received at 2024-03-28T18:03:11.656670+0000: client_request(client.25691:164 rmsnap #0x1//scheduled-2024-03-28-17_57_00_UTC 2024-03-28T18:03:11.656797+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:27:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 735.694416 secs 2024-03-28T18:27:18.141 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2090 sec 2024-03-28T18:27:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:17 smithi045 ceph-mon[31773]: pgmap v1244: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:18.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:17 smithi071 ceph-mon[36597]: pgmap v1244: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:18.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:17 smithi151 ceph-mon[36481]: pgmap v1244: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:19.142 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2091 sec 2024-03-28T18:27:20.142 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2092 sec 2024-03-28T18:27:20.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:19 smithi045 ceph-mon[31773]: pgmap v1245: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:20.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:19 smithi071 ceph-mon[36597]: pgmap v1245: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:20.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:19 smithi151 ceph-mon[36481]: pgmap v1245: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:21.142 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2093 sec 2024-03-28T18:27:22.142 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2094 sec 2024-03-28T18:27:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:21 smithi045 ceph-mon[31773]: pgmap v1246: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1449.270565 secs 2024-03-28T18:27:22.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 740.694506 secs 2024-03-28T18:27:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:21 smithi071 ceph-mon[36597]: pgmap v1246: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1449.270565 secs 2024-03-28T18:27:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 740.694506 secs 2024-03-28T18:27:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:21 smithi151 ceph-mon[36481]: pgmap v1246: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1449.270565 secs 2024-03-28T18:27:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 740.694506 secs 2024-03-28T18:27:23.142 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2095 sec 2024-03-28T18:27:24.142 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2096 sec 2024-03-28T18:27:24.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:23 smithi045 ceph-mon[31773]: pgmap v1247: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:24.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:23 smithi071 ceph-mon[36597]: pgmap v1247: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:24.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:23 smithi151 ceph-mon[36481]: pgmap v1247: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:25.142 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2097 sec 2024-03-28T18:27:26.143 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2098 sec 2024-03-28T18:27:26.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:25 smithi045 ceph-mon[31773]: pgmap v1248: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:26.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:25 smithi071 ceph-mon[36597]: pgmap v1248: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:25 smithi151 ceph-mon[36481]: pgmap v1248: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:27.142 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2099 sec 2024-03-28T18:27:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1454.270625 secs 2024-03-28T18:27:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 745.694596 secs 2024-03-28T18:27:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1454.270625 secs 2024-03-28T18:27:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 745.694596 secs 2024-03-28T18:27:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1454.270625 secs 2024-03-28T18:27:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 745.694596 secs 2024-03-28T18:27:28.143 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2100 sec 2024-03-28T18:27:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:27 smithi045 ceph-mon[31773]: pgmap v1249: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:28.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:27 smithi071 ceph-mon[36597]: pgmap v1249: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:28.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:27 smithi151 ceph-mon[36481]: pgmap v1249: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:29.143 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2101 sec 2024-03-28T18:27:30.143 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2102 sec 2024-03-28T18:27:30.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:29 smithi045 ceph-mon[31773]: pgmap v1250: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:30.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:29 smithi071 ceph-mon[36597]: pgmap v1250: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:30.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:29 smithi151 ceph-mon[36481]: pgmap v1250: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:31.143 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2103 sec 2024-03-28T18:27:32.143 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2104 sec 2024-03-28T18:27:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:31 smithi071 ceph-mon[36597]: pgmap v1251: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1459.270685 secs 2024-03-28T18:27:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 750.694727 secs 2024-03-28T18:27:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:31 smithi151 ceph-mon[36481]: pgmap v1251: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1459.270685 secs 2024-03-28T18:27:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 750.694727 secs 2024-03-28T18:27:32.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:31 smithi045 ceph-mon[31773]: pgmap v1251: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:32.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1459.270685 secs 2024-03-28T18:27:32.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 750.694727 secs 2024-03-28T18:27:33.143 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2105 sec 2024-03-28T18:27:34.143 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2106 sec 2024-03-28T18:27:34.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:33 smithi071 ceph-mon[36597]: pgmap v1252: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:34.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:33 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:27:34.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:33 smithi151 ceph-mon[36481]: pgmap v1252: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:34.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:33 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:27:34.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:33 smithi045 ceph-mon[31773]: pgmap v1252: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:34.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:33 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:27:35.144 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2107 sec 2024-03-28T18:27:35.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:34 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:27:35.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:34 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:27:35.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:34 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:27:35.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:34 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:27:35.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:34 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:27:35.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:34 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:27:35.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:34 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:27:35.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:34 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:27:35.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:34 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:27:36.144 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2108 sec 2024-03-28T18:27:36.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:35 smithi071 ceph-mon[36597]: pgmap v1253: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:36.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:35 smithi071 ceph-mon[36597]: pgmap v1254: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:27:36.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:35 smithi151 ceph-mon[36481]: pgmap v1253: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:36.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:35 smithi151 ceph-mon[36481]: pgmap v1254: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:27:36.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:35 smithi045 ceph-mon[31773]: pgmap v1253: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:36.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:35 smithi045 ceph-mon[31773]: pgmap v1254: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:27:37.144 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2109 sec 2024-03-28T18:27:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1464.270745 secs 2024-03-28T18:27:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 755.694786 secs 2024-03-28T18:27:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1464.270745 secs 2024-03-28T18:27:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 755.694786 secs 2024-03-28T18:27:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1464.270745 secs 2024-03-28T18:27:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 755.694786 secs 2024-03-28T18:27:38.144 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2110 sec 2024-03-28T18:27:38.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:37 smithi045 ceph-mon[31773]: pgmap v1255: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:27:38.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:37 smithi071 ceph-mon[36597]: pgmap v1255: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:27:38.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:37 smithi151 ceph-mon[36481]: pgmap v1255: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:27:39.144 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2111 sec 2024-03-28T18:27:40.144 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2112 sec 2024-03-28T18:27:40.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:39 smithi071 ceph-mon[36597]: pgmap v1256: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:27:40.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:39 smithi151 ceph-mon[36481]: pgmap v1256: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:27:40.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:39 smithi045 ceph-mon[31773]: pgmap v1256: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:27:41.145 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2113 sec 2024-03-28T18:27:42.145 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2114 sec 2024-03-28T18:27:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:41 smithi071 ceph-mon[36597]: pgmap v1257: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:27:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1469.270810 secs 2024-03-28T18:27:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 760.694846 secs 2024-03-28T18:27:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:41 smithi151 ceph-mon[36481]: pgmap v1257: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:27:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1469.270810 secs 2024-03-28T18:27:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 760.694846 secs 2024-03-28T18:27:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:41 smithi045 ceph-mon[31773]: pgmap v1257: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:27:42.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1469.270810 secs 2024-03-28T18:27:42.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 760.694846 secs 2024-03-28T18:27:43.145 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2115 sec 2024-03-28T18:27:44.145 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2116 sec 2024-03-28T18:27:44.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:43 smithi071 ceph-mon[36597]: pgmap v1258: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:27:44.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:43 smithi151 ceph-mon[36481]: pgmap v1258: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:27:44.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:43 smithi045 ceph-mon[31773]: pgmap v1258: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:27:45.145 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2117 sec 2024-03-28T18:27:46.145 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2118 sec 2024-03-28T18:27:46.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:45 smithi071 ceph-mon[36597]: pgmap v1259: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:27:46.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:45 smithi151 ceph-mon[36481]: pgmap v1259: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:27:46.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:45 smithi045 ceph-mon[31773]: pgmap v1259: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:27:47.145 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2119 sec 2024-03-28T18:27:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1474.270846 secs 2024-03-28T18:27:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 765.694935 secs 2024-03-28T18:27:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1474.270846 secs 2024-03-28T18:27:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 765.694935 secs 2024-03-28T18:27:47.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1474.270846 secs 2024-03-28T18:27:47.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 765.694935 secs 2024-03-28T18:27:48.145 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2120 sec 2024-03-28T18:27:48.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:47 smithi071 ceph-mon[36597]: pgmap v1260: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:47 smithi151 ceph-mon[36481]: pgmap v1260: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:48.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:47 smithi045 ceph-mon[31773]: pgmap v1260: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:49.146 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2121 sec 2024-03-28T18:27:50.146 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2122 sec 2024-03-28T18:27:50.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:49 smithi071 ceph-mon[36597]: pgmap v1261: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:50.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:49 smithi151 ceph-mon[36481]: pgmap v1261: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:50.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:49 smithi045 ceph-mon[31773]: pgmap v1261: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:51.146 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2123 sec 2024-03-28T18:27:52.146 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2124 sec 2024-03-28T18:27:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:51 smithi071 ceph-mon[36597]: pgmap v1262: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1479.270934 secs 2024-03-28T18:27:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 770.695021 secs 2024-03-28T18:27:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:51 smithi151 ceph-mon[36481]: pgmap v1262: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1479.270934 secs 2024-03-28T18:27:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 770.695021 secs 2024-03-28T18:27:52.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:51 smithi045 ceph-mon[31773]: pgmap v1262: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:52.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1479.270934 secs 2024-03-28T18:27:52.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 770.695021 secs 2024-03-28T18:27:53.146 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2125 sec 2024-03-28T18:27:54.146 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2126 sec 2024-03-28T18:27:54.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:53 smithi071 ceph-mon[36597]: pgmap v1263: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:54.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:53 smithi151 ceph-mon[36481]: pgmap v1263: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:54.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:53 smithi045 ceph-mon[31773]: pgmap v1263: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:55.146 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2127 sec 2024-03-28T18:27:56.147 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2128 sec 2024-03-28T18:27:56.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:56 smithi071 ceph-mon[36597]: pgmap v1264: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:56.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:56 smithi151 ceph-mon[36481]: pgmap v1264: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:56.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:56 smithi045 ceph-mon[31773]: pgmap v1264: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:27:57.147 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2129 sec 2024-03-28T18:27:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1484.270974 secs 2024-03-28T18:27:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 775.695060 secs 2024-03-28T18:27:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1484.270974 secs 2024-03-28T18:27:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 775.695060 secs 2024-03-28T18:27:57.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1484.270974 secs 2024-03-28T18:27:57.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 775.695060 secs 2024-03-28T18:27:58.147 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2130 sec 2024-03-28T18:27:58.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:27:58 smithi071 ceph-mon[36597]: pgmap v1265: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:58.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:27:58 smithi151 ceph-mon[36481]: pgmap v1265: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:58.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:27:58 smithi045 ceph-mon[31773]: pgmap v1265: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:27:59.147 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2131 sec 2024-03-28T18:28:00.147 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2132 sec 2024-03-28T18:28:00.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:00 smithi071 ceph-mon[36597]: pgmap v1266: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:00.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:00 smithi151 ceph-mon[36481]: pgmap v1266: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:00.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:00 smithi045 ceph-mon[31773]: pgmap v1266: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:01.148 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2133 sec 2024-03-28T18:28:02.148 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2134 sec 2024-03-28T18:28:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:02 smithi071 ceph-mon[36597]: pgmap v1267: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1489.271034 secs 2024-03-28T18:28:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 780.695145 secs 2024-03-28T18:28:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:02 smithi151 ceph-mon[36481]: pgmap v1267: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1489.271034 secs 2024-03-28T18:28:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 780.695145 secs 2024-03-28T18:28:02.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:02 smithi045 ceph-mon[31773]: pgmap v1267: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:02.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1489.271034 secs 2024-03-28T18:28:02.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 780.695145 secs 2024-03-28T18:28:03.148 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2135 sec 2024-03-28T18:28:04.148 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2136 sec 2024-03-28T18:28:04.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:04 smithi071 ceph-mon[36597]: pgmap v1268: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:04.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:04 smithi151 ceph-mon[36481]: pgmap v1268: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:04.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:04 smithi045 ceph-mon[31773]: pgmap v1268: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:05.148 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2137 sec 2024-03-28T18:28:06.148 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2138 sec 2024-03-28T18:28:06.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:06 smithi071 ceph-mon[36597]: pgmap v1269: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:06.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1494.271121 secs 2024-03-28T18:28:06.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:06 smithi151 ceph-mon[36481]: pgmap v1269: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:06.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1494.271121 secs 2024-03-28T18:28:06.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:06 smithi045 ceph-mon[31773]: pgmap v1269: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:06.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1494.271121 secs 2024-03-28T18:28:07.148 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2139 sec 2024-03-28T18:28:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 785.695203 secs 2024-03-28T18:28:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 785.695203 secs 2024-03-28T18:28:07.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 785.695203 secs 2024-03-28T18:28:08.148 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2140 sec 2024-03-28T18:28:08.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:08 smithi071 ceph-mon[36597]: pgmap v1270: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:08.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:08 smithi151 ceph-mon[36481]: pgmap v1270: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:08.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:08 smithi045 ceph-mon[31773]: pgmap v1270: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:09.149 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2141 sec 2024-03-28T18:28:10.149 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2142 sec 2024-03-28T18:28:10.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:10 smithi071 ceph-mon[36597]: pgmap v1271: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:10.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:10 smithi151 ceph-mon[36481]: pgmap v1271: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:10.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:10 smithi045 ceph-mon[31773]: pgmap v1271: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:11.149 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2143 sec 2024-03-28T18:28:11.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1499.271209 secs 2024-03-28T18:28:11.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1499.271209 secs 2024-03-28T18:28:11.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1499.271209 secs 2024-03-28T18:28:12.149 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2144 sec 2024-03-28T18:28:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:12 smithi151 ceph-mon[36481]: pgmap v1272: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 790.695266 secs 2024-03-28T18:28:12.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:12 smithi045 ceph-mon[31773]: pgmap v1272: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:12.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 790.695266 secs 2024-03-28T18:28:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:12 smithi071 ceph-mon[36597]: pgmap v1272: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 790.695266 secs 2024-03-28T18:28:13.149 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2145 sec 2024-03-28T18:28:14.149 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2146 sec 2024-03-28T18:28:14.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:14 smithi151 ceph-mon[36481]: pgmap v1273: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:14.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:14 smithi045 ceph-mon[31773]: pgmap v1273: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:14.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:14 smithi071 ceph-mon[36597]: pgmap v1273: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:15.150 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2147 sec 2024-03-28T18:28:16.150 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2148 sec 2024-03-28T18:28:16.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:16 smithi151 ceph-mon[36481]: pgmap v1274: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:16.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1504.271299 secs 2024-03-28T18:28:16.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:16 smithi045 ceph-mon[31773]: pgmap v1274: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:16.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1504.271299 secs 2024-03-28T18:28:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:16 smithi071 ceph-mon[36597]: pgmap v1274: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1504.271299 secs 2024-03-28T18:28:17.150 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2149 sec 2024-03-28T18:28:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 795.695355 secs 2024-03-28T18:28:17.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 795.695355 secs 2024-03-28T18:28:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 795.695355 secs 2024-03-28T18:28:18.150 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2150 sec 2024-03-28T18:28:18.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:18 smithi045 ceph-mon[31773]: pgmap v1275: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:18.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:18 smithi071 ceph-mon[36597]: pgmap v1275: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:18 smithi151 ceph-mon[36481]: pgmap v1275: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:19.150 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2151 sec 2024-03-28T18:28:20.150 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2152 sec 2024-03-28T18:28:20.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:20 smithi045 ceph-mon[31773]: pgmap v1276: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:20.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:20 smithi071 ceph-mon[36597]: pgmap v1276: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:20.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:20 smithi151 ceph-mon[36481]: pgmap v1276: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:21.150 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2153 sec 2024-03-28T18:28:21.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1509.271360 secs 2024-03-28T18:28:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1509.271360 secs 2024-03-28T18:28:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1509.271360 secs 2024-03-28T18:28:22.151 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2154 sec 2024-03-28T18:28:22.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:22 smithi045 ceph-mon[31773]: pgmap v1277: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:22.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 800.695438 secs 2024-03-28T18:28:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:22 smithi071 ceph-mon[36597]: pgmap v1277: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 800.695438 secs 2024-03-28T18:28:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:22 smithi151 ceph-mon[36481]: pgmap v1277: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 800.695438 secs 2024-03-28T18:28:23.151 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2155 sec 2024-03-28T18:28:24.151 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2156 sec 2024-03-28T18:28:24.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:24 smithi045 ceph-mon[31773]: pgmap v1278: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:24.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:24 smithi071 ceph-mon[36597]: pgmap v1278: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:24.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:24 smithi151 ceph-mon[36481]: pgmap v1278: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:25.151 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2157 sec 2024-03-28T18:28:26.151 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2158 sec 2024-03-28T18:28:26.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:26 smithi045 ceph-mon[31773]: pgmap v1279: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:26.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1514.271453 secs 2024-03-28T18:28:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:26 smithi071 ceph-mon[36597]: pgmap v1279: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1514.271453 secs 2024-03-28T18:28:26.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:26 smithi151 ceph-mon[36481]: pgmap v1279: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:26.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1514.271453 secs 2024-03-28T18:28:27.151 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2159 sec 2024-03-28T18:28:27.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 805.695527 secs 2024-03-28T18:28:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 805.695527 secs 2024-03-28T18:28:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 805.695527 secs 2024-03-28T18:28:28.151 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2160 sec 2024-03-28T18:28:28.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:28 smithi045 ceph-mon[31773]: pgmap v1280: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:28.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:28 smithi071 ceph-mon[36597]: pgmap v1280: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:28.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:28 smithi151 ceph-mon[36481]: pgmap v1280: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:29.152 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2161 sec 2024-03-28T18:28:30.152 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2162 sec 2024-03-28T18:28:30.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:30 smithi045 ceph-mon[31773]: pgmap v1281: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:30.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:30 smithi071 ceph-mon[36597]: pgmap v1281: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:30.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:30 smithi151 ceph-mon[36481]: pgmap v1281: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:31.152 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2163 sec 2024-03-28T18:28:31.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1519.271540 secs 2024-03-28T18:28:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1519.271540 secs 2024-03-28T18:28:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1519.271540 secs 2024-03-28T18:28:32.152 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2164 sec 2024-03-28T18:28:32.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:32 smithi045 ceph-mon[31773]: pgmap v1282: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:32.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 810.695615 secs 2024-03-28T18:28:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:32 smithi071 ceph-mon[36597]: pgmap v1282: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 810.695615 secs 2024-03-28T18:28:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:32 smithi151 ceph-mon[36481]: pgmap v1282: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 810.695615 secs 2024-03-28T18:28:33.152 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2165 sec 2024-03-28T18:28:34.152 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2166 sec 2024-03-28T18:28:34.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:34 smithi045 ceph-mon[31773]: pgmap v1283: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:34.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:34 smithi071 ceph-mon[36597]: pgmap v1283: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:34.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:34 smithi151 ceph-mon[36481]: pgmap v1283: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:35.152 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2167 sec 2024-03-28T18:28:35.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:35 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:28:35.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:35 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:28:35.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:35 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:28:36.153 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2168 sec 2024-03-28T18:28:36.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:36 smithi045 ceph-mon[31773]: pgmap v1284: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:36.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1524.271557 secs 2024-03-28T18:28:36.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:36 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:28:36.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:36 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:28:36.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:36 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:28:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:36 smithi071 ceph-mon[36597]: pgmap v1284: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1524.271557 secs 2024-03-28T18:28:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:36 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:28:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:36 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:28:36.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:36 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:28:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:36 smithi151 ceph-mon[36481]: pgmap v1284: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1524.271557 secs 2024-03-28T18:28:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:36 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:28:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:36 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:28:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:36 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:28:37.153 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2169 sec 2024-03-28T18:28:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:37 smithi071 ceph-mon[36597]: pgmap v1285: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:28:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:37 smithi071 ceph-mon[36597]: pgmap v1286: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:28:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 815.695677 secs 2024-03-28T18:28:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:37 smithi151 ceph-mon[36481]: pgmap v1285: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:28:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:37 smithi151 ceph-mon[36481]: pgmap v1286: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:28:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 815.695677 secs 2024-03-28T18:28:37.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:37 smithi045 ceph-mon[31773]: pgmap v1285: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:28:37.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:37 smithi045 ceph-mon[31773]: pgmap v1286: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:28:37.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 815.695677 secs 2024-03-28T18:28:38.153 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2170 sec 2024-03-28T18:28:39.153 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2171 sec 2024-03-28T18:28:39.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:39 smithi071 ceph-mon[36597]: pgmap v1287: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:28:39.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:39 smithi151 ceph-mon[36481]: pgmap v1287: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:28:39.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:39 smithi045 ceph-mon[31773]: pgmap v1287: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:28:40.153 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2172 sec 2024-03-28T18:28:41.153 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2173 sec 2024-03-28T18:28:41.469 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:41 smithi045 ceph-mon[31773]: pgmap v1288: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:28:41.470 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1529.271621 secs 2024-03-28T18:28:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:41 smithi071 ceph-mon[36597]: pgmap v1288: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:28:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1529.271621 secs 2024-03-28T18:28:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:41 smithi151 ceph-mon[36481]: pgmap v1288: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:28:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1529.271621 secs 2024-03-28T18:28:42.154 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2174 sec 2024-03-28T18:28:42.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 820.695737 secs 2024-03-28T18:28:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 820.695737 secs 2024-03-28T18:28:42.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 820.695737 secs 2024-03-28T18:28:43.153 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2175 sec 2024-03-28T18:28:43.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:43 smithi071 ceph-mon[36597]: pgmap v1289: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:28:43.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:43 smithi151 ceph-mon[36481]: pgmap v1289: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:28:43.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:43 smithi045 ceph-mon[31773]: pgmap v1289: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:28:44.154 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2176 sec 2024-03-28T18:28:45.154 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2177 sec 2024-03-28T18:28:45.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:45 smithi071 ceph-mon[36597]: pgmap v1290: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:28:45.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:45 smithi151 ceph-mon[36481]: pgmap v1290: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:28:45.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:45 smithi045 ceph-mon[31773]: pgmap v1290: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:28:46.154 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2178 sec 2024-03-28T18:28:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1534.271684 secs 2024-03-28T18:28:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1534.271684 secs 2024-03-28T18:28:46.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1534.271684 secs 2024-03-28T18:28:47.154 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2179 sec 2024-03-28T18:28:47.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:47 smithi071 ceph-mon[36597]: pgmap v1291: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:28:47.462 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 825.695828 secs 2024-03-28T18:28:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:47 smithi151 ceph-mon[36481]: pgmap v1291: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:28:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 825.695828 secs 2024-03-28T18:28:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:47 smithi045 ceph-mon[31773]: pgmap v1291: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:28:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 825.695828 secs 2024-03-28T18:28:48.154 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2180 sec 2024-03-28T18:28:48.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:48 smithi071 ceph-mon[36597]: pgmap v1292: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:48 smithi151 ceph-mon[36481]: pgmap v1292: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:48.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:48 smithi045 ceph-mon[31773]: pgmap v1292: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:49.154 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2181 sec 2024-03-28T18:28:50.154 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2182 sec 2024-03-28T18:28:51.155 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2183 sec 2024-03-28T18:28:51.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:51 smithi071 ceph-mon[36597]: pgmap v1293: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:51.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:51 smithi151 ceph-mon[36481]: pgmap v1293: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:51.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:51 smithi045 ceph-mon[31773]: pgmap v1293: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:52.155 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2184 sec 2024-03-28T18:28:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1539.271720 secs 2024-03-28T18:28:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 830.695901 secs 2024-03-28T18:28:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1539.271720 secs 2024-03-28T18:28:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 830.695901 secs 2024-03-28T18:28:52.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1539.271720 secs 2024-03-28T18:28:52.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 830.695901 secs 2024-03-28T18:28:53.155 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2185 sec 2024-03-28T18:28:53.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:53 smithi071 ceph-mon[36597]: pgmap v1294: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:53.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:53 smithi151 ceph-mon[36481]: pgmap v1294: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:53.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:53 smithi045 ceph-mon[31773]: pgmap v1294: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:54.155 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2186 sec 2024-03-28T18:28:55.156 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2187 sec 2024-03-28T18:28:55.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:55 smithi071 ceph-mon[36597]: pgmap v1295: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:55.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:55 smithi151 ceph-mon[36481]: pgmap v1295: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:55.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:55 smithi045 ceph-mon[31773]: pgmap v1295: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:56.155 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2188 sec 2024-03-28T18:28:57.156 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2189 sec 2024-03-28T18:28:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1544.271782 secs 2024-03-28T18:28:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:57 smithi071 ceph-mon[36597]: pgmap v1296: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 835.695916 secs 2024-03-28T18:28:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1544.271782 secs 2024-03-28T18:28:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:57 smithi151 ceph-mon[36481]: pgmap v1296: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 835.695916 secs 2024-03-28T18:28:57.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1544.271782 secs 2024-03-28T18:28:57.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:57 smithi045 ceph-mon[31773]: pgmap v1296: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:28:57.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 835.695916 secs 2024-03-28T18:28:58.156 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2190 sec 2024-03-28T18:28:58.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:28:58 smithi071 ceph-mon[36597]: pgmap v1297: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:58.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:28:58 smithi151 ceph-mon[36481]: pgmap v1297: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:58.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:28:58 smithi045 ceph-mon[31773]: pgmap v1297: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:28:59.156 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2191 sec 2024-03-28T18:29:00.156 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2192 sec 2024-03-28T18:29:01.156 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2193 sec 2024-03-28T18:29:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:01 smithi071 ceph-mon[36597]: pgmap v1298: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:01 smithi151 ceph-mon[36481]: pgmap v1298: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:01.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:01 smithi045 ceph-mon[31773]: pgmap v1298: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:02.156 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2194 sec 2024-03-28T18:29:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1549.271845 secs 2024-03-28T18:29:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 840.695978 secs 2024-03-28T18:29:02.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1549.271845 secs 2024-03-28T18:29:02.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 840.695978 secs 2024-03-28T18:29:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1549.271845 secs 2024-03-28T18:29:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 840.695978 secs 2024-03-28T18:29:03.156 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2195 sec 2024-03-28T18:29:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:03 smithi071 ceph-mon[36597]: pgmap v1299: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:03.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:03 smithi151 ceph-mon[36481]: pgmap v1299: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:03.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:03 smithi045 ceph-mon[31773]: pgmap v1299: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:04.156 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2196 sec 2024-03-28T18:29:05.157 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2197 sec 2024-03-28T18:29:05.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:05 smithi071 ceph-mon[36597]: pgmap v1300: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:05.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:05 smithi151 ceph-mon[36481]: pgmap v1300: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:05.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:05 smithi045 ceph-mon[31773]: pgmap v1300: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:06.157 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2198 sec 2024-03-28T18:29:07.157 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2199 sec 2024-03-28T18:29:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1554.271906 secs 2024-03-28T18:29:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:07 smithi071 ceph-mon[36597]: pgmap v1301: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:07.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 845.696068 secs 2024-03-28T18:29:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1554.271906 secs 2024-03-28T18:29:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:07 smithi151 ceph-mon[36481]: pgmap v1301: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 845.696068 secs 2024-03-28T18:29:07.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1554.271906 secs 2024-03-28T18:29:07.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:07 smithi045 ceph-mon[31773]: pgmap v1301: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:07.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 845.696068 secs 2024-03-28T18:29:08.157 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2200 sec 2024-03-28T18:29:09.157 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2201 sec 2024-03-28T18:29:09.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:09 smithi071 ceph-mon[36597]: pgmap v1302: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:09.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:09 smithi151 ceph-mon[36481]: pgmap v1302: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:09.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:09 smithi045 ceph-mon[31773]: pgmap v1302: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:10.158 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2202 sec 2024-03-28T18:29:11.158 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2203 sec 2024-03-28T18:29:11.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:11 smithi071 ceph-mon[36597]: pgmap v1303: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:11.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1559.271965 secs 2024-03-28T18:29:11.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:11 smithi151 ceph-mon[36481]: pgmap v1303: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:11.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1559.271965 secs 2024-03-28T18:29:11.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:11 smithi045 ceph-mon[31773]: pgmap v1303: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:11.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1559.271965 secs 2024-03-28T18:29:12.158 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2204 sec 2024-03-28T18:29:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 850.696154 secs 2024-03-28T18:29:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 850.696154 secs 2024-03-28T18:29:12.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 850.696154 secs 2024-03-28T18:29:13.158 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2205 sec 2024-03-28T18:29:13.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:13 smithi071 ceph-mon[36597]: pgmap v1304: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:13.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:13 smithi045 ceph-mon[31773]: pgmap v1304: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:13.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:13 smithi151 ceph-mon[36481]: pgmap v1304: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:14.158 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2206 sec 2024-03-28T18:29:15.158 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2207 sec 2024-03-28T18:29:15.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:15 smithi071 ceph-mon[36597]: pgmap v1305: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:15.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:15 smithi151 ceph-mon[36481]: pgmap v1305: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:15.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:15 smithi045 ceph-mon[31773]: pgmap v1305: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:16.158 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2208 sec 2024-03-28T18:29:16.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1564.272052 secs 2024-03-28T18:29:16.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1564.272052 secs 2024-03-28T18:29:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1564.272052 secs 2024-03-28T18:29:17.158 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2209 sec 2024-03-28T18:29:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:17 smithi151 ceph-mon[36481]: pgmap v1306: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 855.696244 secs 2024-03-28T18:29:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:17 smithi045 ceph-mon[31773]: pgmap v1306: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 855.696244 secs 2024-03-28T18:29:17.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:17 smithi071 ceph-mon[36597]: pgmap v1306: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:17.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 855.696244 secs 2024-03-28T18:29:18.159 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2210 sec 2024-03-28T18:29:19.159 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2211 sec 2024-03-28T18:29:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:19 smithi151 ceph-mon[36481]: pgmap v1307: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:19.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:19 smithi045 ceph-mon[31773]: pgmap v1307: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:19.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:19 smithi071 ceph-mon[36597]: pgmap v1307: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:20.159 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2212 sec 2024-03-28T18:29:21.159 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2213 sec 2024-03-28T18:29:21.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:21 smithi045 ceph-mon[31773]: pgmap v1308: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1569.272113 secs 2024-03-28T18:29:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:21 smithi071 ceph-mon[36597]: pgmap v1308: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1569.272113 secs 2024-03-28T18:29:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:21 smithi151 ceph-mon[36481]: pgmap v1308: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1569.272113 secs 2024-03-28T18:29:22.159 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2214 sec 2024-03-28T18:29:22.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 860.696333 secs 2024-03-28T18:29:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 860.696333 secs 2024-03-28T18:29:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 860.696333 secs 2024-03-28T18:29:23.159 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2215 sec 2024-03-28T18:29:23.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:23 smithi045 ceph-mon[31773]: pgmap v1309: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:23.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:23 smithi071 ceph-mon[36597]: pgmap v1309: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:23.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:23 smithi151 ceph-mon[36481]: pgmap v1309: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:24.160 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2216 sec 2024-03-28T18:29:25.160 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2217 sec 2024-03-28T18:29:25.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:25 smithi045 ceph-mon[31773]: pgmap v1310: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:25.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:25 smithi071 ceph-mon[36597]: pgmap v1310: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:25.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:25 smithi151 ceph-mon[36481]: pgmap v1310: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:26.160 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2218 sec 2024-03-28T18:29:26.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1574.272200 secs 2024-03-28T18:29:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1574.272200 secs 2024-03-28T18:29:26.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1574.272200 secs 2024-03-28T18:29:27.160 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2219 sec 2024-03-28T18:29:27.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:27 smithi045 ceph-mon[31773]: pgmap v1311: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:27.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 865.696422 secs 2024-03-28T18:29:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:27 smithi071 ceph-mon[36597]: pgmap v1311: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 865.696422 secs 2024-03-28T18:29:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:27 smithi151 ceph-mon[36481]: pgmap v1311: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 865.696422 secs 2024-03-28T18:29:28.160 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2220 sec 2024-03-28T18:29:29.160 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2221 sec 2024-03-28T18:29:29.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:29 smithi045 ceph-mon[31773]: pgmap v1312: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:29.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:29 smithi071 ceph-mon[36597]: pgmap v1312: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:29.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:29 smithi151 ceph-mon[36481]: pgmap v1312: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:30.161 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2222 sec 2024-03-28T18:29:31.161 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2223 sec 2024-03-28T18:29:31.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:31 smithi045 ceph-mon[31773]: pgmap v1313: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:31.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1579.272262 secs 2024-03-28T18:29:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:31 smithi071 ceph-mon[36597]: pgmap v1313: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:31.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1579.272262 secs 2024-03-28T18:29:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:31 smithi151 ceph-mon[36481]: pgmap v1313: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1579.272262 secs 2024-03-28T18:29:32.161 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2224 sec 2024-03-28T18:29:32.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 870.696486 secs 2024-03-28T18:29:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 870.696486 secs 2024-03-28T18:29:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 870.696486 secs 2024-03-28T18:29:33.161 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2225 sec 2024-03-28T18:29:33.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:33 smithi045 ceph-mon[31773]: pgmap v1314: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:33.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:33 smithi071 ceph-mon[36597]: pgmap v1314: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:33 smithi151 ceph-mon[36481]: pgmap v1314: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:34.161 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2226 sec 2024-03-28T18:29:35.161 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2227 sec 2024-03-28T18:29:35.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:35 smithi045 ceph-mon[31773]: pgmap v1315: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:35.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:35 smithi071 ceph-mon[36597]: pgmap v1315: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:35.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:35 smithi151 ceph-mon[36481]: pgmap v1315: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:36.161 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2228 sec 2024-03-28T18:29:36.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1584.272323 secs 2024-03-28T18:29:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1584.272323 secs 2024-03-28T18:29:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1584.272323 secs 2024-03-28T18:29:37.161 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2229 sec 2024-03-28T18:29:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:37 smithi045 ceph-mon[31773]: pgmap v1316: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:37 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:29:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 875.696549 secs 2024-03-28T18:29:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:37 smithi071 ceph-mon[36597]: pgmap v1316: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:37 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:29:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 875.696549 secs 2024-03-28T18:29:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:37 smithi151 ceph-mon[36481]: pgmap v1316: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:37 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:29:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 875.696549 secs 2024-03-28T18:29:38.162 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2230 sec 2024-03-28T18:29:38.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:38 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:38.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:38 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:29:38.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:38 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:29:38.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:38 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:38.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:38 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:29:38.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:38 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:29:38.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:38 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:38.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:38 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:29:38.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:38 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:29:39.162 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2231 sec 2024-03-28T18:29:39.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:39 smithi045 ceph-mon[31773]: pgmap v1317: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:39.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:39 smithi045 ceph-mon[31773]: pgmap v1318: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:39.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:39 smithi071 ceph-mon[36597]: pgmap v1317: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:39.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:39 smithi071 ceph-mon[36597]: pgmap v1318: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:39.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:39 smithi151 ceph-mon[36481]: pgmap v1317: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:39.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:39 smithi151 ceph-mon[36481]: pgmap v1318: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:40.162 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2232 sec 2024-03-28T18:29:41.162 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2233 sec 2024-03-28T18:29:41.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:41 smithi045 ceph-mon[31773]: pgmap v1319: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:41.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1589.272413 secs 2024-03-28T18:29:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:41 smithi071 ceph-mon[36597]: pgmap v1319: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1589.272413 secs 2024-03-28T18:29:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:41 smithi151 ceph-mon[36481]: pgmap v1319: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1589.272413 secs 2024-03-28T18:29:42.162 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2234 sec 2024-03-28T18:29:42.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 880.696638 secs 2024-03-28T18:29:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 880.696638 secs 2024-03-28T18:29:42.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 880.696638 secs 2024-03-28T18:29:43.162 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2235 sec 2024-03-28T18:29:43.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:43 smithi071 ceph-mon[36597]: pgmap v1320: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:43.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:43 smithi151 ceph-mon[36481]: pgmap v1320: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:43.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:43 smithi045 ceph-mon[31773]: pgmap v1320: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:44.163 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2236 sec 2024-03-28T18:29:45.162 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2237 sec 2024-03-28T18:29:45.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:45 smithi071 ceph-mon[36597]: pgmap v1321: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:45.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:45 smithi151 ceph-mon[36481]: pgmap v1321: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:45.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:45 smithi045 ceph-mon[31773]: pgmap v1321: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:46.163 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2238 sec 2024-03-28T18:29:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1594.272500 secs 2024-03-28T18:29:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1594.272500 secs 2024-03-28T18:29:46.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1594.272500 secs 2024-03-28T18:29:47.163 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2239 sec 2024-03-28T18:29:47.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:47 smithi071 ceph-mon[36597]: pgmap v1322: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:47.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 885.696699 secs 2024-03-28T18:29:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:47 smithi151 ceph-mon[36481]: pgmap v1322: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 885.696699 secs 2024-03-28T18:29:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:47 smithi045 ceph-mon[31773]: pgmap v1322: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:29:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 885.696699 secs 2024-03-28T18:29:48.163 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2240 sec 2024-03-28T18:29:49.163 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2241 sec 2024-03-28T18:29:49.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:49 smithi071 ceph-mon[36597]: pgmap v1323: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:29:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:49 smithi151 ceph-mon[36481]: pgmap v1323: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:29:49.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:49 smithi045 ceph-mon[31773]: pgmap v1323: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:29:50.163 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2242 sec 2024-03-28T18:29:51.163 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2243 sec 2024-03-28T18:29:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:51 smithi071 ceph-mon[36597]: pgmap v1324: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1599.272561 secs 2024-03-28T18:29:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:51 smithi151 ceph-mon[36481]: pgmap v1324: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1599.272561 secs 2024-03-28T18:29:51.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:51 smithi045 ceph-mon[31773]: pgmap v1324: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:51.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1599.272561 secs 2024-03-28T18:29:52.164 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2244 sec 2024-03-28T18:29:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 890.696740 secs 2024-03-28T18:29:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 890.696740 secs 2024-03-28T18:29:52.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 890.696740 secs 2024-03-28T18:29:53.164 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2245 sec 2024-03-28T18:29:53.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:53 smithi071 ceph-mon[36597]: pgmap v1325: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:53.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:53 smithi151 ceph-mon[36481]: pgmap v1325: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:53.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:53 smithi045 ceph-mon[31773]: pgmap v1325: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:54.164 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2246 sec 2024-03-28T18:29:55.164 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2247 sec 2024-03-28T18:29:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:55 smithi071 ceph-mon[36597]: pgmap v1326: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:55.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:55 smithi151 ceph-mon[36481]: pgmap v1326: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:55.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:55 smithi045 ceph-mon[31773]: pgmap v1326: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:56.164 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2248 sec 2024-03-28T18:29:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1604.272622 secs 2024-03-28T18:29:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1604.272622 secs 2024-03-28T18:29:56.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1604.272622 secs 2024-03-28T18:29:57.164 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2249 sec 2024-03-28T18:29:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:57 smithi071 ceph-mon[36597]: pgmap v1327: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 895.696827 secs 2024-03-28T18:29:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:57 smithi151 ceph-mon[36481]: pgmap v1327: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 895.696827 secs 2024-03-28T18:29:57.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:57 smithi045 ceph-mon[31773]: pgmap v1327: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:29:57.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 895.696827 secs 2024-03-28T18:29:58.164 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2250 sec 2024-03-28T18:29:59.164 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2251 sec 2024-03-28T18:29:59.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:29:59 smithi071 ceph-mon[36597]: pgmap v1328: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:59.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:29:59 smithi151 ceph-mon[36481]: pgmap v1328: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:29:59.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:29:59 smithi045 ceph-mon[31773]: pgmap v1328: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:00.165 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2252 sec 2024-03-28T18:30:00.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:00 smithi045 ceph-mon[31773]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-03-28T18:30:00.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:00 smithi071 ceph-mon[36597]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-03-28T18:30:00.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:00 smithi151 ceph-mon[36481]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-03-28T18:30:01.165 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2253 sec 2024-03-28T18:30:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:01 smithi071 ceph-mon[36597]: pgmap v1329: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1609.272684 secs 2024-03-28T18:30:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:01 smithi151 ceph-mon[36481]: pgmap v1329: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1609.272684 secs 2024-03-28T18:30:01.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:01 smithi045 ceph-mon[31773]: pgmap v1329: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1609.272684 secs 2024-03-28T18:30:02.165 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2254 sec 2024-03-28T18:30:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 900.696879 secs 2024-03-28T18:30:02.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 900.696879 secs 2024-03-28T18:30:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 900.696879 secs 2024-03-28T18:30:03.165 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2255 sec 2024-03-28T18:30:03.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:03 smithi071 ceph-mon[36597]: pgmap v1330: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:03 smithi151 ceph-mon[36481]: pgmap v1330: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:03.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:03 smithi045 ceph-mon[31773]: pgmap v1330: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:04.165 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2256 sec 2024-03-28T18:30:04.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:04 smithi151 ceph-mon[36481]: pgmap v1331: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:04.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:04 smithi045 ceph-mon[31773]: pgmap v1331: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:04.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:04 smithi071 ceph-mon[36597]: pgmap v1331: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:05.165 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2257 sec 2024-03-28T18:30:06.165 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2258 sec 2024-03-28T18:30:07.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:06 smithi071 ceph-mon[36597]: pgmap v1332: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:07.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1614.272746 secs 2024-03-28T18:30:07.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:06 smithi151 ceph-mon[36481]: pgmap v1332: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:07.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1614.272746 secs 2024-03-28T18:30:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:06 smithi045 ceph-mon[31773]: pgmap v1332: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:07.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1614.272746 secs 2024-03-28T18:30:07.166 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2259 sec 2024-03-28T18:30:08.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 905.696959 secs 2024-03-28T18:30:08.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 905.696959 secs 2024-03-28T18:30:08.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 905.696959 secs 2024-03-28T18:30:08.166 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2260 sec 2024-03-28T18:30:09.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:08 smithi071 ceph-mon[36597]: pgmap v1333: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:09.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:08 smithi151 ceph-mon[36481]: pgmap v1333: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:09.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:08 smithi045 ceph-mon[31773]: pgmap v1333: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:09.166 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2261 sec 2024-03-28T18:30:10.166 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2262 sec 2024-03-28T18:30:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:10 smithi071 ceph-mon[36597]: pgmap v1334: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:11.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:10 smithi151 ceph-mon[36481]: pgmap v1334: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:11.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:10 smithi045 ceph-mon[31773]: pgmap v1334: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:11.166 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2263 sec 2024-03-28T18:30:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1619.272807 secs 2024-03-28T18:30:12.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1619.272807 secs 2024-03-28T18:30:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1619.272807 secs 2024-03-28T18:30:12.166 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2264 sec 2024-03-28T18:30:12.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:12 smithi151 ceph-mon[36481]: pgmap v1335: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:12.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 910.697050 secs 2024-03-28T18:30:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:12 smithi071 ceph-mon[36597]: pgmap v1335: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 910.697050 secs 2024-03-28T18:30:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:12 smithi045 ceph-mon[31773]: pgmap v1335: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:13.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 910.697050 secs 2024-03-28T18:30:13.166 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2265 sec 2024-03-28T18:30:14.166 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2266 sec 2024-03-28T18:30:14.968 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:14 smithi045 ceph-mon[31773]: pgmap v1336: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:15.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:14 smithi071 ceph-mon[36597]: pgmap v1336: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:15.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:14 smithi151 ceph-mon[36481]: pgmap v1336: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:15.166 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2267 sec 2024-03-28T18:30:16.167 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2268 sec 2024-03-28T18:30:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:16 smithi071 ceph-mon[36597]: pgmap v1337: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1624.272894 secs 2024-03-28T18:30:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:16 smithi151 ceph-mon[36481]: pgmap v1337: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1624.272894 secs 2024-03-28T18:30:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:16 smithi045 ceph-mon[31773]: pgmap v1337: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1624.272894 secs 2024-03-28T18:30:17.166 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2269 sec 2024-03-28T18:30:18.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 915.697111 secs 2024-03-28T18:30:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 915.697111 secs 2024-03-28T18:30:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 915.697111 secs 2024-03-28T18:30:18.167 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2270 sec 2024-03-28T18:30:19.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:18 smithi071 ceph-mon[36597]: pgmap v1338: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:19.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:18 smithi151 ceph-mon[36481]: pgmap v1338: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:19.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:18 smithi045 ceph-mon[31773]: pgmap v1338: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:19.167 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2271 sec 2024-03-28T18:30:20.167 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2272 sec 2024-03-28T18:30:21.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:20 smithi071 ceph-mon[36597]: pgmap v1339: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:21.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:20 smithi151 ceph-mon[36481]: pgmap v1339: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:21.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:20 smithi045 ceph-mon[31773]: pgmap v1339: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:21.167 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2273 sec 2024-03-28T18:30:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1629.272980 secs 2024-03-28T18:30:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1629.272980 secs 2024-03-28T18:30:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1629.272980 secs 2024-03-28T18:30:22.167 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2274 sec 2024-03-28T18:30:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:22 smithi071 ceph-mon[36597]: pgmap v1340: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 920.697195 secs 2024-03-28T18:30:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:22 smithi151 ceph-mon[36481]: pgmap v1340: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 920.697195 secs 2024-03-28T18:30:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:22 smithi045 ceph-mon[31773]: pgmap v1340: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 920.697195 secs 2024-03-28T18:30:23.168 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2275 sec 2024-03-28T18:30:24.168 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2276 sec 2024-03-28T18:30:25.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:24 smithi071 ceph-mon[36597]: pgmap v1341: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:25.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:24 smithi151 ceph-mon[36481]: pgmap v1341: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:25.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:24 smithi045 ceph-mon[31773]: pgmap v1341: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:25.168 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2277 sec 2024-03-28T18:30:26.168 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2278 sec 2024-03-28T18:30:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:26 smithi071 ceph-mon[36597]: pgmap v1342: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1634.273041 secs 2024-03-28T18:30:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:26 smithi151 ceph-mon[36481]: pgmap v1342: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1634.273041 secs 2024-03-28T18:30:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:26 smithi045 ceph-mon[31773]: pgmap v1342: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1634.273041 secs 2024-03-28T18:30:27.168 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2279 sec 2024-03-28T18:30:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 925.697285 secs 2024-03-28T18:30:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 925.697285 secs 2024-03-28T18:30:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 925.697285 secs 2024-03-28T18:30:28.168 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2280 sec 2024-03-28T18:30:29.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:28 smithi071 ceph-mon[36597]: pgmap v1343: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:29.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:28 smithi151 ceph-mon[36481]: pgmap v1343: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:29.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:28 smithi045 ceph-mon[31773]: pgmap v1343: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:29.168 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2281 sec 2024-03-28T18:30:30.169 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2282 sec 2024-03-28T18:30:31.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:30 smithi071 ceph-mon[36597]: pgmap v1344: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:31.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:30 smithi151 ceph-mon[36481]: pgmap v1344: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:31.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:30 smithi045 ceph-mon[31773]: pgmap v1344: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:31.168 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2283 sec 2024-03-28T18:30:31.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:30:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:30:31.381+0000 7f52eec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:30:31.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:30:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:30:31.381+0000 7f52eec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:30:31.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:30:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:30:31.381+0000 7f52eec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:30:31.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:30:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:30:31.381+0000 7f52eec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:30:31.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:30:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:30:31.381+0000 7f52eec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:30:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1639.273125 secs 2024-03-28T18:30:32.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1639.273125 secs 2024-03-28T18:30:32.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1639.273125 secs 2024-03-28T18:30:32.169 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2284 sec 2024-03-28T18:30:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:32 smithi071 ceph-mon[36597]: pgmap v1345: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 930.697376 secs 2024-03-28T18:30:33.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:32 smithi151 ceph-mon[36481]: pgmap v1345: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:33.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 930.697376 secs 2024-03-28T18:30:33.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:32 smithi045 ceph-mon[31773]: pgmap v1345: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:33.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 930.697376 secs 2024-03-28T18:30:33.169 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2285 sec 2024-03-28T18:30:34.169 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2286 sec 2024-03-28T18:30:35.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:34 smithi071 ceph-mon[36597]: pgmap v1346: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:34 smithi151 ceph-mon[36481]: pgmap v1346: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:34 smithi045 ceph-mon[31773]: pgmap v1346: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:30:35.169 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2287 sec 2024-03-28T18:30:36.169 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2288 sec 2024-03-28T18:30:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:36 smithi071 ceph-mon[36597]: pgmap v1347: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:30:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1644.273186 secs 2024-03-28T18:30:37.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:36 smithi151 ceph-mon[36481]: pgmap v1347: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:30:37.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1644.273186 secs 2024-03-28T18:30:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:36 smithi045 ceph-mon[31773]: pgmap v1347: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:30:37.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1644.273186 secs 2024-03-28T18:30:37.169 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2289 sec 2024-03-28T18:30:38.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 935.697439 secs 2024-03-28T18:30:38.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 935.697439 secs 2024-03-28T18:30:38.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 935.697439 secs 2024-03-28T18:30:38.169 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2290 sec 2024-03-28T18:30:39.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:38 smithi151 ceph-mon[36481]: pgmap v1348: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:30:39.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:38 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:39.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:38 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:39.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:38 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:39.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:38 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:39.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:38 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:39.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:38 smithi045 ceph-mon[31773]: pgmap v1348: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:30:39.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:38 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:39.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:38 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:39.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:38 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:39.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:38 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:39.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:38 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:39.169 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2291 sec 2024-03-28T18:30:39.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:38 smithi071 ceph-mon[36597]: pgmap v1348: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:30:39.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:38 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:39.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:38 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:39.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:38 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:39.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:38 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:39.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:38 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:39 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:39 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:40.153 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:39 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:40.153 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:39 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:40.170 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2292 sec 2024-03-28T18:30:40.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:39 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:40.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:39 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:41.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:40 smithi045 ceph-mon[31773]: pgmap v1349: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:30:41.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:40 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:41.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:40 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:41.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:40 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:41.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:40 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:41.170 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2293 sec 2024-03-28T18:30:41.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:40 smithi071 ceph-mon[36597]: pgmap v1349: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:30:41.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:40 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:41.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:40 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:41.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:40 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:41.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:40 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:41.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:40 smithi151 ceph-mon[36481]: pgmap v1349: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:30:41.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:40 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:41.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:40 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:41.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:40 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:41.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:40 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:42.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1649.273273 secs 2024-03-28T18:30:42.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:41 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:41 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:41 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:41 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:41 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:41 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:41 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:41 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:41 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T18:30:42.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:41 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T18:30:42.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:41 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:41 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:41 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:41 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:41 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:41 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:41 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.168 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:41 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.169 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2294 sec 2024-03-28T18:30:42.225 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1649.273273 secs 2024-03-28T18:30:42.225 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:41 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.225 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:41 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.225 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:41 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.225 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:41 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.226 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:41 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.226 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:41 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.226 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:41 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.226 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:41 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.226 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:41 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T18:30:42.226 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:41 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T18:30:42.226 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:41 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.226 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:41 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.226 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:41 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.226 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:41 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.226 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:41 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.227 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:41 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.227 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:41 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.227 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:41 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1649.273273 secs 2024-03-28T18:30:42.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:41 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:41 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:41 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:41 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:41 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:41 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:41 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:41 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:41 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T18:30:42.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:41 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T18:30:42.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:41 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:41 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:41 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:41 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:41 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:41 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:41 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:42.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:41 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:43.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:42 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T18:30:43.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:42 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T18:30:43.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:42 smithi045 ceph-mon[31773]: pgmap v1350: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:30:43.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 940.697501 secs 2024-03-28T18:30:43.170 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2295 sec 2024-03-28T18:30:43.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:42 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T18:30:43.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:42 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T18:30:43.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:42 smithi071 ceph-mon[36597]: pgmap v1350: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:30:43.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 940.697501 secs 2024-03-28T18:30:43.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:42 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T18:30:43.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:42 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T18:30:43.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:42 smithi151 ceph-mon[36481]: pgmap v1350: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:30:43.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 940.697501 secs 2024-03-28T18:30:44.170 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2296 sec 2024-03-28T18:30:44.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:44 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:44.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:44 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:44.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:44 smithi071 ceph-mon[36597]: pgmap v1351: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:30:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:44 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:44 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:44 smithi151 ceph-mon[36481]: pgmap v1351: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:30:44.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:44 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:44.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:44 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:44.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:44 smithi045 ceph-mon[31773]: pgmap v1351: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:30:45.170 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2297 sec 2024-03-28T18:30:45.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:45 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:45 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:45 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:45 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:45 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:45 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:45 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:45 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:45 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T18:30:45.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:45 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:30:45.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:45 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:45.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:45 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:45.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:45 smithi071 ceph-mon[36597]: pgmap v1352: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 181 B/s wr, 15 op/s 2024-03-28T18:30:45.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:45 smithi071 ceph-mon[36597]: pgmap v1353: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:30:45.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:45 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:45.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:45 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:45 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:45 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:45 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:45 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:45 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:45 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:45 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:45 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T18:30:45.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:45 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:30:45.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:45 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:45.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:45 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:45.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:45 smithi151 ceph-mon[36481]: pgmap v1352: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 181 B/s wr, 15 op/s 2024-03-28T18:30:45.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:45 smithi151 ceph-mon[36481]: pgmap v1353: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:30:45.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:45 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:45.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:45 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:45 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:45 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:45 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:45 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:45 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:45 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:45 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:45.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:45 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T18:30:45.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:45 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:30:45.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:45 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:45.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:45 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:45.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:45 smithi045 ceph-mon[31773]: pgmap v1352: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 181 B/s wr, 15 op/s 2024-03-28T18:30:45.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:45 smithi045 ceph-mon[31773]: pgmap v1353: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:30:45.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:45 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:30:46.170 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2298 sec 2024-03-28T18:30:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1654.273312 secs 2024-03-28T18:30:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1654.273312 secs 2024-03-28T18:30:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1654.273312 secs 2024-03-28T18:30:47.170 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2299 sec 2024-03-28T18:30:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 945.697562 secs 2024-03-28T18:30:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:47 smithi151 ceph-mon[36481]: pgmap v1354: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:30:47.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 945.697562 secs 2024-03-28T18:30:47.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:47 smithi045 ceph-mon[31773]: pgmap v1354: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:30:47.961 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 945.697562 secs 2024-03-28T18:30:47.961 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:47 smithi071 ceph-mon[36597]: pgmap v1354: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:30:48.170 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2300 sec 2024-03-28T18:30:49.171 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2301 sec 2024-03-28T18:30:50.171 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2302 sec 2024-03-28T18:30:50.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:49 smithi071 ceph-mon[36597]: pgmap v1355: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:30:50.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:49 smithi151 ceph-mon[36481]: pgmap v1355: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:30:50.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:49 smithi045 ceph-mon[31773]: pgmap v1355: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:30:51.171 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2303 sec 2024-03-28T18:30:52.171 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2304 sec 2024-03-28T18:30:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1659.273399 secs 2024-03-28T18:30:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:51 smithi071 ceph-mon[36597]: pgmap v1356: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:30:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 950.697605 secs 2024-03-28T18:30:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1659.273399 secs 2024-03-28T18:30:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:51 smithi151 ceph-mon[36481]: pgmap v1356: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:30:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 950.697605 secs 2024-03-28T18:30:52.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1659.273399 secs 2024-03-28T18:30:52.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:51 smithi045 ceph-mon[31773]: pgmap v1356: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:30:52.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 950.697605 secs 2024-03-28T18:30:53.171 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2305 sec 2024-03-28T18:30:54.171 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2306 sec 2024-03-28T18:30:54.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:53 smithi071 ceph-mon[36597]: pgmap v1357: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:30:54.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:53 smithi151 ceph-mon[36481]: pgmap v1357: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:30:54.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:53 smithi045 ceph-mon[31773]: pgmap v1357: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:30:55.171 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2307 sec 2024-03-28T18:30:56.172 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2308 sec 2024-03-28T18:30:56.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:55 smithi071 ceph-mon[36597]: pgmap v1358: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:30:56.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:55 smithi151 ceph-mon[36481]: pgmap v1358: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:30:56.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:55 smithi045 ceph-mon[31773]: pgmap v1358: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:30:57.172 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2309 sec 2024-03-28T18:30:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1664.273484 secs 2024-03-28T18:30:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 955.697683 secs 2024-03-28T18:30:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1664.273484 secs 2024-03-28T18:30:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 955.697683 secs 2024-03-28T18:30:57.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1664.273484 secs 2024-03-28T18:30:57.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 955.697683 secs 2024-03-28T18:30:58.172 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2310 sec 2024-03-28T18:30:58.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:57 smithi071 ceph-mon[36597]: pgmap v1359: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:58.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:57 smithi151 ceph-mon[36481]: pgmap v1359: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:58.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:57 smithi045 ceph-mon[31773]: pgmap v1359: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:30:59.172 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2311 sec 2024-03-28T18:31:00.172 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2312 sec 2024-03-28T18:31:00.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:30:59 smithi071 ceph-mon[36597]: pgmap v1360: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:00.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:30:59 smithi151 ceph-mon[36481]: pgmap v1360: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:00.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:30:59 smithi045 ceph-mon[31773]: pgmap v1360: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:01.172 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2313 sec 2024-03-28T18:31:02.173 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2314 sec 2024-03-28T18:31:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1669.273569 secs 2024-03-28T18:31:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:02 smithi071 ceph-mon[36597]: pgmap v1361: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 960.697745 secs 2024-03-28T18:31:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1669.273569 secs 2024-03-28T18:31:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:02 smithi151 ceph-mon[36481]: pgmap v1361: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 960.697745 secs 2024-03-28T18:31:02.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1669.273569 secs 2024-03-28T18:31:02.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:02 smithi045 ceph-mon[31773]: pgmap v1361: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:02.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 960.697745 secs 2024-03-28T18:31:03.173 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2315 sec 2024-03-28T18:31:04.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:04 smithi045 ceph-mon[31773]: pgmap v1362: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:04.172 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2316 sec 2024-03-28T18:31:04.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:04 smithi071 ceph-mon[36597]: pgmap v1362: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:04.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:04 smithi151 ceph-mon[36481]: pgmap v1362: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:05.173 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2317 sec 2024-03-28T18:31:06.173 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2318 sec 2024-03-28T18:31:06.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:06 smithi071 ceph-mon[36597]: pgmap v1363: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:06.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:06 smithi151 ceph-mon[36481]: pgmap v1363: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:06.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:06 smithi045 ceph-mon[31773]: pgmap v1363: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:07.173 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2319 sec 2024-03-28T18:31:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1674.273627 secs 2024-03-28T18:31:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 965.697832 secs 2024-03-28T18:31:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1674.273627 secs 2024-03-28T18:31:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 965.697832 secs 2024-03-28T18:31:07.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1674.273627 secs 2024-03-28T18:31:07.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 965.697832 secs 2024-03-28T18:31:08.173 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2320 sec 2024-03-28T18:31:08.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:08 smithi071 ceph-mon[36597]: pgmap v1364: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:31:08.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:08 smithi151 ceph-mon[36481]: pgmap v1364: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:31:08.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:08 smithi045 ceph-mon[31773]: pgmap v1364: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:31:09.173 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2321 sec 2024-03-28T18:31:10.174 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2322 sec 2024-03-28T18:31:10.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:10 smithi071 ceph-mon[36597]: pgmap v1365: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:31:10.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:10 smithi151 ceph-mon[36481]: pgmap v1365: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:31:10.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:10 smithi045 ceph-mon[31773]: pgmap v1365: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:31:11.174 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2323 sec 2024-03-28T18:31:12.174 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2324 sec 2024-03-28T18:31:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1679.273717 secs 2024-03-28T18:31:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:12 smithi071 ceph-mon[36597]: pgmap v1366: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:31:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 970.697880 secs 2024-03-28T18:31:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1679.273717 secs 2024-03-28T18:31:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:12 smithi151 ceph-mon[36481]: pgmap v1366: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:31:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 970.697880 secs 2024-03-28T18:31:12.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1679.273717 secs 2024-03-28T18:31:12.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:12 smithi045 ceph-mon[31773]: pgmap v1366: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:31:12.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 970.697880 secs 2024-03-28T18:31:13.174 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2325 sec 2024-03-28T18:31:14.174 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2326 sec 2024-03-28T18:31:14.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:14 smithi071 ceph-mon[36597]: pgmap v1367: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:31:14.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:14 smithi151 ceph-mon[36481]: pgmap v1367: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:31:14.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:14 smithi045 ceph-mon[31773]: pgmap v1367: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:31:15.174 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2327 sec 2024-03-28T18:31:16.174 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2328 sec 2024-03-28T18:31:16.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:16 smithi071 ceph-mon[36597]: pgmap v1368: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:31:16.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:16 smithi151 ceph-mon[36481]: pgmap v1368: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:31:16.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:16 smithi045 ceph-mon[31773]: pgmap v1368: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:31:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1684.273778 secs 2024-03-28T18:31:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 975.697942 secs 2024-03-28T18:31:17.174 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2329 sec 2024-03-28T18:31:17.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1684.273778 secs 2024-03-28T18:31:17.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 975.697942 secs 2024-03-28T18:31:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1684.273778 secs 2024-03-28T18:31:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 975.697942 secs 2024-03-28T18:31:18.175 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2330 sec 2024-03-28T18:31:18.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:18 smithi071 ceph-mon[36597]: pgmap v1369: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:31:18.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:18 smithi151 ceph-mon[36481]: pgmap v1369: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:31:18.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:18 smithi045 ceph-mon[31773]: pgmap v1369: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:31:19.175 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2331 sec 2024-03-28T18:31:20.175 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2332 sec 2024-03-28T18:31:20.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:20 smithi071 ceph-mon[36597]: pgmap v1370: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:20.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:20 smithi151 ceph-mon[36481]: pgmap v1370: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:20.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:20 smithi045 ceph-mon[31773]: pgmap v1370: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:21.175 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2333 sec 2024-03-28T18:31:21.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1689.273867 secs 2024-03-28T18:31:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1689.273867 secs 2024-03-28T18:31:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1689.273867 secs 2024-03-28T18:31:22.175 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2334 sec 2024-03-28T18:31:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:22 smithi151 ceph-mon[36481]: pgmap v1371: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 980.698006 secs 2024-03-28T18:31:22.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:22 smithi045 ceph-mon[31773]: pgmap v1371: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:22.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 980.698006 secs 2024-03-28T18:31:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:22 smithi071 ceph-mon[36597]: pgmap v1371: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 980.698006 secs 2024-03-28T18:31:23.175 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2335 sec 2024-03-28T18:31:24.176 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2336 sec 2024-03-28T18:31:24.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:24 smithi045 ceph-mon[31773]: pgmap v1372: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:24.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:24 smithi071 ceph-mon[36597]: pgmap v1372: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:24.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:24 smithi151 ceph-mon[36481]: pgmap v1372: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:25.176 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2337 sec 2024-03-28T18:31:26.176 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2338 sec 2024-03-28T18:31:26.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:26 smithi045 ceph-mon[31773]: pgmap v1373: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:26.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1694.273952 secs 2024-03-28T18:31:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:26 smithi071 ceph-mon[36597]: pgmap v1373: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1694.273952 secs 2024-03-28T18:31:26.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:26 smithi151 ceph-mon[36481]: pgmap v1373: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:26.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1694.273952 secs 2024-03-28T18:31:27.176 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2339 sec 2024-03-28T18:31:27.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 985.698069 secs 2024-03-28T18:31:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 985.698069 secs 2024-03-28T18:31:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 985.698069 secs 2024-03-28T18:31:28.176 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2340 sec 2024-03-28T18:31:28.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:28 smithi045 ceph-mon[31773]: pgmap v1374: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:28.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:28 smithi071 ceph-mon[36597]: pgmap v1374: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:28.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:28 smithi151 ceph-mon[36481]: pgmap v1374: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:29.176 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2341 sec 2024-03-28T18:31:30.176 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2342 sec 2024-03-28T18:31:30.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:30 smithi045 ceph-mon[31773]: pgmap v1375: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:30.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:30 smithi071 ceph-mon[36597]: pgmap v1375: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:30.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:30 smithi151 ceph-mon[36481]: pgmap v1375: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:31.177 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2343 sec 2024-03-28T18:31:31.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1699.274014 secs 2024-03-28T18:31:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1699.274014 secs 2024-03-28T18:31:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1699.274014 secs 2024-03-28T18:31:32.177 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2344 sec 2024-03-28T18:31:32.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:32 smithi045 ceph-mon[31773]: pgmap v1376: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:32.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 990.698132 secs 2024-03-28T18:31:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:32 smithi071 ceph-mon[36597]: pgmap v1376: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 990.698132 secs 2024-03-28T18:31:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:32 smithi151 ceph-mon[36481]: pgmap v1376: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 990.698132 secs 2024-03-28T18:31:33.177 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2345 sec 2024-03-28T18:31:34.177 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2346 sec 2024-03-28T18:31:34.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:34 smithi045 ceph-mon[31773]: pgmap v1377: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:34.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:34 smithi071 ceph-mon[36597]: pgmap v1377: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:34.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:34 smithi151 ceph-mon[36481]: pgmap v1377: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:35.177 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2347 sec 2024-03-28T18:31:36.178 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2348 sec 2024-03-28T18:31:36.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:36 smithi045 ceph-mon[31773]: pgmap v1378: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:36.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1704.274102 secs 2024-03-28T18:31:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:36 smithi071 ceph-mon[36597]: pgmap v1378: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1704.274102 secs 2024-03-28T18:31:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:36 smithi151 ceph-mon[36481]: pgmap v1378: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1704.274102 secs 2024-03-28T18:31:37.177 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2349 sec 2024-03-28T18:31:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 995.698195 secs 2024-03-28T18:31:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 995.698195 secs 2024-03-28T18:31:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 995.698195 secs 2024-03-28T18:31:38.178 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2350 sec 2024-03-28T18:31:38.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:38 smithi045 ceph-mon[31773]: pgmap v1379: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:38.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:38 smithi071 ceph-mon[36597]: pgmap v1379: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:38.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:38 smithi151 ceph-mon[36481]: pgmap v1379: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:39.178 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2351 sec 2024-03-28T18:31:40.178 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2352 sec 2024-03-28T18:31:40.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:40 smithi045 ceph-mon[31773]: pgmap v1380: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:40.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:40 smithi071 ceph-mon[36597]: pgmap v1380: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:40.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:40 smithi151 ceph-mon[36481]: pgmap v1380: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:41.178 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2353 sec 2024-03-28T18:31:41.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1709.274193 secs 2024-03-28T18:31:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1709.274193 secs 2024-03-28T18:31:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1709.274193 secs 2024-03-28T18:31:42.178 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2354 sec 2024-03-28T18:31:42.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:42 smithi045 ceph-mon[31773]: pgmap v1381: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:42.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1000.698257 secs 2024-03-28T18:31:42.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:42 smithi151 ceph-mon[36481]: pgmap v1381: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:42.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1000.698257 secs 2024-03-28T18:31:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:42 smithi071 ceph-mon[36597]: pgmap v1381: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:31:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1000.698257 secs 2024-03-28T18:31:43.178 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2355 sec 2024-03-28T18:31:44.178 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2356 sec 2024-03-28T18:31:44.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:44 smithi045 ceph-mon[31773]: pgmap v1382: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:44.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:44 smithi071 ceph-mon[36597]: pgmap v1382: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:44 smithi151 ceph-mon[36481]: pgmap v1382: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:45.178 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2357 sec 2024-03-28T18:31:46.181 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2358 sec 2024-03-28T18:31:46.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:46 smithi045 ceph-mon[31773]: pgmap v1383: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:46.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:46 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:31:46.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1714.274264 secs 2024-03-28T18:31:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:46 smithi071 ceph-mon[36597]: pgmap v1383: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:46 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:31:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1714.274264 secs 2024-03-28T18:31:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:46 smithi151 ceph-mon[36481]: pgmap v1383: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:31:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:46 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:31:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1714.274264 secs 2024-03-28T18:31:47.179 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2359 sec 2024-03-28T18:31:47.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:47 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:47.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:47 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:31:47.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:47 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:31:47.462 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1005.698317 secs 2024-03-28T18:31:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:47 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:47 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:31:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:47 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:31:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1005.698317 secs 2024-03-28T18:31:47.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:47 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:47.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:47 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:31:47.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:47 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:31:47.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1005.698317 secs 2024-03-28T18:31:48.179 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2360 sec 2024-03-28T18:31:48.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:48 smithi045 ceph-mon[31773]: pgmap v1384: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:31:48.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:48 smithi045 ceph-mon[31773]: pgmap v1385: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:31:48.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:48 smithi071 ceph-mon[36597]: pgmap v1384: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:31:48.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:48 smithi071 ceph-mon[36597]: pgmap v1385: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:31:48.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:48 smithi151 ceph-mon[36481]: pgmap v1384: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:31:48.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:48 smithi151 ceph-mon[36481]: pgmap v1385: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:31:49.179 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2361 sec 2024-03-28T18:31:50.179 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2362 sec 2024-03-28T18:31:50.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:50 smithi071 ceph-mon[36597]: pgmap v1386: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:31:50.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:50 smithi151 ceph-mon[36481]: pgmap v1386: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:31:50.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:50 smithi045 ceph-mon[31773]: pgmap v1386: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:31:51.179 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2363 sec 2024-03-28T18:31:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1719.274353 secs 2024-03-28T18:31:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1719.274353 secs 2024-03-28T18:31:51.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1719.274353 secs 2024-03-28T18:31:52.179 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2364 sec 2024-03-28T18:31:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:52 smithi071 ceph-mon[36597]: pgmap v1387: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:31:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1010.698401 secs 2024-03-28T18:31:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:52 smithi151 ceph-mon[36481]: pgmap v1387: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:31:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1010.698401 secs 2024-03-28T18:31:52.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:52 smithi045 ceph-mon[31773]: pgmap v1387: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:31:52.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1010.698401 secs 2024-03-28T18:31:53.180 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2365 sec 2024-03-28T18:31:54.180 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2366 sec 2024-03-28T18:31:54.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:54 smithi071 ceph-mon[36597]: pgmap v1388: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:31:54.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:54 smithi151 ceph-mon[36481]: pgmap v1388: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:31:54.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:54 smithi045 ceph-mon[31773]: pgmap v1388: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:31:55.180 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2367 sec 2024-03-28T18:31:56.180 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2368 sec 2024-03-28T18:31:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:56 smithi071 ceph-mon[36597]: pgmap v1389: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:31:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1724.274437 secs 2024-03-28T18:31:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:56 smithi151 ceph-mon[36481]: pgmap v1389: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:31:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1724.274437 secs 2024-03-28T18:31:56.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:56 smithi045 ceph-mon[31773]: pgmap v1389: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:31:56.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1724.274437 secs 2024-03-28T18:31:57.180 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2369 sec 2024-03-28T18:31:57.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1015.698463 secs 2024-03-28T18:31:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1015.698463 secs 2024-03-28T18:31:57.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1015.698463 secs 2024-03-28T18:31:58.180 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2370 sec 2024-03-28T18:31:58.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:31:58 smithi071 ceph-mon[36597]: pgmap v1390: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:31:58.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:31:58 smithi151 ceph-mon[36481]: pgmap v1390: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:31:58.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:31:58 smithi045 ceph-mon[31773]: pgmap v1390: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:31:59.180 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2371 sec 2024-03-28T18:32:00.180 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2372 sec 2024-03-28T18:32:00.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:00 smithi071 ceph-mon[36597]: pgmap v1391: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:00.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:00 smithi151 ceph-mon[36481]: pgmap v1391: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:00.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:00 smithi045 ceph-mon[31773]: pgmap v1391: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:01.181 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2373 sec 2024-03-28T18:32:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1729.274509 secs 2024-03-28T18:32:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1729.274509 secs 2024-03-28T18:32:01.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1729.274509 secs 2024-03-28T18:32:02.181 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2374 sec 2024-03-28T18:32:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:02 smithi071 ceph-mon[36597]: pgmap v1392: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:02 smithi071 ceph-mon[36597]: 2 slow requests, 1 included below; oldest blocked for > 1020.698551 secs 2024-03-28T18:32:02.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:02 smithi071 ceph-mon[36597]: slow request 720.510908 seconds old, received at 2024-03-28T18:20:01.134771+0000: client_request(client.16398:1 getattr pAsLsXsFs #0x1 2024-03-28T18:20:01.130880+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:32:02.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:02 smithi045 ceph-mon[31773]: pgmap v1392: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:02.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:02 smithi045 ceph-mon[31773]: 2 slow requests, 1 included below; oldest blocked for > 1020.698551 secs 2024-03-28T18:32:02.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:02 smithi045 ceph-mon[31773]: slow request 720.510908 seconds old, received at 2024-03-28T18:20:01.134771+0000: client_request(client.16398:1 getattr pAsLsXsFs #0x1 2024-03-28T18:20:01.130880+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:32:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:02 smithi151 ceph-mon[36481]: pgmap v1392: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:02 smithi151 ceph-mon[36481]: 2 slow requests, 1 included below; oldest blocked for > 1020.698551 secs 2024-03-28T18:32:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:02 smithi151 ceph-mon[36481]: slow request 720.510908 seconds old, received at 2024-03-28T18:20:01.134771+0000: client_request(client.16398:1 getattr pAsLsXsFs #0x1 2024-03-28T18:20:01.130880+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:32:03.181 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2375 sec 2024-03-28T18:32:04.181 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2376 sec 2024-03-28T18:32:04.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:04 smithi045 ceph-mon[31773]: pgmap v1393: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:04.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:04 smithi071 ceph-mon[36597]: pgmap v1393: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:04.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:04 smithi151 ceph-mon[36481]: pgmap v1393: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:05.181 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2377 sec 2024-03-28T18:32:06.182 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2378 sec 2024-03-28T18:32:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:06 smithi071 ceph-mon[36597]: pgmap v1394: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1734.274572 secs 2024-03-28T18:32:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:06 smithi151 ceph-mon[36481]: pgmap v1394: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1734.274572 secs 2024-03-28T18:32:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:06 smithi045 ceph-mon[31773]: pgmap v1394: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:06.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1734.274572 secs 2024-03-28T18:32:07.181 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2379 sec 2024-03-28T18:32:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1025.698613 secs 2024-03-28T18:32:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1025.698613 secs 2024-03-28T18:32:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1025.698613 secs 2024-03-28T18:32:08.182 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2380 sec 2024-03-28T18:32:08.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:08 smithi071 ceph-mon[36597]: pgmap v1395: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:08.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:08 smithi151 ceph-mon[36481]: pgmap v1395: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:08.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:08 smithi045 ceph-mon[31773]: pgmap v1395: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:09.182 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2381 sec 2024-03-28T18:32:10.182 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2382 sec 2024-03-28T18:32:10.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:10 smithi071 ceph-mon[36597]: pgmap v1396: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:10.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:10 smithi151 ceph-mon[36481]: pgmap v1396: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:10.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:10 smithi045 ceph-mon[31773]: pgmap v1396: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:11.182 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2383 sec 2024-03-28T18:32:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1739.274635 secs 2024-03-28T18:32:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1739.274635 secs 2024-03-28T18:32:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1739.274635 secs 2024-03-28T18:32:12.182 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2384 sec 2024-03-28T18:32:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:12 smithi071 ceph-mon[36597]: pgmap v1397: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1030.698704 secs 2024-03-28T18:32:12.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:12 smithi151 ceph-mon[36481]: pgmap v1397: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:12.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1030.698704 secs 2024-03-28T18:32:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:12 smithi045 ceph-mon[31773]: pgmap v1397: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1030.698704 secs 2024-03-28T18:32:13.182 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2385 sec 2024-03-28T18:32:14.183 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2386 sec 2024-03-28T18:32:14.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:14 smithi071 ceph-mon[36597]: pgmap v1398: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:14.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:14 smithi151 ceph-mon[36481]: pgmap v1398: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:14.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:14 smithi045 ceph-mon[31773]: pgmap v1398: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:15.183 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2387 sec 2024-03-28T18:32:16.183 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2388 sec 2024-03-28T18:32:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:16 smithi071 ceph-mon[36597]: pgmap v1399: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1744.274719 secs 2024-03-28T18:32:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:16 smithi151 ceph-mon[36481]: pgmap v1399: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1744.274719 secs 2024-03-28T18:32:16.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:16 smithi045 ceph-mon[31773]: pgmap v1399: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:16.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1744.274719 secs 2024-03-28T18:32:17.183 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2389 sec 2024-03-28T18:32:17.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1035.698723 secs 2024-03-28T18:32:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1035.698723 secs 2024-03-28T18:32:17.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1035.698723 secs 2024-03-28T18:32:18.183 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2390 sec 2024-03-28T18:32:18.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:18 smithi071 ceph-mon[36597]: pgmap v1400: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:18 smithi151 ceph-mon[36481]: pgmap v1400: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:18.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:18 smithi045 ceph-mon[31773]: pgmap v1400: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:19.183 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2391 sec 2024-03-28T18:32:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:19 smithi045 ceph-mon[31773]: pgmap v1401: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:19.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:19 smithi071 ceph-mon[36597]: pgmap v1401: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:19.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:19 smithi151 ceph-mon[36481]: pgmap v1401: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:20.183 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2392 sec 2024-03-28T18:32:21.184 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2393 sec 2024-03-28T18:32:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:21 smithi071 ceph-mon[36597]: pgmap v1402: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:22.115 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1749.274783 secs 2024-03-28T18:32:22.115 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:21 smithi151 ceph-mon[36481]: pgmap v1402: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:22.115 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1749.274783 secs 2024-03-28T18:32:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:21 smithi045 ceph-mon[31773]: pgmap v1402: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1749.274783 secs 2024-03-28T18:32:22.183 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2394 sec 2024-03-28T18:32:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1040.698784 secs 2024-03-28T18:32:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1040.698784 secs 2024-03-28T18:32:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1040.698784 secs 2024-03-28T18:32:23.184 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2395 sec 2024-03-28T18:32:24.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:23 smithi071 ceph-mon[36597]: pgmap v1403: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:24.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:23 smithi151 ceph-mon[36481]: pgmap v1403: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:24.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:23 smithi045 ceph-mon[31773]: pgmap v1403: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:24.184 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2396 sec 2024-03-28T18:32:25.184 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2397 sec 2024-03-28T18:32:26.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:25 smithi071 ceph-mon[36597]: pgmap v1404: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:26.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:25 smithi151 ceph-mon[36481]: pgmap v1404: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:26.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:25 smithi045 ceph-mon[31773]: pgmap v1404: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:26.184 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2398 sec 2024-03-28T18:32:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1754.274821 secs 2024-03-28T18:32:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1754.274821 secs 2024-03-28T18:32:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1754.274821 secs 2024-03-28T18:32:27.184 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2399 sec 2024-03-28T18:32:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1045.698847 secs 2024-03-28T18:32:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:27 smithi071 ceph-mon[36597]: pgmap v1405: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1045.698847 secs 2024-03-28T18:32:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:27 smithi151 ceph-mon[36481]: pgmap v1405: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1045.698847 secs 2024-03-28T18:32:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:27 smithi045 ceph-mon[31773]: pgmap v1405: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:28.184 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2400 sec 2024-03-28T18:32:29.185 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2401 sec 2024-03-28T18:32:30.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:29 smithi071 ceph-mon[36597]: pgmap v1406: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:30.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:29 smithi151 ceph-mon[36481]: pgmap v1406: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:30.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:29 smithi045 ceph-mon[31773]: pgmap v1406: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:30.185 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2402 sec 2024-03-28T18:32:31.185 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2403 sec 2024-03-28T18:32:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:31 smithi071 ceph-mon[36597]: pgmap v1407: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1759.274884 secs 2024-03-28T18:32:32.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:31 smithi151 ceph-mon[36481]: pgmap v1407: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:32.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1759.274884 secs 2024-03-28T18:32:32.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:31 smithi045 ceph-mon[31773]: pgmap v1407: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:32.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1759.274884 secs 2024-03-28T18:32:32.185 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2404 sec 2024-03-28T18:32:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1050.698907 secs 2024-03-28T18:32:33.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1050.698907 secs 2024-03-28T18:32:33.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1050.698907 secs 2024-03-28T18:32:33.185 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2405 sec 2024-03-28T18:32:34.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:33 smithi071 ceph-mon[36597]: pgmap v1408: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:34.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:33 smithi151 ceph-mon[36481]: pgmap v1408: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:34.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:33 smithi045 ceph-mon[31773]: pgmap v1408: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:34.185 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2406 sec 2024-03-28T18:32:35.185 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2407 sec 2024-03-28T18:32:36.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:35 smithi071 ceph-mon[36597]: pgmap v1409: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:36.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:35 smithi151 ceph-mon[36481]: pgmap v1409: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:36.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:35 smithi045 ceph-mon[31773]: pgmap v1409: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:36.185 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2408 sec 2024-03-28T18:32:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1764.274973 secs 2024-03-28T18:32:37.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1764.274973 secs 2024-03-28T18:32:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1764.274973 secs 2024-03-28T18:32:37.185 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2409 sec 2024-03-28T18:32:38.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1055.698991 secs 2024-03-28T18:32:38.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:37 smithi071 ceph-mon[36597]: pgmap v1410: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:38.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1055.698991 secs 2024-03-28T18:32:38.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:37 smithi151 ceph-mon[36481]: pgmap v1410: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:38.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1055.698991 secs 2024-03-28T18:32:38.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:37 smithi045 ceph-mon[31773]: pgmap v1410: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:38.185 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2410 sec 2024-03-28T18:32:39.186 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2411 sec 2024-03-28T18:32:40.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:39 smithi071 ceph-mon[36597]: pgmap v1411: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:40.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:39 smithi151 ceph-mon[36481]: pgmap v1411: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:40.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:39 smithi045 ceph-mon[31773]: pgmap v1411: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:40.186 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2412 sec 2024-03-28T18:32:41.186 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2413 sec 2024-03-28T18:32:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:41 smithi071 ceph-mon[36597]: pgmap v1412: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1769.275036 secs 2024-03-28T18:32:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:41 smithi151 ceph-mon[36481]: pgmap v1412: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1769.275036 secs 2024-03-28T18:32:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:41 smithi045 ceph-mon[31773]: pgmap v1412: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1769.275036 secs 2024-03-28T18:32:42.186 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2414 sec 2024-03-28T18:32:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1060.699050 secs 2024-03-28T18:32:43.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1060.699050 secs 2024-03-28T18:32:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1060.699050 secs 2024-03-28T18:32:43.186 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2415 sec 2024-03-28T18:32:44.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:43 smithi071 ceph-mon[36597]: pgmap v1413: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:44.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:43 smithi151 ceph-mon[36481]: pgmap v1413: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:44.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:43 smithi045 ceph-mon[31773]: pgmap v1413: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:44.186 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2416 sec 2024-03-28T18:32:45.186 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2417 sec 2024-03-28T18:32:46.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:45 smithi071 ceph-mon[36597]: pgmap v1414: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:46.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:45 smithi151 ceph-mon[36481]: pgmap v1414: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:46.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:45 smithi045 ceph-mon[31773]: pgmap v1414: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:46.186 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2418 sec 2024-03-28T18:32:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1774.275100 secs 2024-03-28T18:32:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1774.275100 secs 2024-03-28T18:32:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1774.275100 secs 2024-03-28T18:32:47.186 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2419 sec 2024-03-28T18:32:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1065.699110 secs 2024-03-28T18:32:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:47 smithi151 ceph-mon[36481]: pgmap v1415: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:47 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:32:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1065.699110 secs 2024-03-28T18:32:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:47 smithi045 ceph-mon[31773]: pgmap v1415: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:47 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:32:48.186 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2420 sec 2024-03-28T18:32:48.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1065.699110 secs 2024-03-28T18:32:48.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:47 smithi071 ceph-mon[36597]: pgmap v1415: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:32:48.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:47 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:32:49.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:48 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:49.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:48 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:32:49.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:48 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:32:49.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:48 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:49.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:48 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:32:49.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:48 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:32:49.187 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2421 sec 2024-03-28T18:32:49.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:48 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:49.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:48 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:32:49.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:48 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:32:50.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:49 smithi045 ceph-mon[31773]: pgmap v1416: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:50.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:49 smithi045 ceph-mon[31773]: pgmap v1417: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:32:50.187 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2422 sec 2024-03-28T18:32:50.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:49 smithi071 ceph-mon[36597]: pgmap v1416: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:50.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:49 smithi071 ceph-mon[36597]: pgmap v1417: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:32:50.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:49 smithi151 ceph-mon[36481]: pgmap v1416: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:32:50.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:49 smithi151 ceph-mon[36481]: pgmap v1417: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:32:51.187 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2423 sec 2024-03-28T18:32:52.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:51 smithi045 ceph-mon[31773]: pgmap v1418: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:32:52.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1779.275186 secs 2024-03-28T18:32:52.187 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2424 sec 2024-03-28T18:32:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:51 smithi071 ceph-mon[36597]: pgmap v1418: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:32:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1779.275186 secs 2024-03-28T18:32:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:51 smithi151 ceph-mon[36481]: pgmap v1418: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:32:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1779.275186 secs 2024-03-28T18:32:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1070.699153 secs 2024-03-28T18:32:53.187 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2425 sec 2024-03-28T18:32:53.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1070.699153 secs 2024-03-28T18:32:53.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1070.699153 secs 2024-03-28T18:32:54.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:53 smithi045 ceph-mon[31773]: pgmap v1419: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:32:54.187 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2426 sec 2024-03-28T18:32:54.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:53 smithi071 ceph-mon[36597]: pgmap v1419: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:32:54.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:53 smithi151 ceph-mon[36481]: pgmap v1419: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:32:55.188 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2427 sec 2024-03-28T18:32:56.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:55 smithi045 ceph-mon[31773]: pgmap v1420: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:32:56.189 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2428 sec 2024-03-28T18:32:56.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:55 smithi071 ceph-mon[36597]: pgmap v1420: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:32:56.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:55 smithi151 ceph-mon[36481]: pgmap v1420: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:32:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1784.275275 secs 2024-03-28T18:32:57.188 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2429 sec 2024-03-28T18:32:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1784.275275 secs 2024-03-28T18:32:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1784.275275 secs 2024-03-28T18:32:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:57 smithi045 ceph-mon[31773]: pgmap v1421: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:32:58.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1075.699242 secs 2024-03-28T18:32:58.188 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2430 sec 2024-03-28T18:32:58.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:57 smithi071 ceph-mon[36597]: pgmap v1421: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:32:58.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1075.699242 secs 2024-03-28T18:32:58.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:57 smithi151 ceph-mon[36481]: pgmap v1421: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:32:58.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1075.699242 secs 2024-03-28T18:32:59.188 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2431 sec 2024-03-28T18:33:00.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:32:59 smithi045 ceph-mon[31773]: pgmap v1422: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:33:00.188 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2432 sec 2024-03-28T18:33:00.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:32:59 smithi071 ceph-mon[36597]: pgmap v1422: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:33:00.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:32:59 smithi151 ceph-mon[36481]: pgmap v1422: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:33:01.189 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2433 sec 2024-03-28T18:33:02.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:01 smithi045 ceph-mon[31773]: pgmap v1423: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:02.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1789.275338 secs 2024-03-28T18:33:02.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:01 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1080.699284 secs 2024-03-28T18:33:02.189 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2434 sec 2024-03-28T18:33:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:01 smithi071 ceph-mon[36597]: pgmap v1423: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1789.275338 secs 2024-03-28T18:33:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:01 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1080.699284 secs 2024-03-28T18:33:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:01 smithi151 ceph-mon[36481]: pgmap v1423: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1789.275338 secs 2024-03-28T18:33:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:01 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1080.699284 secs 2024-03-28T18:33:03.189 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2435 sec 2024-03-28T18:33:04.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:03 smithi045 ceph-mon[31773]: pgmap v1424: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:04.189 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2436 sec 2024-03-28T18:33:04.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:03 smithi071 ceph-mon[36597]: pgmap v1424: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:04.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:03 smithi151 ceph-mon[36481]: pgmap v1424: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:05.189 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2437 sec 2024-03-28T18:33:06.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:05 smithi045 ceph-mon[31773]: pgmap v1425: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:06.189 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2438 sec 2024-03-28T18:33:06.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:05 smithi071 ceph-mon[36597]: pgmap v1425: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:06.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:05 smithi151 ceph-mon[36481]: pgmap v1425: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1794.275424 secs 2024-03-28T18:33:07.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:06 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1085.699344 secs 2024-03-28T18:33:07.189 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2439 sec 2024-03-28T18:33:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1794.275424 secs 2024-03-28T18:33:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:06 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1085.699344 secs 2024-03-28T18:33:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1794.275424 secs 2024-03-28T18:33:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:06 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1085.699344 secs 2024-03-28T18:33:08.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:07 smithi045 ceph-mon[31773]: pgmap v1426: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:08.189 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2440 sec 2024-03-28T18:33:08.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:07 smithi071 ceph-mon[36597]: pgmap v1426: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:08.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:07 smithi151 ceph-mon[36481]: pgmap v1426: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:09.190 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2441 sec 2024-03-28T18:33:10.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:09 smithi045 ceph-mon[31773]: pgmap v1427: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:10.190 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2442 sec 2024-03-28T18:33:10.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:09 smithi071 ceph-mon[36597]: pgmap v1427: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:10.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:09 smithi151 ceph-mon[36481]: pgmap v1427: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:11.190 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2443 sec 2024-03-28T18:33:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:11 smithi045 ceph-mon[31773]: pgmap v1428: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:12.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1799.275484 secs 2024-03-28T18:33:12.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:11 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1090.699430 secs 2024-03-28T18:33:12.190 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2444 sec 2024-03-28T18:33:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:11 smithi071 ceph-mon[36597]: pgmap v1428: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1799.275484 secs 2024-03-28T18:33:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:11 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1090.699430 secs 2024-03-28T18:33:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:11 smithi151 ceph-mon[36481]: pgmap v1428: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1799.275484 secs 2024-03-28T18:33:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:11 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1090.699430 secs 2024-03-28T18:33:13.190 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2445 sec 2024-03-28T18:33:14.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:13 smithi045 ceph-mon[31773]: pgmap v1429: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:14.190 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2446 sec 2024-03-28T18:33:14.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:13 smithi071 ceph-mon[36597]: pgmap v1429: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:14.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:13 smithi151 ceph-mon[36481]: pgmap v1429: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:15.191 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2447 sec 2024-03-28T18:33:16.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:15 smithi045 ceph-mon[31773]: pgmap v1430: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:16.190 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2448 sec 2024-03-28T18:33:16.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:15 smithi071 ceph-mon[36597]: pgmap v1430: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:16.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:15 smithi151 ceph-mon[36481]: pgmap v1430: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:17.191 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2449 sec 2024-03-28T18:33:17.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1804.275545 secs 2024-03-28T18:33:17.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:16 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1095.699520 secs 2024-03-28T18:33:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1804.275545 secs 2024-03-28T18:33:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:16 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1095.699520 secs 2024-03-28T18:33:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1804.275545 secs 2024-03-28T18:33:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:16 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1095.699520 secs 2024-03-28T18:33:18.191 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2450 sec 2024-03-28T18:33:18.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:17 smithi071 ceph-mon[36597]: pgmap v1431: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:18.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:17 smithi151 ceph-mon[36481]: pgmap v1431: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:18.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:17 smithi045 ceph-mon[31773]: pgmap v1431: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:19.191 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2451 sec 2024-03-28T18:33:20.191 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2452 sec 2024-03-28T18:33:20.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:19 smithi071 ceph-mon[36597]: pgmap v1432: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:20.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:19 smithi151 ceph-mon[36481]: pgmap v1432: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:20.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:19 smithi045 ceph-mon[31773]: pgmap v1432: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:21.191 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2453 sec 2024-03-28T18:33:22.192 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2454 sec 2024-03-28T18:33:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:21 smithi071 ceph-mon[36597]: pgmap v1433: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1809.275587 secs 2024-03-28T18:33:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:21 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1100.699607 secs 2024-03-28T18:33:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:21 smithi151 ceph-mon[36481]: pgmap v1433: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1809.275587 secs 2024-03-28T18:33:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:21 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1100.699607 secs 2024-03-28T18:33:22.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:21 smithi045 ceph-mon[31773]: pgmap v1433: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:22.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1809.275587 secs 2024-03-28T18:33:22.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:21 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1100.699607 secs 2024-03-28T18:33:23.192 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2455 sec 2024-03-28T18:33:24.192 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2456 sec 2024-03-28T18:33:24.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:23 smithi071 ceph-mon[36597]: pgmap v1434: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:24.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:23 smithi151 ceph-mon[36481]: pgmap v1434: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:24.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:23 smithi045 ceph-mon[31773]: pgmap v1434: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:25.192 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2457 sec 2024-03-28T18:33:26.192 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2458 sec 2024-03-28T18:33:26.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:25 smithi071 ceph-mon[36597]: pgmap v1435: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:25 smithi151 ceph-mon[36481]: pgmap v1435: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:26.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:25 smithi045 ceph-mon[31773]: pgmap v1435: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:27.192 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2459 sec 2024-03-28T18:33:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1814.275648 secs 2024-03-28T18:33:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:26 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1105.699695 secs 2024-03-28T18:33:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1814.275648 secs 2024-03-28T18:33:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:26 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1105.699695 secs 2024-03-28T18:33:27.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1814.275648 secs 2024-03-28T18:33:27.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:26 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1105.699695 secs 2024-03-28T18:33:28.193 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2460 sec 2024-03-28T18:33:28.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:27 smithi071 ceph-mon[36597]: pgmap v1436: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:28.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:27 smithi151 ceph-mon[36481]: pgmap v1436: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:28.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:27 smithi045 ceph-mon[31773]: pgmap v1436: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:29.193 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2461 sec 2024-03-28T18:33:30.193 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2462 sec 2024-03-28T18:33:30.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:29 smithi071 ceph-mon[36597]: pgmap v1437: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:30.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:29 smithi151 ceph-mon[36481]: pgmap v1437: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:30.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:29 smithi045 ceph-mon[31773]: pgmap v1437: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:31.193 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2463 sec 2024-03-28T18:33:32.193 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2464 sec 2024-03-28T18:33:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:31 smithi071 ceph-mon[36597]: pgmap v1438: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1819.275711 secs 2024-03-28T18:33:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:31 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1110.699756 secs 2024-03-28T18:33:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:31 smithi151 ceph-mon[36481]: pgmap v1438: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1819.275711 secs 2024-03-28T18:33:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:31 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1110.699756 secs 2024-03-28T18:33:32.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:31 smithi045 ceph-mon[31773]: pgmap v1438: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:32.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1819.275711 secs 2024-03-28T18:33:32.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:31 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1110.699756 secs 2024-03-28T18:33:33.193 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2465 sec 2024-03-28T18:33:34.193 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2466 sec 2024-03-28T18:33:34.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:33 smithi071 ceph-mon[36597]: pgmap v1439: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:34.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:33 smithi151 ceph-mon[36481]: pgmap v1439: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:34.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:33 smithi045 ceph-mon[31773]: pgmap v1439: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:35.193 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2467 sec 2024-03-28T18:33:36.193 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2468 sec 2024-03-28T18:33:36.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:35 smithi071 ceph-mon[36597]: pgmap v1440: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:36.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:35 smithi151 ceph-mon[36481]: pgmap v1440: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:36.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:35 smithi045 ceph-mon[31773]: pgmap v1440: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:37.194 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2469 sec 2024-03-28T18:33:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:36 smithi071 ceph-mon[36597]: 3 slow requests, 1 included below; oldest blocked for > 1824.275770 secs 2024-03-28T18:33:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:36 smithi071 ceph-mon[36597]: slow request 184.446879 seconds old, received at 2024-03-28T18:30:31.389436+0000: client_request(client.26705:1 getattr pAsLsXsFs #0x1 2024-03-28T18:30:31.385264+0000 caller_uid=0, caller_gid=0{}) currently dispatched 2024-03-28T18:33:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:36 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1115.699845 secs 2024-03-28T18:33:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:36 smithi151 ceph-mon[36481]: 3 slow requests, 1 included below; oldest blocked for > 1824.275770 secs 2024-03-28T18:33:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:36 smithi151 ceph-mon[36481]: slow request 184.446879 seconds old, received at 2024-03-28T18:30:31.389436+0000: client_request(client.26705:1 getattr pAsLsXsFs #0x1 2024-03-28T18:30:31.385264+0000 caller_uid=0, caller_gid=0{}) currently dispatched 2024-03-28T18:33:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:36 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1115.699845 secs 2024-03-28T18:33:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:36 smithi045 ceph-mon[31773]: 3 slow requests, 1 included below; oldest blocked for > 1824.275770 secs 2024-03-28T18:33:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:36 smithi045 ceph-mon[31773]: slow request 184.446879 seconds old, received at 2024-03-28T18:30:31.389436+0000: client_request(client.26705:1 getattr pAsLsXsFs #0x1 2024-03-28T18:30:31.385264+0000 caller_uid=0, caller_gid=0{}) currently dispatched 2024-03-28T18:33:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:36 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1115.699845 secs 2024-03-28T18:33:38.194 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2470 sec 2024-03-28T18:33:38.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:37 smithi071 ceph-mon[36597]: pgmap v1441: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:38.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:37 smithi151 ceph-mon[36481]: pgmap v1441: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:38.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:37 smithi045 ceph-mon[31773]: pgmap v1441: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:39.194 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2471 sec 2024-03-28T18:33:39.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:39 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] up:active 2024-03-28T18:33:39.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:39 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T18:33:39.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:39 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] up:active 2024-03-28T18:33:39.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:39 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T18:33:39.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:39 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] up:active 2024-03-28T18:33:39.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:39 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T18:33:40.194 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2472 sec 2024-03-28T18:33:40.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:40 smithi071 ceph-mon[36597]: pgmap v1442: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:40.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:40 smithi151 ceph-mon[36481]: pgmap v1442: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:40.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:40 smithi045 ceph-mon[31773]: pgmap v1442: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:41.194 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2473 sec 2024-03-28T18:33:42.194 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2474 sec 2024-03-28T18:33:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:42 smithi071 ceph-mon[36597]: pgmap v1443: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:42 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1829.275855 secs 2024-03-28T18:33:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1120.699907 secs 2024-03-28T18:33:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:42 smithi151 ceph-mon[36481]: pgmap v1443: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:42 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1829.275855 secs 2024-03-28T18:33:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1120.699907 secs 2024-03-28T18:33:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:42 smithi045 ceph-mon[31773]: pgmap v1443: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:42 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1829.275855 secs 2024-03-28T18:33:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1120.699907 secs 2024-03-28T18:33:43.194 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2475 sec 2024-03-28T18:33:44.195 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2476 sec 2024-03-28T18:33:44.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:44 smithi071 ceph-mon[36597]: pgmap v1444: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:44.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:44 smithi151 ceph-mon[36481]: pgmap v1444: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:44.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:44 smithi045 ceph-mon[31773]: pgmap v1444: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:45.195 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2477 sec 2024-03-28T18:33:46.195 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2478 sec 2024-03-28T18:33:46.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:46 smithi071 ceph-mon[36597]: pgmap v1445: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:46.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:46 smithi151 ceph-mon[36481]: pgmap v1445: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:46.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:46 smithi045 ceph-mon[31773]: pgmap v1445: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:47.195 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2479 sec 2024-03-28T18:33:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:47 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1834.275916 secs 2024-03-28T18:33:47.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1125.699993 secs 2024-03-28T18:33:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:47 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1834.275916 secs 2024-03-28T18:33:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1125.699993 secs 2024-03-28T18:33:47.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:47 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1834.275916 secs 2024-03-28T18:33:47.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1125.699993 secs 2024-03-28T18:33:48.195 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2480 sec 2024-03-28T18:33:48.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:48 smithi071 ceph-mon[36597]: pgmap v1446: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:48 smithi151 ceph-mon[36481]: pgmap v1446: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:48.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:48 smithi045 ceph-mon[31773]: pgmap v1446: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:33:49.195 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2481 sec 2024-03-28T18:33:49.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:49 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:49.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:49 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:49.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:49 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:50.195 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2482 sec 2024-03-28T18:33:50.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:50 smithi071 ceph-mon[36597]: pgmap v1447: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:50.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:50 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:50.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:50 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:33:50.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:50 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:33:50.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:50 smithi151 ceph-mon[36481]: pgmap v1447: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:50.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:50 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:50.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:50 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:33:50.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:50 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:33:50.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:50 smithi045 ceph-mon[31773]: pgmap v1447: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:33:50.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:50 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:50.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:50 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:33:50.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:50 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:33:51.195 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2483 sec 2024-03-28T18:33:51.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:51 smithi071 ceph-mon[36597]: pgmap v1448: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:33:51.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:51 smithi071 ceph-mon[36597]: pgmap v1449: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:51.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:51 smithi151 ceph-mon[36481]: pgmap v1448: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:33:51.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:51 smithi151 ceph-mon[36481]: pgmap v1449: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:51.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:51 smithi045 ceph-mon[31773]: pgmap v1448: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:33:51.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:51 smithi045 ceph-mon[31773]: pgmap v1449: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:52.196 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2484 sec 2024-03-28T18:33:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:52 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1839.276003 secs 2024-03-28T18:33:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1130.700077 secs 2024-03-28T18:33:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:52 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1839.276003 secs 2024-03-28T18:33:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1130.700077 secs 2024-03-28T18:33:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:52 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1839.276003 secs 2024-03-28T18:33:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1130.700077 secs 2024-03-28T18:33:53.196 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2485 sec 2024-03-28T18:33:53.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:53 smithi045 ceph-mon[31773]: pgmap v1450: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:53.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:53 smithi151 ceph-mon[36481]: pgmap v1450: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:53.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:53 smithi071 ceph-mon[36597]: pgmap v1450: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:54.196 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2486 sec 2024-03-28T18:33:55.196 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2487 sec 2024-03-28T18:33:55.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:55 smithi045 ceph-mon[31773]: pgmap v1451: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:55 smithi071 ceph-mon[36597]: pgmap v1451: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:55.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:55 smithi151 ceph-mon[36481]: pgmap v1451: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:56.196 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2488 sec 2024-03-28T18:33:56.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:56 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1844.276067 secs 2024-03-28T18:33:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:56 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1844.276067 secs 2024-03-28T18:33:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:56 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1844.276067 secs 2024-03-28T18:33:57.196 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2489 sec 2024-03-28T18:33:57.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:57 smithi045 ceph-mon[31773]: pgmap v1452: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:57.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1135.700138 secs 2024-03-28T18:33:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:57 smithi071 ceph-mon[36597]: pgmap v1452: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1135.700138 secs 2024-03-28T18:33:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:57 smithi151 ceph-mon[36481]: pgmap v1452: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1135.700138 secs 2024-03-28T18:33:58.197 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2490 sec 2024-03-28T18:33:59.197 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2491 sec 2024-03-28T18:33:59.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:33:59 smithi045 ceph-mon[31773]: pgmap v1453: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:59.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:33:59 smithi071 ceph-mon[36597]: pgmap v1453: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:33:59.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:33:59 smithi151 ceph-mon[36481]: pgmap v1453: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:00.197 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2492 sec 2024-03-28T18:34:01.197 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2493 sec 2024-03-28T18:34:01.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:01 smithi045 ceph-mon[31773]: pgmap v1454: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:34:01.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:01 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1849.276150 secs 2024-03-28T18:34:01.513 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:01 smithi151 ceph-mon[36481]: pgmap v1454: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:34:01.513 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:01 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1849.276150 secs 2024-03-28T18:34:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:01 smithi071 ceph-mon[36597]: pgmap v1454: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:34:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:01 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1849.276150 secs 2024-03-28T18:34:02.197 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2494 sec 2024-03-28T18:34:02.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1140.700200 secs 2024-03-28T18:34:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1140.700200 secs 2024-03-28T18:34:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1140.700200 secs 2024-03-28T18:34:03.197 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2495 sec 2024-03-28T18:34:03.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:03 smithi045 ceph-mon[31773]: pgmap v1455: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:03.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:03 smithi071 ceph-mon[36597]: pgmap v1455: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:03 smithi151 ceph-mon[36481]: pgmap v1455: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:04.197 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2496 sec 2024-03-28T18:34:05.198 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2497 sec 2024-03-28T18:34:05.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:05 smithi045 ceph-mon[31773]: pgmap v1456: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:05.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:05 smithi071 ceph-mon[36597]: pgmap v1456: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:05.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:05 smithi151 ceph-mon[36481]: pgmap v1456: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:06.198 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2498 sec 2024-03-28T18:34:06.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:06 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1854.276189 secs 2024-03-28T18:34:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:06 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1854.276189 secs 2024-03-28T18:34:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:06 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1854.276189 secs 2024-03-28T18:34:07.198 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2499 sec 2024-03-28T18:34:07.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:07 smithi045 ceph-mon[31773]: pgmap v1457: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:07.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1145.700301 secs 2024-03-28T18:34:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:07 smithi071 ceph-mon[36597]: pgmap v1457: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1145.700301 secs 2024-03-28T18:34:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:07 smithi151 ceph-mon[36481]: pgmap v1457: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1145.700301 secs 2024-03-28T18:34:08.198 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2500 sec 2024-03-28T18:34:09.198 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2501 sec 2024-03-28T18:34:09.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:09 smithi045 ceph-mon[31773]: pgmap v1458: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:09.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:09 smithi071 ceph-mon[36597]: pgmap v1458: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:09.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:09 smithi151 ceph-mon[36481]: pgmap v1458: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:10.198 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2502 sec 2024-03-28T18:34:11.198 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2503 sec 2024-03-28T18:34:11.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:11 smithi045 ceph-mon[31773]: pgmap v1459: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:11.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:11 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1859.276252 secs 2024-03-28T18:34:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:11 smithi071 ceph-mon[36597]: pgmap v1459: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:11 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1859.276252 secs 2024-03-28T18:34:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:11 smithi151 ceph-mon[36481]: pgmap v1459: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:11 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1859.276252 secs 2024-03-28T18:34:12.199 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2504 sec 2024-03-28T18:34:12.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1150.700394 secs 2024-03-28T18:34:12.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1150.700394 secs 2024-03-28T18:34:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1150.700394 secs 2024-03-28T18:34:13.199 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2505 sec 2024-03-28T18:34:13.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:13 smithi045 ceph-mon[31773]: pgmap v1460: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:13.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:13 smithi071 ceph-mon[36597]: pgmap v1460: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:13.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:13 smithi151 ceph-mon[36481]: pgmap v1460: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:14.199 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2506 sec 2024-03-28T18:34:15.199 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2507 sec 2024-03-28T18:34:15.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:15 smithi045 ceph-mon[31773]: pgmap v1461: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:15.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:15 smithi071 ceph-mon[36597]: pgmap v1461: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:15 smithi151 ceph-mon[36481]: pgmap v1461: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:16.199 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2508 sec 2024-03-28T18:34:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:16 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1864.276339 secs 2024-03-28T18:34:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:16 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1864.276339 secs 2024-03-28T18:34:16.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:16 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1864.276339 secs 2024-03-28T18:34:17.199 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2509 sec 2024-03-28T18:34:17.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:17 smithi071 ceph-mon[36597]: pgmap v1462: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:17.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1155.700455 secs 2024-03-28T18:34:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:17 smithi151 ceph-mon[36481]: pgmap v1462: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1155.700455 secs 2024-03-28T18:34:17.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:17 smithi045 ceph-mon[31773]: pgmap v1462: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:17.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1155.700455 secs 2024-03-28T18:34:18.199 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2510 sec 2024-03-28T18:34:19.199 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2511 sec 2024-03-28T18:34:19.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:19 smithi071 ceph-mon[36597]: pgmap v1463: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:19.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:19 smithi151 ceph-mon[36481]: pgmap v1463: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:19.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:19 smithi045 ceph-mon[31773]: pgmap v1463: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:20.200 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2512 sec 2024-03-28T18:34:21.200 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2513 sec 2024-03-28T18:34:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:21 smithi071 ceph-mon[36597]: pgmap v1464: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:21 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1869.276428 secs 2024-03-28T18:34:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:21 smithi151 ceph-mon[36481]: pgmap v1464: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:21 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1869.276428 secs 2024-03-28T18:34:21.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:21 smithi045 ceph-mon[31773]: pgmap v1464: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:21.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:21 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1869.276428 secs 2024-03-28T18:34:22.200 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2514 sec 2024-03-28T18:34:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1160.700540 secs 2024-03-28T18:34:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1160.700540 secs 2024-03-28T18:34:22.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1160.700540 secs 2024-03-28T18:34:23.200 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2515 sec 2024-03-28T18:34:23.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:23 smithi071 ceph-mon[36597]: pgmap v1465: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:23.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:23 smithi045 ceph-mon[31773]: pgmap v1465: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:23.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:23 smithi151 ceph-mon[36481]: pgmap v1465: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:24.200 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2516 sec 2024-03-28T18:34:25.201 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2517 sec 2024-03-28T18:34:25.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:25 smithi071 ceph-mon[36597]: pgmap v1466: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:25.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:25 smithi151 ceph-mon[36481]: pgmap v1466: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:25.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:25 smithi045 ceph-mon[31773]: pgmap v1466: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:26.200 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2518 sec 2024-03-28T18:34:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:26 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1874.276465 secs 2024-03-28T18:34:26.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:26 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1874.276465 secs 2024-03-28T18:34:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:26 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1874.276465 secs 2024-03-28T18:34:27.200 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2519 sec 2024-03-28T18:34:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:27 smithi071 ceph-mon[36597]: pgmap v1467: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1165.700629 secs 2024-03-28T18:34:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:27 smithi151 ceph-mon[36481]: pgmap v1467: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1165.700629 secs 2024-03-28T18:34:27.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:27 smithi045 ceph-mon[31773]: pgmap v1467: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:27.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1165.700629 secs 2024-03-28T18:34:28.201 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2520 sec 2024-03-28T18:34:29.201 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2521 sec 2024-03-28T18:34:29.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:29 smithi071 ceph-mon[36597]: pgmap v1468: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:29.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:29 smithi151 ceph-mon[36481]: pgmap v1468: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:29.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:29 smithi045 ceph-mon[31773]: pgmap v1468: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:30.201 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2522 sec 2024-03-28T18:34:31.201 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2523 sec 2024-03-28T18:34:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:31 smithi071 ceph-mon[36597]: pgmap v1469: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:31 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1879.276528 secs 2024-03-28T18:34:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:31 smithi151 ceph-mon[36481]: pgmap v1469: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:31 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1879.276528 secs 2024-03-28T18:34:31.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:31 smithi045 ceph-mon[31773]: pgmap v1469: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:31.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:31 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1879.276528 secs 2024-03-28T18:34:32.201 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2524 sec 2024-03-28T18:34:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1170.700718 secs 2024-03-28T18:34:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1170.700718 secs 2024-03-28T18:34:32.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1170.700718 secs 2024-03-28T18:34:33.201 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2525 sec 2024-03-28T18:34:33.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:33 smithi071 ceph-mon[36597]: pgmap v1470: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:33 smithi151 ceph-mon[36481]: pgmap v1470: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:33.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:33 smithi045 ceph-mon[31773]: pgmap v1470: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:34.202 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2526 sec 2024-03-28T18:34:35.202 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2527 sec 2024-03-28T18:34:35.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:35 smithi071 ceph-mon[36597]: pgmap v1471: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:35.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:35 smithi151 ceph-mon[36481]: pgmap v1471: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:35.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:35 smithi045 ceph-mon[31773]: pgmap v1471: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:36.202 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2528 sec 2024-03-28T18:34:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:36 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1884.276588 secs 2024-03-28T18:34:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:36 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1884.276588 secs 2024-03-28T18:34:36.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:36 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1884.276588 secs 2024-03-28T18:34:37.202 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2529 sec 2024-03-28T18:34:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:37 smithi071 ceph-mon[36597]: pgmap v1472: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1175.700779 secs 2024-03-28T18:34:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:37 smithi151 ceph-mon[36481]: pgmap v1472: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1175.700779 secs 2024-03-28T18:34:37.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:37 smithi045 ceph-mon[31773]: pgmap v1472: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:37.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1175.700779 secs 2024-03-28T18:34:38.202 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2530 sec 2024-03-28T18:34:39.202 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2531 sec 2024-03-28T18:34:39.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:39 smithi045 ceph-mon[31773]: pgmap v1473: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:39.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:39 smithi071 ceph-mon[36597]: pgmap v1473: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:39.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:39 smithi151 ceph-mon[36481]: pgmap v1473: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:40.202 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2532 sec 2024-03-28T18:34:41.203 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2533 sec 2024-03-28T18:34:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:41 smithi071 ceph-mon[36597]: pgmap v1474: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:41 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1889.276651 secs 2024-03-28T18:34:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:41 smithi151 ceph-mon[36481]: pgmap v1474: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:41 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1889.276651 secs 2024-03-28T18:34:41.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:41 smithi045 ceph-mon[31773]: pgmap v1474: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:41.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:41 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1889.276651 secs 2024-03-28T18:34:42.203 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2534 sec 2024-03-28T18:34:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1180.700822 secs 2024-03-28T18:34:42.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1180.700822 secs 2024-03-28T18:34:42.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1180.700822 secs 2024-03-28T18:34:43.203 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2535 sec 2024-03-28T18:34:43.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:43 smithi071 ceph-mon[36597]: pgmap v1475: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:43.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:43 smithi151 ceph-mon[36481]: pgmap v1475: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:43.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:43 smithi045 ceph-mon[31773]: pgmap v1475: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:44.203 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2536 sec 2024-03-28T18:34:45.203 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2537 sec 2024-03-28T18:34:45.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:45 smithi071 ceph-mon[36597]: pgmap v1476: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:45.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:45 smithi151 ceph-mon[36481]: pgmap v1476: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:45.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:45 smithi045 ceph-mon[31773]: pgmap v1476: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:46.203 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2538 sec 2024-03-28T18:34:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:46 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1894.276714 secs 2024-03-28T18:34:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:46 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1894.276714 secs 2024-03-28T18:34:46.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:46 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1894.276714 secs 2024-03-28T18:34:47.203 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2539 sec 2024-03-28T18:34:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:47 smithi071 ceph-mon[36597]: pgmap v1477: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1185.700885 secs 2024-03-28T18:34:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:47 smithi151 ceph-mon[36481]: pgmap v1477: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1185.700885 secs 2024-03-28T18:34:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:47 smithi045 ceph-mon[31773]: pgmap v1477: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1185.700885 secs 2024-03-28T18:34:48.204 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2540 sec 2024-03-28T18:34:49.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:48 smithi071 ceph-mon[36597]: pgmap v1478: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:49.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:48 smithi151 ceph-mon[36481]: pgmap v1478: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:49.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:48 smithi045 ceph-mon[31773]: pgmap v1478: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:49.204 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2541 sec 2024-03-28T18:34:50.204 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2542 sec 2024-03-28T18:34:51.204 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2543 sec 2024-03-28T18:34:51.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:51 smithi045 ceph-mon[31773]: pgmap v1479: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:51.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:51 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:51.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:51 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1899.276780 secs 2024-03-28T18:34:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:51 smithi071 ceph-mon[36597]: pgmap v1479: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:51 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:51 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1899.276780 secs 2024-03-28T18:34:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:51 smithi151 ceph-mon[36481]: pgmap v1479: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:34:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:51 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:51 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1899.276780 secs 2024-03-28T18:34:52.204 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2544 sec 2024-03-28T18:34:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1190.700925 secs 2024-03-28T18:34:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:52 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:52 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:52 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:34:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1190.700925 secs 2024-03-28T18:34:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:52 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:52.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:52 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:52.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:52 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:34:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1190.700925 secs 2024-03-28T18:34:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:52 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:52 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:52 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:34:53.204 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2545 sec 2024-03-28T18:34:53.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:53 smithi045 ceph-mon[31773]: pgmap v1480: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:53.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:53 smithi045 ceph-mon[31773]: pgmap v1481: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:53.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:53 smithi071 ceph-mon[36597]: pgmap v1480: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:53.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:53 smithi071 ceph-mon[36597]: pgmap v1481: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:53.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:53 smithi151 ceph-mon[36481]: pgmap v1480: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:34:53.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:53 smithi151 ceph-mon[36481]: pgmap v1481: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:54.205 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2546 sec 2024-03-28T18:34:55.205 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2547 sec 2024-03-28T18:34:55.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:55 smithi045 ceph-mon[31773]: pgmap v1482: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:55 smithi071 ceph-mon[36597]: pgmap v1482: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:55.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:55 smithi151 ceph-mon[36481]: pgmap v1482: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:56.205 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2548 sec 2024-03-28T18:34:56.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:56 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1904.276839 secs 2024-03-28T18:34:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:56 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1904.276839 secs 2024-03-28T18:34:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:56 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1904.276839 secs 2024-03-28T18:34:57.205 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2549 sec 2024-03-28T18:34:57.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:57 smithi045 ceph-mon[31773]: pgmap v1483: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:57.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1195.700987 secs 2024-03-28T18:34:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:57 smithi071 ceph-mon[36597]: pgmap v1483: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:57.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1195.700987 secs 2024-03-28T18:34:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:57 smithi151 ceph-mon[36481]: pgmap v1483: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1195.700987 secs 2024-03-28T18:34:58.205 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2550 sec 2024-03-28T18:34:59.206 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2551 sec 2024-03-28T18:34:59.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:34:59 smithi045 ceph-mon[31773]: pgmap v1484: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:59.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:34:59 smithi071 ceph-mon[36597]: pgmap v1484: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:34:59.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:34:59 smithi151 ceph-mon[36481]: pgmap v1484: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:35:00.205 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2552 sec 2024-03-28T18:35:01.206 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2553 sec 2024-03-28T18:35:01.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:01 smithi045 ceph-mon[31773]: pgmap v1485: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:35:01.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:01 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1909.276899 secs 2024-03-28T18:35:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:01 smithi071 ceph-mon[36597]: pgmap v1485: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:35:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:01 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1909.276899 secs 2024-03-28T18:35:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:01 smithi151 ceph-mon[36481]: pgmap v1485: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:35:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:01 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1909.276899 secs 2024-03-28T18:35:02.206 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2554 sec 2024-03-28T18:35:02.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1200.701047 secs 2024-03-28T18:35:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1200.701047 secs 2024-03-28T18:35:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1200.701047 secs 2024-03-28T18:35:03.206 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2555 sec 2024-03-28T18:35:03.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:03 smithi045 ceph-mon[31773]: pgmap v1486: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:35:03.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:03 smithi071 ceph-mon[36597]: pgmap v1486: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:35:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:03 smithi151 ceph-mon[36481]: pgmap v1486: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:35:04.206 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2556 sec 2024-03-28T18:35:05.206 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2557 sec 2024-03-28T18:35:05.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:05 smithi045 ceph-mon[31773]: pgmap v1487: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:05.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:05 smithi071 ceph-mon[36597]: pgmap v1487: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:05.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:05 smithi151 ceph-mon[36481]: pgmap v1487: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:06.206 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2558 sec 2024-03-28T18:35:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:06 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1914.276960 secs 2024-03-28T18:35:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:06 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1914.276960 secs 2024-03-28T18:35:06.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:06 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1914.276960 secs 2024-03-28T18:35:07.207 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2559 sec 2024-03-28T18:35:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:07 smithi071 ceph-mon[36597]: pgmap v1488: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:07.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1205.701130 secs 2024-03-28T18:35:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:07 smithi151 ceph-mon[36481]: pgmap v1488: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1205.701130 secs 2024-03-28T18:35:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:07 smithi045 ceph-mon[31773]: pgmap v1488: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:07.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1205.701130 secs 2024-03-28T18:35:08.207 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2560 sec 2024-03-28T18:35:09.207 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2561 sec 2024-03-28T18:35:09.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:09 smithi071 ceph-mon[36597]: pgmap v1489: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:09.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:09 smithi151 ceph-mon[36481]: pgmap v1489: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:09.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:09 smithi045 ceph-mon[31773]: pgmap v1489: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:10.207 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2562 sec 2024-03-28T18:35:11.207 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2563 sec 2024-03-28T18:35:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:11 smithi071 ceph-mon[36597]: pgmap v1490: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:11 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1919.277021 secs 2024-03-28T18:35:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:11 smithi151 ceph-mon[36481]: pgmap v1490: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:11 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1919.277021 secs 2024-03-28T18:35:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:11 smithi045 ceph-mon[31773]: pgmap v1490: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:11 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1919.277021 secs 2024-03-28T18:35:12.207 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2564 sec 2024-03-28T18:35:12.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1210.701189 secs 2024-03-28T18:35:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1210.701189 secs 2024-03-28T18:35:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1210.701189 secs 2024-03-28T18:35:13.207 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2565 sec 2024-03-28T18:35:13.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:13 smithi071 ceph-mon[36597]: pgmap v1491: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:13.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:13 smithi151 ceph-mon[36481]: pgmap v1491: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:13.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:13 smithi045 ceph-mon[31773]: pgmap v1491: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:14.209 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2566 sec 2024-03-28T18:35:15.207 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2567 sec 2024-03-28T18:35:15.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:15 smithi071 ceph-mon[36597]: pgmap v1492: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:15 smithi151 ceph-mon[36481]: pgmap v1492: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:15.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:15 smithi045 ceph-mon[31773]: pgmap v1492: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:16.208 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2568 sec 2024-03-28T18:35:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:16 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1924.277063 secs 2024-03-28T18:35:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:16 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1924.277063 secs 2024-03-28T18:35:16.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:16 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1924.277063 secs 2024-03-28T18:35:17.208 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2569 sec 2024-03-28T18:35:17.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:17 smithi071 ceph-mon[36597]: pgmap v1493: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:17.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1215.701276 secs 2024-03-28T18:35:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:17 smithi151 ceph-mon[36481]: pgmap v1493: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1215.701276 secs 2024-03-28T18:35:17.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:17 smithi045 ceph-mon[31773]: pgmap v1493: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:17.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1215.701276 secs 2024-03-28T18:35:18.208 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2570 sec 2024-03-28T18:35:19.208 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2571 sec 2024-03-28T18:35:19.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:19 smithi071 ceph-mon[36597]: pgmap v1494: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:19.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:19 smithi151 ceph-mon[36481]: pgmap v1494: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:19.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:19 smithi045 ceph-mon[31773]: pgmap v1494: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:20.208 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2572 sec 2024-03-28T18:35:21.208 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2573 sec 2024-03-28T18:35:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:21 smithi071 ceph-mon[36597]: pgmap v1495: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:21 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1929.277124 secs 2024-03-28T18:35:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:21 smithi151 ceph-mon[36481]: pgmap v1495: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:21 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1929.277124 secs 2024-03-28T18:35:21.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:21 smithi045 ceph-mon[31773]: pgmap v1495: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:21.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:21 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1929.277124 secs 2024-03-28T18:35:22.208 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2574 sec 2024-03-28T18:35:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1220.701367 secs 2024-03-28T18:35:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1220.701367 secs 2024-03-28T18:35:22.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1220.701367 secs 2024-03-28T18:35:23.209 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2575 sec 2024-03-28T18:35:23.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:23 smithi071 ceph-mon[36597]: pgmap v1496: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:23.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:23 smithi151 ceph-mon[36481]: pgmap v1496: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:23.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:23 smithi045 ceph-mon[31773]: pgmap v1496: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:24.209 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2576 sec 2024-03-28T18:35:25.209 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2577 sec 2024-03-28T18:35:25.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:25 smithi071 ceph-mon[36597]: pgmap v1497: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:25.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:25 smithi151 ceph-mon[36481]: pgmap v1497: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:25.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:25 smithi045 ceph-mon[31773]: pgmap v1497: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:26.209 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2578 sec 2024-03-28T18:35:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:26 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1934.277185 secs 2024-03-28T18:35:26.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:26 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1934.277185 secs 2024-03-28T18:35:26.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:26 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1934.277185 secs 2024-03-28T18:35:27.209 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2579 sec 2024-03-28T18:35:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:27 smithi071 ceph-mon[36597]: pgmap v1498: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1225.701456 secs 2024-03-28T18:35:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:27 smithi151 ceph-mon[36481]: pgmap v1498: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1225.701456 secs 2024-03-28T18:35:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:27 smithi045 ceph-mon[31773]: pgmap v1498: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1225.701456 secs 2024-03-28T18:35:28.210 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2580 sec 2024-03-28T18:35:29.210 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2581 sec 2024-03-28T18:35:29.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:29 smithi071 ceph-mon[36597]: pgmap v1499: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:29.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:29 smithi151 ceph-mon[36481]: pgmap v1499: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:29.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:29 smithi045 ceph-mon[31773]: pgmap v1499: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:30.210 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2582 sec 2024-03-28T18:35:31.210 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2583 sec 2024-03-28T18:35:31.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:31 smithi071 ceph-mon[36597]: pgmap v1500: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:31.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:31 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1939.277245 secs 2024-03-28T18:35:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:31 smithi151 ceph-mon[36481]: pgmap v1500: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:31 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1939.277245 secs 2024-03-28T18:35:31.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:31 smithi045 ceph-mon[31773]: pgmap v1500: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:31.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:31 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1939.277245 secs 2024-03-28T18:35:31.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:35:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:35:31.466+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:35:31.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:35:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:35:31.466+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:35:31.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:35:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:35:31.466+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:35:31.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:35:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:35:31.466+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:35:31.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:35:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:35:31.466+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:35:32.210 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2584 sec 2024-03-28T18:35:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1230.701542 secs 2024-03-28T18:35:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1230.701542 secs 2024-03-28T18:35:32.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1230.701542 secs 2024-03-28T18:35:33.210 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2585 sec 2024-03-28T18:35:33.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:33 smithi071 ceph-mon[36597]: pgmap v1501: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:33 smithi151 ceph-mon[36481]: pgmap v1501: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:33.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:33 smithi045 ceph-mon[31773]: pgmap v1501: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:34.210 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2586 sec 2024-03-28T18:35:35.210 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2587 sec 2024-03-28T18:35:35.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:35 smithi071 ceph-mon[36597]: pgmap v1502: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:35.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:35 smithi151 ceph-mon[36481]: pgmap v1502: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:35.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:35 smithi045 ceph-mon[31773]: pgmap v1502: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:36.211 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2588 sec 2024-03-28T18:35:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:36 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1944.277303 secs 2024-03-28T18:35:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:36 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1944.277303 secs 2024-03-28T18:35:36.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:36 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1944.277303 secs 2024-03-28T18:35:37.211 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2589 sec 2024-03-28T18:35:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:37 smithi071 ceph-mon[36597]: pgmap v1503: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T18:35:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1235.701632 secs 2024-03-28T18:35:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:37 smithi151 ceph-mon[36481]: pgmap v1503: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T18:35:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1235.701632 secs 2024-03-28T18:35:37.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:37 smithi045 ceph-mon[31773]: pgmap v1503: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T18:35:37.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1235.701632 secs 2024-03-28T18:35:38.211 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2590 sec 2024-03-28T18:35:39.211 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2591 sec 2024-03-28T18:35:39.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:39 smithi071 ceph-mon[36597]: pgmap v1504: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-28T18:35:39.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:39 smithi151 ceph-mon[36481]: pgmap v1504: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-28T18:35:39.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:39 smithi045 ceph-mon[31773]: pgmap v1504: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-28T18:35:40.212 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2592 sec 2024-03-28T18:35:41.211 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2593 sec 2024-03-28T18:35:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:41 smithi071 ceph-mon[36597]: pgmap v1505: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-28T18:35:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:41 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1949.277364 secs 2024-03-28T18:35:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:41 smithi151 ceph-mon[36481]: pgmap v1505: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-28T18:35:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:41 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1949.277364 secs 2024-03-28T18:35:41.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:41 smithi045 ceph-mon[31773]: pgmap v1505: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-28T18:35:41.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:41 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1949.277364 secs 2024-03-28T18:35:42.213 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2594 sec 2024-03-28T18:35:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1240.701728 secs 2024-03-28T18:35:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1240.701728 secs 2024-03-28T18:35:42.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1240.701728 secs 2024-03-28T18:35:43.212 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2595 sec 2024-03-28T18:35:43.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:43 smithi071 ceph-mon[36597]: pgmap v1506: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T18:35:43.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:43 smithi151 ceph-mon[36481]: pgmap v1506: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T18:35:43.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:43 smithi045 ceph-mon[31773]: pgmap v1506: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T18:35:44.212 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2596 sec 2024-03-28T18:35:45.212 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2597 sec 2024-03-28T18:35:45.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:45 smithi151 ceph-mon[36481]: pgmap v1507: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-28T18:35:45.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:45 smithi045 ceph-mon[31773]: pgmap v1507: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-28T18:35:45.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:45 smithi071 ceph-mon[36597]: pgmap v1507: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-28T18:35:46.212 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2598 sec 2024-03-28T18:35:46.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:46 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1954.277425 secs 2024-03-28T18:35:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:46 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1954.277425 secs 2024-03-28T18:35:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:46 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1954.277425 secs 2024-03-28T18:35:47.212 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2599 sec 2024-03-28T18:35:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:47 smithi045 ceph-mon[31773]: pgmap v1508: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T18:35:47.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1245.701821 secs 2024-03-28T18:35:47.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:47 smithi071 ceph-mon[36597]: pgmap v1508: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T18:35:47.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1245.701821 secs 2024-03-28T18:35:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:47 smithi151 ceph-mon[36481]: pgmap v1508: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T18:35:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1245.701821 secs 2024-03-28T18:35:48.212 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2600 sec 2024-03-28T18:35:49.212 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2601 sec 2024-03-28T18:35:49.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:49 smithi045 ceph-mon[31773]: pgmap v1509: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:49.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:49 smithi071 ceph-mon[36597]: pgmap v1509: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:49.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:49 smithi151 ceph-mon[36481]: pgmap v1509: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:50.213 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2602 sec 2024-03-28T18:35:51.213 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2603 sec 2024-03-28T18:35:51.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:51 smithi045 ceph-mon[31773]: pgmap v1510: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:51.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:51 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1959.277487 secs 2024-03-28T18:35:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:51 smithi071 ceph-mon[36597]: pgmap v1510: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:51 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1959.277487 secs 2024-03-28T18:35:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:51 smithi151 ceph-mon[36481]: pgmap v1510: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:51 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1959.277487 secs 2024-03-28T18:35:52.213 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2604 sec 2024-03-28T18:35:52.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1250.701881 secs 2024-03-28T18:35:52.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:52 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1250.701881 secs 2024-03-28T18:35:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:52 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1250.701881 secs 2024-03-28T18:35:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:52 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:53.213 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2605 sec 2024-03-28T18:35:53.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:53 smithi045 ceph-mon[31773]: pgmap v1511: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:53.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:53 smithi071 ceph-mon[36597]: pgmap v1511: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:53.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:53 smithi151 ceph-mon[36481]: pgmap v1511: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:35:54.213 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2606 sec 2024-03-28T18:35:54.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:54 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:54.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:54 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:54.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:54 smithi045 ceph-mon[31773]: pgmap v1512: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:54.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:54 smithi045 ceph-mon[31773]: pgmap v1513: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:35:54.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:54 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:35:54.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:54 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:54.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:54 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:54.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:54 smithi071 ceph-mon[36597]: pgmap v1512: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:54.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:54 smithi071 ceph-mon[36597]: pgmap v1513: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:35:54.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:54 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:35:54.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:54 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:54.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:54 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:54.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:54 smithi151 ceph-mon[36481]: pgmap v1512: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:35:54.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:54 smithi151 ceph-mon[36481]: pgmap v1513: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:35:54.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:54 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:35:55.213 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2607 sec 2024-03-28T18:35:56.213 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2608 sec 2024-03-28T18:35:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:56 smithi071 ceph-mon[36597]: pgmap v1514: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:35:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:56 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1964.277503 secs 2024-03-28T18:35:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:56 smithi151 ceph-mon[36481]: pgmap v1514: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:35:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:56 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1964.277503 secs 2024-03-28T18:35:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:56 smithi045 ceph-mon[31773]: pgmap v1514: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:35:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:56 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1964.277503 secs 2024-03-28T18:35:57.214 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2609 sec 2024-03-28T18:35:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1255.701971 secs 2024-03-28T18:35:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1255.701971 secs 2024-03-28T18:35:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1255.701971 secs 2024-03-28T18:35:58.214 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2610 sec 2024-03-28T18:35:58.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:35:58 smithi071 ceph-mon[36597]: pgmap v1515: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:35:58.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:35:58 smithi151 ceph-mon[36481]: pgmap v1515: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:35:58.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:35:58 smithi045 ceph-mon[31773]: pgmap v1515: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:35:59.214 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2611 sec 2024-03-28T18:36:00.214 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2612 sec 2024-03-28T18:36:00.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:00 smithi071 ceph-mon[36597]: pgmap v1516: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:00 smithi151 ceph-mon[36481]: pgmap v1516: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:00.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:00 smithi045 ceph-mon[31773]: pgmap v1516: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:01.214 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2613 sec 2024-03-28T18:36:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:01 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1969.277566 secs 2024-03-28T18:36:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:01 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1969.277566 secs 2024-03-28T18:36:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:01 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1969.277566 secs 2024-03-28T18:36:02.214 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2614 sec 2024-03-28T18:36:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:02 smithi071 ceph-mon[36597]: pgmap v1517: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1260.702056 secs 2024-03-28T18:36:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:02 smithi151 ceph-mon[36481]: pgmap v1517: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1260.702056 secs 2024-03-28T18:36:02.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:02 smithi045 ceph-mon[31773]: pgmap v1517: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:02.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1260.702056 secs 2024-03-28T18:36:03.214 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2615 sec 2024-03-28T18:36:04.214 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2616 sec 2024-03-28T18:36:04.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:04 smithi071 ceph-mon[36597]: pgmap v1518: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:36:04.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:04 smithi151 ceph-mon[36481]: pgmap v1518: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:36:04.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:04 smithi045 ceph-mon[31773]: pgmap v1518: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:36:05.214 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2617 sec 2024-03-28T18:36:06.215 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2618 sec 2024-03-28T18:36:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:06 smithi071 ceph-mon[36597]: pgmap v1519: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:06 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1974.277629 secs 2024-03-28T18:36:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:06 smithi151 ceph-mon[36481]: pgmap v1519: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:06 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1974.277629 secs 2024-03-28T18:36:06.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:06 smithi045 ceph-mon[31773]: pgmap v1519: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:06.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:06 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1974.277629 secs 2024-03-28T18:36:07.215 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2619 sec 2024-03-28T18:36:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1265.702072 secs 2024-03-28T18:36:07.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1265.702072 secs 2024-03-28T18:36:07.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1265.702072 secs 2024-03-28T18:36:08.215 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2620 sec 2024-03-28T18:36:08.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:08 smithi071 ceph-mon[36597]: pgmap v1520: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:08 smithi151 ceph-mon[36481]: pgmap v1520: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:08.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:08 smithi045 ceph-mon[31773]: pgmap v1520: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:09.215 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2621 sec 2024-03-28T18:36:10.215 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2622 sec 2024-03-28T18:36:10.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:10 smithi071 ceph-mon[36597]: pgmap v1521: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:10.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:10 smithi151 ceph-mon[36481]: pgmap v1521: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:10.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:10 smithi045 ceph-mon[31773]: pgmap v1521: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:11.215 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2623 sec 2024-03-28T18:36:11.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:11 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1979.277691 secs 2024-03-28T18:36:11.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:11 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1979.277691 secs 2024-03-28T18:36:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:11 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1979.277691 secs 2024-03-28T18:36:12.218 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2624 sec 2024-03-28T18:36:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:12 smithi045 ceph-mon[31773]: pgmap v1522: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:12.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1270.702162 secs 2024-03-28T18:36:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:12 smithi071 ceph-mon[36597]: pgmap v1522: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1270.702162 secs 2024-03-28T18:36:12.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:12 smithi151 ceph-mon[36481]: pgmap v1522: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:12.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1270.702162 secs 2024-03-28T18:36:13.216 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2625 sec 2024-03-28T18:36:14.216 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2626 sec 2024-03-28T18:36:14.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:14 smithi071 ceph-mon[36597]: pgmap v1523: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:14.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:14 smithi151 ceph-mon[36481]: pgmap v1523: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:14.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:14 smithi045 ceph-mon[31773]: pgmap v1523: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:15.216 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2627 sec 2024-03-28T18:36:16.216 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2628 sec 2024-03-28T18:36:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:16 smithi071 ceph-mon[36597]: pgmap v1524: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:16 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1984.277752 secs 2024-03-28T18:36:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:16 smithi151 ceph-mon[36481]: pgmap v1524: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:16 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1984.277752 secs 2024-03-28T18:36:16.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:16 smithi045 ceph-mon[31773]: pgmap v1524: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:16.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:16 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1984.277752 secs 2024-03-28T18:36:17.216 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2629 sec 2024-03-28T18:36:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1275.702251 secs 2024-03-28T18:36:17.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1275.702251 secs 2024-03-28T18:36:18.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1275.702251 secs 2024-03-28T18:36:18.217 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2630 sec 2024-03-28T18:36:18.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:18 smithi151 ceph-mon[36481]: pgmap v1525: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:18.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:18 smithi045 ceph-mon[31773]: pgmap v1525: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:19.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:18 smithi071 ceph-mon[36597]: pgmap v1525: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:19.217 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2631 sec 2024-03-28T18:36:20.217 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2632 sec 2024-03-28T18:36:20.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:20 smithi045 ceph-mon[31773]: pgmap v1526: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:21.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:20 smithi071 ceph-mon[36597]: pgmap v1526: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:21.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:20 smithi151 ceph-mon[36481]: pgmap v1526: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:21.217 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2633 sec 2024-03-28T18:36:21.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:21 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1989.277814 secs 2024-03-28T18:36:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:21 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1989.277814 secs 2024-03-28T18:36:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:21 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1989.277814 secs 2024-03-28T18:36:22.217 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2634 sec 2024-03-28T18:36:22.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:22 smithi045 ceph-mon[31773]: pgmap v1527: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:22.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1280.702315 secs 2024-03-28T18:36:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:22 smithi071 ceph-mon[36597]: pgmap v1527: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1280.702315 secs 2024-03-28T18:36:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:22 smithi151 ceph-mon[36481]: pgmap v1527: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1280.702315 secs 2024-03-28T18:36:23.217 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2635 sec 2024-03-28T18:36:24.217 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2636 sec 2024-03-28T18:36:24.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:24 smithi045 ceph-mon[31773]: pgmap v1528: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:25.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:24 smithi071 ceph-mon[36597]: pgmap v1528: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:25.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:24 smithi151 ceph-mon[36481]: pgmap v1528: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:25.217 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2637 sec 2024-03-28T18:36:26.218 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2638 sec 2024-03-28T18:36:26.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:26 smithi045 ceph-mon[31773]: pgmap v1529: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:26.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:26 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1994.277902 secs 2024-03-28T18:36:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:26 smithi071 ceph-mon[36597]: pgmap v1529: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:26 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1994.277902 secs 2024-03-28T18:36:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:26 smithi151 ceph-mon[36481]: pgmap v1529: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:26 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1994.277902 secs 2024-03-28T18:36:27.218 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2639 sec 2024-03-28T18:36:27.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1285.702330 secs 2024-03-28T18:36:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1285.702330 secs 2024-03-28T18:36:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1285.702330 secs 2024-03-28T18:36:28.218 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2640 sec 2024-03-28T18:36:28.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:28 smithi045 ceph-mon[31773]: pgmap v1530: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:29.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:28 smithi071 ceph-mon[36597]: pgmap v1530: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:29.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:28 smithi151 ceph-mon[36481]: pgmap v1530: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:29.218 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2641 sec 2024-03-28T18:36:30.218 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2642 sec 2024-03-28T18:36:30.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:30 smithi045 ceph-mon[31773]: pgmap v1531: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:31.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:30 smithi071 ceph-mon[36597]: pgmap v1531: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:31.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:30 smithi151 ceph-mon[36481]: pgmap v1531: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:31.218 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2643 sec 2024-03-28T18:36:31.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:31 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 1999.277941 secs 2024-03-28T18:36:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:31 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 1999.277941 secs 2024-03-28T18:36:32.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:31 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 1999.277941 secs 2024-03-28T18:36:32.218 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2644 sec 2024-03-28T18:36:32.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:32 smithi045 ceph-mon[31773]: pgmap v1532: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:32.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1290.702363 secs 2024-03-28T18:36:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:32 smithi071 ceph-mon[36597]: pgmap v1532: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1290.702363 secs 2024-03-28T18:36:33.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:32 smithi151 ceph-mon[36481]: pgmap v1532: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:33.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1290.702363 secs 2024-03-28T18:36:33.218 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2645 sec 2024-03-28T18:36:34.219 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2646 sec 2024-03-28T18:36:34.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:34 smithi045 ceph-mon[31773]: pgmap v1533: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:35.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:34 smithi071 ceph-mon[36597]: pgmap v1533: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:34 smithi151 ceph-mon[36481]: pgmap v1533: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:35.219 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2647 sec 2024-03-28T18:36:36.219 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2648 sec 2024-03-28T18:36:36.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:36 smithi045 ceph-mon[31773]: pgmap v1534: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:36.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:36 smithi045 ceph-mon[31773]: 3 slow requests, 1 included below; oldest blocked for > 2004.278003 secs 2024-03-28T18:36:36.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:36 smithi045 ceph-mon[31773]: slow request 364.449111 seconds old, received at 2024-03-28T18:30:31.389436+0000: client_request(client.26705:1 getattr pAsLsXsFs #0x1 2024-03-28T18:30:31.385264+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:36:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:36 smithi071 ceph-mon[36597]: pgmap v1534: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:36 smithi071 ceph-mon[36597]: 3 slow requests, 1 included below; oldest blocked for > 2004.278003 secs 2024-03-28T18:36:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:36 smithi071 ceph-mon[36597]: slow request 364.449111 seconds old, received at 2024-03-28T18:30:31.389436+0000: client_request(client.26705:1 getattr pAsLsXsFs #0x1 2024-03-28T18:30:31.385264+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:36:37.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:36 smithi151 ceph-mon[36481]: pgmap v1534: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:37.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:36 smithi151 ceph-mon[36481]: 3 slow requests, 1 included below; oldest blocked for > 2004.278003 secs 2024-03-28T18:36:37.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:36 smithi151 ceph-mon[36481]: slow request 364.449111 seconds old, received at 2024-03-28T18:30:31.389436+0000: client_request(client.26705:1 getattr pAsLsXsFs #0x1 2024-03-28T18:30:31.385264+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:36:37.219 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2649 sec 2024-03-28T18:36:37.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1295.702453 secs 2024-03-28T18:36:38.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1295.702453 secs 2024-03-28T18:36:38.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1295.702453 secs 2024-03-28T18:36:38.219 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2650 sec 2024-03-28T18:36:38.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:38 smithi045 ceph-mon[31773]: pgmap v1535: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:39.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:38 smithi071 ceph-mon[36597]: pgmap v1535: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:39.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:38 smithi151 ceph-mon[36481]: pgmap v1535: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:39.219 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2651 sec 2024-03-28T18:36:40.220 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2652 sec 2024-03-28T18:36:40.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:40 smithi045 ceph-mon[31773]: pgmap v1536: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:41.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:40 smithi071 ceph-mon[36597]: pgmap v1536: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:41.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:40 smithi151 ceph-mon[36481]: pgmap v1536: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:41.220 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2653 sec 2024-03-28T18:36:41.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:41 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 2009.278092 secs 2024-03-28T18:36:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:41 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 2009.278092 secs 2024-03-28T18:36:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:41 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 2009.278092 secs 2024-03-28T18:36:42.220 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2654 sec 2024-03-28T18:36:42.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:42 smithi045 ceph-mon[31773]: pgmap v1537: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:42.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1300.702466 secs 2024-03-28T18:36:42.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:42 smithi151 ceph-mon[36481]: pgmap v1537: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:42.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1300.702466 secs 2024-03-28T18:36:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:42 smithi071 ceph-mon[36597]: pgmap v1537: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1300.702466 secs 2024-03-28T18:36:43.220 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2655 sec 2024-03-28T18:36:44.220 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2656 sec 2024-03-28T18:36:45.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:44 smithi071 ceph-mon[36597]: pgmap v1538: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:45.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:44 smithi151 ceph-mon[36481]: pgmap v1538: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:45.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:44 smithi045 ceph-mon[31773]: pgmap v1538: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:45.220 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2657 sec 2024-03-28T18:36:46.220 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2658 sec 2024-03-28T18:36:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:46 smithi071 ceph-mon[36597]: pgmap v1539: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:46 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 2014.278180 secs 2024-03-28T18:36:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:46 smithi151 ceph-mon[36481]: pgmap v1539: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:46 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 2014.278180 secs 2024-03-28T18:36:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:46 smithi045 ceph-mon[31773]: pgmap v1539: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:46 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 2014.278180 secs 2024-03-28T18:36:47.220 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2659 sec 2024-03-28T18:36:47.961 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1305.702480 secs 2024-03-28T18:36:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1305.702480 secs 2024-03-28T18:36:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1305.702480 secs 2024-03-28T18:36:48.220 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2660 sec 2024-03-28T18:36:49.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:48 smithi071 ceph-mon[36597]: pgmap v1540: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:49.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:48 smithi151 ceph-mon[36481]: pgmap v1540: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:49.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:48 smithi045 ceph-mon[31773]: pgmap v1540: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:49.221 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2661 sec 2024-03-28T18:36:50.221 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2662 sec 2024-03-28T18:36:51.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:50 smithi071 ceph-mon[36597]: pgmap v1541: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:51.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:50 smithi151 ceph-mon[36481]: pgmap v1541: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:51.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:50 smithi045 ceph-mon[31773]: pgmap v1541: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:51.221 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2663 sec 2024-03-28T18:36:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:51 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 2019.278242 secs 2024-03-28T18:36:52.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:51 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 2019.278242 secs 2024-03-28T18:36:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:51 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 2019.278242 secs 2024-03-28T18:36:52.221 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2664 sec 2024-03-28T18:36:53.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:52 smithi071 ceph-mon[36597]: pgmap v1542: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:53.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1310.702542 secs 2024-03-28T18:36:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:52 smithi151 ceph-mon[36481]: pgmap v1542: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1310.702542 secs 2024-03-28T18:36:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:52 smithi045 ceph-mon[31773]: pgmap v1542: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:36:53.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1310.702542 secs 2024-03-28T18:36:53.221 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2665 sec 2024-03-28T18:36:54.221 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2666 sec 2024-03-28T18:36:55.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:54 smithi071 ceph-mon[36597]: pgmap v1543: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:55.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:54 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:36:55.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:54 smithi151 ceph-mon[36481]: pgmap v1543: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:55.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:54 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:36:55.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:54 smithi045 ceph-mon[31773]: pgmap v1543: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:36:55.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:54 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:36:55.221 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2667 sec 2024-03-28T18:36:56.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:55 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:56.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:55 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:36:56.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:55 smithi071 ceph-mon[36597]: pgmap v1544: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:36:56.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:55 smithi071 ceph-mon[36597]: pgmap v1545: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:56.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:55 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:36:56.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:55 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:56.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:55 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:36:56.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:55 smithi151 ceph-mon[36481]: pgmap v1544: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:36:56.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:55 smithi151 ceph-mon[36481]: pgmap v1545: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:56.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:55 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:36:56.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:55 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:56.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:55 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:36:56.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:55 smithi045 ceph-mon[31773]: pgmap v1544: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:36:56.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:55 smithi045 ceph-mon[31773]: pgmap v1545: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:56.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:55 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:36:56.222 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2668 sec 2024-03-28T18:36:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:56 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 2024.278331 secs 2024-03-28T18:36:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:56 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 2024.278331 secs 2024-03-28T18:36:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:56 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 2024.278331 secs 2024-03-28T18:36:57.222 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2669 sec 2024-03-28T18:36:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1315.702603 secs 2024-03-28T18:36:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:36:57 smithi071 ceph-mon[36597]: pgmap v1546: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:58.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1315.702603 secs 2024-03-28T18:36:58.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:36:57 smithi151 ceph-mon[36481]: pgmap v1546: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1315.702603 secs 2024-03-28T18:36:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:36:57 smithi045 ceph-mon[31773]: pgmap v1546: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:36:58.222 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2670 sec 2024-03-28T18:36:59.222 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2671 sec 2024-03-28T18:37:00.222 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2672 sec 2024-03-28T18:37:00.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:00 smithi045 ceph-mon[31773]: pgmap v1547: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:37:00.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:00 smithi071 ceph-mon[36597]: pgmap v1547: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:37:00.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:00 smithi151 ceph-mon[36481]: pgmap v1547: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:37:01.222 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2673 sec 2024-03-28T18:37:01.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:01 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 2029.278415 secs 2024-03-28T18:37:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:01 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 2029.278415 secs 2024-03-28T18:37:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:01 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 2029.278415 secs 2024-03-28T18:37:02.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:02 smithi045 ceph-mon[31773]: pgmap v1548: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:37:02.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1320.702692 secs 2024-03-28T18:37:02.222 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2674 sec 2024-03-28T18:37:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:02 smithi071 ceph-mon[36597]: pgmap v1548: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:37:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1320.702692 secs 2024-03-28T18:37:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:02 smithi151 ceph-mon[36481]: pgmap v1548: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:37:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1320.702692 secs 2024-03-28T18:37:03.223 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2675 sec 2024-03-28T18:37:04.223 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2676 sec 2024-03-28T18:37:04.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:04 smithi045 ceph-mon[31773]: pgmap v1549: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:37:04.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:04 smithi071 ceph-mon[36597]: pgmap v1549: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:37:04.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:04 smithi151 ceph-mon[36481]: pgmap v1549: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:37:05.223 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2677 sec 2024-03-28T18:37:06.223 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2678 sec 2024-03-28T18:37:06.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:06 smithi045 ceph-mon[31773]: pgmap v1550: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:37:06.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:06 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 2034.278486 secs 2024-03-28T18:37:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:06 smithi071 ceph-mon[36597]: pgmap v1550: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:37:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:06 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 2034.278486 secs 2024-03-28T18:37:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:06 smithi151 ceph-mon[36481]: pgmap v1550: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:37:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:06 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 2034.278486 secs 2024-03-28T18:37:07.223 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2679 sec 2024-03-28T18:37:07.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1325.702783 secs 2024-03-28T18:37:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1325.702783 secs 2024-03-28T18:37:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1325.702783 secs 2024-03-28T18:37:08.223 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2680 sec 2024-03-28T18:37:08.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:08 smithi045 ceph-mon[31773]: pgmap v1551: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:08.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:08 smithi071 ceph-mon[36597]: pgmap v1551: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:08.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:08 smithi151 ceph-mon[36481]: pgmap v1551: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:09.223 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2681 sec 2024-03-28T18:37:10.224 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2682 sec 2024-03-28T18:37:10.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:10 smithi045 ceph-mon[31773]: pgmap v1552: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:10.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:10 smithi071 ceph-mon[36597]: pgmap v1552: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:10.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:10 smithi151 ceph-mon[36481]: pgmap v1552: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:11.224 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2683 sec 2024-03-28T18:37:11.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:11 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 2039.278549 secs 2024-03-28T18:37:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:11 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 2039.278549 secs 2024-03-28T18:37:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:11 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 2039.278549 secs 2024-03-28T18:37:12.224 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2684 sec 2024-03-28T18:37:12.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:12 smithi045 ceph-mon[31773]: pgmap v1553: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:12.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1330.702843 secs 2024-03-28T18:37:12.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:12 smithi151 ceph-mon[36481]: pgmap v1553: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:12.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1330.702843 secs 2024-03-28T18:37:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:12 smithi071 ceph-mon[36597]: pgmap v1553: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1330.702843 secs 2024-03-28T18:37:13.224 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2685 sec 2024-03-28T18:37:14.224 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2686 sec 2024-03-28T18:37:14.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:14 smithi045 ceph-mon[31773]: pgmap v1554: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:14.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:14 smithi071 ceph-mon[36597]: pgmap v1554: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:14.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:14 smithi151 ceph-mon[36481]: pgmap v1554: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:15.224 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2687 sec 2024-03-28T18:37:16.225 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2688 sec 2024-03-28T18:37:16.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:16 smithi045 ceph-mon[31773]: pgmap v1555: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:16.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:16 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 2044.278610 secs 2024-03-28T18:37:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:16 smithi071 ceph-mon[36597]: pgmap v1555: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:16 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 2044.278610 secs 2024-03-28T18:37:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:16 smithi151 ceph-mon[36481]: pgmap v1555: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:16 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 2044.278610 secs 2024-03-28T18:37:17.225 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2689 sec 2024-03-28T18:37:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1335.702929 secs 2024-03-28T18:37:17.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1335.702929 secs 2024-03-28T18:37:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1335.702929 secs 2024-03-28T18:37:18.225 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2690 sec 2024-03-28T18:37:18.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:18 smithi045 ceph-mon[31773]: pgmap v1556: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:18.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:18 smithi071 ceph-mon[36597]: pgmap v1556: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:18 smithi151 ceph-mon[36481]: pgmap v1556: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:19.225 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2691 sec 2024-03-28T18:37:20.225 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2692 sec 2024-03-28T18:37:20.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:20 smithi071 ceph-mon[36597]: pgmap v1557: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:20.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:20 smithi151 ceph-mon[36481]: pgmap v1557: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:20.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:20 smithi045 ceph-mon[31773]: pgmap v1557: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:21.225 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2693 sec 2024-03-28T18:37:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:21 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 2049.278626 secs 2024-03-28T18:37:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:21 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 2049.278626 secs 2024-03-28T18:37:21.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:21 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 2049.278626 secs 2024-03-28T18:37:22.226 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2694 sec 2024-03-28T18:37:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:22 smithi071 ceph-mon[36597]: pgmap v1558: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:37:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1340.703015 secs 2024-03-28T18:37:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:22 smithi151 ceph-mon[36481]: pgmap v1558: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:37:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1340.703015 secs 2024-03-28T18:37:22.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:22 smithi045 ceph-mon[31773]: pgmap v1558: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:37:22.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1340.703015 secs 2024-03-28T18:37:23.226 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2695 sec 2024-03-28T18:37:24.226 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2696 sec 2024-03-28T18:37:24.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:24 smithi071 ceph-mon[36597]: pgmap v1559: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:37:24.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:24 smithi151 ceph-mon[36481]: pgmap v1559: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:37:24.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:24 smithi045 ceph-mon[31773]: pgmap v1559: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:37:25.226 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2697 sec 2024-03-28T18:37:26.226 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2698 sec 2024-03-28T18:37:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:26 smithi071 ceph-mon[36597]: pgmap v1560: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:37:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:26 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 2054.278686 secs 2024-03-28T18:37:26.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:26 smithi151 ceph-mon[36481]: pgmap v1560: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:37:26.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:26 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 2054.278686 secs 2024-03-28T18:37:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:26 smithi045 ceph-mon[31773]: pgmap v1560: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:37:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:26 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 2054.278686 secs 2024-03-28T18:37:27.226 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2699 sec 2024-03-28T18:37:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1345.703077 secs 2024-03-28T18:37:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1345.703077 secs 2024-03-28T18:37:27.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1345.703077 secs 2024-03-28T18:37:28.226 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2700 sec 2024-03-28T18:37:28.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:28 smithi071 ceph-mon[36597]: pgmap v1561: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:37:28.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:28 smithi151 ceph-mon[36481]: pgmap v1561: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:37:28.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:28 smithi045 ceph-mon[31773]: pgmap v1561: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:37:29.227 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2701 sec 2024-03-28T18:37:30.227 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2702 sec 2024-03-28T18:37:30.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:30 smithi045 ceph-mon[31773]: pgmap v1562: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:37:30.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:30 smithi071 ceph-mon[36597]: pgmap v1562: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:37:30.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:30 smithi151 ceph-mon[36481]: pgmap v1562: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:37:31.227 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2703 sec 2024-03-28T18:37:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:31 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 2059.278747 secs 2024-03-28T18:37:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:31 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 2059.278747 secs 2024-03-28T18:37:31.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:31 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 2059.278747 secs 2024-03-28T18:37:32.227 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2704 sec 2024-03-28T18:37:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:32 smithi071 ceph-mon[36597]: pgmap v1563: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:37:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1350.703162 secs 2024-03-28T18:37:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:32 smithi151 ceph-mon[36481]: pgmap v1563: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:37:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1350.703162 secs 2024-03-28T18:37:32.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:32 smithi045 ceph-mon[31773]: pgmap v1563: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:37:32.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1350.703162 secs 2024-03-28T18:37:33.227 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2705 sec 2024-03-28T18:37:34.227 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2706 sec 2024-03-28T18:37:34.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:34 smithi071 ceph-mon[36597]: pgmap v1564: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:34.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:34 smithi151 ceph-mon[36481]: pgmap v1564: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:34.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:34 smithi045 ceph-mon[31773]: pgmap v1564: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:35.227 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2707 sec 2024-03-28T18:37:36.227 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2708 sec 2024-03-28T18:37:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:36 smithi071 ceph-mon[36597]: pgmap v1565: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:36 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 2064.278762 secs 2024-03-28T18:37:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:36 smithi151 ceph-mon[36481]: pgmap v1565: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:36 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 2064.278762 secs 2024-03-28T18:37:36.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:36 smithi045 ceph-mon[31773]: pgmap v1565: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:36.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:36 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 2064.278762 secs 2024-03-28T18:37:37.227 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2709 sec 2024-03-28T18:37:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1355.703223 secs 2024-03-28T18:37:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1355.703223 secs 2024-03-28T18:37:37.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1355.703223 secs 2024-03-28T18:37:38.228 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2710 sec 2024-03-28T18:37:38.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:38 smithi071 ceph-mon[36597]: pgmap v1566: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:38.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:38 smithi151 ceph-mon[36481]: pgmap v1566: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:38.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:38 smithi045 ceph-mon[31773]: pgmap v1566: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:39.228 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2711 sec 2024-03-28T18:37:40.228 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2712 sec 2024-03-28T18:37:40.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:40 smithi071 ceph-mon[36597]: pgmap v1567: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:40.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:40 smithi151 ceph-mon[36481]: pgmap v1567: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:40.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:40 smithi045 ceph-mon[31773]: pgmap v1567: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:41.228 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2713 sec 2024-03-28T18:37:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:41 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 2069.278822 secs 2024-03-28T18:37:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:41 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 2069.278822 secs 2024-03-28T18:37:41.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:41 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 2069.278822 secs 2024-03-28T18:37:42.228 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2714 sec 2024-03-28T18:37:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:42 smithi071 ceph-mon[36597]: pgmap v1568: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1360.703312 secs 2024-03-28T18:37:42.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:42 smithi151 ceph-mon[36481]: pgmap v1568: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:42.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1360.703312 secs 2024-03-28T18:37:42.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:42 smithi045 ceph-mon[31773]: pgmap v1568: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:42.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1360.703312 secs 2024-03-28T18:37:43.228 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2715 sec 2024-03-28T18:37:44.228 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2716 sec 2024-03-28T18:37:44.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:44 smithi071 ceph-mon[36597]: pgmap v1569: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:44 smithi151 ceph-mon[36481]: pgmap v1569: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:44.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:44 smithi045 ceph-mon[31773]: pgmap v1569: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:45.229 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2717 sec 2024-03-28T18:37:46.229 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2718 sec 2024-03-28T18:37:46.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:46 smithi045 ceph-mon[31773]: pgmap v1570: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:46.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:46 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 2074.278857 secs 2024-03-28T18:37:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:46 smithi071 ceph-mon[36597]: pgmap v1570: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:46 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 2074.278857 secs 2024-03-28T18:37:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:46 smithi151 ceph-mon[36481]: pgmap v1570: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:46 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 2074.278857 secs 2024-03-28T18:37:47.229 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2719 sec 2024-03-28T18:37:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1365.703376 secs 2024-03-28T18:37:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1365.703376 secs 2024-03-28T18:37:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1365.703376 secs 2024-03-28T18:37:48.229 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2720 sec 2024-03-28T18:37:48.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:48 smithi071 ceph-mon[36597]: pgmap v1571: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:48.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:48 smithi151 ceph-mon[36481]: pgmap v1571: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:48.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:48 smithi045 ceph-mon[31773]: pgmap v1571: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:49.229 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2721 sec 2024-03-28T18:37:50.229 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2722 sec 2024-03-28T18:37:50.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:50 smithi071 ceph-mon[36597]: pgmap v1572: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:50.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:50 smithi151 ceph-mon[36481]: pgmap v1572: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:50.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:50 smithi045 ceph-mon[31773]: pgmap v1572: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:51.229 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2723 sec 2024-03-28T18:37:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:51 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 2079.278918 secs 2024-03-28T18:37:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:51 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 2079.278918 secs 2024-03-28T18:37:51.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:51 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 2079.278918 secs 2024-03-28T18:37:52.230 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2724 sec 2024-03-28T18:37:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:52 smithi071 ceph-mon[36597]: pgmap v1573: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1370.703463 secs 2024-03-28T18:37:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:52 smithi151 ceph-mon[36481]: pgmap v1573: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1370.703463 secs 2024-03-28T18:37:52.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:52 smithi045 ceph-mon[31773]: pgmap v1573: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:52.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1370.703463 secs 2024-03-28T18:37:53.230 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2725 sec 2024-03-28T18:37:54.230 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2726 sec 2024-03-28T18:37:54.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:54 smithi071 ceph-mon[36597]: pgmap v1574: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:54.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:54 smithi151 ceph-mon[36481]: pgmap v1574: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:54.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:54 smithi045 ceph-mon[31773]: pgmap v1574: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:55.230 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2727 sec 2024-03-28T18:37:56.230 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2728 sec 2024-03-28T18:37:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:56 smithi151 ceph-mon[36481]: pgmap v1575: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:56 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:37:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:56 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 2084.278983 secs 2024-03-28T18:37:56.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:56 smithi045 ceph-mon[31773]: pgmap v1575: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:56.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:56 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:37:56.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:56 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 2084.278983 secs 2024-03-28T18:37:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:56 smithi071 ceph-mon[36597]: pgmap v1575: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:37:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:56 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:37:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:56 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 2084.278983 secs 2024-03-28T18:37:57.230 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2729 sec 2024-03-28T18:37:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1375.703553 secs 2024-03-28T18:37:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:57 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:57 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:37:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:57 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:37:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1375.703553 secs 2024-03-28T18:37:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:57 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:57 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:37:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:57 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:37:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1375.703553 secs 2024-03-28T18:37:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:57 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:57 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:37:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:57 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:37:58.230 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2730 sec 2024-03-28T18:37:58.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:58 smithi045 ceph-mon[31773]: pgmap v1576: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:58.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:37:58 smithi045 ceph-mon[31773]: pgmap v1577: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:37:58.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:58 smithi071 ceph-mon[36597]: pgmap v1576: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:58.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:37:58 smithi071 ceph-mon[36597]: pgmap v1577: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:37:58.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:58 smithi151 ceph-mon[36481]: pgmap v1576: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:37:58.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:37:58 smithi151 ceph-mon[36481]: pgmap v1577: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:37:59.230 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2731 sec 2024-03-28T18:38:00.231 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2732 sec 2024-03-28T18:38:00.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:00 smithi045 ceph-mon[31773]: pgmap v1578: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:38:00.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:00 smithi071 ceph-mon[36597]: pgmap v1578: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:38:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:00 smithi151 ceph-mon[36481]: pgmap v1578: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:38:01.231 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2733 sec 2024-03-28T18:38:01.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:01 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 2089.279072 secs 2024-03-28T18:38:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:01 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 2089.279072 secs 2024-03-28T18:38:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:01 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 2089.279072 secs 2024-03-28T18:38:02.231 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2734 sec 2024-03-28T18:38:02.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:02 smithi045 ceph-mon[31773]: pgmap v1579: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:38:02.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1380.703614 secs 2024-03-28T18:38:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:02 smithi071 ceph-mon[36597]: pgmap v1579: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:38:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1380.703614 secs 2024-03-28T18:38:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:02 smithi151 ceph-mon[36481]: pgmap v1579: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:38:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1380.703614 secs 2024-03-28T18:38:03.231 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2735 sec 2024-03-28T18:38:04.231 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2736 sec 2024-03-28T18:38:04.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:04 smithi045 ceph-mon[31773]: pgmap v1580: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:38:04.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:04 smithi071 ceph-mon[36597]: pgmap v1580: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:38:04.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:04 smithi151 ceph-mon[36481]: pgmap v1580: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:38:05.231 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2737 sec 2024-03-28T18:38:06.232 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2738 sec 2024-03-28T18:38:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:06 smithi045 ceph-mon[31773]: pgmap v1581: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:38:06.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:06 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 2094.279131 secs 2024-03-28T18:38:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:06 smithi071 ceph-mon[36597]: pgmap v1581: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:38:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:06 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 2094.279131 secs 2024-03-28T18:38:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:06 smithi151 ceph-mon[36481]: pgmap v1581: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:38:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:06 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 2094.279131 secs 2024-03-28T18:38:07.232 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2739 sec 2024-03-28T18:38:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1385.703701 secs 2024-03-28T18:38:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1385.703701 secs 2024-03-28T18:38:07.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1385.703701 secs 2024-03-28T18:38:08.232 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2740 sec 2024-03-28T18:38:08.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:08 smithi045 ceph-mon[31773]: pgmap v1582: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:38:08.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:08 smithi071 ceph-mon[36597]: pgmap v1582: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:38:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:08 smithi151 ceph-mon[36481]: pgmap v1582: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:38:09.232 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2741 sec 2024-03-28T18:38:09.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:09 smithi071 ceph-mon[36597]: pgmap v1583: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:09.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:09 smithi151 ceph-mon[36481]: pgmap v1583: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:09.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:09 smithi045 ceph-mon[31773]: pgmap v1583: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:10.232 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2742 sec 2024-03-28T18:38:11.232 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2743 sec 2024-03-28T18:38:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:11 smithi071 ceph-mon[36597]: pgmap v1584: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:11 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 2099.279193 secs 2024-03-28T18:38:12.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:11 smithi151 ceph-mon[36481]: pgmap v1584: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:12.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:11 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 2099.279193 secs 2024-03-28T18:38:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:11 smithi045 ceph-mon[31773]: pgmap v1584: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:11 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 2099.279193 secs 2024-03-28T18:38:12.232 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2744 sec 2024-03-28T18:38:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1390.703785 secs 2024-03-28T18:38:13.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1390.703785 secs 2024-03-28T18:38:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1390.703785 secs 2024-03-28T18:38:13.232 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2745 sec 2024-03-28T18:38:14.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:13 smithi071 ceph-mon[36597]: pgmap v1585: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:13 smithi151 ceph-mon[36481]: pgmap v1585: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:14.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:13 smithi045 ceph-mon[31773]: pgmap v1585: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:14.232 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2746 sec 2024-03-28T18:38:15.233 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2747 sec 2024-03-28T18:38:16.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:15 smithi071 ceph-mon[36597]: pgmap v1586: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:16.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:15 smithi151 ceph-mon[36481]: pgmap v1586: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:16.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:15 smithi045 ceph-mon[31773]: pgmap v1586: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:16.232 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2748 sec 2024-03-28T18:38:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:16 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 2104.279281 secs 2024-03-28T18:38:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:16 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 2104.279281 secs 2024-03-28T18:38:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:16 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 2104.279281 secs 2024-03-28T18:38:17.233 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2749 sec 2024-03-28T18:38:18.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1395.703846 secs 2024-03-28T18:38:18.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:17 smithi071 ceph-mon[36597]: pgmap v1587: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:38:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1395.703846 secs 2024-03-28T18:38:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:17 smithi151 ceph-mon[36481]: pgmap v1587: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:38:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1395.703846 secs 2024-03-28T18:38:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:17 smithi045 ceph-mon[31773]: pgmap v1587: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:38:18.233 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2750 sec 2024-03-28T18:38:19.233 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2751 sec 2024-03-28T18:38:20.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:19 smithi071 ceph-mon[36597]: pgmap v1588: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:38:20.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:19 smithi151 ceph-mon[36481]: pgmap v1588: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:38:20.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:19 smithi045 ceph-mon[31773]: pgmap v1588: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:38:20.233 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2752 sec 2024-03-28T18:38:21.233 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2753 sec 2024-03-28T18:38:22.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:21 smithi071 ceph-mon[36597]: pgmap v1589: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:38:22.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:21 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 2109.279343 secs 2024-03-28T18:38:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:21 smithi151 ceph-mon[36481]: pgmap v1589: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:38:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:21 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 2109.279343 secs 2024-03-28T18:38:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:21 smithi045 ceph-mon[31773]: pgmap v1589: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:38:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:21 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 2109.279343 secs 2024-03-28T18:38:22.233 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2754 sec 2024-03-28T18:38:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1400.703908 secs 2024-03-28T18:38:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1400.703908 secs 2024-03-28T18:38:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1400.703908 secs 2024-03-28T18:38:23.234 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2755 sec 2024-03-28T18:38:24.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:23 smithi151 ceph-mon[36481]: pgmap v1590: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:38:24.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:23 smithi045 ceph-mon[31773]: pgmap v1590: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:38:24.234 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2756 sec 2024-03-28T18:38:24.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:23 smithi071 ceph-mon[36597]: pgmap v1590: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:38:25.234 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2757 sec 2024-03-28T18:38:26.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:25 smithi151 ceph-mon[36481]: pgmap v1591: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:38:26.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:25 smithi045 ceph-mon[31773]: pgmap v1591: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:38:26.234 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2758 sec 2024-03-28T18:38:26.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:25 smithi071 ceph-mon[36597]: pgmap v1591: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:38:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:26 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 2114.279403 secs 2024-03-28T18:38:27.234 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2759 sec 2024-03-28T18:38:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:26 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 2114.279403 secs 2024-03-28T18:38:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:26 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 2114.279403 secs 2024-03-28T18:38:28.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1405.703997 secs 2024-03-28T18:38:28.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:27 smithi045 ceph-mon[31773]: pgmap v1592: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:38:28.234 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2760 sec 2024-03-28T18:38:28.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1405.703997 secs 2024-03-28T18:38:28.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:27 smithi071 ceph-mon[36597]: pgmap v1592: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:38:28.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1405.703997 secs 2024-03-28T18:38:28.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:27 smithi151 ceph-mon[36481]: pgmap v1592: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:38:29.235 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2761 sec 2024-03-28T18:38:30.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:29 smithi045 ceph-mon[31773]: pgmap v1593: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:30.235 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2762 sec 2024-03-28T18:38:30.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:29 smithi071 ceph-mon[36597]: pgmap v1593: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:30.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:29 smithi151 ceph-mon[36481]: pgmap v1593: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:31.235 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2763 sec 2024-03-28T18:38:32.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:31 smithi045 ceph-mon[31773]: pgmap v1594: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:32.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:31 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 2119.279491 secs 2024-03-28T18:38:32.235 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2764 sec 2024-03-28T18:38:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:31 smithi071 ceph-mon[36597]: pgmap v1594: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:31 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 2119.279491 secs 2024-03-28T18:38:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:31 smithi151 ceph-mon[36481]: pgmap v1594: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:31 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 2119.279491 secs 2024-03-28T18:38:33.135 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1410.704082 secs 2024-03-28T18:38:33.235 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2765 sec 2024-03-28T18:38:33.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1410.704082 secs 2024-03-28T18:38:33.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1410.704082 secs 2024-03-28T18:38:34.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:33 smithi045 ceph-mon[31773]: pgmap v1595: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:34.235 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2766 sec 2024-03-28T18:38:34.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:33 smithi071 ceph-mon[36597]: pgmap v1595: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:34.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:33 smithi151 ceph-mon[36481]: pgmap v1595: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:35.235 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2767 sec 2024-03-28T18:38:36.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:35 smithi045 ceph-mon[31773]: pgmap v1596: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:36.235 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2768 sec 2024-03-28T18:38:36.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:35 smithi071 ceph-mon[36597]: pgmap v1596: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:36.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:35 smithi151 ceph-mon[36481]: pgmap v1596: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:37.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:36 smithi045 ceph-mon[31773]: 4 slow requests, 1 included below; oldest blocked for > 2124.279553 secs 2024-03-28T18:38:37.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:36 smithi045 ceph-mon[31773]: slow request 184.366528 seconds old, received at 2024-03-28T18:35:31.473570+0000: client_request(client.26711:1 getattr pAsLsXsFs #0x1 2024-03-28T18:35:31.469661+0000 caller_uid=0, caller_gid=0{}) currently joining batch getattr 2024-03-28T18:38:37.235 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2769 sec 2024-03-28T18:38:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:36 smithi071 ceph-mon[36597]: 4 slow requests, 1 included below; oldest blocked for > 2124.279553 secs 2024-03-28T18:38:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:36 smithi071 ceph-mon[36597]: slow request 184.366528 seconds old, received at 2024-03-28T18:35:31.473570+0000: client_request(client.26711:1 getattr pAsLsXsFs #0x1 2024-03-28T18:35:31.469661+0000 caller_uid=0, caller_gid=0{}) currently joining batch getattr 2024-03-28T18:38:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:36 smithi151 ceph-mon[36481]: 4 slow requests, 1 included below; oldest blocked for > 2124.279553 secs 2024-03-28T18:38:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:36 smithi151 ceph-mon[36481]: slow request 184.366528 seconds old, received at 2024-03-28T18:35:31.473570+0000: client_request(client.26711:1 getattr pAsLsXsFs #0x1 2024-03-28T18:35:31.469661+0000 caller_uid=0, caller_gid=0{}) currently joining batch getattr 2024-03-28T18:38:38.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1415.704166 secs 2024-03-28T18:38:38.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:37 smithi045 ceph-mon[31773]: pgmap v1597: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:38.236 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2770 sec 2024-03-28T18:38:38.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1415.704166 secs 2024-03-28T18:38:38.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:37 smithi071 ceph-mon[36597]: pgmap v1597: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:38.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1415.704166 secs 2024-03-28T18:38:38.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:37 smithi151 ceph-mon[36481]: pgmap v1597: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:39.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:38 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] up:active 2024-03-28T18:38:39.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:38 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T18:38:39.236 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2771 sec 2024-03-28T18:38:39.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:38 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] up:active 2024-03-28T18:38:39.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:38 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T18:38:39.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:38 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] up:active 2024-03-28T18:38:39.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:38 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T18:38:40.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:39 smithi045 ceph-mon[31773]: pgmap v1598: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:40.236 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2772 sec 2024-03-28T18:38:40.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:39 smithi071 ceph-mon[36597]: pgmap v1598: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:40.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:39 smithi151 ceph-mon[36481]: pgmap v1598: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:41.236 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2773 sec 2024-03-28T18:38:42.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:41 smithi045 ceph-mon[31773]: pgmap v1599: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:42.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:41 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2129.279617 secs 2024-03-28T18:38:42.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:41 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1420.704181 secs 2024-03-28T18:38:42.236 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2774 sec 2024-03-28T18:38:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:41 smithi071 ceph-mon[36597]: pgmap v1599: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:41 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2129.279617 secs 2024-03-28T18:38:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:41 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1420.704181 secs 2024-03-28T18:38:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:41 smithi151 ceph-mon[36481]: pgmap v1599: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:41 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2129.279617 secs 2024-03-28T18:38:42.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:41 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1420.704181 secs 2024-03-28T18:38:43.236 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2775 sec 2024-03-28T18:38:44.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:43 smithi045 ceph-mon[31773]: pgmap v1600: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:44.237 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2776 sec 2024-03-28T18:38:44.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:43 smithi071 ceph-mon[36597]: pgmap v1600: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:44.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:43 smithi151 ceph-mon[36481]: pgmap v1600: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:45.237 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2777 sec 2024-03-28T18:38:46.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:45 smithi045 ceph-mon[31773]: pgmap v1601: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:46.237 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2778 sec 2024-03-28T18:38:46.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:45 smithi071 ceph-mon[36597]: pgmap v1601: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:46.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:45 smithi151 ceph-mon[36481]: pgmap v1601: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:47.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:46 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2134.279704 secs 2024-03-28T18:38:47.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:46 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1425.704265 secs 2024-03-28T18:38:47.237 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2779 sec 2024-03-28T18:38:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:46 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2134.279704 secs 2024-03-28T18:38:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:46 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1425.704265 secs 2024-03-28T18:38:47.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:46 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2134.279704 secs 2024-03-28T18:38:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:46 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1425.704265 secs 2024-03-28T18:38:48.237 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2780 sec 2024-03-28T18:38:48.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:47 smithi071 ceph-mon[36597]: pgmap v1602: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:47 smithi151 ceph-mon[36481]: pgmap v1602: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:48.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:47 smithi045 ceph-mon[31773]: pgmap v1602: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:49.238 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2781 sec 2024-03-28T18:38:50.238 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2782 sec 2024-03-28T18:38:50.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:49 smithi071 ceph-mon[36597]: pgmap v1603: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:50.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:49 smithi151 ceph-mon[36481]: pgmap v1603: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:50.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:49 smithi045 ceph-mon[31773]: pgmap v1603: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:51.237 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2783 sec 2024-03-28T18:38:52.238 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2784 sec 2024-03-28T18:38:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:51 smithi071 ceph-mon[36597]: pgmap v1604: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:51 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2139.279766 secs 2024-03-28T18:38:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1430.704354 secs 2024-03-28T18:38:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:51 smithi151 ceph-mon[36481]: pgmap v1604: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:51 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2139.279766 secs 2024-03-28T18:38:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1430.704354 secs 2024-03-28T18:38:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:51 smithi045 ceph-mon[31773]: pgmap v1604: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:51 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2139.279766 secs 2024-03-28T18:38:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1430.704354 secs 2024-03-28T18:38:53.238 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2785 sec 2024-03-28T18:38:54.238 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2786 sec 2024-03-28T18:38:54.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:53 smithi071 ceph-mon[36597]: pgmap v1605: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:54.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:53 smithi151 ceph-mon[36481]: pgmap v1605: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:54.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:53 smithi045 ceph-mon[31773]: pgmap v1605: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:55.238 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2787 sec 2024-03-28T18:38:56.238 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2788 sec 2024-03-28T18:38:56.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:55 smithi071 ceph-mon[36597]: pgmap v1606: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:56.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:55 smithi151 ceph-mon[36481]: pgmap v1606: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:56.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:55 smithi045 ceph-mon[31773]: pgmap v1606: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:57.238 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2789 sec 2024-03-28T18:38:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:56 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2144.279850 secs 2024-03-28T18:38:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1435.704442 secs 2024-03-28T18:38:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:56 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2144.279850 secs 2024-03-28T18:38:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1435.704442 secs 2024-03-28T18:38:57.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:56 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2144.279850 secs 2024-03-28T18:38:57.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1435.704442 secs 2024-03-28T18:38:58.239 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2790 sec 2024-03-28T18:38:58.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:57 smithi071 ceph-mon[36597]: pgmap v1607: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:58.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:57 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:58.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:57 smithi151 ceph-mon[36481]: pgmap v1607: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:58.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:57 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:58.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:57 smithi045 ceph-mon[31773]: pgmap v1607: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:38:58.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:57 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:59.239 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2791 sec 2024-03-28T18:38:59.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:58 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:59.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:58 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:59.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:58 smithi071 ceph-mon[36597]: pgmap v1608: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:59.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:58 smithi071 ceph-mon[36597]: pgmap v1609: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:38:59.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:38:58 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:38:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:58 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:58 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:58 smithi151 ceph-mon[36481]: pgmap v1608: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:58 smithi151 ceph-mon[36481]: pgmap v1609: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:38:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:38:58 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:38:59.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:58 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:59.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:58 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:59.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:58 smithi045 ceph-mon[31773]: pgmap v1608: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:38:59.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:58 smithi045 ceph-mon[31773]: pgmap v1609: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:38:59.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:38:58 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:39:00.239 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2792 sec 2024-03-28T18:39:00.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:00 smithi071 ceph-mon[36597]: pgmap v1610: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:00 smithi151 ceph-mon[36481]: pgmap v1610: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:00.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:00 smithi045 ceph-mon[31773]: pgmap v1610: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:01.239 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2793 sec 2024-03-28T18:39:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:01 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2149.279928 secs 2024-03-28T18:39:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:01 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2149.279928 secs 2024-03-28T18:39:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:01 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2149.279928 secs 2024-03-28T18:39:02.239 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2794 sec 2024-03-28T18:39:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:02 smithi071 ceph-mon[36597]: 2 slow requests, 1 included below; oldest blocked for > 1440.704477 secs 2024-03-28T18:39:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:02 smithi071 ceph-mon[36597]: slow request 1440.704477 seconds old, received at 2024-03-28T18:15:00.947128+0000: client_request(client.26687:1 getattr pAsLsXsFs #0x1 2024-03-28T18:15:00.943510+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:39:02.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:02 smithi071 ceph-mon[36597]: pgmap v1611: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:02 smithi151 ceph-mon[36481]: 2 slow requests, 1 included below; oldest blocked for > 1440.704477 secs 2024-03-28T18:39:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:02 smithi151 ceph-mon[36481]: slow request 1440.704477 seconds old, received at 2024-03-28T18:15:00.947128+0000: client_request(client.26687:1 getattr pAsLsXsFs #0x1 2024-03-28T18:15:00.943510+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:39:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:02 smithi151 ceph-mon[36481]: pgmap v1611: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:02.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:02 smithi045 ceph-mon[31773]: 2 slow requests, 1 included below; oldest blocked for > 1440.704477 secs 2024-03-28T18:39:02.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:02 smithi045 ceph-mon[31773]: slow request 1440.704477 seconds old, received at 2024-03-28T18:15:00.947128+0000: client_request(client.26687:1 getattr pAsLsXsFs #0x1 2024-03-28T18:15:00.943510+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:39:02.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:02 smithi045 ceph-mon[31773]: pgmap v1611: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:03.239 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2795 sec 2024-03-28T18:39:04.239 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2796 sec 2024-03-28T18:39:04.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:04 smithi045 ceph-mon[31773]: pgmap v1612: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:04.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:04 smithi071 ceph-mon[36597]: pgmap v1612: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:04.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:04 smithi151 ceph-mon[36481]: pgmap v1612: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:05.239 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2797 sec 2024-03-28T18:39:06.240 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2798 sec 2024-03-28T18:39:06.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:06 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2154.280014 secs 2024-03-28T18:39:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:06 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2154.280014 secs 2024-03-28T18:39:06.578 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:06 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2154.280014 secs 2024-03-28T18:39:07.240 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2799 sec 2024-03-28T18:39:07.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:07 smithi045 ceph-mon[31773]: pgmap v1613: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:07.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1445.704541 secs 2024-03-28T18:39:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:07 smithi071 ceph-mon[36597]: pgmap v1613: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1445.704541 secs 2024-03-28T18:39:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:07 smithi151 ceph-mon[36481]: pgmap v1613: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:39:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1445.704541 secs 2024-03-28T18:39:08.240 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2800 sec 2024-03-28T18:39:08.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:08 smithi071 ceph-mon[36597]: pgmap v1614: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:39:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:08 smithi151 ceph-mon[36481]: pgmap v1614: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:39:08.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:08 smithi045 ceph-mon[31773]: pgmap v1614: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:39:09.240 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2801 sec 2024-03-28T18:39:10.240 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2802 sec 2024-03-28T18:39:10.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:10 smithi071 ceph-mon[36597]: pgmap v1615: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:10.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:10 smithi151 ceph-mon[36481]: pgmap v1615: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:10.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:10 smithi045 ceph-mon[31773]: pgmap v1615: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:11.240 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2803 sec 2024-03-28T18:39:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:11 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2159.280103 secs 2024-03-28T18:39:11.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:11 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2159.280103 secs 2024-03-28T18:39:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:11 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2159.280103 secs 2024-03-28T18:39:12.240 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2804 sec 2024-03-28T18:39:12.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1450.704603 secs 2024-03-28T18:39:12.725 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:12 smithi151 ceph-mon[36481]: pgmap v1616: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1450.704603 secs 2024-03-28T18:39:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:12 smithi071 ceph-mon[36597]: pgmap v1616: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:12.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1450.704603 secs 2024-03-28T18:39:12.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:12 smithi045 ceph-mon[31773]: pgmap v1616: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:13.241 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2805 sec 2024-03-28T18:39:14.241 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2806 sec 2024-03-28T18:39:14.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:14 smithi071 ceph-mon[36597]: pgmap v1617: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:14.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:14 smithi151 ceph-mon[36481]: pgmap v1617: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:14.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:14 smithi045 ceph-mon[31773]: pgmap v1617: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:15.241 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2807 sec 2024-03-28T18:39:16.241 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2808 sec 2024-03-28T18:39:16.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:16 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2164.280162 secs 2024-03-28T18:39:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:16 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2164.280162 secs 2024-03-28T18:39:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:16 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2164.280162 secs 2024-03-28T18:39:17.241 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2809 sec 2024-03-28T18:39:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:17 smithi045 ceph-mon[31773]: pgmap v1618: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:17.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1455.704663 secs 2024-03-28T18:39:17.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:17 smithi071 ceph-mon[36597]: pgmap v1618: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:17.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1455.704663 secs 2024-03-28T18:39:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:17 smithi151 ceph-mon[36481]: pgmap v1618: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1455.704663 secs 2024-03-28T18:39:18.241 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2810 sec 2024-03-28T18:39:18.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:18 smithi071 ceph-mon[36597]: pgmap v1619: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:18.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:18 smithi151 ceph-mon[36481]: pgmap v1619: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:18.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:18 smithi045 ceph-mon[31773]: pgmap v1619: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:19.241 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2811 sec 2024-03-28T18:39:20.242 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2812 sec 2024-03-28T18:39:20.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:20 smithi045 ceph-mon[31773]: pgmap v1620: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:20.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:20 smithi071 ceph-mon[36597]: pgmap v1620: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:20 smithi151 ceph-mon[36481]: pgmap v1620: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:21.242 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2813 sec 2024-03-28T18:39:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:21 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2169.280249 secs 2024-03-28T18:39:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:21 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2169.280249 secs 2024-03-28T18:39:21.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:21 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2169.280249 secs 2024-03-28T18:39:22.242 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2814 sec 2024-03-28T18:39:22.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1460.704725 secs 2024-03-28T18:39:22.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:22 smithi071 ceph-mon[36597]: pgmap v1621: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1460.704725 secs 2024-03-28T18:39:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:22 smithi151 ceph-mon[36481]: pgmap v1621: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:22.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1460.704725 secs 2024-03-28T18:39:22.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:22 smithi045 ceph-mon[31773]: pgmap v1621: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:23.242 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2815 sec 2024-03-28T18:39:24.242 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2816 sec 2024-03-28T18:39:24.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:24 smithi071 ceph-mon[36597]: pgmap v1622: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:24.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:24 smithi151 ceph-mon[36481]: pgmap v1622: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:24.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:24 smithi045 ceph-mon[31773]: pgmap v1622: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:25.242 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2817 sec 2024-03-28T18:39:26.242 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2818 sec 2024-03-28T18:39:26.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:26 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2174.280336 secs 2024-03-28T18:39:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:26 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2174.280336 secs 2024-03-28T18:39:26.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:26 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2174.280336 secs 2024-03-28T18:39:27.243 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2819 sec 2024-03-28T18:39:27.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:27 smithi045 ceph-mon[31773]: pgmap v1623: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:27.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1465.704787 secs 2024-03-28T18:39:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:27 smithi071 ceph-mon[36597]: pgmap v1623: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1465.704787 secs 2024-03-28T18:39:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:27 smithi151 ceph-mon[36481]: pgmap v1623: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1465.704787 secs 2024-03-28T18:39:28.243 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2820 sec 2024-03-28T18:39:28.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:28 smithi071 ceph-mon[36597]: pgmap v1624: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:28 smithi151 ceph-mon[36481]: pgmap v1624: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:28.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:28 smithi045 ceph-mon[31773]: pgmap v1624: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:29.243 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2821 sec 2024-03-28T18:39:30.243 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2822 sec 2024-03-28T18:39:30.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:30 smithi071 ceph-mon[36597]: pgmap v1625: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:30.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:30 smithi151 ceph-mon[36481]: pgmap v1625: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:30.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:30 smithi045 ceph-mon[31773]: pgmap v1625: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:31.243 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2823 sec 2024-03-28T18:39:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:31 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2179.280421 secs 2024-03-28T18:39:31.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:31 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2179.280421 secs 2024-03-28T18:39:31.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:31 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2179.280421 secs 2024-03-28T18:39:32.243 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2824 sec 2024-03-28T18:39:32.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1470.704847 secs 2024-03-28T18:39:32.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:32 smithi071 ceph-mon[36597]: pgmap v1626: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1470.704847 secs 2024-03-28T18:39:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:32 smithi151 ceph-mon[36481]: pgmap v1626: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:32.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1470.704847 secs 2024-03-28T18:39:32.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:32 smithi045 ceph-mon[31773]: pgmap v1626: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:33.243 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2825 sec 2024-03-28T18:39:34.244 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2826 sec 2024-03-28T18:39:34.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:34 smithi071 ceph-mon[36597]: pgmap v1627: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:34.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:34 smithi151 ceph-mon[36481]: pgmap v1627: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:34.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:34 smithi045 ceph-mon[31773]: pgmap v1627: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:35.244 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2827 sec 2024-03-28T18:39:36.244 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2828 sec 2024-03-28T18:39:36.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:36 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2184.280505 secs 2024-03-28T18:39:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:36 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2184.280505 secs 2024-03-28T18:39:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:36 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2184.280505 secs 2024-03-28T18:39:37.244 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2829 sec 2024-03-28T18:39:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:37 smithi045 ceph-mon[31773]: pgmap v1628: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1475.704861 secs 2024-03-28T18:39:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:37 smithi071 ceph-mon[36597]: pgmap v1628: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1475.704861 secs 2024-03-28T18:39:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:37 smithi151 ceph-mon[36481]: pgmap v1628: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1475.704861 secs 2024-03-28T18:39:38.244 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2830 sec 2024-03-28T18:39:38.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:38 smithi071 ceph-mon[36597]: pgmap v1629: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:38.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:38 smithi151 ceph-mon[36481]: pgmap v1629: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:38.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:38 smithi045 ceph-mon[31773]: pgmap v1629: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:39.244 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2831 sec 2024-03-28T18:39:40.245 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2832 sec 2024-03-28T18:39:40.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:40 smithi071 ceph-mon[36597]: pgmap v1630: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:40 smithi151 ceph-mon[36481]: pgmap v1630: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:40.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:40 smithi045 ceph-mon[31773]: pgmap v1630: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:41.245 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2833 sec 2024-03-28T18:39:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:41 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2189.280566 secs 2024-03-28T18:39:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:41 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2189.280566 secs 2024-03-28T18:39:41.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:41 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2189.280566 secs 2024-03-28T18:39:42.245 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2834 sec 2024-03-28T18:39:42.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1480.704923 secs 2024-03-28T18:39:42.725 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:42 smithi151 ceph-mon[36481]: pgmap v1631: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1480.704923 secs 2024-03-28T18:39:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:42 smithi071 ceph-mon[36597]: pgmap v1631: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:42.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1480.704923 secs 2024-03-28T18:39:42.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:42 smithi045 ceph-mon[31773]: pgmap v1631: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:43.245 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2835 sec 2024-03-28T18:39:44.245 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2836 sec 2024-03-28T18:39:44.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:44 smithi071 ceph-mon[36597]: pgmap v1632: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:44 smithi151 ceph-mon[36481]: pgmap v1632: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:44.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:44 smithi045 ceph-mon[31773]: pgmap v1632: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:45.245 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2837 sec 2024-03-28T18:39:46.245 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2838 sec 2024-03-28T18:39:46.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:46 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2194.280625 secs 2024-03-28T18:39:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:46 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2194.280625 secs 2024-03-28T18:39:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:46 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2194.280625 secs 2024-03-28T18:39:47.245 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2839 sec 2024-03-28T18:39:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:47 smithi045 ceph-mon[31773]: pgmap v1633: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1485.705010 secs 2024-03-28T18:39:47.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:47 smithi071 ceph-mon[36597]: pgmap v1633: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:47.462 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1485.705010 secs 2024-03-28T18:39:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:47 smithi151 ceph-mon[36481]: pgmap v1633: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1485.705010 secs 2024-03-28T18:39:48.246 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2840 sec 2024-03-28T18:39:48.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:48 smithi071 ceph-mon[36597]: pgmap v1634: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:48 smithi151 ceph-mon[36481]: pgmap v1634: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:48.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:48 smithi045 ceph-mon[31773]: pgmap v1634: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:49.246 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2841 sec 2024-03-28T18:39:50.246 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2842 sec 2024-03-28T18:39:50.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:50 smithi071 ceph-mon[36597]: pgmap v1635: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:50.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:50 smithi151 ceph-mon[36481]: pgmap v1635: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:50.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:50 smithi045 ceph-mon[31773]: pgmap v1635: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:51.246 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2843 sec 2024-03-28T18:39:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:51 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2199.280678 secs 2024-03-28T18:39:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:51 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2199.280678 secs 2024-03-28T18:39:51.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:51 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2199.280678 secs 2024-03-28T18:39:52.246 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2844 sec 2024-03-28T18:39:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1490.705098 secs 2024-03-28T18:39:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:52 smithi071 ceph-mon[36597]: pgmap v1636: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1490.705098 secs 2024-03-28T18:39:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:52 smithi151 ceph-mon[36481]: pgmap v1636: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1490.705098 secs 2024-03-28T18:39:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:52 smithi045 ceph-mon[31773]: pgmap v1636: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:53.246 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2845 sec 2024-03-28T18:39:54.246 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2846 sec 2024-03-28T18:39:54.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:54 smithi071 ceph-mon[36597]: pgmap v1637: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:54.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:54 smithi151 ceph-mon[36481]: pgmap v1637: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:54.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:54 smithi045 ceph-mon[31773]: pgmap v1637: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:55.247 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2847 sec 2024-03-28T18:39:56.247 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2848 sec 2024-03-28T18:39:56.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:56 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2204.280753 secs 2024-03-28T18:39:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:56 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2204.280753 secs 2024-03-28T18:39:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:56 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2204.280753 secs 2024-03-28T18:39:57.247 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2849 sec 2024-03-28T18:39:57.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:57 smithi045 ceph-mon[31773]: pgmap v1638: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:57.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1495.705204 secs 2024-03-28T18:39:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:57 smithi071 ceph-mon[36597]: pgmap v1638: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1495.705204 secs 2024-03-28T18:39:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:57 smithi151 ceph-mon[36481]: pgmap v1638: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:39:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1495.705204 secs 2024-03-28T18:39:58.247 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2850 sec 2024-03-28T18:39:58.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:58 smithi071 ceph-mon[36597]: pgmap v1639: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:58.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:58 smithi151 ceph-mon[36481]: pgmap v1639: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:58.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:58 smithi045 ceph-mon[31773]: pgmap v1639: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:39:59.247 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2851 sec 2024-03-28T18:39:59.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:39:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:39:59.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:39:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:39:59.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:39:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:40:00.247 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2852 sec 2024-03-28T18:40:00.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:00 smithi071 ceph-mon[36597]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-03-28T18:40:00.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:00 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:00.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:00 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:40:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:00 smithi071 ceph-mon[36597]: pgmap v1640: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-28T18:40:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:00 smithi071 ceph-mon[36597]: pgmap v1641: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:40:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:00 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:40:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:00 smithi151 ceph-mon[36481]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-03-28T18:40:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:00 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:00 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:40:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:00 smithi151 ceph-mon[36481]: pgmap v1640: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-28T18:40:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:00 smithi151 ceph-mon[36481]: pgmap v1641: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:40:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:00 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:40:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:00 smithi045 ceph-mon[31773]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-03-28T18:40:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:00 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:00 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:40:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:00 smithi045 ceph-mon[31773]: pgmap v1640: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-28T18:40:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:00 smithi045 ceph-mon[31773]: pgmap v1641: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T18:40:00.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:00 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:40:01.247 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2853 sec 2024-03-28T18:40:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:01 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2209.280845 secs 2024-03-28T18:40:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:01 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2209.280845 secs 2024-03-28T18:40:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:01 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2209.280845 secs 2024-03-28T18:40:02.247 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2854 sec 2024-03-28T18:40:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1500.705291 secs 2024-03-28T18:40:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:02 smithi071 ceph-mon[36597]: pgmap v1642: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1500.705291 secs 2024-03-28T18:40:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:02 smithi151 ceph-mon[36481]: pgmap v1642: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:02.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1500.705291 secs 2024-03-28T18:40:02.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:02 smithi045 ceph-mon[31773]: pgmap v1642: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:03.248 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2855 sec 2024-03-28T18:40:04.248 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2856 sec 2024-03-28T18:40:05.248 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2857 sec 2024-03-28T18:40:05.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:05 smithi045 ceph-mon[31773]: pgmap v1643: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:05.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:05 smithi071 ceph-mon[36597]: pgmap v1643: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:05.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:05 smithi151 ceph-mon[36481]: pgmap v1643: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:06.248 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2858 sec 2024-03-28T18:40:07.248 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2859 sec 2024-03-28T18:40:07.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:07 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2214.280914 secs 2024-03-28T18:40:07.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:07 smithi045 ceph-mon[31773]: pgmap v1644: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:07.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1505.705376 secs 2024-03-28T18:40:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:07 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2214.280914 secs 2024-03-28T18:40:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:07 smithi071 ceph-mon[36597]: pgmap v1644: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1505.705376 secs 2024-03-28T18:40:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:07 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2214.280914 secs 2024-03-28T18:40:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:07 smithi151 ceph-mon[36481]: pgmap v1644: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1505.705376 secs 2024-03-28T18:40:08.248 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2860 sec 2024-03-28T18:40:09.249 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2861 sec 2024-03-28T18:40:09.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:09 smithi045 ceph-mon[31773]: pgmap v1645: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:09.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:09 smithi071 ceph-mon[36597]: pgmap v1645: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:09.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:09 smithi151 ceph-mon[36481]: pgmap v1645: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:40:10.249 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2862 sec 2024-03-28T18:40:11.249 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2863 sec 2024-03-28T18:40:11.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:11 smithi045 ceph-mon[31773]: pgmap v1646: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:40:11.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:11 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2219.281003 secs 2024-03-28T18:40:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:11 smithi071 ceph-mon[36597]: pgmap v1646: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:40:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:11 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2219.281003 secs 2024-03-28T18:40:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:11 smithi151 ceph-mon[36481]: pgmap v1646: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:40:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:11 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2219.281003 secs 2024-03-28T18:40:12.249 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2864 sec 2024-03-28T18:40:12.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1510.705479 secs 2024-03-28T18:40:12.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1510.705479 secs 2024-03-28T18:40:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1510.705479 secs 2024-03-28T18:40:13.249 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2865 sec 2024-03-28T18:40:13.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:13 smithi045 ceph-mon[31773]: pgmap v1647: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:13.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:13 smithi071 ceph-mon[36597]: pgmap v1647: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:13.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:13 smithi151 ceph-mon[36481]: pgmap v1647: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:14.249 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2866 sec 2024-03-28T18:40:15.250 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2867 sec 2024-03-28T18:40:15.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:15 smithi045 ceph-mon[31773]: pgmap v1648: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:15.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:15 smithi071 ceph-mon[36597]: pgmap v1648: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:15 smithi151 ceph-mon[36481]: pgmap v1648: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:16.250 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2868 sec 2024-03-28T18:40:16.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:16 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2224.281078 secs 2024-03-28T18:40:16.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:16 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2224.281078 secs 2024-03-28T18:40:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:16 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2224.281078 secs 2024-03-28T18:40:17.250 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2869 sec 2024-03-28T18:40:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:17 smithi045 ceph-mon[31773]: pgmap v1649: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:17.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1515.705568 secs 2024-03-28T18:40:17.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:17 smithi071 ceph-mon[36597]: pgmap v1649: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:17.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1515.705568 secs 2024-03-28T18:40:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:17 smithi151 ceph-mon[36481]: pgmap v1649: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1515.705568 secs 2024-03-28T18:40:18.250 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2870 sec 2024-03-28T18:40:19.250 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2871 sec 2024-03-28T18:40:19.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:19 smithi045 ceph-mon[31773]: pgmap v1650: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:19.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:19 smithi071 ceph-mon[36597]: pgmap v1650: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:19.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:19 smithi151 ceph-mon[36481]: pgmap v1650: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:20.250 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2872 sec 2024-03-28T18:40:21.250 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2873 sec 2024-03-28T18:40:21.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:21 smithi045 ceph-mon[31773]: pgmap v1651: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:21 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2229.281139 secs 2024-03-28T18:40:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:21 smithi071 ceph-mon[36597]: pgmap v1651: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:21 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2229.281139 secs 2024-03-28T18:40:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:21 smithi151 ceph-mon[36481]: pgmap v1651: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:21 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2229.281139 secs 2024-03-28T18:40:22.250 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2874 sec 2024-03-28T18:40:22.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1520.705631 secs 2024-03-28T18:40:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1520.705631 secs 2024-03-28T18:40:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1520.705631 secs 2024-03-28T18:40:23.250 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2875 sec 2024-03-28T18:40:23.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:23 smithi045 ceph-mon[31773]: pgmap v1652: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:23.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:23 smithi071 ceph-mon[36597]: pgmap v1652: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:23.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:23 smithi151 ceph-mon[36481]: pgmap v1652: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:24.251 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2876 sec 2024-03-28T18:40:25.251 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2877 sec 2024-03-28T18:40:25.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:25 smithi045 ceph-mon[31773]: pgmap v1653: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:25.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:25 smithi071 ceph-mon[36597]: pgmap v1653: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:25.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:25 smithi151 ceph-mon[36481]: pgmap v1653: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:26.251 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2878 sec 2024-03-28T18:40:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:26 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2234.281156 secs 2024-03-28T18:40:26.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:26 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2234.281156 secs 2024-03-28T18:40:26.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:26 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2234.281156 secs 2024-03-28T18:40:27.251 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2879 sec 2024-03-28T18:40:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:27 smithi071 ceph-mon[36597]: pgmap v1654: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1525.705689 secs 2024-03-28T18:40:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:27 smithi151 ceph-mon[36481]: pgmap v1654: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1525.705689 secs 2024-03-28T18:40:27.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:27 smithi045 ceph-mon[31773]: pgmap v1654: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1525.705689 secs 2024-03-28T18:40:28.251 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2880 sec 2024-03-28T18:40:29.252 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2881 sec 2024-03-28T18:40:29.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:29 smithi071 ceph-mon[36597]: pgmap v1655: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:29.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:29 smithi151 ceph-mon[36481]: pgmap v1655: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:29.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:29 smithi045 ceph-mon[31773]: pgmap v1655: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:30.251 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2882 sec 2024-03-28T18:40:31.252 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2883 sec 2024-03-28T18:40:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:31 smithi071 ceph-mon[36597]: pgmap v1656: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:31 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2239.281243 secs 2024-03-28T18:40:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:31 smithi151 ceph-mon[36481]: pgmap v1656: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:31 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2239.281243 secs 2024-03-28T18:40:31.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:31 smithi045 ceph-mon[31773]: pgmap v1656: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:31.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:31 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2239.281243 secs 2024-03-28T18:40:32.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:40:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:40:31.679+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:40:32.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:40:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:40:31.679+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:40:32.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:40:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:40:31.679+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:40:32.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:40:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:40:31.679+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:40:32.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:40:31 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:40:31.679+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:40:32.252 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2884 sec 2024-03-28T18:40:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1530.705778 secs 2024-03-28T18:40:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1530.705778 secs 2024-03-28T18:40:32.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1530.705778 secs 2024-03-28T18:40:33.252 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2885 sec 2024-03-28T18:40:33.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:33 smithi071 ceph-mon[36597]: pgmap v1657: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:33 smithi151 ceph-mon[36481]: pgmap v1657: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:33.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:33 smithi045 ceph-mon[31773]: pgmap v1657: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:34.252 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2886 sec 2024-03-28T18:40:35.252 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2887 sec 2024-03-28T18:40:35.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:35 smithi071 ceph-mon[36597]: pgmap v1658: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:35.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:35 smithi151 ceph-mon[36481]: pgmap v1658: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:35.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:35 smithi045 ceph-mon[31773]: pgmap v1658: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:36.252 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2888 sec 2024-03-28T18:40:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:36 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2244.281306 secs 2024-03-28T18:40:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:36 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2244.281306 secs 2024-03-28T18:40:36.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:36 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2244.281306 secs 2024-03-28T18:40:37.252 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2889 sec 2024-03-28T18:40:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:37 smithi071 ceph-mon[36597]: pgmap v1659: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:40:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1535.705874 secs 2024-03-28T18:40:37.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:37 smithi045 ceph-mon[31773]: pgmap v1659: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:40:37.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1535.705874 secs 2024-03-28T18:40:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:37 smithi151 ceph-mon[36481]: pgmap v1659: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:40:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1535.705874 secs 2024-03-28T18:40:38.253 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2890 sec 2024-03-28T18:40:39.253 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2891 sec 2024-03-28T18:40:39.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:39 smithi071 ceph-mon[36597]: pgmap v1660: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:40:39.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:39 smithi151 ceph-mon[36481]: pgmap v1660: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:40:39.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:39 smithi045 ceph-mon[31773]: pgmap v1660: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:40:40.253 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2892 sec 2024-03-28T18:40:41.254 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2893 sec 2024-03-28T18:40:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:41 smithi071 ceph-mon[36597]: pgmap v1661: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:40:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:41 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2249.281369 secs 2024-03-28T18:40:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:41 smithi151 ceph-mon[36481]: pgmap v1661: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:40:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:41 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2249.281369 secs 2024-03-28T18:40:41.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:41 smithi045 ceph-mon[31773]: pgmap v1661: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:40:41.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:41 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2249.281369 secs 2024-03-28T18:40:42.253 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2894 sec 2024-03-28T18:40:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1540.705960 secs 2024-03-28T18:40:42.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1540.705960 secs 2024-03-28T18:40:42.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1540.705960 secs 2024-03-28T18:40:43.253 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2895 sec 2024-03-28T18:40:43.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:43 smithi071 ceph-mon[36597]: pgmap v1662: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:40:43.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:43 smithi151 ceph-mon[36481]: pgmap v1662: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:40:43.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:43 smithi045 ceph-mon[31773]: pgmap v1662: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:40:44.254 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2896 sec 2024-03-28T18:40:45.254 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2897 sec 2024-03-28T18:40:45.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:45 smithi071 ceph-mon[36597]: pgmap v1663: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:40:45.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:45 smithi151 ceph-mon[36481]: pgmap v1663: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:40:45.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:45 smithi045 ceph-mon[31773]: pgmap v1663: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T18:40:46.254 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2898 sec 2024-03-28T18:40:46.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:46 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2254.281458 secs 2024-03-28T18:40:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:46 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2254.281458 secs 2024-03-28T18:40:46.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:46 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2254.281458 secs 2024-03-28T18:40:47.254 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2899 sec 2024-03-28T18:40:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:47 smithi071 ceph-mon[36597]: pgmap v1664: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:40:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1545.706051 secs 2024-03-28T18:40:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:47 smithi151 ceph-mon[36481]: pgmap v1664: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:40:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1545.706051 secs 2024-03-28T18:40:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:47 smithi045 ceph-mon[31773]: pgmap v1664: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T18:40:47.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1545.706051 secs 2024-03-28T18:40:48.254 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2900 sec 2024-03-28T18:40:49.254 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2901 sec 2024-03-28T18:40:49.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:48 smithi071 ceph-mon[36597]: pgmap v1665: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:49.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:48 smithi151 ceph-mon[36481]: pgmap v1665: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:49.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:48 smithi045 ceph-mon[31773]: pgmap v1665: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:50.254 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2902 sec 2024-03-28T18:40:51.255 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2903 sec 2024-03-28T18:40:51.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:51 smithi045 ceph-mon[31773]: pgmap v1666: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:51.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:51 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2259.281521 secs 2024-03-28T18:40:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:51 smithi071 ceph-mon[36597]: pgmap v1666: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:51 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2259.281521 secs 2024-03-28T18:40:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:51 smithi151 ceph-mon[36481]: pgmap v1666: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:51 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2259.281521 secs 2024-03-28T18:40:52.255 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2904 sec 2024-03-28T18:40:52.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1550.706134 secs 2024-03-28T18:40:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1550.706134 secs 2024-03-28T18:40:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1550.706134 secs 2024-03-28T18:40:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:53 smithi045 ceph-mon[31773]: pgmap v1667: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:53.255 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2905 sec 2024-03-28T18:40:53.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:53 smithi071 ceph-mon[36597]: pgmap v1667: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:53.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:53 smithi151 ceph-mon[36481]: pgmap v1667: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:54.255 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2906 sec 2024-03-28T18:40:55.255 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2907 sec 2024-03-28T18:40:55.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:55 smithi045 ceph-mon[31773]: pgmap v1668: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:55 smithi071 ceph-mon[36597]: pgmap v1668: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:55.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:55 smithi151 ceph-mon[36481]: pgmap v1668: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:56.255 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2908 sec 2024-03-28T18:40:56.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:56 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2264.281582 secs 2024-03-28T18:40:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:56 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2264.281582 secs 2024-03-28T18:40:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:56 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2264.281582 secs 2024-03-28T18:40:57.255 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2909 sec 2024-03-28T18:40:57.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:57 smithi045 ceph-mon[31773]: pgmap v1669: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:57.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1555.706195 secs 2024-03-28T18:40:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:57 smithi071 ceph-mon[36597]: pgmap v1669: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1555.706195 secs 2024-03-28T18:40:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:57 smithi151 ceph-mon[36481]: pgmap v1669: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:40:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1555.706195 secs 2024-03-28T18:40:58.256 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2910 sec 2024-03-28T18:40:59.256 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2911 sec 2024-03-28T18:40:59.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:40:58 smithi071 ceph-mon[36597]: pgmap v1670: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:40:58 smithi151 ceph-mon[36481]: pgmap v1670: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:40:59.418 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:40:58 smithi045 ceph-mon[31773]: pgmap v1670: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:00.256 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2912 sec 2024-03-28T18:41:01.256 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2913 sec 2024-03-28T18:41:01.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:01 smithi045 ceph-mon[31773]: pgmap v1671: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:01.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:01 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:41:01.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:01 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2269.281645 secs 2024-03-28T18:41:01.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:01 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:01.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:01 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:01.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:01 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:01.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:01 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:01 smithi071 ceph-mon[36597]: pgmap v1671: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:01 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:41:01.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:01 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2269.281645 secs 2024-03-28T18:41:01.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:01 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:01.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:01 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:01.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:01 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:01.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:01 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:01 smithi151 ceph-mon[36481]: pgmap v1671: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:01 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:41:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:01 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2269.281645 secs 2024-03-28T18:41:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:01 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:01 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:01 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:01 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:02.256 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2914 sec 2024-03-28T18:41:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:02 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:02.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:02 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:02.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1560.706260 secs 2024-03-28T18:41:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:02 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:02 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1560.706260 secs 2024-03-28T18:41:02.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:02 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:02.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:02 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:02.585 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1560.706260 secs 2024-03-28T18:41:03.256 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2915 sec 2024-03-28T18:41:03.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:03 smithi071 ceph-mon[36597]: pgmap v1672: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:03.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:03 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:03.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:03 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:03.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:03 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:03.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:03 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:03 smithi151 ceph-mon[36481]: pgmap v1672: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:03 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:03 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:03 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:03 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:03.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:03 smithi045 ceph-mon[31773]: pgmap v1672: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:03.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:03 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:03.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:03 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:03.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:03 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:03.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:03 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:04.256 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2916 sec 2024-03-28T18:41:04.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:04 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:04 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:04 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:04 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.576 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:04 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.576 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:04 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.576 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:04 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.576 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:04 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.576 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:04 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.576 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:04 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.576 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:04 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.577 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:04 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.577 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:04 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.577 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:04 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.577 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:04 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.577 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:04 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.671 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:04 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.671 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:04 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.671 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:04 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.671 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:04 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.671 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:04 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.671 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:04 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.672 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:04 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.672 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:04 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.672 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:04 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.672 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:04 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.672 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:04 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.672 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:04 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.673 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:04 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.673 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:04 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.673 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:04 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.673 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:04 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:04 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:04 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:04 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:04 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:04 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:04 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:04 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:04 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:04 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:04 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:04 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:04 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:04 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:04 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:04 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:04.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:04 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:05.257 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2917 sec 2024-03-28T18:41:05.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:05 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T18:41:05.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:05 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T18:41:05.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:05 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T18:41:05.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:05 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T18:41:05.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:05 smithi151 ceph-mon[36481]: pgmap v1673: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:05.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:05 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T18:41:05.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:05 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T18:41:05.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:05 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T18:41:05.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:05 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T18:41:05.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:05 smithi045 ceph-mon[31773]: pgmap v1673: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:05.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:05 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T18:41:05.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:05 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T18:41:05.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:05 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T18:41:05.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:05 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T18:41:05.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:05 smithi071 ceph-mon[36597]: pgmap v1673: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:06.256 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2918 sec 2024-03-28T18:41:07.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:06 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2274.281705 secs 2024-03-28T18:41:07.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:06 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:07.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:06 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:07.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:06 smithi045 ceph-mon[31773]: pgmap v1674: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:41:07.256 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2919 sec 2024-03-28T18:41:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:06 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2274.281705 secs 2024-03-28T18:41:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:06 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:06 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:06 smithi071 ceph-mon[36597]: pgmap v1674: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:41:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:06 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2274.281705 secs 2024-03-28T18:41:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:06 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:06 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:06 smithi151 ceph-mon[36481]: pgmap v1674: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:41:08.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1565.706319 secs 2024-03-28T18:41:08.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:07 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:07 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:07 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:07 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:07 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:07 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:07 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:07 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:07 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T18:41:08.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:07 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:41:08.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:07 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:08.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:07 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:41:08.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:07 smithi045 ceph-mon[31773]: pgmap v1675: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:41:08.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:07 smithi045 ceph-mon[31773]: pgmap v1676: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:41:08.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:07 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:08.257 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2920 sec 2024-03-28T18:41:08.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1565.706319 secs 2024-03-28T18:41:08.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:07 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:07 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:07 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:07 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:07 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:07 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:07 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:07 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:07 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T18:41:08.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:07 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:41:08.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:07 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:08.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:07 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:41:08.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:07 smithi071 ceph-mon[36597]: pgmap v1675: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:41:08.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:07 smithi071 ceph-mon[36597]: pgmap v1676: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:41:08.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:07 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:08.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1565.706319 secs 2024-03-28T18:41:08.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:07 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:07 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:07 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:07 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:07 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:07 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:07 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:07 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:41:08.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:07 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T18:41:08.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:07 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:41:08.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:07 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:08.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:07 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:41:08.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:07 smithi151 ceph-mon[36481]: pgmap v1675: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:41:08.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:07 smithi151 ceph-mon[36481]: pgmap v1676: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:41:08.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:07 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:41:09.257 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2921 sec 2024-03-28T18:41:10.257 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2922 sec 2024-03-28T18:41:10.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:10 smithi071 ceph-mon[36597]: pgmap v1677: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-28T18:41:10.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:10 smithi151 ceph-mon[36481]: pgmap v1677: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-28T18:41:10.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:10 smithi045 ceph-mon[31773]: pgmap v1677: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-28T18:41:11.257 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2923 sec 2024-03-28T18:41:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:11 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2279.281789 secs 2024-03-28T18:41:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:11 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2279.281789 secs 2024-03-28T18:41:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:11 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2279.281789 secs 2024-03-28T18:41:12.257 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2924 sec 2024-03-28T18:41:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:12 smithi071 ceph-mon[36597]: pgmap v1678: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:41:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1570.706411 secs 2024-03-28T18:41:12.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:12 smithi151 ceph-mon[36481]: pgmap v1678: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:41:12.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1570.706411 secs 2024-03-28T18:41:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:12 smithi045 ceph-mon[31773]: pgmap v1678: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:41:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1570.706411 secs 2024-03-28T18:41:13.257 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2925 sec 2024-03-28T18:41:14.258 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2926 sec 2024-03-28T18:41:14.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:14 smithi071 ceph-mon[36597]: pgmap v1679: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:41:14.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:14 smithi151 ceph-mon[36481]: pgmap v1679: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:41:14.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:14 smithi045 ceph-mon[31773]: pgmap v1679: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:41:15.258 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2927 sec 2024-03-28T18:41:16.258 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2928 sec 2024-03-28T18:41:16.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:16 smithi071 ceph-mon[36597]: pgmap v1680: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:41:16.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:16 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2284.281849 secs 2024-03-28T18:41:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:16 smithi151 ceph-mon[36481]: pgmap v1680: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:41:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:16 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2284.281849 secs 2024-03-28T18:41:16.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:16 smithi045 ceph-mon[31773]: pgmap v1680: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:41:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:16 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2284.281849 secs 2024-03-28T18:41:17.258 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2929 sec 2024-03-28T18:41:17.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1575.706501 secs 2024-03-28T18:41:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1575.706501 secs 2024-03-28T18:41:17.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1575.706501 secs 2024-03-28T18:41:18.258 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2930 sec 2024-03-28T18:41:18.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:18 smithi071 ceph-mon[36597]: pgmap v1681: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:41:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:18 smithi151 ceph-mon[36481]: pgmap v1681: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:41:18.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:18 smithi045 ceph-mon[31773]: pgmap v1681: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:41:19.258 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2931 sec 2024-03-28T18:41:20.258 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2932 sec 2024-03-28T18:41:20.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:20 smithi071 ceph-mon[36597]: pgmap v1682: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:20.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:20 smithi151 ceph-mon[36481]: pgmap v1682: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:20.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:20 smithi045 ceph-mon[31773]: pgmap v1682: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:21.258 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2933 sec 2024-03-28T18:41:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:21 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2289.281937 secs 2024-03-28T18:41:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:21 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2289.281937 secs 2024-03-28T18:41:21.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:21 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2289.281937 secs 2024-03-28T18:41:22.258 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2934 sec 2024-03-28T18:41:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:22 smithi071 ceph-mon[36597]: pgmap v1683: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1580.706590 secs 2024-03-28T18:41:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:22 smithi151 ceph-mon[36481]: pgmap v1683: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1580.706590 secs 2024-03-28T18:41:22.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:22 smithi045 ceph-mon[31773]: pgmap v1683: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:22.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1580.706590 secs 2024-03-28T18:41:23.259 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2935 sec 2024-03-28T18:41:24.259 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2936 sec 2024-03-28T18:41:24.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:24 smithi045 ceph-mon[31773]: pgmap v1684: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:24.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:24 smithi071 ceph-mon[36597]: pgmap v1684: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:24.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:24 smithi151 ceph-mon[36481]: pgmap v1684: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:25.259 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2937 sec 2024-03-28T18:41:26.259 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2938 sec 2024-03-28T18:41:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:26 smithi071 ceph-mon[36597]: pgmap v1685: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:26 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2294.282024 secs 2024-03-28T18:41:26.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:26 smithi151 ceph-mon[36481]: pgmap v1685: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:26.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:26 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2294.282024 secs 2024-03-28T18:41:26.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:26 smithi045 ceph-mon[31773]: pgmap v1685: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:26 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2294.282024 secs 2024-03-28T18:41:27.259 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2939 sec 2024-03-28T18:41:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1585.706681 secs 2024-03-28T18:41:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1585.706681 secs 2024-03-28T18:41:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1585.706681 secs 2024-03-28T18:41:28.259 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2940 sec 2024-03-28T18:41:28.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:28 smithi071 ceph-mon[36597]: pgmap v1686: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:28.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:28 smithi151 ceph-mon[36481]: pgmap v1686: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:28.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:28 smithi045 ceph-mon[31773]: pgmap v1686: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:29.259 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2941 sec 2024-03-28T18:41:30.260 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2942 sec 2024-03-28T18:41:30.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:30 smithi071 ceph-mon[36597]: pgmap v1687: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:30.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:30 smithi151 ceph-mon[36481]: pgmap v1687: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:30.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:30 smithi045 ceph-mon[31773]: pgmap v1687: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:31.260 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2943 sec 2024-03-28T18:41:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:31 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2299.282114 secs 2024-03-28T18:41:31.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:31 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2299.282114 secs 2024-03-28T18:41:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:31 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2299.282114 secs 2024-03-28T18:41:32.260 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2944 sec 2024-03-28T18:41:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:32 smithi151 ceph-mon[36481]: pgmap v1688: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1590.706769 secs 2024-03-28T18:41:32.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:32 smithi045 ceph-mon[31773]: pgmap v1688: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1590.706769 secs 2024-03-28T18:41:32.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:32 smithi071 ceph-mon[36597]: pgmap v1688: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:32.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1590.706769 secs 2024-03-28T18:41:33.260 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2945 sec 2024-03-28T18:41:34.260 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2946 sec 2024-03-28T18:41:34.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:34 smithi045 ceph-mon[31773]: pgmap v1689: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:34.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:34 smithi071 ceph-mon[36597]: pgmap v1689: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:34.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:34 smithi151 ceph-mon[36481]: pgmap v1689: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:35.260 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2947 sec 2024-03-28T18:41:36.260 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2948 sec 2024-03-28T18:41:36.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:36 smithi045 ceph-mon[31773]: pgmap v1690: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:36.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:36 smithi045 ceph-mon[31773]: 4 slow requests, 1 included below; oldest blocked for > 2304.282204 secs 2024-03-28T18:41:36.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:36 smithi045 ceph-mon[31773]: slow request 364.369179 seconds old, received at 2024-03-28T18:35:31.473570+0000: client_request(client.26711:1 getattr pAsLsXsFs #0x1 2024-03-28T18:35:31.469661+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:41:36.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:36 smithi071 ceph-mon[36597]: pgmap v1690: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:36.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:36 smithi071 ceph-mon[36597]: 4 slow requests, 1 included below; oldest blocked for > 2304.282204 secs 2024-03-28T18:41:36.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:36 smithi071 ceph-mon[36597]: slow request 364.369179 seconds old, received at 2024-03-28T18:35:31.473570+0000: client_request(client.26711:1 getattr pAsLsXsFs #0x1 2024-03-28T18:35:31.469661+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:41:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:36 smithi151 ceph-mon[36481]: pgmap v1690: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:36 smithi151 ceph-mon[36481]: 4 slow requests, 1 included below; oldest blocked for > 2304.282204 secs 2024-03-28T18:41:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:36 smithi151 ceph-mon[36481]: slow request 364.369179 seconds old, received at 2024-03-28T18:35:31.473570+0000: client_request(client.26711:1 getattr pAsLsXsFs #0x1 2024-03-28T18:35:31.469661+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:41:37.261 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2949 sec 2024-03-28T18:41:37.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1595.706853 secs 2024-03-28T18:41:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1595.706853 secs 2024-03-28T18:41:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1595.706853 secs 2024-03-28T18:41:38.261 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2950 sec 2024-03-28T18:41:38.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:38 smithi045 ceph-mon[31773]: pgmap v1691: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:38.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:38 smithi071 ceph-mon[36597]: pgmap v1691: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:38.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:38 smithi151 ceph-mon[36481]: pgmap v1691: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:39.261 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2951 sec 2024-03-28T18:41:40.261 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2952 sec 2024-03-28T18:41:40.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:40 smithi045 ceph-mon[31773]: pgmap v1692: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:40.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:40 smithi071 ceph-mon[36597]: pgmap v1692: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:40 smithi151 ceph-mon[36481]: pgmap v1692: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:41.261 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2953 sec 2024-03-28T18:41:41.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:41 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2309.282224 secs 2024-03-28T18:41:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:41 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2309.282224 secs 2024-03-28T18:41:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:41 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2309.282224 secs 2024-03-28T18:41:42.261 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2954 sec 2024-03-28T18:41:42.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:42 smithi045 ceph-mon[31773]: pgmap v1693: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:42.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1600.706909 secs 2024-03-28T18:41:42.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:42 smithi151 ceph-mon[36481]: pgmap v1693: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:42.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1600.706909 secs 2024-03-28T18:41:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:42 smithi071 ceph-mon[36597]: pgmap v1693: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1600.706909 secs 2024-03-28T18:41:43.261 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2955 sec 2024-03-28T18:41:44.262 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2956 sec 2024-03-28T18:41:44.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:44 smithi045 ceph-mon[31773]: pgmap v1694: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:44.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:44 smithi071 ceph-mon[36597]: pgmap v1694: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:44 smithi151 ceph-mon[36481]: pgmap v1694: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:45.262 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2957 sec 2024-03-28T18:41:46.262 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2958 sec 2024-03-28T18:41:46.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:46 smithi045 ceph-mon[31773]: pgmap v1695: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:46.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:46 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2314.282308 secs 2024-03-28T18:41:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:46 smithi071 ceph-mon[36597]: pgmap v1695: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:46 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2314.282308 secs 2024-03-28T18:41:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:46 smithi151 ceph-mon[36481]: pgmap v1695: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:46 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2314.282308 secs 2024-03-28T18:41:47.262 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2959 sec 2024-03-28T18:41:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1605.706925 secs 2024-03-28T18:41:47.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1605.706925 secs 2024-03-28T18:41:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1605.706925 secs 2024-03-28T18:41:48.262 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2960 sec 2024-03-28T18:41:48.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:48 smithi045 ceph-mon[31773]: pgmap v1696: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:48.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:48 smithi071 ceph-mon[36597]: pgmap v1696: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:48 smithi151 ceph-mon[36481]: pgmap v1696: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:49.262 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2961 sec 2024-03-28T18:41:50.262 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2962 sec 2024-03-28T18:41:50.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:50 smithi045 ceph-mon[31773]: pgmap v1697: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:50.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:50 smithi071 ceph-mon[36597]: pgmap v1697: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:50.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:50 smithi151 ceph-mon[36481]: pgmap v1697: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:51.262 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2963 sec 2024-03-28T18:41:51.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:51 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2319.282397 secs 2024-03-28T18:41:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:51 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2319.282397 secs 2024-03-28T18:41:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:51 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2319.282397 secs 2024-03-28T18:41:52.263 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2964 sec 2024-03-28T18:41:52.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:52 smithi045 ceph-mon[31773]: pgmap v1698: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:52.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1610.706998 secs 2024-03-28T18:41:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:52 smithi071 ceph-mon[36597]: pgmap v1698: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1610.706998 secs 2024-03-28T18:41:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:52 smithi151 ceph-mon[36481]: pgmap v1698: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1610.706998 secs 2024-03-28T18:41:53.263 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2965 sec 2024-03-28T18:41:54.263 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2966 sec 2024-03-28T18:41:54.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:54 smithi045 ceph-mon[31773]: pgmap v1699: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:54.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:54 smithi071 ceph-mon[36597]: pgmap v1699: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:54.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:54 smithi151 ceph-mon[36481]: pgmap v1699: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:55.263 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2967 sec 2024-03-28T18:41:56.264 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2968 sec 2024-03-28T18:41:56.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:56 smithi045 ceph-mon[31773]: pgmap v1700: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:56.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:56 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2324.282463 secs 2024-03-28T18:41:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:56 smithi071 ceph-mon[36597]: pgmap v1700: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:56 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2324.282463 secs 2024-03-28T18:41:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:56 smithi151 ceph-mon[36481]: pgmap v1700: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:41:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:56 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2324.282463 secs 2024-03-28T18:41:57.263 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2969 sec 2024-03-28T18:41:57.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1615.707060 secs 2024-03-28T18:41:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1615.707060 secs 2024-03-28T18:41:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1615.707060 secs 2024-03-28T18:41:58.263 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2970 sec 2024-03-28T18:41:58.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:41:58 smithi045 ceph-mon[31773]: pgmap v1701: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:58.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:41:58 smithi071 ceph-mon[36597]: pgmap v1701: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:58.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:41:58 smithi151 ceph-mon[36481]: pgmap v1701: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:41:59.264 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2971 sec 2024-03-28T18:42:00.264 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2972 sec 2024-03-28T18:42:00.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:00 smithi045 ceph-mon[31773]: pgmap v1702: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:00.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:00 smithi071 ceph-mon[36597]: pgmap v1702: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:00 smithi151 ceph-mon[36481]: pgmap v1702: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:01.264 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2973 sec 2024-03-28T18:42:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:01 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2329.282543 secs 2024-03-28T18:42:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:01 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2329.282543 secs 2024-03-28T18:42:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:01 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2329.282543 secs 2024-03-28T18:42:02.264 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2974 sec 2024-03-28T18:42:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:02 smithi071 ceph-mon[36597]: pgmap v1703: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1620.707123 secs 2024-03-28T18:42:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:02 smithi151 ceph-mon[36481]: pgmap v1703: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1620.707123 secs 2024-03-28T18:42:02.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:02 smithi045 ceph-mon[31773]: pgmap v1703: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:02.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1620.707123 secs 2024-03-28T18:42:03.264 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2975 sec 2024-03-28T18:42:03.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:03 smithi071 ceph-mon[36597]: pgmap v1704: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:03.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:03 smithi151 ceph-mon[36481]: pgmap v1704: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:03.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:03 smithi045 ceph-mon[31773]: pgmap v1704: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:04.264 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2976 sec 2024-03-28T18:42:05.264 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2977 sec 2024-03-28T18:42:06.265 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2978 sec 2024-03-28T18:42:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:06 smithi071 ceph-mon[36597]: pgmap v1705: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:06 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2334.282586 secs 2024-03-28T18:42:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:06 smithi151 ceph-mon[36481]: pgmap v1705: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:06 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2334.282586 secs 2024-03-28T18:42:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:06 smithi045 ceph-mon[31773]: pgmap v1705: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:06.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:06 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2334.282586 secs 2024-03-28T18:42:07.265 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2979 sec 2024-03-28T18:42:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1625.707222 secs 2024-03-28T18:42:07.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1625.707222 secs 2024-03-28T18:42:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1625.707222 secs 2024-03-28T18:42:08.265 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2980 sec 2024-03-28T18:42:08.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:08 smithi071 ceph-mon[36597]: pgmap v1706: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:08.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:08 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:42:08.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:08 smithi151 ceph-mon[36481]: pgmap v1706: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:08.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:08 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:42:08.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:08 smithi045 ceph-mon[31773]: pgmap v1706: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:08.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:08 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:42:09.265 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2981 sec 2024-03-28T18:42:09.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:09 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:42:09.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:09 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:42:09.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:09 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:42:09.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:09 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:42:09.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:09 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:42:09.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:09 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:42:09.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:09 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:42:09.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:09 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:42:09.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:09 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:42:10.265 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2982 sec 2024-03-28T18:42:10.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:10 smithi071 ceph-mon[36597]: pgmap v1707: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:10.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:10 smithi071 ceph-mon[36597]: pgmap v1708: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:42:10.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:10 smithi151 ceph-mon[36481]: pgmap v1707: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:10.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:10 smithi151 ceph-mon[36481]: pgmap v1708: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:42:10.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:10 smithi045 ceph-mon[31773]: pgmap v1707: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:10.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:10 smithi045 ceph-mon[31773]: pgmap v1708: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:42:11.265 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2983 sec 2024-03-28T18:42:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:11 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2339.282626 secs 2024-03-28T18:42:11.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:11 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2339.282626 secs 2024-03-28T18:42:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:11 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2339.282626 secs 2024-03-28T18:42:12.265 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2984 sec 2024-03-28T18:42:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:12 smithi071 ceph-mon[36597]: pgmap v1709: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:42:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1630.707309 secs 2024-03-28T18:42:12.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:12 smithi151 ceph-mon[36481]: pgmap v1709: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:42:12.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1630.707309 secs 2024-03-28T18:42:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:12 smithi045 ceph-mon[31773]: pgmap v1709: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:42:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1630.707309 secs 2024-03-28T18:42:13.266 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2985 sec 2024-03-28T18:42:14.266 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2986 sec 2024-03-28T18:42:14.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:14 smithi071 ceph-mon[36597]: pgmap v1710: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:42:14.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:14 smithi151 ceph-mon[36481]: pgmap v1710: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:42:14.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:14 smithi045 ceph-mon[31773]: pgmap v1710: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:42:15.266 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2987 sec 2024-03-28T18:42:16.266 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2988 sec 2024-03-28T18:42:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:16 smithi071 ceph-mon[36597]: pgmap v1711: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:42:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:16 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2344.282687 secs 2024-03-28T18:42:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:16 smithi151 ceph-mon[36481]: pgmap v1711: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:42:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:16 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2344.282687 secs 2024-03-28T18:42:16.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:16 smithi045 ceph-mon[31773]: pgmap v1711: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:42:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:16 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2344.282687 secs 2024-03-28T18:42:17.267 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2989 sec 2024-03-28T18:42:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1635.707379 secs 2024-03-28T18:42:17.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1635.707379 secs 2024-03-28T18:42:17.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1635.707379 secs 2024-03-28T18:42:18.267 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2990 sec 2024-03-28T18:42:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:18 smithi151 ceph-mon[36481]: pgmap v1712: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:42:18.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:18 smithi045 ceph-mon[31773]: pgmap v1712: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:42:18.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:18 smithi071 ceph-mon[36597]: pgmap v1712: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:42:19.267 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2991 sec 2024-03-28T18:42:20.267 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2992 sec 2024-03-28T18:42:20.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:20 smithi151 ceph-mon[36481]: pgmap v1713: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:42:20.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:20 smithi045 ceph-mon[31773]: pgmap v1713: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:42:20.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:20 smithi071 ceph-mon[36597]: pgmap v1713: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:42:21.267 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2993 sec 2024-03-28T18:42:21.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:21 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2349.282784 secs 2024-03-28T18:42:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:21 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2349.282784 secs 2024-03-28T18:42:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:21 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2349.282784 secs 2024-03-28T18:42:22.270 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2994 sec 2024-03-28T18:42:22.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:22 smithi045 ceph-mon[31773]: pgmap v1714: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:22.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1640.707416 secs 2024-03-28T18:42:22.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:22 smithi071 ceph-mon[36597]: pgmap v1714: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:22.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1640.707416 secs 2024-03-28T18:42:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:22 smithi151 ceph-mon[36481]: pgmap v1714: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1640.707416 secs 2024-03-28T18:42:23.267 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2995 sec 2024-03-28T18:42:24.268 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2996 sec 2024-03-28T18:42:24.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:24 smithi045 ceph-mon[31773]: pgmap v1715: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:24.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:24 smithi071 ceph-mon[36597]: pgmap v1715: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:24.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:24 smithi151 ceph-mon[36481]: pgmap v1715: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:25.268 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2997 sec 2024-03-28T18:42:26.268 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2998 sec 2024-03-28T18:42:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:26 smithi045 ceph-mon[31773]: pgmap v1716: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:26 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2354.282843 secs 2024-03-28T18:42:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:26 smithi071 ceph-mon[36597]: pgmap v1716: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:26 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2354.282843 secs 2024-03-28T18:42:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:26 smithi151 ceph-mon[36481]: pgmap v1716: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:26 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2354.282843 secs 2024-03-28T18:42:27.268 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 2999 sec 2024-03-28T18:42:27.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1645.707477 secs 2024-03-28T18:42:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1645.707477 secs 2024-03-28T18:42:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1645.707477 secs 2024-03-28T18:42:28.268 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3000 sec 2024-03-28T18:42:28.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:28 smithi045 ceph-mon[31773]: pgmap v1717: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:28.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:28 smithi071 ceph-mon[36597]: pgmap v1717: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:28 smithi151 ceph-mon[36481]: pgmap v1717: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:29.268 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3001 sec 2024-03-28T18:42:30.268 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3002 sec 2024-03-28T18:42:30.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:30 smithi045 ceph-mon[31773]: pgmap v1718: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:30.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:30 smithi071 ceph-mon[36597]: pgmap v1718: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:30.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:30 smithi151 ceph-mon[36481]: pgmap v1718: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:31.268 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3003 sec 2024-03-28T18:42:31.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:31 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2359.282907 secs 2024-03-28T18:42:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:31 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2359.282907 secs 2024-03-28T18:42:31.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:31 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2359.282907 secs 2024-03-28T18:42:32.269 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3004 sec 2024-03-28T18:42:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:32 smithi045 ceph-mon[31773]: pgmap v1719: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1650.707538 secs 2024-03-28T18:42:32.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:32 smithi071 ceph-mon[36597]: pgmap v1719: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:32.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1650.707538 secs 2024-03-28T18:42:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:32 smithi151 ceph-mon[36481]: pgmap v1719: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1650.707538 secs 2024-03-28T18:42:33.269 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3005 sec 2024-03-28T18:42:34.269 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3006 sec 2024-03-28T18:42:34.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:34 smithi045 ceph-mon[31773]: pgmap v1720: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:34.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:34 smithi071 ceph-mon[36597]: pgmap v1720: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:34.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:34 smithi151 ceph-mon[36481]: pgmap v1720: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:35.269 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3007 sec 2024-03-28T18:42:36.269 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3008 sec 2024-03-28T18:42:36.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:36 smithi045 ceph-mon[31773]: pgmap v1721: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:36.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:36 smithi045 ceph-mon[31773]: 4 slow requests, 1 included below; oldest blocked for > 2364.282950 secs 2024-03-28T18:42:36.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:36 smithi045 ceph-mon[31773]: slow request 724.454059 seconds old, received at 2024-03-28T18:30:31.389436+0000: client_request(client.26705:1 getattr pAsLsXsFs #0x1 2024-03-28T18:30:31.385264+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:42:36.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:36 smithi071 ceph-mon[36597]: pgmap v1721: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:36.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:36 smithi071 ceph-mon[36597]: 4 slow requests, 1 included below; oldest blocked for > 2364.282950 secs 2024-03-28T18:42:36.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:36 smithi071 ceph-mon[36597]: slow request 724.454059 seconds old, received at 2024-03-28T18:30:31.389436+0000: client_request(client.26705:1 getattr pAsLsXsFs #0x1 2024-03-28T18:30:31.385264+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:42:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:36 smithi151 ceph-mon[36481]: pgmap v1721: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:36 smithi151 ceph-mon[36481]: 4 slow requests, 1 included below; oldest blocked for > 2364.282950 secs 2024-03-28T18:42:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:36 smithi151 ceph-mon[36481]: slow request 724.454059 seconds old, received at 2024-03-28T18:30:31.389436+0000: client_request(client.26705:1 getattr pAsLsXsFs #0x1 2024-03-28T18:30:31.385264+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:42:37.269 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3009 sec 2024-03-28T18:42:37.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1655.707572 secs 2024-03-28T18:42:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1655.707572 secs 2024-03-28T18:42:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1655.707572 secs 2024-03-28T18:42:38.270 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3010 sec 2024-03-28T18:42:38.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:38 smithi045 ceph-mon[31773]: pgmap v1722: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:38.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:38 smithi071 ceph-mon[36597]: pgmap v1722: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:38.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:38 smithi151 ceph-mon[36481]: pgmap v1722: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:39.270 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3011 sec 2024-03-28T18:42:40.270 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3012 sec 2024-03-28T18:42:40.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:40 smithi045 ceph-mon[31773]: pgmap v1723: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:40.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:40 smithi071 ceph-mon[36597]: pgmap v1723: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:40 smithi151 ceph-mon[36481]: pgmap v1723: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:41.270 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3013 sec 2024-03-28T18:42:41.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:41 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2369.283043 secs 2024-03-28T18:42:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:41 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2369.283043 secs 2024-03-28T18:42:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:41 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2369.283043 secs 2024-03-28T18:42:42.270 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3014 sec 2024-03-28T18:42:42.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:42 smithi045 ceph-mon[31773]: pgmap v1724: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:42.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1660.707632 secs 2024-03-28T18:42:42.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:42 smithi151 ceph-mon[36481]: pgmap v1724: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:42.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1660.707632 secs 2024-03-28T18:42:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:42 smithi071 ceph-mon[36597]: pgmap v1724: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1660.707632 secs 2024-03-28T18:42:43.270 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3015 sec 2024-03-28T18:42:44.271 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3016 sec 2024-03-28T18:42:44.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:44 smithi045 ceph-mon[31773]: pgmap v1725: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:44.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:44 smithi071 ceph-mon[36597]: pgmap v1725: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:44 smithi151 ceph-mon[36481]: pgmap v1725: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:45.271 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3017 sec 2024-03-28T18:42:46.271 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3018 sec 2024-03-28T18:42:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:46 smithi071 ceph-mon[36597]: pgmap v1726: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:46 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2374.283105 secs 2024-03-28T18:42:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:46 smithi151 ceph-mon[36481]: pgmap v1726: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:46 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2374.283105 secs 2024-03-28T18:42:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:46 smithi045 ceph-mon[31773]: pgmap v1726: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:46.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:46 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2374.283105 secs 2024-03-28T18:42:47.271 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3019 sec 2024-03-28T18:42:47.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1665.707721 secs 2024-03-28T18:42:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1665.707721 secs 2024-03-28T18:42:47.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1665.707721 secs 2024-03-28T18:42:48.271 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3020 sec 2024-03-28T18:42:48.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:48 smithi071 ceph-mon[36597]: pgmap v1727: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:48 smithi151 ceph-mon[36481]: pgmap v1727: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:48.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:48 smithi045 ceph-mon[31773]: pgmap v1727: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:49.271 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3021 sec 2024-03-28T18:42:49.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:49 smithi071 ceph-mon[36597]: pgmap v1728: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:49.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:49 smithi151 ceph-mon[36481]: pgmap v1728: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:49.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:49 smithi045 ceph-mon[31773]: pgmap v1728: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:50.271 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3022 sec 2024-03-28T18:42:51.272 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3023 sec 2024-03-28T18:42:52.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:51 smithi045 ceph-mon[31773]: pgmap v1729: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:52.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:51 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2379.283144 secs 2024-03-28T18:42:52.272 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3024 sec 2024-03-28T18:42:52.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:51 smithi071 ceph-mon[36597]: pgmap v1729: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:52.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:51 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2379.283144 secs 2024-03-28T18:42:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:51 smithi151 ceph-mon[36481]: pgmap v1729: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:51 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2379.283144 secs 2024-03-28T18:42:53.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1670.707811 secs 2024-03-28T18:42:53.272 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3025 sec 2024-03-28T18:42:53.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1670.707811 secs 2024-03-28T18:42:53.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1670.707811 secs 2024-03-28T18:42:54.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:53 smithi045 ceph-mon[31773]: pgmap v1730: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:54.272 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3026 sec 2024-03-28T18:42:54.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:53 smithi071 ceph-mon[36597]: pgmap v1730: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:54.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:53 smithi151 ceph-mon[36481]: pgmap v1730: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:55.272 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3027 sec 2024-03-28T18:42:56.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:55 smithi045 ceph-mon[31773]: pgmap v1731: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:56.272 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3028 sec 2024-03-28T18:42:56.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:55 smithi071 ceph-mon[36597]: pgmap v1731: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:56.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:55 smithi151 ceph-mon[36481]: pgmap v1731: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:42:57.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:56 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2384.283233 secs 2024-03-28T18:42:57.272 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3029 sec 2024-03-28T18:42:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:56 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2384.283233 secs 2024-03-28T18:42:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:56 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2384.283233 secs 2024-03-28T18:42:58.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1675.707890 secs 2024-03-28T18:42:58.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:42:57 smithi045 ceph-mon[31773]: pgmap v1732: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:58.272 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3030 sec 2024-03-28T18:42:58.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1675.707890 secs 2024-03-28T18:42:58.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:42:57 smithi071 ceph-mon[36597]: pgmap v1732: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:58.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1675.707890 secs 2024-03-28T18:42:58.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:42:57 smithi151 ceph-mon[36481]: pgmap v1732: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:42:59.273 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3031 sec 2024-03-28T18:43:00.273 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3032 sec 2024-03-28T18:43:00.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:00 smithi071 ceph-mon[36597]: pgmap v1733: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:00.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:00 smithi151 ceph-mon[36481]: pgmap v1733: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:00.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:00 smithi045 ceph-mon[31773]: pgmap v1733: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:01.273 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3033 sec 2024-03-28T18:43:02.273 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3034 sec 2024-03-28T18:43:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:02 smithi071 ceph-mon[36597]: pgmap v1734: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:02.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:02 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2389.283306 secs 2024-03-28T18:43:02.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1680.707954 secs 2024-03-28T18:43:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:02 smithi151 ceph-mon[36481]: pgmap v1734: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:02 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2389.283306 secs 2024-03-28T18:43:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1680.707954 secs 2024-03-28T18:43:02.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:02 smithi045 ceph-mon[31773]: pgmap v1734: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:02.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:02 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2389.283306 secs 2024-03-28T18:43:02.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1680.707954 secs 2024-03-28T18:43:03.273 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3035 sec 2024-03-28T18:43:04.274 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3036 sec 2024-03-28T18:43:04.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:04 smithi071 ceph-mon[36597]: pgmap v1735: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:04.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:04 smithi151 ceph-mon[36481]: pgmap v1735: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:04.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:04 smithi045 ceph-mon[31773]: pgmap v1735: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:05.274 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3037 sec 2024-03-28T18:43:06.274 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3038 sec 2024-03-28T18:43:06.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:06 smithi071 ceph-mon[36597]: pgmap v1736: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:06.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:06 smithi151 ceph-mon[36481]: pgmap v1736: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:06.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:06 smithi045 ceph-mon[31773]: pgmap v1736: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:07.274 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3039 sec 2024-03-28T18:43:07.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:07 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2394.283395 secs 2024-03-28T18:43:07.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1685.708040 secs 2024-03-28T18:43:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:07 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2394.283395 secs 2024-03-28T18:43:07.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1685.708040 secs 2024-03-28T18:43:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:07 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2394.283395 secs 2024-03-28T18:43:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1685.708040 secs 2024-03-28T18:43:08.274 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3040 sec 2024-03-28T18:43:08.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:08 smithi151 ceph-mon[36481]: pgmap v1737: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:08.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:08 smithi045 ceph-mon[31773]: pgmap v1737: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:08.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:08 smithi071 ceph-mon[36597]: pgmap v1737: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:09.274 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3041 sec 2024-03-28T18:43:10.274 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3042 sec 2024-03-28T18:43:10.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:10 smithi045 ceph-mon[31773]: pgmap v1738: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:10.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:10 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:43:10.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:10 smithi071 ceph-mon[36597]: pgmap v1738: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:10.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:10 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:43:10.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:10 smithi151 ceph-mon[36481]: pgmap v1738: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:10.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:10 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:43:11.274 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3043 sec 2024-03-28T18:43:11.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:11 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:43:11.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:11 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:43:11.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:11 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:43:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:11 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:43:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:11 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:43:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:11 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:43:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:11 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:43:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:11 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:43:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:11 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:43:12.274 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3044 sec 2024-03-28T18:43:12.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:12 smithi045 ceph-mon[31773]: pgmap v1739: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:43:12.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:12 smithi045 ceph-mon[31773]: pgmap v1740: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:12.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:12 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2399.283488 secs 2024-03-28T18:43:12.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1690.708125 secs 2024-03-28T18:43:12.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:12 smithi151 ceph-mon[36481]: pgmap v1739: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:43:12.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:12 smithi151 ceph-mon[36481]: pgmap v1740: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:12.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:12 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2399.283488 secs 2024-03-28T18:43:12.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1690.708125 secs 2024-03-28T18:43:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:12 smithi071 ceph-mon[36597]: pgmap v1739: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:43:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:12 smithi071 ceph-mon[36597]: pgmap v1740: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:12 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2399.283488 secs 2024-03-28T18:43:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1690.708125 secs 2024-03-28T18:43:13.275 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3045 sec 2024-03-28T18:43:14.275 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3046 sec 2024-03-28T18:43:14.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:14 smithi045 ceph-mon[31773]: pgmap v1741: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:14.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:14 smithi071 ceph-mon[36597]: pgmap v1741: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:14.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:14 smithi151 ceph-mon[36481]: pgmap v1741: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:15.275 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3047 sec 2024-03-28T18:43:16.275 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3048 sec 2024-03-28T18:43:16.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:16 smithi045 ceph-mon[31773]: pgmap v1742: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:16 smithi071 ceph-mon[36597]: pgmap v1742: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:16 smithi151 ceph-mon[36481]: pgmap v1742: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:17.275 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3049 sec 2024-03-28T18:43:17.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:17 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2404.283503 secs 2024-03-28T18:43:17.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1695.708214 secs 2024-03-28T18:43:17.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:17 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2404.283503 secs 2024-03-28T18:43:17.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1695.708214 secs 2024-03-28T18:43:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:17 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2404.283503 secs 2024-03-28T18:43:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1695.708214 secs 2024-03-28T18:43:18.275 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3050 sec 2024-03-28T18:43:18.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:18 smithi045 ceph-mon[31773]: pgmap v1743: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:18.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:18 smithi071 ceph-mon[36597]: pgmap v1743: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:18.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:18 smithi151 ceph-mon[36481]: pgmap v1743: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:19.275 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3051 sec 2024-03-28T18:43:20.276 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3052 sec 2024-03-28T18:43:20.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:20 smithi045 ceph-mon[31773]: pgmap v1744: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:20.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:20 smithi071 ceph-mon[36597]: pgmap v1744: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:20.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:20 smithi151 ceph-mon[36481]: pgmap v1744: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:43:21.276 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3053 sec 2024-03-28T18:43:21.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:21 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2409.283596 secs 2024-03-28T18:43:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:21 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2409.283596 secs 2024-03-28T18:43:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:21 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2409.283596 secs 2024-03-28T18:43:22.276 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3054 sec 2024-03-28T18:43:22.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:22 smithi045 ceph-mon[31773]: pgmap v1745: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:43:22.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1700.708276 secs 2024-03-28T18:43:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:22 smithi071 ceph-mon[36597]: pgmap v1745: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:43:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1700.708276 secs 2024-03-28T18:43:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:22 smithi151 ceph-mon[36481]: pgmap v1745: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:43:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1700.708276 secs 2024-03-28T18:43:23.276 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3055 sec 2024-03-28T18:43:24.276 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3056 sec 2024-03-28T18:43:24.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:24 smithi045 ceph-mon[31773]: pgmap v1746: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:24.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:24 smithi071 ceph-mon[36597]: pgmap v1746: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:24.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:24 smithi151 ceph-mon[36481]: pgmap v1746: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:25.276 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3057 sec 2024-03-28T18:43:26.277 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3058 sec 2024-03-28T18:43:26.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:26 smithi045 ceph-mon[31773]: pgmap v1747: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:26.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:26 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2414.283658 secs 2024-03-28T18:43:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:26 smithi071 ceph-mon[36597]: pgmap v1747: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:26 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2414.283658 secs 2024-03-28T18:43:26.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:26 smithi151 ceph-mon[36481]: pgmap v1747: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:26.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:26 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2414.283658 secs 2024-03-28T18:43:27.277 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3059 sec 2024-03-28T18:43:27.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1705.708337 secs 2024-03-28T18:43:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1705.708337 secs 2024-03-28T18:43:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1705.708337 secs 2024-03-28T18:43:28.277 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3060 sec 2024-03-28T18:43:28.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:28 smithi045 ceph-mon[31773]: pgmap v1748: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:28.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:28 smithi071 ceph-mon[36597]: pgmap v1748: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:28.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:28 smithi151 ceph-mon[36481]: pgmap v1748: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:29.277 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3061 sec 2024-03-28T18:43:30.277 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3062 sec 2024-03-28T18:43:30.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:30 smithi045 ceph-mon[31773]: pgmap v1749: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:30.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:30 smithi071 ceph-mon[36597]: pgmap v1749: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:30.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:30 smithi151 ceph-mon[36481]: pgmap v1749: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:31.277 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3063 sec 2024-03-28T18:43:31.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:31 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2419.283722 secs 2024-03-28T18:43:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:31 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2419.283722 secs 2024-03-28T18:43:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:31 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2419.283722 secs 2024-03-28T18:43:32.277 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3064 sec 2024-03-28T18:43:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:32 smithi071 ceph-mon[36597]: pgmap v1750: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1710.708426 secs 2024-03-28T18:43:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:32 smithi151 ceph-mon[36481]: pgmap v1750: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1710.708426 secs 2024-03-28T18:43:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:32 smithi045 ceph-mon[31773]: pgmap v1750: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1710.708426 secs 2024-03-28T18:43:33.277 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3065 sec 2024-03-28T18:43:34.278 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3066 sec 2024-03-28T18:43:34.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:34 smithi071 ceph-mon[36597]: pgmap v1751: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:34.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:34 smithi151 ceph-mon[36481]: pgmap v1751: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:34.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:34 smithi045 ceph-mon[31773]: pgmap v1751: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:35.279 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3067 sec 2024-03-28T18:43:36.278 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3068 sec 2024-03-28T18:43:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:36 smithi071 ceph-mon[36597]: pgmap v1752: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:36 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2424.283782 secs 2024-03-28T18:43:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:36 smithi151 ceph-mon[36481]: pgmap v1752: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:36 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2424.283782 secs 2024-03-28T18:43:36.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:36 smithi045 ceph-mon[31773]: pgmap v1752: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:36.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:36 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2424.283782 secs 2024-03-28T18:43:37.278 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3069 sec 2024-03-28T18:43:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1715.708466 secs 2024-03-28T18:43:37.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1715.708466 secs 2024-03-28T18:43:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1715.708466 secs 2024-03-28T18:43:38.278 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3070 sec 2024-03-28T18:43:38.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:38 smithi071 ceph-mon[36597]: pgmap v1753: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:38.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:38 smithi151 ceph-mon[36481]: pgmap v1753: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:38.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:38 smithi045 ceph-mon[31773]: pgmap v1753: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:39.278 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3071 sec 2024-03-28T18:43:40.278 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3072 sec 2024-03-28T18:43:40.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:40 smithi071 ceph-mon[36597]: pgmap v1754: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:40.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:40 smithi151 ceph-mon[36481]: pgmap v1754: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:40.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:40 smithi045 ceph-mon[31773]: pgmap v1754: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:41.278 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3073 sec 2024-03-28T18:43:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:41 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2429.283834 secs 2024-03-28T18:43:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:41 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2429.283834 secs 2024-03-28T18:43:41.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:41 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2429.283834 secs 2024-03-28T18:43:42.279 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3074 sec 2024-03-28T18:43:42.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:42 smithi151 ceph-mon[36481]: pgmap v1755: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:42.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1720.708556 secs 2024-03-28T18:43:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:42 smithi071 ceph-mon[36597]: pgmap v1755: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1720.708556 secs 2024-03-28T18:43:42.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:42 smithi045 ceph-mon[31773]: pgmap v1755: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:42.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1720.708556 secs 2024-03-28T18:43:43.279 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3075 sec 2024-03-28T18:43:44.279 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3076 sec 2024-03-28T18:43:44.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:44 smithi071 ceph-mon[36597]: pgmap v1756: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:44 smithi151 ceph-mon[36481]: pgmap v1756: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:44.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:44 smithi045 ceph-mon[31773]: pgmap v1756: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:45.279 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3077 sec 2024-03-28T18:43:46.279 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3078 sec 2024-03-28T18:43:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:46 smithi071 ceph-mon[36597]: pgmap v1757: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:46 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2434.283894 secs 2024-03-28T18:43:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:46 smithi151 ceph-mon[36481]: pgmap v1757: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:46 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2434.283894 secs 2024-03-28T18:43:46.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:46 smithi045 ceph-mon[31773]: pgmap v1757: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:46.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:46 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2434.283894 secs 2024-03-28T18:43:47.279 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3079 sec 2024-03-28T18:43:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1725.708646 secs 2024-03-28T18:43:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1725.708646 secs 2024-03-28T18:43:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1725.708646 secs 2024-03-28T18:43:48.279 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3080 sec 2024-03-28T18:43:48.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:48 smithi071 ceph-mon[36597]: pgmap v1758: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:48.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:48 smithi151 ceph-mon[36481]: pgmap v1758: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:48.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:48 smithi045 ceph-mon[31773]: pgmap v1758: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:49.280 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3081 sec 2024-03-28T18:43:50.280 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3082 sec 2024-03-28T18:43:50.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:50 smithi071 ceph-mon[36597]: pgmap v1759: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:50.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:50 smithi151 ceph-mon[36481]: pgmap v1759: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:50.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:50 smithi045 ceph-mon[31773]: pgmap v1759: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:51.280 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3083 sec 2024-03-28T18:43:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:51 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2439.283955 secs 2024-03-28T18:43:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:51 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2439.283955 secs 2024-03-28T18:43:51.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:51 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2439.283955 secs 2024-03-28T18:43:52.280 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3084 sec 2024-03-28T18:43:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:52 smithi071 ceph-mon[36597]: pgmap v1760: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1730.708735 secs 2024-03-28T18:43:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:52 smithi151 ceph-mon[36481]: pgmap v1760: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1730.708735 secs 2024-03-28T18:43:52.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:52 smithi045 ceph-mon[31773]: pgmap v1760: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:52.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1730.708735 secs 2024-03-28T18:43:53.280 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3085 sec 2024-03-28T18:43:54.280 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3086 sec 2024-03-28T18:43:54.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:54 smithi071 ceph-mon[36597]: pgmap v1761: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:54.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:54 smithi151 ceph-mon[36481]: pgmap v1761: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:54.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:54 smithi045 ceph-mon[31773]: pgmap v1761: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:55.280 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3087 sec 2024-03-28T18:43:56.281 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3088 sec 2024-03-28T18:43:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:56 smithi071 ceph-mon[36597]: pgmap v1762: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:56 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2444.284017 secs 2024-03-28T18:43:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:56 smithi151 ceph-mon[36481]: pgmap v1762: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:56 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2444.284017 secs 2024-03-28T18:43:56.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:56 smithi045 ceph-mon[31773]: pgmap v1762: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:43:56.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:56 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2444.284017 secs 2024-03-28T18:43:57.281 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3089 sec 2024-03-28T18:43:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1735.708796 secs 2024-03-28T18:43:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1735.708796 secs 2024-03-28T18:43:57.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1735.708796 secs 2024-03-28T18:43:58.281 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3090 sec 2024-03-28T18:43:58.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:43:58 smithi071 ceph-mon[36597]: pgmap v1763: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:58.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:43:58 smithi151 ceph-mon[36481]: pgmap v1763: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:58.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:43:58 smithi045 ceph-mon[31773]: pgmap v1763: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:43:59.281 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3091 sec 2024-03-28T18:44:00.281 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3092 sec 2024-03-28T18:44:00.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:00 smithi071 ceph-mon[36597]: pgmap v1764: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:00.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:00 smithi151 ceph-mon[36481]: pgmap v1764: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:00.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:00 smithi045 ceph-mon[31773]: pgmap v1764: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:01.281 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3093 sec 2024-03-28T18:44:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:01 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2449.284106 secs 2024-03-28T18:44:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:01 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2449.284106 secs 2024-03-28T18:44:01.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:01 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2449.284106 secs 2024-03-28T18:44:02.281 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3094 sec 2024-03-28T18:44:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:02 smithi071 ceph-mon[36597]: pgmap v1765: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:02.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:02 smithi071 ceph-mon[36597]: 2 slow requests, 1 included below; oldest blocked for > 1740.708886 secs 2024-03-28T18:44:02.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:02 smithi071 ceph-mon[36597]: slow request 1440.521243 seconds old, received at 2024-03-28T18:20:01.134771+0000: client_request(client.16398:1 getattr pAsLsXsFs #0x1 2024-03-28T18:20:01.130880+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:44:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:02 smithi151 ceph-mon[36481]: pgmap v1765: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:02 smithi151 ceph-mon[36481]: 2 slow requests, 1 included below; oldest blocked for > 1740.708886 secs 2024-03-28T18:44:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:02 smithi151 ceph-mon[36481]: slow request 1440.521243 seconds old, received at 2024-03-28T18:20:01.134771+0000: client_request(client.16398:1 getattr pAsLsXsFs #0x1 2024-03-28T18:20:01.130880+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:44:02.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:02 smithi045 ceph-mon[31773]: pgmap v1765: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:02.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:02 smithi045 ceph-mon[31773]: 2 slow requests, 1 included below; oldest blocked for > 1740.708886 secs 2024-03-28T18:44:02.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:02 smithi045 ceph-mon[31773]: slow request 1440.521243 seconds old, received at 2024-03-28T18:20:01.134771+0000: client_request(client.16398:1 getattr pAsLsXsFs #0x1 2024-03-28T18:20:01.130880+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:44:03.281 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3095 sec 2024-03-28T18:44:04.282 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3096 sec 2024-03-28T18:44:04.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:04 smithi071 ceph-mon[36597]: pgmap v1766: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:04.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:04 smithi151 ceph-mon[36481]: pgmap v1766: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:04.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:04 smithi045 ceph-mon[31773]: pgmap v1766: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:05.282 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3097 sec 2024-03-28T18:44:06.282 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3098 sec 2024-03-28T18:44:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:06 smithi071 ceph-mon[36597]: pgmap v1767: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:06.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:06 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2454.284167 secs 2024-03-28T18:44:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:06 smithi151 ceph-mon[36481]: pgmap v1767: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:06 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2454.284167 secs 2024-03-28T18:44:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:06 smithi045 ceph-mon[31773]: pgmap v1767: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:06.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:06 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2454.284167 secs 2024-03-28T18:44:07.282 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3099 sec 2024-03-28T18:44:07.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1745.708974 secs 2024-03-28T18:44:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1745.708974 secs 2024-03-28T18:44:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1745.708974 secs 2024-03-28T18:44:08.282 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3100 sec 2024-03-28T18:44:08.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:08 smithi151 ceph-mon[36481]: pgmap v1768: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:08.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:08 smithi045 ceph-mon[31773]: pgmap v1768: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:08.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:08 smithi071 ceph-mon[36597]: pgmap v1768: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:09.282 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3101 sec 2024-03-28T18:44:10.282 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3102 sec 2024-03-28T18:44:10.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:10 smithi045 ceph-mon[31773]: pgmap v1769: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:10.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:10 smithi071 ceph-mon[36597]: pgmap v1769: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:10.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:10 smithi151 ceph-mon[36481]: pgmap v1769: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:11.282 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3103 sec 2024-03-28T18:44:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:11 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:44:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:11 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2459.284249 secs 2024-03-28T18:44:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:11 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:44:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:11 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2459.284249 secs 2024-03-28T18:44:11.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:11 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:44:11.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:11 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2459.284249 secs 2024-03-28T18:44:12.282 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3104 sec 2024-03-28T18:44:12.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:12 smithi045 ceph-mon[31773]: pgmap v1770: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:12.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1750.709059 secs 2024-03-28T18:44:12.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:12 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:44:12.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:12 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:44:12.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:12 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:44:12.725 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:12 smithi151 ceph-mon[36481]: pgmap v1770: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:12.725 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1750.709059 secs 2024-03-28T18:44:12.725 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:12 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:44:12.725 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:12 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:44:12.725 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:12 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:44:12.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:12 smithi071 ceph-mon[36597]: pgmap v1770: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:12.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1750.709059 secs 2024-03-28T18:44:12.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:12 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:44:12.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:12 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:44:12.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:12 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:44:13.282 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3105 sec 2024-03-28T18:44:13.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:13 smithi045 ceph-mon[31773]: pgmap v1771: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:13.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:13 smithi045 ceph-mon[31773]: pgmap v1772: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:13.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:13 smithi071 ceph-mon[36597]: pgmap v1771: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:13.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:13 smithi071 ceph-mon[36597]: pgmap v1772: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:13.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:13 smithi151 ceph-mon[36481]: pgmap v1771: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:13.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:13 smithi151 ceph-mon[36481]: pgmap v1772: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:14.283 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3106 sec 2024-03-28T18:44:15.283 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3107 sec 2024-03-28T18:44:15.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:15 smithi045 ceph-mon[31773]: pgmap v1773: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:15.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:15 smithi071 ceph-mon[36597]: pgmap v1773: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:15.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:15 smithi151 ceph-mon[36481]: pgmap v1773: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:16.283 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3108 sec 2024-03-28T18:44:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:16 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2464.284310 secs 2024-03-28T18:44:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:16 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2464.284310 secs 2024-03-28T18:44:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:16 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2464.284310 secs 2024-03-28T18:44:17.283 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3109 sec 2024-03-28T18:44:17.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:17 smithi045 ceph-mon[31773]: pgmap v1774: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:17.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1755.709120 secs 2024-03-28T18:44:17.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:17 smithi071 ceph-mon[36597]: pgmap v1774: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:17.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1755.709120 secs 2024-03-28T18:44:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:17 smithi151 ceph-mon[36481]: pgmap v1774: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1755.709120 secs 2024-03-28T18:44:18.283 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3110 sec 2024-03-28T18:44:19.283 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3111 sec 2024-03-28T18:44:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:19 smithi045 ceph-mon[31773]: pgmap v1775: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:19.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:19 smithi071 ceph-mon[36597]: pgmap v1775: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:19.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:19 smithi151 ceph-mon[36481]: pgmap v1775: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:20.283 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3112 sec 2024-03-28T18:44:21.284 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3113 sec 2024-03-28T18:44:21.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:21 smithi045 ceph-mon[31773]: pgmap v1776: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:21.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:21 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2469.284398 secs 2024-03-28T18:44:21.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:21 smithi071 ceph-mon[36597]: pgmap v1776: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:21.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:21 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2469.284398 secs 2024-03-28T18:44:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:21 smithi151 ceph-mon[36481]: pgmap v1776: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:44:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:21 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2469.284398 secs 2024-03-28T18:44:22.284 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3114 sec 2024-03-28T18:44:22.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1760.709209 secs 2024-03-28T18:44:22.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1760.709209 secs 2024-03-28T18:44:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1760.709209 secs 2024-03-28T18:44:23.284 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3115 sec 2024-03-28T18:44:23.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:23 smithi045 ceph-mon[31773]: pgmap v1777: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:44:23.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:23 smithi071 ceph-mon[36597]: pgmap v1777: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:44:23.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:23 smithi151 ceph-mon[36481]: pgmap v1777: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:44:24.284 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3116 sec 2024-03-28T18:44:25.284 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3117 sec 2024-03-28T18:44:25.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:25 smithi045 ceph-mon[31773]: pgmap v1778: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:25.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:25 smithi071 ceph-mon[36597]: pgmap v1778: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:25.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:25 smithi151 ceph-mon[36481]: pgmap v1778: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:26.284 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3118 sec 2024-03-28T18:44:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:26 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2474.284488 secs 2024-03-28T18:44:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:26 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2474.284488 secs 2024-03-28T18:44:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:26 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2474.284488 secs 2024-03-28T18:44:27.285 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3119 sec 2024-03-28T18:44:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:27 smithi045 ceph-mon[31773]: pgmap v1779: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1765.709297 secs 2024-03-28T18:44:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:27 smithi071 ceph-mon[36597]: pgmap v1779: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1765.709297 secs 2024-03-28T18:44:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:27 smithi151 ceph-mon[36481]: pgmap v1779: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1765.709297 secs 2024-03-28T18:44:28.285 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3120 sec 2024-03-28T18:44:29.285 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3121 sec 2024-03-28T18:44:29.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:29 smithi045 ceph-mon[31773]: pgmap v1780: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:29.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:29 smithi071 ceph-mon[36597]: pgmap v1780: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:29.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:29 smithi151 ceph-mon[36481]: pgmap v1780: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:30.285 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3122 sec 2024-03-28T18:44:31.285 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3123 sec 2024-03-28T18:44:31.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:31 smithi045 ceph-mon[31773]: pgmap v1781: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:31.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:31 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2479.284549 secs 2024-03-28T18:44:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:31 smithi071 ceph-mon[36597]: pgmap v1781: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:31 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2479.284549 secs 2024-03-28T18:44:31.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:31 smithi151 ceph-mon[36481]: pgmap v1781: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:31.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:31 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2479.284549 secs 2024-03-28T18:44:32.285 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3124 sec 2024-03-28T18:44:32.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1770.709375 secs 2024-03-28T18:44:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1770.709375 secs 2024-03-28T18:44:32.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1770.709375 secs 2024-03-28T18:44:33.285 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3125 sec 2024-03-28T18:44:33.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:33 smithi071 ceph-mon[36597]: pgmap v1782: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:33.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:33 smithi151 ceph-mon[36481]: pgmap v1782: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:33.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:33 smithi045 ceph-mon[31773]: pgmap v1782: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:34.285 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3126 sec 2024-03-28T18:44:35.285 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3127 sec 2024-03-28T18:44:35.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:35 smithi071 ceph-mon[36597]: pgmap v1783: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:35.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:35 smithi151 ceph-mon[36481]: pgmap v1783: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:35.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:35 smithi045 ceph-mon[31773]: pgmap v1783: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:36.286 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3128 sec 2024-03-28T18:44:36.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:36 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2484.284562 secs 2024-03-28T18:44:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:36 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2484.284562 secs 2024-03-28T18:44:36.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:36 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2484.284562 secs 2024-03-28T18:44:37.286 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3129 sec 2024-03-28T18:44:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:37 smithi071 ceph-mon[36597]: pgmap v1784: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1775.709464 secs 2024-03-28T18:44:37.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:37 smithi045 ceph-mon[31773]: pgmap v1784: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:37.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1775.709464 secs 2024-03-28T18:44:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:37 smithi151 ceph-mon[36481]: pgmap v1784: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1775.709464 secs 2024-03-28T18:44:38.286 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3130 sec 2024-03-28T18:44:39.286 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3131 sec 2024-03-28T18:44:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:39 smithi071 ceph-mon[36597]: pgmap v1785: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:39 smithi151 ceph-mon[36481]: pgmap v1785: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:39.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:39 smithi045 ceph-mon[31773]: pgmap v1785: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:40.286 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3132 sec 2024-03-28T18:44:41.286 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3133 sec 2024-03-28T18:44:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:41 smithi071 ceph-mon[36597]: pgmap v1786: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:41 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2489.284623 secs 2024-03-28T18:44:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:41 smithi151 ceph-mon[36481]: pgmap v1786: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:41 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2489.284623 secs 2024-03-28T18:44:41.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:41 smithi045 ceph-mon[31773]: pgmap v1786: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:41.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:41 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2489.284623 secs 2024-03-28T18:44:42.286 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3134 sec 2024-03-28T18:44:42.725 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1780.709552 secs 2024-03-28T18:44:42.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1780.709552 secs 2024-03-28T18:44:42.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1780.709552 secs 2024-03-28T18:44:43.287 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3135 sec 2024-03-28T18:44:43.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:43 smithi071 ceph-mon[36597]: pgmap v1787: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:43.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:43 smithi151 ceph-mon[36481]: pgmap v1787: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:43.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:43 smithi045 ceph-mon[31773]: pgmap v1787: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:44.287 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3136 sec 2024-03-28T18:44:44.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:44 smithi071 ceph-mon[36597]: pgmap v1788: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:44 smithi151 ceph-mon[36481]: pgmap v1788: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:44.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:44 smithi045 ceph-mon[31773]: pgmap v1788: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:45.287 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3137 sec 2024-03-28T18:44:46.287 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3138 sec 2024-03-28T18:44:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:46 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2494.284683 secs 2024-03-28T18:44:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:46 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2494.284683 secs 2024-03-28T18:44:46.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:46 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2494.284683 secs 2024-03-28T18:44:47.287 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3139 sec 2024-03-28T18:44:47.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:47 smithi071 ceph-mon[36597]: pgmap v1789: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:47.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1785.709637 secs 2024-03-28T18:44:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:47 smithi151 ceph-mon[36481]: pgmap v1789: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1785.709637 secs 2024-03-28T18:44:47.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:47 smithi045 ceph-mon[31773]: pgmap v1789: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:47.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1785.709637 secs 2024-03-28T18:44:48.287 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3140 sec 2024-03-28T18:44:49.287 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3141 sec 2024-03-28T18:44:49.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:49 smithi071 ceph-mon[36597]: pgmap v1790: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:49 smithi151 ceph-mon[36481]: pgmap v1790: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:49.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:49 smithi045 ceph-mon[31773]: pgmap v1790: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:50.287 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3142 sec 2024-03-28T18:44:51.288 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3143 sec 2024-03-28T18:44:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:51 smithi071 ceph-mon[36597]: pgmap v1791: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:51 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2499.284788 secs 2024-03-28T18:44:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:51 smithi151 ceph-mon[36481]: pgmap v1791: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:51 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2499.284788 secs 2024-03-28T18:44:51.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:51 smithi045 ceph-mon[31773]: pgmap v1791: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:51.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:51 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2499.284788 secs 2024-03-28T18:44:52.288 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3144 sec 2024-03-28T18:44:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1790.709699 secs 2024-03-28T18:44:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1790.709699 secs 2024-03-28T18:44:52.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1790.709699 secs 2024-03-28T18:44:53.289 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3145 sec 2024-03-28T18:44:53.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:53 smithi071 ceph-mon[36597]: pgmap v1792: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:53.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:53 smithi151 ceph-mon[36481]: pgmap v1792: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:53.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:53 smithi045 ceph-mon[31773]: pgmap v1792: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:54.288 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3146 sec 2024-03-28T18:44:55.288 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3147 sec 2024-03-28T18:44:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:55 smithi071 ceph-mon[36597]: pgmap v1793: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:55.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:55 smithi151 ceph-mon[36481]: pgmap v1793: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:55.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:55 smithi045 ceph-mon[31773]: pgmap v1793: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:56.288 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3148 sec 2024-03-28T18:44:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:56 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2504.284873 secs 2024-03-28T18:44:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:56 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2504.284873 secs 2024-03-28T18:44:56.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:56 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2504.284873 secs 2024-03-28T18:44:57.288 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3149 sec 2024-03-28T18:44:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:57 smithi071 ceph-mon[36597]: pgmap v1794: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1795.709758 secs 2024-03-28T18:44:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:57 smithi151 ceph-mon[36481]: pgmap v1794: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1795.709758 secs 2024-03-28T18:44:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:57 smithi045 ceph-mon[31773]: pgmap v1794: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:44:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1795.709758 secs 2024-03-28T18:44:58.289 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3150 sec 2024-03-28T18:44:59.289 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3151 sec 2024-03-28T18:44:59.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:44:59 smithi045 ceph-mon[31773]: pgmap v1795: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:59.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:44:59 smithi071 ceph-mon[36597]: pgmap v1795: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:44:59.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:44:59 smithi151 ceph-mon[36481]: pgmap v1795: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:00.289 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3152 sec 2024-03-28T18:45:01.289 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3153 sec 2024-03-28T18:45:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:01 smithi071 ceph-mon[36597]: pgmap v1796: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:01 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2509.284957 secs 2024-03-28T18:45:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:01 smithi151 ceph-mon[36481]: pgmap v1796: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:01 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2509.284957 secs 2024-03-28T18:45:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:01 smithi045 ceph-mon[31773]: pgmap v1796: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:01 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2509.284957 secs 2024-03-28T18:45:02.289 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3154 sec 2024-03-28T18:45:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1800.709859 secs 2024-03-28T18:45:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1800.709859 secs 2024-03-28T18:45:02.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1800.709859 secs 2024-03-28T18:45:03.289 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3155 sec 2024-03-28T18:45:03.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:03 smithi071 ceph-mon[36597]: pgmap v1797: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:03 smithi151 ceph-mon[36481]: pgmap v1797: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:03.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:03 smithi045 ceph-mon[31773]: pgmap v1797: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:04.289 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3156 sec 2024-03-28T18:45:05.289 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3157 sec 2024-03-28T18:45:05.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:05 smithi071 ceph-mon[36597]: pgmap v1798: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:05.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:05 smithi151 ceph-mon[36481]: pgmap v1798: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:05.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:05 smithi045 ceph-mon[31773]: pgmap v1798: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:06.290 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3158 sec 2024-03-28T18:45:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:06 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2514.285046 secs 2024-03-28T18:45:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:06 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2514.285046 secs 2024-03-28T18:45:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:06 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2514.285046 secs 2024-03-28T18:45:07.290 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3159 sec 2024-03-28T18:45:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:07 smithi071 ceph-mon[36597]: pgmap v1799: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1805.709921 secs 2024-03-28T18:45:07.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:07 smithi045 ceph-mon[31773]: pgmap v1799: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:07.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1805.709921 secs 2024-03-28T18:45:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:07 smithi151 ceph-mon[36481]: pgmap v1799: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1805.709921 secs 2024-03-28T18:45:08.290 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3160 sec 2024-03-28T18:45:09.290 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3161 sec 2024-03-28T18:45:09.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:09 smithi071 ceph-mon[36597]: pgmap v1800: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:09.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:09 smithi151 ceph-mon[36481]: pgmap v1800: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:09.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:09 smithi045 ceph-mon[31773]: pgmap v1800: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:10.290 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3162 sec 2024-03-28T18:45:11.290 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3163 sec 2024-03-28T18:45:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:11 smithi151 ceph-mon[36481]: pgmap v1801: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:11 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2519.285063 secs 2024-03-28T18:45:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:11 smithi045 ceph-mon[31773]: pgmap v1801: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:11 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2519.285063 secs 2024-03-28T18:45:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:11 smithi071 ceph-mon[36597]: pgmap v1801: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:11 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2519.285063 secs 2024-03-28T18:45:12.291 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3164 sec 2024-03-28T18:45:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1810.709982 secs 2024-03-28T18:45:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1810.709982 secs 2024-03-28T18:45:12.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1810.709982 secs 2024-03-28T18:45:13.291 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3165 sec 2024-03-28T18:45:13.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:13 smithi045 ceph-mon[31773]: pgmap v1802: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:13.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:13 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:45:13.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:13 smithi071 ceph-mon[36597]: pgmap v1802: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:13.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:13 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:45:13.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:13 smithi151 ceph-mon[36481]: pgmap v1802: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:13.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:13 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:45:14.291 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3166 sec 2024-03-28T18:45:15.291 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3167 sec 2024-03-28T18:45:15.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:15 smithi045 ceph-mon[31773]: pgmap v1803: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:15.621 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:15 smithi071 ceph-mon[36597]: pgmap v1803: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:15.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:15 smithi151 ceph-mon[36481]: pgmap v1803: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:16.294 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3168 sec 2024-03-28T18:45:16.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:16 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2524.285195 secs 2024-03-28T18:45:16.775 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:16 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2524.285195 secs 2024-03-28T18:45:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:16 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2524.285195 secs 2024-03-28T18:45:17.291 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3169 sec 2024-03-28T18:45:17.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:17 smithi045 ceph-mon[31773]: pgmap v1804: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:17.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1815.710045 secs 2024-03-28T18:45:17.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:17 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:45:17.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:17 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:45:17.747 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:17 smithi071 ceph-mon[36597]: pgmap v1804: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:17.747 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1815.710045 secs 2024-03-28T18:45:17.747 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:17 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:45:17.747 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:17 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:45:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:17 smithi151 ceph-mon[36481]: pgmap v1804: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1815.710045 secs 2024-03-28T18:45:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:17 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:45:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:17 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:45:18.291 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3170 sec 2024-03-28T18:45:19.292 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3171 sec 2024-03-28T18:45:19.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:19 smithi045 ceph-mon[31773]: pgmap v1805: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:19.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:19 smithi071 ceph-mon[36597]: pgmap v1805: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:19.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:19 smithi151 ceph-mon[36481]: pgmap v1805: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:20.292 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3172 sec 2024-03-28T18:45:20.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:20 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:45:20.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:20 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:45:20.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:20 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:45:20.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:20 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:45:20.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:20 smithi071 ceph-mon[36597]: pgmap v1806: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T18:45:20.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:20 smithi071 ceph-mon[36597]: pgmap v1807: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:45:20.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:20 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:45:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:20 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:45:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:20 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:45:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:20 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:45:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:20 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:45:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:20 smithi151 ceph-mon[36481]: pgmap v1806: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T18:45:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:20 smithi151 ceph-mon[36481]: pgmap v1807: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:45:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:20 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:45:20.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:20 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:45:20.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:20 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:45:20.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:20 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:45:20.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:20 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:45:20.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:20 smithi045 ceph-mon[31773]: pgmap v1806: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T18:45:20.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:20 smithi045 ceph-mon[31773]: pgmap v1807: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:45:20.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:20 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:45:21.292 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3173 sec 2024-03-28T18:45:21.826 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:21 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2529.285306 secs 2024-03-28T18:45:21.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:21 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2529.285306 secs 2024-03-28T18:45:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:21 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2529.285306 secs 2024-03-28T18:45:22.292 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3174 sec 2024-03-28T18:45:22.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:22 smithi045 ceph-mon[31773]: pgmap v1808: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:45:22.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1820.710132 secs 2024-03-28T18:45:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:22 smithi071 ceph-mon[36597]: pgmap v1808: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:45:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1820.710132 secs 2024-03-28T18:45:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:22 smithi151 ceph-mon[36481]: pgmap v1808: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:45:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1820.710132 secs 2024-03-28T18:45:23.292 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3175 sec 2024-03-28T18:45:24.293 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3176 sec 2024-03-28T18:45:24.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:24 smithi045 ceph-mon[31773]: pgmap v1809: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:45:25.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:24 smithi071 ceph-mon[36597]: pgmap v1809: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:45:25.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:24 smithi151 ceph-mon[36481]: pgmap v1809: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:45:25.292 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3177 sec 2024-03-28T18:45:26.293 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3178 sec 2024-03-28T18:45:26.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:26 smithi045 ceph-mon[31773]: pgmap v1810: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:45:26.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:26 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2534.285396 secs 2024-03-28T18:45:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:26 smithi071 ceph-mon[36597]: pgmap v1810: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:45:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:26 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2534.285396 secs 2024-03-28T18:45:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:26 smithi151 ceph-mon[36481]: pgmap v1810: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:45:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:26 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2534.285396 secs 2024-03-28T18:45:27.293 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3179 sec 2024-03-28T18:45:27.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1825.710195 secs 2024-03-28T18:45:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1825.710195 secs 2024-03-28T18:45:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1825.710195 secs 2024-03-28T18:45:28.293 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3180 sec 2024-03-28T18:45:29.293 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3181 sec 2024-03-28T18:45:29.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:29 smithi045 ceph-mon[31773]: pgmap v1811: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:45:29.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:29 smithi071 ceph-mon[36597]: pgmap v1811: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:45:29.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:29 smithi151 ceph-mon[36481]: pgmap v1811: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:45:30.293 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3182 sec 2024-03-28T18:45:31.294 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3183 sec 2024-03-28T18:45:31.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:31 smithi045 ceph-mon[31773]: pgmap v1812: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:45:31.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:31 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2539.285487 secs 2024-03-28T18:45:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:31 smithi071 ceph-mon[36597]: pgmap v1812: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:45:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:31 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2539.285487 secs 2024-03-28T18:45:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:31 smithi151 ceph-mon[36481]: pgmap v1812: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:45:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:31 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2539.285487 secs 2024-03-28T18:45:32.294 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3184 sec 2024-03-28T18:45:32.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1830.710282 secs 2024-03-28T18:45:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1830.710282 secs 2024-03-28T18:45:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1830.710282 secs 2024-03-28T18:45:33.294 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3185 sec 2024-03-28T18:45:33.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:33 smithi045 ceph-mon[31773]: pgmap v1813: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:33.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:33 smithi071 ceph-mon[36597]: pgmap v1813: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:33 smithi151 ceph-mon[36481]: pgmap v1813: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:34.294 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3186 sec 2024-03-28T18:45:35.294 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3187 sec 2024-03-28T18:45:35.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:35 smithi045 ceph-mon[31773]: pgmap v1814: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:35.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:35 smithi071 ceph-mon[36597]: pgmap v1814: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:35.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:35 smithi151 ceph-mon[36481]: pgmap v1814: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:36.294 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3188 sec 2024-03-28T18:45:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:36 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2544.285575 secs 2024-03-28T18:45:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:36 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2544.285575 secs 2024-03-28T18:45:36.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:36 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2544.285575 secs 2024-03-28T18:45:37.294 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3189 sec 2024-03-28T18:45:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:37 smithi071 ceph-mon[36597]: pgmap v1815: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1835.710342 secs 2024-03-28T18:45:37.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:37 smithi045 ceph-mon[31773]: pgmap v1815: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:37.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1835.710342 secs 2024-03-28T18:45:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:37 smithi151 ceph-mon[36481]: pgmap v1815: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1835.710342 secs 2024-03-28T18:45:38.294 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3190 sec 2024-03-28T18:45:39.294 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3191 sec 2024-03-28T18:45:39.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:39 smithi071 ceph-mon[36597]: pgmap v1816: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:39.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:39 smithi151 ceph-mon[36481]: pgmap v1816: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:39.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:39 smithi045 ceph-mon[31773]: pgmap v1816: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:40.295 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3192 sec 2024-03-28T18:45:41.295 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3193 sec 2024-03-28T18:45:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:41 smithi071 ceph-mon[36597]: pgmap v1817: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:41.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:41 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2549.285659 secs 2024-03-28T18:45:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:41 smithi151 ceph-mon[36481]: pgmap v1817: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:41 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2549.285659 secs 2024-03-28T18:45:41.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:41 smithi045 ceph-mon[31773]: pgmap v1817: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:41.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:41 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2549.285659 secs 2024-03-28T18:45:42.295 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3194 sec 2024-03-28T18:45:42.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1840.710400 secs 2024-03-28T18:45:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1840.710400 secs 2024-03-28T18:45:42.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1840.710400 secs 2024-03-28T18:45:43.295 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3195 sec 2024-03-28T18:45:43.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:43 smithi071 ceph-mon[36597]: pgmap v1818: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:43.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:43 smithi151 ceph-mon[36481]: pgmap v1818: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:43.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:43 smithi045 ceph-mon[31773]: pgmap v1818: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:44.295 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3196 sec 2024-03-28T18:45:45.296 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3197 sec 2024-03-28T18:45:45.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:45 smithi071 ceph-mon[36597]: pgmap v1819: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:45.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:45 smithi151 ceph-mon[36481]: pgmap v1819: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:45.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:45 smithi045 ceph-mon[31773]: pgmap v1819: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:46.295 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3198 sec 2024-03-28T18:45:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:46 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2554.285744 secs 2024-03-28T18:45:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:46 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2554.285744 secs 2024-03-28T18:45:46.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:46 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2554.285744 secs 2024-03-28T18:45:47.295 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3199 sec 2024-03-28T18:45:47.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:47 smithi071 ceph-mon[36597]: pgmap v1820: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:47.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1845.710469 secs 2024-03-28T18:45:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:47 smithi151 ceph-mon[36481]: pgmap v1820: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1845.710469 secs 2024-03-28T18:45:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:47 smithi045 ceph-mon[31773]: pgmap v1820: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:47.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1845.710469 secs 2024-03-28T18:45:48.296 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3200 sec 2024-03-28T18:45:49.296 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3201 sec 2024-03-28T18:45:49.468 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:49 smithi045 ceph-mon[31773]: pgmap v1821: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:49.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:49 smithi071 ceph-mon[36597]: pgmap v1821: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:49 smithi151 ceph-mon[36481]: pgmap v1821: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:50.296 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3202 sec 2024-03-28T18:45:51.296 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3203 sec 2024-03-28T18:45:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:51 smithi071 ceph-mon[36597]: pgmap v1822: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:51 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2559.285803 secs 2024-03-28T18:45:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:51 smithi151 ceph-mon[36481]: pgmap v1822: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:51 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2559.285803 secs 2024-03-28T18:45:51.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:51 smithi045 ceph-mon[31773]: pgmap v1822: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:51.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:51 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2559.285803 secs 2024-03-28T18:45:52.296 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3204 sec 2024-03-28T18:45:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1850.710530 secs 2024-03-28T18:45:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1850.710530 secs 2024-03-28T18:45:52.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1850.710530 secs 2024-03-28T18:45:53.296 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3205 sec 2024-03-28T18:45:53.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:53 smithi071 ceph-mon[36597]: pgmap v1823: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:53.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:53 smithi151 ceph-mon[36481]: pgmap v1823: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:53.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:53 smithi045 ceph-mon[31773]: pgmap v1823: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:54.297 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3206 sec 2024-03-28T18:45:55.297 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3207 sec 2024-03-28T18:45:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:55 smithi071 ceph-mon[36597]: pgmap v1824: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:55.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:55 smithi151 ceph-mon[36481]: pgmap v1824: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:55.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:55 smithi045 ceph-mon[31773]: pgmap v1824: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:56.297 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3208 sec 2024-03-28T18:45:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:56 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2564.285872 secs 2024-03-28T18:45:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:56 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2564.285872 secs 2024-03-28T18:45:56.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:56 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2564.285872 secs 2024-03-28T18:45:57.297 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3209 sec 2024-03-28T18:45:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:57 smithi071 ceph-mon[36597]: pgmap v1825: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1855.710614 secs 2024-03-28T18:45:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:57 smithi151 ceph-mon[36481]: pgmap v1825: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1855.710614 secs 2024-03-28T18:45:57.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:57 smithi045 ceph-mon[31773]: pgmap v1825: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:45:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1855.710614 secs 2024-03-28T18:45:58.297 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3210 sec 2024-03-28T18:45:59.297 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3211 sec 2024-03-28T18:45:59.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:45:59 smithi071 ceph-mon[36597]: pgmap v1826: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:59.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:45:59 smithi151 ceph-mon[36481]: pgmap v1826: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:45:59.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:45:59 smithi045 ceph-mon[31773]: pgmap v1826: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:00.297 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3212 sec 2024-03-28T18:46:01.298 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3213 sec 2024-03-28T18:46:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:01 smithi071 ceph-mon[36597]: pgmap v1827: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:01 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2569.285928 secs 2024-03-28T18:46:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:01 smithi151 ceph-mon[36481]: pgmap v1827: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:01 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2569.285928 secs 2024-03-28T18:46:01.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:01 smithi045 ceph-mon[31773]: pgmap v1827: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:01 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2569.285928 secs 2024-03-28T18:46:02.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:46:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:46:01.742+0000 7f52eec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:46:02.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:46:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:46:01.742+0000 7f52eec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:46:02.166 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:46:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:46:01.742+0000 7f52eec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:46:02.167 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:46:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:46:01.742+0000 7f52eec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:46:02.167 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:46:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:46:01.742+0000 7f52eec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:46:02.298 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3214 sec 2024-03-28T18:46:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1860.710651 secs 2024-03-28T18:46:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1860.710651 secs 2024-03-28T18:46:02.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1860.710651 secs 2024-03-28T18:46:03.298 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3215 sec 2024-03-28T18:46:03.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:03 smithi071 ceph-mon[36597]: pgmap v1828: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:03 smithi151 ceph-mon[36481]: pgmap v1828: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:03.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:03 smithi045 ceph-mon[31773]: pgmap v1828: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:04.298 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3216 sec 2024-03-28T18:46:05.298 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3217 sec 2024-03-28T18:46:05.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:05 smithi071 ceph-mon[36597]: pgmap v1829: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:05.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:05 smithi151 ceph-mon[36481]: pgmap v1829: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:05.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:05 smithi045 ceph-mon[31773]: pgmap v1829: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:06.298 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3218 sec 2024-03-28T18:46:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:06 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2574.286016 secs 2024-03-28T18:46:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:06 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2574.286016 secs 2024-03-28T18:46:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:06 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2574.286016 secs 2024-03-28T18:46:07.298 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3219 sec 2024-03-28T18:46:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:07 smithi071 ceph-mon[36597]: pgmap v1830: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:46:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1865.710712 secs 2024-03-28T18:46:07.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:07 smithi045 ceph-mon[31773]: pgmap v1830: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:46:07.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1865.710712 secs 2024-03-28T18:46:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:07 smithi151 ceph-mon[36481]: pgmap v1830: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:46:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1865.710712 secs 2024-03-28T18:46:08.299 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3220 sec 2024-03-28T18:46:09.299 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3221 sec 2024-03-28T18:46:09.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:09 smithi071 ceph-mon[36597]: pgmap v1831: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:46:09.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:09 smithi151 ceph-mon[36481]: pgmap v1831: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:46:09.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:09 smithi045 ceph-mon[31773]: pgmap v1831: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:46:10.299 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3222 sec 2024-03-28T18:46:11.301 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3223 sec 2024-03-28T18:46:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:11 smithi071 ceph-mon[36597]: pgmap v1832: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:46:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:11 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2579.286104 secs 2024-03-28T18:46:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:11 smithi151 ceph-mon[36481]: pgmap v1832: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:46:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:11 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2579.286104 secs 2024-03-28T18:46:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:11 smithi045 ceph-mon[31773]: pgmap v1832: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:46:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:11 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2579.286104 secs 2024-03-28T18:46:12.299 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3224 sec 2024-03-28T18:46:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1870.710770 secs 2024-03-28T18:46:12.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1870.710770 secs 2024-03-28T18:46:12.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1870.710770 secs 2024-03-28T18:46:13.299 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3225 sec 2024-03-28T18:46:13.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:13 smithi071 ceph-mon[36597]: pgmap v1833: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:46:13.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:13 smithi151 ceph-mon[36481]: pgmap v1833: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:46:13.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:13 smithi045 ceph-mon[31773]: pgmap v1833: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:46:14.299 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3226 sec 2024-03-28T18:46:15.299 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3227 sec 2024-03-28T18:46:15.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:15 smithi151 ceph-mon[36481]: pgmap v1834: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:46:15.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:15 smithi045 ceph-mon[31773]: pgmap v1834: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:46:15.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:15 smithi071 ceph-mon[36597]: pgmap v1834: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:46:16.299 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3228 sec 2024-03-28T18:46:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:16 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2584.286165 secs 2024-03-28T18:46:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:16 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2584.286165 secs 2024-03-28T18:46:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:16 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2584.286165 secs 2024-03-28T18:46:17.300 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3229 sec 2024-03-28T18:46:17.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:17 smithi045 ceph-mon[31773]: pgmap v1835: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:46:17.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1875.710859 secs 2024-03-28T18:46:17.712 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:17 smithi071 ceph-mon[36597]: pgmap v1835: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:46:17.712 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1875.710859 secs 2024-03-28T18:46:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:17 smithi151 ceph-mon[36481]: pgmap v1835: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:46:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1875.710859 secs 2024-03-28T18:46:18.300 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3230 sec 2024-03-28T18:46:19.300 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3231 sec 2024-03-28T18:46:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:19 smithi045 ceph-mon[31773]: pgmap v1836: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:19.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:19 smithi071 ceph-mon[36597]: pgmap v1836: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:19.826 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:19 smithi151 ceph-mon[36481]: pgmap v1836: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:20.300 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3232 sec 2024-03-28T18:46:20.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:20 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:46:20.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:20 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:46:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:20 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:46:21.300 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3233 sec 2024-03-28T18:46:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:21 smithi045 ceph-mon[31773]: pgmap v1837: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:21.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:21 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2589.286224 secs 2024-03-28T18:46:21.491 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:21 smithi151 ceph-mon[36481]: pgmap v1837: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:21.492 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:21 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2589.286224 secs 2024-03-28T18:46:21.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:21 smithi071 ceph-mon[36597]: pgmap v1837: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:21.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:21 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2589.286224 secs 2024-03-28T18:46:22.300 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3234 sec 2024-03-28T18:46:22.498 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1880.710901 secs 2024-03-28T18:46:22.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1880.710901 secs 2024-03-28T18:46:22.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1880.710901 secs 2024-03-28T18:46:23.303 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3235 sec 2024-03-28T18:46:23.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:23 smithi151 ceph-mon[36481]: pgmap v1838: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:23.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:23 smithi045 ceph-mon[31773]: pgmap v1838: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:23.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:23 smithi071 ceph-mon[36597]: pgmap v1838: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:24.301 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3236 sec 2024-03-28T18:46:25.301 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3237 sec 2024-03-28T18:46:25.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:25 smithi071 ceph-mon[36597]: pgmap v1839: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:25.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:25 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:46:25.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:25 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:46:25.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:25 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:46:25.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:25 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:46:25.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:25 smithi071 ceph-mon[36597]: pgmap v1840: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:46:25.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:25 smithi071 ceph-mon[36597]: pgmap v1841: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:46:25.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:25 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:46:25.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:25 smithi151 ceph-mon[36481]: pgmap v1839: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:25.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:25 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:46:25.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:25 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:46:25.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:25 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:46:25.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:25 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:46:25.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:25 smithi151 ceph-mon[36481]: pgmap v1840: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:46:25.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:25 smithi151 ceph-mon[36481]: pgmap v1841: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:46:25.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:25 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:46:25.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:25 smithi045 ceph-mon[31773]: pgmap v1839: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:25.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:25 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:46:25.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:25 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:46:25.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:25 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:46:25.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:25 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:46:25.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:25 smithi045 ceph-mon[31773]: pgmap v1840: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:46:25.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:25 smithi045 ceph-mon[31773]: pgmap v1841: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:46:25.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:25 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:46:26.301 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3238 sec 2024-03-28T18:46:27.301 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3239 sec 2024-03-28T18:46:27.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:27 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2594.286282 secs 2024-03-28T18:46:27.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:27 smithi045 ceph-mon[31773]: pgmap v1842: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:46:27.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1885.710964 secs 2024-03-28T18:46:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:27 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2594.286282 secs 2024-03-28T18:46:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:27 smithi071 ceph-mon[36597]: pgmap v1842: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:46:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1885.710964 secs 2024-03-28T18:46:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:27 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2594.286282 secs 2024-03-28T18:46:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:27 smithi151 ceph-mon[36481]: pgmap v1842: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:46:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1885.710964 secs 2024-03-28T18:46:28.301 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3240 sec 2024-03-28T18:46:29.301 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3241 sec 2024-03-28T18:46:29.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:29 smithi045 ceph-mon[31773]: pgmap v1843: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:46:29.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:29 smithi071 ceph-mon[36597]: pgmap v1843: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:46:29.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:29 smithi151 ceph-mon[36481]: pgmap v1843: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:46:30.301 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3242 sec 2024-03-28T18:46:31.302 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3243 sec 2024-03-28T18:46:31.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:31 smithi045 ceph-mon[31773]: pgmap v1844: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:46:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:31 smithi071 ceph-mon[36597]: pgmap v1844: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:46:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:31 smithi151 ceph-mon[36481]: pgmap v1844: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:46:32.302 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3244 sec 2024-03-28T18:46:32.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:32 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2599.286369 secs 2024-03-28T18:46:32.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1890.710975 secs 2024-03-28T18:46:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:32 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2599.286369 secs 2024-03-28T18:46:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1890.710975 secs 2024-03-28T18:46:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:32 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2599.286369 secs 2024-03-28T18:46:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1890.710975 secs 2024-03-28T18:46:33.302 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3245 sec 2024-03-28T18:46:33.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:33 smithi045 ceph-mon[31773]: pgmap v1845: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:46:33.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:33 smithi071 ceph-mon[36597]: pgmap v1845: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:46:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:33 smithi151 ceph-mon[36481]: pgmap v1845: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:46:34.302 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3246 sec 2024-03-28T18:46:35.302 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3247 sec 2024-03-28T18:46:35.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:35 smithi045 ceph-mon[31773]: pgmap v1846: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:46:35.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:35 smithi071 ceph-mon[36597]: pgmap v1846: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:46:35.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:35 smithi151 ceph-mon[36481]: pgmap v1846: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:46:36.302 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3248 sec 2024-03-28T18:46:37.303 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3249 sec 2024-03-28T18:46:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:37 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2604.286434 secs 2024-03-28T18:46:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:37 smithi045 ceph-mon[31773]: pgmap v1847: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1895.711035 secs 2024-03-28T18:46:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:37 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2604.286434 secs 2024-03-28T18:46:37.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:37 smithi071 ceph-mon[36597]: pgmap v1847: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:37.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1895.711035 secs 2024-03-28T18:46:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:37 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2604.286434 secs 2024-03-28T18:46:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:37 smithi151 ceph-mon[36481]: pgmap v1847: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1895.711035 secs 2024-03-28T18:46:38.303 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3250 sec 2024-03-28T18:46:39.303 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3251 sec 2024-03-28T18:46:39.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:39 smithi045 ceph-mon[31773]: pgmap v1848: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:39.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:39 smithi071 ceph-mon[36597]: pgmap v1848: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:39.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:39 smithi151 ceph-mon[36481]: pgmap v1848: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:40.303 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3252 sec 2024-03-28T18:46:41.303 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3253 sec 2024-03-28T18:46:41.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:41 smithi045 ceph-mon[31773]: pgmap v1849: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:41.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:41 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2609.286522 secs 2024-03-28T18:46:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:41 smithi071 ceph-mon[36597]: pgmap v1849: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:41 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2609.286522 secs 2024-03-28T18:46:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:41 smithi151 ceph-mon[36481]: pgmap v1849: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:41 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2609.286522 secs 2024-03-28T18:46:42.303 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3254 sec 2024-03-28T18:46:42.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1900.711123 secs 2024-03-28T18:46:42.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1900.711123 secs 2024-03-28T18:46:42.570 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1900.711123 secs 2024-03-28T18:46:43.303 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3255 sec 2024-03-28T18:46:43.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:43 smithi045 ceph-mon[31773]: pgmap v1850: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:43.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:43 smithi071 ceph-mon[36597]: pgmap v1850: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:43.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:43 smithi151 ceph-mon[36481]: pgmap v1850: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:44.304 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3256 sec 2024-03-28T18:46:45.304 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3257 sec 2024-03-28T18:46:45.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:45 smithi045 ceph-mon[31773]: pgmap v1851: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:45.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:45 smithi071 ceph-mon[36597]: pgmap v1851: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:45.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:45 smithi151 ceph-mon[36481]: pgmap v1851: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:46.304 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3258 sec 2024-03-28T18:46:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:46 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2614.286581 secs 2024-03-28T18:46:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:46 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2614.286581 secs 2024-03-28T18:46:46.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:46 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2614.286581 secs 2024-03-28T18:46:47.304 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3259 sec 2024-03-28T18:46:47.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:47 smithi071 ceph-mon[36597]: pgmap v1852: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:47.462 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1905.711213 secs 2024-03-28T18:46:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:47 smithi151 ceph-mon[36481]: pgmap v1852: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:47.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1905.711213 secs 2024-03-28T18:46:47.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:47 smithi045 ceph-mon[31773]: pgmap v1852: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:47.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1905.711213 secs 2024-03-28T18:46:48.304 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3260 sec 2024-03-28T18:46:49.304 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3261 sec 2024-03-28T18:46:49.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:49 smithi071 ceph-mon[36597]: pgmap v1853: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:49 smithi151 ceph-mon[36481]: pgmap v1853: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:49.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:49 smithi045 ceph-mon[31773]: pgmap v1853: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:50.304 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3262 sec 2024-03-28T18:46:51.305 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3263 sec 2024-03-28T18:46:51.469 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:51 smithi045 ceph-mon[31773]: pgmap v1854: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:51.469 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:51 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2619.286671 secs 2024-03-28T18:46:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:51 smithi071 ceph-mon[36597]: pgmap v1854: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:51 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2619.286671 secs 2024-03-28T18:46:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:51 smithi151 ceph-mon[36481]: pgmap v1854: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:51 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2619.286671 secs 2024-03-28T18:46:52.305 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3264 sec 2024-03-28T18:46:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1910.711302 secs 2024-03-28T18:46:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1910.711302 secs 2024-03-28T18:46:52.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1910.711302 secs 2024-03-28T18:46:53.305 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3265 sec 2024-03-28T18:46:53.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:53 smithi071 ceph-mon[36597]: pgmap v1855: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:53.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:53 smithi151 ceph-mon[36481]: pgmap v1855: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:53.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:53 smithi045 ceph-mon[31773]: pgmap v1855: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:54.305 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3266 sec 2024-03-28T18:46:55.305 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3267 sec 2024-03-28T18:46:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:55 smithi071 ceph-mon[36597]: pgmap v1856: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:55.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:55 smithi151 ceph-mon[36481]: pgmap v1856: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:55.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:55 smithi045 ceph-mon[31773]: pgmap v1856: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:56.305 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3268 sec 2024-03-28T18:46:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:56 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2624.286754 secs 2024-03-28T18:46:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:56 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2624.286754 secs 2024-03-28T18:46:56.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:56 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2624.286754 secs 2024-03-28T18:46:57.305 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3269 sec 2024-03-28T18:46:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:57 smithi071 ceph-mon[36597]: pgmap v1857: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1915.711389 secs 2024-03-28T18:46:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:57 smithi151 ceph-mon[36481]: pgmap v1857: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1915.711389 secs 2024-03-28T18:46:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:57 smithi045 ceph-mon[31773]: pgmap v1857: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:46:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1915.711389 secs 2024-03-28T18:46:58.306 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3270 sec 2024-03-28T18:46:59.306 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3271 sec 2024-03-28T18:46:59.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:46:59 smithi071 ceph-mon[36597]: pgmap v1858: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:59.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:46:59 smithi151 ceph-mon[36481]: pgmap v1858: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:46:59.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:46:59 smithi045 ceph-mon[31773]: pgmap v1858: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:00.306 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3272 sec 2024-03-28T18:47:01.306 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3273 sec 2024-03-28T18:47:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:01 smithi071 ceph-mon[36597]: pgmap v1859: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:01 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2629.286816 secs 2024-03-28T18:47:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:01 smithi151 ceph-mon[36481]: pgmap v1859: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:01 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2629.286816 secs 2024-03-28T18:47:01.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:01 smithi045 ceph-mon[31773]: pgmap v1859: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:01 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2629.286816 secs 2024-03-28T18:47:02.306 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3274 sec 2024-03-28T18:47:02.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1920.711452 secs 2024-03-28T18:47:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1920.711452 secs 2024-03-28T18:47:02.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1920.711452 secs 2024-03-28T18:47:03.306 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3275 sec 2024-03-28T18:47:03.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:03 smithi071 ceph-mon[36597]: pgmap v1860: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:03 smithi151 ceph-mon[36481]: pgmap v1860: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:03.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:03 smithi045 ceph-mon[31773]: pgmap v1860: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:04.306 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3276 sec 2024-03-28T18:47:05.307 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3277 sec 2024-03-28T18:47:05.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:05 smithi071 ceph-mon[36597]: pgmap v1861: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:05.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:05 smithi151 ceph-mon[36481]: pgmap v1861: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:05.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:05 smithi045 ceph-mon[31773]: pgmap v1861: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:06.307 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3278 sec 2024-03-28T18:47:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:06 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2634.286876 secs 2024-03-28T18:47:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:06 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2634.286876 secs 2024-03-28T18:47:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:06 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2634.286876 secs 2024-03-28T18:47:07.307 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3279 sec 2024-03-28T18:47:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:07 smithi071 ceph-mon[36597]: pgmap v1862: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1925.711515 secs 2024-03-28T18:47:07.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:07 smithi045 ceph-mon[31773]: pgmap v1862: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:07.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1925.711515 secs 2024-03-28T18:47:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:07 smithi151 ceph-mon[36481]: pgmap v1862: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1925.711515 secs 2024-03-28T18:47:08.308 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3280 sec 2024-03-28T18:47:09.307 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3281 sec 2024-03-28T18:47:09.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:09 smithi071 ceph-mon[36597]: pgmap v1863: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:09.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:09 smithi151 ceph-mon[36481]: pgmap v1863: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:09.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:09 smithi045 ceph-mon[31773]: pgmap v1863: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:10.307 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3282 sec 2024-03-28T18:47:11.307 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3283 sec 2024-03-28T18:47:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:11 smithi071 ceph-mon[36597]: pgmap v1864: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:11 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2639.286935 secs 2024-03-28T18:47:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:11 smithi151 ceph-mon[36481]: pgmap v1864: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:11 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2639.286935 secs 2024-03-28T18:47:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:11 smithi045 ceph-mon[31773]: pgmap v1864: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:11 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2639.286935 secs 2024-03-28T18:47:12.308 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3284 sec 2024-03-28T18:47:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1930.711552 secs 2024-03-28T18:47:12.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1930.711552 secs 2024-03-28T18:47:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1930.711552 secs 2024-03-28T18:47:13.309 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3285 sec 2024-03-28T18:47:13.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:13 smithi071 ceph-mon[36597]: pgmap v1865: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:13.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:13 smithi151 ceph-mon[36481]: pgmap v1865: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:13.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:13 smithi045 ceph-mon[31773]: pgmap v1865: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:14.308 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3286 sec 2024-03-28T18:47:15.308 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3287 sec 2024-03-28T18:47:15.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:15 smithi071 ceph-mon[36597]: pgmap v1866: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:15 smithi151 ceph-mon[36481]: pgmap v1866: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:15.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:15 smithi045 ceph-mon[31773]: pgmap v1866: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:16.308 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3288 sec 2024-03-28T18:47:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:16 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2644.286995 secs 2024-03-28T18:47:16.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:16 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2644.286995 secs 2024-03-28T18:47:16.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:16 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2644.286995 secs 2024-03-28T18:47:17.308 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3289 sec 2024-03-28T18:47:17.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:17 smithi071 ceph-mon[36597]: pgmap v1867: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:17.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1935.711615 secs 2024-03-28T18:47:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:17 smithi151 ceph-mon[36481]: pgmap v1867: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1935.711615 secs 2024-03-28T18:47:17.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:17 smithi045 ceph-mon[31773]: pgmap v1867: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:17.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1935.711615 secs 2024-03-28T18:47:18.308 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3290 sec 2024-03-28T18:47:19.309 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3291 sec 2024-03-28T18:47:19.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:19 smithi151 ceph-mon[36481]: pgmap v1868: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:19.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:19 smithi045 ceph-mon[31773]: pgmap v1868: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:19.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:19 smithi071 ceph-mon[36597]: pgmap v1868: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:20.309 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3292 sec 2024-03-28T18:47:21.309 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3293 sec 2024-03-28T18:47:21.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:21 smithi045 ceph-mon[31773]: pgmap v1869: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:21.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:21 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2649.287082 secs 2024-03-28T18:47:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:21 smithi071 ceph-mon[36597]: pgmap v1869: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:21 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2649.287082 secs 2024-03-28T18:47:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:21 smithi151 ceph-mon[36481]: pgmap v1869: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:21 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2649.287082 secs 2024-03-28T18:47:22.309 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3294 sec 2024-03-28T18:47:22.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1940.711678 secs 2024-03-28T18:47:22.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1940.711678 secs 2024-03-28T18:47:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1940.711678 secs 2024-03-28T18:47:23.309 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3295 sec 2024-03-28T18:47:23.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:23 smithi045 ceph-mon[31773]: pgmap v1870: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:23.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:23 smithi071 ceph-mon[36597]: pgmap v1870: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:23.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:23 smithi151 ceph-mon[36481]: pgmap v1870: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:24.309 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3296 sec 2024-03-28T18:47:25.310 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3297 sec 2024-03-28T18:47:25.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:25 smithi045 ceph-mon[31773]: pgmap v1871: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:25.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:25 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:47:25.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:25 smithi071 ceph-mon[36597]: pgmap v1871: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:25.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:25 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:47:25.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:25 smithi151 ceph-mon[36481]: pgmap v1871: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:25.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:25 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:47:26.310 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3298 sec 2024-03-28T18:47:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:26 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:47:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:26 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:47:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:26 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:47:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:26 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2654.287171 secs 2024-03-28T18:47:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:26 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:47:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:26 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:47:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:26 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:47:26.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:26 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2654.287171 secs 2024-03-28T18:47:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:26 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:47:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:26 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:47:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:26 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:47:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:26 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2654.287171 secs 2024-03-28T18:47:27.310 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3299 sec 2024-03-28T18:47:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:27 smithi045 ceph-mon[31773]: pgmap v1872: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:47:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:27 smithi045 ceph-mon[31773]: pgmap v1873: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:27.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1945.711765 secs 2024-03-28T18:47:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:27 smithi071 ceph-mon[36597]: pgmap v1872: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:47:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:27 smithi071 ceph-mon[36597]: pgmap v1873: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1945.711765 secs 2024-03-28T18:47:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:27 smithi151 ceph-mon[36481]: pgmap v1872: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:47:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:27 smithi151 ceph-mon[36481]: pgmap v1873: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1945.711765 secs 2024-03-28T18:47:28.311 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3300 sec 2024-03-28T18:47:29.310 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3301 sec 2024-03-28T18:47:29.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:29 smithi045 ceph-mon[31773]: pgmap v1874: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:29.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:29 smithi071 ceph-mon[36597]: pgmap v1874: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:29.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:29 smithi151 ceph-mon[36481]: pgmap v1874: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:30.310 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3302 sec 2024-03-28T18:47:31.311 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3303 sec 2024-03-28T18:47:31.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:31 smithi045 ceph-mon[31773]: pgmap v1875: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:47:31.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:31 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2659.287259 secs 2024-03-28T18:47:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:31 smithi071 ceph-mon[36597]: pgmap v1875: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:47:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:31 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2659.287259 secs 2024-03-28T18:47:31.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:31 smithi151 ceph-mon[36481]: pgmap v1875: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:47:31.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:31 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2659.287259 secs 2024-03-28T18:47:32.311 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3304 sec 2024-03-28T18:47:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1950.711827 secs 2024-03-28T18:47:32.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1950.711827 secs 2024-03-28T18:47:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1950.711827 secs 2024-03-28T18:47:33.311 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3305 sec 2024-03-28T18:47:33.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:33 smithi045 ceph-mon[31773]: pgmap v1876: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:33.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:33 smithi071 ceph-mon[36597]: pgmap v1876: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:33.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:33 smithi151 ceph-mon[36481]: pgmap v1876: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:34.311 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3306 sec 2024-03-28T18:47:35.311 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3307 sec 2024-03-28T18:47:35.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:35 smithi045 ceph-mon[31773]: pgmap v1877: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:35.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:35 smithi071 ceph-mon[36597]: pgmap v1877: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:35.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:35 smithi151 ceph-mon[36481]: pgmap v1877: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:47:36.311 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3308 sec 2024-03-28T18:47:36.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:36 smithi045 ceph-mon[31773]: 4 slow requests, 1 included below; oldest blocked for > 2664.287348 secs 2024-03-28T18:47:36.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:36 smithi045 ceph-mon[31773]: slow request 724.374323 seconds old, received at 2024-03-28T18:35:31.473570+0000: client_request(client.26711:1 getattr pAsLsXsFs #0x1 2024-03-28T18:35:31.469661+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:47:36.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:36 smithi071 ceph-mon[36597]: 4 slow requests, 1 included below; oldest blocked for > 2664.287348 secs 2024-03-28T18:47:36.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:36 smithi071 ceph-mon[36597]: slow request 724.374323 seconds old, received at 2024-03-28T18:35:31.473570+0000: client_request(client.26711:1 getattr pAsLsXsFs #0x1 2024-03-28T18:35:31.469661+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:47:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:36 smithi151 ceph-mon[36481]: 4 slow requests, 1 included below; oldest blocked for > 2664.287348 secs 2024-03-28T18:47:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:36 smithi151 ceph-mon[36481]: slow request 724.374323 seconds old, received at 2024-03-28T18:35:31.473570+0000: client_request(client.26711:1 getattr pAsLsXsFs #0x1 2024-03-28T18:35:31.469661+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:47:37.311 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3309 sec 2024-03-28T18:47:37.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:37 smithi045 ceph-mon[31773]: pgmap v1878: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:47:37.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1955.711905 secs 2024-03-28T18:47:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:37 smithi071 ceph-mon[36597]: pgmap v1878: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:47:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1955.711905 secs 2024-03-28T18:47:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:37 smithi151 ceph-mon[36481]: pgmap v1878: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:47:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1955.711905 secs 2024-03-28T18:47:38.312 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3310 sec 2024-03-28T18:47:39.312 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3311 sec 2024-03-28T18:47:39.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:39 smithi045 ceph-mon[31773]: pgmap v1879: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:39 smithi071 ceph-mon[36597]: pgmap v1879: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:39 smithi151 ceph-mon[36481]: pgmap v1879: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:40.313 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3312 sec 2024-03-28T18:47:41.312 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3313 sec 2024-03-28T18:47:41.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:41 smithi045 ceph-mon[31773]: pgmap v1880: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:41.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:41 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2669.287435 secs 2024-03-28T18:47:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:41 smithi071 ceph-mon[36597]: pgmap v1880: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:41 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2669.287435 secs 2024-03-28T18:47:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:41 smithi151 ceph-mon[36481]: pgmap v1880: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:41 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2669.287435 secs 2024-03-28T18:47:42.312 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3314 sec 2024-03-28T18:47:42.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1960.711970 secs 2024-03-28T18:47:42.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1960.711970 secs 2024-03-28T18:47:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1960.711970 secs 2024-03-28T18:47:43.312 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3315 sec 2024-03-28T18:47:43.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:43 smithi045 ceph-mon[31773]: pgmap v1881: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:43.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:43 smithi071 ceph-mon[36597]: pgmap v1881: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:43.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:43 smithi151 ceph-mon[36481]: pgmap v1881: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:44.312 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3316 sec 2024-03-28T18:47:45.313 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3317 sec 2024-03-28T18:47:45.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:45 smithi071 ceph-mon[36597]: pgmap v1882: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:45.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:45 smithi151 ceph-mon[36481]: pgmap v1882: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:45.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:45 smithi045 ceph-mon[31773]: pgmap v1882: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:46.313 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3318 sec 2024-03-28T18:47:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:46 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2674.287486 secs 2024-03-28T18:47:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:46 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2674.287486 secs 2024-03-28T18:47:46.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:46 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2674.287486 secs 2024-03-28T18:47:47.313 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3319 sec 2024-03-28T18:47:47.712 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:47 smithi071 ceph-mon[36597]: pgmap v1883: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:47.712 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1965.712010 secs 2024-03-28T18:47:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:47 smithi151 ceph-mon[36481]: pgmap v1883: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1965.712010 secs 2024-03-28T18:47:47.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:47 smithi045 ceph-mon[31773]: pgmap v1883: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:47.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1965.712010 secs 2024-03-28T18:47:48.314 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3320 sec 2024-03-28T18:47:49.314 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3321 sec 2024-03-28T18:47:49.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:49 smithi071 ceph-mon[36597]: pgmap v1884: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:49.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:49 smithi151 ceph-mon[36481]: pgmap v1884: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:49.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:49 smithi045 ceph-mon[31773]: pgmap v1884: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:50.313 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3322 sec 2024-03-28T18:47:51.313 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3323 sec 2024-03-28T18:47:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:51 smithi071 ceph-mon[36597]: pgmap v1885: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:51 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2679.287548 secs 2024-03-28T18:47:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:51 smithi151 ceph-mon[36481]: pgmap v1885: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:51 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2679.287548 secs 2024-03-28T18:47:51.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:51 smithi045 ceph-mon[31773]: pgmap v1885: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:51.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:51 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2679.287548 secs 2024-03-28T18:47:52.314 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3324 sec 2024-03-28T18:47:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1970.712074 secs 2024-03-28T18:47:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1970.712074 secs 2024-03-28T18:47:52.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1970.712074 secs 2024-03-28T18:47:53.314 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3325 sec 2024-03-28T18:47:53.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:53 smithi045 ceph-mon[31773]: pgmap v1886: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:53.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:53 smithi071 ceph-mon[36597]: pgmap v1886: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:53.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:53 smithi151 ceph-mon[36481]: pgmap v1886: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:54.314 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3326 sec 2024-03-28T18:47:55.314 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3327 sec 2024-03-28T18:47:55.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:55 smithi071 ceph-mon[36597]: pgmap v1887: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:55.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:55 smithi151 ceph-mon[36481]: pgmap v1887: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:55.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:55 smithi045 ceph-mon[31773]: pgmap v1887: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:56.314 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3328 sec 2024-03-28T18:47:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:56 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2684.287612 secs 2024-03-28T18:47:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:56 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2684.287612 secs 2024-03-28T18:47:56.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:56 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2684.287612 secs 2024-03-28T18:47:57.314 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3329 sec 2024-03-28T18:47:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:57 smithi071 ceph-mon[36597]: pgmap v1888: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1975.712135 secs 2024-03-28T18:47:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:57 smithi151 ceph-mon[36481]: pgmap v1888: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1975.712135 secs 2024-03-28T18:47:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:57 smithi045 ceph-mon[31773]: pgmap v1888: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:47:57.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1975.712135 secs 2024-03-28T18:47:58.314 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3330 sec 2024-03-28T18:47:59.315 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3331 sec 2024-03-28T18:47:59.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:47:59 smithi071 ceph-mon[36597]: pgmap v1889: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:59.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:47:59 smithi151 ceph-mon[36481]: pgmap v1889: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:47:59.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:47:59 smithi045 ceph-mon[31773]: pgmap v1889: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:00.315 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3332 sec 2024-03-28T18:48:01.315 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3333 sec 2024-03-28T18:48:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:01 smithi071 ceph-mon[36597]: pgmap v1890: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:01 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2689.287672 secs 2024-03-28T18:48:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:01 smithi151 ceph-mon[36481]: pgmap v1890: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:01 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2689.287672 secs 2024-03-28T18:48:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:01 smithi045 ceph-mon[31773]: pgmap v1890: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:01.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:01 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2689.287672 secs 2024-03-28T18:48:02.315 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3334 sec 2024-03-28T18:48:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1980.712223 secs 2024-03-28T18:48:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1980.712223 secs 2024-03-28T18:48:02.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1980.712223 secs 2024-03-28T18:48:03.315 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3335 sec 2024-03-28T18:48:03.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:03 smithi071 ceph-mon[36597]: pgmap v1891: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:03.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:03 smithi151 ceph-mon[36481]: pgmap v1891: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:03.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:03 smithi045 ceph-mon[31773]: pgmap v1891: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:04.315 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3336 sec 2024-03-28T18:48:04.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:04 smithi071 ceph-mon[36597]: pgmap v1892: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:04.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:04 smithi151 ceph-mon[36481]: pgmap v1892: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:04.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:04 smithi045 ceph-mon[31773]: pgmap v1892: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:05.315 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3337 sec 2024-03-28T18:48:06.315 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3338 sec 2024-03-28T18:48:07.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:06 smithi071 ceph-mon[36597]: pgmap v1893: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:07.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:06 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2694.287732 secs 2024-03-28T18:48:07.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:06 smithi151 ceph-mon[36481]: pgmap v1893: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:07.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:06 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2694.287732 secs 2024-03-28T18:48:07.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:06 smithi045 ceph-mon[31773]: pgmap v1893: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:07.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:06 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2694.287732 secs 2024-03-28T18:48:07.315 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3339 sec 2024-03-28T18:48:08.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1985.712310 secs 2024-03-28T18:48:08.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1985.712310 secs 2024-03-28T18:48:08.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1985.712310 secs 2024-03-28T18:48:08.316 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3340 sec 2024-03-28T18:48:09.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:08 smithi071 ceph-mon[36597]: pgmap v1894: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:09.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:08 smithi151 ceph-mon[36481]: pgmap v1894: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:09.135 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:08 smithi045 ceph-mon[31773]: pgmap v1894: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:09.316 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3341 sec 2024-03-28T18:48:10.316 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3342 sec 2024-03-28T18:48:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:10 smithi071 ceph-mon[36597]: pgmap v1895: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:11.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:10 smithi151 ceph-mon[36481]: pgmap v1895: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:11.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:10 smithi045 ceph-mon[31773]: pgmap v1895: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:11.316 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3343 sec 2024-03-28T18:48:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:11 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2699.287783 secs 2024-03-28T18:48:12.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:11 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2699.287783 secs 2024-03-28T18:48:12.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:11 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2699.287783 secs 2024-03-28T18:48:12.316 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3344 sec 2024-03-28T18:48:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1990.712399 secs 2024-03-28T18:48:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:12 smithi071 ceph-mon[36597]: pgmap v1896: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:13.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1990.712399 secs 2024-03-28T18:48:13.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:12 smithi151 ceph-mon[36481]: pgmap v1896: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1990.712399 secs 2024-03-28T18:48:13.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:12 smithi045 ceph-mon[31773]: pgmap v1896: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:13.316 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3345 sec 2024-03-28T18:48:14.317 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3346 sec 2024-03-28T18:48:15.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:14 smithi071 ceph-mon[36597]: pgmap v1897: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:15.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:14 smithi151 ceph-mon[36481]: pgmap v1897: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:15.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:14 smithi045 ceph-mon[31773]: pgmap v1897: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:15.317 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3347 sec 2024-03-28T18:48:16.317 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3348 sec 2024-03-28T18:48:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:16 smithi071 ceph-mon[36597]: pgmap v1898: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:16 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2704.287872 secs 2024-03-28T18:48:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:16 smithi151 ceph-mon[36481]: pgmap v1898: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:16 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2704.287872 secs 2024-03-28T18:48:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:16 smithi045 ceph-mon[31773]: pgmap v1898: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:16 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2704.287872 secs 2024-03-28T18:48:17.317 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3349 sec 2024-03-28T18:48:18.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 1995.712487 secs 2024-03-28T18:48:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 1995.712487 secs 2024-03-28T18:48:18.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 1995.712487 secs 2024-03-28T18:48:18.317 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3350 sec 2024-03-28T18:48:19.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:18 smithi071 ceph-mon[36597]: pgmap v1899: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:19.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:18 smithi151 ceph-mon[36481]: pgmap v1899: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:19.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:18 smithi045 ceph-mon[31773]: pgmap v1899: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:19.317 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3351 sec 2024-03-28T18:48:20.318 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3352 sec 2024-03-28T18:48:21.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:20 smithi071 ceph-mon[36597]: pgmap v1900: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:21.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:20 smithi151 ceph-mon[36481]: pgmap v1900: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:21.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:20 smithi045 ceph-mon[31773]: pgmap v1900: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:21.317 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3353 sec 2024-03-28T18:48:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:21 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2709.287960 secs 2024-03-28T18:48:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:21 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2709.287960 secs 2024-03-28T18:48:22.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:21 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2709.287960 secs 2024-03-28T18:48:22.318 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3354 sec 2024-03-28T18:48:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2000.712528 secs 2024-03-28T18:48:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:22 smithi071 ceph-mon[36597]: pgmap v1901: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2000.712528 secs 2024-03-28T18:48:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:22 smithi151 ceph-mon[36481]: pgmap v1901: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:23.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2000.712528 secs 2024-03-28T18:48:23.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:22 smithi045 ceph-mon[31773]: pgmap v1901: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:23.318 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3355 sec 2024-03-28T18:48:24.318 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3356 sec 2024-03-28T18:48:25.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:24 smithi071 ceph-mon[36597]: pgmap v1902: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:25.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:24 smithi151 ceph-mon[36481]: pgmap v1902: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:25.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:24 smithi045 ceph-mon[31773]: pgmap v1902: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:25.318 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3357 sec 2024-03-28T18:48:26.318 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3358 sec 2024-03-28T18:48:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:26 smithi151 ceph-mon[36481]: pgmap v1903: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:26 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2714.288043 secs 2024-03-28T18:48:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:26 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:48:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:26 smithi045 ceph-mon[31773]: pgmap v1903: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:26 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2714.288043 secs 2024-03-28T18:48:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:26 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:48:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:26 smithi071 ceph-mon[36597]: pgmap v1903: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:26 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2714.288043 secs 2024-03-28T18:48:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:26 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:48:27.318 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3359 sec 2024-03-28T18:48:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2005.712590 secs 2024-03-28T18:48:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:27 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:48:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:27 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:48:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:27 smithi151 ceph-mon[36481]: pgmap v1904: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:27 smithi151 ceph-mon[36481]: pgmap v1905: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:28.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:27 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:48:28.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2005.712590 secs 2024-03-28T18:48:28.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:27 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:48:28.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:27 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:48:28.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:27 smithi045 ceph-mon[31773]: pgmap v1904: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:28.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:27 smithi045 ceph-mon[31773]: pgmap v1905: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:28.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:27 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:48:28.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2005.712590 secs 2024-03-28T18:48:28.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:27 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:48:28.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:27 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:48:28.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:27 smithi071 ceph-mon[36597]: pgmap v1904: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:28.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:27 smithi071 ceph-mon[36597]: pgmap v1905: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:28.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:27 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:48:28.318 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3360 sec 2024-03-28T18:48:29.319 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3361 sec 2024-03-28T18:48:30.319 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3362 sec 2024-03-28T18:48:30.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:30 smithi045 ceph-mon[31773]: pgmap v1906: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:30.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:30 smithi071 ceph-mon[36597]: pgmap v1906: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:30.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:30 smithi151 ceph-mon[36481]: pgmap v1906: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:31.319 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3363 sec 2024-03-28T18:48:31.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:31 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2719.288102 secs 2024-03-28T18:48:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:31 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2719.288102 secs 2024-03-28T18:48:31.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:31 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2719.288102 secs 2024-03-28T18:48:32.319 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3364 sec 2024-03-28T18:48:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:32 smithi045 ceph-mon[31773]: pgmap v1907: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2010.712651 secs 2024-03-28T18:48:32.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:32 smithi071 ceph-mon[36597]: pgmap v1907: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:32.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2010.712651 secs 2024-03-28T18:48:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:32 smithi151 ceph-mon[36481]: pgmap v1907: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2010.712651 secs 2024-03-28T18:48:33.319 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3365 sec 2024-03-28T18:48:34.319 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3366 sec 2024-03-28T18:48:34.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:34 smithi045 ceph-mon[31773]: pgmap v1908: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:34.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:34 smithi071 ceph-mon[36597]: pgmap v1908: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:34.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:34 smithi151 ceph-mon[36481]: pgmap v1908: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:35.319 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3367 sec 2024-03-28T18:48:36.320 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3368 sec 2024-03-28T18:48:36.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:36 smithi045 ceph-mon[31773]: pgmap v1909: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:36.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:36 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2724.288161 secs 2024-03-28T18:48:36.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:36 smithi071 ceph-mon[36597]: pgmap v1909: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:36.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:36 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2724.288161 secs 2024-03-28T18:48:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:36 smithi151 ceph-mon[36481]: pgmap v1909: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:48:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:36 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2724.288161 secs 2024-03-28T18:48:37.320 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3369 sec 2024-03-28T18:48:37.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2015.712710 secs 2024-03-28T18:48:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2015.712710 secs 2024-03-28T18:48:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2015.712710 secs 2024-03-28T18:48:38.320 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3370 sec 2024-03-28T18:48:38.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:38 smithi045 ceph-mon[31773]: pgmap v1910: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:48:38.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:38 smithi071 ceph-mon[36597]: pgmap v1910: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:48:38.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:38 smithi151 ceph-mon[36481]: pgmap v1910: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:48:39.320 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3371 sec 2024-03-28T18:48:40.320 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3372 sec 2024-03-28T18:48:40.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:40 smithi045 ceph-mon[31773]: pgmap v1911: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:40.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:40 smithi071 ceph-mon[36597]: pgmap v1911: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:40 smithi151 ceph-mon[36481]: pgmap v1911: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:41.320 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3373 sec 2024-03-28T18:48:41.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:41 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2729.288220 secs 2024-03-28T18:48:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:41 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2729.288220 secs 2024-03-28T18:48:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:41 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2729.288220 secs 2024-03-28T18:48:42.320 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3374 sec 2024-03-28T18:48:42.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:42 smithi045 ceph-mon[31773]: pgmap v1912: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:42.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2020.712752 secs 2024-03-28T18:48:42.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:42 smithi151 ceph-mon[36481]: pgmap v1912: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:42.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2020.712752 secs 2024-03-28T18:48:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:42 smithi071 ceph-mon[36597]: pgmap v1912: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2020.712752 secs 2024-03-28T18:48:43.321 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3375 sec 2024-03-28T18:48:44.321 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3376 sec 2024-03-28T18:48:44.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:44 smithi045 ceph-mon[31773]: pgmap v1913: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:44.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:44 smithi071 ceph-mon[36597]: pgmap v1913: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:44 smithi151 ceph-mon[36481]: pgmap v1913: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:45.321 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3377 sec 2024-03-28T18:48:46.321 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3378 sec 2024-03-28T18:48:46.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:46 smithi045 ceph-mon[31773]: pgmap v1914: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:46.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:46 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2734.288281 secs 2024-03-28T18:48:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:46 smithi071 ceph-mon[36597]: pgmap v1914: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:46 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2734.288281 secs 2024-03-28T18:48:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:46 smithi151 ceph-mon[36481]: pgmap v1914: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:46 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2734.288281 secs 2024-03-28T18:48:47.321 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3379 sec 2024-03-28T18:48:47.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2025.712811 secs 2024-03-28T18:48:47.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2025.712811 secs 2024-03-28T18:48:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2025.712811 secs 2024-03-28T18:48:48.321 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3380 sec 2024-03-28T18:48:48.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:48 smithi071 ceph-mon[36597]: pgmap v1915: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:48.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:48 smithi151 ceph-mon[36481]: pgmap v1915: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:48.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:48 smithi045 ceph-mon[31773]: pgmap v1915: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:49.321 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3381 sec 2024-03-28T18:48:50.322 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3382 sec 2024-03-28T18:48:50.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:50 smithi071 ceph-mon[36597]: pgmap v1916: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:50.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:50 smithi151 ceph-mon[36481]: pgmap v1916: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:50.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:50 smithi045 ceph-mon[31773]: pgmap v1916: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:51.322 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3383 sec 2024-03-28T18:48:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:51 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2739.288343 secs 2024-03-28T18:48:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:51 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2739.288343 secs 2024-03-28T18:48:51.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:51 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2739.288343 secs 2024-03-28T18:48:52.322 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3384 sec 2024-03-28T18:48:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:52 smithi071 ceph-mon[36597]: pgmap v1917: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2030.712917 secs 2024-03-28T18:48:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:52 smithi151 ceph-mon[36481]: pgmap v1917: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2030.712917 secs 2024-03-28T18:48:52.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:52 smithi045 ceph-mon[31773]: pgmap v1917: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2030.712917 secs 2024-03-28T18:48:53.322 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3385 sec 2024-03-28T18:48:54.323 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3386 sec 2024-03-28T18:48:54.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:54 smithi071 ceph-mon[36597]: pgmap v1918: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:54.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:54 smithi151 ceph-mon[36481]: pgmap v1918: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:54.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:54 smithi045 ceph-mon[31773]: pgmap v1918: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:55.322 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3387 sec 2024-03-28T18:48:56.322 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3388 sec 2024-03-28T18:48:56.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:56 smithi045 ceph-mon[31773]: pgmap v1919: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:56.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:56 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2744.288402 secs 2024-03-28T18:48:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:56 smithi071 ceph-mon[36597]: pgmap v1919: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:56 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2744.288402 secs 2024-03-28T18:48:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:56 smithi151 ceph-mon[36481]: pgmap v1919: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:48:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:56 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2744.288402 secs 2024-03-28T18:48:57.323 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3389 sec 2024-03-28T18:48:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2035.713004 secs 2024-03-28T18:48:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2035.713004 secs 2024-03-28T18:48:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2035.713004 secs 2024-03-28T18:48:58.323 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3390 sec 2024-03-28T18:48:58.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:48:58 smithi071 ceph-mon[36597]: pgmap v1920: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:58.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:48:58 smithi151 ceph-mon[36481]: pgmap v1920: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:58.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:48:58 smithi045 ceph-mon[31773]: pgmap v1920: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:48:59.323 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3391 sec 2024-03-28T18:49:00.323 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3392 sec 2024-03-28T18:49:00.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:00 smithi071 ceph-mon[36597]: pgmap v1921: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:00 smithi151 ceph-mon[36481]: pgmap v1921: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:00.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:00 smithi045 ceph-mon[31773]: pgmap v1921: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:01.323 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3393 sec 2024-03-28T18:49:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:01 smithi071 ceph-mon[36597]: 4 slow requests, 0 included below; oldest blocked for > 2749.288504 secs 2024-03-28T18:49:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:01 smithi151 ceph-mon[36481]: 4 slow requests, 0 included below; oldest blocked for > 2749.288504 secs 2024-03-28T18:49:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:01 smithi045 ceph-mon[31773]: 4 slow requests, 0 included below; oldest blocked for > 2749.288504 secs 2024-03-28T18:49:02.323 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3394 sec 2024-03-28T18:49:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:02 smithi071 ceph-mon[36597]: pgmap v1922: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:02.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2040.713063 secs 2024-03-28T18:49:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:02 smithi151 ceph-mon[36481]: pgmap v1922: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2040.713063 secs 2024-03-28T18:49:02.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:02 smithi045 ceph-mon[31773]: pgmap v1922: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:02.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2040.713063 secs 2024-03-28T18:49:03.323 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3395 sec 2024-03-28T18:49:04.324 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3396 sec 2024-03-28T18:49:04.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:04 smithi071 ceph-mon[36597]: pgmap v1923: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:04.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:04 smithi151 ceph-mon[36481]: pgmap v1923: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:04.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:04 smithi045 ceph-mon[31773]: pgmap v1923: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:05.324 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3397 sec 2024-03-28T18:49:06.324 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3398 sec 2024-03-28T18:49:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:06 smithi071 ceph-mon[36597]: pgmap v1924: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:06.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:06 smithi071 ceph-mon[36597]: 5 slow requests, 1 included below; oldest blocked for > 2754.288592 secs 2024-03-28T18:49:06.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:06 smithi071 ceph-mon[36597]: slow request 184.100451 seconds old, received at 2024-03-28T18:46:01.748685+0000: client_request(client.26723:1 getattr pAsLsXsFs #0x1 2024-03-28T18:46:01.745025+0000 caller_uid=0, caller_gid=0{}) currently joining batch getattr 2024-03-28T18:49:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:06 smithi151 ceph-mon[36481]: pgmap v1924: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:06 smithi151 ceph-mon[36481]: 5 slow requests, 1 included below; oldest blocked for > 2754.288592 secs 2024-03-28T18:49:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:06 smithi151 ceph-mon[36481]: slow request 184.100451 seconds old, received at 2024-03-28T18:46:01.748685+0000: client_request(client.26723:1 getattr pAsLsXsFs #0x1 2024-03-28T18:46:01.745025+0000 caller_uid=0, caller_gid=0{}) currently joining batch getattr 2024-03-28T18:49:06.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:06 smithi045 ceph-mon[31773]: pgmap v1924: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:06.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:06 smithi045 ceph-mon[31773]: 5 slow requests, 1 included below; oldest blocked for > 2754.288592 secs 2024-03-28T18:49:06.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:06 smithi045 ceph-mon[31773]: slow request 184.100451 seconds old, received at 2024-03-28T18:46:01.748685+0000: client_request(client.26723:1 getattr pAsLsXsFs #0x1 2024-03-28T18:46:01.745025+0000 caller_uid=0, caller_gid=0{}) currently joining batch getattr 2024-03-28T18:49:07.324 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3399 sec 2024-03-28T18:49:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2045.713123 secs 2024-03-28T18:49:07.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2045.713123 secs 2024-03-28T18:49:07.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2045.713123 secs 2024-03-28T18:49:08.324 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3400 sec 2024-03-28T18:49:08.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:08 smithi071 ceph-mon[36597]: pgmap v1925: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:08 smithi151 ceph-mon[36481]: pgmap v1925: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:08.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:08 smithi045 ceph-mon[31773]: pgmap v1925: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:09.324 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3401 sec 2024-03-28T18:49:10.324 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3402 sec 2024-03-28T18:49:10.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:10 smithi071 ceph-mon[36597]: pgmap v1926: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:10.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:10 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] up:active 2024-03-28T18:49:10.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:10 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T18:49:10.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:10 smithi151 ceph-mon[36481]: pgmap v1926: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:10.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:10 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] up:active 2024-03-28T18:49:10.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:10 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T18:49:10.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:10 smithi045 ceph-mon[31773]: pgmap v1926: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:10.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:10 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6838/2709938800,v1:172.21.15.71:6839/2709938800] up:active 2024-03-28T18:49:10.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:10 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T18:49:11.324 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3403 sec 2024-03-28T18:49:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2759.288650 secs 2024-03-28T18:49:11.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2759.288650 secs 2024-03-28T18:49:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2759.288650 secs 2024-03-28T18:49:12.325 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3404 sec 2024-03-28T18:49:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:12 smithi071 ceph-mon[36597]: pgmap v1927: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2050.713214 secs 2024-03-28T18:49:12.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:12 smithi151 ceph-mon[36481]: pgmap v1927: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:12.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2050.713214 secs 2024-03-28T18:49:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:12 smithi045 ceph-mon[31773]: pgmap v1927: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2050.713214 secs 2024-03-28T18:49:13.325 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3405 sec 2024-03-28T18:49:13.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:13 smithi071 ceph-mon[36597]: pgmap v1928: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:13.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:13 smithi151 ceph-mon[36481]: pgmap v1928: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:13.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:13 smithi045 ceph-mon[31773]: pgmap v1928: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:14.325 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3406 sec 2024-03-28T18:49:15.325 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3407 sec 2024-03-28T18:49:16.325 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3408 sec 2024-03-28T18:49:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:16 smithi045 ceph-mon[31773]: pgmap v1929: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:16.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2764.288711 secs 2024-03-28T18:49:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:16 smithi071 ceph-mon[36597]: pgmap v1929: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2764.288711 secs 2024-03-28T18:49:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:16 smithi151 ceph-mon[36481]: pgmap v1929: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2764.288711 secs 2024-03-28T18:49:17.325 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3409 sec 2024-03-28T18:49:17.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2055.713304 secs 2024-03-28T18:49:17.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2055.713304 secs 2024-03-28T18:49:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2055.713304 secs 2024-03-28T18:49:18.325 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3410 sec 2024-03-28T18:49:18.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:18 smithi045 ceph-mon[31773]: pgmap v1930: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:18.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:18 smithi071 ceph-mon[36597]: pgmap v1930: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:18.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:18 smithi151 ceph-mon[36481]: pgmap v1930: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:19.326 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3411 sec 2024-03-28T18:49:20.326 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3412 sec 2024-03-28T18:49:20.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:20 smithi045 ceph-mon[31773]: pgmap v1931: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:20.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:20 smithi071 ceph-mon[36597]: pgmap v1931: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:20 smithi151 ceph-mon[36481]: pgmap v1931: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:21.326 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3413 sec 2024-03-28T18:49:21.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2769.288800 secs 2024-03-28T18:49:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2769.288800 secs 2024-03-28T18:49:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2769.288800 secs 2024-03-28T18:49:22.326 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3414 sec 2024-03-28T18:49:22.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:22 smithi045 ceph-mon[31773]: pgmap v1932: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:22.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2060.713395 secs 2024-03-28T18:49:22.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:22 smithi071 ceph-mon[36597]: pgmap v1932: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:22.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2060.713395 secs 2024-03-28T18:49:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:22 smithi151 ceph-mon[36481]: pgmap v1932: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2060.713395 secs 2024-03-28T18:49:23.326 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3415 sec 2024-03-28T18:49:24.326 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3416 sec 2024-03-28T18:49:24.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:24 smithi045 ceph-mon[31773]: pgmap v1933: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:24.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:24 smithi071 ceph-mon[36597]: pgmap v1933: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:24.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:24 smithi151 ceph-mon[36481]: pgmap v1933: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:25.327 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3417 sec 2024-03-28T18:49:26.327 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3418 sec 2024-03-28T18:49:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:26 smithi045 ceph-mon[31773]: pgmap v1934: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:26.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2774.288860 secs 2024-03-28T18:49:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:26 smithi071 ceph-mon[36597]: pgmap v1934: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:26.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2774.288860 secs 2024-03-28T18:49:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:26 smithi151 ceph-mon[36481]: pgmap v1934: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2774.288860 secs 2024-03-28T18:49:27.327 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3419 sec 2024-03-28T18:49:27.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2065.713479 secs 2024-03-28T18:49:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2065.713479 secs 2024-03-28T18:49:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2065.713479 secs 2024-03-28T18:49:28.327 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3420 sec 2024-03-28T18:49:28.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:28 smithi071 ceph-mon[36597]: pgmap v1935: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:28.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:28 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:49:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:28 smithi151 ceph-mon[36481]: pgmap v1935: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:28 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:49:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:28 smithi045 ceph-mon[31773]: pgmap v1935: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:28.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:28 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:49:29.327 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3421 sec 2024-03-28T18:49:30.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:29 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:49:30.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:29 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:49:30.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:29 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:49:30.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:29 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:49:30.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:29 smithi151 ceph-mon[36481]: pgmap v1936: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:30.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:29 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:49:30.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:29 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:49:30.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:29 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:49:30.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:29 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:49:30.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:29 smithi045 ceph-mon[31773]: pgmap v1936: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:30.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:29 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:49:30.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:29 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:49:30.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:29 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:49:30.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:29 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:49:30.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:29 smithi071 ceph-mon[36597]: pgmap v1936: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:30.327 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3422 sec 2024-03-28T18:49:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:30 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:49:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:30 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:49:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:30 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:49:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:30 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:49:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:30 smithi045 ceph-mon[31773]: pgmap v1937: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:49:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:30 smithi045 ceph-mon[31773]: pgmap v1938: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:49:31.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:30 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:49:31.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:30 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:49:31.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:30 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:49:31.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:30 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:49:31.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:30 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:49:31.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:30 smithi071 ceph-mon[36597]: pgmap v1937: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:49:31.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:30 smithi071 ceph-mon[36597]: pgmap v1938: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:49:31.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:30 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:49:31.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:30 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:49:31.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:30 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:49:31.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:30 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:49:31.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:30 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:49:31.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:30 smithi151 ceph-mon[36481]: pgmap v1937: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:49:31.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:30 smithi151 ceph-mon[36481]: pgmap v1938: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:49:31.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:30 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:49:31.327 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3423 sec 2024-03-28T18:49:32.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2779.288959 secs 2024-03-28T18:49:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2779.288959 secs 2024-03-28T18:49:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2779.288959 secs 2024-03-28T18:49:32.327 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3424 sec 2024-03-28T18:49:33.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2070.713567 secs 2024-03-28T18:49:33.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:32 smithi045 ceph-mon[31773]: pgmap v1939: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:49:33.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2070.713567 secs 2024-03-28T18:49:33.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:32 smithi071 ceph-mon[36597]: pgmap v1939: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:49:33.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2070.713567 secs 2024-03-28T18:49:33.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:32 smithi151 ceph-mon[36481]: pgmap v1939: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T18:49:33.327 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3425 sec 2024-03-28T18:49:34.328 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3426 sec 2024-03-28T18:49:35.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:34 smithi045 ceph-mon[31773]: pgmap v1940: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:49:35.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:34 smithi071 ceph-mon[36597]: pgmap v1940: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:49:35.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:34 smithi151 ceph-mon[36481]: pgmap v1940: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T18:49:35.328 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3427 sec 2024-03-28T18:49:36.328 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3428 sec 2024-03-28T18:49:37.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2784.289015 secs 2024-03-28T18:49:37.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:36 smithi045 ceph-mon[31773]: pgmap v1941: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:49:37.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2784.289015 secs 2024-03-28T18:49:37.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:36 smithi071 ceph-mon[36597]: pgmap v1941: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:49:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2784.289015 secs 2024-03-28T18:49:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:36 smithi151 ceph-mon[36481]: pgmap v1941: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:49:37.328 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3429 sec 2024-03-28T18:49:38.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2075.713669 secs 2024-03-28T18:49:38.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2075.713669 secs 2024-03-28T18:49:38.328 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3430 sec 2024-03-28T18:49:38.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2075.713669 secs 2024-03-28T18:49:39.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:38 smithi071 ceph-mon[36597]: pgmap v1942: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:49:39.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:38 smithi151 ceph-mon[36481]: pgmap v1942: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:49:39.328 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3431 sec 2024-03-28T18:49:39.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:38 smithi045 ceph-mon[31773]: pgmap v1942: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:49:40.329 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3432 sec 2024-03-28T18:49:41.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:40 smithi071 ceph-mon[36597]: pgmap v1943: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:49:41.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:40 smithi151 ceph-mon[36481]: pgmap v1943: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:49:41.329 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3433 sec 2024-03-28T18:49:41.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:40 smithi045 ceph-mon[31773]: pgmap v1943: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T18:49:42.224 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2789.289103 secs 2024-03-28T18:49:42.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2789.289103 secs 2024-03-28T18:49:42.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2789.289103 secs 2024-03-28T18:49:42.329 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3434 sec 2024-03-28T18:49:43.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2080.713759 secs 2024-03-28T18:49:43.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:42 smithi071 ceph-mon[36597]: pgmap v1944: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:49:43.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2080.713759 secs 2024-03-28T18:49:43.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:42 smithi151 ceph-mon[36481]: pgmap v1944: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:49:43.329 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3435 sec 2024-03-28T18:49:43.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2080.713759 secs 2024-03-28T18:49:43.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:42 smithi045 ceph-mon[31773]: pgmap v1944: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:49:44.329 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3436 sec 2024-03-28T18:49:45.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:44 smithi071 ceph-mon[36597]: pgmap v1945: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:49:45.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:44 smithi151 ceph-mon[36481]: pgmap v1945: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:49:45.329 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3437 sec 2024-03-28T18:49:45.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:44 smithi045 ceph-mon[31773]: pgmap v1945: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:49:46.330 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3438 sec 2024-03-28T18:49:47.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2794.289190 secs 2024-03-28T18:49:47.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:46 smithi071 ceph-mon[36597]: pgmap v1946: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:49:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2794.289190 secs 2024-03-28T18:49:47.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:46 smithi151 ceph-mon[36481]: pgmap v1946: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:49:47.330 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3439 sec 2024-03-28T18:49:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2794.289190 secs 2024-03-28T18:49:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:46 smithi045 ceph-mon[31773]: pgmap v1946: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:49:48.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2085.713820 secs 2024-03-28T18:49:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2085.713820 secs 2024-03-28T18:49:48.330 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3440 sec 2024-03-28T18:49:48.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2085.713820 secs 2024-03-28T18:49:49.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:48 smithi071 ceph-mon[36597]: pgmap v1947: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:49.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:48 smithi151 ceph-mon[36481]: pgmap v1947: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:49.330 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3441 sec 2024-03-28T18:49:49.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:48 smithi045 ceph-mon[31773]: pgmap v1947: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:50.330 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3442 sec 2024-03-28T18:49:51.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:50 smithi071 ceph-mon[36597]: pgmap v1948: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:51.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:50 smithi151 ceph-mon[36481]: pgmap v1948: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:51.330 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3443 sec 2024-03-28T18:49:51.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:50 smithi045 ceph-mon[31773]: pgmap v1948: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:52.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2799.289278 secs 2024-03-28T18:49:52.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:51 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2090.713906 secs 2024-03-28T18:49:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2799.289278 secs 2024-03-28T18:49:52.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:51 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2090.713906 secs 2024-03-28T18:49:52.330 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3444 sec 2024-03-28T18:49:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2799.289278 secs 2024-03-28T18:49:52.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:51 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2090.713906 secs 2024-03-28T18:49:53.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:52 smithi071 ceph-mon[36597]: pgmap v1949: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:53.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:52 smithi151 ceph-mon[36481]: pgmap v1949: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:53.330 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3445 sec 2024-03-28T18:49:53.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:52 smithi045 ceph-mon[31773]: pgmap v1949: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:54.331 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3446 sec 2024-03-28T18:49:55.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:54 smithi071 ceph-mon[36597]: pgmap v1950: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:55.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:54 smithi151 ceph-mon[36481]: pgmap v1950: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:55.330 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3447 sec 2024-03-28T18:49:55.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:54 smithi045 ceph-mon[31773]: pgmap v1950: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:56.331 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3448 sec 2024-03-28T18:49:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2804.289336 secs 2024-03-28T18:49:57.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:56 smithi071 ceph-mon[36597]: pgmap v1951: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:57.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:56 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2095.713989 secs 2024-03-28T18:49:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2804.289336 secs 2024-03-28T18:49:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:56 smithi151 ceph-mon[36481]: pgmap v1951: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:57.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:56 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2095.713989 secs 2024-03-28T18:49:57.331 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3449 sec 2024-03-28T18:49:57.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2804.289336 secs 2024-03-28T18:49:57.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:56 smithi045 ceph-mon[31773]: pgmap v1951: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:49:57.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:56 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2095.713989 secs 2024-03-28T18:49:58.331 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3450 sec 2024-03-28T18:49:59.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:49:58 smithi071 ceph-mon[36597]: pgmap v1952: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:59.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:49:58 smithi151 ceph-mon[36481]: pgmap v1952: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:49:59.331 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3451 sec 2024-03-28T18:49:59.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:49:58 smithi045 ceph-mon[31773]: pgmap v1952: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:00.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:00 smithi071 ceph-mon[36597]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-03-28T18:50:00.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:00 smithi151 ceph-mon[36481]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-03-28T18:50:00.331 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3452 sec 2024-03-28T18:50:00.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:00 smithi045 ceph-mon[31773]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-03-28T18:50:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:01 smithi071 ceph-mon[36597]: pgmap v1953: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:01.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:01 smithi151 ceph-mon[36481]: pgmap v1953: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:01.331 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3453 sec 2024-03-28T18:50:01.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:01 smithi045 ceph-mon[31773]: pgmap v1953: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:02 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2809.289374 secs 2024-03-28T18:50:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2100.714074 secs 2024-03-28T18:50:02.331 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3454 sec 2024-03-28T18:50:02.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:02 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2809.289374 secs 2024-03-28T18:50:02.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2100.714074 secs 2024-03-28T18:50:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:02 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2809.289374 secs 2024-03-28T18:50:02.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2100.714074 secs 2024-03-28T18:50:03.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:03 smithi151 ceph-mon[36481]: pgmap v1954: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:03.333 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3455 sec 2024-03-28T18:50:03.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:03 smithi045 ceph-mon[31773]: pgmap v1954: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:03.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:03 smithi071 ceph-mon[36597]: pgmap v1954: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:04.332 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3456 sec 2024-03-28T18:50:05.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:05 smithi151 ceph-mon[36481]: pgmap v1955: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:05.332 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3457 sec 2024-03-28T18:50:05.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:05 smithi045 ceph-mon[31773]: pgmap v1955: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:05.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:05 smithi071 ceph-mon[36597]: pgmap v1955: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:06.333 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3458 sec 2024-03-28T18:50:07.332 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3459 sec 2024-03-28T18:50:07.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:07 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2814.289437 secs 2024-03-28T18:50:07.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:07 smithi045 ceph-mon[31773]: pgmap v1956: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:50:07.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2105.714158 secs 2024-03-28T18:50:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:07 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2814.289437 secs 2024-03-28T18:50:07.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:07 smithi071 ceph-mon[36597]: pgmap v1956: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:50:07.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2105.714158 secs 2024-03-28T18:50:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:07 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2814.289437 secs 2024-03-28T18:50:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:07 smithi151 ceph-mon[36481]: pgmap v1956: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:50:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2105.714158 secs 2024-03-28T18:50:08.332 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3460 sec 2024-03-28T18:50:09.333 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3461 sec 2024-03-28T18:50:09.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:09 smithi045 ceph-mon[31773]: pgmap v1957: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:50:09.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:09 smithi071 ceph-mon[36597]: pgmap v1957: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:50:09.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:09 smithi151 ceph-mon[36481]: pgmap v1957: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:50:10.333 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3462 sec 2024-03-28T18:50:11.333 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3463 sec 2024-03-28T18:50:11.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:11 smithi045 ceph-mon[31773]: pgmap v1958: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:50:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:11 smithi071 ceph-mon[36597]: pgmap v1958: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:50:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:11 smithi151 ceph-mon[36481]: pgmap v1958: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:50:12.333 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3464 sec 2024-03-28T18:50:12.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:12 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2819.289487 secs 2024-03-28T18:50:12.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2110.714247 secs 2024-03-28T18:50:12.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:12 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2819.289487 secs 2024-03-28T18:50:12.475 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2110.714247 secs 2024-03-28T18:50:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:12 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2819.289487 secs 2024-03-28T18:50:12.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2110.714247 secs 2024-03-28T18:50:13.333 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3465 sec 2024-03-28T18:50:13.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:13 smithi045 ceph-mon[31773]: pgmap v1959: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:50:13.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:13 smithi071 ceph-mon[36597]: pgmap v1959: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:50:13.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:13 smithi151 ceph-mon[36481]: pgmap v1959: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:50:14.333 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3466 sec 2024-03-28T18:50:15.333 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3467 sec 2024-03-28T18:50:15.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:15 smithi045 ceph-mon[31773]: pgmap v1960: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:50:15.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:15 smithi071 ceph-mon[36597]: pgmap v1960: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:50:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:15 smithi151 ceph-mon[36481]: pgmap v1960: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-28T18:50:16.333 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3468 sec 2024-03-28T18:50:17.333 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3469 sec 2024-03-28T18:50:17.362 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:17 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2824.289570 secs 2024-03-28T18:50:17.363 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:17 smithi071 ceph-mon[36597]: pgmap v1961: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:50:17.363 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2115.714335 secs 2024-03-28T18:50:17.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:17 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2824.289570 secs 2024-03-28T18:50:17.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:17 smithi045 ceph-mon[31773]: pgmap v1961: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:50:17.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2115.714335 secs 2024-03-28T18:50:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:17 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2824.289570 secs 2024-03-28T18:50:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:17 smithi151 ceph-mon[36481]: pgmap v1961: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T18:50:17.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2115.714335 secs 2024-03-28T18:50:18.334 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3470 sec 2024-03-28T18:50:19.334 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3471 sec 2024-03-28T18:50:19.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:19 smithi045 ceph-mon[31773]: pgmap v1962: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:19.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:19 smithi071 ceph-mon[36597]: pgmap v1962: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:19.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:19 smithi151 ceph-mon[36481]: pgmap v1962: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:20.334 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3472 sec 2024-03-28T18:50:21.334 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3473 sec 2024-03-28T18:50:21.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:21 smithi045 ceph-mon[31773]: pgmap v1963: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:21 smithi071 ceph-mon[36597]: pgmap v1963: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:21 smithi151 ceph-mon[36481]: pgmap v1963: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:22.334 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3474 sec 2024-03-28T18:50:22.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:22 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2829.289655 secs 2024-03-28T18:50:22.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2120.714424 secs 2024-03-28T18:50:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:22 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2829.289655 secs 2024-03-28T18:50:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2120.714424 secs 2024-03-28T18:50:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:22 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2829.289655 secs 2024-03-28T18:50:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2120.714424 secs 2024-03-28T18:50:23.335 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3475 sec 2024-03-28T18:50:23.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:23 smithi045 ceph-mon[31773]: pgmap v1964: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:23.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:23 smithi071 ceph-mon[36597]: pgmap v1964: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:23.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:23 smithi151 ceph-mon[36481]: pgmap v1964: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:24.335 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3476 sec 2024-03-28T18:50:25.335 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3477 sec 2024-03-28T18:50:25.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:25 smithi045 ceph-mon[31773]: pgmap v1965: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:25.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:25 smithi071 ceph-mon[36597]: pgmap v1965: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:25.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:25 smithi151 ceph-mon[36481]: pgmap v1965: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:26.335 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3478 sec 2024-03-28T18:50:27.335 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3479 sec 2024-03-28T18:50:27.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:27 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2834.289714 secs 2024-03-28T18:50:27.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:27 smithi045 ceph-mon[31773]: pgmap v1966: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:27.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2125.714512 secs 2024-03-28T18:50:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:27 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2834.289714 secs 2024-03-28T18:50:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:27 smithi071 ceph-mon[36597]: pgmap v1966: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2125.714512 secs 2024-03-28T18:50:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:27 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2834.289714 secs 2024-03-28T18:50:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:27 smithi151 ceph-mon[36481]: pgmap v1966: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2125.714512 secs 2024-03-28T18:50:28.335 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3480 sec 2024-03-28T18:50:29.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:29 smithi045 ceph-mon[31773]: pgmap v1967: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:29.335 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3481 sec 2024-03-28T18:50:29.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:29 smithi071 ceph-mon[36597]: pgmap v1967: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:29.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:29 smithi151 ceph-mon[36481]: pgmap v1967: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:30.335 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3482 sec 2024-03-28T18:50:30.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:30 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:50:30.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:30 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:50:30.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:30 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:50:31.335 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3483 sec 2024-03-28T18:50:31.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:31 smithi045 ceph-mon[31773]: pgmap v1968: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:31.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2839.289776 secs 2024-03-28T18:50:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:31 smithi071 ceph-mon[36597]: pgmap v1968: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2839.289776 secs 2024-03-28T18:50:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:31 smithi151 ceph-mon[36481]: pgmap v1968: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2839.289776 secs 2024-03-28T18:50:32.336 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3484 sec 2024-03-28T18:50:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:32 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:50:32.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:32 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:50:32.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:32 smithi071 ceph-mon[36597]: pgmap v1969: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:50:32.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:32 smithi071 ceph-mon[36597]: pgmap v1970: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:32.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:32 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:50:32.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2130.714582 secs 2024-03-28T18:50:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:32 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:50:32.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:32 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:50:32.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:32 smithi151 ceph-mon[36481]: pgmap v1969: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:50:32.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:32 smithi151 ceph-mon[36481]: pgmap v1970: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:32.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:32 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:50:32.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2130.714582 secs 2024-03-28T18:50:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:32 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:50:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:32 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:50:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:32 smithi045 ceph-mon[31773]: pgmap v1969: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:50:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:32 smithi045 ceph-mon[31773]: pgmap v1970: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:32 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:50:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2130.714582 secs 2024-03-28T18:50:33.336 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3485 sec 2024-03-28T18:50:33.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:33 smithi045 ceph-mon[31773]: pgmap v1971: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:34.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:33 smithi071 ceph-mon[36597]: pgmap v1971: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:34.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:33 smithi151 ceph-mon[36481]: pgmap v1971: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:34.336 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3486 sec 2024-03-28T18:50:35.336 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3487 sec 2024-03-28T18:50:35.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:35 smithi045 ceph-mon[31773]: pgmap v1972: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:36.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:35 smithi071 ceph-mon[36597]: pgmap v1972: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:36.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:35 smithi151 ceph-mon[36481]: pgmap v1972: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:36.336 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3488 sec 2024-03-28T18:50:36.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2844.289864 secs 2024-03-28T18:50:37.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2844.289864 secs 2024-03-28T18:50:37.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2844.289864 secs 2024-03-28T18:50:37.336 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3489 sec 2024-03-28T18:50:37.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2135.714642 secs 2024-03-28T18:50:37.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:37 smithi045 ceph-mon[31773]: pgmap v1973: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:38.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2135.714642 secs 2024-03-28T18:50:38.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:37 smithi071 ceph-mon[36597]: pgmap v1973: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:38.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2135.714642 secs 2024-03-28T18:50:38.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:37 smithi151 ceph-mon[36481]: pgmap v1973: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:38.336 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3490 sec 2024-03-28T18:50:39.337 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3491 sec 2024-03-28T18:50:39.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:39 smithi045 ceph-mon[31773]: pgmap v1974: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:40.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:39 smithi071 ceph-mon[36597]: pgmap v1974: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:40.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:39 smithi151 ceph-mon[36481]: pgmap v1974: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:50:40.337 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3492 sec 2024-03-28T18:50:41.337 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3493 sec 2024-03-28T18:50:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2849.289953 secs 2024-03-28T18:50:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2849.289953 secs 2024-03-28T18:50:41.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2849.289953 secs 2024-03-28T18:50:42.337 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3494 sec 2024-03-28T18:50:42.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:42 smithi151 ceph-mon[36481]: pgmap v1975: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:50:42.475 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2140.714701 secs 2024-03-28T18:50:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:42 smithi071 ceph-mon[36597]: pgmap v1975: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:50:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2140.714701 secs 2024-03-28T18:50:42.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:42 smithi045 ceph-mon[31773]: pgmap v1975: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:50:42.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2140.714701 secs 2024-03-28T18:50:43.337 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3495 sec 2024-03-28T18:50:43.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:43 smithi045 ceph-mon[31773]: pgmap v1976: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:44.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:43 smithi071 ceph-mon[36597]: pgmap v1976: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:44.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:43 smithi151 ceph-mon[36481]: pgmap v1976: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:44.337 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3496 sec 2024-03-28T18:50:45.337 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3497 sec 2024-03-28T18:50:45.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:45 smithi045 ceph-mon[31773]: pgmap v1977: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:46.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:45 smithi071 ceph-mon[36597]: pgmap v1977: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:46.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:45 smithi151 ceph-mon[36481]: pgmap v1977: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:46.338 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3498 sec 2024-03-28T18:50:46.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2854.290038 secs 2024-03-28T18:50:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2854.290038 secs 2024-03-28T18:50:47.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2854.290038 secs 2024-03-28T18:50:47.338 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3499 sec 2024-03-28T18:50:47.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2145.714764 secs 2024-03-28T18:50:47.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:47 smithi045 ceph-mon[31773]: pgmap v1978: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:47.961 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2145.714764 secs 2024-03-28T18:50:47.962 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:47 smithi071 ceph-mon[36597]: pgmap v1978: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2145.714764 secs 2024-03-28T18:50:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:47 smithi151 ceph-mon[36481]: pgmap v1978: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:48.338 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3500 sec 2024-03-28T18:50:49.338 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3501 sec 2024-03-28T18:50:49.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:49 smithi045 ceph-mon[31773]: pgmap v1979: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:50.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:49 smithi071 ceph-mon[36597]: pgmap v1979: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:50.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:49 smithi151 ceph-mon[36481]: pgmap v1979: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:50.338 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3502 sec 2024-03-28T18:50:51.338 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3503 sec 2024-03-28T18:50:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2859.290097 secs 2024-03-28T18:50:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2859.290097 secs 2024-03-28T18:50:51.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2859.290097 secs 2024-03-28T18:50:52.338 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3504 sec 2024-03-28T18:50:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:52 smithi071 ceph-mon[36597]: pgmap v1980: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:52.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2150.714825 secs 2024-03-28T18:50:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:52 smithi151 ceph-mon[36481]: pgmap v1980: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2150.714825 secs 2024-03-28T18:50:52.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:52 smithi045 ceph-mon[31773]: pgmap v1980: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:52.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2150.714825 secs 2024-03-28T18:50:53.338 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3505 sec 2024-03-28T18:50:53.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:53 smithi045 ceph-mon[31773]: pgmap v1981: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:54.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:53 smithi071 ceph-mon[36597]: pgmap v1981: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:54.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:53 smithi151 ceph-mon[36481]: pgmap v1981: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:54.339 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3506 sec 2024-03-28T18:50:55.339 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3507 sec 2024-03-28T18:50:55.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:55 smithi045 ceph-mon[31773]: pgmap v1982: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:56.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:55 smithi071 ceph-mon[36597]: pgmap v1982: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:56.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:55 smithi151 ceph-mon[36481]: pgmap v1982: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:50:56.339 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3508 sec 2024-03-28T18:50:56.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2864.290182 secs 2024-03-28T18:50:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2864.290182 secs 2024-03-28T18:50:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2864.290182 secs 2024-03-28T18:50:57.339 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3509 sec 2024-03-28T18:50:57.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2155.714922 secs 2024-03-28T18:50:57.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:57 smithi045 ceph-mon[31773]: pgmap v1983: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2155.714922 secs 2024-03-28T18:50:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:57 smithi071 ceph-mon[36597]: pgmap v1983: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:58.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2155.714922 secs 2024-03-28T18:50:58.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:57 smithi151 ceph-mon[36481]: pgmap v1983: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:50:58.339 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3510 sec 2024-03-28T18:50:59.340 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3511 sec 2024-03-28T18:50:59.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:50:59 smithi045 ceph-mon[31773]: pgmap v1984: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:00.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:50:59 smithi071 ceph-mon[36597]: pgmap v1984: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:00.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:50:59 smithi151 ceph-mon[36481]: pgmap v1984: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:00.339 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3512 sec 2024-03-28T18:51:01.340 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3513 sec 2024-03-28T18:51:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2869.290267 secs 2024-03-28T18:51:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2869.290267 secs 2024-03-28T18:51:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2869.290267 secs 2024-03-28T18:51:02.340 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3514 sec 2024-03-28T18:51:02.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:02 smithi045 ceph-mon[31773]: pgmap v1985: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:02.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2160.715009 secs 2024-03-28T18:51:02.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:51:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:51:01.983+0000 7f52eec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:51:02.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:51:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:51:01.983+0000 7f52eec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:51:02.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:51:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:51:01.983+0000 7f52eec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:51:02.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:51:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:51:01.983+0000 7f52eec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:51:02.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:51:01 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:51:01.983+0000 7f52eec00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:51:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:02 smithi071 ceph-mon[36597]: pgmap v1985: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2160.715009 secs 2024-03-28T18:51:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:02 smithi151 ceph-mon[36481]: pgmap v1985: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2160.715009 secs 2024-03-28T18:51:03.340 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3515 sec 2024-03-28T18:51:03.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:03 smithi045 ceph-mon[31773]: pgmap v1986: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:04.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:03 smithi071 ceph-mon[36597]: pgmap v1986: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:04.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:03 smithi151 ceph-mon[36481]: pgmap v1986: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:04.340 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3516 sec 2024-03-28T18:51:05.340 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3517 sec 2024-03-28T18:51:05.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:05 smithi045 ceph-mon[31773]: pgmap v1987: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 255 B/s wr, 16 op/s 2024-03-28T18:51:06.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:05 smithi071 ceph-mon[36597]: pgmap v1987: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 255 B/s wr, 16 op/s 2024-03-28T18:51:06.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:05 smithi151 ceph-mon[36481]: pgmap v1987: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 255 B/s wr, 16 op/s 2024-03-28T18:51:06.340 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3518 sec 2024-03-28T18:51:06.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2874.290356 secs 2024-03-28T18:51:07.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2874.290356 secs 2024-03-28T18:51:07.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2874.290356 secs 2024-03-28T18:51:07.341 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3519 sec 2024-03-28T18:51:07.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2165.715095 secs 2024-03-28T18:51:07.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:07 smithi045 ceph-mon[31773]: pgmap v1988: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-28T18:51:08.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2165.715095 secs 2024-03-28T18:51:08.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:07 smithi071 ceph-mon[36597]: pgmap v1988: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-28T18:51:08.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2165.715095 secs 2024-03-28T18:51:08.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:07 smithi151 ceph-mon[36481]: pgmap v1988: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-28T18:51:08.341 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3520 sec 2024-03-28T18:51:09.341 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3521 sec 2024-03-28T18:51:09.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:09 smithi045 ceph-mon[31773]: pgmap v1989: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-28T18:51:10.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:09 smithi071 ceph-mon[36597]: pgmap v1989: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-28T18:51:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:09 smithi151 ceph-mon[36481]: pgmap v1989: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-28T18:51:10.341 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3522 sec 2024-03-28T18:51:11.341 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3523 sec 2024-03-28T18:51:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2879.290416 secs 2024-03-28T18:51:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2879.290416 secs 2024-03-28T18:51:11.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2879.290416 secs 2024-03-28T18:51:12.341 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3524 sec 2024-03-28T18:51:12.475 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:12 smithi151 ceph-mon[36481]: pgmap v1990: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 255 B/s wr, 16 op/s 2024-03-28T18:51:12.475 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2170.715205 secs 2024-03-28T18:51:12.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:12 smithi071 ceph-mon[36597]: pgmap v1990: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 255 B/s wr, 16 op/s 2024-03-28T18:51:12.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2170.715205 secs 2024-03-28T18:51:12.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:12 smithi045 ceph-mon[31773]: pgmap v1990: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 255 B/s wr, 16 op/s 2024-03-28T18:51:12.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2170.715205 secs 2024-03-28T18:51:13.341 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3525 sec 2024-03-28T18:51:13.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:13 smithi045 ceph-mon[31773]: pgmap v1991: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-28T18:51:14.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:13 smithi071 ceph-mon[36597]: pgmap v1991: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-28T18:51:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:13 smithi151 ceph-mon[36481]: pgmap v1991: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-28T18:51:14.341 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3526 sec 2024-03-28T18:51:15.342 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3527 sec 2024-03-28T18:51:15.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:15 smithi045 ceph-mon[31773]: pgmap v1992: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 255 B/s wr, 16 op/s 2024-03-28T18:51:16.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:15 smithi071 ceph-mon[36597]: pgmap v1992: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 255 B/s wr, 16 op/s 2024-03-28T18:51:16.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:15 smithi151 ceph-mon[36481]: pgmap v1992: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 255 B/s wr, 16 op/s 2024-03-28T18:51:16.342 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3528 sec 2024-03-28T18:51:16.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:16 smithi045 ceph-mon[31773]: 5 slow requests, 2 included below; oldest blocked for > 2884.290476 secs 2024-03-28T18:51:16.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:16 smithi045 ceph-mon[31773]: slow request 2884.290475 seconds old, received at 2024-03-28T18:03:11.560545+0000: peer_request:client.15069:1629 currently acquired locks 2024-03-28T18:51:16.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:16 smithi045 ceph-mon[31773]: slow request 2884.194350 seconds old, received at 2024-03-28T18:03:11.656670+0000: client_request(client.25691:164 rmsnap #0x1//scheduled-2024-03-28-17_57_00_UTC 2024-03-28T18:03:11.656797+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:51:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:16 smithi071 ceph-mon[36597]: 5 slow requests, 2 included below; oldest blocked for > 2884.290476 secs 2024-03-28T18:51:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:16 smithi071 ceph-mon[36597]: slow request 2884.290475 seconds old, received at 2024-03-28T18:03:11.560545+0000: peer_request:client.15069:1629 currently acquired locks 2024-03-28T18:51:17.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:16 smithi071 ceph-mon[36597]: slow request 2884.194350 seconds old, received at 2024-03-28T18:03:11.656670+0000: client_request(client.25691:164 rmsnap #0x1//scheduled-2024-03-28-17_57_00_UTC 2024-03-28T18:03:11.656797+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:51:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:16 smithi151 ceph-mon[36481]: 5 slow requests, 2 included below; oldest blocked for > 2884.290476 secs 2024-03-28T18:51:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:16 smithi151 ceph-mon[36481]: slow request 2884.290475 seconds old, received at 2024-03-28T18:03:11.560545+0000: peer_request:client.15069:1629 currently acquired locks 2024-03-28T18:51:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:16 smithi151 ceph-mon[36481]: slow request 2884.194350 seconds old, received at 2024-03-28T18:03:11.656670+0000: client_request(client.25691:164 rmsnap #0x1//scheduled-2024-03-28-17_57_00_UTC 2024-03-28T18:03:11.656797+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-28T18:51:17.342 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3529 sec 2024-03-28T18:51:17.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2175.715290 secs 2024-03-28T18:51:17.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:17 smithi045 ceph-mon[31773]: pgmap v1993: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:17.961 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2175.715290 secs 2024-03-28T18:51:17.961 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:17 smithi071 ceph-mon[36597]: pgmap v1993: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2175.715290 secs 2024-03-28T18:51:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:17 smithi151 ceph-mon[36481]: pgmap v1993: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:18.342 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3530 sec 2024-03-28T18:51:19.342 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3531 sec 2024-03-28T18:51:19.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:19 smithi045 ceph-mon[31773]: pgmap v1994: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:20.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:19 smithi071 ceph-mon[36597]: pgmap v1994: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:20.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:19 smithi151 ceph-mon[36481]: pgmap v1994: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:20.342 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3532 sec 2024-03-28T18:51:21.342 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3533 sec 2024-03-28T18:51:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2889.290536 secs 2024-03-28T18:51:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2889.290536 secs 2024-03-28T18:51:21.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2889.290536 secs 2024-03-28T18:51:22.343 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3534 sec 2024-03-28T18:51:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:22 smithi071 ceph-mon[36597]: pgmap v1995: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2180.715352 secs 2024-03-28T18:51:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:22 smithi151 ceph-mon[36481]: pgmap v1995: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2180.715352 secs 2024-03-28T18:51:22.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:22 smithi045 ceph-mon[31773]: pgmap v1995: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:22.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2180.715352 secs 2024-03-28T18:51:23.343 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3535 sec 2024-03-28T18:51:23.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:23 smithi045 ceph-mon[31773]: pgmap v1996: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:24.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:23 smithi071 ceph-mon[36597]: pgmap v1996: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:24.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:23 smithi151 ceph-mon[36481]: pgmap v1996: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:24.343 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3536 sec 2024-03-28T18:51:25.343 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3537 sec 2024-03-28T18:51:25.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:25 smithi045 ceph-mon[31773]: pgmap v1997: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:26.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:25 smithi071 ceph-mon[36597]: pgmap v1997: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:26.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:25 smithi151 ceph-mon[36481]: pgmap v1997: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:26.343 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3538 sec 2024-03-28T18:51:26.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2894.290598 secs 2024-03-28T18:51:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2894.290598 secs 2024-03-28T18:51:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2894.290598 secs 2024-03-28T18:51:27.343 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3539 sec 2024-03-28T18:51:27.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2185.715414 secs 2024-03-28T18:51:27.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:27 smithi045 ceph-mon[31773]: pgmap v1998: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2185.715414 secs 2024-03-28T18:51:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:27 smithi071 ceph-mon[36597]: pgmap v1998: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2185.715414 secs 2024-03-28T18:51:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:27 smithi151 ceph-mon[36481]: pgmap v1998: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:28.343 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3540 sec 2024-03-28T18:51:29.344 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3541 sec 2024-03-28T18:51:29.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:29 smithi045 ceph-mon[31773]: pgmap v1999: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:30.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:29 smithi071 ceph-mon[36597]: pgmap v1999: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:30.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:29 smithi151 ceph-mon[36481]: pgmap v1999: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:30.344 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3542 sec 2024-03-28T18:51:31.344 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3543 sec 2024-03-28T18:51:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2899.290660 secs 2024-03-28T18:51:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2899.290660 secs 2024-03-28T18:51:31.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2899.290660 secs 2024-03-28T18:51:32.344 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3544 sec 2024-03-28T18:51:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:32 smithi071 ceph-mon[36597]: pgmap v2000: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2190.715498 secs 2024-03-28T18:51:32.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:32 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:51:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:32 smithi151 ceph-mon[36481]: pgmap v2000: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2190.715498 secs 2024-03-28T18:51:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:32 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:51:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:32 smithi045 ceph-mon[31773]: pgmap v2000: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2190.715498 secs 2024-03-28T18:51:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:32 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:51:33.344 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3545 sec 2024-03-28T18:51:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:33 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:33 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:33 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:33 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:33 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:33 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:33.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:33 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:33.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:33 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:33.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:33 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:33.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:33 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:33.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:33 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:33.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:33 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:33.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:33 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:33.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:33 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:33.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:33 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:33.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:33 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:33.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:33 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:33.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:33 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:34.345 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3546 sec 2024-03-28T18:51:34.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:34 smithi045 ceph-mon[31773]: pgmap v2001: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:34.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:34 smithi071 ceph-mon[36597]: pgmap v2001: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:34.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:34 smithi151 ceph-mon[36481]: pgmap v2001: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:35.344 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3547 sec 2024-03-28T18:51:35.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:35 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:35.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:35 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:35.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:35 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:35.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:35 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:35.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:35 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:35 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:35 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:35 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:35 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:35 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:35 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:35 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:35 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:35 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:35 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:35 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:35 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:35 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:35 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:35 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:35 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:35.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:35 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:35.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:35 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:35.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:35 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:35.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:35 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:35 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:35 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:35 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:35 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:35 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:35 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:35 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:35 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:35 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:35 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:35 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:35 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:35 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:35 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:35 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:35 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:35.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:35 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:35.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:35 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:35.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:35 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:35.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:35 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:35 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:35 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:35 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:35 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:35 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:35 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:35 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:35 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:35 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:35 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:35 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:35 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:35 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:35 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:35.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:35 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:36.344 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3548 sec 2024-03-28T18:51:36.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:36 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T18:51:36.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:36 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T18:51:36.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:36 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T18:51:36.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:36 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T18:51:36.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:36 smithi071 ceph-mon[36597]: pgmap v2002: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:36.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2904.290723 secs 2024-03-28T18:51:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:36 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T18:51:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:36 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T18:51:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:36 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T18:51:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:36 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T18:51:36.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:36 smithi151 ceph-mon[36481]: pgmap v2002: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:36.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2904.290723 secs 2024-03-28T18:51:36.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:36 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T18:51:36.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:36 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T18:51:36.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:36 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T18:51:36.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:36 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T18:51:36.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:36 smithi045 ceph-mon[31773]: pgmap v2002: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:36.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2904.290723 secs 2024-03-28T18:51:37.344 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3549 sec 2024-03-28T18:51:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2195.715537 secs 2024-03-28T18:51:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:37 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:37 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2195.715537 secs 2024-03-28T18:51:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:37 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:37 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:37.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2195.715537 secs 2024-03-28T18:51:37.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:37 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:37.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:37 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:38.345 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3550 sec 2024-03-28T18:51:38.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:38 smithi071 ceph-mon[36597]: pgmap v2003: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:38.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:38 smithi151 ceph-mon[36481]: pgmap v2003: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:38.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:38 smithi045 ceph-mon[31773]: pgmap v2003: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:39.345 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3551 sec 2024-03-28T18:51:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:39 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:39 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:39 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:39 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:39 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:39 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:39 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:39 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:39 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:51:39.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:39 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:51:39.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:39 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:39.818 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:39 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:39 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:39 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:39 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:39 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:39 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:39 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:39 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:39 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:51:39.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:39 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:51:39.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:39 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:39 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:39 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:39 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:39 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:39 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:39 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:39 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:39 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:51:39.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:39 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:51:39.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:39 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:51:39.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:39 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:51:40.345 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3552 sec 2024-03-28T18:51:40.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:40 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T18:51:40.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:40 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:51:40.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:40 smithi071 ceph-mon[36597]: pgmap v2004: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T18:51:40.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:40 smithi071 ceph-mon[36597]: pgmap v2005: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:51:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:40 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T18:51:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:40 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:51:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:40 smithi151 ceph-mon[36481]: pgmap v2004: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T18:51:40.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:40 smithi151 ceph-mon[36481]: pgmap v2005: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:51:40.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:40 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T18:51:40.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:40 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T18:51:40.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:40 smithi045 ceph-mon[31773]: pgmap v2004: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T18:51:40.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:40 smithi045 ceph-mon[31773]: pgmap v2005: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:51:41.345 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3553 sec 2024-03-28T18:51:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:41 smithi071 ceph-mon[36597]: pgmap v2006: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:51:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2909.290783 secs 2024-03-28T18:51:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:41 smithi151 ceph-mon[36481]: pgmap v2006: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:51:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2909.290783 secs 2024-03-28T18:51:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:41 smithi045 ceph-mon[31773]: pgmap v2006: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:51:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2909.290783 secs 2024-03-28T18:51:42.345 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3554 sec 2024-03-28T18:51:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2200.715607 secs 2024-03-28T18:51:42.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2200.715607 secs 2024-03-28T18:51:42.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2200.715607 secs 2024-03-28T18:51:43.345 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3555 sec 2024-03-28T18:51:43.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:43 smithi151 ceph-mon[36481]: pgmap v2007: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:51:43.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:43 smithi045 ceph-mon[31773]: pgmap v2007: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:51:44.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:43 smithi071 ceph-mon[36597]: pgmap v2007: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:51:44.345 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3556 sec 2024-03-28T18:51:45.346 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3557 sec 2024-03-28T18:51:46.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:45 smithi071 ceph-mon[36597]: pgmap v2008: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:51:46.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:45 smithi151 ceph-mon[36481]: pgmap v2008: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:51:46.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:45 smithi045 ceph-mon[31773]: pgmap v2008: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T18:51:46.346 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3558 sec 2024-03-28T18:51:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2914.290873 secs 2024-03-28T18:51:47.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2914.290873 secs 2024-03-28T18:51:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2914.290873 secs 2024-03-28T18:51:47.346 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3559 sec 2024-03-28T18:51:48.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:47 smithi071 ceph-mon[36597]: pgmap v2009: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:51:48.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2205.715685 secs 2024-03-28T18:51:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:47 smithi151 ceph-mon[36481]: pgmap v2009: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:51:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2205.715685 secs 2024-03-28T18:51:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:47 smithi045 ceph-mon[31773]: pgmap v2009: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T18:51:48.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2205.715685 secs 2024-03-28T18:51:48.346 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3560 sec 2024-03-28T18:51:49.346 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3561 sec 2024-03-28T18:51:50.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:49 smithi071 ceph-mon[36597]: pgmap v2010: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:51:50.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:49 smithi151 ceph-mon[36481]: pgmap v2010: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:51:50.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:49 smithi045 ceph-mon[31773]: pgmap v2010: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:51:50.346 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3562 sec 2024-03-28T18:51:51.346 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3563 sec 2024-03-28T18:51:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:51 smithi071 ceph-mon[36597]: pgmap v2011: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2919.290962 secs 2024-03-28T18:51:52.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:51 smithi151 ceph-mon[36481]: pgmap v2011: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:52.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2919.290962 secs 2024-03-28T18:51:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:51 smithi045 ceph-mon[31773]: pgmap v2011: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:52.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2919.290962 secs 2024-03-28T18:51:52.346 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3564 sec 2024-03-28T18:51:53.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2210.715744 secs 2024-03-28T18:51:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2210.715744 secs 2024-03-28T18:51:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2210.715744 secs 2024-03-28T18:51:53.346 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3565 sec 2024-03-28T18:51:54.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:53 smithi071 ceph-mon[36597]: pgmap v2012: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:54.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:53 smithi151 ceph-mon[36481]: pgmap v2012: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:54.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:53 smithi045 ceph-mon[31773]: pgmap v2012: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:54.347 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3566 sec 2024-03-28T18:51:55.347 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3567 sec 2024-03-28T18:51:56.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:55 smithi071 ceph-mon[36597]: pgmap v2013: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:56.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:55 smithi151 ceph-mon[36481]: pgmap v2013: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:56.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:55 smithi045 ceph-mon[31773]: pgmap v2013: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:51:56.347 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3568 sec 2024-03-28T18:51:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2924.291050 secs 2024-03-28T18:51:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2924.291050 secs 2024-03-28T18:51:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2924.291050 secs 2024-03-28T18:51:57.347 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3569 sec 2024-03-28T18:51:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:57 smithi071 ceph-mon[36597]: pgmap v2014: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2215.715803 secs 2024-03-28T18:51:58.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:57 smithi151 ceph-mon[36481]: pgmap v2014: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:58.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2215.715803 secs 2024-03-28T18:51:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:57 smithi045 ceph-mon[31773]: pgmap v2014: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:51:58.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2215.715803 secs 2024-03-28T18:51:58.347 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3570 sec 2024-03-28T18:51:59.347 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3571 sec 2024-03-28T18:52:00.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:51:59 smithi071 ceph-mon[36597]: pgmap v2015: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:00.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:51:59 smithi151 ceph-mon[36481]: pgmap v2015: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:00.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:51:59 smithi045 ceph-mon[31773]: pgmap v2015: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:00.347 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3572 sec 2024-03-28T18:52:01.348 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3573 sec 2024-03-28T18:52:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:01 smithi071 ceph-mon[36597]: pgmap v2016: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2929.291092 secs 2024-03-28T18:52:02.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:01 smithi151 ceph-mon[36481]: pgmap v2016: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:02.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2929.291092 secs 2024-03-28T18:52:02.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:01 smithi045 ceph-mon[31773]: pgmap v2016: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:02.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2929.291092 secs 2024-03-28T18:52:02.348 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3574 sec 2024-03-28T18:52:03.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2220.715887 secs 2024-03-28T18:52:03.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2220.715887 secs 2024-03-28T18:52:03.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2220.715887 secs 2024-03-28T18:52:03.348 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3575 sec 2024-03-28T18:52:04.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:03 smithi071 ceph-mon[36597]: pgmap v2017: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:04.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:03 smithi151 ceph-mon[36481]: pgmap v2017: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:04.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:03 smithi045 ceph-mon[31773]: pgmap v2017: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:04.348 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3576 sec 2024-03-28T18:52:05.348 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3577 sec 2024-03-28T18:52:06.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:05 smithi071 ceph-mon[36597]: pgmap v2018: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:06.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:05 smithi151 ceph-mon[36481]: pgmap v2018: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:06.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:05 smithi045 ceph-mon[31773]: pgmap v2018: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:06.348 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3579 sec 2024-03-28T18:52:07.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:06 smithi071 ceph-mon[36597]: 5 slow requests, 1 included below; oldest blocked for > 2934.291152 secs 2024-03-28T18:52:07.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:06 smithi071 ceph-mon[36597]: slow request 364.103012 seconds old, received at 2024-03-28T18:46:01.748685+0000: client_request(client.26723:1 getattr pAsLsXsFs #0x1 2024-03-28T18:46:01.745025+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:52:07.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:06 smithi151 ceph-mon[36481]: 5 slow requests, 1 included below; oldest blocked for > 2934.291152 secs 2024-03-28T18:52:07.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:06 smithi151 ceph-mon[36481]: slow request 364.103012 seconds old, received at 2024-03-28T18:46:01.748685+0000: client_request(client.26723:1 getattr pAsLsXsFs #0x1 2024-03-28T18:46:01.745025+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:52:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:06 smithi045 ceph-mon[31773]: 5 slow requests, 1 included below; oldest blocked for > 2934.291152 secs 2024-03-28T18:52:07.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:06 smithi045 ceph-mon[31773]: slow request 364.103012 seconds old, received at 2024-03-28T18:46:01.748685+0000: client_request(client.26723:1 getattr pAsLsXsFs #0x1 2024-03-28T18:46:01.745025+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:52:07.349 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3580 sec 2024-03-28T18:52:08.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2225.715925 secs 2024-03-28T18:52:08.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:07 smithi071 ceph-mon[36597]: pgmap v2019: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:52:08.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2225.715925 secs 2024-03-28T18:52:08.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:07 smithi151 ceph-mon[36481]: pgmap v2019: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:52:08.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2225.715925 secs 2024-03-28T18:52:08.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:07 smithi045 ceph-mon[31773]: pgmap v2019: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:52:08.349 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3581 sec 2024-03-28T18:52:09.349 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3582 sec 2024-03-28T18:52:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:09 smithi151 ceph-mon[36481]: pgmap v2020: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:52:10.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:09 smithi045 ceph-mon[31773]: pgmap v2020: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:52:10.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:09 smithi071 ceph-mon[36597]: pgmap v2020: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:52:10.349 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3583 sec 2024-03-28T18:52:11.349 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3584 sec 2024-03-28T18:52:12.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:11 smithi151 ceph-mon[36481]: pgmap v2021: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:52:12.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2939.291240 secs 2024-03-28T18:52:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:11 smithi045 ceph-mon[31773]: pgmap v2021: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:52:12.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2939.291240 secs 2024-03-28T18:52:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:11 smithi071 ceph-mon[36597]: pgmap v2021: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:52:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2939.291240 secs 2024-03-28T18:52:12.349 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3585 sec 2024-03-28T18:52:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2230.715988 secs 2024-03-28T18:52:13.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2230.715988 secs 2024-03-28T18:52:13.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2230.715988 secs 2024-03-28T18:52:13.349 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3586 sec 2024-03-28T18:52:14.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:13 smithi045 ceph-mon[31773]: pgmap v2022: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:52:14.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:13 smithi071 ceph-mon[36597]: pgmap v2022: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:52:14.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:13 smithi151 ceph-mon[36481]: pgmap v2022: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:52:14.349 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3587 sec 2024-03-28T18:52:15.350 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3588 sec 2024-03-28T18:52:16.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:15 smithi045 ceph-mon[31773]: pgmap v2023: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:52:16.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:15 smithi071 ceph-mon[36597]: pgmap v2023: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:52:16.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:15 smithi151 ceph-mon[36481]: pgmap v2023: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T18:52:16.350 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3589 sec 2024-03-28T18:52:17.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2944.291302 secs 2024-03-28T18:52:17.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2944.291302 secs 2024-03-28T18:52:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2944.291302 secs 2024-03-28T18:52:17.350 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3590 sec 2024-03-28T18:52:18.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2235.716051 secs 2024-03-28T18:52:18.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:17 smithi045 ceph-mon[31773]: pgmap v2024: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:52:18.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2235.716051 secs 2024-03-28T18:52:18.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:17 smithi071 ceph-mon[36597]: pgmap v2024: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:52:18.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2235.716051 secs 2024-03-28T18:52:18.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:17 smithi151 ceph-mon[36481]: pgmap v2024: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T18:52:18.350 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3591 sec 2024-03-28T18:52:19.350 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3592 sec 2024-03-28T18:52:20.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:19 smithi045 ceph-mon[31773]: pgmap v2025: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:20.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:19 smithi071 ceph-mon[36597]: pgmap v2025: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:20.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:19 smithi151 ceph-mon[36481]: pgmap v2025: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:20.350 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3593 sec 2024-03-28T18:52:21.351 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3594 sec 2024-03-28T18:52:22.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:21 smithi045 ceph-mon[31773]: pgmap v2026: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:22.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2949.291362 secs 2024-03-28T18:52:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:21 smithi071 ceph-mon[36597]: pgmap v2026: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2949.291362 secs 2024-03-28T18:52:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:21 smithi151 ceph-mon[36481]: pgmap v2026: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2949.291362 secs 2024-03-28T18:52:22.351 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3595 sec 2024-03-28T18:52:23.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2240.716114 secs 2024-03-28T18:52:23.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2240.716114 secs 2024-03-28T18:52:23.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2240.716114 secs 2024-03-28T18:52:23.351 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3596 sec 2024-03-28T18:52:24.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:23 smithi045 ceph-mon[31773]: pgmap v2027: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:24.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:23 smithi071 ceph-mon[36597]: pgmap v2027: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:24.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:23 smithi151 ceph-mon[36481]: pgmap v2027: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:24.351 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3597 sec 2024-03-28T18:52:25.351 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3598 sec 2024-03-28T18:52:26.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:25 smithi045 ceph-mon[31773]: pgmap v2028: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:26.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:25 smithi071 ceph-mon[36597]: pgmap v2028: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:25 smithi151 ceph-mon[36481]: pgmap v2028: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:26.351 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3599 sec 2024-03-28T18:52:27.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2954.291457 secs 2024-03-28T18:52:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2954.291457 secs 2024-03-28T18:52:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2954.291457 secs 2024-03-28T18:52:27.351 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3600 sec 2024-03-28T18:52:28.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2245.716176 secs 2024-03-28T18:52:28.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:27 smithi045 ceph-mon[31773]: pgmap v2029: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:28.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2245.716176 secs 2024-03-28T18:52:28.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:27 smithi071 ceph-mon[36597]: pgmap v2029: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:28.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2245.716176 secs 2024-03-28T18:52:28.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:27 smithi151 ceph-mon[36481]: pgmap v2029: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:28.351 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3601 sec 2024-03-28T18:52:29.352 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3602 sec 2024-03-28T18:52:30.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:29 smithi045 ceph-mon[31773]: pgmap v2030: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:30.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:29 smithi071 ceph-mon[36597]: pgmap v2030: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:30.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:29 smithi151 ceph-mon[36481]: pgmap v2030: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:30.351 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3603 sec 2024-03-28T18:52:31.352 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3604 sec 2024-03-28T18:52:32.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:31 smithi045 ceph-mon[31773]: pgmap v2031: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:32.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2959.291542 secs 2024-03-28T18:52:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:31 smithi071 ceph-mon[36597]: pgmap v2031: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2959.291542 secs 2024-03-28T18:52:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:31 smithi151 ceph-mon[36481]: pgmap v2031: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2959.291542 secs 2024-03-28T18:52:32.352 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3605 sec 2024-03-28T18:52:33.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2250.716237 secs 2024-03-28T18:52:33.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2250.716237 secs 2024-03-28T18:52:33.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2250.716237 secs 2024-03-28T18:52:33.352 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3606 sec 2024-03-28T18:52:34.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:33 smithi045 ceph-mon[31773]: pgmap v2032: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:34.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:33 smithi071 ceph-mon[36597]: pgmap v2032: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:34.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:33 smithi151 ceph-mon[36481]: pgmap v2032: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:34.352 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3607 sec 2024-03-28T18:52:35.354 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3608 sec 2024-03-28T18:52:36.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:35 smithi045 ceph-mon[31773]: pgmap v2033: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:36.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:35 smithi071 ceph-mon[36597]: pgmap v2033: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:36.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:35 smithi151 ceph-mon[36481]: pgmap v2033: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:36.352 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3609 sec 2024-03-28T18:52:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2964.291602 secs 2024-03-28T18:52:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2964.291602 secs 2024-03-28T18:52:37.353 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3610 sec 2024-03-28T18:52:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2964.291602 secs 2024-03-28T18:52:38.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2255.716296 secs 2024-03-28T18:52:38.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:37 smithi071 ceph-mon[36597]: pgmap v2034: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:38.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2255.716296 secs 2024-03-28T18:52:38.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:37 smithi151 ceph-mon[36481]: pgmap v2034: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:38.353 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3611 sec 2024-03-28T18:52:38.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2255.716296 secs 2024-03-28T18:52:38.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:37 smithi045 ceph-mon[31773]: pgmap v2034: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:39.353 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3612 sec 2024-03-28T18:52:40.219 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:39 smithi045 ceph-mon[31773]: pgmap v2035: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:40.219 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:39 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:52:40.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:39 smithi071 ceph-mon[36597]: pgmap v2035: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:40.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:39 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:52:40.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:39 smithi151 ceph-mon[36481]: pgmap v2035: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:40.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:39 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:52:40.353 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3613 sec 2024-03-28T18:52:41.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:40 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:52:41.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:40 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:52:41.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:40 smithi071 ceph-mon[36597]: pgmap v2036: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:52:41.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:40 smithi071 ceph-mon[36597]: pgmap v2037: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:52:41.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:40 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:52:41.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:40 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:52:41.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:40 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:52:41.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:40 smithi151 ceph-mon[36481]: pgmap v2036: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:52:41.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:40 smithi151 ceph-mon[36481]: pgmap v2037: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:52:41.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:40 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:52:41.353 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3614 sec 2024-03-28T18:52:41.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:40 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:52:41.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:40 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:52:41.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:40 smithi045 ceph-mon[31773]: pgmap v2036: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:52:41.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:40 smithi045 ceph-mon[31773]: pgmap v2037: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:52:41.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:40 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:52:42.224 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2969.291622 secs 2024-03-28T18:52:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2969.291622 secs 2024-03-28T18:52:42.353 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3615 sec 2024-03-28T18:52:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2969.291622 secs 2024-03-28T18:52:43.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2260.716384 secs 2024-03-28T18:52:43.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:42 smithi071 ceph-mon[36597]: pgmap v2038: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:52:43.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2260.716384 secs 2024-03-28T18:52:43.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:42 smithi151 ceph-mon[36481]: pgmap v2038: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:52:43.353 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3616 sec 2024-03-28T18:52:43.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2260.716384 secs 2024-03-28T18:52:43.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:42 smithi045 ceph-mon[31773]: pgmap v2038: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:52:44.353 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3617 sec 2024-03-28T18:52:45.354 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3618 sec 2024-03-28T18:52:45.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:45 smithi045 ceph-mon[31773]: pgmap v2039: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:52:45.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:45 smithi071 ceph-mon[36597]: pgmap v2039: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:52:45.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:45 smithi151 ceph-mon[36481]: pgmap v2039: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:52:46.354 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3619 sec 2024-03-28T18:52:46.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2974.291683 secs 2024-03-28T18:52:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2974.291683 secs 2024-03-28T18:52:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2974.291683 secs 2024-03-28T18:52:47.354 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3620 sec 2024-03-28T18:52:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:47 smithi045 ceph-mon[31773]: pgmap v2040: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:52:47.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2265.716426 secs 2024-03-28T18:52:47.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:47 smithi071 ceph-mon[36597]: pgmap v2040: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:52:47.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2265.716426 secs 2024-03-28T18:52:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:47 smithi151 ceph-mon[36481]: pgmap v2040: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:52:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2265.716426 secs 2024-03-28T18:52:48.354 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3621 sec 2024-03-28T18:52:49.354 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3622 sec 2024-03-28T18:52:49.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:49 smithi045 ceph-mon[31773]: pgmap v2041: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:52:49.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:49 smithi071 ceph-mon[36597]: pgmap v2041: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:52:49.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:49 smithi151 ceph-mon[36481]: pgmap v2041: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:52:50.355 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3623 sec 2024-03-28T18:52:51.355 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3624 sec 2024-03-28T18:52:51.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:51 smithi045 ceph-mon[31773]: pgmap v2042: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:52:51.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2979.291756 secs 2024-03-28T18:52:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:51 smithi071 ceph-mon[36597]: pgmap v2042: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:52:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2979.291756 secs 2024-03-28T18:52:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:51 smithi151 ceph-mon[36481]: pgmap v2042: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:52:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2979.291756 secs 2024-03-28T18:52:52.355 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3625 sec 2024-03-28T18:52:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2270.716514 secs 2024-03-28T18:52:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2270.716514 secs 2024-03-28T18:52:52.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2270.716514 secs 2024-03-28T18:52:53.355 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3626 sec 2024-03-28T18:52:53.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:53 smithi071 ceph-mon[36597]: pgmap v2043: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:53.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:53 smithi151 ceph-mon[36481]: pgmap v2043: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:53.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:53 smithi045 ceph-mon[31773]: pgmap v2043: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:54.355 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3627 sec 2024-03-28T18:52:55.355 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3628 sec 2024-03-28T18:52:55.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:55 smithi071 ceph-mon[36597]: pgmap v2044: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:55.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:55 smithi151 ceph-mon[36481]: pgmap v2044: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:55.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:55 smithi045 ceph-mon[31773]: pgmap v2044: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:56.355 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3629 sec 2024-03-28T18:52:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2984.291817 secs 2024-03-28T18:52:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2984.291817 secs 2024-03-28T18:52:56.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2984.291817 secs 2024-03-28T18:52:57.355 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3630 sec 2024-03-28T18:52:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:57 smithi071 ceph-mon[36597]: pgmap v2045: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:57.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2275.716602 secs 2024-03-28T18:52:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:57 smithi151 ceph-mon[36481]: pgmap v2045: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2275.716602 secs 2024-03-28T18:52:57.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:57 smithi045 ceph-mon[31773]: pgmap v2045: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:52:57.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2275.716602 secs 2024-03-28T18:52:58.355 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3631 sec 2024-03-28T18:52:59.356 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3632 sec 2024-03-28T18:52:59.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:52:59 smithi071 ceph-mon[36597]: pgmap v2046: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:59.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:52:59 smithi151 ceph-mon[36481]: pgmap v2046: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:52:59.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:52:59 smithi045 ceph-mon[31773]: pgmap v2046: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:00.356 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3633 sec 2024-03-28T18:53:01.356 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3634 sec 2024-03-28T18:53:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:01 smithi071 ceph-mon[36597]: pgmap v2047: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2989.291903 secs 2024-03-28T18:53:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:01 smithi151 ceph-mon[36481]: pgmap v2047: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2989.291903 secs 2024-03-28T18:53:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:01 smithi045 ceph-mon[31773]: pgmap v2047: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:01.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2989.291903 secs 2024-03-28T18:53:02.356 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3635 sec 2024-03-28T18:53:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2280.716663 secs 2024-03-28T18:53:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2280.716663 secs 2024-03-28T18:53:02.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2280.716663 secs 2024-03-28T18:53:03.356 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3636 sec 2024-03-28T18:53:03.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:03 smithi071 ceph-mon[36597]: pgmap v2048: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:03.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:03 smithi151 ceph-mon[36481]: pgmap v2048: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:03.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:03 smithi045 ceph-mon[31773]: pgmap v2048: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:04.356 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3637 sec 2024-03-28T18:53:05.357 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3638 sec 2024-03-28T18:53:05.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:05 smithi071 ceph-mon[36597]: pgmap v2049: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:05 smithi151 ceph-mon[36481]: pgmap v2049: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:05.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:05 smithi045 ceph-mon[31773]: pgmap v2049: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:06.357 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3639 sec 2024-03-28T18:53:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2994.291993 secs 2024-03-28T18:53:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2994.291993 secs 2024-03-28T18:53:06.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2994.291993 secs 2024-03-28T18:53:07.357 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3640 sec 2024-03-28T18:53:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:07 smithi071 ceph-mon[36597]: pgmap v2050: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:07.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2285.716679 secs 2024-03-28T18:53:07.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:07 smithi151 ceph-mon[36481]: pgmap v2050: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:07.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2285.716679 secs 2024-03-28T18:53:07.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:07 smithi045 ceph-mon[31773]: pgmap v2050: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:07.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2285.716679 secs 2024-03-28T18:53:08.357 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3641 sec 2024-03-28T18:53:09.357 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3642 sec 2024-03-28T18:53:09.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:09 smithi071 ceph-mon[36597]: pgmap v2051: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:09.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:09 smithi151 ceph-mon[36481]: pgmap v2051: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:09.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:09 smithi045 ceph-mon[31773]: pgmap v2051: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:10.357 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3643 sec 2024-03-28T18:53:11.358 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3644 sec 2024-03-28T18:53:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:11 smithi071 ceph-mon[36597]: pgmap v2052: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:11.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 2999.292052 secs 2024-03-28T18:53:11.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:11 smithi151 ceph-mon[36481]: pgmap v2052: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:11.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 2999.292052 secs 2024-03-28T18:53:11.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:11 smithi045 ceph-mon[31773]: pgmap v2052: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:11.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 2999.292052 secs 2024-03-28T18:53:12.358 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3645 sec 2024-03-28T18:53:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2290.716741 secs 2024-03-28T18:53:12.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2290.716741 secs 2024-03-28T18:53:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2290.716741 secs 2024-03-28T18:53:13.358 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3646 sec 2024-03-28T18:53:13.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:13 smithi071 ceph-mon[36597]: pgmap v2053: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:13.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:13 smithi151 ceph-mon[36481]: pgmap v2053: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:13.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:13 smithi045 ceph-mon[31773]: pgmap v2053: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:14.358 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3647 sec 2024-03-28T18:53:15.358 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3648 sec 2024-03-28T18:53:15.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:15 smithi071 ceph-mon[36597]: pgmap v2054: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:15.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:15 smithi151 ceph-mon[36481]: pgmap v2054: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:15.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:15 smithi045 ceph-mon[31773]: pgmap v2054: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:16.358 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3649 sec 2024-03-28T18:53:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3004.292139 secs 2024-03-28T18:53:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3004.292139 secs 2024-03-28T18:53:16.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3004.292139 secs 2024-03-28T18:53:17.358 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3650 sec 2024-03-28T18:53:17.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:17 smithi071 ceph-mon[36597]: pgmap v2055: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:17.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2295.716801 secs 2024-03-28T18:53:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:17 smithi151 ceph-mon[36481]: pgmap v2055: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2295.716801 secs 2024-03-28T18:53:17.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:17 smithi045 ceph-mon[31773]: pgmap v2055: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:17.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2295.716801 secs 2024-03-28T18:53:18.358 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3651 sec 2024-03-28T18:53:19.359 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3652 sec 2024-03-28T18:53:19.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:19 smithi071 ceph-mon[36597]: pgmap v2056: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:19.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:19 smithi151 ceph-mon[36481]: pgmap v2056: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:19.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:19 smithi045 ceph-mon[31773]: pgmap v2056: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:20.359 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3653 sec 2024-03-28T18:53:21.359 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3654 sec 2024-03-28T18:53:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:21 smithi071 ceph-mon[36597]: pgmap v2057: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3009.292226 secs 2024-03-28T18:53:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:21 smithi151 ceph-mon[36481]: pgmap v2057: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3009.292226 secs 2024-03-28T18:53:21.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:21 smithi045 ceph-mon[31773]: pgmap v2057: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:21.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3009.292226 secs 2024-03-28T18:53:22.359 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3655 sec 2024-03-28T18:53:22.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2300.716862 secs 2024-03-28T18:53:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2300.716862 secs 2024-03-28T18:53:22.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2300.716862 secs 2024-03-28T18:53:23.359 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3656 sec 2024-03-28T18:53:23.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:23 smithi071 ceph-mon[36597]: pgmap v2058: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:23.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:23 smithi151 ceph-mon[36481]: pgmap v2058: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:23.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:23 smithi045 ceph-mon[31773]: pgmap v2058: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:24.361 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3657 sec 2024-03-28T18:53:25.359 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3658 sec 2024-03-28T18:53:25.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:25 smithi071 ceph-mon[36597]: pgmap v2059: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:25.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:25 smithi151 ceph-mon[36481]: pgmap v2059: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:25.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:25 smithi045 ceph-mon[31773]: pgmap v2059: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:26.359 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3659 sec 2024-03-28T18:53:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3014.292287 secs 2024-03-28T18:53:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3014.292287 secs 2024-03-28T18:53:26.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3014.292287 secs 2024-03-28T18:53:27.360 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3660 sec 2024-03-28T18:53:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:27 smithi151 ceph-mon[36481]: pgmap v2060: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2305.716894 secs 2024-03-28T18:53:27.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:27 smithi045 ceph-mon[31773]: pgmap v2060: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:27.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2305.716894 secs 2024-03-28T18:53:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:27 smithi071 ceph-mon[36597]: pgmap v2060: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2305.716894 secs 2024-03-28T18:53:28.360 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3661 sec 2024-03-28T18:53:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:28 smithi151 ceph-mon[36481]: pgmap v2061: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:28.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:28 smithi045 ceph-mon[31773]: pgmap v2061: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:29.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:28 smithi071 ceph-mon[36597]: pgmap v2061: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:29.360 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3662 sec 2024-03-28T18:53:30.360 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3663 sec 2024-03-28T18:53:31.360 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3664 sec 2024-03-28T18:53:31.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:31 smithi045 ceph-mon[31773]: pgmap v2062: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:31.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3019.292372 secs 2024-03-28T18:53:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:31 smithi071 ceph-mon[36597]: pgmap v2062: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3019.292372 secs 2024-03-28T18:53:31.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:31 smithi151 ceph-mon[36481]: pgmap v2062: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:31.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3019.292372 secs 2024-03-28T18:53:32.360 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3665 sec 2024-03-28T18:53:32.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2310.716932 secs 2024-03-28T18:53:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2310.716932 secs 2024-03-28T18:53:32.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2310.716932 secs 2024-03-28T18:53:33.360 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3666 sec 2024-03-28T18:53:33.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:33 smithi071 ceph-mon[36597]: pgmap v2063: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:33.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:33 smithi151 ceph-mon[36481]: pgmap v2063: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:33.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:33 smithi045 ceph-mon[31773]: pgmap v2063: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:34.360 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3667 sec 2024-03-28T18:53:35.361 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3668 sec 2024-03-28T18:53:35.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:35 smithi045 ceph-mon[31773]: pgmap v2064: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:35.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:35 smithi071 ceph-mon[36597]: pgmap v2064: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:35.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:35 smithi151 ceph-mon[36481]: pgmap v2064: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:36.361 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3669 sec 2024-03-28T18:53:36.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3024.292409 secs 2024-03-28T18:53:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3024.292409 secs 2024-03-28T18:53:36.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3024.292409 secs 2024-03-28T18:53:37.361 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3670 sec 2024-03-28T18:53:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:37 smithi071 ceph-mon[36597]: pgmap v2065: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2315.716996 secs 2024-03-28T18:53:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:37 smithi151 ceph-mon[36481]: pgmap v2065: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2315.716996 secs 2024-03-28T18:53:37.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:37 smithi045 ceph-mon[31773]: pgmap v2065: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:37.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2315.716996 secs 2024-03-28T18:53:38.361 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3671 sec 2024-03-28T18:53:39.361 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3672 sec 2024-03-28T18:53:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:39 smithi071 ceph-mon[36597]: pgmap v2066: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:39 smithi151 ceph-mon[36481]: pgmap v2066: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:39.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:39 smithi045 ceph-mon[31773]: pgmap v2066: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:40.362 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3673 sec 2024-03-28T18:53:41.361 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3674 sec 2024-03-28T18:53:41.718 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:41 smithi045 ceph-mon[31773]: pgmap v2067: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:41.718 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:41 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:53:41.718 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3029.292476 secs 2024-03-28T18:53:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:41 smithi071 ceph-mon[36597]: pgmap v2067: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:41 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:53:41.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3029.292476 secs 2024-03-28T18:53:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:41 smithi151 ceph-mon[36481]: pgmap v2067: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:41 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:53:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3029.292476 secs 2024-03-28T18:53:42.362 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3675 sec 2024-03-28T18:53:42.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2320.717037 secs 2024-03-28T18:53:42.725 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:42 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:53:42.725 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:42 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:53:42.725 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:42 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:53:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2320.717037 secs 2024-03-28T18:53:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:42 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:53:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:42 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:53:42.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:42 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:53:42.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2320.717037 secs 2024-03-28T18:53:42.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:42 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:53:42.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:42 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:53:42.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:42 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:53:43.362 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3676 sec 2024-03-28T18:53:43.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:43 smithi071 ceph-mon[36597]: pgmap v2068: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:43.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:43 smithi071 ceph-mon[36597]: pgmap v2069: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:53:43.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:43 smithi151 ceph-mon[36481]: pgmap v2068: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:43.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:43 smithi151 ceph-mon[36481]: pgmap v2069: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:53:43.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:43 smithi045 ceph-mon[31773]: pgmap v2068: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:43.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:43 smithi045 ceph-mon[31773]: pgmap v2069: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:53:44.362 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3677 sec 2024-03-28T18:53:45.362 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3678 sec 2024-03-28T18:53:45.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:45 smithi071 ceph-mon[36597]: pgmap v2070: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:53:45.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:45 smithi151 ceph-mon[36481]: pgmap v2070: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:53:45.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:45 smithi045 ceph-mon[31773]: pgmap v2070: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:53:46.362 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3679 sec 2024-03-28T18:53:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3034.292562 secs 2024-03-28T18:53:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3034.292562 secs 2024-03-28T18:53:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3034.292562 secs 2024-03-28T18:53:47.362 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3680 sec 2024-03-28T18:53:47.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:47 smithi071 ceph-mon[36597]: pgmap v2071: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:53:47.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2325.717098 secs 2024-03-28T18:53:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:47 smithi151 ceph-mon[36481]: pgmap v2071: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:53:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2325.717098 secs 2024-03-28T18:53:47.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:47 smithi045 ceph-mon[31773]: pgmap v2071: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:53:47.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2325.717098 secs 2024-03-28T18:53:48.363 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3681 sec 2024-03-28T18:53:49.363 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3682 sec 2024-03-28T18:53:49.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:49 smithi071 ceph-mon[36597]: pgmap v2072: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:53:49.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:49 smithi151 ceph-mon[36481]: pgmap v2072: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:53:49.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:49 smithi045 ceph-mon[31773]: pgmap v2072: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:53:50.363 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3683 sec 2024-03-28T18:53:51.363 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3684 sec 2024-03-28T18:53:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:51 smithi071 ceph-mon[36597]: pgmap v2073: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:53:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3039.292647 secs 2024-03-28T18:53:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:51 smithi151 ceph-mon[36481]: pgmap v2073: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:53:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3039.292647 secs 2024-03-28T18:53:51.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:51 smithi045 ceph-mon[31773]: pgmap v2073: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:53:51.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3039.292647 secs 2024-03-28T18:53:52.363 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3685 sec 2024-03-28T18:53:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2330.717161 secs 2024-03-28T18:53:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2330.717161 secs 2024-03-28T18:53:52.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2330.717161 secs 2024-03-28T18:53:53.363 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3686 sec 2024-03-28T18:53:53.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:53 smithi071 ceph-mon[36597]: pgmap v2074: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:53:53.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:53 smithi151 ceph-mon[36481]: pgmap v2074: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:53:53.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:53 smithi045 ceph-mon[31773]: pgmap v2074: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:53:54.363 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3687 sec 2024-03-28T18:53:55.364 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3688 sec 2024-03-28T18:53:55.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:55 smithi071 ceph-mon[36597]: pgmap v2075: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:55.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:55 smithi151 ceph-mon[36481]: pgmap v2075: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:55.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:55 smithi045 ceph-mon[31773]: pgmap v2075: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:56.364 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3689 sec 2024-03-28T18:53:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3044.292710 secs 2024-03-28T18:53:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3044.292710 secs 2024-03-28T18:53:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3044.292710 secs 2024-03-28T18:53:57.364 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3690 sec 2024-03-28T18:53:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:57 smithi071 ceph-mon[36597]: pgmap v2076: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:57.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2335.717251 secs 2024-03-28T18:53:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:57 smithi151 ceph-mon[36481]: pgmap v2076: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2335.717251 secs 2024-03-28T18:53:57.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:57 smithi045 ceph-mon[31773]: pgmap v2076: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:53:57.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2335.717251 secs 2024-03-28T18:53:58.364 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3691 sec 2024-03-28T18:53:59.364 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3692 sec 2024-03-28T18:53:59.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:53:59 smithi071 ceph-mon[36597]: pgmap v2077: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:59.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:53:59 smithi151 ceph-mon[36481]: pgmap v2077: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:53:59.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:53:59 smithi045 ceph-mon[31773]: pgmap v2077: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:00.364 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3693 sec 2024-03-28T18:54:01.364 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3694 sec 2024-03-28T18:54:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:01 smithi071 ceph-mon[36597]: pgmap v2078: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:01.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3049.292769 secs 2024-03-28T18:54:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:01 smithi151 ceph-mon[36481]: pgmap v2078: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3049.292769 secs 2024-03-28T18:54:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:01 smithi045 ceph-mon[31773]: pgmap v2078: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:01.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3049.292769 secs 2024-03-28T18:54:02.364 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3695 sec 2024-03-28T18:54:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2340.717340 secs 2024-03-28T18:54:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2340.717340 secs 2024-03-28T18:54:02.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2340.717340 secs 2024-03-28T18:54:03.365 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3696 sec 2024-03-28T18:54:03.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:03 smithi071 ceph-mon[36597]: pgmap v2079: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:03.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:03 smithi151 ceph-mon[36481]: pgmap v2079: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:03.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:03 smithi045 ceph-mon[31773]: pgmap v2079: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:04.365 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3697 sec 2024-03-28T18:54:05.365 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3698 sec 2024-03-28T18:54:05.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:05 smithi071 ceph-mon[36597]: pgmap v2080: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:05.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:05 smithi151 ceph-mon[36481]: pgmap v2080: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:05.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:05 smithi045 ceph-mon[31773]: pgmap v2080: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:06.365 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3699 sec 2024-03-28T18:54:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3054.292832 secs 2024-03-28T18:54:06.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3054.292832 secs 2024-03-28T18:54:07.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3054.292832 secs 2024-03-28T18:54:07.365 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3700 sec 2024-03-28T18:54:07.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:07 smithi151 ceph-mon[36481]: pgmap v2081: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:07.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2345.717430 secs 2024-03-28T18:54:07.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:07 smithi045 ceph-mon[31773]: pgmap v2081: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:07.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2345.717430 secs 2024-03-28T18:54:08.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:07 smithi071 ceph-mon[36597]: pgmap v2081: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:08.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2345.717430 secs 2024-03-28T18:54:08.365 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3701 sec 2024-03-28T18:54:08.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:08 smithi045 ceph-mon[31773]: pgmap v2082: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:09.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:08 smithi071 ceph-mon[36597]: pgmap v2082: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:09.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:08 smithi151 ceph-mon[36481]: pgmap v2082: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:09.365 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3702 sec 2024-03-28T18:54:10.366 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3703 sec 2024-03-28T18:54:11.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:11 smithi071 ceph-mon[36597]: pgmap v2083: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:11.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:11 smithi151 ceph-mon[36481]: pgmap v2083: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:11.366 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3704 sec 2024-03-28T18:54:11.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:11 smithi045 ceph-mon[31773]: pgmap v2083: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:12 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3059.292921 secs 2024-03-28T18:54:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2350.717469 secs 2024-03-28T18:54:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:12 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3059.292921 secs 2024-03-28T18:54:12.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2350.717469 secs 2024-03-28T18:54:12.366 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3705 sec 2024-03-28T18:54:12.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:12 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3059.292921 secs 2024-03-28T18:54:12.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2350.717469 secs 2024-03-28T18:54:13.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:13 smithi071 ceph-mon[36597]: pgmap v2084: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:13.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:13 smithi151 ceph-mon[36481]: pgmap v2084: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:13.366 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3706 sec 2024-03-28T18:54:13.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:13 smithi045 ceph-mon[31773]: pgmap v2084: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:14.367 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3707 sec 2024-03-28T18:54:15.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:15 smithi071 ceph-mon[36597]: pgmap v2085: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:15.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:15 smithi151 ceph-mon[36481]: pgmap v2085: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:15.366 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3708 sec 2024-03-28T18:54:15.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:15 smithi045 ceph-mon[31773]: pgmap v2085: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:16.367 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3709 sec 2024-03-28T18:54:17.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:17 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3064.293010 secs 2024-03-28T18:54:17.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:17 smithi071 ceph-mon[36597]: pgmap v2086: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:17.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2355.717557 secs 2024-03-28T18:54:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:17 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3064.293010 secs 2024-03-28T18:54:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:17 smithi151 ceph-mon[36481]: pgmap v2086: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:17.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2355.717557 secs 2024-03-28T18:54:17.366 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3710 sec 2024-03-28T18:54:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:17 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3064.293010 secs 2024-03-28T18:54:17.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:17 smithi045 ceph-mon[31773]: pgmap v2086: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:17.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2355.717557 secs 2024-03-28T18:54:18.367 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3711 sec 2024-03-28T18:54:19.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:19 smithi071 ceph-mon[36597]: pgmap v2087: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:19.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:19 smithi151 ceph-mon[36481]: pgmap v2087: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:19.367 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3712 sec 2024-03-28T18:54:19.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:19 smithi045 ceph-mon[31773]: pgmap v2087: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:20.367 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3713 sec 2024-03-28T18:54:21.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:21 smithi071 ceph-mon[36597]: pgmap v2088: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:21.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:21 smithi151 ceph-mon[36481]: pgmap v2088: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:21.367 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3714 sec 2024-03-28T18:54:21.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:21 smithi045 ceph-mon[31773]: pgmap v2088: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:22 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3069.293072 secs 2024-03-28T18:54:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2360.717599 secs 2024-03-28T18:54:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:22 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3069.293072 secs 2024-03-28T18:54:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2360.717599 secs 2024-03-28T18:54:22.367 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3715 sec 2024-03-28T18:54:22.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:22 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3069.293072 secs 2024-03-28T18:54:22.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2360.717599 secs 2024-03-28T18:54:23.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:23 smithi071 ceph-mon[36597]: pgmap v2089: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:23.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:23 smithi151 ceph-mon[36481]: pgmap v2089: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:23.368 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3716 sec 2024-03-28T18:54:23.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:23 smithi045 ceph-mon[31773]: pgmap v2089: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:24.368 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3717 sec 2024-03-28T18:54:25.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:25 smithi151 ceph-mon[36481]: pgmap v2090: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:25.368 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3718 sec 2024-03-28T18:54:25.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:25 smithi045 ceph-mon[31773]: pgmap v2090: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:25.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:25 smithi071 ceph-mon[36597]: pgmap v2090: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:26.371 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3719 sec 2024-03-28T18:54:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:27 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3074.293157 secs 2024-03-28T18:54:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:27 smithi151 ceph-mon[36481]: pgmap v2091: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:27.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2365.717688 secs 2024-03-28T18:54:27.368 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3720 sec 2024-03-28T18:54:27.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:27 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3074.293157 secs 2024-03-28T18:54:27.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:27 smithi045 ceph-mon[31773]: pgmap v2091: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:27.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2365.717688 secs 2024-03-28T18:54:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:27 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3074.293157 secs 2024-03-28T18:54:27.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:27 smithi071 ceph-mon[36597]: pgmap v2091: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:27.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2365.717688 secs 2024-03-28T18:54:28.368 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3721 sec 2024-03-28T18:54:29.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:29 smithi151 ceph-mon[36481]: pgmap v2092: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:29.368 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3722 sec 2024-03-28T18:54:29.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:29 smithi045 ceph-mon[31773]: pgmap v2092: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:29.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:29 smithi071 ceph-mon[36597]: pgmap v2092: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:30.368 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3723 sec 2024-03-28T18:54:31.369 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3724 sec 2024-03-28T18:54:31.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:31 smithi045 ceph-mon[31773]: pgmap v2093: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:31 smithi071 ceph-mon[36597]: pgmap v2093: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:31 smithi151 ceph-mon[36481]: pgmap v2093: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:32.369 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3725 sec 2024-03-28T18:54:32.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:32 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3079.293220 secs 2024-03-28T18:54:32.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2370.717776 secs 2024-03-28T18:54:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:32 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3079.293220 secs 2024-03-28T18:54:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2370.717776 secs 2024-03-28T18:54:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:32 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3079.293220 secs 2024-03-28T18:54:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2370.717776 secs 2024-03-28T18:54:33.369 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3726 sec 2024-03-28T18:54:33.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:33 smithi045 ceph-mon[31773]: pgmap v2094: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:33.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:33 smithi071 ceph-mon[36597]: pgmap v2094: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:33 smithi151 ceph-mon[36481]: pgmap v2094: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:34.370 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3727 sec 2024-03-28T18:54:35.369 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3728 sec 2024-03-28T18:54:35.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:35 smithi045 ceph-mon[31773]: pgmap v2095: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:35.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:35 smithi071 ceph-mon[36597]: pgmap v2095: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:35.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:35 smithi151 ceph-mon[36481]: pgmap v2095: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:36.370 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3729 sec 2024-03-28T18:54:37.371 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3730 sec 2024-03-28T18:54:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:37 smithi045 ceph-mon[31773]: 5 slow requests, 1 included below; oldest blocked for > 3084.293277 secs 2024-03-28T18:54:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:37 smithi045 ceph-mon[31773]: slow request 1444.464386 seconds old, received at 2024-03-28T18:30:31.389436+0000: client_request(client.26705:1 getattr pAsLsXsFs #0x1 2024-03-28T18:30:31.385264+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:54:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:37 smithi045 ceph-mon[31773]: pgmap v2096: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2375.717871 secs 2024-03-28T18:54:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:37 smithi071 ceph-mon[36597]: 5 slow requests, 1 included below; oldest blocked for > 3084.293277 secs 2024-03-28T18:54:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:37 smithi071 ceph-mon[36597]: slow request 1444.464386 seconds old, received at 2024-03-28T18:30:31.389436+0000: client_request(client.26705:1 getattr pAsLsXsFs #0x1 2024-03-28T18:30:31.385264+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:54:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:37 smithi071 ceph-mon[36597]: pgmap v2096: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:37.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2375.717871 secs 2024-03-28T18:54:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:37 smithi151 ceph-mon[36481]: 5 slow requests, 1 included below; oldest blocked for > 3084.293277 secs 2024-03-28T18:54:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:37 smithi151 ceph-mon[36481]: slow request 1444.464386 seconds old, received at 2024-03-28T18:30:31.389436+0000: client_request(client.26705:1 getattr pAsLsXsFs #0x1 2024-03-28T18:30:31.385264+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:54:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:37 smithi151 ceph-mon[36481]: pgmap v2096: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2375.717871 secs 2024-03-28T18:54:38.370 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3731 sec 2024-03-28T18:54:39.370 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3732 sec 2024-03-28T18:54:39.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:39 smithi045 ceph-mon[31773]: pgmap v2097: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:39.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:39 smithi071 ceph-mon[36597]: pgmap v2097: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:39.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:39 smithi151 ceph-mon[36481]: pgmap v2097: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:40.370 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3733 sec 2024-03-28T18:54:41.370 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3734 sec 2024-03-28T18:54:41.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:41 smithi045 ceph-mon[31773]: pgmap v2098: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:41 smithi071 ceph-mon[36597]: pgmap v2098: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:41 smithi151 ceph-mon[36481]: pgmap v2098: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:42.370 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3735 sec 2024-03-28T18:54:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:42 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3089.293367 secs 2024-03-28T18:54:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2380.717908 secs 2024-03-28T18:54:42.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:42 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3089.293367 secs 2024-03-28T18:54:42.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2380.717908 secs 2024-03-28T18:54:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:42 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3089.293367 secs 2024-03-28T18:54:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2380.717908 secs 2024-03-28T18:54:43.370 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3736 sec 2024-03-28T18:54:43.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:43 smithi045 ceph-mon[31773]: pgmap v2099: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:43.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:43 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:54:43.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:43 smithi071 ceph-mon[36597]: pgmap v2099: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:43.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:43 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:54:43.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:43 smithi151 ceph-mon[36481]: pgmap v2099: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:43.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:43 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:54:44.370 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3737 sec 2024-03-28T18:54:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:44 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:54:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:44 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:54:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:44 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:54:44.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:44 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:54:44.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:44 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:54:44.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:44 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:54:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:44 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:54:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:44 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:54:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:44 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:54:45.371 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3738 sec 2024-03-28T18:54:45.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:45 smithi045 ceph-mon[31773]: pgmap v2100: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:45.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:45 smithi045 ceph-mon[31773]: pgmap v2101: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:54:45.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:45 smithi071 ceph-mon[36597]: pgmap v2100: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:45.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:45 smithi071 ceph-mon[36597]: pgmap v2101: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:54:45.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:45 smithi151 ceph-mon[36481]: pgmap v2100: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:45.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:45 smithi151 ceph-mon[36481]: pgmap v2101: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:54:46.370 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3739 sec 2024-03-28T18:54:47.371 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3740 sec 2024-03-28T18:54:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:47 smithi045 ceph-mon[31773]: pgmap v2102: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:54:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:47 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3094.293406 secs 2024-03-28T18:54:47.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2385.717969 secs 2024-03-28T18:54:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:47 smithi071 ceph-mon[36597]: pgmap v2102: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:54:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:47 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3094.293406 secs 2024-03-28T18:54:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2385.717969 secs 2024-03-28T18:54:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:47 smithi151 ceph-mon[36481]: pgmap v2102: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:54:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:47 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3094.293406 secs 2024-03-28T18:54:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2385.717969 secs 2024-03-28T18:54:48.371 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3741 sec 2024-03-28T18:54:49.371 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3742 sec 2024-03-28T18:54:49.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:49 smithi045 ceph-mon[31773]: pgmap v2103: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:54:49.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:49 smithi071 ceph-mon[36597]: pgmap v2103: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:54:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:49 smithi151 ceph-mon[36481]: pgmap v2103: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:54:50.371 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3743 sec 2024-03-28T18:54:51.371 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3744 sec 2024-03-28T18:54:51.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:51 smithi045 ceph-mon[31773]: pgmap v2104: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:54:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:51 smithi071 ceph-mon[36597]: pgmap v2104: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:54:51.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:51 smithi151 ceph-mon[36481]: pgmap v2104: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:54:52.371 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3745 sec 2024-03-28T18:54:52.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:52 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3099.293488 secs 2024-03-28T18:54:52.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2390.718030 secs 2024-03-28T18:54:52.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:52 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3099.293488 secs 2024-03-28T18:54:52.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2390.718030 secs 2024-03-28T18:54:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:52 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3099.293488 secs 2024-03-28T18:54:52.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2390.718030 secs 2024-03-28T18:54:53.372 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3746 sec 2024-03-28T18:54:53.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:53 smithi045 ceph-mon[31773]: pgmap v2105: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:54:53.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:53 smithi071 ceph-mon[36597]: pgmap v2105: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:54:53.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:53 smithi151 ceph-mon[36481]: pgmap v2105: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:54:54.372 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3747 sec 2024-03-28T18:54:55.372 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3748 sec 2024-03-28T18:54:55.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:55 smithi045 ceph-mon[31773]: pgmap v2106: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:54:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:55 smithi071 ceph-mon[36597]: pgmap v2106: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:54:55.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:55 smithi151 ceph-mon[36481]: pgmap v2106: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:54:56.372 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3749 sec 2024-03-28T18:54:57.372 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3750 sec 2024-03-28T18:54:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:57 smithi071 ceph-mon[36597]: pgmap v2107: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:57 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3104.293550 secs 2024-03-28T18:54:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2395.718089 secs 2024-03-28T18:54:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:57 smithi151 ceph-mon[36481]: pgmap v2107: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:57 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3104.293550 secs 2024-03-28T18:54:57.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2395.718089 secs 2024-03-28T18:54:57.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:57 smithi045 ceph-mon[31773]: pgmap v2107: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:54:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:57 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3104.293550 secs 2024-03-28T18:54:57.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2395.718089 secs 2024-03-28T18:54:58.372 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3751 sec 2024-03-28T18:54:59.372 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3752 sec 2024-03-28T18:54:59.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:54:59 smithi071 ceph-mon[36597]: pgmap v2108: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:59.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:54:59 smithi151 ceph-mon[36481]: pgmap v2108: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:54:59.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:54:59 smithi045 ceph-mon[31773]: pgmap v2108: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:00.373 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3753 sec 2024-03-28T18:55:01.373 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3754 sec 2024-03-28T18:55:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:01 smithi071 ceph-mon[36597]: pgmap v2109: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:01 smithi151 ceph-mon[36481]: pgmap v2109: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:01.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:01 smithi045 ceph-mon[31773]: pgmap v2109: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:02.373 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3755 sec 2024-03-28T18:55:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:02 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3109.293610 secs 2024-03-28T18:55:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2400.718165 secs 2024-03-28T18:55:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:02 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3109.293610 secs 2024-03-28T18:55:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2400.718165 secs 2024-03-28T18:55:02.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:02 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3109.293610 secs 2024-03-28T18:55:02.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2400.718165 secs 2024-03-28T18:55:03.373 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3756 sec 2024-03-28T18:55:03.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:03 smithi071 ceph-mon[36597]: pgmap v2110: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:03.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:03 smithi151 ceph-mon[36481]: pgmap v2110: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:03.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:03 smithi045 ceph-mon[31773]: pgmap v2110: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:04.373 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3757 sec 2024-03-28T18:55:05.373 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3758 sec 2024-03-28T18:55:05.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:05 smithi071 ceph-mon[36597]: pgmap v2111: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:05.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:05 smithi151 ceph-mon[36481]: pgmap v2111: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:05.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:05 smithi045 ceph-mon[31773]: pgmap v2111: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:06.373 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3759 sec 2024-03-28T18:55:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3114.293669 secs 2024-03-28T18:55:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3114.293669 secs 2024-03-28T18:55:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3114.293669 secs 2024-03-28T18:55:07.373 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3760 sec 2024-03-28T18:55:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:07 smithi071 ceph-mon[36597]: pgmap v2112: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2405.718252 secs 2024-03-28T18:55:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:07 smithi151 ceph-mon[36481]: pgmap v2112: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2405.718252 secs 2024-03-28T18:55:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:07 smithi045 ceph-mon[31773]: pgmap v2112: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2405.718252 secs 2024-03-28T18:55:08.373 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3761 sec 2024-03-28T18:55:09.374 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3762 sec 2024-03-28T18:55:09.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:09 smithi045 ceph-mon[31773]: pgmap v2113: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:09.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:09 smithi071 ceph-mon[36597]: pgmap v2113: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:09.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:09 smithi151 ceph-mon[36481]: pgmap v2113: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:10.374 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3763 sec 2024-03-28T18:55:11.374 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3764 sec 2024-03-28T18:55:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:11 smithi071 ceph-mon[36597]: pgmap v2114: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3119.293730 secs 2024-03-28T18:55:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:11 smithi151 ceph-mon[36481]: pgmap v2114: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3119.293730 secs 2024-03-28T18:55:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:11 smithi045 ceph-mon[31773]: pgmap v2114: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3119.293730 secs 2024-03-28T18:55:12.374 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3765 sec 2024-03-28T18:55:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2410.718315 secs 2024-03-28T18:55:12.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2410.718315 secs 2024-03-28T18:55:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2410.718315 secs 2024-03-28T18:55:13.374 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3766 sec 2024-03-28T18:55:13.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:13 smithi071 ceph-mon[36597]: pgmap v2115: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:13.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:13 smithi151 ceph-mon[36481]: pgmap v2115: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:13.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:13 smithi045 ceph-mon[31773]: pgmap v2115: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:14.374 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3767 sec 2024-03-28T18:55:15.374 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3768 sec 2024-03-28T18:55:15.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:15 smithi071 ceph-mon[36597]: pgmap v2116: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:15.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:15 smithi151 ceph-mon[36481]: pgmap v2116: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:15.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:15 smithi045 ceph-mon[31773]: pgmap v2116: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:16.375 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3769 sec 2024-03-28T18:55:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3124.293817 secs 2024-03-28T18:55:16.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3124.293817 secs 2024-03-28T18:55:16.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3124.293817 secs 2024-03-28T18:55:17.375 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3770 sec 2024-03-28T18:55:17.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:17 smithi071 ceph-mon[36597]: pgmap v2117: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:17.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2415.718377 secs 2024-03-28T18:55:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:17 smithi151 ceph-mon[36481]: pgmap v2117: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2415.718377 secs 2024-03-28T18:55:17.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:17 smithi045 ceph-mon[31773]: pgmap v2117: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:17.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2415.718377 secs 2024-03-28T18:55:18.375 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3771 sec 2024-03-28T18:55:19.375 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3772 sec 2024-03-28T18:55:19.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:19 smithi071 ceph-mon[36597]: pgmap v2118: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:19.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:19 smithi151 ceph-mon[36481]: pgmap v2118: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:19.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:19 smithi045 ceph-mon[31773]: pgmap v2118: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:20.375 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3773 sec 2024-03-28T18:55:21.375 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3774 sec 2024-03-28T18:55:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:21 smithi071 ceph-mon[36597]: pgmap v2119: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3129.293855 secs 2024-03-28T18:55:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:21 smithi151 ceph-mon[36481]: pgmap v2119: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:21.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3129.293855 secs 2024-03-28T18:55:21.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:21 smithi045 ceph-mon[31773]: pgmap v2119: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:21.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3129.293855 secs 2024-03-28T18:55:22.375 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3775 sec 2024-03-28T18:55:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2420.718458 secs 2024-03-28T18:55:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2420.718458 secs 2024-03-28T18:55:22.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2420.718458 secs 2024-03-28T18:55:23.376 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3776 sec 2024-03-28T18:55:23.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:23 smithi071 ceph-mon[36597]: pgmap v2120: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:23.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:23 smithi151 ceph-mon[36481]: pgmap v2120: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:23.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:23 smithi045 ceph-mon[31773]: pgmap v2120: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:24.376 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3777 sec 2024-03-28T18:55:25.376 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3778 sec 2024-03-28T18:55:25.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:25 smithi071 ceph-mon[36597]: pgmap v2121: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:25.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:25 smithi151 ceph-mon[36481]: pgmap v2121: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:25.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:25 smithi045 ceph-mon[31773]: pgmap v2121: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:26.376 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3779 sec 2024-03-28T18:55:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3134.293914 secs 2024-03-28T18:55:26.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3134.293914 secs 2024-03-28T18:55:26.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3134.293914 secs 2024-03-28T18:55:27.376 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3780 sec 2024-03-28T18:55:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:27 smithi071 ceph-mon[36597]: pgmap v2122: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2425.718542 secs 2024-03-28T18:55:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:27 smithi151 ceph-mon[36481]: pgmap v2122: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2425.718542 secs 2024-03-28T18:55:27.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:27 smithi045 ceph-mon[31773]: pgmap v2122: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:27.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2425.718542 secs 2024-03-28T18:55:28.376 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3781 sec 2024-03-28T18:55:29.376 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3782 sec 2024-03-28T18:55:29.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:29 smithi071 ceph-mon[36597]: pgmap v2123: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:29.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:29 smithi151 ceph-mon[36481]: pgmap v2123: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:29.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:29 smithi045 ceph-mon[31773]: pgmap v2123: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:30.376 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3783 sec 2024-03-28T18:55:31.376 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3784 sec 2024-03-28T18:55:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:31 smithi071 ceph-mon[36597]: pgmap v2124: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3139.293973 secs 2024-03-28T18:55:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:31 smithi151 ceph-mon[36481]: pgmap v2124: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:31.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3139.293973 secs 2024-03-28T18:55:31.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:31 smithi045 ceph-mon[31773]: pgmap v2124: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:31.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3139.293973 secs 2024-03-28T18:55:32.377 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3785 sec 2024-03-28T18:55:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2430.718630 secs 2024-03-28T18:55:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2430.718630 secs 2024-03-28T18:55:32.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2430.718630 secs 2024-03-28T18:55:33.377 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3786 sec 2024-03-28T18:55:33.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:33 smithi151 ceph-mon[36481]: pgmap v2125: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:33.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:33 smithi045 ceph-mon[31773]: pgmap v2125: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:33.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:33 smithi071 ceph-mon[36597]: pgmap v2125: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:34.377 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3787 sec 2024-03-28T18:55:35.377 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3788 sec 2024-03-28T18:55:35.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:35 smithi151 ceph-mon[36481]: pgmap v2126: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:35.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:35 smithi045 ceph-mon[31773]: pgmap v2126: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:35.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:35 smithi071 ceph-mon[36597]: pgmap v2126: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:36.377 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3789 sec 2024-03-28T18:55:36.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3144.294060 secs 2024-03-28T18:55:36.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3144.294060 secs 2024-03-28T18:55:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3144.294060 secs 2024-03-28T18:55:37.378 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3790 sec 2024-03-28T18:55:37.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:37 smithi045 ceph-mon[31773]: pgmap v2127: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:37.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2435.718690 secs 2024-03-28T18:55:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:37 smithi071 ceph-mon[36597]: pgmap v2127: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2435.718690 secs 2024-03-28T18:55:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:37 smithi151 ceph-mon[36481]: pgmap v2127: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2435.718690 secs 2024-03-28T18:55:38.377 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3791 sec 2024-03-28T18:55:39.378 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3792 sec 2024-03-28T18:55:39.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:39 smithi045 ceph-mon[31773]: pgmap v2128: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:39 smithi071 ceph-mon[36597]: pgmap v2128: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:39.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:39 smithi151 ceph-mon[36481]: pgmap v2128: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:40.378 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3793 sec 2024-03-28T18:55:41.378 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3794 sec 2024-03-28T18:55:41.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:41 smithi045 ceph-mon[31773]: pgmap v2129: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:41.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3149.294148 secs 2024-03-28T18:55:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:41 smithi071 ceph-mon[36597]: pgmap v2129: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3149.294148 secs 2024-03-28T18:55:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:41 smithi151 ceph-mon[36481]: pgmap v2129: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3149.294148 secs 2024-03-28T18:55:42.378 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3795 sec 2024-03-28T18:55:42.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2440.718778 secs 2024-03-28T18:55:42.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2440.718778 secs 2024-03-28T18:55:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2440.718778 secs 2024-03-28T18:55:43.378 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3796 sec 2024-03-28T18:55:43.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:43 smithi045 ceph-mon[31773]: pgmap v2130: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:43.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:43 smithi071 ceph-mon[36597]: pgmap v2130: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:43.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:43 smithi151 ceph-mon[36481]: pgmap v2130: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:55:44.379 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3797 sec 2024-03-28T18:55:44.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:44 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:55:44.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:44 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:55:44.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:44 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:55:45.379 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3798 sec 2024-03-28T18:55:45.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:45 smithi045 ceph-mon[31773]: pgmap v2131: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:45.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:45 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:55:45.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:45 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:55:45.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:45 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:55:45.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:45 smithi071 ceph-mon[36597]: pgmap v2131: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:45.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:45 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:55:45.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:45 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:55:45.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:45 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:55:45.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:45 smithi151 ceph-mon[36481]: pgmap v2131: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:45.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:45 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:55:45.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:45 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:55:45.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:45 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:55:46.379 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3799 sec 2024-03-28T18:55:46.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:46 smithi045 ceph-mon[31773]: pgmap v2132: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:55:46.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:46 smithi045 ceph-mon[31773]: pgmap v2133: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:46.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3154.294186 secs 2024-03-28T18:55:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:46 smithi071 ceph-mon[36597]: pgmap v2132: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:55:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:46 smithi071 ceph-mon[36597]: pgmap v2133: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3154.294186 secs 2024-03-28T18:55:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:46 smithi151 ceph-mon[36481]: pgmap v2132: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:55:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:46 smithi151 ceph-mon[36481]: pgmap v2133: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3154.294186 secs 2024-03-28T18:55:47.379 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3800 sec 2024-03-28T18:55:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2445.718874 secs 2024-03-28T18:55:47.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2445.718874 secs 2024-03-28T18:55:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2445.718874 secs 2024-03-28T18:55:48.379 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3801 sec 2024-03-28T18:55:48.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:48 smithi045 ceph-mon[31773]: pgmap v2134: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:48.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:48 smithi071 ceph-mon[36597]: pgmap v2134: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:48 smithi151 ceph-mon[36481]: pgmap v2134: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:49.379 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3802 sec 2024-03-28T18:55:50.379 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3803 sec 2024-03-28T18:55:50.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:50 smithi045 ceph-mon[31773]: pgmap v2135: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:50.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:50 smithi071 ceph-mon[36597]: pgmap v2135: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:50.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:50 smithi151 ceph-mon[36481]: pgmap v2135: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:51.380 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3804 sec 2024-03-28T18:55:51.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3159.294270 secs 2024-03-28T18:55:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3159.294270 secs 2024-03-28T18:55:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3159.294270 secs 2024-03-28T18:55:52.380 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3805 sec 2024-03-28T18:55:52.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:52 smithi045 ceph-mon[31773]: pgmap v2136: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:52.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2450.718964 secs 2024-03-28T18:55:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:52 smithi071 ceph-mon[36597]: pgmap v2136: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2450.718964 secs 2024-03-28T18:55:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:52 smithi151 ceph-mon[36481]: pgmap v2136: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2450.718964 secs 2024-03-28T18:55:53.380 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3806 sec 2024-03-28T18:55:54.380 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3807 sec 2024-03-28T18:55:54.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:54 smithi045 ceph-mon[31773]: pgmap v2137: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:54.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:54 smithi071 ceph-mon[36597]: pgmap v2137: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:54.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:54 smithi151 ceph-mon[36481]: pgmap v2137: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:55:55.380 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3808 sec 2024-03-28T18:55:56.380 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3809 sec 2024-03-28T18:55:56.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:56 smithi045 ceph-mon[31773]: pgmap v2138: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:55:56.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3164.294294 secs 2024-03-28T18:55:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:56 smithi071 ceph-mon[36597]: pgmap v2138: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:55:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3164.294294 secs 2024-03-28T18:55:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:56 smithi151 ceph-mon[36481]: pgmap v2138: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:55:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3164.294294 secs 2024-03-28T18:55:57.381 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3810 sec 2024-03-28T18:55:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2455.719053 secs 2024-03-28T18:55:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2455.719053 secs 2024-03-28T18:55:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2455.719053 secs 2024-03-28T18:55:58.381 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3811 sec 2024-03-28T18:55:58.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:55:58 smithi071 ceph-mon[36597]: pgmap v2139: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:58.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:55:58 smithi151 ceph-mon[36481]: pgmap v2139: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:58.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:55:58 smithi045 ceph-mon[31773]: pgmap v2139: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:55:59.381 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3812 sec 2024-03-28T18:56:00.381 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3813 sec 2024-03-28T18:56:00.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:00 smithi071 ceph-mon[36597]: pgmap v2140: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:00 smithi151 ceph-mon[36481]: pgmap v2140: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:00.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:00 smithi045 ceph-mon[31773]: pgmap v2140: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:01.381 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3814 sec 2024-03-28T18:56:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3169.294383 secs 2024-03-28T18:56:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3169.294383 secs 2024-03-28T18:56:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3169.294383 secs 2024-03-28T18:56:02.381 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3815 sec 2024-03-28T18:56:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:02 smithi071 ceph-mon[36597]: pgmap v2141: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2460.719144 secs 2024-03-28T18:56:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:02 smithi151 ceph-mon[36481]: pgmap v2141: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2460.719144 secs 2024-03-28T18:56:02.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:02 smithi045 ceph-mon[31773]: pgmap v2141: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:02.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2460.719144 secs 2024-03-28T18:56:03.381 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3816 sec 2024-03-28T18:56:04.381 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3817 sec 2024-03-28T18:56:04.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:04 smithi071 ceph-mon[36597]: pgmap v2142: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:04.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:04 smithi151 ceph-mon[36481]: pgmap v2142: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:04.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:04 smithi045 ceph-mon[31773]: pgmap v2142: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:05.381 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3818 sec 2024-03-28T18:56:06.381 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3819 sec 2024-03-28T18:56:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:06 smithi071 ceph-mon[36597]: pgmap v2143: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:56:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3174.294471 secs 2024-03-28T18:56:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:06 smithi151 ceph-mon[36481]: pgmap v2143: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:56:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3174.294471 secs 2024-03-28T18:56:06.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:06 smithi045 ceph-mon[31773]: pgmap v2143: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:56:06.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3174.294471 secs 2024-03-28T18:56:07.382 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3820 sec 2024-03-28T18:56:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2465.719207 secs 2024-03-28T18:56:07.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2465.719207 secs 2024-03-28T18:56:07.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2465.719207 secs 2024-03-28T18:56:08.382 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3821 sec 2024-03-28T18:56:08.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:08 smithi071 ceph-mon[36597]: pgmap v2144: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:56:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:08 smithi151 ceph-mon[36481]: pgmap v2144: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:56:08.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:08 smithi045 ceph-mon[31773]: pgmap v2144: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:56:09.382 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3822 sec 2024-03-28T18:56:10.382 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3823 sec 2024-03-28T18:56:10.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:10 smithi071 ceph-mon[36597]: pgmap v2145: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:56:10.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:10 smithi151 ceph-mon[36481]: pgmap v2145: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:56:10.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:10 smithi045 ceph-mon[31773]: pgmap v2145: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:56:11.382 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3824 sec 2024-03-28T18:56:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3179.294532 secs 2024-03-28T18:56:11.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3179.294532 secs 2024-03-28T18:56:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3179.294532 secs 2024-03-28T18:56:12.382 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3825 sec 2024-03-28T18:56:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:12 smithi071 ceph-mon[36597]: pgmap v2146: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:56:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2470.719246 secs 2024-03-28T18:56:12.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:12 smithi151 ceph-mon[36481]: pgmap v2146: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:56:12.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2470.719246 secs 2024-03-28T18:56:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:12 smithi045 ceph-mon[31773]: pgmap v2146: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:56:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2470.719246 secs 2024-03-28T18:56:13.383 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3826 sec 2024-03-28T18:56:14.383 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3827 sec 2024-03-28T18:56:14.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:14 smithi071 ceph-mon[36597]: pgmap v2147: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:56:14.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:14 smithi151 ceph-mon[36481]: pgmap v2147: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:56:14.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:14 smithi045 ceph-mon[31773]: pgmap v2147: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:56:15.383 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3828 sec 2024-03-28T18:56:16.383 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3829 sec 2024-03-28T18:56:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:16 smithi071 ceph-mon[36597]: pgmap v2148: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:56:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3184.294591 secs 2024-03-28T18:56:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:16 smithi151 ceph-mon[36481]: pgmap v2148: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:56:16.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3184.294591 secs 2024-03-28T18:56:16.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:16 smithi045 ceph-mon[31773]: pgmap v2148: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:56:16.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3184.294591 secs 2024-03-28T18:56:17.383 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3830 sec 2024-03-28T18:56:17.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2475.719333 secs 2024-03-28T18:56:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2475.719333 secs 2024-03-28T18:56:17.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2475.719333 secs 2024-03-28T18:56:18.383 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3831 sec 2024-03-28T18:56:18.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:18 smithi071 ceph-mon[36597]: pgmap v2149: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:18.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:18 smithi151 ceph-mon[36481]: pgmap v2149: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:18.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:18 smithi045 ceph-mon[31773]: pgmap v2149: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:19.383 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3832 sec 2024-03-28T18:56:20.384 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3833 sec 2024-03-28T18:56:20.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:20 smithi071 ceph-mon[36597]: pgmap v2150: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:20 smithi151 ceph-mon[36481]: pgmap v2150: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:20.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:20 smithi045 ceph-mon[31773]: pgmap v2150: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:21.384 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3834 sec 2024-03-28T18:56:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3189.294679 secs 2024-03-28T18:56:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3189.294679 secs 2024-03-28T18:56:21.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3189.294679 secs 2024-03-28T18:56:22.384 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3835 sec 2024-03-28T18:56:22.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:22 smithi071 ceph-mon[36597]: pgmap v2151: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:22.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2480.719420 secs 2024-03-28T18:56:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:22 smithi151 ceph-mon[36481]: pgmap v2151: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2480.719420 secs 2024-03-28T18:56:22.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:22 smithi045 ceph-mon[31773]: pgmap v2151: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:22.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2480.719420 secs 2024-03-28T18:56:23.384 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3836 sec 2024-03-28T18:56:24.384 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3837 sec 2024-03-28T18:56:24.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:24 smithi071 ceph-mon[36597]: pgmap v2152: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:24.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:24 smithi151 ceph-mon[36481]: pgmap v2152: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:24.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:24 smithi045 ceph-mon[31773]: pgmap v2152: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:25.384 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3838 sec 2024-03-28T18:56:26.384 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3839 sec 2024-03-28T18:56:26.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:26 smithi045 ceph-mon[31773]: pgmap v2153: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:26.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3194.294766 secs 2024-03-28T18:56:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:26 smithi071 ceph-mon[36597]: pgmap v2153: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3194.294766 secs 2024-03-28T18:56:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:26 smithi151 ceph-mon[36481]: pgmap v2153: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3194.294766 secs 2024-03-28T18:56:27.384 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3840 sec 2024-03-28T18:56:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2485.719506 secs 2024-03-28T18:56:27.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2485.719506 secs 2024-03-28T18:56:27.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2485.719506 secs 2024-03-28T18:56:28.385 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3841 sec 2024-03-28T18:56:28.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:28 smithi071 ceph-mon[36597]: pgmap v2154: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:28 smithi151 ceph-mon[36481]: pgmap v2154: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:28.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:28 smithi045 ceph-mon[31773]: pgmap v2154: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:29.385 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3842 sec 2024-03-28T18:56:30.385 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3843 sec 2024-03-28T18:56:30.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:30 smithi071 ceph-mon[36597]: pgmap v2155: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:30.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:30 smithi151 ceph-mon[36481]: pgmap v2155: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:30.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:30 smithi045 ceph-mon[31773]: pgmap v2155: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:31.385 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3844 sec 2024-03-28T18:56:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3199.294850 secs 2024-03-28T18:56:31.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3199.294850 secs 2024-03-28T18:56:31.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3199.294850 secs 2024-03-28T18:56:32.385 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3845 sec 2024-03-28T18:56:32.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:56:32 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:56:32.068+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:56:32.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:56:32 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:56:32.068+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:56:32.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:56:32 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:56:32.068+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:56:32.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:56:32 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:56:32.068+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:56:32.416 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 18:56:32 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T18:56:32.068+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:56:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:32 smithi151 ceph-mon[36481]: pgmap v2156: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:32.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2490.719596 secs 2024-03-28T18:56:32.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:32 smithi045 ceph-mon[31773]: pgmap v2156: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:32.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2490.719596 secs 2024-03-28T18:56:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:32 smithi071 ceph-mon[36597]: pgmap v2156: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:56:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2490.719596 secs 2024-03-28T18:56:33.386 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3846 sec 2024-03-28T18:56:34.386 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3847 sec 2024-03-28T18:56:34.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:34 smithi151 ceph-mon[36481]: pgmap v2157: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:34.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:34 smithi045 ceph-mon[31773]: pgmap v2157: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:35.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:34 smithi071 ceph-mon[36597]: pgmap v2157: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:35.386 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3848 sec 2024-03-28T18:56:36.386 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3849 sec 2024-03-28T18:56:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:36 smithi151 ceph-mon[36481]: pgmap v2158: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:56:36.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3204.294892 secs 2024-03-28T18:56:36.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:36 smithi045 ceph-mon[31773]: pgmap v2158: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:56:36.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3204.294892 secs 2024-03-28T18:56:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:36 smithi071 ceph-mon[36597]: pgmap v2158: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:56:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3204.294892 secs 2024-03-28T18:56:37.386 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3850 sec 2024-03-28T18:56:37.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2495.719655 secs 2024-03-28T18:56:38.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2495.719655 secs 2024-03-28T18:56:38.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2495.719655 secs 2024-03-28T18:56:38.386 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3851 sec 2024-03-28T18:56:38.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:38 smithi045 ceph-mon[31773]: pgmap v2159: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:56:39.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:38 smithi071 ceph-mon[36597]: pgmap v2159: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:56:39.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:38 smithi151 ceph-mon[36481]: pgmap v2159: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:56:39.386 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3852 sec 2024-03-28T18:56:40.386 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3853 sec 2024-03-28T18:56:40.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:40 smithi045 ceph-mon[31773]: pgmap v2160: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:56:41.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:40 smithi071 ceph-mon[36597]: pgmap v2160: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:56:41.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:40 smithi151 ceph-mon[36481]: pgmap v2160: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:56:41.386 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3854 sec 2024-03-28T18:56:41.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3209.294983 secs 2024-03-28T18:56:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3209.294983 secs 2024-03-28T18:56:42.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3209.294983 secs 2024-03-28T18:56:42.386 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3855 sec 2024-03-28T18:56:42.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:42 smithi045 ceph-mon[31773]: pgmap v2161: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:56:42.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2500.719716 secs 2024-03-28T18:56:42.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:42 smithi151 ceph-mon[36481]: pgmap v2161: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:56:42.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2500.719716 secs 2024-03-28T18:56:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:42 smithi071 ceph-mon[36597]: pgmap v2161: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:56:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2500.719716 secs 2024-03-28T18:56:43.386 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3856 sec 2024-03-28T18:56:43.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:43 smithi045 ceph-mon[31773]: pgmap v2162: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:56:44.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:43 smithi071 ceph-mon[36597]: pgmap v2162: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:56:44.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:43 smithi151 ceph-mon[36481]: pgmap v2162: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T18:56:44.387 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3857 sec 2024-03-28T18:56:45.387 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3858 sec 2024-03-28T18:56:46.387 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3859 sec 2024-03-28T18:56:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:46 smithi071 ceph-mon[36597]: pgmap v2163: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:56:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:46 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:56:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3214.295027 secs 2024-03-28T18:56:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:46 smithi151 ceph-mon[36481]: pgmap v2163: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:56:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:46 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:56:46.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3214.295027 secs 2024-03-28T18:56:46.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:46 smithi045 ceph-mon[31773]: pgmap v2163: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T18:56:46.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:46 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:56:46.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3214.295027 secs 2024-03-28T18:56:47.387 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3860 sec 2024-03-28T18:56:47.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2505.719781 secs 2024-03-28T18:56:47.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:47 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:56:47.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:47 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:56:47.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:47 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:56:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2505.719781 secs 2024-03-28T18:56:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:47 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:56:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:47 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:56:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:47 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:56:47.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2505.719781 secs 2024-03-28T18:56:47.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:47 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:56:47.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:47 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:56:47.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:47 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:56:48.387 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3861 sec 2024-03-28T18:56:48.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:48 smithi071 ceph-mon[36597]: pgmap v2164: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:48.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:48 smithi071 ceph-mon[36597]: pgmap v2165: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:48 smithi151 ceph-mon[36481]: pgmap v2164: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:48.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:48 smithi151 ceph-mon[36481]: pgmap v2165: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:48.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:48 smithi045 ceph-mon[31773]: pgmap v2164: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:56:48.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:48 smithi045 ceph-mon[31773]: pgmap v2165: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:49.387 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3862 sec 2024-03-28T18:56:50.388 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3863 sec 2024-03-28T18:56:50.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:50 smithi071 ceph-mon[36597]: pgmap v2166: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:50.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:50 smithi151 ceph-mon[36481]: pgmap v2166: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:50.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:50 smithi045 ceph-mon[31773]: pgmap v2166: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:51.388 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3864 sec 2024-03-28T18:56:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3219.295103 secs 2024-03-28T18:56:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3219.295103 secs 2024-03-28T18:56:51.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3219.295103 secs 2024-03-28T18:56:52.388 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3865 sec 2024-03-28T18:56:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:52 smithi071 ceph-mon[36597]: pgmap v2167: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2510.719857 secs 2024-03-28T18:56:52.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:52 smithi151 ceph-mon[36481]: pgmap v2167: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2510.719857 secs 2024-03-28T18:56:52.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:52 smithi045 ceph-mon[31773]: pgmap v2167: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:52.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2510.719857 secs 2024-03-28T18:56:53.388 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3866 sec 2024-03-28T18:56:54.388 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3867 sec 2024-03-28T18:56:54.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:54 smithi071 ceph-mon[36597]: pgmap v2168: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:54.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:54 smithi151 ceph-mon[36481]: pgmap v2168: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:54.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:54 smithi045 ceph-mon[31773]: pgmap v2168: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:55.388 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3868 sec 2024-03-28T18:56:56.388 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3869 sec 2024-03-28T18:56:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:56 smithi071 ceph-mon[36597]: pgmap v2169: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3224.295185 secs 2024-03-28T18:56:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:56 smithi151 ceph-mon[36481]: pgmap v2169: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:56.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3224.295185 secs 2024-03-28T18:56:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:56 smithi045 ceph-mon[31773]: pgmap v2169: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:56:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3224.295185 secs 2024-03-28T18:56:57.388 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3870 sec 2024-03-28T18:56:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2515.719917 secs 2024-03-28T18:56:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2515.719917 secs 2024-03-28T18:56:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2515.719917 secs 2024-03-28T18:56:58.389 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3871 sec 2024-03-28T18:56:58.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:56:58 smithi071 ceph-mon[36597]: pgmap v2170: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:56:58.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:56:58 smithi151 ceph-mon[36481]: pgmap v2170: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:56:58.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:56:58 smithi045 ceph-mon[31773]: pgmap v2170: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:56:59.389 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3872 sec 2024-03-28T18:57:00.389 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3873 sec 2024-03-28T18:57:00.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:00 smithi071 ceph-mon[36597]: pgmap v2171: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:00 smithi151 ceph-mon[36481]: pgmap v2171: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:00.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:00 smithi045 ceph-mon[31773]: pgmap v2171: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:01.389 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3874 sec 2024-03-28T18:57:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3229.295273 secs 2024-03-28T18:57:01.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3229.295273 secs 2024-03-28T18:57:01.908 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3229.295273 secs 2024-03-28T18:57:02.389 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3875 sec 2024-03-28T18:57:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:02 smithi071 ceph-mon[36597]: pgmap v2172: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2520.720003 secs 2024-03-28T18:57:02.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:02 smithi151 ceph-mon[36481]: pgmap v2172: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2520.720003 secs 2024-03-28T18:57:02.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:02 smithi045 ceph-mon[31773]: pgmap v2172: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:02.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2520.720003 secs 2024-03-28T18:57:03.390 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3876 sec 2024-03-28T18:57:04.389 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3877 sec 2024-03-28T18:57:04.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:04 smithi151 ceph-mon[36481]: pgmap v2173: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:04.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:04 smithi045 ceph-mon[31773]: pgmap v2173: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:05.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:04 smithi071 ceph-mon[36597]: pgmap v2173: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:05.390 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3878 sec 2024-03-28T18:57:06.390 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3879 sec 2024-03-28T18:57:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:06 smithi151 ceph-mon[36481]: pgmap v2174: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3234.295359 secs 2024-03-28T18:57:06.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:06 smithi045 ceph-mon[31773]: pgmap v2174: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:06.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3234.295359 secs 2024-03-28T18:57:07.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:06 smithi071 ceph-mon[36597]: pgmap v2174: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:07.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3234.295359 secs 2024-03-28T18:57:07.390 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3880 sec 2024-03-28T18:57:07.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2525.720091 secs 2024-03-28T18:57:08.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2525.720091 secs 2024-03-28T18:57:08.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2525.720091 secs 2024-03-28T18:57:08.390 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3881 sec 2024-03-28T18:57:08.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:08 smithi045 ceph-mon[31773]: pgmap v2175: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:09.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:08 smithi071 ceph-mon[36597]: pgmap v2175: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:09.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:08 smithi151 ceph-mon[36481]: pgmap v2175: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:09.390 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3882 sec 2024-03-28T18:57:10.390 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3883 sec 2024-03-28T18:57:10.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:10 smithi045 ceph-mon[31773]: pgmap v2176: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:10 smithi071 ceph-mon[36597]: pgmap v2176: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:11.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:10 smithi151 ceph-mon[36481]: pgmap v2176: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:11.391 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3884 sec 2024-03-28T18:57:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3239.295420 secs 2024-03-28T18:57:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3239.295420 secs 2024-03-28T18:57:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3239.295420 secs 2024-03-28T18:57:12.391 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3885 sec 2024-03-28T18:57:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:12 smithi045 ceph-mon[31773]: pgmap v2177: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2530.720154 secs 2024-03-28T18:57:12.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:12 smithi151 ceph-mon[36481]: pgmap v2177: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:12.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2530.720154 secs 2024-03-28T18:57:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:12 smithi071 ceph-mon[36597]: pgmap v2177: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2530.720154 secs 2024-03-28T18:57:13.391 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3886 sec 2024-03-28T18:57:14.391 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3887 sec 2024-03-28T18:57:14.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:14 smithi045 ceph-mon[31773]: pgmap v2178: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:15.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:14 smithi071 ceph-mon[36597]: pgmap v2178: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:15.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:14 smithi151 ceph-mon[36481]: pgmap v2178: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:15.391 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3888 sec 2024-03-28T18:57:15.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:15 smithi045 ceph-mon[31773]: pgmap v2179: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:16.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:15 smithi071 ceph-mon[36597]: pgmap v2179: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:16.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:15 smithi151 ceph-mon[36481]: pgmap v2179: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:16.391 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3889 sec 2024-03-28T18:57:16.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3244.295508 secs 2024-03-28T18:57:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3244.295508 secs 2024-03-28T18:57:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3244.295508 secs 2024-03-28T18:57:17.391 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3890 sec 2024-03-28T18:57:17.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2535.720244 secs 2024-03-28T18:57:17.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:17 smithi045 ceph-mon[31773]: pgmap v2180: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:17.960 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2535.720244 secs 2024-03-28T18:57:17.961 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:17 smithi071 ceph-mon[36597]: pgmap v2180: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2535.720244 secs 2024-03-28T18:57:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:17 smithi151 ceph-mon[36481]: pgmap v2180: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:18.391 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3891 sec 2024-03-28T18:57:19.392 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3892 sec 2024-03-28T18:57:20.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:20 smithi071 ceph-mon[36597]: pgmap v2181: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:20.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:20 smithi151 ceph-mon[36481]: pgmap v2181: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:20.392 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3893 sec 2024-03-28T18:57:20.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:20 smithi045 ceph-mon[31773]: pgmap v2181: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:21.392 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3894 sec 2024-03-28T18:57:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:22 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3249.295591 secs 2024-03-28T18:57:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:22 smithi071 ceph-mon[36597]: pgmap v2182: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2540.720279 secs 2024-03-28T18:57:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:22 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3249.295591 secs 2024-03-28T18:57:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:22 smithi151 ceph-mon[36481]: pgmap v2182: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:22.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2540.720279 secs 2024-03-28T18:57:22.392 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3895 sec 2024-03-28T18:57:22.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:22 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3249.295591 secs 2024-03-28T18:57:22.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:22 smithi045 ceph-mon[31773]: pgmap v2182: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:22.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2540.720279 secs 2024-03-28T18:57:23.393 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3896 sec 2024-03-28T18:57:24.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:24 smithi071 ceph-mon[36597]: pgmap v2183: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:24.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:24 smithi151 ceph-mon[36481]: pgmap v2183: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:24.392 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3897 sec 2024-03-28T18:57:24.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:24 smithi045 ceph-mon[31773]: pgmap v2183: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:25.393 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3898 sec 2024-03-28T18:57:26.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:26 smithi071 ceph-mon[36597]: pgmap v2184: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:26.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:26 smithi151 ceph-mon[36481]: pgmap v2184: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:26.393 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3899 sec 2024-03-28T18:57:26.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:26 smithi045 ceph-mon[31773]: pgmap v2184: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:27 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3254.295651 secs 2024-03-28T18:57:27.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2545.720366 secs 2024-03-28T18:57:27.393 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3900 sec 2024-03-28T18:57:27.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:27 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3254.295651 secs 2024-03-28T18:57:27.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2545.720366 secs 2024-03-28T18:57:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:27 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3254.295651 secs 2024-03-28T18:57:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2545.720366 secs 2024-03-28T18:57:28.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:28 smithi151 ceph-mon[36481]: pgmap v2185: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:28.393 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3901 sec 2024-03-28T18:57:28.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:28 smithi045 ceph-mon[31773]: pgmap v2185: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:28.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:28 smithi071 ceph-mon[36597]: pgmap v2185: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:29.393 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3902 sec 2024-03-28T18:57:30.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:30 smithi151 ceph-mon[36481]: pgmap v2186: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:30.393 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3903 sec 2024-03-28T18:57:30.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:30 smithi045 ceph-mon[31773]: pgmap v2186: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:30.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:30 smithi071 ceph-mon[36597]: pgmap v2186: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:31.393 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3904 sec 2024-03-28T18:57:32.394 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3905 sec 2024-03-28T18:57:32.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:32 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3259.295735 secs 2024-03-28T18:57:32.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:32 smithi045 ceph-mon[31773]: pgmap v2187: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:32.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2550.720427 secs 2024-03-28T18:57:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:32 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3259.295735 secs 2024-03-28T18:57:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:32 smithi071 ceph-mon[36597]: pgmap v2187: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2550.720427 secs 2024-03-28T18:57:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:32 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3259.295735 secs 2024-03-28T18:57:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:32 smithi151 ceph-mon[36481]: pgmap v2187: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2550.720427 secs 2024-03-28T18:57:33.394 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3906 sec 2024-03-28T18:57:34.394 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3907 sec 2024-03-28T18:57:34.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:34 smithi045 ceph-mon[31773]: pgmap v2188: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:34.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:34 smithi071 ceph-mon[36597]: pgmap v2188: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:34.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:34 smithi151 ceph-mon[36481]: pgmap v2188: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:35.394 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3908 sec 2024-03-28T18:57:36.394 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3909 sec 2024-03-28T18:57:36.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:36 smithi045 ceph-mon[31773]: pgmap v2189: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:36 smithi071 ceph-mon[36597]: pgmap v2189: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:36.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:36 smithi151 ceph-mon[36481]: pgmap v2189: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:37.394 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3910 sec 2024-03-28T18:57:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:37 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3264.295753 secs 2024-03-28T18:57:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2555.720516 secs 2024-03-28T18:57:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:37 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3264.295753 secs 2024-03-28T18:57:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2555.720516 secs 2024-03-28T18:57:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:37 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3264.295753 secs 2024-03-28T18:57:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2555.720516 secs 2024-03-28T18:57:38.394 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3911 sec 2024-03-28T18:57:38.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:38 smithi045 ceph-mon[31773]: pgmap v2190: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:38.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:38 smithi071 ceph-mon[36597]: pgmap v2190: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:38.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:38 smithi151 ceph-mon[36481]: pgmap v2190: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:39.394 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3912 sec 2024-03-28T18:57:40.395 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3913 sec 2024-03-28T18:57:40.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:40 smithi045 ceph-mon[31773]: pgmap v2191: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:40.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:40 smithi071 ceph-mon[36597]: pgmap v2191: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:40.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:40 smithi151 ceph-mon[36481]: pgmap v2191: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:41.395 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3914 sec 2024-03-28T18:57:42.375 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:42 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3269.295814 secs 2024-03-28T18:57:42.375 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:42 smithi151 ceph-mon[36481]: pgmap v2192: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:42.375 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2560.720607 secs 2024-03-28T18:57:42.397 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3915 sec 2024-03-28T18:57:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:42 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3269.295814 secs 2024-03-28T18:57:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:42 smithi045 ceph-mon[31773]: pgmap v2192: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:42.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2560.720607 secs 2024-03-28T18:57:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:42 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3269.295814 secs 2024-03-28T18:57:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:42 smithi071 ceph-mon[36597]: pgmap v2192: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2560.720607 secs 2024-03-28T18:57:43.395 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3916 sec 2024-03-28T18:57:44.395 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3917 sec 2024-03-28T18:57:44.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:44 smithi045 ceph-mon[31773]: pgmap v2193: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:44.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:44 smithi071 ceph-mon[36597]: pgmap v2193: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:44 smithi151 ceph-mon[36481]: pgmap v2193: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:45.395 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3918 sec 2024-03-28T18:57:46.395 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3919 sec 2024-03-28T18:57:46.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:46 smithi045 ceph-mon[31773]: pgmap v2194: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:46 smithi071 ceph-mon[36597]: pgmap v2194: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:46 smithi151 ceph-mon[36481]: pgmap v2194: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:57:47.395 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3920 sec 2024-03-28T18:57:47.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:47 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3274.295850 secs 2024-03-28T18:57:47.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2565.720667 secs 2024-03-28T18:57:47.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:47 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3274.295850 secs 2024-03-28T18:57:47.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2565.720667 secs 2024-03-28T18:57:47.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:47 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3274.295850 secs 2024-03-28T18:57:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2565.720667 secs 2024-03-28T18:57:48.396 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3921 sec 2024-03-28T18:57:48.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:48 smithi045 ceph-mon[31773]: pgmap v2195: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:48.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:48 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:57:48.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:48 smithi071 ceph-mon[36597]: pgmap v2195: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:48.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:48 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:57:48.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:48 smithi151 ceph-mon[36481]: pgmap v2195: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:48.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:48 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:57:49.396 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3922 sec 2024-03-28T18:57:49.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:49 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:57:49.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:49 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:57:49.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:49 smithi045 ceph-mon[31773]: pgmap v2196: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:49.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:49 smithi045 ceph-mon[31773]: pgmap v2197: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:57:49.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:49 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:57:49.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:49 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:57:49.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:49 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:57:49.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:49 smithi071 ceph-mon[36597]: pgmap v2196: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:49.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:49 smithi071 ceph-mon[36597]: pgmap v2197: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:57:49.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:49 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:57:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:49 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:57:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:49 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:57:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:49 smithi151 ceph-mon[36481]: pgmap v2196: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:57:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:49 smithi151 ceph-mon[36481]: pgmap v2197: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:57:49.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:49 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:57:50.396 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3923 sec 2024-03-28T18:57:51.396 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3924 sec 2024-03-28T18:57:51.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:51 smithi045 ceph-mon[31773]: pgmap v2198: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:57:51.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3279.295937 secs 2024-03-28T18:57:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:51 smithi071 ceph-mon[36597]: pgmap v2198: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:57:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3279.295937 secs 2024-03-28T18:57:52.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:51 smithi151 ceph-mon[36481]: pgmap v2198: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:57:52.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3279.295937 secs 2024-03-28T18:57:52.396 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3925 sec 2024-03-28T18:57:52.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2570.720708 secs 2024-03-28T18:57:53.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2570.720708 secs 2024-03-28T18:57:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2570.720708 secs 2024-03-28T18:57:53.396 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3926 sec 2024-03-28T18:57:53.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:53 smithi045 ceph-mon[31773]: pgmap v2199: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:57:54.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:53 smithi071 ceph-mon[36597]: pgmap v2199: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:57:54.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:53 smithi151 ceph-mon[36481]: pgmap v2199: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:57:54.396 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3927 sec 2024-03-28T18:57:54.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:54 smithi045 ceph-mon[31773]: pgmap v2200: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:57:55.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:54 smithi071 ceph-mon[36597]: pgmap v2200: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:57:55.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:54 smithi151 ceph-mon[36481]: pgmap v2200: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T18:57:55.397 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3928 sec 2024-03-28T18:57:56.397 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3929 sec 2024-03-28T18:57:56.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3284.296027 secs 2024-03-28T18:57:56.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3284.296027 secs 2024-03-28T18:57:56.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3284.296027 secs 2024-03-28T18:57:57.397 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3930 sec 2024-03-28T18:57:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:57 smithi071 ceph-mon[36597]: pgmap v2201: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:57:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2575.720797 secs 2024-03-28T18:57:57.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:57 smithi151 ceph-mon[36481]: pgmap v2201: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:57:57.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2575.720797 secs 2024-03-28T18:57:57.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:57 smithi045 ceph-mon[31773]: pgmap v2201: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:57:57.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2575.720797 secs 2024-03-28T18:57:58.397 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3931 sec 2024-03-28T18:57:59.397 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3932 sec 2024-03-28T18:57:59.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:57:59 smithi045 ceph-mon[31773]: pgmap v2202: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:58:00.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:57:59 smithi071 ceph-mon[36597]: pgmap v2202: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:58:00.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:57:59 smithi151 ceph-mon[36481]: pgmap v2202: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:58:00.397 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3933 sec 2024-03-28T18:58:01.397 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3934 sec 2024-03-28T18:58:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:01 smithi045 ceph-mon[31773]: pgmap v2203: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3289.296087 secs 2024-03-28T18:58:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:01 smithi071 ceph-mon[36597]: pgmap v2203: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3289.296087 secs 2024-03-28T18:58:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:01 smithi151 ceph-mon[36481]: pgmap v2203: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:02.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3289.296087 secs 2024-03-28T18:58:02.398 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3935 sec 2024-03-28T18:58:02.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2580.720836 secs 2024-03-28T18:58:03.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2580.720836 secs 2024-03-28T18:58:03.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2580.720836 secs 2024-03-28T18:58:03.398 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3936 sec 2024-03-28T18:58:03.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:03 smithi045 ceph-mon[31773]: pgmap v2204: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:04.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:03 smithi071 ceph-mon[36597]: pgmap v2204: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:04.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:03 smithi151 ceph-mon[36481]: pgmap v2204: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:04.398 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3937 sec 2024-03-28T18:58:04.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:04 smithi045 ceph-mon[31773]: pgmap v2205: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:05.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:04 smithi071 ceph-mon[36597]: pgmap v2205: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:05.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:04 smithi151 ceph-mon[36481]: pgmap v2205: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:05.398 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3938 sec 2024-03-28T18:58:06.398 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3939 sec 2024-03-28T18:58:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:06 smithi071 ceph-mon[36597]: 5 slow requests, 1 included below; oldest blocked for > 3294.296149 secs 2024-03-28T18:58:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:06 smithi071 ceph-mon[36597]: slow request 724.108008 seconds old, received at 2024-03-28T18:46:01.748685+0000: client_request(client.26723:1 getattr pAsLsXsFs #0x1 2024-03-28T18:46:01.745025+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:58:06.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:06 smithi151 ceph-mon[36481]: 5 slow requests, 1 included below; oldest blocked for > 3294.296149 secs 2024-03-28T18:58:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:06 smithi151 ceph-mon[36481]: slow request 724.108008 seconds old, received at 2024-03-28T18:46:01.748685+0000: client_request(client.26723:1 getattr pAsLsXsFs #0x1 2024-03-28T18:46:01.745025+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:58:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:06 smithi045 ceph-mon[31773]: 5 slow requests, 1 included below; oldest blocked for > 3294.296149 secs 2024-03-28T18:58:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:06 smithi045 ceph-mon[31773]: slow request 724.108008 seconds old, received at 2024-03-28T18:46:01.748685+0000: client_request(client.26723:1 getattr pAsLsXsFs #0x1 2024-03-28T18:46:01.745025+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:58:07.399 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3940 sec 2024-03-28T18:58:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:07 smithi071 ceph-mon[36597]: pgmap v2206: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2585.720910 secs 2024-03-28T18:58:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:07 smithi151 ceph-mon[36481]: pgmap v2206: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2585.720910 secs 2024-03-28T18:58:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:07 smithi045 ceph-mon[31773]: pgmap v2206: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2585.720910 secs 2024-03-28T18:58:08.398 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3941 sec 2024-03-28T18:58:09.399 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3942 sec 2024-03-28T18:58:09.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:09 smithi045 ceph-mon[31773]: pgmap v2207: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:10.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:09 smithi071 ceph-mon[36597]: pgmap v2207: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:09 smithi151 ceph-mon[36481]: pgmap v2207: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:10.399 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3943 sec 2024-03-28T18:58:11.399 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3944 sec 2024-03-28T18:58:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:11 smithi045 ceph-mon[31773]: pgmap v2208: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3299.296237 secs 2024-03-28T18:58:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:11 smithi071 ceph-mon[36597]: pgmap v2208: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3299.296237 secs 2024-03-28T18:58:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:11 smithi151 ceph-mon[36481]: pgmap v2208: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:12.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3299.296237 secs 2024-03-28T18:58:12.399 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3945 sec 2024-03-28T18:58:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2590.720999 secs 2024-03-28T18:58:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:12 smithi045 ceph-mon[31773]: pgmap v2209: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:12.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2590.720999 secs 2024-03-28T18:58:12.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:12 smithi151 ceph-mon[36481]: pgmap v2209: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2590.720999 secs 2024-03-28T18:58:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:12 smithi071 ceph-mon[36597]: pgmap v2209: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:13.399 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3946 sec 2024-03-28T18:58:14.399 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3947 sec 2024-03-28T18:58:15.399 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3948 sec 2024-03-28T18:58:15.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:15 smithi045 ceph-mon[31773]: pgmap v2210: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:16.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:15 smithi071 ceph-mon[36597]: pgmap v2210: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:16.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:15 smithi151 ceph-mon[36481]: pgmap v2210: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:16.400 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3949 sec 2024-03-28T18:58:16.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3304.296299 secs 2024-03-28T18:58:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3304.296299 secs 2024-03-28T18:58:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3304.296299 secs 2024-03-28T18:58:17.400 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3950 sec 2024-03-28T18:58:17.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:17 smithi045 ceph-mon[31773]: pgmap v2211: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:17.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2595.721036 secs 2024-03-28T18:58:17.961 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:17 smithi071 ceph-mon[36597]: pgmap v2211: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:17.961 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2595.721036 secs 2024-03-28T18:58:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:17 smithi151 ceph-mon[36481]: pgmap v2211: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2595.721036 secs 2024-03-28T18:58:18.400 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3951 sec 2024-03-28T18:58:18.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:18 smithi045 ceph-mon[31773]: pgmap v2212: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:19.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:18 smithi071 ceph-mon[36597]: pgmap v2212: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:19.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:18 smithi151 ceph-mon[36481]: pgmap v2212: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:19.400 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3952 sec 2024-03-28T18:58:20.400 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3953 sec 2024-03-28T18:58:21.400 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3954 sec 2024-03-28T18:58:21.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:21 smithi045 ceph-mon[31773]: pgmap v2213: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:21.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3309.296361 secs 2024-03-28T18:58:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:21 smithi071 ceph-mon[36597]: pgmap v2213: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3309.296361 secs 2024-03-28T18:58:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:21 smithi151 ceph-mon[36481]: pgmap v2213: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3309.296361 secs 2024-03-28T18:58:22.400 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3955 sec 2024-03-28T18:58:22.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2600.721124 secs 2024-03-28T18:58:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2600.721124 secs 2024-03-28T18:58:23.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2600.721124 secs 2024-03-28T18:58:23.401 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3956 sec 2024-03-28T18:58:23.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:23 smithi045 ceph-mon[31773]: pgmap v2214: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:24.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:23 smithi071 ceph-mon[36597]: pgmap v2214: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:24.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:23 smithi151 ceph-mon[36481]: pgmap v2214: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:24.401 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3957 sec 2024-03-28T18:58:24.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:24 smithi045 ceph-mon[31773]: pgmap v2215: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:25.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:24 smithi071 ceph-mon[36597]: pgmap v2215: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:25.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:24 smithi151 ceph-mon[36481]: pgmap v2215: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:25.401 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3958 sec 2024-03-28T18:58:26.401 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3959 sec 2024-03-28T18:58:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3314.296422 secs 2024-03-28T18:58:26.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3314.296422 secs 2024-03-28T18:58:26.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3314.296422 secs 2024-03-28T18:58:27.401 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3960 sec 2024-03-28T18:58:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:27 smithi071 ceph-mon[36597]: pgmap v2216: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2605.721185 secs 2024-03-28T18:58:27.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:27 smithi151 ceph-mon[36481]: pgmap v2216: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2605.721185 secs 2024-03-28T18:58:27.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:27 smithi045 ceph-mon[31773]: pgmap v2216: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:27.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2605.721185 secs 2024-03-28T18:58:28.401 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3961 sec 2024-03-28T18:58:29.401 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3962 sec 2024-03-28T18:58:29.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:29 smithi045 ceph-mon[31773]: pgmap v2217: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:30.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:29 smithi071 ceph-mon[36597]: pgmap v2217: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:30.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:29 smithi151 ceph-mon[36481]: pgmap v2217: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:30.402 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3963 sec 2024-03-28T18:58:30.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:30 smithi045 ceph-mon[31773]: pgmap v2218: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:31.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:30 smithi071 ceph-mon[36597]: pgmap v2218: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:31.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:30 smithi151 ceph-mon[36481]: pgmap v2218: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:31.402 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3964 sec 2024-03-28T18:58:31.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3319.296485 secs 2024-03-28T18:58:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3319.296485 secs 2024-03-28T18:58:32.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3319.296485 secs 2024-03-28T18:58:32.402 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3965 sec 2024-03-28T18:58:32.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2610.721244 secs 2024-03-28T18:58:32.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:32 smithi045 ceph-mon[31773]: pgmap v2219: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2610.721244 secs 2024-03-28T18:58:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:32 smithi071 ceph-mon[36597]: pgmap v2219: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:33.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2610.721244 secs 2024-03-28T18:58:33.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:32 smithi151 ceph-mon[36481]: pgmap v2219: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:33.402 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3966 sec 2024-03-28T18:58:34.402 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3967 sec 2024-03-28T18:58:35.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:34 smithi071 ceph-mon[36597]: pgmap v2220: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:34 smithi151 ceph-mon[36481]: pgmap v2220: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:34 smithi045 ceph-mon[31773]: pgmap v2220: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:35.402 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3968 sec 2024-03-28T18:58:36.402 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3969 sec 2024-03-28T18:58:36.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3324.296545 secs 2024-03-28T18:58:36.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3324.296545 secs 2024-03-28T18:58:36.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3324.296545 secs 2024-03-28T18:58:37.403 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3970 sec 2024-03-28T18:58:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:37 smithi071 ceph-mon[36597]: pgmap v2221: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2615.721285 secs 2024-03-28T18:58:37.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:37 smithi151 ceph-mon[36481]: pgmap v2221: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2615.721285 secs 2024-03-28T18:58:37.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:37 smithi045 ceph-mon[31773]: pgmap v2221: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:37.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2615.721285 secs 2024-03-28T18:58:38.403 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3971 sec 2024-03-28T18:58:39.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:38 smithi071 ceph-mon[36597]: pgmap v2222: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:39.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:38 smithi151 ceph-mon[36481]: pgmap v2222: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:39.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:38 smithi045 ceph-mon[31773]: pgmap v2222: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:39.403 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3972 sec 2024-03-28T18:58:40.403 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3973 sec 2024-03-28T18:58:41.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:40 smithi071 ceph-mon[36597]: pgmap v2223: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:41.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:40 smithi151 ceph-mon[36481]: pgmap v2223: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:41.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:40 smithi045 ceph-mon[31773]: pgmap v2223: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:41.403 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3974 sec 2024-03-28T18:58:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3329.296630 secs 2024-03-28T18:58:42.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3329.296630 secs 2024-03-28T18:58:42.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3329.296630 secs 2024-03-28T18:58:42.403 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3975 sec 2024-03-28T18:58:42.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2620.721347 secs 2024-03-28T18:58:42.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:42 smithi151 ceph-mon[36481]: pgmap v2224: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2620.721347 secs 2024-03-28T18:58:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:42 smithi071 ceph-mon[36597]: pgmap v2224: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2620.721347 secs 2024-03-28T18:58:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:42 smithi045 ceph-mon[31773]: pgmap v2224: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:43.403 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3976 sec 2024-03-28T18:58:44.403 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3977 sec 2024-03-28T18:58:45.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:44 smithi071 ceph-mon[36597]: pgmap v2225: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:45.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:44 smithi151 ceph-mon[36481]: pgmap v2225: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:45.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:44 smithi045 ceph-mon[31773]: pgmap v2225: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:45.404 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3978 sec 2024-03-28T18:58:46.404 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3979 sec 2024-03-28T18:58:46.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3334.296688 secs 2024-03-28T18:58:46.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3334.296688 secs 2024-03-28T18:58:46.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3334.296688 secs 2024-03-28T18:58:47.404 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3980 sec 2024-03-28T18:58:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:47 smithi071 ceph-mon[36597]: pgmap v2226: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2625.721435 secs 2024-03-28T18:58:47.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:47 smithi151 ceph-mon[36481]: pgmap v2226: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2625.721435 secs 2024-03-28T18:58:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:47 smithi045 ceph-mon[31773]: pgmap v2226: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:58:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2625.721435 secs 2024-03-28T18:58:48.404 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3981 sec 2024-03-28T18:58:49.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:48 smithi071 ceph-mon[36597]: pgmap v2227: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:49.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:48 smithi151 ceph-mon[36481]: pgmap v2227: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:49.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:48 smithi045 ceph-mon[31773]: pgmap v2227: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:58:49.404 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3982 sec 2024-03-28T18:58:50.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:49 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:58:50.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:49 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:58:50.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:49 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:58:50.404 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3983 sec 2024-03-28T18:58:51.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:50 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:58:51.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:50 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:58:51.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:50 smithi071 ceph-mon[36597]: pgmap v2228: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:58:51.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:50 smithi071 ceph-mon[36597]: pgmap v2229: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:58:51.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:50 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:58:51.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:50 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:58:51.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:50 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:58:51.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:50 smithi151 ceph-mon[36481]: pgmap v2228: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:58:51.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:50 smithi151 ceph-mon[36481]: pgmap v2229: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:58:51.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:50 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:58:51.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:50 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:58:51.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:50 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:58:51.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:50 smithi045 ceph-mon[31773]: pgmap v2228: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T18:58:51.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:50 smithi045 ceph-mon[31773]: pgmap v2229: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:58:51.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:50 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:58:51.404 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3984 sec 2024-03-28T18:58:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3339.296751 secs 2024-03-28T18:58:52.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3339.296751 secs 2024-03-28T18:58:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3339.296751 secs 2024-03-28T18:58:52.405 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3985 sec 2024-03-28T18:58:53.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2630.721511 secs 2024-03-28T18:58:53.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:52 smithi071 ceph-mon[36597]: pgmap v2230: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:58:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2630.721511 secs 2024-03-28T18:58:53.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:52 smithi151 ceph-mon[36481]: pgmap v2230: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:58:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2630.721511 secs 2024-03-28T18:58:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:52 smithi045 ceph-mon[31773]: pgmap v2230: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:58:53.405 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3986 sec 2024-03-28T18:58:54.405 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3987 sec 2024-03-28T18:58:55.405 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3988 sec 2024-03-28T18:58:55.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:55 smithi045 ceph-mon[31773]: pgmap v2231: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:58:56.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:55 smithi071 ceph-mon[36597]: pgmap v2231: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:58:56.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:55 smithi151 ceph-mon[36481]: pgmap v2231: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:58:56.405 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3989 sec 2024-03-28T18:58:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3344.296811 secs 2024-03-28T18:58:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3344.296811 secs 2024-03-28T18:58:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3344.296811 secs 2024-03-28T18:58:57.405 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3990 sec 2024-03-28T18:58:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:57 smithi045 ceph-mon[31773]: pgmap v2232: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:58:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2635.721613 secs 2024-03-28T18:58:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:57 smithi071 ceph-mon[36597]: pgmap v2232: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:58:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2635.721613 secs 2024-03-28T18:58:58.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:57 smithi151 ceph-mon[36481]: pgmap v2232: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:58:58.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2635.721613 secs 2024-03-28T18:58:58.406 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3991 sec 2024-03-28T18:58:58.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:58:58 smithi045 ceph-mon[31773]: pgmap v2233: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:58:59.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:58:58 smithi071 ceph-mon[36597]: pgmap v2233: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:58:59.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:58:58 smithi151 ceph-mon[36481]: pgmap v2233: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:58:59.406 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3992 sec 2024-03-28T18:59:00.406 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3993 sec 2024-03-28T18:59:01.406 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3994 sec 2024-03-28T18:59:01.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:01 smithi045 ceph-mon[31773]: pgmap v2234: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:59:01.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3349.296869 secs 2024-03-28T18:59:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:01 smithi071 ceph-mon[36597]: pgmap v2234: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:59:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3349.296869 secs 2024-03-28T18:59:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:01 smithi151 ceph-mon[36481]: pgmap v2234: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T18:59:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3349.296869 secs 2024-03-28T18:59:02.406 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3995 sec 2024-03-28T18:59:02.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2640.721702 secs 2024-03-28T18:59:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2640.721702 secs 2024-03-28T18:59:02.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2640.721702 secs 2024-03-28T18:59:03.406 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3996 sec 2024-03-28T18:59:03.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:03 smithi045 ceph-mon[31773]: pgmap v2235: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:03.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:03 smithi071 ceph-mon[36597]: pgmap v2235: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:03.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:03 smithi151 ceph-mon[36481]: pgmap v2235: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:04.407 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3997 sec 2024-03-28T18:59:05.407 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3998 sec 2024-03-28T18:59:05.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:05 smithi045 ceph-mon[31773]: pgmap v2236: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:05.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:05 smithi071 ceph-mon[36597]: pgmap v2236: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:05.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:05 smithi151 ceph-mon[36481]: pgmap v2236: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:06.407 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 3999 sec 2024-03-28T18:59:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3354.296930 secs 2024-03-28T18:59:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3354.296930 secs 2024-03-28T18:59:06.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3354.296930 secs 2024-03-28T18:59:07.407 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4000 sec 2024-03-28T18:59:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:07 smithi045 ceph-mon[31773]: pgmap v2237: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2645.721789 secs 2024-03-28T18:59:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:07 smithi071 ceph-mon[36597]: pgmap v2237: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2645.721789 secs 2024-03-28T18:59:07.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:07 smithi151 ceph-mon[36481]: pgmap v2237: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:07.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2645.721789 secs 2024-03-28T18:59:08.407 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4001 sec 2024-03-28T18:59:09.407 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4002 sec 2024-03-28T18:59:09.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:09 smithi045 ceph-mon[31773]: pgmap v2238: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:09.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:09 smithi071 ceph-mon[36597]: pgmap v2238: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:09.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:09 smithi151 ceph-mon[36481]: pgmap v2238: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:10.407 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4003 sec 2024-03-28T18:59:11.407 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4004 sec 2024-03-28T18:59:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:11 smithi045 ceph-mon[31773]: pgmap v2239: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3359.296991 secs 2024-03-28T18:59:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:11 smithi071 ceph-mon[36597]: pgmap v2239: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3359.296991 secs 2024-03-28T18:59:11.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:11 smithi151 ceph-mon[36481]: pgmap v2239: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:11.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3359.296991 secs 2024-03-28T18:59:12.407 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4005 sec 2024-03-28T18:59:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2650.721898 secs 2024-03-28T18:59:12.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2650.721898 secs 2024-03-28T18:59:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2650.721898 secs 2024-03-28T18:59:13.407 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4006 sec 2024-03-28T18:59:13.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:13 smithi045 ceph-mon[31773]: pgmap v2240: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:13.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:13 smithi071 ceph-mon[36597]: pgmap v2240: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:13.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:13 smithi151 ceph-mon[36481]: pgmap v2240: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:14.408 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4007 sec 2024-03-28T18:59:15.408 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4008 sec 2024-03-28T18:59:15.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:15 smithi045 ceph-mon[31773]: pgmap v2241: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:15.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:15 smithi071 ceph-mon[36597]: pgmap v2241: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:15.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:15 smithi151 ceph-mon[36481]: pgmap v2241: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:16.408 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4009 sec 2024-03-28T18:59:16.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3364.297006 secs 2024-03-28T18:59:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3364.297006 secs 2024-03-28T18:59:16.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3364.297006 secs 2024-03-28T18:59:17.408 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4010 sec 2024-03-28T18:59:17.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:17 smithi045 ceph-mon[31773]: pgmap v2242: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:17.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2655.721989 secs 2024-03-28T18:59:17.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:17 smithi071 ceph-mon[36597]: pgmap v2242: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:17.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2655.721989 secs 2024-03-28T18:59:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:17 smithi151 ceph-mon[36481]: pgmap v2242: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2655.721989 secs 2024-03-28T18:59:18.408 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4011 sec 2024-03-28T18:59:19.408 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4012 sec 2024-03-28T18:59:19.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:19 smithi045 ceph-mon[31773]: pgmap v2243: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:19.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:19 smithi071 ceph-mon[36597]: pgmap v2243: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:19.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:19 smithi151 ceph-mon[36481]: pgmap v2243: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:20.408 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4013 sec 2024-03-28T18:59:21.409 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4014 sec 2024-03-28T18:59:21.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:21 smithi045 ceph-mon[31773]: pgmap v2244: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:21.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3369.297069 secs 2024-03-28T18:59:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:21 smithi071 ceph-mon[36597]: pgmap v2244: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3369.297069 secs 2024-03-28T18:59:21.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:21 smithi151 ceph-mon[36481]: pgmap v2244: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:21.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3369.297069 secs 2024-03-28T18:59:22.409 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4015 sec 2024-03-28T18:59:22.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2660.722051 secs 2024-03-28T18:59:22.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2660.722051 secs 2024-03-28T18:59:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2660.722051 secs 2024-03-28T18:59:23.409 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4016 sec 2024-03-28T18:59:23.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:23 smithi045 ceph-mon[31773]: pgmap v2245: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:23.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:23 smithi071 ceph-mon[36597]: pgmap v2245: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:23.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:23 smithi151 ceph-mon[36481]: pgmap v2245: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:24.409 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4017 sec 2024-03-28T18:59:25.409 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4018 sec 2024-03-28T18:59:25.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:25 smithi071 ceph-mon[36597]: pgmap v2246: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:25.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:25 smithi151 ceph-mon[36481]: pgmap v2246: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:25.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:25 smithi045 ceph-mon[31773]: pgmap v2246: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:26.409 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4019 sec 2024-03-28T18:59:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3374.297086 secs 2024-03-28T18:59:26.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3374.297086 secs 2024-03-28T18:59:26.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3374.297086 secs 2024-03-28T18:59:27.409 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4020 sec 2024-03-28T18:59:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:27 smithi071 ceph-mon[36597]: pgmap v2247: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2665.722112 secs 2024-03-28T18:59:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:27 smithi151 ceph-mon[36481]: pgmap v2247: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2665.722112 secs 2024-03-28T18:59:27.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:27 smithi045 ceph-mon[31773]: pgmap v2247: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:27.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2665.722112 secs 2024-03-28T18:59:28.409 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4021 sec 2024-03-28T18:59:29.410 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4022 sec 2024-03-28T18:59:29.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:29 smithi071 ceph-mon[36597]: pgmap v2248: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:29.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:29 smithi151 ceph-mon[36481]: pgmap v2248: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:29.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:29 smithi045 ceph-mon[31773]: pgmap v2248: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:30.410 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4023 sec 2024-03-28T18:59:31.410 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4024 sec 2024-03-28T18:59:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:31 smithi071 ceph-mon[36597]: pgmap v2249: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3379.297173 secs 2024-03-28T18:59:31.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:31 smithi151 ceph-mon[36481]: pgmap v2249: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:31.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3379.297173 secs 2024-03-28T18:59:31.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:31 smithi045 ceph-mon[31773]: pgmap v2249: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:31.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3379.297173 secs 2024-03-28T18:59:32.410 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4025 sec 2024-03-28T18:59:32.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2670.722174 secs 2024-03-28T18:59:32.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2670.722174 secs 2024-03-28T18:59:32.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2670.722174 secs 2024-03-28T18:59:33.410 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4026 sec 2024-03-28T18:59:33.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:33 smithi071 ceph-mon[36597]: pgmap v2250: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:33.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:33 smithi151 ceph-mon[36481]: pgmap v2250: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:33.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:33 smithi045 ceph-mon[31773]: pgmap v2250: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:34.410 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4027 sec 2024-03-28T18:59:35.410 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4028 sec 2024-03-28T18:59:35.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:35 smithi071 ceph-mon[36597]: pgmap v2251: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:35.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:35 smithi151 ceph-mon[36481]: pgmap v2251: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:35.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:35 smithi045 ceph-mon[31773]: pgmap v2251: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:36.410 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4029 sec 2024-03-28T18:59:36.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:36 smithi071 ceph-mon[36597]: 5 slow requests, 1 included below; oldest blocked for > 3384.297262 secs 2024-03-28T18:59:36.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:36 smithi071 ceph-mon[36597]: slow request 1444.384236 seconds old, received at 2024-03-28T18:35:31.473570+0000: client_request(client.26711:1 getattr pAsLsXsFs #0x1 2024-03-28T18:35:31.469661+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:59:36.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:36 smithi151 ceph-mon[36481]: 5 slow requests, 1 included below; oldest blocked for > 3384.297262 secs 2024-03-28T18:59:36.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:36 smithi151 ceph-mon[36481]: slow request 1444.384236 seconds old, received at 2024-03-28T18:35:31.473570+0000: client_request(client.26711:1 getattr pAsLsXsFs #0x1 2024-03-28T18:35:31.469661+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:59:36.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:36 smithi045 ceph-mon[31773]: 5 slow requests, 1 included below; oldest blocked for > 3384.297262 secs 2024-03-28T18:59:36.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:36 smithi045 ceph-mon[31773]: slow request 1444.384236 seconds old, received at 2024-03-28T18:35:31.473570+0000: client_request(client.26711:1 getattr pAsLsXsFs #0x1 2024-03-28T18:35:31.469661+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T18:59:37.411 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4030 sec 2024-03-28T18:59:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:37 smithi071 ceph-mon[36597]: pgmap v2252: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2675.722275 secs 2024-03-28T18:59:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:37 smithi151 ceph-mon[36481]: pgmap v2252: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2675.722275 secs 2024-03-28T18:59:37.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:37 smithi045 ceph-mon[31773]: pgmap v2252: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:37.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2675.722275 secs 2024-03-28T18:59:38.411 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4031 sec 2024-03-28T18:59:39.411 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4032 sec 2024-03-28T18:59:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:39 smithi071 ceph-mon[36597]: pgmap v2253: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:39.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:39 smithi151 ceph-mon[36481]: pgmap v2253: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:39.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:39 smithi045 ceph-mon[31773]: pgmap v2253: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:40.411 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4033 sec 2024-03-28T18:59:41.411 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4034 sec 2024-03-28T18:59:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:41 smithi071 ceph-mon[36597]: pgmap v2254: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3389.297350 secs 2024-03-28T18:59:41.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:41 smithi151 ceph-mon[36481]: pgmap v2254: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:41.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3389.297350 secs 2024-03-28T18:59:41.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:41 smithi045 ceph-mon[31773]: pgmap v2254: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:41.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3389.297350 secs 2024-03-28T18:59:42.412 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4035 sec 2024-03-28T18:59:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2680.722339 secs 2024-03-28T18:59:42.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2680.722339 secs 2024-03-28T18:59:42.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2680.722339 secs 2024-03-28T18:59:43.412 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4036 sec 2024-03-28T18:59:43.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:43 smithi071 ceph-mon[36597]: pgmap v2255: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:43.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:43 smithi151 ceph-mon[36481]: pgmap v2255: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:43.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:43 smithi045 ceph-mon[31773]: pgmap v2255: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:44.412 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4037 sec 2024-03-28T18:59:45.412 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4038 sec 2024-03-28T18:59:45.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:45 smithi071 ceph-mon[36597]: pgmap v2256: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:45.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:45 smithi151 ceph-mon[36481]: pgmap v2256: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:45.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:45 smithi045 ceph-mon[31773]: pgmap v2256: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:46.412 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4039 sec 2024-03-28T18:59:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3394.297440 secs 2024-03-28T18:59:46.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3394.297440 secs 2024-03-28T18:59:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3394.297440 secs 2024-03-28T18:59:47.412 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4040 sec 2024-03-28T18:59:47.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:47 smithi071 ceph-mon[36597]: pgmap v2257: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:47.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2685.722379 secs 2024-03-28T18:59:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:47 smithi151 ceph-mon[36481]: pgmap v2257: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:47.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2685.722379 secs 2024-03-28T18:59:47.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:47 smithi045 ceph-mon[31773]: pgmap v2257: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:47.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2685.722379 secs 2024-03-28T18:59:48.412 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4041 sec 2024-03-28T18:59:49.412 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4042 sec 2024-03-28T18:59:49.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:49 smithi071 ceph-mon[36597]: pgmap v2258: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:49.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:49 smithi151 ceph-mon[36481]: pgmap v2258: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:49.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:49 smithi045 ceph-mon[31773]: pgmap v2258: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:50.413 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4043 sec 2024-03-28T18:59:51.413 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4044 sec 2024-03-28T18:59:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:51 smithi071 ceph-mon[36597]: pgmap v2259: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:51 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:59:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3399.297469 secs 2024-03-28T18:59:51.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:51 smithi151 ceph-mon[36481]: pgmap v2259: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:51.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:51 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:59:51.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3399.297469 secs 2024-03-28T18:59:51.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:51 smithi045 ceph-mon[31773]: pgmap v2259: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T18:59:51.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:51 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:59:51.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3399.297469 secs 2024-03-28T18:59:52.413 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4045 sec 2024-03-28T18:59:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2690.722463 secs 2024-03-28T18:59:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:52 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:59:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:52 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:59:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:52 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:59:52.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2690.722463 secs 2024-03-28T18:59:52.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:52 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:59:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:52 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:59:52.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:52 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:59:52.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2690.722463 secs 2024-03-28T18:59:52.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:52 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:59:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:52 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:59:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:52 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T18:59:53.413 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4046 sec 2024-03-28T18:59:53.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:53 smithi071 ceph-mon[36597]: pgmap v2260: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:53.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:53 smithi071 ceph-mon[36597]: pgmap v2261: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:59:53.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:53 smithi151 ceph-mon[36481]: pgmap v2260: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:53.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:53 smithi151 ceph-mon[36481]: pgmap v2261: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:59:53.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:53 smithi045 ceph-mon[31773]: pgmap v2260: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T18:59:53.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:53 smithi045 ceph-mon[31773]: pgmap v2261: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:59:54.413 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4047 sec 2024-03-28T18:59:55.413 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4048 sec 2024-03-28T18:59:55.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:55 smithi071 ceph-mon[36597]: pgmap v2262: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:59:55.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:55 smithi151 ceph-mon[36481]: pgmap v2262: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:59:55.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:55 smithi045 ceph-mon[31773]: pgmap v2262: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:59:56.413 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4049 sec 2024-03-28T18:59:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3404.297531 secs 2024-03-28T18:59:56.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3404.297531 secs 2024-03-28T18:59:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3404.297531 secs 2024-03-28T18:59:57.414 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4050 sec 2024-03-28T18:59:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:57 smithi071 ceph-mon[36597]: pgmap v2263: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:59:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2695.722551 secs 2024-03-28T18:59:57.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:57 smithi151 ceph-mon[36481]: pgmap v2263: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:59:57.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2695.722551 secs 2024-03-28T18:59:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:57 smithi045 ceph-mon[31773]: pgmap v2263: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:59:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2695.722551 secs 2024-03-28T18:59:58.414 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4051 sec 2024-03-28T18:59:59.414 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4052 sec 2024-03-28T18:59:59.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 18:59:59 smithi071 ceph-mon[36597]: pgmap v2264: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:59:59.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 18:59:59 smithi151 ceph-mon[36481]: pgmap v2264: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T18:59:59.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 18:59:59 smithi045 ceph-mon[31773]: pgmap v2264: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:00:00.414 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4053 sec 2024-03-28T19:00:00.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:00 smithi151 ceph-mon[36481]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-03-28T19:00:00.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:00 smithi045 ceph-mon[31773]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-03-28T19:00:01.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:00 smithi071 ceph-mon[36597]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-03-28T19:00:01.414 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4054 sec 2024-03-28T19:00:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:01 smithi071 ceph-mon[36597]: pgmap v2265: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:00:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3409.297595 secs 2024-03-28T19:00:01.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:01 smithi151 ceph-mon[36481]: pgmap v2265: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:00:01.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3409.297595 secs 2024-03-28T19:00:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:01 smithi045 ceph-mon[31773]: pgmap v2265: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:00:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3409.297595 secs 2024-03-28T19:00:02.414 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4055 sec 2024-03-28T19:00:02.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2700.722644 secs 2024-03-28T19:00:03.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2700.722644 secs 2024-03-28T19:00:03.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2700.722644 secs 2024-03-28T19:00:03.414 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4056 sec 2024-03-28T19:00:03.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:03 smithi045 ceph-mon[31773]: pgmap v2266: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:00:04.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:03 smithi071 ceph-mon[36597]: pgmap v2266: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:00:04.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:03 smithi151 ceph-mon[36481]: pgmap v2266: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:00:04.414 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4057 sec 2024-03-28T19:00:05.415 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4058 sec 2024-03-28T19:00:05.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:05 smithi045 ceph-mon[31773]: pgmap v2267: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:06.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:05 smithi071 ceph-mon[36597]: pgmap v2267: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:06.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:05 smithi151 ceph-mon[36481]: pgmap v2267: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:06.415 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4059 sec 2024-03-28T19:00:06.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3414.297657 secs 2024-03-28T19:00:07.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3414.297657 secs 2024-03-28T19:00:07.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3414.297657 secs 2024-03-28T19:00:07.415 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4060 sec 2024-03-28T19:00:07.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:07 smithi045 ceph-mon[31773]: pgmap v2268: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:07.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2705.722733 secs 2024-03-28T19:00:08.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:07 smithi071 ceph-mon[36597]: pgmap v2268: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:08.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2705.722733 secs 2024-03-28T19:00:08.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:07 smithi151 ceph-mon[36481]: pgmap v2268: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:08.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2705.722733 secs 2024-03-28T19:00:08.415 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4061 sec 2024-03-28T19:00:09.415 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4062 sec 2024-03-28T19:00:09.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:09 smithi045 ceph-mon[31773]: pgmap v2269: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:10.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:09 smithi071 ceph-mon[36597]: pgmap v2269: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:10.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:09 smithi151 ceph-mon[36481]: pgmap v2269: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:10.415 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4063 sec 2024-03-28T19:00:11.415 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4064 sec 2024-03-28T19:00:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:11 smithi045 ceph-mon[31773]: pgmap v2270: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3419.297722 secs 2024-03-28T19:00:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:11 smithi071 ceph-mon[36597]: pgmap v2270: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3419.297722 secs 2024-03-28T19:00:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:11 smithi151 ceph-mon[36481]: pgmap v2270: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:12.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3419.297722 secs 2024-03-28T19:00:12.416 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4065 sec 2024-03-28T19:00:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2710.722825 secs 2024-03-28T19:00:12.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2710.722825 secs 2024-03-28T19:00:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2710.722825 secs 2024-03-28T19:00:13.416 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4066 sec 2024-03-28T19:00:13.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:13 smithi045 ceph-mon[31773]: pgmap v2271: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:14.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:13 smithi071 ceph-mon[36597]: pgmap v2271: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:13 smithi151 ceph-mon[36481]: pgmap v2271: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:14.416 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4067 sec 2024-03-28T19:00:15.416 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4068 sec 2024-03-28T19:00:15.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:15 smithi045 ceph-mon[31773]: pgmap v2272: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:16.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:15 smithi071 ceph-mon[36597]: pgmap v2272: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:16.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:15 smithi151 ceph-mon[36481]: pgmap v2272: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:16.416 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4069 sec 2024-03-28T19:00:16.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3424.297783 secs 2024-03-28T19:00:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3424.297783 secs 2024-03-28T19:00:17.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3424.297783 secs 2024-03-28T19:00:17.417 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4070 sec 2024-03-28T19:00:17.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:17 smithi045 ceph-mon[31773]: pgmap v2273: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:17.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2715.722884 secs 2024-03-28T19:00:17.961 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:17 smithi071 ceph-mon[36597]: pgmap v2273: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:17.961 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2715.722884 secs 2024-03-28T19:00:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:17 smithi151 ceph-mon[36481]: pgmap v2273: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2715.722884 secs 2024-03-28T19:00:18.416 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4071 sec 2024-03-28T19:00:19.416 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4072 sec 2024-03-28T19:00:19.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:19 smithi045 ceph-mon[31773]: pgmap v2274: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:20.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:19 smithi071 ceph-mon[36597]: pgmap v2274: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:20.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:19 smithi151 ceph-mon[36481]: pgmap v2274: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:20.416 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4073 sec 2024-03-28T19:00:20.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:20 smithi045 ceph-mon[31773]: pgmap v2275: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:21.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:20 smithi071 ceph-mon[36597]: pgmap v2275: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:21.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:20 smithi151 ceph-mon[36481]: pgmap v2275: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:21.417 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4074 sec 2024-03-28T19:00:21.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3429.297873 secs 2024-03-28T19:00:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3429.297873 secs 2024-03-28T19:00:22.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3429.297873 secs 2024-03-28T19:00:22.417 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4075 sec 2024-03-28T19:00:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2720.722977 secs 2024-03-28T19:00:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:22 smithi071 ceph-mon[36597]: pgmap v2276: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:23.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2720.722977 secs 2024-03-28T19:00:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:22 smithi151 ceph-mon[36481]: pgmap v2276: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2720.722977 secs 2024-03-28T19:00:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:22 smithi045 ceph-mon[31773]: pgmap v2276: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:23.417 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4076 sec 2024-03-28T19:00:24.417 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4077 sec 2024-03-28T19:00:25.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:25 smithi071 ceph-mon[36597]: pgmap v2277: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:25.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:25 smithi151 ceph-mon[36481]: pgmap v2277: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:25.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:25 smithi045 ceph-mon[31773]: pgmap v2277: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:25.417 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4078 sec 2024-03-28T19:00:26.417 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4079 sec 2024-03-28T19:00:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:27 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3434.297935 secs 2024-03-28T19:00:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:27 smithi071 ceph-mon[36597]: pgmap v2278: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:27.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:27 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3434.297935 secs 2024-03-28T19:00:27.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:27 smithi151 ceph-mon[36481]: pgmap v2278: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:27.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:27 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3434.297935 secs 2024-03-28T19:00:27.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:27 smithi045 ceph-mon[31773]: pgmap v2278: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:27.417 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4080 sec 2024-03-28T19:00:28.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:28 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2725.723066 secs 2024-03-28T19:00:28.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:28 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2725.723066 secs 2024-03-28T19:00:28.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:28 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2725.723066 secs 2024-03-28T19:00:28.418 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4081 sec 2024-03-28T19:00:29.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:29 smithi071 ceph-mon[36597]: pgmap v2279: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:29.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:29 smithi151 ceph-mon[36481]: pgmap v2279: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:29.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:29 smithi045 ceph-mon[31773]: pgmap v2279: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:29.418 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4082 sec 2024-03-28T19:00:30.418 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4083 sec 2024-03-28T19:00:31.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:31 smithi071 ceph-mon[36597]: pgmap v2280: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:31.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:31 smithi151 ceph-mon[36481]: pgmap v2280: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:31.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:31 smithi045 ceph-mon[31773]: pgmap v2280: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:31.418 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4084 sec 2024-03-28T19:00:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:32 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3439.298038 secs 2024-03-28T19:00:32.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2730.723128 secs 2024-03-28T19:00:32.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:32 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3439.298038 secs 2024-03-28T19:00:32.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2730.723128 secs 2024-03-28T19:00:32.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:32 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3439.298038 secs 2024-03-28T19:00:32.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2730.723128 secs 2024-03-28T19:00:32.418 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4085 sec 2024-03-28T19:00:33.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:33 smithi071 ceph-mon[36597]: pgmap v2281: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:33.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:33 smithi151 ceph-mon[36481]: pgmap v2281: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:33.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:33 smithi045 ceph-mon[31773]: pgmap v2281: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:33.418 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4086 sec 2024-03-28T19:00:34.419 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4087 sec 2024-03-28T19:00:35.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:35 smithi071 ceph-mon[36597]: pgmap v2282: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:35.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:35 smithi151 ceph-mon[36481]: pgmap v2282: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:35.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:35 smithi045 ceph-mon[31773]: pgmap v2282: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:35.418 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4088 sec 2024-03-28T19:00:36.419 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4089 sec 2024-03-28T19:00:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:37 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3444.298053 secs 2024-03-28T19:00:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:37 smithi071 ceph-mon[36597]: pgmap v2283: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2735.723188 secs 2024-03-28T19:00:37.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:37 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3444.298053 secs 2024-03-28T19:00:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:37 smithi151 ceph-mon[36481]: pgmap v2283: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:37.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2735.723188 secs 2024-03-28T19:00:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:37 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3444.298053 secs 2024-03-28T19:00:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:37 smithi045 ceph-mon[31773]: pgmap v2283: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2735.723188 secs 2024-03-28T19:00:37.419 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4090 sec 2024-03-28T19:00:38.421 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4091 sec 2024-03-28T19:00:39.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:39 smithi071 ceph-mon[36597]: pgmap v2284: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:39.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:39 smithi151 ceph-mon[36481]: pgmap v2284: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:39.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:39 smithi045 ceph-mon[31773]: pgmap v2284: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:39.419 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4092 sec 2024-03-28T19:00:40.419 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4093 sec 2024-03-28T19:00:41.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:41 smithi071 ceph-mon[36597]: pgmap v2285: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:41.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:41 smithi151 ceph-mon[36481]: pgmap v2285: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:41.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:41 smithi045 ceph-mon[31773]: pgmap v2285: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:41.419 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4094 sec 2024-03-28T19:00:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:42 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3449.298111 secs 2024-03-28T19:00:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2740.723248 secs 2024-03-28T19:00:42.419 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4095 sec 2024-03-28T19:00:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:42 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3449.298111 secs 2024-03-28T19:00:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2740.723248 secs 2024-03-28T19:00:42.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:42 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3449.298111 secs 2024-03-28T19:00:42.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2740.723248 secs 2024-03-28T19:00:43.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:43 smithi045 ceph-mon[31773]: pgmap v2286: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:43.419 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4096 sec 2024-03-28T19:00:43.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:43 smithi071 ceph-mon[36597]: pgmap v2286: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:43.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:43 smithi151 ceph-mon[36481]: pgmap v2286: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:44.420 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4097 sec 2024-03-28T19:00:45.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:45 smithi045 ceph-mon[31773]: pgmap v2287: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:45.420 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4098 sec 2024-03-28T19:00:45.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:45 smithi071 ceph-mon[36597]: pgmap v2287: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:45.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:45 smithi151 ceph-mon[36481]: pgmap v2287: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:46.420 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4099 sec 2024-03-28T19:00:47.361 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:47 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3454.298173 secs 2024-03-28T19:00:47.362 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:47 smithi071 ceph-mon[36597]: pgmap v2288: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:47.362 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2745.723336 secs 2024-03-28T19:00:47.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:47 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3454.298173 secs 2024-03-28T19:00:47.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:47 smithi045 ceph-mon[31773]: pgmap v2288: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:47.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2745.723336 secs 2024-03-28T19:00:47.420 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4100 sec 2024-03-28T19:00:47.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:47 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3454.298173 secs 2024-03-28T19:00:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:47 smithi151 ceph-mon[36481]: pgmap v2288: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2745.723336 secs 2024-03-28T19:00:48.420 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4101 sec 2024-03-28T19:00:49.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:49 smithi045 ceph-mon[31773]: pgmap v2289: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:49.420 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4102 sec 2024-03-28T19:00:49.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:49 smithi071 ceph-mon[36597]: pgmap v2289: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:49.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:49 smithi151 ceph-mon[36481]: pgmap v2289: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:50.421 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4103 sec 2024-03-28T19:00:51.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:51 smithi045 ceph-mon[31773]: pgmap v2290: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:51.421 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4104 sec 2024-03-28T19:00:51.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:51 smithi071 ceph-mon[36597]: pgmap v2290: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:51.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:51 smithi151 ceph-mon[36481]: pgmap v2290: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:52.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:52 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3459.298263 secs 2024-03-28T19:00:52.316 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2750.723398 secs 2024-03-28T19:00:52.421 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4105 sec 2024-03-28T19:00:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:52 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3459.298263 secs 2024-03-28T19:00:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2750.723398 secs 2024-03-28T19:00:52.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:52 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3459.298263 secs 2024-03-28T19:00:52.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2750.723398 secs 2024-03-28T19:00:53.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:53 smithi045 ceph-mon[31773]: pgmap v2291: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:53.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:53 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:00:53.421 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4106 sec 2024-03-28T19:00:53.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:53 smithi071 ceph-mon[36597]: pgmap v2291: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:53.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:53 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:00:53.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:53 smithi151 ceph-mon[36481]: pgmap v2291: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:00:53.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:53 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:00:54.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:54 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:00:54.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:54 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:00:54.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:54 smithi045 ceph-mon[31773]: pgmap v2292: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:54.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:54 smithi045 ceph-mon[31773]: pgmap v2293: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:00:54.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:54 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:00:54.421 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4107 sec 2024-03-28T19:00:54.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:54 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:00:54.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:54 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:00:54.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:54 smithi071 ceph-mon[36597]: pgmap v2292: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:54.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:54 smithi071 ceph-mon[36597]: pgmap v2293: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:00:54.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:54 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:00:54.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:54 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:00:54.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:54 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:00:54.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:54 smithi151 ceph-mon[36481]: pgmap v2292: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:00:54.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:54 smithi151 ceph-mon[36481]: pgmap v2293: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:00:54.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:54 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:00:55.421 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4108 sec 2024-03-28T19:00:56.421 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4109 sec 2024-03-28T19:00:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:56 smithi045 ceph-mon[31773]: pgmap v2294: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:00:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3464.298324 secs 2024-03-28T19:00:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:56 smithi071 ceph-mon[36597]: pgmap v2294: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:00:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3464.298324 secs 2024-03-28T19:00:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:56 smithi151 ceph-mon[36481]: pgmap v2294: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:00:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3464.298324 secs 2024-03-28T19:00:57.421 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4110 sec 2024-03-28T19:00:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2755.723459 secs 2024-03-28T19:00:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2755.723459 secs 2024-03-28T19:00:58.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2755.723459 secs 2024-03-28T19:00:58.422 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4111 sec 2024-03-28T19:00:58.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:58 smithi045 ceph-mon[31773]: pgmap v2295: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:00:59.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:58 smithi071 ceph-mon[36597]: pgmap v2295: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:00:59.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:58 smithi151 ceph-mon[36481]: pgmap v2295: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:00:59.422 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4112 sec 2024-03-28T19:01:00.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:00:59 smithi071 ceph-mon[36597]: pgmap v2296: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:01:00.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:00:59 smithi151 ceph-mon[36481]: pgmap v2296: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:01:00.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:00:59 smithi045 ceph-mon[31773]: pgmap v2296: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:01:00.422 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4113 sec 2024-03-28T19:01:01.422 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4114 sec 2024-03-28T19:01:01.525 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3469.298340 secs 2024-03-28T19:01:01.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3469.298340 secs 2024-03-28T19:01:01.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3469.298340 secs 2024-03-28T19:01:02.422 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4115 sec 2024-03-28T19:01:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:02 smithi071 ceph-mon[36597]: pgmap v2297: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:01:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2760.723549 secs 2024-03-28T19:01:02.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:02 smithi151 ceph-mon[36481]: pgmap v2297: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:01:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2760.723549 secs 2024-03-28T19:01:02.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:02 smithi045 ceph-mon[31773]: pgmap v2297: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:01:02.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2760.723549 secs 2024-03-28T19:01:03.422 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4116 sec 2024-03-28T19:01:04.423 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4117 sec 2024-03-28T19:01:04.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:04 smithi045 ceph-mon[31773]: pgmap v2298: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:01:05.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:04 smithi071 ceph-mon[36597]: pgmap v2298: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:01:05.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:04 smithi151 ceph-mon[36481]: pgmap v2298: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:01:05.423 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4118 sec 2024-03-28T19:01:06.423 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4119 sec 2024-03-28T19:01:06.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:06 smithi045 ceph-mon[31773]: pgmap v2299: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:06.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3474.298404 secs 2024-03-28T19:01:07.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:06 smithi071 ceph-mon[36597]: pgmap v2299: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:07.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3474.298404 secs 2024-03-28T19:01:07.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:06 smithi151 ceph-mon[36481]: pgmap v2299: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:07.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3474.298404 secs 2024-03-28T19:01:07.423 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4120 sec 2024-03-28T19:01:07.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2765.723610 secs 2024-03-28T19:01:08.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2765.723610 secs 2024-03-28T19:01:08.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2765.723610 secs 2024-03-28T19:01:08.423 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4121 sec 2024-03-28T19:01:08.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:08 smithi045 ceph-mon[31773]: pgmap v2300: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:09.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:08 smithi071 ceph-mon[36597]: pgmap v2300: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:09.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:08 smithi151 ceph-mon[36481]: pgmap v2300: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:09.423 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4122 sec 2024-03-28T19:01:10.423 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4123 sec 2024-03-28T19:01:10.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:10 smithi045 ceph-mon[31773]: pgmap v2301: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:10 smithi071 ceph-mon[36597]: pgmap v2301: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:11.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:10 smithi151 ceph-mon[36481]: pgmap v2301: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:11.423 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4124 sec 2024-03-28T19:01:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3479.298483 secs 2024-03-28T19:01:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3479.298483 secs 2024-03-28T19:01:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3479.298483 secs 2024-03-28T19:01:12.424 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4125 sec 2024-03-28T19:01:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:12 smithi045 ceph-mon[31773]: pgmap v2302: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2770.723671 secs 2024-03-28T19:01:12.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:12 smithi151 ceph-mon[36481]: pgmap v2302: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:12.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2770.723671 secs 2024-03-28T19:01:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:12 smithi071 ceph-mon[36597]: pgmap v2302: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2770.723671 secs 2024-03-28T19:01:13.424 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4126 sec 2024-03-28T19:01:14.424 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4127 sec 2024-03-28T19:01:14.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:14 smithi045 ceph-mon[31773]: pgmap v2303: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:15.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:14 smithi071 ceph-mon[36597]: pgmap v2303: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:15.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:14 smithi151 ceph-mon[36481]: pgmap v2303: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:15.424 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4128 sec 2024-03-28T19:01:15.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:15 smithi045 ceph-mon[31773]: pgmap v2304: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:16.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:15 smithi071 ceph-mon[36597]: pgmap v2304: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:16.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:15 smithi151 ceph-mon[36481]: pgmap v2304: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:16.424 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4129 sec 2024-03-28T19:01:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3484.298499 secs 2024-03-28T19:01:17.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3484.298499 secs 2024-03-28T19:01:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3484.298499 secs 2024-03-28T19:01:17.424 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4130 sec 2024-03-28T19:01:17.961 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2775.723731 secs 2024-03-28T19:01:17.961 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:17 smithi071 ceph-mon[36597]: pgmap v2305: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2775.723731 secs 2024-03-28T19:01:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:17 smithi151 ceph-mon[36481]: pgmap v2305: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2775.723731 secs 2024-03-28T19:01:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:17 smithi045 ceph-mon[31773]: pgmap v2305: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:18.425 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4131 sec 2024-03-28T19:01:19.425 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4132 sec 2024-03-28T19:01:20.425 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4133 sec 2024-03-28T19:01:20.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:20 smithi045 ceph-mon[31773]: pgmap v2306: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:21.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:20 smithi071 ceph-mon[36597]: pgmap v2306: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:21.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:20 smithi151 ceph-mon[36481]: pgmap v2306: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:21.425 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4134 sec 2024-03-28T19:01:21.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3489.298561 secs 2024-03-28T19:01:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3489.298561 secs 2024-03-28T19:01:22.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3489.298561 secs 2024-03-28T19:01:22.425 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4135 sec 2024-03-28T19:01:22.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:22 smithi045 ceph-mon[31773]: pgmap v2307: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:22.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2780.723771 secs 2024-03-28T19:01:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:22 smithi071 ceph-mon[36597]: pgmap v2307: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2780.723771 secs 2024-03-28T19:01:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:22 smithi151 ceph-mon[36481]: pgmap v2307: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2780.723771 secs 2024-03-28T19:01:23.425 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4136 sec 2024-03-28T19:01:24.425 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4137 sec 2024-03-28T19:01:24.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:24 smithi045 ceph-mon[31773]: pgmap v2308: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:25.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:24 smithi071 ceph-mon[36597]: pgmap v2308: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:25.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:24 smithi151 ceph-mon[36481]: pgmap v2308: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:25.425 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4138 sec 2024-03-28T19:01:26.426 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4139 sec 2024-03-28T19:01:26.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:26 smithi045 ceph-mon[31773]: pgmap v2309: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:26.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3494.298624 secs 2024-03-28T19:01:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:26 smithi071 ceph-mon[36597]: pgmap v2309: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3494.298624 secs 2024-03-28T19:01:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:26 smithi151 ceph-mon[36481]: pgmap v2309: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:27.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3494.298624 secs 2024-03-28T19:01:27.426 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4140 sec 2024-03-28T19:01:27.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2785.723855 secs 2024-03-28T19:01:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2785.723855 secs 2024-03-28T19:01:28.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2785.723855 secs 2024-03-28T19:01:28.426 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4141 sec 2024-03-28T19:01:28.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:28 smithi045 ceph-mon[31773]: pgmap v2310: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:29.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:28 smithi071 ceph-mon[36597]: pgmap v2310: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:29.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:28 smithi151 ceph-mon[36481]: pgmap v2310: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:29.426 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4142 sec 2024-03-28T19:01:30.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:29 smithi071 ceph-mon[36597]: pgmap v2311: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:30.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:29 smithi151 ceph-mon[36481]: pgmap v2311: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:30.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:29 smithi045 ceph-mon[31773]: pgmap v2311: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:30.426 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4143 sec 2024-03-28T19:01:31.426 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4144 sec 2024-03-28T19:01:31.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3499.298683 secs 2024-03-28T19:01:31.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3499.298683 secs 2024-03-28T19:01:31.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3499.298683 secs 2024-03-28T19:01:32.426 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4145 sec 2024-03-28T19:01:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:32 smithi071 ceph-mon[36597]: pgmap v2312: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2790.723941 secs 2024-03-28T19:01:32.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:32 smithi151 ceph-mon[36481]: pgmap v2312: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:32.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2790.723941 secs 2024-03-28T19:01:32.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:32 smithi045 ceph-mon[31773]: pgmap v2312: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:32.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2790.723941 secs 2024-03-28T19:01:33.427 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4146 sec 2024-03-28T19:01:34.427 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4147 sec 2024-03-28T19:01:34.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:34 smithi045 ceph-mon[31773]: pgmap v2313: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:35.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:34 smithi071 ceph-mon[36597]: pgmap v2313: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:35.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:34 smithi151 ceph-mon[36481]: pgmap v2313: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:35.427 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4148 sec 2024-03-28T19:01:36.427 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4149 sec 2024-03-28T19:01:36.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:36 smithi045 ceph-mon[31773]: pgmap v2314: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T19:01:36.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3504.298746 secs 2024-03-28T19:01:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:36 smithi071 ceph-mon[36597]: pgmap v2314: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T19:01:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3504.298746 secs 2024-03-28T19:01:37.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:36 smithi151 ceph-mon[36481]: pgmap v2314: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T19:01:37.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3504.298746 secs 2024-03-28T19:01:37.427 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4150 sec 2024-03-28T19:01:37.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2795.724004 secs 2024-03-28T19:01:38.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2795.724004 secs 2024-03-28T19:01:38.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2795.724004 secs 2024-03-28T19:01:38.427 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4151 sec 2024-03-28T19:01:38.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:38 smithi045 ceph-mon[31773]: pgmap v2315: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T19:01:39.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:38 smithi071 ceph-mon[36597]: pgmap v2315: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T19:01:39.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:38 smithi151 ceph-mon[36481]: pgmap v2315: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T19:01:39.427 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4152 sec 2024-03-28T19:01:40.428 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4153 sec 2024-03-28T19:01:40.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:40 smithi045 ceph-mon[31773]: pgmap v2316: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T19:01:41.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:40 smithi071 ceph-mon[36597]: pgmap v2316: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T19:01:41.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:40 smithi151 ceph-mon[36481]: pgmap v2316: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T19:01:41.428 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4154 sec 2024-03-28T19:01:41.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3509.298808 secs 2024-03-28T19:01:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3509.298808 secs 2024-03-28T19:01:42.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3509.298808 secs 2024-03-28T19:01:42.428 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4155 sec 2024-03-28T19:01:42.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:42 smithi045 ceph-mon[31773]: pgmap v2317: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T19:01:42.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2800.724096 secs 2024-03-28T19:01:42.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:42 smithi151 ceph-mon[36481]: pgmap v2317: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T19:01:42.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2800.724096 secs 2024-03-28T19:01:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:42 smithi071 ceph-mon[36597]: pgmap v2317: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T19:01:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2800.724096 secs 2024-03-28T19:01:43.428 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4156 sec 2024-03-28T19:01:44.428 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4157 sec 2024-03-28T19:01:44.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:44 smithi045 ceph-mon[31773]: pgmap v2318: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T19:01:45.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:44 smithi071 ceph-mon[36597]: pgmap v2318: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T19:01:45.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:44 smithi151 ceph-mon[36481]: pgmap v2318: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T19:01:45.428 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4158 sec 2024-03-28T19:01:46.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:45 smithi071 ceph-mon[36597]: pgmap v2319: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T19:01:46.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:45 smithi151 ceph-mon[36481]: pgmap v2319: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T19:01:46.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:45 smithi045 ceph-mon[31773]: pgmap v2319: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T19:01:46.428 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4159 sec 2024-03-28T19:01:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3514.298849 secs 2024-03-28T19:01:47.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3514.298849 secs 2024-03-28T19:01:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3514.298849 secs 2024-03-28T19:01:47.429 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4160 sec 2024-03-28T19:01:47.961 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2805.724156 secs 2024-03-28T19:01:47.961 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:47 smithi071 ceph-mon[36597]: pgmap v2320: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:48.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2805.724156 secs 2024-03-28T19:01:48.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:47 smithi151 ceph-mon[36481]: pgmap v2320: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2805.724156 secs 2024-03-28T19:01:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:47 smithi045 ceph-mon[31773]: pgmap v2320: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:48.429 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4161 sec 2024-03-28T19:01:49.429 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4162 sec 2024-03-28T19:01:50.429 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4163 sec 2024-03-28T19:01:50.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:50 smithi045 ceph-mon[31773]: pgmap v2321: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:51.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:50 smithi071 ceph-mon[36597]: pgmap v2321: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:51.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:50 smithi151 ceph-mon[36481]: pgmap v2321: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:51.429 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4164 sec 2024-03-28T19:01:51.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3519.298886 secs 2024-03-28T19:01:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3519.298886 secs 2024-03-28T19:01:52.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3519.298886 secs 2024-03-28T19:01:52.430 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4165 sec 2024-03-28T19:01:52.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:52 smithi045 ceph-mon[31773]: pgmap v2322: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:52.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2810.724219 secs 2024-03-28T19:01:53.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:52 smithi071 ceph-mon[36597]: pgmap v2322: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:53.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2810.724219 secs 2024-03-28T19:01:53.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:52 smithi151 ceph-mon[36481]: pgmap v2322: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:53.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2810.724219 secs 2024-03-28T19:01:53.430 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4166 sec 2024-03-28T19:01:54.430 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4167 sec 2024-03-28T19:01:54.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:54 smithi045 ceph-mon[31773]: pgmap v2323: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:54.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:54 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:01:54.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:54 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:54.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:54 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:54.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:54 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:54.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:54 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:55.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:54 smithi071 ceph-mon[36597]: pgmap v2323: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:55.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:54 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:01:55.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:54 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:55.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:54 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:55.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:54 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:55.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:54 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:55.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:54 smithi151 ceph-mon[36481]: pgmap v2323: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:55.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:54 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:01:55.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:54 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:55.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:54 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:55.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:54 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:55.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:54 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:55.430 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4168 sec 2024-03-28T19:01:56.058 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:55 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:56.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:55 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:56.059 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:55 smithi045 ceph-mon[31773]: pgmap v2324: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:56.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:55 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:56.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:55 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:56.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:55 smithi071 ceph-mon[36597]: pgmap v2324: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:56.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:55 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:56.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:55 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:56.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:55 smithi151 ceph-mon[36481]: pgmap v2324: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:01:56.430 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4169 sec 2024-03-28T19:01:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3524.298951 secs 2024-03-28T19:01:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:56 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:56 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:56 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:56 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3524.298951 secs 2024-03-28T19:01:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:56 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:56 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:56 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:57.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:56 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3524.298951 secs 2024-03-28T19:01:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:56 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:56 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:56 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:56 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:01:57.430 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4170 sec 2024-03-28T19:01:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2815.724285 secs 2024-03-28T19:01:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:57 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:57 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:57 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:57 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:57 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:57 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:57 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:57 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:57 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T19:01:58.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:57 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T19:01:58.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:57 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:57 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:57 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:57 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:57 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:57 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:57 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:57 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:57 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T19:01:58.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:57 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T19:01:58.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:01:57 smithi071 ceph-mon[36597]: pgmap v2325: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:58.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2815.724285 secs 2024-03-28T19:01:58.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:57 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:57 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:57 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:57 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:57 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:57 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:57 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:57 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:57 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T19:01:58.075 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:57 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T19:01:58.075 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:57 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.075 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:57 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.075 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:57 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.075 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:57 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.075 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:57 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.075 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:57 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.075 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:57 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.075 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:57 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.076 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:57 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T19:01:58.076 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:57 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T19:01:58.076 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:01:57 smithi151 ceph-mon[36481]: pgmap v2325: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2815.724285 secs 2024-03-28T19:01:58.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:57 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:57 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:57 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:57 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:57 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:57 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:57 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:57 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:57 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T19:01:58.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:57 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T19:01:58.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:57 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:57 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:57 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:57 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:57 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:57 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:57 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.167 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:57 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:01:58.168 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:57 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T19:01:58.168 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:57 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T19:01:58.168 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:01:57 smithi045 ceph-mon[31773]: pgmap v2325: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:01:58.430 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4171 sec 2024-03-28T19:01:59.430 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4172 sec 2024-03-28T19:02:00.430 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4173 sec 2024-03-28T19:02:00.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:00 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:02:00.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:00 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:02:00.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:00 smithi071 ceph-mon[36597]: pgmap v2326: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:00.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:00 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:02:00.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:00 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:02:00.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:00 smithi151 ceph-mon[36481]: pgmap v2326: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:00.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:00 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:02:00.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:00 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:02:00.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:00 smithi045 ceph-mon[31773]: pgmap v2326: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:01.431 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4174 sec 2024-03-28T19:02:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:01 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:01 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:01 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:01 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:01 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:01 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:01 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:01 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:01 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T19:02:01.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:01 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T19:02:01.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:01 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:02:01.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:01 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:02:01.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:01 smithi151 ceph-mon[36481]: pgmap v2327: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T19:02:01.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:01 smithi151 ceph-mon[36481]: pgmap v2328: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:02:01.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:01 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:02:01.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3529.299042 secs 2024-03-28T19:02:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:01 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:01 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:01 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:01 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:01 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:01 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:01 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:01 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:01 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T19:02:01.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:01 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T19:02:01.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:01 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:02:01.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:01 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:02:01.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:01 smithi045 ceph-mon[31773]: pgmap v2327: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T19:02:01.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:01 smithi045 ceph-mon[31773]: pgmap v2328: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:02:01.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:01 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:02:01.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3529.299042 secs 2024-03-28T19:02:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:01 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:01 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:01 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:01 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:01 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:01 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:01 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:01 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:02:01.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:01 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T19:02:01.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:01 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T19:02:01.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:01 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:02:01.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:01 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:02:01.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:01 smithi071 ceph-mon[36597]: pgmap v2327: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T19:02:01.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:01 smithi071 ceph-mon[36597]: pgmap v2328: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:02:01.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:01 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:02:01.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3529.299042 secs 2024-03-28T19:02:02.431 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4175 sec 2024-03-28T19:02:02.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2820.724372 secs 2024-03-28T19:02:02.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2820.724372 secs 2024-03-28T19:02:02.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:02:02 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:02:02.260+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:02:02.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:02:02 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:02:02.260+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:02:02.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:02:02 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:02:02.260+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:02:02.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:02:02 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:02:02.260+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:02:02.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:02:02 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:02:02.260+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:02:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2820.724372 secs 2024-03-28T19:02:03.431 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4176 sec 2024-03-28T19:02:03.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:03 smithi045 ceph-mon[31773]: pgmap v2329: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:02:03.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:03 smithi071 ceph-mon[36597]: pgmap v2329: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:02:03.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:03 smithi151 ceph-mon[36481]: pgmap v2329: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:02:04.431 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4177 sec 2024-03-28T19:02:05.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:04 smithi071 ceph-mon[36597]: pgmap v2330: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:02:05.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:04 smithi151 ceph-mon[36481]: pgmap v2330: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:02:05.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:04 smithi045 ceph-mon[31773]: pgmap v2330: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:02:05.431 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4178 sec 2024-03-28T19:02:06.431 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4179 sec 2024-03-28T19:02:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3534.299082 secs 2024-03-28T19:02:06.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3534.299082 secs 2024-03-28T19:02:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3534.299082 secs 2024-03-28T19:02:07.431 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4180 sec 2024-03-28T19:02:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:07 smithi071 ceph-mon[36597]: pgmap v2331: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 225 B/s wr, 19 op/s 2024-03-28T19:02:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2825.724414 secs 2024-03-28T19:02:07.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:07 smithi151 ceph-mon[36481]: pgmap v2331: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 225 B/s wr, 19 op/s 2024-03-28T19:02:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2825.724414 secs 2024-03-28T19:02:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:07 smithi045 ceph-mon[31773]: pgmap v2331: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 225 B/s wr, 19 op/s 2024-03-28T19:02:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2825.724414 secs 2024-03-28T19:02:08.431 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4181 sec 2024-03-28T19:02:09.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:08 smithi071 ceph-mon[36597]: pgmap v2332: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 225 B/s wr, 19 op/s 2024-03-28T19:02:09.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:08 smithi151 ceph-mon[36481]: pgmap v2332: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 225 B/s wr, 19 op/s 2024-03-28T19:02:09.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:08 smithi045 ceph-mon[31773]: pgmap v2332: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 225 B/s wr, 19 op/s 2024-03-28T19:02:09.432 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4182 sec 2024-03-28T19:02:10.432 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4183 sec 2024-03-28T19:02:11.432 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4184 sec 2024-03-28T19:02:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:11 smithi071 ceph-mon[36597]: pgmap v2333: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 204 B/s wr, 17 op/s 2024-03-28T19:02:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3539.299170 secs 2024-03-28T19:02:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:11 smithi151 ceph-mon[36481]: pgmap v2333: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 204 B/s wr, 17 op/s 2024-03-28T19:02:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3539.299170 secs 2024-03-28T19:02:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:11 smithi045 ceph-mon[31773]: pgmap v2333: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 204 B/s wr, 17 op/s 2024-03-28T19:02:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3539.299170 secs 2024-03-28T19:02:12.432 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4185 sec 2024-03-28T19:02:12.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2830.724473 secs 2024-03-28T19:02:12.985 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2830.724473 secs 2024-03-28T19:02:13.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2830.724473 secs 2024-03-28T19:02:13.432 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4186 sec 2024-03-28T19:02:14.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:13 smithi071 ceph-mon[36597]: pgmap v2334: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T19:02:14.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:13 smithi151 ceph-mon[36481]: pgmap v2334: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T19:02:14.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:13 smithi045 ceph-mon[31773]: pgmap v2334: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T19:02:14.432 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4187 sec 2024-03-28T19:02:15.432 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4188 sec 2024-03-28T19:02:16.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:15 smithi071 ceph-mon[36597]: pgmap v2335: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T19:02:16.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:15 smithi151 ceph-mon[36481]: pgmap v2335: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T19:02:16.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:15 smithi045 ceph-mon[31773]: pgmap v2335: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 170 B/s wr, 14 op/s 2024-03-28T19:02:16.432 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4189 sec 2024-03-28T19:02:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3544.299259 secs 2024-03-28T19:02:17.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3544.299259 secs 2024-03-28T19:02:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3544.299259 secs 2024-03-28T19:02:17.433 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4190 sec 2024-03-28T19:02:18.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:17 smithi071 ceph-mon[36597]: pgmap v2336: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T19:02:18.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2835.724560 secs 2024-03-28T19:02:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:17 smithi151 ceph-mon[36481]: pgmap v2336: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T19:02:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2835.724560 secs 2024-03-28T19:02:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:17 smithi045 ceph-mon[31773]: pgmap v2336: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 B/s wr, 16 op/s 2024-03-28T19:02:18.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2835.724560 secs 2024-03-28T19:02:18.433 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4191 sec 2024-03-28T19:02:19.433 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4192 sec 2024-03-28T19:02:20.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:19 smithi071 ceph-mon[36597]: pgmap v2337: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:20.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:19 smithi151 ceph-mon[36481]: pgmap v2337: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:20.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:19 smithi045 ceph-mon[31773]: pgmap v2337: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:20.433 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4193 sec 2024-03-28T19:02:21.434 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4194 sec 2024-03-28T19:02:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:21 smithi071 ceph-mon[36597]: pgmap v2338: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3549.299351 secs 2024-03-28T19:02:22.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:21 smithi151 ceph-mon[36481]: pgmap v2338: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:22.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3549.299351 secs 2024-03-28T19:02:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:21 smithi045 ceph-mon[31773]: pgmap v2338: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3549.299351 secs 2024-03-28T19:02:22.433 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4195 sec 2024-03-28T19:02:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2840.724622 secs 2024-03-28T19:02:23.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2840.724622 secs 2024-03-28T19:02:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2840.724622 secs 2024-03-28T19:02:23.433 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4196 sec 2024-03-28T19:02:24.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:23 smithi071 ceph-mon[36597]: pgmap v2339: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:24.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:23 smithi151 ceph-mon[36481]: pgmap v2339: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:24.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:23 smithi045 ceph-mon[31773]: pgmap v2339: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:24.434 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4197 sec 2024-03-28T19:02:25.434 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4198 sec 2024-03-28T19:02:26.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:25 smithi071 ceph-mon[36597]: pgmap v2340: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:26.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:25 smithi151 ceph-mon[36481]: pgmap v2340: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:26.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:25 smithi045 ceph-mon[31773]: pgmap v2340: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:26.434 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4199 sec 2024-03-28T19:02:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3554.299440 secs 2024-03-28T19:02:27.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3554.299440 secs 2024-03-28T19:02:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3554.299440 secs 2024-03-28T19:02:27.434 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4200 sec 2024-03-28T19:02:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2845.724709 secs 2024-03-28T19:02:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:27 smithi071 ceph-mon[36597]: pgmap v2341: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:28.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2845.724709 secs 2024-03-28T19:02:28.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:27 smithi151 ceph-mon[36481]: pgmap v2341: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2845.724709 secs 2024-03-28T19:02:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:27 smithi045 ceph-mon[31773]: pgmap v2341: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:28.434 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4201 sec 2024-03-28T19:02:29.434 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4202 sec 2024-03-28T19:02:30.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:29 smithi071 ceph-mon[36597]: pgmap v2342: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:30.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:29 smithi151 ceph-mon[36481]: pgmap v2342: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:30.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:29 smithi045 ceph-mon[31773]: pgmap v2342: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:30.434 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4203 sec 2024-03-28T19:02:31.435 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4204 sec 2024-03-28T19:02:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:31 smithi071 ceph-mon[36597]: pgmap v2343: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3559.299481 secs 2024-03-28T19:02:32.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:31 smithi151 ceph-mon[36481]: pgmap v2343: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:32.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3559.299481 secs 2024-03-28T19:02:32.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:31 smithi045 ceph-mon[31773]: pgmap v2343: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:32.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3559.299481 secs 2024-03-28T19:02:32.435 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4205 sec 2024-03-28T19:02:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2850.724799 secs 2024-03-28T19:02:33.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2850.724799 secs 2024-03-28T19:02:33.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2850.724799 secs 2024-03-28T19:02:33.435 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4206 sec 2024-03-28T19:02:34.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:33 smithi071 ceph-mon[36597]: pgmap v2344: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:34.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:33 smithi151 ceph-mon[36481]: pgmap v2344: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:34.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:33 smithi045 ceph-mon[31773]: pgmap v2344: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:34.435 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4207 sec 2024-03-28T19:02:35.435 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4208 sec 2024-03-28T19:02:36.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:35 smithi151 ceph-mon[36481]: pgmap v2345: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:36.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:35 smithi045 ceph-mon[31773]: pgmap v2345: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:36.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:35 smithi071 ceph-mon[36597]: pgmap v2345: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:36.435 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4209 sec 2024-03-28T19:02:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3564.299570 secs 2024-03-28T19:02:37.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3564.299570 secs 2024-03-28T19:02:37.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3564.299570 secs 2024-03-28T19:02:37.435 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4210 sec 2024-03-28T19:02:38.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2855.724857 secs 2024-03-28T19:02:38.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:37 smithi045 ceph-mon[31773]: pgmap v2346: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:38.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2855.724857 secs 2024-03-28T19:02:38.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:37 smithi071 ceph-mon[36597]: pgmap v2346: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:38.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2855.724857 secs 2024-03-28T19:02:38.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:37 smithi151 ceph-mon[36481]: pgmap v2346: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:38.435 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4211 sec 2024-03-28T19:02:39.436 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4212 sec 2024-03-28T19:02:40.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:39 smithi045 ceph-mon[31773]: pgmap v2347: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:40.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:39 smithi071 ceph-mon[36597]: pgmap v2347: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:40.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:39 smithi151 ceph-mon[36481]: pgmap v2347: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:40.436 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4213 sec 2024-03-28T19:02:41.436 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4214 sec 2024-03-28T19:02:42.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:41 smithi151 ceph-mon[36481]: pgmap v2348: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:42.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3569.299654 secs 2024-03-28T19:02:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:41 smithi045 ceph-mon[31773]: pgmap v2348: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3569.299654 secs 2024-03-28T19:02:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:41 smithi071 ceph-mon[36597]: pgmap v2348: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3569.299654 secs 2024-03-28T19:02:42.436 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4215 sec 2024-03-28T19:02:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2860.724876 secs 2024-03-28T19:02:43.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2860.724876 secs 2024-03-28T19:02:43.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2860.724876 secs 2024-03-28T19:02:43.436 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4216 sec 2024-03-28T19:02:44.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:43 smithi045 ceph-mon[31773]: pgmap v2349: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:44.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:43 smithi071 ceph-mon[36597]: pgmap v2349: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:44.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:43 smithi151 ceph-mon[36481]: pgmap v2349: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:44.436 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4217 sec 2024-03-28T19:02:45.437 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4218 sec 2024-03-28T19:02:46.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:45 smithi045 ceph-mon[31773]: pgmap v2350: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:46.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:45 smithi071 ceph-mon[36597]: pgmap v2350: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:46.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:45 smithi151 ceph-mon[36481]: pgmap v2350: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:46.437 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4219 sec 2024-03-28T19:02:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3574.299714 secs 2024-03-28T19:02:47.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3574.299714 secs 2024-03-28T19:02:47.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3574.299714 secs 2024-03-28T19:02:47.437 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4220 sec 2024-03-28T19:02:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2865.724935 secs 2024-03-28T19:02:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:47 smithi045 ceph-mon[31773]: pgmap v2351: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:48.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2865.724935 secs 2024-03-28T19:02:48.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:47 smithi071 ceph-mon[36597]: pgmap v2351: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2865.724935 secs 2024-03-28T19:02:48.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:47 smithi151 ceph-mon[36481]: pgmap v2351: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:48.437 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4221 sec 2024-03-28T19:02:49.437 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4222 sec 2024-03-28T19:02:50.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:49 smithi045 ceph-mon[31773]: pgmap v2352: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:50.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:49 smithi071 ceph-mon[36597]: pgmap v2352: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:50.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:49 smithi151 ceph-mon[36481]: pgmap v2352: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:50.437 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4223 sec 2024-03-28T19:02:51.438 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4224 sec 2024-03-28T19:02:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:51 smithi045 ceph-mon[31773]: pgmap v2353: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3579.299775 secs 2024-03-28T19:02:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:51 smithi071 ceph-mon[36597]: pgmap v2353: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:52.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3579.299775 secs 2024-03-28T19:02:52.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:51 smithi151 ceph-mon[36481]: pgmap v2353: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:52.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3579.299775 secs 2024-03-28T19:02:52.437 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4225 sec 2024-03-28T19:02:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2870.724997 secs 2024-03-28T19:02:53.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2870.724997 secs 2024-03-28T19:02:53.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2870.724997 secs 2024-03-28T19:02:53.438 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4226 sec 2024-03-28T19:02:54.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:53 smithi045 ceph-mon[31773]: pgmap v2354: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:54.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:53 smithi071 ceph-mon[36597]: pgmap v2354: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:54.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:53 smithi151 ceph-mon[36481]: pgmap v2354: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:54.438 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4227 sec 2024-03-28T19:02:55.438 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4228 sec 2024-03-28T19:02:56.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:55 smithi045 ceph-mon[31773]: pgmap v2355: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:56.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:55 smithi071 ceph-mon[36597]: pgmap v2355: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:56.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:55 smithi151 ceph-mon[36481]: pgmap v2355: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:02:56.438 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4229 sec 2024-03-28T19:02:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3584.299809 secs 2024-03-28T19:02:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3584.299809 secs 2024-03-28T19:02:57.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3584.299809 secs 2024-03-28T19:02:57.438 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4230 sec 2024-03-28T19:02:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2875.725086 secs 2024-03-28T19:02:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:57 smithi045 ceph-mon[31773]: pgmap v2356: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:58.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2875.725086 secs 2024-03-28T19:02:58.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:57 smithi071 ceph-mon[36597]: pgmap v2356: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:58.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2875.725086 secs 2024-03-28T19:02:58.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:57 smithi151 ceph-mon[36481]: pgmap v2356: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:02:58.438 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4231 sec 2024-03-28T19:02:59.438 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4232 sec 2024-03-28T19:03:00.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:02:59 smithi045 ceph-mon[31773]: pgmap v2357: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:00.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:02:59 smithi071 ceph-mon[36597]: pgmap v2357: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:00.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:02:59 smithi151 ceph-mon[36481]: pgmap v2357: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:00.438 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4233 sec 2024-03-28T19:03:01.439 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4234 sec 2024-03-28T19:03:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:01 smithi071 ceph-mon[36597]: pgmap v2358: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3589.299898 secs 2024-03-28T19:03:02.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:01 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:03:02.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:01 smithi151 ceph-mon[36481]: pgmap v2358: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3589.299898 secs 2024-03-28T19:03:02.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:01 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:03:02.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:01 smithi045 ceph-mon[31773]: pgmap v2358: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:02.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3589.299898 secs 2024-03-28T19:03:02.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:01 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:03:02.439 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4235 sec 2024-03-28T19:03:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:02 smithi071 ceph-mon[36597]: 2 slow requests, 1 included below; oldest blocked for > 2880.725164 secs 2024-03-28T19:03:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:02 smithi071 ceph-mon[36597]: slow request 2880.725164 seconds old, received at 2024-03-28T18:15:00.947128+0000: client_request(client.26687:1 getattr pAsLsXsFs #0x1 2024-03-28T18:15:00.943510+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T19:03:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:02 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:03:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:02 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:03:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:02 smithi071 ceph-mon[36597]: pgmap v2359: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:03.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:02 smithi071 ceph-mon[36597]: pgmap v2360: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:03:03.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:02 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:03:03.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:02 smithi151 ceph-mon[36481]: 2 slow requests, 1 included below; oldest blocked for > 2880.725164 secs 2024-03-28T19:03:03.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:02 smithi151 ceph-mon[36481]: slow request 2880.725164 seconds old, received at 2024-03-28T18:15:00.947128+0000: client_request(client.26687:1 getattr pAsLsXsFs #0x1 2024-03-28T18:15:00.943510+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T19:03:03.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:02 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:03:03.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:02 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:03:03.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:02 smithi151 ceph-mon[36481]: pgmap v2359: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:03.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:02 smithi151 ceph-mon[36481]: pgmap v2360: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:03:03.323 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:02 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:03:03.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:02 smithi045 ceph-mon[31773]: 2 slow requests, 1 included below; oldest blocked for > 2880.725164 secs 2024-03-28T19:03:03.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:02 smithi045 ceph-mon[31773]: slow request 2880.725164 seconds old, received at 2024-03-28T18:15:00.947128+0000: client_request(client.26687:1 getattr pAsLsXsFs #0x1 2024-03-28T18:15:00.943510+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T19:03:03.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:02 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:03:03.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:02 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:03:03.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:02 smithi045 ceph-mon[31773]: pgmap v2359: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:03.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:02 smithi045 ceph-mon[31773]: pgmap v2360: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:03:03.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:02 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:03:03.439 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4236 sec 2024-03-28T19:03:04.439 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4237 sec 2024-03-28T19:03:05.439 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4238 sec 2024-03-28T19:03:05.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:05 smithi045 ceph-mon[31773]: pgmap v2361: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:03:05.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:05 smithi071 ceph-mon[36597]: pgmap v2361: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:03:05.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:05 smithi151 ceph-mon[36481]: pgmap v2361: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:03:06.440 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4239 sec 2024-03-28T19:03:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3594.299985 secs 2024-03-28T19:03:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3594.299985 secs 2024-03-28T19:03:06.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3594.299985 secs 2024-03-28T19:03:07.440 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4240 sec 2024-03-28T19:03:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:07 smithi045 ceph-mon[31773]: pgmap v2362: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-28T19:03:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2885.725226 secs 2024-03-28T19:03:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:07 smithi071 ceph-mon[36597]: pgmap v2362: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-28T19:03:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2885.725226 secs 2024-03-28T19:03:07.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:07 smithi151 ceph-mon[36481]: pgmap v2362: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-28T19:03:07.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2885.725226 secs 2024-03-28T19:03:08.440 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4241 sec 2024-03-28T19:03:09.440 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4242 sec 2024-03-28T19:03:09.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:09 smithi045 ceph-mon[31773]: pgmap v2363: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-28T19:03:09.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:09 smithi071 ceph-mon[36597]: pgmap v2363: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-28T19:03:09.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:09 smithi151 ceph-mon[36481]: pgmap v2363: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-28T19:03:10.440 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4243 sec 2024-03-28T19:03:11.440 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4244 sec 2024-03-28T19:03:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:11 smithi045 ceph-mon[31773]: pgmap v2364: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-28T19:03:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3599.300033 secs 2024-03-28T19:03:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:11 smithi071 ceph-mon[36597]: pgmap v2364: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-28T19:03:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3599.300033 secs 2024-03-28T19:03:11.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:11 smithi151 ceph-mon[36481]: pgmap v2364: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-28T19:03:11.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3599.300033 secs 2024-03-28T19:03:12.440 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4245 sec 2024-03-28T19:03:12.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2890.725287 secs 2024-03-28T19:03:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2890.725287 secs 2024-03-28T19:03:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2890.725287 secs 2024-03-28T19:03:13.440 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4246 sec 2024-03-28T19:03:13.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:13 smithi071 ceph-mon[36597]: pgmap v2365: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-28T19:03:13.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:13 smithi151 ceph-mon[36481]: pgmap v2365: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-28T19:03:13.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:13 smithi045 ceph-mon[31773]: pgmap v2365: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-28T19:03:14.440 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4247 sec 2024-03-28T19:03:15.440 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4248 sec 2024-03-28T19:03:15.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:15 smithi071 ceph-mon[36597]: pgmap v2366: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T19:03:15.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:15 smithi151 ceph-mon[36481]: pgmap v2366: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T19:03:15.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:15 smithi045 ceph-mon[31773]: pgmap v2366: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T19:03:16.441 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4249 sec 2024-03-28T19:03:16.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3604.300074 secs 2024-03-28T19:03:16.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3604.300074 secs 2024-03-28T19:03:16.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3604.300074 secs 2024-03-28T19:03:17.441 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4250 sec 2024-03-28T19:03:17.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:17 smithi071 ceph-mon[36597]: pgmap v2367: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T19:03:17.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2895.725347 secs 2024-03-28T19:03:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:17 smithi151 ceph-mon[36481]: pgmap v2367: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T19:03:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2895.725347 secs 2024-03-28T19:03:17.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:17 smithi045 ceph-mon[31773]: pgmap v2367: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T19:03:17.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2895.725347 secs 2024-03-28T19:03:18.441 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4251 sec 2024-03-28T19:03:19.441 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4252 sec 2024-03-28T19:03:19.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:19 smithi071 ceph-mon[36597]: pgmap v2368: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:19.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:19 smithi151 ceph-mon[36481]: pgmap v2368: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:19.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:19 smithi045 ceph-mon[31773]: pgmap v2368: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:20.441 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4253 sec 2024-03-28T19:03:21.441 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4254 sec 2024-03-28T19:03:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:21 smithi071 ceph-mon[36597]: pgmap v2369: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3609.300163 secs 2024-03-28T19:03:21.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:21 smithi151 ceph-mon[36481]: pgmap v2369: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:21.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3609.300163 secs 2024-03-28T19:03:21.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:21 smithi045 ceph-mon[31773]: pgmap v2369: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:21.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3609.300163 secs 2024-03-28T19:03:22.441 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4255 sec 2024-03-28T19:03:22.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2900.725410 secs 2024-03-28T19:03:22.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2900.725410 secs 2024-03-28T19:03:22.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2900.725410 secs 2024-03-28T19:03:23.442 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4256 sec 2024-03-28T19:03:23.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:23 smithi071 ceph-mon[36597]: pgmap v2370: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:23.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:23 smithi151 ceph-mon[36481]: pgmap v2370: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:23.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:23 smithi045 ceph-mon[31773]: pgmap v2370: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:24.442 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4257 sec 2024-03-28T19:03:25.442 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4258 sec 2024-03-28T19:03:25.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:25 smithi071 ceph-mon[36597]: pgmap v2371: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:25.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:25 smithi151 ceph-mon[36481]: pgmap v2371: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:25.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:25 smithi045 ceph-mon[31773]: pgmap v2371: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:26.442 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4259 sec 2024-03-28T19:03:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3614.300223 secs 2024-03-28T19:03:26.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3614.300223 secs 2024-03-28T19:03:26.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3614.300223 secs 2024-03-28T19:03:27.442 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4260 sec 2024-03-28T19:03:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:27 smithi071 ceph-mon[36597]: pgmap v2372: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2905.725472 secs 2024-03-28T19:03:27.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:27 smithi151 ceph-mon[36481]: pgmap v2372: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:27.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2905.725472 secs 2024-03-28T19:03:27.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:27 smithi045 ceph-mon[31773]: pgmap v2372: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:27.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2905.725472 secs 2024-03-28T19:03:28.442 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4261 sec 2024-03-28T19:03:29.443 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4262 sec 2024-03-28T19:03:29.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:29 smithi071 ceph-mon[36597]: pgmap v2373: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:29.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:29 smithi151 ceph-mon[36481]: pgmap v2373: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:29.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:29 smithi045 ceph-mon[31773]: pgmap v2373: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:30.443 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4263 sec 2024-03-28T19:03:31.443 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4264 sec 2024-03-28T19:03:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:31 smithi071 ceph-mon[36597]: pgmap v2374: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3619.300284 secs 2024-03-28T19:03:31.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:31 smithi151 ceph-mon[36481]: pgmap v2374: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:31.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3619.300284 secs 2024-03-28T19:03:31.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:31 smithi045 ceph-mon[31773]: pgmap v2374: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:31.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3619.300284 secs 2024-03-28T19:03:32.443 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4265 sec 2024-03-28T19:03:32.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2910.725558 secs 2024-03-28T19:03:32.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2910.725558 secs 2024-03-28T19:03:32.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2910.725558 secs 2024-03-28T19:03:33.443 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4266 sec 2024-03-28T19:03:33.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:33 smithi071 ceph-mon[36597]: pgmap v2375: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:33.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:33 smithi151 ceph-mon[36481]: pgmap v2375: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:33.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:33 smithi045 ceph-mon[31773]: pgmap v2375: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:34.443 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4267 sec 2024-03-28T19:03:35.443 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4268 sec 2024-03-28T19:03:35.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:35 smithi071 ceph-mon[36597]: pgmap v2376: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:35.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:35 smithi151 ceph-mon[36481]: pgmap v2376: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:35.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:35 smithi045 ceph-mon[31773]: pgmap v2376: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:36.443 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4269 sec 2024-03-28T19:03:36.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3624.300374 secs 2024-03-28T19:03:36.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3624.300374 secs 2024-03-28T19:03:36.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3624.300374 secs 2024-03-28T19:03:37.444 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4270 sec 2024-03-28T19:03:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:37 smithi071 ceph-mon[36597]: pgmap v2377: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2915.725648 secs 2024-03-28T19:03:37.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:37 smithi151 ceph-mon[36481]: pgmap v2377: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2915.725648 secs 2024-03-28T19:03:37.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:37 smithi045 ceph-mon[31773]: pgmap v2377: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:37.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2915.725648 secs 2024-03-28T19:03:38.444 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4271 sec 2024-03-28T19:03:39.444 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4272 sec 2024-03-28T19:03:39.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:39 smithi071 ceph-mon[36597]: pgmap v2378: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:39.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:39 smithi151 ceph-mon[36481]: pgmap v2378: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:39.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:39 smithi045 ceph-mon[31773]: pgmap v2378: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:40.444 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4273 sec 2024-03-28T19:03:41.444 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4274 sec 2024-03-28T19:03:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:41 smithi071 ceph-mon[36597]: pgmap v2379: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:03:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3629.300436 secs 2024-03-28T19:03:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:41 smithi151 ceph-mon[36481]: pgmap v2379: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:03:41.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3629.300436 secs 2024-03-28T19:03:41.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:41 smithi045 ceph-mon[31773]: pgmap v2379: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:03:41.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3629.300436 secs 2024-03-28T19:03:42.444 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4275 sec 2024-03-28T19:03:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2920.725737 secs 2024-03-28T19:03:42.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2920.725737 secs 2024-03-28T19:03:42.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2920.725737 secs 2024-03-28T19:03:43.445 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4276 sec 2024-03-28T19:03:43.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:43 smithi071 ceph-mon[36597]: pgmap v2380: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-03-28T19:03:43.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:43 smithi151 ceph-mon[36481]: pgmap v2380: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-03-28T19:03:43.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:43 smithi045 ceph-mon[31773]: pgmap v2380: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-03-28T19:03:44.445 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4277 sec 2024-03-28T19:03:45.445 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4278 sec 2024-03-28T19:03:45.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:45 smithi071 ceph-mon[36597]: pgmap v2381: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-03-28T19:03:45.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:45 smithi151 ceph-mon[36481]: pgmap v2381: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-03-28T19:03:45.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:45 smithi045 ceph-mon[31773]: pgmap v2381: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-03-28T19:03:46.445 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4279 sec 2024-03-28T19:03:46.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3634.300499 secs 2024-03-28T19:03:46.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3634.300499 secs 2024-03-28T19:03:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3634.300499 secs 2024-03-28T19:03:47.445 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4280 sec 2024-03-28T19:03:47.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:47 smithi071 ceph-mon[36597]: pgmap v2382: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:03:47.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2925.725797 secs 2024-03-28T19:03:47.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:47 smithi151 ceph-mon[36481]: pgmap v2382: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:03:47.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2925.725797 secs 2024-03-28T19:03:47.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:47 smithi045 ceph-mon[31773]: pgmap v2382: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:03:47.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2925.725797 secs 2024-03-28T19:03:48.445 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4281 sec 2024-03-28T19:03:49.445 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4282 sec 2024-03-28T19:03:49.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:49 smithi151 ceph-mon[36481]: pgmap v2383: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-03-28T19:03:49.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:49 smithi045 ceph-mon[31773]: pgmap v2383: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-03-28T19:03:50.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:49 smithi071 ceph-mon[36597]: pgmap v2383: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-03-28T19:03:50.446 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4283 sec 2024-03-28T19:03:51.446 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4284 sec 2024-03-28T19:03:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:51 smithi151 ceph-mon[36481]: pgmap v2384: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:03:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3639.300586 secs 2024-03-28T19:03:51.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:51 smithi045 ceph-mon[31773]: pgmap v2384: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:03:51.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3639.300586 secs 2024-03-28T19:03:52.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:51 smithi071 ceph-mon[36597]: pgmap v2384: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:03:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3639.300586 secs 2024-03-28T19:03:52.446 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4285 sec 2024-03-28T19:03:52.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2930.725886 secs 2024-03-28T19:03:53.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2930.725886 secs 2024-03-28T19:03:53.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2930.725886 secs 2024-03-28T19:03:53.446 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4286 sec 2024-03-28T19:03:53.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:53 smithi045 ceph-mon[31773]: pgmap v2385: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:54.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:53 smithi071 ceph-mon[36597]: pgmap v2385: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:54.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:53 smithi151 ceph-mon[36481]: pgmap v2385: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:54.446 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4287 sec 2024-03-28T19:03:55.447 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4288 sec 2024-03-28T19:03:55.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:55 smithi045 ceph-mon[31773]: pgmap v2386: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:56.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:55 smithi071 ceph-mon[36597]: pgmap v2386: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:56.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:55 smithi151 ceph-mon[36481]: pgmap v2386: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:03:56.447 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4289 sec 2024-03-28T19:03:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3644.300650 secs 2024-03-28T19:03:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3644.300650 secs 2024-03-28T19:03:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3644.300650 secs 2024-03-28T19:03:57.447 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4290 sec 2024-03-28T19:03:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:57 smithi045 ceph-mon[31773]: pgmap v2387: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2935.725976 secs 2024-03-28T19:03:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:57 smithi071 ceph-mon[36597]: pgmap v2387: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2935.725976 secs 2024-03-28T19:03:58.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:57 smithi151 ceph-mon[36481]: pgmap v2387: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:03:58.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2935.725976 secs 2024-03-28T19:03:58.447 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4291 sec 2024-03-28T19:03:59.447 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4292 sec 2024-03-28T19:03:59.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:03:59 smithi045 ceph-mon[31773]: pgmap v2388: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:00.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:03:59 smithi071 ceph-mon[36597]: pgmap v2388: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:00.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:03:59 smithi151 ceph-mon[36481]: pgmap v2388: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:00.447 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4293 sec 2024-03-28T19:04:01.447 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4294 sec 2024-03-28T19:04:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:01 smithi045 ceph-mon[31773]: pgmap v2389: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3649.300711 secs 2024-03-28T19:04:02.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:01 smithi071 ceph-mon[36597]: pgmap v2389: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3649.300711 secs 2024-03-28T19:04:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:01 smithi151 ceph-mon[36481]: pgmap v2389: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3649.300711 secs 2024-03-28T19:04:02.447 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4295 sec 2024-03-28T19:04:02.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2940.725994 secs 2024-03-28T19:04:03.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2940.725994 secs 2024-03-28T19:04:03.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2940.725994 secs 2024-03-28T19:04:03.447 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4296 sec 2024-03-28T19:04:03.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:03 smithi045 ceph-mon[31773]: pgmap v2390: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:03.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:03 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:04:04.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:03 smithi071 ceph-mon[36597]: pgmap v2390: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:04.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:03 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:04:04.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:03 smithi151 ceph-mon[36481]: pgmap v2390: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:04.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:03 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:04:04.448 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4297 sec 2024-03-28T19:04:04.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:04 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:04:04.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:04 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:04:04.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:04 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:04:05.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:04 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:04:05.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:04 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:04:05.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:04 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:04:05.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:04 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:04:05.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:04 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:04:05.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:04 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:04:05.447 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4298 sec 2024-03-28T19:04:05.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:05 smithi045 ceph-mon[31773]: pgmap v2391: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:05.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:05 smithi045 ceph-mon[31773]: pgmap v2392: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:06.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:05 smithi071 ceph-mon[36597]: pgmap v2391: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:06.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:05 smithi071 ceph-mon[36597]: pgmap v2392: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:06.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:05 smithi151 ceph-mon[36481]: pgmap v2391: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:06.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:05 smithi151 ceph-mon[36481]: pgmap v2392: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:06.448 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4299 sec 2024-03-28T19:04:06.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3654.300799 secs 2024-03-28T19:04:07.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3654.300799 secs 2024-03-28T19:04:07.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3654.300799 secs 2024-03-28T19:04:07.448 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4300 sec 2024-03-28T19:04:07.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:07 smithi045 ceph-mon[31773]: pgmap v2393: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:07.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2945.726084 secs 2024-03-28T19:04:08.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:07 smithi071 ceph-mon[36597]: pgmap v2393: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:08.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2945.726084 secs 2024-03-28T19:04:08.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:07 smithi151 ceph-mon[36481]: pgmap v2393: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:08.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2945.726084 secs 2024-03-28T19:04:08.448 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4301 sec 2024-03-28T19:04:09.448 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4302 sec 2024-03-28T19:04:09.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:09 smithi045 ceph-mon[31773]: pgmap v2394: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:10.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:09 smithi071 ceph-mon[36597]: pgmap v2394: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:10.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:09 smithi151 ceph-mon[36481]: pgmap v2394: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:10.448 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4303 sec 2024-03-28T19:04:11.448 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4304 sec 2024-03-28T19:04:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:11 smithi071 ceph-mon[36597]: pgmap v2395: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3659.300888 secs 2024-03-28T19:04:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:11 smithi151 ceph-mon[36481]: pgmap v2395: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3659.300888 secs 2024-03-28T19:04:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:11 smithi045 ceph-mon[31773]: pgmap v2395: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3659.300888 secs 2024-03-28T19:04:12.448 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4305 sec 2024-03-28T19:04:12.973 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2950.726146 secs 2024-03-28T19:04:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2950.726146 secs 2024-03-28T19:04:13.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2950.726146 secs 2024-03-28T19:04:13.449 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4306 sec 2024-03-28T19:04:14.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:13 smithi071 ceph-mon[36597]: pgmap v2396: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:14.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:13 smithi151 ceph-mon[36481]: pgmap v2396: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:14.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:13 smithi045 ceph-mon[31773]: pgmap v2396: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:04:14.449 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4307 sec 2024-03-28T19:04:15.449 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4308 sec 2024-03-28T19:04:16.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:15 smithi071 ceph-mon[36597]: pgmap v2397: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:04:16.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:15 smithi151 ceph-mon[36481]: pgmap v2397: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:04:16.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:15 smithi045 ceph-mon[31773]: pgmap v2397: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:04:16.449 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4309 sec 2024-03-28T19:04:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3664.300976 secs 2024-03-28T19:04:17.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3664.300976 secs 2024-03-28T19:04:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3664.300976 secs 2024-03-28T19:04:17.449 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4310 sec 2024-03-28T19:04:17.961 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:17 smithi071 ceph-mon[36597]: pgmap v2398: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:17.961 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2955.726233 secs 2024-03-28T19:04:18.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:17 smithi151 ceph-mon[36481]: pgmap v2398: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:18.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2955.726233 secs 2024-03-28T19:04:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:17 smithi045 ceph-mon[31773]: pgmap v2398: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2955.726233 secs 2024-03-28T19:04:18.450 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4311 sec 2024-03-28T19:04:19.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:18 smithi071 ceph-mon[36597]: pgmap v2399: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:19.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:18 smithi151 ceph-mon[36481]: pgmap v2399: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:19.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:18 smithi045 ceph-mon[31773]: pgmap v2399: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:19.450 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4312 sec 2024-03-28T19:04:20.450 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4313 sec 2024-03-28T19:04:21.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:21 smithi045 ceph-mon[31773]: pgmap v2400: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:21.450 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4314 sec 2024-03-28T19:04:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:21 smithi071 ceph-mon[36597]: pgmap v2400: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:21.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:21 smithi151 ceph-mon[36481]: pgmap v2400: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:22.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:22 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3669.301064 secs 2024-03-28T19:04:22.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2960.726293 secs 2024-03-28T19:04:22.450 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4315 sec 2024-03-28T19:04:22.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:22 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3669.301064 secs 2024-03-28T19:04:22.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2960.726293 secs 2024-03-28T19:04:22.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:22 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3669.301064 secs 2024-03-28T19:04:22.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2960.726293 secs 2024-03-28T19:04:23.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:23 smithi045 ceph-mon[31773]: pgmap v2401: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:23.450 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4316 sec 2024-03-28T19:04:23.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:23 smithi071 ceph-mon[36597]: pgmap v2401: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:23.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:23 smithi151 ceph-mon[36481]: pgmap v2401: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:24.450 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4317 sec 2024-03-28T19:04:25.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:25 smithi045 ceph-mon[31773]: pgmap v2402: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:25.451 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4318 sec 2024-03-28T19:04:25.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:25 smithi071 ceph-mon[36597]: pgmap v2402: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:25.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:25 smithi151 ceph-mon[36481]: pgmap v2402: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:26.451 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4319 sec 2024-03-28T19:04:27.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:27 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3674.301105 secs 2024-03-28T19:04:27.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:27 smithi045 ceph-mon[31773]: pgmap v2403: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:27.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2965.726382 secs 2024-03-28T19:04:27.451 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4320 sec 2024-03-28T19:04:27.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:27 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3674.301105 secs 2024-03-28T19:04:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:27 smithi071 ceph-mon[36597]: pgmap v2403: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2965.726382 secs 2024-03-28T19:04:27.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:27 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3674.301105 secs 2024-03-28T19:04:27.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:27 smithi151 ceph-mon[36481]: pgmap v2403: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2965.726382 secs 2024-03-28T19:04:28.451 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4321 sec 2024-03-28T19:04:29.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:29 smithi045 ceph-mon[31773]: pgmap v2404: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:29.451 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4322 sec 2024-03-28T19:04:29.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:29 smithi071 ceph-mon[36597]: pgmap v2404: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:29.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:29 smithi151 ceph-mon[36481]: pgmap v2404: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:30.451 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4323 sec 2024-03-28T19:04:31.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:31 smithi045 ceph-mon[31773]: pgmap v2405: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:31.451 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4324 sec 2024-03-28T19:04:31.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:31 smithi071 ceph-mon[36597]: pgmap v2405: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:31.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:31 smithi151 ceph-mon[36481]: pgmap v2405: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:32.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:32 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3679.301144 secs 2024-03-28T19:04:32.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2970.726443 secs 2024-03-28T19:04:32.451 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4325 sec 2024-03-28T19:04:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:32 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3679.301144 secs 2024-03-28T19:04:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2970.726443 secs 2024-03-28T19:04:32.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:32 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3679.301144 secs 2024-03-28T19:04:32.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2970.726443 secs 2024-03-28T19:04:33.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:33 smithi045 ceph-mon[31773]: pgmap v2406: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:33.451 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4326 sec 2024-03-28T19:04:33.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:33 smithi071 ceph-mon[36597]: pgmap v2406: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:33.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:33 smithi151 ceph-mon[36481]: pgmap v2406: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:34.452 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4327 sec 2024-03-28T19:04:35.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:35 smithi045 ceph-mon[31773]: pgmap v2407: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:35.452 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4328 sec 2024-03-28T19:04:35.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:35 smithi071 ceph-mon[36597]: pgmap v2407: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:35.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:35 smithi151 ceph-mon[36481]: pgmap v2407: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:36.452 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4329 sec 2024-03-28T19:04:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:37 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3684.301232 secs 2024-03-28T19:04:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:37 smithi045 ceph-mon[31773]: pgmap v2408: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2975.726532 secs 2024-03-28T19:04:37.452 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4330 sec 2024-03-28T19:04:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:37 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3684.301232 secs 2024-03-28T19:04:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:37 smithi071 ceph-mon[36597]: pgmap v2408: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:37.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2975.726532 secs 2024-03-28T19:04:37.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:37 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3684.301232 secs 2024-03-28T19:04:37.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:37 smithi151 ceph-mon[36481]: pgmap v2408: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:37.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2975.726532 secs 2024-03-28T19:04:38.452 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4331 sec 2024-03-28T19:04:39.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:39 smithi045 ceph-mon[31773]: pgmap v2409: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:39.452 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4332 sec 2024-03-28T19:04:39.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:39 smithi071 ceph-mon[36597]: pgmap v2409: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:39.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:39 smithi151 ceph-mon[36481]: pgmap v2409: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:40.453 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4333 sec 2024-03-28T19:04:41.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:41 smithi045 ceph-mon[31773]: pgmap v2410: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:41.453 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4334 sec 2024-03-28T19:04:41.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:41 smithi071 ceph-mon[36597]: pgmap v2410: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:41.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:41 smithi151 ceph-mon[36481]: pgmap v2410: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:42 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3689.301318 secs 2024-03-28T19:04:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2980.726617 secs 2024-03-28T19:04:42.453 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4335 sec 2024-03-28T19:04:42.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:42 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3689.301318 secs 2024-03-28T19:04:42.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2980.726617 secs 2024-03-28T19:04:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:42 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3689.301318 secs 2024-03-28T19:04:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2980.726617 secs 2024-03-28T19:04:43.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:43 smithi045 ceph-mon[31773]: pgmap v2411: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:43.453 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4336 sec 2024-03-28T19:04:43.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:43 smithi071 ceph-mon[36597]: pgmap v2411: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:43.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:43 smithi151 ceph-mon[36481]: pgmap v2411: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:44.453 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4337 sec 2024-03-28T19:04:45.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:45 smithi045 ceph-mon[31773]: pgmap v2412: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:45.453 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4338 sec 2024-03-28T19:04:45.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:45 smithi071 ceph-mon[36597]: pgmap v2412: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:45.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:45 smithi151 ceph-mon[36481]: pgmap v2412: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:46.453 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4339 sec 2024-03-28T19:04:47.453 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4340 sec 2024-03-28T19:04:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:47 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3694.301379 secs 2024-03-28T19:04:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:47 smithi071 ceph-mon[36597]: pgmap v2413: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2985.726679 secs 2024-03-28T19:04:47.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:47 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3694.301379 secs 2024-03-28T19:04:47.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:47 smithi151 ceph-mon[36481]: pgmap v2413: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:47.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2985.726679 secs 2024-03-28T19:04:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:47 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3694.301379 secs 2024-03-28T19:04:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:47 smithi045 ceph-mon[31773]: pgmap v2413: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2985.726679 secs 2024-03-28T19:04:48.454 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4341 sec 2024-03-28T19:04:49.454 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4342 sec 2024-03-28T19:04:49.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:49 smithi071 ceph-mon[36597]: pgmap v2414: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:49.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:49 smithi151 ceph-mon[36481]: pgmap v2414: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:49.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:49 smithi045 ceph-mon[31773]: pgmap v2414: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:50.454 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4343 sec 2024-03-28T19:04:51.454 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4344 sec 2024-03-28T19:04:51.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:51 smithi071 ceph-mon[36597]: pgmap v2415: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:51.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:51 smithi151 ceph-mon[36481]: pgmap v2415: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:51.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:51 smithi045 ceph-mon[31773]: pgmap v2415: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:52.454 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4345 sec 2024-03-28T19:04:52.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:52 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3699.301440 secs 2024-03-28T19:04:52.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2990.726741 secs 2024-03-28T19:04:52.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:52 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3699.301440 secs 2024-03-28T19:04:52.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2990.726741 secs 2024-03-28T19:04:52.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:52 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3699.301440 secs 2024-03-28T19:04:52.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2990.726741 secs 2024-03-28T19:04:53.454 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4346 sec 2024-03-28T19:04:53.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:53 smithi071 ceph-mon[36597]: pgmap v2416: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:53.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:53 smithi151 ceph-mon[36481]: pgmap v2416: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:53.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:53 smithi045 ceph-mon[31773]: pgmap v2416: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:54.454 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4347 sec 2024-03-28T19:04:55.454 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4348 sec 2024-03-28T19:04:55.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:55 smithi071 ceph-mon[36597]: pgmap v2417: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:55.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:55 smithi151 ceph-mon[36481]: pgmap v2417: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:55.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:55 smithi045 ceph-mon[31773]: pgmap v2417: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:56.454 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4349 sec 2024-03-28T19:04:57.455 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4350 sec 2024-03-28T19:04:57.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:57 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3704.301527 secs 2024-03-28T19:04:57.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:57 smithi071 ceph-mon[36597]: pgmap v2418: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 2995.726803 secs 2024-03-28T19:04:57.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:57 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3704.301527 secs 2024-03-28T19:04:57.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:57 smithi151 ceph-mon[36481]: pgmap v2418: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:57.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 2995.726803 secs 2024-03-28T19:04:57.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:57 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3704.301527 secs 2024-03-28T19:04:57.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:57 smithi045 ceph-mon[31773]: pgmap v2418: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:04:57.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 2995.726803 secs 2024-03-28T19:04:58.455 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4351 sec 2024-03-28T19:04:59.455 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4352 sec 2024-03-28T19:04:59.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:04:59 smithi071 ceph-mon[36597]: pgmap v2419: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:59.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:04:59 smithi151 ceph-mon[36481]: pgmap v2419: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:04:59.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:04:59 smithi045 ceph-mon[31773]: pgmap v2419: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:00.455 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4353 sec 2024-03-28T19:05:01.456 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4354 sec 2024-03-28T19:05:01.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:01 smithi071 ceph-mon[36597]: pgmap v2420: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:01.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:01 smithi151 ceph-mon[36481]: pgmap v2420: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:01.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:01 smithi045 ceph-mon[31773]: pgmap v2420: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:02.456 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4355 sec 2024-03-28T19:05:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:02 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3709.301591 secs 2024-03-28T19:05:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3000.726887 secs 2024-03-28T19:05:02.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:02 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3709.301591 secs 2024-03-28T19:05:02.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3000.726887 secs 2024-03-28T19:05:02.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:02 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3709.301591 secs 2024-03-28T19:05:02.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3000.726887 secs 2024-03-28T19:05:03.456 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4356 sec 2024-03-28T19:05:03.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:03 smithi071 ceph-mon[36597]: pgmap v2421: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:03.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:03 smithi151 ceph-mon[36481]: pgmap v2421: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:03.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:03 smithi045 ceph-mon[31773]: pgmap v2421: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:04.456 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4357 sec 2024-03-28T19:05:05.456 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4358 sec 2024-03-28T19:05:05.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:05 smithi071 ceph-mon[36597]: pgmap v2422: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:05.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:05 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:05:05.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:05 smithi151 ceph-mon[36481]: pgmap v2422: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:05.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:05 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:05:05.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:05 smithi045 ceph-mon[31773]: pgmap v2422: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:05.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:05 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:05:06.456 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4359 sec 2024-03-28T19:05:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:06 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:05:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:06 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:05:06.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:06 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:05:06.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:06 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:05:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:06 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:05:06.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:06 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:05:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:06 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:05:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:06 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:05:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:06 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:05:07.456 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4360 sec 2024-03-28T19:05:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:07 smithi071 ceph-mon[36597]: pgmap v2423: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:05:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:07 smithi071 ceph-mon[36597]: pgmap v2424: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:07 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3714.301653 secs 2024-03-28T19:05:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3005.726954 secs 2024-03-28T19:05:07.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:07 smithi151 ceph-mon[36481]: pgmap v2423: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:05:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:07 smithi151 ceph-mon[36481]: pgmap v2424: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:07 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3714.301653 secs 2024-03-28T19:05:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3005.726954 secs 2024-03-28T19:05:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:07 smithi045 ceph-mon[31773]: pgmap v2423: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:05:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:07 smithi045 ceph-mon[31773]: pgmap v2424: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:07 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3714.301653 secs 2024-03-28T19:05:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3005.726954 secs 2024-03-28T19:05:08.456 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4361 sec 2024-03-28T19:05:09.456 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4362 sec 2024-03-28T19:05:09.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:09 smithi071 ceph-mon[36597]: pgmap v2425: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:09.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:09 smithi151 ceph-mon[36481]: pgmap v2425: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:09.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:09 smithi045 ceph-mon[31773]: pgmap v2425: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:10.456 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4363 sec 2024-03-28T19:05:11.457 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4364 sec 2024-03-28T19:05:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:11 smithi071 ceph-mon[36597]: pgmap v2426: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:05:11.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3719.301713 secs 2024-03-28T19:05:11.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:11 smithi151 ceph-mon[36481]: pgmap v2426: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:05:11.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3719.301713 secs 2024-03-28T19:05:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:11 smithi045 ceph-mon[31773]: pgmap v2426: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:05:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3719.301713 secs 2024-03-28T19:05:12.457 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4365 sec 2024-03-28T19:05:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3010.727038 secs 2024-03-28T19:05:12.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3010.727038 secs 2024-03-28T19:05:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3010.727038 secs 2024-03-28T19:05:13.457 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4366 sec 2024-03-28T19:05:13.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:13 smithi071 ceph-mon[36597]: pgmap v2427: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:13.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:13 smithi045 ceph-mon[31773]: pgmap v2427: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:13.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:13 smithi151 ceph-mon[36481]: pgmap v2427: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:14.457 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4367 sec 2024-03-28T19:05:15.457 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4368 sec 2024-03-28T19:05:15.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:15 smithi071 ceph-mon[36597]: pgmap v2428: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:15.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:15 smithi151 ceph-mon[36481]: pgmap v2428: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:15.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:15 smithi045 ceph-mon[31773]: pgmap v2428: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:05:16.457 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4369 sec 2024-03-28T19:05:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3724.301746 secs 2024-03-28T19:05:16.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3724.301746 secs 2024-03-28T19:05:16.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3724.301746 secs 2024-03-28T19:05:17.458 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4370 sec 2024-03-28T19:05:17.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:17 smithi071 ceph-mon[36597]: pgmap v2429: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:05:17.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3015.727103 secs 2024-03-28T19:05:17.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:17 smithi151 ceph-mon[36481]: pgmap v2429: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:05:17.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3015.727103 secs 2024-03-28T19:05:17.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:17 smithi045 ceph-mon[31773]: pgmap v2429: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:05:17.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3015.727103 secs 2024-03-28T19:05:18.458 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4371 sec 2024-03-28T19:05:19.458 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4372 sec 2024-03-28T19:05:19.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:19 smithi071 ceph-mon[36597]: pgmap v2430: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:19.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:19 smithi151 ceph-mon[36481]: pgmap v2430: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:19.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:19 smithi045 ceph-mon[31773]: pgmap v2430: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:20.458 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4373 sec 2024-03-28T19:05:21.458 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4374 sec 2024-03-28T19:05:21.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:21 smithi071 ceph-mon[36597]: pgmap v2431: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:21.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3729.301806 secs 2024-03-28T19:05:21.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:21 smithi151 ceph-mon[36481]: pgmap v2431: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:21.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3729.301806 secs 2024-03-28T19:05:21.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:21 smithi045 ceph-mon[31773]: pgmap v2431: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:21.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3729.301806 secs 2024-03-28T19:05:22.458 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4375 sec 2024-03-28T19:05:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3020.727174 secs 2024-03-28T19:05:22.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3020.727174 secs 2024-03-28T19:05:22.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3020.727174 secs 2024-03-28T19:05:23.458 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4376 sec 2024-03-28T19:05:23.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:23 smithi071 ceph-mon[36597]: pgmap v2432: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:23.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:23 smithi151 ceph-mon[36481]: pgmap v2432: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:23.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:23 smithi045 ceph-mon[31773]: pgmap v2432: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:24.459 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4377 sec 2024-03-28T19:05:25.458 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4378 sec 2024-03-28T19:05:25.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:25 smithi071 ceph-mon[36597]: pgmap v2433: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:25.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:25 smithi151 ceph-mon[36481]: pgmap v2433: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:25.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:25 smithi045 ceph-mon[31773]: pgmap v2433: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:26.459 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4379 sec 2024-03-28T19:05:26.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3734.301866 secs 2024-03-28T19:05:26.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3734.301866 secs 2024-03-28T19:05:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3734.301866 secs 2024-03-28T19:05:27.459 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4380 sec 2024-03-28T19:05:27.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:27 smithi151 ceph-mon[36481]: pgmap v2434: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:27.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3025.727187 secs 2024-03-28T19:05:27.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:27 smithi045 ceph-mon[31773]: pgmap v2434: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:27.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3025.727187 secs 2024-03-28T19:05:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:27 smithi071 ceph-mon[36597]: pgmap v2434: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3025.727187 secs 2024-03-28T19:05:28.459 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4381 sec 2024-03-28T19:05:29.459 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4382 sec 2024-03-28T19:05:29.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:29 smithi045 ceph-mon[31773]: pgmap v2435: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:29.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:29 smithi071 ceph-mon[36597]: pgmap v2435: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:29.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:29 smithi151 ceph-mon[36481]: pgmap v2435: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:30.459 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4383 sec 2024-03-28T19:05:31.459 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4384 sec 2024-03-28T19:05:31.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:31 smithi045 ceph-mon[31773]: pgmap v2436: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:31.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3739.301953 secs 2024-03-28T19:05:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:31 smithi071 ceph-mon[36597]: pgmap v2436: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3739.301953 secs 2024-03-28T19:05:31.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:31 smithi151 ceph-mon[36481]: pgmap v2436: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:31.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3739.301953 secs 2024-03-28T19:05:32.460 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4385 sec 2024-03-28T19:05:32.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3030.727247 secs 2024-03-28T19:05:32.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3030.727247 secs 2024-03-28T19:05:32.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3030.727247 secs 2024-03-28T19:05:33.460 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4386 sec 2024-03-28T19:05:33.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:33 smithi045 ceph-mon[31773]: pgmap v2437: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:33.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:33 smithi071 ceph-mon[36597]: pgmap v2437: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:33.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:33 smithi151 ceph-mon[36481]: pgmap v2437: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:34.460 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4387 sec 2024-03-28T19:05:35.460 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4388 sec 2024-03-28T19:05:35.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:35 smithi045 ceph-mon[31773]: pgmap v2438: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:35.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:35 smithi071 ceph-mon[36597]: pgmap v2438: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:35.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:35 smithi151 ceph-mon[36481]: pgmap v2438: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:36.460 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4389 sec 2024-03-28T19:05:36.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3744.301988 secs 2024-03-28T19:05:36.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3744.301988 secs 2024-03-28T19:05:36.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3744.301988 secs 2024-03-28T19:05:37.460 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4390 sec 2024-03-28T19:05:37.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:37 smithi045 ceph-mon[31773]: pgmap v2439: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:37.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3035.727308 secs 2024-03-28T19:05:37.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:37 smithi071 ceph-mon[36597]: pgmap v2439: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3035.727308 secs 2024-03-28T19:05:37.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:37 smithi151 ceph-mon[36481]: pgmap v2439: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:37.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3035.727308 secs 2024-03-28T19:05:38.460 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4391 sec 2024-03-28T19:05:39.460 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4392 sec 2024-03-28T19:05:39.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:39 smithi045 ceph-mon[31773]: pgmap v2440: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:39.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:39 smithi071 ceph-mon[36597]: pgmap v2440: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:39.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:39 smithi151 ceph-mon[36481]: pgmap v2440: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:40.461 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4393 sec 2024-03-28T19:05:41.461 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4394 sec 2024-03-28T19:05:41.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:41 smithi045 ceph-mon[31773]: pgmap v2441: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:41.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3749.302048 secs 2024-03-28T19:05:41.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:41 smithi071 ceph-mon[36597]: pgmap v2441: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3749.302048 secs 2024-03-28T19:05:41.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:41 smithi151 ceph-mon[36481]: pgmap v2441: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:41.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3749.302048 secs 2024-03-28T19:05:42.461 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4395 sec 2024-03-28T19:05:42.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3040.727396 secs 2024-03-28T19:05:42.723 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3040.727396 secs 2024-03-28T19:05:42.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3040.727396 secs 2024-03-28T19:05:43.461 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4396 sec 2024-03-28T19:05:43.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:43 smithi045 ceph-mon[31773]: pgmap v2442: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:43.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:43 smithi071 ceph-mon[36597]: pgmap v2442: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:43.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:43 smithi151 ceph-mon[36481]: pgmap v2442: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:44.461 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4397 sec 2024-03-28T19:05:45.461 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4398 sec 2024-03-28T19:05:45.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:45 smithi045 ceph-mon[31773]: pgmap v2443: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:45.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:45 smithi071 ceph-mon[36597]: pgmap v2443: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:45.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:45 smithi151 ceph-mon[36481]: pgmap v2443: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:46.461 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4399 sec 2024-03-28T19:05:46.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3754.302137 secs 2024-03-28T19:05:46.710 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3754.302137 secs 2024-03-28T19:05:46.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3754.302137 secs 2024-03-28T19:05:47.462 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4400 sec 2024-03-28T19:05:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:47 smithi045 ceph-mon[31773]: pgmap v2444: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3045.727481 secs 2024-03-28T19:05:47.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:47 smithi071 ceph-mon[36597]: pgmap v2444: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:47.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3045.727481 secs 2024-03-28T19:05:47.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:47 smithi151 ceph-mon[36481]: pgmap v2444: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:47.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3045.727481 secs 2024-03-28T19:05:48.462 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4401 sec 2024-03-28T19:05:49.462 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4402 sec 2024-03-28T19:05:49.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:49 smithi045 ceph-mon[31773]: pgmap v2445: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:49.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:49 smithi071 ceph-mon[36597]: pgmap v2445: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:49.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:49 smithi151 ceph-mon[36481]: pgmap v2445: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:50.462 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4403 sec 2024-03-28T19:05:51.462 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4404 sec 2024-03-28T19:05:51.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:51 smithi045 ceph-mon[31773]: pgmap v2446: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:51.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3759.302224 secs 2024-03-28T19:05:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:51 smithi071 ceph-mon[36597]: pgmap v2446: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3759.302224 secs 2024-03-28T19:05:51.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:51 smithi151 ceph-mon[36481]: pgmap v2446: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:51.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3759.302224 secs 2024-03-28T19:05:52.462 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4405 sec 2024-03-28T19:05:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3050.727543 secs 2024-03-28T19:05:52.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3050.727543 secs 2024-03-28T19:05:52.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3050.727543 secs 2024-03-28T19:05:53.463 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4406 sec 2024-03-28T19:05:53.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:53 smithi071 ceph-mon[36597]: pgmap v2447: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:53.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:53 smithi151 ceph-mon[36481]: pgmap v2447: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:53.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:53 smithi045 ceph-mon[31773]: pgmap v2447: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:54.463 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4407 sec 2024-03-28T19:05:55.463 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4408 sec 2024-03-28T19:05:55.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:55 smithi071 ceph-mon[36597]: pgmap v2448: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:55.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:55 smithi151 ceph-mon[36481]: pgmap v2448: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:55.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:55 smithi045 ceph-mon[31773]: pgmap v2448: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:56.463 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4409 sec 2024-03-28T19:05:56.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3764.302284 secs 2024-03-28T19:05:56.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3764.302284 secs 2024-03-28T19:05:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3764.302284 secs 2024-03-28T19:05:57.463 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4410 sec 2024-03-28T19:05:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:57 smithi071 ceph-mon[36597]: pgmap v2449: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3055.727604 secs 2024-03-28T19:05:57.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:57 smithi151 ceph-mon[36481]: pgmap v2449: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:57.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3055.727604 secs 2024-03-28T19:05:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:57 smithi045 ceph-mon[31773]: pgmap v2449: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:05:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3055.727604 secs 2024-03-28T19:05:58.463 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4411 sec 2024-03-28T19:05:59.463 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4412 sec 2024-03-28T19:05:59.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:05:59 smithi071 ceph-mon[36597]: pgmap v2450: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:59.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:05:59 smithi151 ceph-mon[36481]: pgmap v2450: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:05:59.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:05:59 smithi045 ceph-mon[31773]: pgmap v2450: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:00.464 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4413 sec 2024-03-28T19:06:01.464 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4414 sec 2024-03-28T19:06:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:01 smithi071 ceph-mon[36597]: pgmap v2451: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3769.302325 secs 2024-03-28T19:06:01.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:01 smithi151 ceph-mon[36481]: pgmap v2451: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:01.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3769.302325 secs 2024-03-28T19:06:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:01 smithi045 ceph-mon[31773]: pgmap v2451: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3769.302325 secs 2024-03-28T19:06:02.464 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4415 sec 2024-03-28T19:06:02.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3060.727619 secs 2024-03-28T19:06:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3060.727619 secs 2024-03-28T19:06:02.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3060.727619 secs 2024-03-28T19:06:03.464 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4416 sec 2024-03-28T19:06:03.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:03 smithi071 ceph-mon[36597]: pgmap v2452: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:03.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:03 smithi151 ceph-mon[36481]: pgmap v2452: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:03.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:03 smithi045 ceph-mon[31773]: pgmap v2452: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:04.464 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4417 sec 2024-03-28T19:06:05.465 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4418 sec 2024-03-28T19:06:05.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:05 smithi071 ceph-mon[36597]: pgmap v2453: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:05.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:05 smithi151 ceph-mon[36481]: pgmap v2453: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:05.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:05 smithi045 ceph-mon[31773]: pgmap v2453: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:06.464 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4419 sec 2024-03-28T19:06:06.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3774.302415 secs 2024-03-28T19:06:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:06 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:06:06.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3774.302415 secs 2024-03-28T19:06:06.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:06 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:06:06.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3774.302415 secs 2024-03-28T19:06:06.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:06 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:06:07.464 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4420 sec 2024-03-28T19:06:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:07 smithi071 ceph-mon[36597]: pgmap v2454: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3065.727691 secs 2024-03-28T19:06:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:07 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:06:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:07 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:06:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:07 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:06:07.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:07 smithi151 ceph-mon[36481]: pgmap v2454: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:07.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3065.727691 secs 2024-03-28T19:06:07.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:07 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:06:07.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:07 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:06:07.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:07 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:06:07.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:07 smithi045 ceph-mon[31773]: pgmap v2454: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:07.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3065.727691 secs 2024-03-28T19:06:07.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:07 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:06:07.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:07 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:06:07.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:07 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:06:08.464 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4421 sec 2024-03-28T19:06:08.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:08 smithi071 ceph-mon[36597]: pgmap v2455: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:08.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:08 smithi071 ceph-mon[36597]: pgmap v2456: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:08.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:08 smithi151 ceph-mon[36481]: pgmap v2455: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:08.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:08 smithi151 ceph-mon[36481]: pgmap v2456: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:08.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:08 smithi045 ceph-mon[31773]: pgmap v2455: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:08.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:08 smithi045 ceph-mon[31773]: pgmap v2456: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:09.464 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4422 sec 2024-03-28T19:06:10.465 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4423 sec 2024-03-28T19:06:10.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:10 smithi071 ceph-mon[36597]: pgmap v2457: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:10.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:10 smithi151 ceph-mon[36481]: pgmap v2457: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:10.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:10 smithi045 ceph-mon[31773]: pgmap v2457: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:11.465 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4424 sec 2024-03-28T19:06:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3779.302490 secs 2024-03-28T19:06:11.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3779.302490 secs 2024-03-28T19:06:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3779.302490 secs 2024-03-28T19:06:12.465 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4425 sec 2024-03-28T19:06:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:12 smithi071 ceph-mon[36597]: pgmap v2458: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3070.727751 secs 2024-03-28T19:06:12.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:12 smithi151 ceph-mon[36481]: pgmap v2458: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:12.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3070.727751 secs 2024-03-28T19:06:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:12 smithi045 ceph-mon[31773]: pgmap v2458: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3070.727751 secs 2024-03-28T19:06:13.465 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4426 sec 2024-03-28T19:06:14.465 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4427 sec 2024-03-28T19:06:14.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:14 smithi071 ceph-mon[36597]: pgmap v2459: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:14.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:14 smithi151 ceph-mon[36481]: pgmap v2459: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:14.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:14 smithi045 ceph-mon[31773]: pgmap v2459: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:15.465 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4428 sec 2024-03-28T19:06:16.466 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4429 sec 2024-03-28T19:06:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:16 smithi071 ceph-mon[36597]: pgmap v2460: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3784.302574 secs 2024-03-28T19:06:16.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:16 smithi151 ceph-mon[36481]: pgmap v2460: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:16.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3784.302574 secs 2024-03-28T19:06:16.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:16 smithi045 ceph-mon[31773]: pgmap v2460: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:06:16.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3784.302574 secs 2024-03-28T19:06:17.466 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4430 sec 2024-03-28T19:06:17.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3075.727810 secs 2024-03-28T19:06:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3075.727810 secs 2024-03-28T19:06:17.918 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3075.727810 secs 2024-03-28T19:06:18.466 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4431 sec 2024-03-28T19:06:18.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:18 smithi071 ceph-mon[36597]: pgmap v2461: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:06:18.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:18 smithi151 ceph-mon[36481]: pgmap v2461: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:06:18.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:18 smithi045 ceph-mon[31773]: pgmap v2461: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:06:19.466 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4432 sec 2024-03-28T19:06:20.466 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4433 sec 2024-03-28T19:06:20.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:20 smithi071 ceph-mon[36597]: pgmap v2462: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:20.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:20 smithi151 ceph-mon[36481]: pgmap v2462: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:20.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:20 smithi045 ceph-mon[31773]: pgmap v2462: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:21.467 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4434 sec 2024-03-28T19:06:21.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3789.302635 secs 2024-03-28T19:06:21.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3789.302635 secs 2024-03-28T19:06:21.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3789.302635 secs 2024-03-28T19:06:22.466 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4435 sec 2024-03-28T19:06:22.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:22 smithi071 ceph-mon[36597]: pgmap v2463: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:22.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3080.727898 secs 2024-03-28T19:06:22.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:22 smithi151 ceph-mon[36481]: pgmap v2463: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:22.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3080.727898 secs 2024-03-28T19:06:22.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:22 smithi045 ceph-mon[31773]: pgmap v2463: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:22.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3080.727898 secs 2024-03-28T19:06:23.466 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4436 sec 2024-03-28T19:06:24.467 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4437 sec 2024-03-28T19:06:24.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:24 smithi071 ceph-mon[36597]: pgmap v2464: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:24.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:24 smithi151 ceph-mon[36481]: pgmap v2464: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:24.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:24 smithi045 ceph-mon[31773]: pgmap v2464: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:25.467 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4438 sec 2024-03-28T19:06:26.467 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4439 sec 2024-03-28T19:06:26.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:26 smithi071 ceph-mon[36597]: pgmap v2465: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3794.302719 secs 2024-03-28T19:06:26.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:26 smithi151 ceph-mon[36481]: pgmap v2465: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:26.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3794.302719 secs 2024-03-28T19:06:26.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:26 smithi045 ceph-mon[31773]: pgmap v2465: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:26.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3794.302719 secs 2024-03-28T19:06:27.467 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4440 sec 2024-03-28T19:06:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3085.727961 secs 2024-03-28T19:06:27.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3085.727961 secs 2024-03-28T19:06:27.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3085.727961 secs 2024-03-28T19:06:28.467 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4441 sec 2024-03-28T19:06:28.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:28 smithi151 ceph-mon[36481]: pgmap v2466: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:28.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:28 smithi045 ceph-mon[31773]: pgmap v2466: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:29.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:28 smithi071 ceph-mon[36597]: pgmap v2466: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:29.467 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4442 sec 2024-03-28T19:06:30.468 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4443 sec 2024-03-28T19:06:30.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:30 smithi151 ceph-mon[36481]: pgmap v2467: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:30.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:30 smithi045 ceph-mon[31773]: pgmap v2467: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:31.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:30 smithi071 ceph-mon[36597]: pgmap v2467: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:31.468 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4444 sec 2024-03-28T19:06:31.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3799.302797 secs 2024-03-28T19:06:32.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3799.302797 secs 2024-03-28T19:06:32.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3799.302797 secs 2024-03-28T19:06:32.468 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4445 sec 2024-03-28T19:06:32.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:32 smithi045 ceph-mon[31773]: pgmap v2468: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:32.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3090.728024 secs 2024-03-28T19:06:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:32 smithi071 ceph-mon[36597]: pgmap v2468: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3090.728024 secs 2024-03-28T19:06:33.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:32 smithi151 ceph-mon[36481]: pgmap v2468: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:33.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3090.728024 secs 2024-03-28T19:06:33.468 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4446 sec 2024-03-28T19:06:34.469 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4447 sec 2024-03-28T19:06:34.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:34 smithi045 ceph-mon[31773]: pgmap v2469: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:35.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:34 smithi071 ceph-mon[36597]: pgmap v2469: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:35.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:34 smithi151 ceph-mon[36481]: pgmap v2469: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:35.468 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4448 sec 2024-03-28T19:06:36.469 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4449 sec 2024-03-28T19:06:36.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:36 smithi045 ceph-mon[31773]: pgmap v2470: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:36.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3804.302855 secs 2024-03-28T19:06:37.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:36 smithi071 ceph-mon[36597]: pgmap v2470: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:37.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3804.302855 secs 2024-03-28T19:06:37.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:36 smithi151 ceph-mon[36481]: pgmap v2470: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:37.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3804.302855 secs 2024-03-28T19:06:37.469 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4450 sec 2024-03-28T19:06:37.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3095.728107 secs 2024-03-28T19:06:38.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3095.728107 secs 2024-03-28T19:06:38.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3095.728107 secs 2024-03-28T19:06:38.469 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4451 sec 2024-03-28T19:06:38.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:38 smithi045 ceph-mon[31773]: pgmap v2471: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:39.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:38 smithi071 ceph-mon[36597]: pgmap v2471: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:39.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:38 smithi151 ceph-mon[36481]: pgmap v2471: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:39.469 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4452 sec 2024-03-28T19:06:40.470 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4453 sec 2024-03-28T19:06:40.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:40 smithi045 ceph-mon[31773]: pgmap v2472: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:41.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:40 smithi071 ceph-mon[36597]: pgmap v2472: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:41.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:40 smithi151 ceph-mon[36481]: pgmap v2472: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:41.470 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4454 sec 2024-03-28T19:06:41.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3809.302916 secs 2024-03-28T19:06:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3809.302916 secs 2024-03-28T19:06:42.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3809.302916 secs 2024-03-28T19:06:42.469 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4455 sec 2024-03-28T19:06:42.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:42 smithi045 ceph-mon[31773]: pgmap v2473: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:42.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3100.728195 secs 2024-03-28T19:06:42.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:42 smithi151 ceph-mon[36481]: pgmap v2473: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:42.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3100.728195 secs 2024-03-28T19:06:43.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:42 smithi071 ceph-mon[36597]: pgmap v2473: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:43.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3100.728195 secs 2024-03-28T19:06:43.470 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4456 sec 2024-03-28T19:06:44.470 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4457 sec 2024-03-28T19:06:44.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:44 smithi045 ceph-mon[31773]: pgmap v2474: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:45.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:44 smithi071 ceph-mon[36597]: pgmap v2474: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:45.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:44 smithi151 ceph-mon[36481]: pgmap v2474: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:45.470 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4458 sec 2024-03-28T19:06:46.470 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4459 sec 2024-03-28T19:06:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:46 smithi045 ceph-mon[31773]: pgmap v2475: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3814.302977 secs 2024-03-28T19:06:47.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:46 smithi071 ceph-mon[36597]: pgmap v2475: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3814.302977 secs 2024-03-28T19:06:47.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:46 smithi151 ceph-mon[36481]: pgmap v2475: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:47.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3814.302977 secs 2024-03-28T19:06:47.470 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4460 sec 2024-03-28T19:06:47.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3105.728262 secs 2024-03-28T19:06:47.960 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3105.728262 secs 2024-03-28T19:06:48.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3105.728262 secs 2024-03-28T19:06:48.470 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4461 sec 2024-03-28T19:06:48.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:48 smithi045 ceph-mon[31773]: pgmap v2476: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:49.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:48 smithi071 ceph-mon[36597]: pgmap v2476: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:49.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:48 smithi151 ceph-mon[36481]: pgmap v2476: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:49.470 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4462 sec 2024-03-28T19:06:50.471 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4463 sec 2024-03-28T19:06:50.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:50 smithi045 ceph-mon[31773]: pgmap v2477: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:51.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:50 smithi071 ceph-mon[36597]: pgmap v2477: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:51.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:50 smithi151 ceph-mon[36481]: pgmap v2477: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:51.471 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4464 sec 2024-03-28T19:06:51.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3819.303038 secs 2024-03-28T19:06:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3819.303038 secs 2024-03-28T19:06:52.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3819.303038 secs 2024-03-28T19:06:52.471 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4465 sec 2024-03-28T19:06:52.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:52 smithi045 ceph-mon[31773]: pgmap v2478: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:52.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3110.728323 secs 2024-03-28T19:06:53.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:52 smithi071 ceph-mon[36597]: pgmap v2478: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:53.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3110.728323 secs 2024-03-28T19:06:53.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:52 smithi151 ceph-mon[36481]: pgmap v2478: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:53.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3110.728323 secs 2024-03-28T19:06:53.471 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4466 sec 2024-03-28T19:06:54.471 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4467 sec 2024-03-28T19:06:54.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:54 smithi045 ceph-mon[31773]: pgmap v2479: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:55.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:54 smithi071 ceph-mon[36597]: pgmap v2479: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:55.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:54 smithi151 ceph-mon[36481]: pgmap v2479: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:55.471 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4468 sec 2024-03-28T19:06:56.471 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4469 sec 2024-03-28T19:06:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:56 smithi071 ceph-mon[36597]: pgmap v2480: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3824.303100 secs 2024-03-28T19:06:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:56 smithi151 ceph-mon[36481]: pgmap v2480: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3824.303100 secs 2024-03-28T19:06:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:56 smithi045 ceph-mon[31773]: pgmap v2480: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:06:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3824.303100 secs 2024-03-28T19:06:57.471 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4470 sec 2024-03-28T19:06:58.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3115.728386 secs 2024-03-28T19:06:58.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3115.728386 secs 2024-03-28T19:06:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3115.728386 secs 2024-03-28T19:06:58.471 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4471 sec 2024-03-28T19:06:59.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:06:58 smithi071 ceph-mon[36597]: pgmap v2481: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:59.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:06:58 smithi151 ceph-mon[36481]: pgmap v2481: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:59.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:06:58 smithi045 ceph-mon[31773]: pgmap v2481: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:06:59.473 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4472 sec 2024-03-28T19:07:00.472 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4473 sec 2024-03-28T19:07:01.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:00 smithi071 ceph-mon[36597]: pgmap v2482: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:01.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:00 smithi151 ceph-mon[36481]: pgmap v2482: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:01.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:00 smithi045 ceph-mon[31773]: pgmap v2482: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:01.472 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4474 sec 2024-03-28T19:07:02.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3829.303172 secs 2024-03-28T19:07:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3829.303172 secs 2024-03-28T19:07:02.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3829.303172 secs 2024-03-28T19:07:02.472 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4475 sec 2024-03-28T19:07:03.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:02 smithi071 ceph-mon[36597]: pgmap v2483: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:03.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3120.728447 secs 2024-03-28T19:07:03.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:02 smithi151 ceph-mon[36481]: pgmap v2483: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:03.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3120.728447 secs 2024-03-28T19:07:03.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:02 smithi045 ceph-mon[31773]: pgmap v2483: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:03.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3120.728447 secs 2024-03-28T19:07:03.472 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4476 sec 2024-03-28T19:07:04.472 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4477 sec 2024-03-28T19:07:05.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:04 smithi071 ceph-mon[36597]: pgmap v2484: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:05.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:04 smithi151 ceph-mon[36481]: pgmap v2484: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:05.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:04 smithi045 ceph-mon[31773]: pgmap v2484: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:05.473 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4478 sec 2024-03-28T19:07:06.473 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4479 sec 2024-03-28T19:07:07.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:06 smithi071 ceph-mon[36597]: pgmap v2485: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T19:07:07.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3834.303232 secs 2024-03-28T19:07:07.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:06 smithi151 ceph-mon[36481]: pgmap v2485: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T19:07:07.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3834.303232 secs 2024-03-28T19:07:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:06 smithi045 ceph-mon[31773]: pgmap v2485: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T19:07:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3834.303232 secs 2024-03-28T19:07:07.473 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4480 sec 2024-03-28T19:07:08.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3125.728533 secs 2024-03-28T19:07:08.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3125.728533 secs 2024-03-28T19:07:08.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3125.728533 secs 2024-03-28T19:07:08.473 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4481 sec 2024-03-28T19:07:09.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:08 smithi071 ceph-mon[36597]: pgmap v2486: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T19:07:09.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:08 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:07:09.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:08 smithi151 ceph-mon[36481]: pgmap v2486: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T19:07:09.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:08 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:07:09.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:08 smithi045 ceph-mon[31773]: pgmap v2486: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T19:07:09.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:08 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:07:09.473 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4482 sec 2024-03-28T19:07:10.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:09 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:07:10.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:09 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:07:10.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:09 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:07:10.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:09 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:07:10.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:09 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:07:10.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:09 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:07:10.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:09 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:07:10.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:09 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:07:10.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:09 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:07:10.473 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4483 sec 2024-03-28T19:07:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:10 smithi071 ceph-mon[36597]: pgmap v2487: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 175 B/s wr, 3 op/s 2024-03-28T19:07:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:10 smithi071 ceph-mon[36597]: pgmap v2488: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 212 B/s wr, 4 op/s 2024-03-28T19:07:11.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:10 smithi151 ceph-mon[36481]: pgmap v2487: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 175 B/s wr, 3 op/s 2024-03-28T19:07:11.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:10 smithi151 ceph-mon[36481]: pgmap v2488: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 212 B/s wr, 4 op/s 2024-03-28T19:07:11.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:10 smithi045 ceph-mon[31773]: pgmap v2487: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 175 B/s wr, 3 op/s 2024-03-28T19:07:11.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:10 smithi045 ceph-mon[31773]: pgmap v2488: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 212 B/s wr, 4 op/s 2024-03-28T19:07:11.473 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4484 sec 2024-03-28T19:07:12.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3839.303321 secs 2024-03-28T19:07:12.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:11 smithi071 ceph-mon[36597]: pgmap v2489: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 212 B/s wr, 4 op/s 2024-03-28T19:07:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3839.303321 secs 2024-03-28T19:07:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:11 smithi151 ceph-mon[36481]: pgmap v2489: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 212 B/s wr, 4 op/s 2024-03-28T19:07:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3839.303321 secs 2024-03-28T19:07:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:11 smithi045 ceph-mon[31773]: pgmap v2489: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 212 B/s wr, 4 op/s 2024-03-28T19:07:12.473 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4485 sec 2024-03-28T19:07:13.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3130.728620 secs 2024-03-28T19:07:13.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3130.728620 secs 2024-03-28T19:07:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3130.728620 secs 2024-03-28T19:07:13.474 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4486 sec 2024-03-28T19:07:14.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:13 smithi071 ceph-mon[36597]: pgmap v2490: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 212 B/s wr, 4 op/s 2024-03-28T19:07:14.067 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:13 smithi045 ceph-mon[31773]: pgmap v2490: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 212 B/s wr, 4 op/s 2024-03-28T19:07:14.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:13 smithi151 ceph-mon[36481]: pgmap v2490: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 212 B/s wr, 4 op/s 2024-03-28T19:07:14.474 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4487 sec 2024-03-28T19:07:15.474 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4488 sec 2024-03-28T19:07:16.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:16 smithi045 ceph-mon[31773]: pgmap v2491: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:07:16.474 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4489 sec 2024-03-28T19:07:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:16 smithi071 ceph-mon[36597]: pgmap v2491: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:07:16.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:16 smithi151 ceph-mon[36481]: pgmap v2491: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:07:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:17 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3844.303381 secs 2024-03-28T19:07:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3135.728704 secs 2024-03-28T19:07:17.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:17 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3844.303381 secs 2024-03-28T19:07:17.461 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3135.728704 secs 2024-03-28T19:07:17.474 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4490 sec 2024-03-28T19:07:17.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:17 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3844.303381 secs 2024-03-28T19:07:17.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3135.728704 secs 2024-03-28T19:07:18.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:18 smithi045 ceph-mon[31773]: pgmap v2492: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:07:18.474 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4491 sec 2024-03-28T19:07:18.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:18 smithi071 ceph-mon[36597]: pgmap v2492: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:07:18.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:18 smithi151 ceph-mon[36481]: pgmap v2492: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:07:19.475 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4492 sec 2024-03-28T19:07:20.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:20 smithi045 ceph-mon[31773]: pgmap v2493: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:07:20.475 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4493 sec 2024-03-28T19:07:20.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:20 smithi071 ceph-mon[36597]: pgmap v2493: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:07:20.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:20 smithi151 ceph-mon[36481]: pgmap v2493: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:07:21.475 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4494 sec 2024-03-28T19:07:22.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:22 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3849.303474 secs 2024-03-28T19:07:22.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:22 smithi045 ceph-mon[31773]: pgmap v2494: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:22.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3140.728766 secs 2024-03-28T19:07:22.475 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4495 sec 2024-03-28T19:07:22.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:22 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3849.303474 secs 2024-03-28T19:07:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:22 smithi071 ceph-mon[36597]: pgmap v2494: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:22.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3140.728766 secs 2024-03-28T19:07:22.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:22 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3849.303474 secs 2024-03-28T19:07:22.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:22 smithi151 ceph-mon[36481]: pgmap v2494: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3140.728766 secs 2024-03-28T19:07:23.475 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4496 sec 2024-03-28T19:07:24.475 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4497 sec 2024-03-28T19:07:24.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:24 smithi071 ceph-mon[36597]: pgmap v2495: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:24.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:24 smithi151 ceph-mon[36481]: pgmap v2495: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:24.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:24 smithi045 ceph-mon[31773]: pgmap v2495: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:25.475 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4498 sec 2024-03-28T19:07:26.476 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4499 sec 2024-03-28T19:07:26.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:26 smithi071 ceph-mon[36597]: pgmap v2496: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:26.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:26 smithi151 ceph-mon[36481]: pgmap v2496: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:26.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:26 smithi045 ceph-mon[31773]: pgmap v2496: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:27.476 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4500 sec 2024-03-28T19:07:27.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:27 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3854.303489 secs 2024-03-28T19:07:27.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3145.728850 secs 2024-03-28T19:07:27.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:27 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3854.303489 secs 2024-03-28T19:07:27.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3145.728850 secs 2024-03-28T19:07:27.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:27 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3854.303489 secs 2024-03-28T19:07:27.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3145.728850 secs 2024-03-28T19:07:28.476 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4501 sec 2024-03-28T19:07:28.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:28 smithi071 ceph-mon[36597]: pgmap v2497: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:28.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:28 smithi151 ceph-mon[36481]: pgmap v2497: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:28.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:28 smithi045 ceph-mon[31773]: pgmap v2497: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:29.476 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4502 sec 2024-03-28T19:07:30.476 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4503 sec 2024-03-28T19:07:30.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:30 smithi071 ceph-mon[36597]: pgmap v2498: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:30.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:30 smithi151 ceph-mon[36481]: pgmap v2498: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:30.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:30 smithi045 ceph-mon[31773]: pgmap v2498: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:31.476 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4504 sec 2024-03-28T19:07:32.476 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4505 sec 2024-03-28T19:07:32.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:32 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3859.303578 secs 2024-03-28T19:07:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:32 smithi071 ceph-mon[36597]: pgmap v2499: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:32.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3150.728912 secs 2024-03-28T19:07:32.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:32 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3859.303578 secs 2024-03-28T19:07:32.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:32 smithi151 ceph-mon[36481]: pgmap v2499: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:32.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3150.728912 secs 2024-03-28T19:07:32.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:32 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3859.303578 secs 2024-03-28T19:07:32.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:32 smithi045 ceph-mon[31773]: pgmap v2499: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:32.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3150.728912 secs 2024-03-28T19:07:32.665 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:07:32 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:07:32.345+0000 7f52f3200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:07:32.665 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:07:32 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:07:32.345+0000 7f52f3200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:07:32.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:07:32 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:07:32.345+0000 7f52f3200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:07:32.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:07:32 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:07:32.345+0000 7f52f3200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:07:32.666 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:07:32 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:07:32.345+0000 7f52f3200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:07:33.476 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4506 sec 2024-03-28T19:07:34.477 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4507 sec 2024-03-28T19:07:34.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:34 smithi071 ceph-mon[36597]: pgmap v2500: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:34.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:34 smithi151 ceph-mon[36481]: pgmap v2500: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:34.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:34 smithi045 ceph-mon[31773]: pgmap v2500: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:35.477 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4508 sec 2024-03-28T19:07:36.477 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4509 sec 2024-03-28T19:07:36.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:36 smithi071 ceph-mon[36597]: pgmap v2501: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:36.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:36 smithi151 ceph-mon[36481]: pgmap v2501: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:36.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:36 smithi045 ceph-mon[31773]: pgmap v2501: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:37 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3864.303670 secs 2024-03-28T19:07:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3155.728996 secs 2024-03-28T19:07:37.477 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4510 sec 2024-03-28T19:07:37.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:37 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3864.303670 secs 2024-03-28T19:07:37.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3155.728996 secs 2024-03-28T19:07:37.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:37 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3864.303670 secs 2024-03-28T19:07:37.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3155.728996 secs 2024-03-28T19:07:38.477 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4511 sec 2024-03-28T19:07:38.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:38 smithi071 ceph-mon[36597]: pgmap v2502: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:38.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:38 smithi151 ceph-mon[36481]: pgmap v2502: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:38.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:38 smithi045 ceph-mon[31773]: pgmap v2502: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:39.477 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4512 sec 2024-03-28T19:07:40.478 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4513 sec 2024-03-28T19:07:40.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:40 smithi071 ceph-mon[36597]: pgmap v2503: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:40.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:40 smithi151 ceph-mon[36481]: pgmap v2503: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:40.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:40 smithi045 ceph-mon[31773]: pgmap v2503: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:41.478 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4514 sec 2024-03-28T19:07:42.478 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4515 sec 2024-03-28T19:07:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:42 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3869.303729 secs 2024-03-28T19:07:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:42 smithi071 ceph-mon[36597]: pgmap v2504: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:42.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3160.729080 secs 2024-03-28T19:07:42.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:42 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3869.303729 secs 2024-03-28T19:07:42.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:42 smithi151 ceph-mon[36481]: pgmap v2504: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:42.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3160.729080 secs 2024-03-28T19:07:42.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:42 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3869.303729 secs 2024-03-28T19:07:42.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:42 smithi045 ceph-mon[31773]: pgmap v2504: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:42.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3160.729080 secs 2024-03-28T19:07:43.478 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4516 sec 2024-03-28T19:07:44.478 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4517 sec 2024-03-28T19:07:44.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:44 smithi071 ceph-mon[36597]: pgmap v2505: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:44.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:44 smithi151 ceph-mon[36481]: pgmap v2505: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:44.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:44 smithi045 ceph-mon[31773]: pgmap v2505: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:45.478 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4518 sec 2024-03-28T19:07:46.478 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4519 sec 2024-03-28T19:07:46.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:46 smithi071 ceph-mon[36597]: pgmap v2506: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:46.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:46 smithi151 ceph-mon[36481]: pgmap v2506: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:46.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:46 smithi045 ceph-mon[31773]: pgmap v2506: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:47.478 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4520 sec 2024-03-28T19:07:47.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:47 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3874.303790 secs 2024-03-28T19:07:47.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3165.729170 secs 2024-03-28T19:07:47.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:47 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3874.303790 secs 2024-03-28T19:07:47.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3165.729170 secs 2024-03-28T19:07:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:47 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3874.303790 secs 2024-03-28T19:07:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3165.729170 secs 2024-03-28T19:07:48.478 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4521 sec 2024-03-28T19:07:48.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:48 smithi071 ceph-mon[36597]: pgmap v2507: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:48.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:48 smithi151 ceph-mon[36481]: pgmap v2507: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:48.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:48 smithi045 ceph-mon[31773]: pgmap v2507: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:49.479 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4522 sec 2024-03-28T19:07:50.479 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4523 sec 2024-03-28T19:07:50.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:50 smithi071 ceph-mon[36597]: pgmap v2508: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:50.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:50 smithi151 ceph-mon[36481]: pgmap v2508: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:50.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:50 smithi045 ceph-mon[31773]: pgmap v2508: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:51.479 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4524 sec 2024-03-28T19:07:51.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3879.303851 secs 2024-03-28T19:07:51.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3879.303851 secs 2024-03-28T19:07:51.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3879.303851 secs 2024-03-28T19:07:52.479 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4525 sec 2024-03-28T19:07:52.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:52 smithi071 ceph-mon[36597]: pgmap v2509: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:52.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3170.729232 secs 2024-03-28T19:07:52.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:52 smithi151 ceph-mon[36481]: pgmap v2509: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:52.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3170.729232 secs 2024-03-28T19:07:52.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:52 smithi045 ceph-mon[31773]: pgmap v2509: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:52.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3170.729232 secs 2024-03-28T19:07:53.479 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4526 sec 2024-03-28T19:07:54.479 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4527 sec 2024-03-28T19:07:54.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:54 smithi071 ceph-mon[36597]: pgmap v2510: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:54.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:54 smithi151 ceph-mon[36481]: pgmap v2510: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:54.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:54 smithi045 ceph-mon[31773]: pgmap v2510: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:55.479 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4528 sec 2024-03-28T19:07:56.480 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4529 sec 2024-03-28T19:07:56.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:56 smithi071 ceph-mon[36597]: pgmap v2511: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:56.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3884.303913 secs 2024-03-28T19:07:56.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:56 smithi151 ceph-mon[36481]: pgmap v2511: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:56.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3884.303913 secs 2024-03-28T19:07:56.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:56 smithi045 ceph-mon[31773]: pgmap v2511: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:07:56.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3884.303913 secs 2024-03-28T19:07:57.480 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4530 sec 2024-03-28T19:07:57.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3175.729294 secs 2024-03-28T19:07:57.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3175.729294 secs 2024-03-28T19:07:57.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3175.729294 secs 2024-03-28T19:07:58.480 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4531 sec 2024-03-28T19:07:58.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:07:58 smithi071 ceph-mon[36597]: pgmap v2512: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:58.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:07:58 smithi151 ceph-mon[36481]: pgmap v2512: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:58.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:07:58 smithi045 ceph-mon[31773]: pgmap v2512: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:07:59.480 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4532 sec 2024-03-28T19:08:00.480 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4533 sec 2024-03-28T19:08:00.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:00 smithi071 ceph-mon[36597]: pgmap v2513: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:00.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:00 smithi151 ceph-mon[36481]: pgmap v2513: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:00.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:00 smithi045 ceph-mon[31773]: pgmap v2513: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:01.480 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4534 sec 2024-03-28T19:08:01.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3889.303976 secs 2024-03-28T19:08:01.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3889.303976 secs 2024-03-28T19:08:01.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3889.303976 secs 2024-03-28T19:08:02.481 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4535 sec 2024-03-28T19:08:02.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:02 smithi071 ceph-mon[36597]: pgmap v2514: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:02 smithi071 ceph-mon[36597]: 2 slow requests, 1 included below; oldest blocked for > 3180.729359 secs 2024-03-28T19:08:02.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:02 smithi071 ceph-mon[36597]: slow request 2880.541715 seconds old, received at 2024-03-28T18:20:01.134771+0000: client_request(client.16398:1 getattr pAsLsXsFs #0x1 2024-03-28T18:20:01.130880+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T19:08:02.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:02 smithi151 ceph-mon[36481]: pgmap v2514: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:02.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:02 smithi151 ceph-mon[36481]: 2 slow requests, 1 included below; oldest blocked for > 3180.729359 secs 2024-03-28T19:08:02.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:02 smithi151 ceph-mon[36481]: slow request 2880.541715 seconds old, received at 2024-03-28T18:20:01.134771+0000: client_request(client.16398:1 getattr pAsLsXsFs #0x1 2024-03-28T18:20:01.130880+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T19:08:02.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:02 smithi045 ceph-mon[31773]: pgmap v2514: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:02.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:02 smithi045 ceph-mon[31773]: 2 slow requests, 1 included below; oldest blocked for > 3180.729359 secs 2024-03-28T19:08:02.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:02 smithi045 ceph-mon[31773]: slow request 2880.541715 seconds old, received at 2024-03-28T18:20:01.134771+0000: client_request(client.16398:1 getattr pAsLsXsFs #0x1 2024-03-28T18:20:01.130880+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T19:08:03.481 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4536 sec 2024-03-28T19:08:04.481 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4537 sec 2024-03-28T19:08:04.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:04 smithi071 ceph-mon[36597]: pgmap v2515: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:04.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:04 smithi151 ceph-mon[36481]: pgmap v2515: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:04.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:04 smithi045 ceph-mon[31773]: pgmap v2515: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:05.481 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4538 sec 2024-03-28T19:08:06.481 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4539 sec 2024-03-28T19:08:06.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:06 smithi151 ceph-mon[36481]: pgmap v2516: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:06.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3894.304065 secs 2024-03-28T19:08:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:06 smithi045 ceph-mon[31773]: pgmap v2516: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3894.304065 secs 2024-03-28T19:08:06.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:06 smithi071 ceph-mon[36597]: pgmap v2516: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:06.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3894.304065 secs 2024-03-28T19:08:07.482 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4540 sec 2024-03-28T19:08:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3185.729446 secs 2024-03-28T19:08:07.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3185.729446 secs 2024-03-28T19:08:07.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3185.729446 secs 2024-03-28T19:08:08.481 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4541 sec 2024-03-28T19:08:08.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:08 smithi045 ceph-mon[31773]: pgmap v2517: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:08.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:08 smithi071 ceph-mon[36597]: pgmap v2517: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:08.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:08 smithi151 ceph-mon[36481]: pgmap v2517: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:09.482 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4542 sec 2024-03-28T19:08:10.483 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4543 sec 2024-03-28T19:08:10.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:10 smithi045 ceph-mon[31773]: pgmap v2518: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:10.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:10 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:08:10.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:10 smithi071 ceph-mon[36597]: pgmap v2518: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:10.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:10 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:08:10.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:10 smithi151 ceph-mon[36481]: pgmap v2518: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:10.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:10 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:08:11.482 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4544 sec 2024-03-28T19:08:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:11 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:08:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:11 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:08:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:11 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:08:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3899.304128 secs 2024-03-28T19:08:11.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:11 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:08:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:11 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:08:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:11 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:08:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3899.304128 secs 2024-03-28T19:08:11.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:11 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:08:11.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:11 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:08:11.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:11 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:08:11.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3899.304128 secs 2024-03-28T19:08:12.482 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4545 sec 2024-03-28T19:08:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:12 smithi045 ceph-mon[31773]: pgmap v2519: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:08:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:12 smithi045 ceph-mon[31773]: pgmap v2520: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:08:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3190.729505 secs 2024-03-28T19:08:12.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:12 smithi151 ceph-mon[36481]: pgmap v2519: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:08:12.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:12 smithi151 ceph-mon[36481]: pgmap v2520: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:08:12.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3190.729505 secs 2024-03-28T19:08:12.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:12 smithi071 ceph-mon[36597]: pgmap v2519: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:08:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:12 smithi071 ceph-mon[36597]: pgmap v2520: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:08:12.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3190.729505 secs 2024-03-28T19:08:13.482 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4546 sec 2024-03-28T19:08:14.482 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4547 sec 2024-03-28T19:08:14.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:14 smithi045 ceph-mon[31773]: pgmap v2521: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:08:14.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:14 smithi071 ceph-mon[36597]: pgmap v2521: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:08:14.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:14 smithi151 ceph-mon[36481]: pgmap v2521: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:08:15.482 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4548 sec 2024-03-28T19:08:16.483 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4549 sec 2024-03-28T19:08:16.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:16 smithi045 ceph-mon[31773]: pgmap v2522: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:08:16.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3904.304216 secs 2024-03-28T19:08:16.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:16 smithi071 ceph-mon[36597]: pgmap v2522: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:08:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3904.304216 secs 2024-03-28T19:08:16.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:16 smithi151 ceph-mon[36481]: pgmap v2522: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:08:16.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3904.304216 secs 2024-03-28T19:08:17.483 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4550 sec 2024-03-28T19:08:17.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3195.729545 secs 2024-03-28T19:08:17.710 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3195.729545 secs 2024-03-28T19:08:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3195.729545 secs 2024-03-28T19:08:18.483 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4551 sec 2024-03-28T19:08:18.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:18 smithi045 ceph-mon[31773]: pgmap v2523: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:08:18.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:18 smithi071 ceph-mon[36597]: pgmap v2523: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:08:18.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:18 smithi151 ceph-mon[36481]: pgmap v2523: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:08:19.483 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4552 sec 2024-03-28T19:08:20.483 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4553 sec 2024-03-28T19:08:20.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:20 smithi045 ceph-mon[31773]: pgmap v2524: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:08:20.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:20 smithi071 ceph-mon[36597]: pgmap v2524: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:08:20.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:20 smithi151 ceph-mon[36481]: pgmap v2524: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:08:21.483 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4554 sec 2024-03-28T19:08:21.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3909.304331 secs 2024-03-28T19:08:21.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3909.304331 secs 2024-03-28T19:08:21.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3909.304331 secs 2024-03-28T19:08:22.483 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4555 sec 2024-03-28T19:08:22.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:22 smithi045 ceph-mon[31773]: pgmap v2525: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:08:22.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3200.729635 secs 2024-03-28T19:08:22.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:22 smithi071 ceph-mon[36597]: pgmap v2525: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:08:22.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3200.729635 secs 2024-03-28T19:08:22.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:22 smithi151 ceph-mon[36481]: pgmap v2525: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:08:22.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3200.729635 secs 2024-03-28T19:08:23.484 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4556 sec 2024-03-28T19:08:24.484 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4557 sec 2024-03-28T19:08:24.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:24 smithi045 ceph-mon[31773]: pgmap v2526: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:24.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:24 smithi071 ceph-mon[36597]: pgmap v2526: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:24.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:24 smithi151 ceph-mon[36481]: pgmap v2526: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:25.484 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4558 sec 2024-03-28T19:08:26.484 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4559 sec 2024-03-28T19:08:26.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:26 smithi045 ceph-mon[31773]: pgmap v2527: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:26.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3914.304393 secs 2024-03-28T19:08:26.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:26 smithi071 ceph-mon[36597]: pgmap v2527: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:26.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3914.304393 secs 2024-03-28T19:08:26.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:26 smithi151 ceph-mon[36481]: pgmap v2527: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:26.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3914.304393 secs 2024-03-28T19:08:27.484 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4560 sec 2024-03-28T19:08:27.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3205.729697 secs 2024-03-28T19:08:27.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3205.729697 secs 2024-03-28T19:08:27.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3205.729697 secs 2024-03-28T19:08:28.484 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4561 sec 2024-03-28T19:08:28.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:28 smithi045 ceph-mon[31773]: pgmap v2528: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:28.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:28 smithi071 ceph-mon[36597]: pgmap v2528: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:28.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:28 smithi151 ceph-mon[36481]: pgmap v2528: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:29.485 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4562 sec 2024-03-28T19:08:30.485 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4563 sec 2024-03-28T19:08:30.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:30 smithi045 ceph-mon[31773]: pgmap v2529: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:30.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:30 smithi071 ceph-mon[36597]: pgmap v2529: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:30.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:30 smithi151 ceph-mon[36481]: pgmap v2529: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:31.485 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4564 sec 2024-03-28T19:08:31.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3919.304477 secs 2024-03-28T19:08:31.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3919.304477 secs 2024-03-28T19:08:31.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3919.304477 secs 2024-03-28T19:08:32.485 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4565 sec 2024-03-28T19:08:32.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:32 smithi071 ceph-mon[36597]: pgmap v2530: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:32.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3210.729760 secs 2024-03-28T19:08:32.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:32 smithi151 ceph-mon[36481]: pgmap v2530: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:32.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3210.729760 secs 2024-03-28T19:08:32.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:32 smithi045 ceph-mon[31773]: pgmap v2530: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:32.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3210.729760 secs 2024-03-28T19:08:33.485 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4566 sec 2024-03-28T19:08:34.485 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4567 sec 2024-03-28T19:08:34.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:34 smithi071 ceph-mon[36597]: pgmap v2531: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:34.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:34 smithi151 ceph-mon[36481]: pgmap v2531: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:34.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:34 smithi045 ceph-mon[31773]: pgmap v2531: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:35.485 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4568 sec 2024-03-28T19:08:36.485 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4569 sec 2024-03-28T19:08:36.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:36 smithi071 ceph-mon[36597]: pgmap v2532: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:36.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3924.304539 secs 2024-03-28T19:08:36.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:36 smithi151 ceph-mon[36481]: pgmap v2532: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:36.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3924.304539 secs 2024-03-28T19:08:36.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:36 smithi045 ceph-mon[31773]: pgmap v2532: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:36.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3924.304539 secs 2024-03-28T19:08:37.486 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4570 sec 2024-03-28T19:08:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3215.729824 secs 2024-03-28T19:08:37.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3215.729824 secs 2024-03-28T19:08:37.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3215.729824 secs 2024-03-28T19:08:38.486 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4571 sec 2024-03-28T19:08:38.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:38 smithi071 ceph-mon[36597]: pgmap v2533: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:38.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:38 smithi151 ceph-mon[36481]: pgmap v2533: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:38.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:38 smithi045 ceph-mon[31773]: pgmap v2533: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:39.486 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4572 sec 2024-03-28T19:08:40.486 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4573 sec 2024-03-28T19:08:40.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:40 smithi071 ceph-mon[36597]: pgmap v2534: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:40.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:40 smithi151 ceph-mon[36481]: pgmap v2534: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:40.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:40 smithi045 ceph-mon[31773]: pgmap v2534: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:41.486 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4574 sec 2024-03-28T19:08:41.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3929.304600 secs 2024-03-28T19:08:41.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3929.304600 secs 2024-03-28T19:08:41.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3929.304600 secs 2024-03-28T19:08:42.486 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4575 sec 2024-03-28T19:08:42.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:42 smithi151 ceph-mon[36481]: pgmap v2535: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:42.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3220.729904 secs 2024-03-28T19:08:42.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:42 smithi071 ceph-mon[36597]: pgmap v2535: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3220.729904 secs 2024-03-28T19:08:42.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:42 smithi045 ceph-mon[31773]: pgmap v2535: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:42.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3220.729904 secs 2024-03-28T19:08:43.486 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4576 sec 2024-03-28T19:08:44.487 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4577 sec 2024-03-28T19:08:44.718 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:44 smithi045 ceph-mon[31773]: pgmap v2536: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:44.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:44 smithi071 ceph-mon[36597]: pgmap v2536: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:44.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:44 smithi151 ceph-mon[36481]: pgmap v2536: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:45.490 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4578 sec 2024-03-28T19:08:46.487 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4579 sec 2024-03-28T19:08:46.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:46 smithi071 ceph-mon[36597]: pgmap v2537: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:46.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3934.304664 secs 2024-03-28T19:08:46.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:46 smithi151 ceph-mon[36481]: pgmap v2537: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:46.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3934.304664 secs 2024-03-28T19:08:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:46 smithi045 ceph-mon[31773]: pgmap v2537: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3934.304664 secs 2024-03-28T19:08:47.487 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4580 sec 2024-03-28T19:08:47.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3225.729989 secs 2024-03-28T19:08:47.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3225.729989 secs 2024-03-28T19:08:47.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3225.729989 secs 2024-03-28T19:08:48.487 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4581 sec 2024-03-28T19:08:48.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:48 smithi071 ceph-mon[36597]: pgmap v2538: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:48.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:48 smithi151 ceph-mon[36481]: pgmap v2538: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:48.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:48 smithi045 ceph-mon[31773]: pgmap v2538: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:49.487 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4582 sec 2024-03-28T19:08:50.487 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4583 sec 2024-03-28T19:08:50.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:50 smithi071 ceph-mon[36597]: pgmap v2539: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:50.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:50 smithi151 ceph-mon[36481]: pgmap v2539: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:50.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:50 smithi045 ceph-mon[31773]: pgmap v2539: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:51.487 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4584 sec 2024-03-28T19:08:51.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3939.304697 secs 2024-03-28T19:08:51.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3939.304697 secs 2024-03-28T19:08:51.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3939.304697 secs 2024-03-28T19:08:52.487 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4585 sec 2024-03-28T19:08:52.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:52 smithi071 ceph-mon[36597]: pgmap v2540: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3230.730073 secs 2024-03-28T19:08:52.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:52 smithi151 ceph-mon[36481]: pgmap v2540: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:52.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3230.730073 secs 2024-03-28T19:08:52.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:52 smithi045 ceph-mon[31773]: pgmap v2540: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:52.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3230.730073 secs 2024-03-28T19:08:53.488 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4586 sec 2024-03-28T19:08:54.488 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4587 sec 2024-03-28T19:08:54.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:54 smithi045 ceph-mon[31773]: pgmap v2541: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:54.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:54 smithi071 ceph-mon[36597]: pgmap v2541: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:54.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:54 smithi151 ceph-mon[36481]: pgmap v2541: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:55.488 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4588 sec 2024-03-28T19:08:56.488 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4589 sec 2024-03-28T19:08:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:56 smithi071 ceph-mon[36597]: pgmap v2542: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3944.304760 secs 2024-03-28T19:08:56.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:56 smithi151 ceph-mon[36481]: pgmap v2542: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:56.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3944.304760 secs 2024-03-28T19:08:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:56 smithi045 ceph-mon[31773]: pgmap v2542: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:08:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3944.304760 secs 2024-03-28T19:08:57.488 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4590 sec 2024-03-28T19:08:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3235.730136 secs 2024-03-28T19:08:57.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3235.730136 secs 2024-03-28T19:08:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3235.730136 secs 2024-03-28T19:08:58.488 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4591 sec 2024-03-28T19:08:58.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:08:58 smithi071 ceph-mon[36597]: pgmap v2543: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:58.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:08:58 smithi151 ceph-mon[36481]: pgmap v2543: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:58.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:08:58 smithi045 ceph-mon[31773]: pgmap v2543: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:08:59.489 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4592 sec 2024-03-28T19:09:00.489 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4593 sec 2024-03-28T19:09:00.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:00 smithi071 ceph-mon[36597]: pgmap v2544: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:00.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:00 smithi151 ceph-mon[36481]: pgmap v2544: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:00.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:00 smithi045 ceph-mon[31773]: pgmap v2544: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:01.489 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4594 sec 2024-03-28T19:09:01.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3949.304848 secs 2024-03-28T19:09:01.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3949.304848 secs 2024-03-28T19:09:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3949.304848 secs 2024-03-28T19:09:02.489 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4595 sec 2024-03-28T19:09:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:02 smithi071 ceph-mon[36597]: pgmap v2545: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3240.730198 secs 2024-03-28T19:09:02.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:02 smithi151 ceph-mon[36481]: pgmap v2545: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:02.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3240.730198 secs 2024-03-28T19:09:02.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:02 smithi045 ceph-mon[31773]: pgmap v2545: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:02.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3240.730198 secs 2024-03-28T19:09:03.490 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4596 sec 2024-03-28T19:09:04.490 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4597 sec 2024-03-28T19:09:04.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:04 smithi071 ceph-mon[36597]: pgmap v2546: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:04.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:04 smithi151 ceph-mon[36481]: pgmap v2546: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:04.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:04 smithi045 ceph-mon[31773]: pgmap v2546: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:05.490 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4598 sec 2024-03-28T19:09:06.490 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4599 sec 2024-03-28T19:09:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:06 smithi071 ceph-mon[36597]: pgmap v2547: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3954.304938 secs 2024-03-28T19:09:06.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:06 smithi151 ceph-mon[36481]: pgmap v2547: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:06.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3954.304938 secs 2024-03-28T19:09:06.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:06 smithi045 ceph-mon[31773]: pgmap v2547: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:06.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3954.304938 secs 2024-03-28T19:09:07.490 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4600 sec 2024-03-28T19:09:07.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3245.730260 secs 2024-03-28T19:09:07.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3245.730260 secs 2024-03-28T19:09:07.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3245.730260 secs 2024-03-28T19:09:08.490 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4601 sec 2024-03-28T19:09:08.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:08 smithi151 ceph-mon[36481]: pgmap v2548: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:08.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:08 smithi045 ceph-mon[31773]: pgmap v2548: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:09.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:08 smithi071 ceph-mon[36597]: pgmap v2548: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:09.491 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4602 sec 2024-03-28T19:09:10.490 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4603 sec 2024-03-28T19:09:10.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:10 smithi151 ceph-mon[36481]: pgmap v2549: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:10.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:10 smithi045 ceph-mon[31773]: pgmap v2549: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:11.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:10 smithi071 ceph-mon[36597]: pgmap v2549: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:11.490 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4604 sec 2024-03-28T19:09:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3959.305027 secs 2024-03-28T19:09:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:11 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:09:12.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3959.305027 secs 2024-03-28T19:09:12.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:11 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:09:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3959.305027 secs 2024-03-28T19:09:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:11 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:09:12.490 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4605 sec 2024-03-28T19:09:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:12 smithi045 ceph-mon[31773]: pgmap v2550: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3250.730352 secs 2024-03-28T19:09:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:12 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:09:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:12 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:09:12.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:12 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:09:12.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:12 smithi151 ceph-mon[36481]: pgmap v2550: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:12.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3250.730352 secs 2024-03-28T19:09:12.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:12 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:09:12.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:12 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:09:12.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:12 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:09:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:12 smithi071 ceph-mon[36597]: pgmap v2550: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3250.730352 secs 2024-03-28T19:09:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:12 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:09:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:12 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:09:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:12 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:09:13.490 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4606 sec 2024-03-28T19:09:13.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:13 smithi045 ceph-mon[31773]: pgmap v2551: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:13.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:13 smithi045 ceph-mon[31773]: pgmap v2552: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:14.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:13 smithi071 ceph-mon[36597]: pgmap v2551: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:14.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:13 smithi071 ceph-mon[36597]: pgmap v2552: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:14.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:13 smithi151 ceph-mon[36481]: pgmap v2551: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:14.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:13 smithi151 ceph-mon[36481]: pgmap v2552: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:14.491 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4607 sec 2024-03-28T19:09:15.491 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4608 sec 2024-03-28T19:09:15.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:15 smithi045 ceph-mon[31773]: pgmap v2553: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:16.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:15 smithi071 ceph-mon[36597]: pgmap v2553: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:16.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:15 smithi151 ceph-mon[36481]: pgmap v2553: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:16.490 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4609 sec 2024-03-28T19:09:16.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3964.305115 secs 2024-03-28T19:09:17.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3964.305115 secs 2024-03-28T19:09:17.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3964.305115 secs 2024-03-28T19:09:17.491 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4610 sec 2024-03-28T19:09:17.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:17 smithi045 ceph-mon[31773]: pgmap v2554: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:17.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3255.730432 secs 2024-03-28T19:09:17.960 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:17 smithi071 ceph-mon[36597]: pgmap v2554: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:17.960 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3255.730432 secs 2024-03-28T19:09:18.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:17 smithi151 ceph-mon[36481]: pgmap v2554: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:18.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3255.730432 secs 2024-03-28T19:09:18.491 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4611 sec 2024-03-28T19:09:19.491 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4612 sec 2024-03-28T19:09:19.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:19 smithi045 ceph-mon[31773]: pgmap v2555: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:20.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:19 smithi071 ceph-mon[36597]: pgmap v2555: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:20.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:19 smithi151 ceph-mon[36481]: pgmap v2555: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:20.491 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4613 sec 2024-03-28T19:09:21.492 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4614 sec 2024-03-28T19:09:21.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:21 smithi045 ceph-mon[31773]: pgmap v2556: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:21.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3969.305176 secs 2024-03-28T19:09:22.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:21 smithi071 ceph-mon[36597]: pgmap v2556: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3969.305176 secs 2024-03-28T19:09:22.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:21 smithi151 ceph-mon[36481]: pgmap v2556: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:09:22.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3969.305176 secs 2024-03-28T19:09:22.491 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4615 sec 2024-03-28T19:09:22.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3260.730540 secs 2024-03-28T19:09:23.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3260.730540 secs 2024-03-28T19:09:23.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3260.730540 secs 2024-03-28T19:09:23.492 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4616 sec 2024-03-28T19:09:23.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:23 smithi045 ceph-mon[31773]: pgmap v2557: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:09:24.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:23 smithi071 ceph-mon[36597]: pgmap v2557: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:09:24.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:23 smithi151 ceph-mon[36481]: pgmap v2557: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:09:24.492 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4617 sec 2024-03-28T19:09:25.492 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4618 sec 2024-03-28T19:09:25.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:25 smithi045 ceph-mon[31773]: pgmap v2558: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:26.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:25 smithi071 ceph-mon[36597]: pgmap v2558: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:26.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:25 smithi151 ceph-mon[36481]: pgmap v2558: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:26.492 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4619 sec 2024-03-28T19:09:26.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3974.305267 secs 2024-03-28T19:09:27.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3974.305267 secs 2024-03-28T19:09:27.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3974.305267 secs 2024-03-28T19:09:27.492 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4620 sec 2024-03-28T19:09:27.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:27 smithi045 ceph-mon[31773]: pgmap v2559: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:27.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3265.730639 secs 2024-03-28T19:09:28.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:27 smithi071 ceph-mon[36597]: pgmap v2559: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:28.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3265.730639 secs 2024-03-28T19:09:28.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:27 smithi151 ceph-mon[36481]: pgmap v2559: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:28.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3265.730639 secs 2024-03-28T19:09:28.492 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4621 sec 2024-03-28T19:09:29.492 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4622 sec 2024-03-28T19:09:29.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:29 smithi045 ceph-mon[31773]: pgmap v2560: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:30.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:29 smithi071 ceph-mon[36597]: pgmap v2560: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:30.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:29 smithi151 ceph-mon[36481]: pgmap v2560: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:30.492 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4623 sec 2024-03-28T19:09:31.492 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4624 sec 2024-03-28T19:09:31.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:31 smithi045 ceph-mon[31773]: pgmap v2561: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:31.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3979.305304 secs 2024-03-28T19:09:32.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:31 smithi071 ceph-mon[36597]: pgmap v2561: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3979.305304 secs 2024-03-28T19:09:32.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:31 smithi151 ceph-mon[36481]: pgmap v2561: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:32.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3979.305304 secs 2024-03-28T19:09:32.493 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4625 sec 2024-03-28T19:09:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3270.730749 secs 2024-03-28T19:09:33.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3270.730749 secs 2024-03-28T19:09:33.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3270.730749 secs 2024-03-28T19:09:33.493 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4626 sec 2024-03-28T19:09:34.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:33 smithi071 ceph-mon[36597]: pgmap v2562: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:34.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:33 smithi151 ceph-mon[36481]: pgmap v2562: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:34.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:33 smithi045 ceph-mon[31773]: pgmap v2562: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:34.493 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4627 sec 2024-03-28T19:09:35.493 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4628 sec 2024-03-28T19:09:36.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:35 smithi071 ceph-mon[36597]: pgmap v2563: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:36.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:35 smithi151 ceph-mon[36481]: pgmap v2563: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:36.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:35 smithi045 ceph-mon[31773]: pgmap v2563: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:36.493 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4629 sec 2024-03-28T19:09:37.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3984.305364 secs 2024-03-28T19:09:37.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3984.305364 secs 2024-03-28T19:09:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3984.305364 secs 2024-03-28T19:09:37.493 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4630 sec 2024-03-28T19:09:38.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:37 smithi071 ceph-mon[36597]: pgmap v2564: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:38.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3275.730820 secs 2024-03-28T19:09:38.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:37 smithi151 ceph-mon[36481]: pgmap v2564: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:38.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3275.730820 secs 2024-03-28T19:09:38.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:37 smithi045 ceph-mon[31773]: pgmap v2564: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:38.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3275.730820 secs 2024-03-28T19:09:38.494 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4631 sec 2024-03-28T19:09:39.494 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4632 sec 2024-03-28T19:09:40.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:39 smithi071 ceph-mon[36597]: pgmap v2565: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:40.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:39 smithi151 ceph-mon[36481]: pgmap v2565: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:40.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:39 smithi045 ceph-mon[31773]: pgmap v2565: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:40.494 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4633 sec 2024-03-28T19:09:41.494 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4634 sec 2024-03-28T19:09:42.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:41 smithi071 ceph-mon[36597]: pgmap v2566: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3989.305403 secs 2024-03-28T19:09:42.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:41 smithi151 ceph-mon[36481]: pgmap v2566: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:42.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3989.305403 secs 2024-03-28T19:09:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:41 smithi045 ceph-mon[31773]: pgmap v2566: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3989.305403 secs 2024-03-28T19:09:42.494 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4635 sec 2024-03-28T19:09:42.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3280.730930 secs 2024-03-28T19:09:43.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3280.730930 secs 2024-03-28T19:09:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3280.730930 secs 2024-03-28T19:09:43.494 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4636 sec 2024-03-28T19:09:44.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:43 smithi071 ceph-mon[36597]: pgmap v2567: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:44.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:43 smithi151 ceph-mon[36481]: pgmap v2567: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:44.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:43 smithi045 ceph-mon[31773]: pgmap v2567: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:44.495 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4637 sec 2024-03-28T19:09:45.495 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4638 sec 2024-03-28T19:09:45.967 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:45 smithi045 ceph-mon[31773]: pgmap v2568: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:46.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:45 smithi071 ceph-mon[36597]: pgmap v2568: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:46.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:45 smithi151 ceph-mon[36481]: pgmap v2568: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:46.495 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4639 sec 2024-03-28T19:09:47.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3994.305509 secs 2024-03-28T19:09:47.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3994.305509 secs 2024-03-28T19:09:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3994.305509 secs 2024-03-28T19:09:47.495 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4640 sec 2024-03-28T19:09:48.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:47 smithi071 ceph-mon[36597]: pgmap v2569: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:48.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3285.730945 secs 2024-03-28T19:09:48.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:47 smithi151 ceph-mon[36481]: pgmap v2569: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:48.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3285.730945 secs 2024-03-28T19:09:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:47 smithi045 ceph-mon[31773]: pgmap v2569: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3285.730945 secs 2024-03-28T19:09:48.495 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4641 sec 2024-03-28T19:09:49.495 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4642 sec 2024-03-28T19:09:50.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:49 smithi071 ceph-mon[36597]: pgmap v2570: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:50.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:49 smithi151 ceph-mon[36481]: pgmap v2570: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:50.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:49 smithi045 ceph-mon[31773]: pgmap v2570: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:50.495 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4643 sec 2024-03-28T19:09:51.495 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4644 sec 2024-03-28T19:09:52.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:51 smithi071 ceph-mon[36597]: pgmap v2571: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 3999.305570 secs 2024-03-28T19:09:52.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:51 smithi151 ceph-mon[36481]: pgmap v2571: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:52.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 3999.305570 secs 2024-03-28T19:09:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:51 smithi045 ceph-mon[31773]: pgmap v2571: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 3999.305570 secs 2024-03-28T19:09:52.495 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4645 sec 2024-03-28T19:09:53.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3290.731010 secs 2024-03-28T19:09:53.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3290.731010 secs 2024-03-28T19:09:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3290.731010 secs 2024-03-28T19:09:53.496 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4646 sec 2024-03-28T19:09:54.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:53 smithi071 ceph-mon[36597]: pgmap v2572: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:54.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:53 smithi151 ceph-mon[36481]: pgmap v2572: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:54.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:53 smithi045 ceph-mon[31773]: pgmap v2572: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:54.496 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4647 sec 2024-03-28T19:09:55.496 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4648 sec 2024-03-28T19:09:56.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:55 smithi071 ceph-mon[36597]: pgmap v2573: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:56.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:55 smithi151 ceph-mon[36481]: pgmap v2573: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:56.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:55 smithi045 ceph-mon[31773]: pgmap v2573: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:09:56.496 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4649 sec 2024-03-28T19:09:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4004.305630 secs 2024-03-28T19:09:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4004.305630 secs 2024-03-28T19:09:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4004.305630 secs 2024-03-28T19:09:57.496 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4650 sec 2024-03-28T19:09:58.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:57 smithi071 ceph-mon[36597]: pgmap v2574: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:58.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3295.731025 secs 2024-03-28T19:09:58.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:57 smithi151 ceph-mon[36481]: pgmap v2574: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:58.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3295.731025 secs 2024-03-28T19:09:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:57 smithi045 ceph-mon[31773]: pgmap v2574: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:09:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3295.731025 secs 2024-03-28T19:09:58.496 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4651 sec 2024-03-28T19:09:59.497 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4652 sec 2024-03-28T19:10:00.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:09:59 smithi071 ceph-mon[36597]: pgmap v2575: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:00.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:09:59 smithi151 ceph-mon[36481]: pgmap v2575: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:00.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:09:59 smithi045 ceph-mon[31773]: pgmap v2575: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:00.497 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4653 sec 2024-03-28T19:10:01.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:00 smithi071 ceph-mon[36597]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-03-28T19:10:01.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:00 smithi151 ceph-mon[36481]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-03-28T19:10:01.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:00 smithi045 ceph-mon[31773]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-03-28T19:10:01.497 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4654 sec 2024-03-28T19:10:02.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:01 smithi071 ceph-mon[36597]: pgmap v2576: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4009.305692 secs 2024-03-28T19:10:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:01 smithi151 ceph-mon[36481]: pgmap v2576: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4009.305692 secs 2024-03-28T19:10:02.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:01 smithi045 ceph-mon[31773]: pgmap v2576: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:02.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4009.305692 secs 2024-03-28T19:10:02.497 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4655 sec 2024-03-28T19:10:03.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3300.731113 secs 2024-03-28T19:10:03.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:02 smithi071 ceph-mon[36597]: pgmap v2577: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:03.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3300.731113 secs 2024-03-28T19:10:03.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:02 smithi151 ceph-mon[36481]: pgmap v2577: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:03.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3300.731113 secs 2024-03-28T19:10:03.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:02 smithi045 ceph-mon[31773]: pgmap v2577: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:03.497 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4656 sec 2024-03-28T19:10:04.497 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4657 sec 2024-03-28T19:10:05.497 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4658 sec 2024-03-28T19:10:05.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:05 smithi071 ceph-mon[36597]: pgmap v2578: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:05.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:05 smithi151 ceph-mon[36481]: pgmap v2578: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:05.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:05 smithi045 ceph-mon[31773]: pgmap v2578: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:06.498 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4659 sec 2024-03-28T19:10:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:06 smithi071 ceph-mon[36597]: 5 slow requests, 1 included below; oldest blocked for > 4014.305786 secs 2024-03-28T19:10:06.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:06 smithi071 ceph-mon[36597]: slow request 1444.117646 seconds old, received at 2024-03-28T18:46:01.748685+0000: client_request(client.26723:1 getattr pAsLsXsFs #0x1 2024-03-28T18:46:01.745025+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T19:10:06.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:06 smithi151 ceph-mon[36481]: 5 slow requests, 1 included below; oldest blocked for > 4014.305786 secs 2024-03-28T19:10:06.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:06 smithi151 ceph-mon[36481]: slow request 1444.117646 seconds old, received at 2024-03-28T18:46:01.748685+0000: client_request(client.26723:1 getattr pAsLsXsFs #0x1 2024-03-28T18:46:01.745025+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T19:10:06.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:06 smithi045 ceph-mon[31773]: 5 slow requests, 1 included below; oldest blocked for > 4014.305786 secs 2024-03-28T19:10:06.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:06 smithi045 ceph-mon[31773]: slow request 1444.117646 seconds old, received at 2024-03-28T18:46:01.748685+0000: client_request(client.26723:1 getattr pAsLsXsFs #0x1 2024-03-28T18:46:01.745025+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T19:10:07.497 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4660 sec 2024-03-28T19:10:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:07 smithi071 ceph-mon[36597]: pgmap v2579: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3305.731177 secs 2024-03-28T19:10:07.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:07 smithi151 ceph-mon[36481]: pgmap v2579: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:07.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3305.731177 secs 2024-03-28T19:10:07.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:07 smithi045 ceph-mon[31773]: pgmap v2579: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:07.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3305.731177 secs 2024-03-28T19:10:08.498 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4661 sec 2024-03-28T19:10:09.498 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4662 sec 2024-03-28T19:10:09.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:09 smithi071 ceph-mon[36597]: pgmap v2580: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:09.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:09 smithi151 ceph-mon[36481]: pgmap v2580: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:09.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:09 smithi045 ceph-mon[31773]: pgmap v2580: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:10.498 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4663 sec 2024-03-28T19:10:11.498 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4664 sec 2024-03-28T19:10:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:11 smithi071 ceph-mon[36597]: pgmap v2581: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4019.305824 secs 2024-03-28T19:10:11.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:11 smithi151 ceph-mon[36481]: pgmap v2581: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:11.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4019.305824 secs 2024-03-28T19:10:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:11 smithi045 ceph-mon[31773]: pgmap v2581: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4019.305824 secs 2024-03-28T19:10:12.498 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4665 sec 2024-03-28T19:10:12.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3310.731211 secs 2024-03-28T19:10:12.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3310.731211 secs 2024-03-28T19:10:12.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3310.731211 secs 2024-03-28T19:10:13.498 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4666 sec 2024-03-28T19:10:13.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:13 smithi071 ceph-mon[36597]: pgmap v2582: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:13.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:13 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:10:13.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:13 smithi151 ceph-mon[36481]: pgmap v2582: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:13.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:13 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:10:13.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:13 smithi045 ceph-mon[31773]: pgmap v2582: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:13.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:13 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:10:14.499 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4667 sec 2024-03-28T19:10:14.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:14 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:10:14.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:14 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:10:14.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:14 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:10:14.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:14 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:10:14.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:14 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:10:14.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:14 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:10:14.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:14 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:10:14.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:14 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:10:14.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:14 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:10:15.499 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4668 sec 2024-03-28T19:10:15.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:15 smithi071 ceph-mon[36597]: pgmap v2583: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:15.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:15 smithi071 ceph-mon[36597]: pgmap v2584: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:15.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:15 smithi045 ceph-mon[31773]: pgmap v2583: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:15.816 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:15 smithi045 ceph-mon[31773]: pgmap v2584: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:15.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:15 smithi151 ceph-mon[36481]: pgmap v2583: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:15.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:15 smithi151 ceph-mon[36481]: pgmap v2584: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:16.499 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4669 sec 2024-03-28T19:10:16.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4024.305915 secs 2024-03-28T19:10:16.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4024.305915 secs 2024-03-28T19:10:16.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4024.305915 secs 2024-03-28T19:10:17.499 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4670 sec 2024-03-28T19:10:17.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:17 smithi071 ceph-mon[36597]: pgmap v2585: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:17.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3315.731301 secs 2024-03-28T19:10:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:17 smithi151 ceph-mon[36481]: pgmap v2585: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3315.731301 secs 2024-03-28T19:10:17.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:17 smithi045 ceph-mon[31773]: pgmap v2585: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:17.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3315.731301 secs 2024-03-28T19:10:18.499 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4671 sec 2024-03-28T19:10:19.500 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4672 sec 2024-03-28T19:10:19.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:19 smithi071 ceph-mon[36597]: pgmap v2586: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:19.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:19 smithi151 ceph-mon[36481]: pgmap v2586: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:19.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:19 smithi045 ceph-mon[31773]: pgmap v2586: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:20.499 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4673 sec 2024-03-28T19:10:21.499 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4674 sec 2024-03-28T19:10:21.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:21 smithi151 ceph-mon[36481]: pgmap v2587: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T19:10:21.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4029.306003 secs 2024-03-28T19:10:21.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:21 smithi045 ceph-mon[31773]: pgmap v2587: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T19:10:21.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4029.306003 secs 2024-03-28T19:10:22.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:21 smithi071 ceph-mon[36597]: pgmap v2587: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-28T19:10:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4029.306003 secs 2024-03-28T19:10:22.499 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4675 sec 2024-03-28T19:10:22.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3320.731363 secs 2024-03-28T19:10:23.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3320.731363 secs 2024-03-28T19:10:23.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3320.731363 secs 2024-03-28T19:10:23.499 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4676 sec 2024-03-28T19:10:23.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:23 smithi045 ceph-mon[31773]: pgmap v2588: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:24.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:23 smithi071 ceph-mon[36597]: pgmap v2588: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:24.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:23 smithi151 ceph-mon[36481]: pgmap v2588: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:10:24.500 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4677 sec 2024-03-28T19:10:25.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:25 smithi071 ceph-mon[36597]: pgmap v2589: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:10:25.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:25 smithi151 ceph-mon[36481]: pgmap v2589: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:10:25.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:25 smithi045 ceph-mon[31773]: pgmap v2589: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:10:25.500 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4678 sec 2024-03-28T19:10:26.500 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4679 sec 2024-03-28T19:10:27.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:27 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4034.306064 secs 2024-03-28T19:10:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:27 smithi071 ceph-mon[36597]: pgmap v2590: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:27.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:27 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4034.306064 secs 2024-03-28T19:10:27.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:27 smithi151 ceph-mon[36481]: pgmap v2590: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:27.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:27 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4034.306064 secs 2024-03-28T19:10:27.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:27 smithi045 ceph-mon[31773]: pgmap v2590: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:27.500 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4680 sec 2024-03-28T19:10:28.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:28 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3325.731421 secs 2024-03-28T19:10:28.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:28 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3325.731421 secs 2024-03-28T19:10:28.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:28 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3325.731421 secs 2024-03-28T19:10:28.500 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4681 sec 2024-03-28T19:10:29.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:29 smithi071 ceph-mon[36597]: pgmap v2591: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:29.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:29 smithi151 ceph-mon[36481]: pgmap v2591: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:29.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:29 smithi045 ceph-mon[31773]: pgmap v2591: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:29.500 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4682 sec 2024-03-28T19:10:30.501 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4683 sec 2024-03-28T19:10:31.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:31 smithi071 ceph-mon[36597]: pgmap v2592: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:31.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:31 smithi151 ceph-mon[36481]: pgmap v2592: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:31.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:31 smithi045 ceph-mon[31773]: pgmap v2592: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:31.501 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4684 sec 2024-03-28T19:10:32.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:32 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4039.306152 secs 2024-03-28T19:10:32.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:32 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4039.306152 secs 2024-03-28T19:10:32.501 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4685 sec 2024-03-28T19:10:32.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:32 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4039.306152 secs 2024-03-28T19:10:33.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:33 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3330.731509 secs 2024-03-28T19:10:33.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:33 smithi151 ceph-mon[36481]: pgmap v2593: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:33.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:33 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3330.731509 secs 2024-03-28T19:10:33.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:33 smithi045 ceph-mon[31773]: pgmap v2593: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:33.501 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4686 sec 2024-03-28T19:10:33.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:33 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3330.731509 secs 2024-03-28T19:10:33.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:33 smithi071 ceph-mon[36597]: pgmap v2593: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:34.501 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4687 sec 2024-03-28T19:10:35.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:35 smithi045 ceph-mon[31773]: pgmap v2594: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:35.501 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4688 sec 2024-03-28T19:10:35.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:35 smithi071 ceph-mon[36597]: pgmap v2594: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:35.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:35 smithi151 ceph-mon[36481]: pgmap v2594: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:36.501 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4689 sec 2024-03-28T19:10:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:37 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4044.306234 secs 2024-03-28T19:10:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:37 smithi045 ceph-mon[31773]: pgmap v2595: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:37.501 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4690 sec 2024-03-28T19:10:37.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:37 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4044.306234 secs 2024-03-28T19:10:37.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:37 smithi071 ceph-mon[36597]: pgmap v2595: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:37.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:37 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4044.306234 secs 2024-03-28T19:10:37.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:37 smithi151 ceph-mon[36481]: pgmap v2595: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:38.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:38 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3335.731568 secs 2024-03-28T19:10:38.502 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4691 sec 2024-03-28T19:10:38.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:38 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3335.731568 secs 2024-03-28T19:10:38.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:38 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3335.731568 secs 2024-03-28T19:10:39.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:39 smithi045 ceph-mon[31773]: pgmap v2596: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:39.502 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4692 sec 2024-03-28T19:10:39.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:39 smithi071 ceph-mon[36597]: pgmap v2596: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:39.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:39 smithi151 ceph-mon[36481]: pgmap v2596: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:40.502 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4693 sec 2024-03-28T19:10:41.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:41 smithi045 ceph-mon[31773]: pgmap v2597: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:41.502 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4694 sec 2024-03-28T19:10:41.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:41 smithi071 ceph-mon[36597]: pgmap v2597: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:41.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:41 smithi151 ceph-mon[36481]: pgmap v2597: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:42 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4049.306321 secs 2024-03-28T19:10:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3340.731655 secs 2024-03-28T19:10:42.473 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:42 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4049.306321 secs 2024-03-28T19:10:42.473 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3340.731655 secs 2024-03-28T19:10:42.502 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4695 sec 2024-03-28T19:10:42.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:42 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4049.306321 secs 2024-03-28T19:10:42.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3340.731655 secs 2024-03-28T19:10:43.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:43 smithi045 ceph-mon[31773]: pgmap v2598: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:43.502 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4696 sec 2024-03-28T19:10:43.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:43 smithi071 ceph-mon[36597]: pgmap v2598: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:43.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:43 smithi151 ceph-mon[36481]: pgmap v2598: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:44.503 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4697 sec 2024-03-28T19:10:45.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:45 smithi045 ceph-mon[31773]: pgmap v2599: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:45.503 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4698 sec 2024-03-28T19:10:45.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:45 smithi071 ceph-mon[36597]: pgmap v2599: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:45.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:45 smithi151 ceph-mon[36481]: pgmap v2599: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:46.503 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4699 sec 2024-03-28T19:10:47.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:47 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4054.306359 secs 2024-03-28T19:10:47.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:47 smithi045 ceph-mon[31773]: pgmap v2600: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:47.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3345.731716 secs 2024-03-28T19:10:47.503 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4700 sec 2024-03-28T19:10:47.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:47 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4054.306359 secs 2024-03-28T19:10:47.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:47 smithi071 ceph-mon[36597]: pgmap v2600: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:47.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3345.731716 secs 2024-03-28T19:10:47.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:47 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4054.306359 secs 2024-03-28T19:10:47.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:47 smithi151 ceph-mon[36481]: pgmap v2600: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:47.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3345.731716 secs 2024-03-28T19:10:48.503 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4701 sec 2024-03-28T19:10:49.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:49 smithi045 ceph-mon[31773]: pgmap v2601: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:49.503 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4702 sec 2024-03-28T19:10:49.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:49 smithi071 ceph-mon[36597]: pgmap v2601: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:49.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:49 smithi151 ceph-mon[36481]: pgmap v2601: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:50.503 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4703 sec 2024-03-28T19:10:51.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:51 smithi045 ceph-mon[31773]: pgmap v2602: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:51.503 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4704 sec 2024-03-28T19:10:51.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:51 smithi071 ceph-mon[36597]: pgmap v2602: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:51.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:51 smithi151 ceph-mon[36481]: pgmap v2602: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:52.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:52 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4059.306420 secs 2024-03-28T19:10:52.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3350.731806 secs 2024-03-28T19:10:52.503 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4705 sec 2024-03-28T19:10:52.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:52 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4059.306420 secs 2024-03-28T19:10:52.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3350.731806 secs 2024-03-28T19:10:52.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:52 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4059.306420 secs 2024-03-28T19:10:52.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3350.731806 secs 2024-03-28T19:10:53.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:53 smithi045 ceph-mon[31773]: pgmap v2603: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:53.504 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4706 sec 2024-03-28T19:10:53.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:53 smithi071 ceph-mon[36597]: pgmap v2603: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:53.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:53 smithi151 ceph-mon[36481]: pgmap v2603: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:54.504 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4707 sec 2024-03-28T19:10:55.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:55 smithi045 ceph-mon[31773]: pgmap v2604: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:55.504 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4708 sec 2024-03-28T19:10:55.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:55 smithi071 ceph-mon[36597]: pgmap v2604: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:55.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:55 smithi151 ceph-mon[36481]: pgmap v2604: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:56.504 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4709 sec 2024-03-28T19:10:57.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:57 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4064.306506 secs 2024-03-28T19:10:57.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:57 smithi045 ceph-mon[31773]: pgmap v2605: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:57.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3355.731894 secs 2024-03-28T19:10:57.504 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4710 sec 2024-03-28T19:10:57.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:57 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4064.306506 secs 2024-03-28T19:10:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:57 smithi071 ceph-mon[36597]: pgmap v2605: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:57.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3355.731894 secs 2024-03-28T19:10:57.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:57 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4064.306506 secs 2024-03-28T19:10:57.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:57 smithi151 ceph-mon[36481]: pgmap v2605: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:10:57.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3355.731894 secs 2024-03-28T19:10:58.504 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4711 sec 2024-03-28T19:10:59.505 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4712 sec 2024-03-28T19:10:59.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:10:59 smithi071 ceph-mon[36597]: pgmap v2606: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:59.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:10:59 smithi151 ceph-mon[36481]: pgmap v2606: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:10:59.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:10:59 smithi045 ceph-mon[31773]: pgmap v2606: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:00.505 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4713 sec 2024-03-28T19:11:01.505 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4714 sec 2024-03-28T19:11:01.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:01 smithi071 ceph-mon[36597]: pgmap v2607: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:01.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:01 smithi151 ceph-mon[36481]: pgmap v2607: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:01.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:01 smithi045 ceph-mon[31773]: pgmap v2607: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:02.505 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4715 sec 2024-03-28T19:11:02.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:02 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4069.306569 secs 2024-03-28T19:11:02.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3360.731954 secs 2024-03-28T19:11:02.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:02 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4069.306569 secs 2024-03-28T19:11:02.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3360.731954 secs 2024-03-28T19:11:02.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:02 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4069.306569 secs 2024-03-28T19:11:02.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3360.731954 secs 2024-03-28T19:11:03.505 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4716 sec 2024-03-28T19:11:03.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:03 smithi071 ceph-mon[36597]: pgmap v2608: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:03.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:03 smithi151 ceph-mon[36481]: pgmap v2608: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:03.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:03 smithi045 ceph-mon[31773]: pgmap v2608: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:04.505 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4717 sec 2024-03-28T19:11:05.505 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4718 sec 2024-03-28T19:11:05.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:05 smithi071 ceph-mon[36597]: pgmap v2609: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:05.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:05 smithi151 ceph-mon[36481]: pgmap v2609: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:05.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:05 smithi045 ceph-mon[31773]: pgmap v2609: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:06.506 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4719 sec 2024-03-28T19:11:07.506 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4720 sec 2024-03-28T19:11:07.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:07 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4074.306632 secs 2024-03-28T19:11:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:07 smithi071 ceph-mon[36597]: pgmap v2610: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:07.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3365.732016 secs 2024-03-28T19:11:07.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:07 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4074.306632 secs 2024-03-28T19:11:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:07 smithi151 ceph-mon[36481]: pgmap v2610: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:07.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3365.732016 secs 2024-03-28T19:11:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:07 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4074.306632 secs 2024-03-28T19:11:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:07 smithi045 ceph-mon[31773]: pgmap v2610: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3365.732016 secs 2024-03-28T19:11:08.506 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4721 sec 2024-03-28T19:11:09.506 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4722 sec 2024-03-28T19:11:09.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:09 smithi071 ceph-mon[36597]: pgmap v2611: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:09.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:09 smithi151 ceph-mon[36481]: pgmap v2611: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:09.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:09 smithi045 ceph-mon[31773]: pgmap v2611: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:10.506 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4723 sec 2024-03-28T19:11:11.506 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4724 sec 2024-03-28T19:11:11.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:11 smithi071 ceph-mon[36597]: pgmap v2612: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:11.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:11 smithi151 ceph-mon[36481]: pgmap v2612: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:11 smithi045 ceph-mon[31773]: pgmap v2612: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:12.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:12 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4079.306692 secs 2024-03-28T19:11:12.474 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3370.732075 secs 2024-03-28T19:11:12.506 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4725 sec 2024-03-28T19:11:12.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:12 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4079.306692 secs 2024-03-28T19:11:12.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3370.732075 secs 2024-03-28T19:11:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:12 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4079.306692 secs 2024-03-28T19:11:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3370.732075 secs 2024-03-28T19:11:13.507 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4726 sec 2024-03-28T19:11:13.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:13 smithi071 ceph-mon[36597]: pgmap v2613: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:13.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:13 smithi151 ceph-mon[36481]: pgmap v2613: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:13.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:13 smithi045 ceph-mon[31773]: pgmap v2613: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:14.507 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4727 sec 2024-03-28T19:11:15.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:15 smithi045 ceph-mon[31773]: pgmap v2614: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:15.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:15 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:11:15.507 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4728 sec 2024-03-28T19:11:15.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:15 smithi071 ceph-mon[36597]: pgmap v2614: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:15.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:15 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:11:15.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:15 smithi151 ceph-mon[36481]: pgmap v2614: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:15.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:15 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:11:16.507 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4729 sec 2024-03-28T19:11:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:16 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:11:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:16 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:11:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:16 smithi071 ceph-mon[36597]: pgmap v2615: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:11:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:16 smithi071 ceph-mon[36597]: pgmap v2616: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:11:16.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:16 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:11:16.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:16 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:11:16.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:16 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:11:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:16 smithi151 ceph-mon[36481]: pgmap v2615: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:11:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:16 smithi151 ceph-mon[36481]: pgmap v2616: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:11:16.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:16 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:11:16.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:16 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:11:16.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:16 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:11:16.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:16 smithi045 ceph-mon[31773]: pgmap v2615: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:11:16.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:16 smithi045 ceph-mon[31773]: pgmap v2616: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:11:16.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:16 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:11:17.507 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4730 sec 2024-03-28T19:11:17.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:17 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4084.306754 secs 2024-03-28T19:11:17.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3375.732137 secs 2024-03-28T19:11:17.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:17 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4084.306754 secs 2024-03-28T19:11:17.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3375.732137 secs 2024-03-28T19:11:17.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:17 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4084.306754 secs 2024-03-28T19:11:17.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3375.732137 secs 2024-03-28T19:11:18.507 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4731 sec 2024-03-28T19:11:18.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:18 smithi071 ceph-mon[36597]: pgmap v2617: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:11:18.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:18 smithi151 ceph-mon[36481]: pgmap v2617: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:11:18.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:18 smithi045 ceph-mon[31773]: pgmap v2617: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:11:19.508 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4732 sec 2024-03-28T19:11:20.508 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4733 sec 2024-03-28T19:11:21.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:20 smithi071 ceph-mon[36597]: pgmap v2618: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:11:21.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:20 smithi151 ceph-mon[36481]: pgmap v2618: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:11:21.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:20 smithi045 ceph-mon[31773]: pgmap v2618: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:11:21.508 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4734 sec 2024-03-28T19:11:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4089.306816 secs 2024-03-28T19:11:22.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4089.306816 secs 2024-03-28T19:11:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4089.306816 secs 2024-03-28T19:11:22.508 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4735 sec 2024-03-28T19:11:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3380.732198 secs 2024-03-28T19:11:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:22 smithi071 ceph-mon[36597]: pgmap v2619: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:11:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3380.732198 secs 2024-03-28T19:11:23.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:22 smithi151 ceph-mon[36481]: pgmap v2619: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:11:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3380.732198 secs 2024-03-28T19:11:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:22 smithi045 ceph-mon[31773]: pgmap v2619: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:11:23.508 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4736 sec 2024-03-28T19:11:24.508 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4737 sec 2024-03-28T19:11:25.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:24 smithi071 ceph-mon[36597]: pgmap v2620: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:11:25.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:24 smithi151 ceph-mon[36481]: pgmap v2620: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:11:25.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:24 smithi045 ceph-mon[31773]: pgmap v2620: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:11:25.508 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4738 sec 2024-03-28T19:11:26.508 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4739 sec 2024-03-28T19:11:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:26 smithi071 ceph-mon[36597]: pgmap v2621: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:11:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4094.306903 secs 2024-03-28T19:11:27.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:26 smithi151 ceph-mon[36481]: pgmap v2621: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:11:27.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4094.306903 secs 2024-03-28T19:11:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:26 smithi045 ceph-mon[31773]: pgmap v2621: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:11:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4094.306903 secs 2024-03-28T19:11:27.508 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4740 sec 2024-03-28T19:11:28.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3385.732285 secs 2024-03-28T19:11:28.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3385.732285 secs 2024-03-28T19:11:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3385.732285 secs 2024-03-28T19:11:28.509 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4741 sec 2024-03-28T19:11:29.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:28 smithi071 ceph-mon[36597]: pgmap v2622: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:29.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:28 smithi151 ceph-mon[36481]: pgmap v2622: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:29.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:28 smithi045 ceph-mon[31773]: pgmap v2622: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:29.509 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4742 sec 2024-03-28T19:11:30.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:29 smithi071 ceph-mon[36597]: pgmap v2623: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:30.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:29 smithi151 ceph-mon[36481]: pgmap v2623: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:30.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:29 smithi045 ceph-mon[31773]: pgmap v2623: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:30.509 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4743 sec 2024-03-28T19:11:31.509 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4744 sec 2024-03-28T19:11:31.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4099.306966 secs 2024-03-28T19:11:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4099.306966 secs 2024-03-28T19:11:31.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4099.306966 secs 2024-03-28T19:11:32.509 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4745 sec 2024-03-28T19:11:32.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3390.732346 secs 2024-03-28T19:11:32.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:32 smithi045 ceph-mon[31773]: pgmap v2624: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:32.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3390.732346 secs 2024-03-28T19:11:32.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:32 smithi071 ceph-mon[36597]: pgmap v2624: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:32.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3390.732346 secs 2024-03-28T19:11:32.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:32 smithi151 ceph-mon[36481]: pgmap v2624: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:33.510 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4746 sec 2024-03-28T19:11:34.510 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4747 sec 2024-03-28T19:11:35.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:34 smithi071 ceph-mon[36597]: pgmap v2625: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:35.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:34 smithi151 ceph-mon[36481]: pgmap v2625: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:34 smithi045 ceph-mon[31773]: pgmap v2625: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:35.510 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4748 sec 2024-03-28T19:11:36.510 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4749 sec 2024-03-28T19:11:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:36 smithi071 ceph-mon[36597]: pgmap v2626: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4104.307030 secs 2024-03-28T19:11:37.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:36 smithi151 ceph-mon[36481]: pgmap v2626: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:37.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4104.307030 secs 2024-03-28T19:11:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:36 smithi045 ceph-mon[31773]: pgmap v2626: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4104.307030 secs 2024-03-28T19:11:37.510 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4750 sec 2024-03-28T19:11:38.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3395.732434 secs 2024-03-28T19:11:38.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3395.732434 secs 2024-03-28T19:11:38.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3395.732434 secs 2024-03-28T19:11:38.510 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4751 sec 2024-03-28T19:11:39.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:38 smithi071 ceph-mon[36597]: pgmap v2627: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:39.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:38 smithi151 ceph-mon[36481]: pgmap v2627: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:39.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:38 smithi045 ceph-mon[31773]: pgmap v2627: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:39.511 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4752 sec 2024-03-28T19:11:40.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:39 smithi045 ceph-mon[31773]: pgmap v2628: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:40.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:39 smithi071 ceph-mon[36597]: pgmap v2628: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:40.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:39 smithi151 ceph-mon[36481]: pgmap v2628: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:40.510 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4753 sec 2024-03-28T19:11:41.511 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4754 sec 2024-03-28T19:11:41.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4109.307143 secs 2024-03-28T19:11:41.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4109.307143 secs 2024-03-28T19:11:41.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4109.307143 secs 2024-03-28T19:11:42.511 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4755 sec 2024-03-28T19:11:42.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3400.732524 secs 2024-03-28T19:11:42.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:42 smithi045 ceph-mon[31773]: pgmap v2629: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:42.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3400.732524 secs 2024-03-28T19:11:42.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:42 smithi151 ceph-mon[36481]: pgmap v2629: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:42.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3400.732524 secs 2024-03-28T19:11:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:42 smithi071 ceph-mon[36597]: pgmap v2629: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:43.511 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4756 sec 2024-03-28T19:11:44.511 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4757 sec 2024-03-28T19:11:45.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:44 smithi071 ceph-mon[36597]: pgmap v2630: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:45.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:44 smithi151 ceph-mon[36481]: pgmap v2630: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:45.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:44 smithi045 ceph-mon[31773]: pgmap v2630: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:45.511 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4758 sec 2024-03-28T19:11:46.511 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4759 sec 2024-03-28T19:11:47.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:46 smithi071 ceph-mon[36597]: pgmap v2631: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4114.307234 secs 2024-03-28T19:11:47.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:46 smithi151 ceph-mon[36481]: pgmap v2631: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:47.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4114.307234 secs 2024-03-28T19:11:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:46 smithi045 ceph-mon[31773]: pgmap v2631: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4114.307234 secs 2024-03-28T19:11:47.511 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4760 sec 2024-03-28T19:11:48.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3405.732613 secs 2024-03-28T19:11:48.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3405.732613 secs 2024-03-28T19:11:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3405.732613 secs 2024-03-28T19:11:48.511 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4761 sec 2024-03-28T19:11:49.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:48 smithi071 ceph-mon[36597]: pgmap v2632: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:49.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:48 smithi151 ceph-mon[36481]: pgmap v2632: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:49.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:48 smithi045 ceph-mon[31773]: pgmap v2632: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:49.511 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4762 sec 2024-03-28T19:11:50.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:49 smithi071 ceph-mon[36597]: pgmap v2633: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:50.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:49 smithi151 ceph-mon[36481]: pgmap v2633: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:50.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:49 smithi045 ceph-mon[31773]: pgmap v2633: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:50.512 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4763 sec 2024-03-28T19:11:51.512 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4764 sec 2024-03-28T19:11:51.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4119.307293 secs 2024-03-28T19:11:51.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4119.307293 secs 2024-03-28T19:11:51.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4119.307293 secs 2024-03-28T19:11:52.512 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4765 sec 2024-03-28T19:11:52.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3410.732700 secs 2024-03-28T19:11:52.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:52 smithi045 ceph-mon[31773]: pgmap v2634: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:52.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3410.732700 secs 2024-03-28T19:11:52.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:52 smithi071 ceph-mon[36597]: pgmap v2634: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:52.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3410.732700 secs 2024-03-28T19:11:52.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:52 smithi151 ceph-mon[36481]: pgmap v2634: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:53.512 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4766 sec 2024-03-28T19:11:54.512 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4767 sec 2024-03-28T19:11:55.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:54 smithi071 ceph-mon[36597]: pgmap v2635: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:55.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:54 smithi151 ceph-mon[36481]: pgmap v2635: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:55.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:54 smithi045 ceph-mon[31773]: pgmap v2635: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:55.512 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4768 sec 2024-03-28T19:11:56.513 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4769 sec 2024-03-28T19:11:57.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:56 smithi071 ceph-mon[36597]: pgmap v2636: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4124.307308 secs 2024-03-28T19:11:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:56 smithi151 ceph-mon[36481]: pgmap v2636: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4124.307308 secs 2024-03-28T19:11:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:56 smithi045 ceph-mon[31773]: pgmap v2636: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:11:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4124.307308 secs 2024-03-28T19:11:57.513 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4770 sec 2024-03-28T19:11:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3415.732787 secs 2024-03-28T19:11:58.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3415.732787 secs 2024-03-28T19:11:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3415.732787 secs 2024-03-28T19:11:58.513 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4771 sec 2024-03-28T19:11:59.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:58 smithi071 ceph-mon[36597]: pgmap v2637: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:59.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:58 smithi151 ceph-mon[36481]: pgmap v2637: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:59.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:58 smithi045 ceph-mon[31773]: pgmap v2637: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:11:59.513 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4772 sec 2024-03-28T19:12:00.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:11:59 smithi071 ceph-mon[36597]: pgmap v2638: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:00.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:11:59 smithi151 ceph-mon[36481]: pgmap v2638: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:00.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:11:59 smithi045 ceph-mon[31773]: pgmap v2638: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:00.513 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4773 sec 2024-03-28T19:12:01.513 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4774 sec 2024-03-28T19:12:01.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4129.307391 secs 2024-03-28T19:12:01.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4129.307391 secs 2024-03-28T19:12:01.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4129.307391 secs 2024-03-28T19:12:02.513 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4775 sec 2024-03-28T19:12:02.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3420.732875 secs 2024-03-28T19:12:02.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:02 smithi045 ceph-mon[31773]: pgmap v2639: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:02.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3420.732875 secs 2024-03-28T19:12:02.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:02 smithi071 ceph-mon[36597]: pgmap v2639: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:02.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3420.732875 secs 2024-03-28T19:12:02.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:02 smithi151 ceph-mon[36481]: pgmap v2639: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:03.514 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4776 sec 2024-03-28T19:12:04.514 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4777 sec 2024-03-28T19:12:05.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:04 smithi071 ceph-mon[36597]: pgmap v2640: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:05.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:04 smithi151 ceph-mon[36481]: pgmap v2640: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:05.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:04 smithi045 ceph-mon[31773]: pgmap v2640: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:05.514 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4778 sec 2024-03-28T19:12:06.514 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4779 sec 2024-03-28T19:12:07.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:06 smithi071 ceph-mon[36597]: pgmap v2641: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:07.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4134.307482 secs 2024-03-28T19:12:07.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:06 smithi151 ceph-mon[36481]: pgmap v2641: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:07.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4134.307482 secs 2024-03-28T19:12:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:06 smithi045 ceph-mon[31773]: pgmap v2641: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4134.307482 secs 2024-03-28T19:12:07.514 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4780 sec 2024-03-28T19:12:08.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3425.732956 secs 2024-03-28T19:12:08.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3425.732956 secs 2024-03-28T19:12:08.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3425.732956 secs 2024-03-28T19:12:08.514 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4781 sec 2024-03-28T19:12:09.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:08 smithi071 ceph-mon[36597]: pgmap v2642: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:09.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:08 smithi151 ceph-mon[36481]: pgmap v2642: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:09.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:08 smithi045 ceph-mon[31773]: pgmap v2642: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:09.514 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4782 sec 2024-03-28T19:12:10.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:09 smithi071 ceph-mon[36597]: pgmap v2643: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:10.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:09 smithi151 ceph-mon[36481]: pgmap v2643: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:10.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:09 smithi045 ceph-mon[31773]: pgmap v2643: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:10.514 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4783 sec 2024-03-28T19:12:11.515 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4784 sec 2024-03-28T19:12:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4139.307543 secs 2024-03-28T19:12:11.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4139.307543 secs 2024-03-28T19:12:11.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4139.307543 secs 2024-03-28T19:12:12.515 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4785 sec 2024-03-28T19:12:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3430.733040 secs 2024-03-28T19:12:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:12 smithi045 ceph-mon[31773]: pgmap v2644: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:12.723 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3430.733040 secs 2024-03-28T19:12:12.723 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:12 smithi151 ceph-mon[36481]: pgmap v2644: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:12.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3430.733040 secs 2024-03-28T19:12:12.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:12 smithi071 ceph-mon[36597]: pgmap v2644: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:13.515 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4786 sec 2024-03-28T19:12:14.515 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4787 sec 2024-03-28T19:12:15.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:14 smithi071 ceph-mon[36597]: pgmap v2645: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:15.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:14 smithi151 ceph-mon[36481]: pgmap v2645: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:15.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:14 smithi045 ceph-mon[31773]: pgmap v2645: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:15.515 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4788 sec 2024-03-28T19:12:16.515 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4789 sec 2024-03-28T19:12:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:16 smithi071 ceph-mon[36597]: pgmap v2646: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4144.307608 secs 2024-03-28T19:12:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:16 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:12:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:16 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:16 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:16 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:16 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:17.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:16 smithi045 ceph-mon[31773]: pgmap v2646: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:17.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4144.307608 secs 2024-03-28T19:12:17.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:16 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:12:17.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:16 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:17.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:16 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:17.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:16 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:17.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:16 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:17.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:16 smithi151 ceph-mon[36481]: pgmap v2646: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:17.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4144.307608 secs 2024-03-28T19:12:17.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:16 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:12:17.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:16 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:17.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:16 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:17.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:16 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:17.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:16 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:17.515 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4790 sec 2024-03-28T19:12:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3435.733110 secs 2024-03-28T19:12:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:17 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:17 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:17 smithi045 ceph-mon[31773]: pgmap v2647: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:18.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3435.733110 secs 2024-03-28T19:12:18.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:17 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:18.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:17 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:18.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:17 smithi071 ceph-mon[36597]: pgmap v2647: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:18.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3435.733110 secs 2024-03-28T19:12:18.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:17 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:18.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:17 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:18.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:17 smithi151 ceph-mon[36481]: pgmap v2647: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:18.516 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4791 sec 2024-03-28T19:12:19.516 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4792 sec 2024-03-28T19:12:19.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:19 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:19.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:19 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:19.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:19 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:19.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:19 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:19.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:19 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:19 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:19 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:19 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:19 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:19 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:19 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:19 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:19 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:19 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:19 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:19 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:19 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:19 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:19 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.812 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:19 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:19 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:19.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:19 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:19.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:19 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:19.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:19 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:19.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:19 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:19 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:19 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:19 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:19 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:19 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:19 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:19 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:19 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:19 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:19 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:19 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:19 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:19 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:19 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:19 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:19 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:19.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:19 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:19.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:19 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:19.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:19 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:19.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:19 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:19 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:19 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:19 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:19 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:19 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:19 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:19 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:19 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:19 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:19 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:19 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:19 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:19 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:19 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:19.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:19 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:20.516 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4793 sec 2024-03-28T19:12:20.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:20 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T19:12:20.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:20 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T19:12:20.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:20 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T19:12:20.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:20 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T19:12:20.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:20 smithi071 ceph-mon[36597]: pgmap v2648: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:20.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:20 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T19:12:20.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:20 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T19:12:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:20 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T19:12:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:20 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T19:12:20.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:20 smithi151 ceph-mon[36481]: pgmap v2648: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:20.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:20 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T19:12:20.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:20 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T19:12:20.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:20 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T19:12:20.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:20 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T19:12:20.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:20 smithi045 ceph-mon[31773]: pgmap v2648: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:21.516 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4794 sec 2024-03-28T19:12:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4149.307667 secs 2024-03-28T19:12:21.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4149.307667 secs 2024-03-28T19:12:21.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4149.307667 secs 2024-03-28T19:12:22.516 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4795 sec 2024-03-28T19:12:22.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:22 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:22.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:22 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:22.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3440.733170 secs 2024-03-28T19:12:22.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:22 smithi045 ceph-mon[31773]: pgmap v2649: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:23.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:22 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:22 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3440.733170 secs 2024-03-28T19:12:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:22 smithi071 ceph-mon[36597]: pgmap v2649: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:23.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:22 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:23.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:22 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:23.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3440.733170 secs 2024-03-28T19:12:23.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:22 smithi151 ceph-mon[36481]: pgmap v2649: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:23.516 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4796 sec 2024-03-28T19:12:23.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:23 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:23.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:23 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:23.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:23 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:23.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:23 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:23.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:23 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:23.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:23 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:23.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:23 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:23.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:23 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:23.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:23 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:12:23.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:23 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:12:23.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:23 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:24.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:23 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:24.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:23 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:24.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:23 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:24.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:23 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:24.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:23 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:24.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:23 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:24.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:23 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:24.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:23 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:24.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:23 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:12:24.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:23 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:12:24.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:23 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:24.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:23 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:24.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:23 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:24.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:23 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:24.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:23 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:24.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:23 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:24.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:23 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:24.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:23 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:24.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:23 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:12:24.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:23 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:12:24.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:23 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:12:24.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:23 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:12:24.517 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4797 sec 2024-03-28T19:12:24.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:24 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T19:12:24.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:24 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T19:12:24.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:24 smithi045 ceph-mon[31773]: pgmap v2650: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T19:12:24.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:24 smithi045 ceph-mon[31773]: pgmap v2651: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:12:25.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:24 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T19:12:25.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:24 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T19:12:25.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:24 smithi071 ceph-mon[36597]: pgmap v2650: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T19:12:25.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:24 smithi071 ceph-mon[36597]: pgmap v2651: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:12:25.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:24 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T19:12:25.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:24 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T19:12:25.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:24 smithi151 ceph-mon[36481]: pgmap v2650: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T19:12:25.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:24 smithi151 ceph-mon[36481]: pgmap v2651: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:12:25.517 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4798 sec 2024-03-28T19:12:26.517 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4799 sec 2024-03-28T19:12:26.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:26 smithi045 ceph-mon[31773]: pgmap v2652: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:12:26.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4154.307730 secs 2024-03-28T19:12:27.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:26 smithi071 ceph-mon[36597]: pgmap v2652: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:12:27.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4154.307730 secs 2024-03-28T19:12:27.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:26 smithi151 ceph-mon[36481]: pgmap v2652: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:12:27.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4154.307730 secs 2024-03-28T19:12:27.517 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4800 sec 2024-03-28T19:12:27.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3445.733254 secs 2024-03-28T19:12:28.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3445.733254 secs 2024-03-28T19:12:28.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3445.733254 secs 2024-03-28T19:12:28.517 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4801 sec 2024-03-28T19:12:28.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:28 smithi045 ceph-mon[31773]: pgmap v2653: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:12:29.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:28 smithi071 ceph-mon[36597]: pgmap v2653: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:12:29.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:28 smithi151 ceph-mon[36481]: pgmap v2653: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:12:29.517 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4802 sec 2024-03-28T19:12:30.517 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4803 sec 2024-03-28T19:12:31.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:30 smithi071 ceph-mon[36597]: pgmap v2654: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:12:31.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:30 smithi151 ceph-mon[36481]: pgmap v2654: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:12:31.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:30 smithi045 ceph-mon[31773]: pgmap v2654: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:12:31.517 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4804 sec 2024-03-28T19:12:32.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4159.307792 secs 2024-03-28T19:12:32.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4159.307792 secs 2024-03-28T19:12:32.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4159.307792 secs 2024-03-28T19:12:32.517 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4805 sec 2024-03-28T19:12:33.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:32 smithi071 ceph-mon[36597]: pgmap v2655: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:12:33.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3450.733316 secs 2024-03-28T19:12:33.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:32 smithi151 ceph-mon[36481]: pgmap v2655: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:12:33.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3450.733316 secs 2024-03-28T19:12:33.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:32 smithi045 ceph-mon[31773]: pgmap v2655: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:12:33.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3450.733316 secs 2024-03-28T19:12:33.517 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4806 sec 2024-03-28T19:12:34.518 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4807 sec 2024-03-28T19:12:35.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:34 smithi071 ceph-mon[36597]: pgmap v2656: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:12:35.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:34 smithi151 ceph-mon[36481]: pgmap v2656: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:12:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:34 smithi045 ceph-mon[31773]: pgmap v2656: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:12:35.518 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4808 sec 2024-03-28T19:12:36.518 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4809 sec 2024-03-28T19:12:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:36 smithi071 ceph-mon[36597]: pgmap v2657: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:37.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4164.307879 secs 2024-03-28T19:12:37.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:36 smithi151 ceph-mon[36481]: pgmap v2657: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:37.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4164.307879 secs 2024-03-28T19:12:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:36 smithi045 ceph-mon[31773]: pgmap v2657: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4164.307879 secs 2024-03-28T19:12:37.518 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4810 sec 2024-03-28T19:12:38.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3455.733373 secs 2024-03-28T19:12:38.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3455.733373 secs 2024-03-28T19:12:38.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3455.733373 secs 2024-03-28T19:12:38.518 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4811 sec 2024-03-28T19:12:39.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:38 smithi071 ceph-mon[36597]: pgmap v2658: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:39.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:38 smithi151 ceph-mon[36481]: pgmap v2658: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:39.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:38 smithi045 ceph-mon[31773]: pgmap v2658: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:39.518 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4812 sec 2024-03-28T19:12:40.519 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4813 sec 2024-03-28T19:12:41.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:40 smithi071 ceph-mon[36597]: pgmap v2659: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:41.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:40 smithi151 ceph-mon[36481]: pgmap v2659: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:41.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:40 smithi045 ceph-mon[31773]: pgmap v2659: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:41.519 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4814 sec 2024-03-28T19:12:42.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4169.307916 secs 2024-03-28T19:12:42.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4169.307916 secs 2024-03-28T19:12:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4169.307916 secs 2024-03-28T19:12:42.519 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4815 sec 2024-03-28T19:12:42.973 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:42 smithi151 ceph-mon[36481]: pgmap v2660: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:42.973 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3460.733461 secs 2024-03-28T19:12:43.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:42 smithi071 ceph-mon[36597]: pgmap v2660: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:43.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3460.733461 secs 2024-03-28T19:12:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:42 smithi045 ceph-mon[31773]: pgmap v2660: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3460.733461 secs 2024-03-28T19:12:43.519 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4816 sec 2024-03-28T19:12:44.519 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4817 sec 2024-03-28T19:12:45.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:44 smithi071 ceph-mon[36597]: pgmap v2661: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:45.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:44 smithi151 ceph-mon[36481]: pgmap v2661: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:45.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:44 smithi045 ceph-mon[31773]: pgmap v2661: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:45.519 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4818 sec 2024-03-28T19:12:46.520 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4819 sec 2024-03-28T19:12:47.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:46 smithi071 ceph-mon[36597]: pgmap v2662: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4174.308000 secs 2024-03-28T19:12:47.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:46 smithi151 ceph-mon[36481]: pgmap v2662: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:47.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4174.308000 secs 2024-03-28T19:12:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:46 smithi045 ceph-mon[31773]: pgmap v2662: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4174.308000 secs 2024-03-28T19:12:47.519 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4820 sec 2024-03-28T19:12:47.998 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3465.733544 secs 2024-03-28T19:12:48.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3465.733544 secs 2024-03-28T19:12:48.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3465.733544 secs 2024-03-28T19:12:48.520 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4821 sec 2024-03-28T19:12:49.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:48 smithi071 ceph-mon[36597]: pgmap v2663: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:49.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:48 smithi151 ceph-mon[36481]: pgmap v2663: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:49.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:48 smithi045 ceph-mon[31773]: pgmap v2663: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:49.520 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4822 sec 2024-03-28T19:12:50.520 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4823 sec 2024-03-28T19:12:51.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:50 smithi071 ceph-mon[36597]: pgmap v2664: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:51.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:50 smithi151 ceph-mon[36481]: pgmap v2664: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:51.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:50 smithi045 ceph-mon[31773]: pgmap v2664: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:51.520 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4824 sec 2024-03-28T19:12:52.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4179.308063 secs 2024-03-28T19:12:52.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4179.308063 secs 2024-03-28T19:12:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4179.308063 secs 2024-03-28T19:12:52.520 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4825 sec 2024-03-28T19:12:53.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:52 smithi071 ceph-mon[36597]: pgmap v2665: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:53.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3470.733606 secs 2024-03-28T19:12:53.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:52 smithi151 ceph-mon[36481]: pgmap v2665: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:53.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3470.733606 secs 2024-03-28T19:12:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:52 smithi045 ceph-mon[31773]: pgmap v2665: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3470.733606 secs 2024-03-28T19:12:53.520 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4826 sec 2024-03-28T19:12:54.520 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4827 sec 2024-03-28T19:12:55.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:54 smithi071 ceph-mon[36597]: pgmap v2666: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:55.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:54 smithi151 ceph-mon[36481]: pgmap v2666: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:55.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:54 smithi045 ceph-mon[31773]: pgmap v2666: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:55.520 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4828 sec 2024-03-28T19:12:56.521 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4829 sec 2024-03-28T19:12:57.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:56 smithi071 ceph-mon[36597]: pgmap v2667: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4184.308150 secs 2024-03-28T19:12:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:56 smithi151 ceph-mon[36481]: pgmap v2667: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4184.308150 secs 2024-03-28T19:12:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:56 smithi045 ceph-mon[31773]: pgmap v2667: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:12:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4184.308150 secs 2024-03-28T19:12:57.521 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4830 sec 2024-03-28T19:12:58.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3475.733664 secs 2024-03-28T19:12:58.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3475.733664 secs 2024-03-28T19:12:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3475.733664 secs 2024-03-28T19:12:58.521 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4831 sec 2024-03-28T19:12:59.053 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:12:58 smithi045 ceph-mon[31773]: pgmap v2668: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:59.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:12:58 smithi071 ceph-mon[36597]: pgmap v2668: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:59.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:12:58 smithi151 ceph-mon[36481]: pgmap v2668: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:12:59.521 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4832 sec 2024-03-28T19:13:00.521 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4833 sec 2024-03-28T19:13:01.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:00 smithi071 ceph-mon[36597]: pgmap v2669: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:01.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:00 smithi151 ceph-mon[36481]: pgmap v2669: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:01.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:00 smithi045 ceph-mon[31773]: pgmap v2669: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:01.522 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4834 sec 2024-03-28T19:13:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4189.308238 secs 2024-03-28T19:13:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4189.308238 secs 2024-03-28T19:13:02.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4189.308238 secs 2024-03-28T19:13:02.521 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4835 sec 2024-03-28T19:13:02.801 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:13:02 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:13:02.537+0000 7f52f3200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:13:02.801 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:13:02 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:13:02.537+0000 7f52f3200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:13:02.802 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:13:02 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:13:02.537+0000 7f52f3200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:13:02.802 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:13:02 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:13:02.537+0000 7f52f3200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:13:02.802 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:13:02 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:13:02.537+0000 7f52f3200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:13:03.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:02 smithi071 ceph-mon[36597]: pgmap v2670: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:03.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3480.733752 secs 2024-03-28T19:13:03.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:02 smithi151 ceph-mon[36481]: pgmap v2670: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:03.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3480.733752 secs 2024-03-28T19:13:03.151 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:02 smithi045 ceph-mon[31773]: pgmap v2670: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:03.151 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3480.733752 secs 2024-03-28T19:13:03.522 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4836 sec 2024-03-28T19:13:04.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:03 smithi151 ceph-mon[36481]: pgmap v2671: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:04.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:03 smithi045 ceph-mon[31773]: pgmap v2671: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:04.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:03 smithi071 ceph-mon[36597]: pgmap v2671: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:04.522 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4837 sec 2024-03-28T19:13:05.522 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4838 sec 2024-03-28T19:13:06.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:06 smithi071 ceph-mon[36597]: pgmap v2672: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:06.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:06 smithi151 ceph-mon[36481]: pgmap v2672: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:06.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:06 smithi045 ceph-mon[31773]: pgmap v2672: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:06.522 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4839 sec 2024-03-28T19:13:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:07 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4194.308323 secs 2024-03-28T19:13:07.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:07 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4194.308323 secs 2024-03-28T19:13:07.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:07 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4194.308323 secs 2024-03-28T19:13:07.522 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4840 sec 2024-03-28T19:13:08.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:08 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3485.733840 secs 2024-03-28T19:13:08.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:08 smithi071 ceph-mon[36597]: pgmap v2673: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 341 B/s wr, 16 op/s 2024-03-28T19:13:08.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:08 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3485.733840 secs 2024-03-28T19:13:08.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:08 smithi151 ceph-mon[36481]: pgmap v2673: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 341 B/s wr, 16 op/s 2024-03-28T19:13:08.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:08 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3485.733840 secs 2024-03-28T19:13:08.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:08 smithi045 ceph-mon[31773]: pgmap v2673: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 341 B/s wr, 16 op/s 2024-03-28T19:13:08.522 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4841 sec 2024-03-28T19:13:09.523 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4842 sec 2024-03-28T19:13:10.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:10 smithi071 ceph-mon[36597]: pgmap v2674: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 341 B/s wr, 15 op/s 2024-03-28T19:13:10.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:10 smithi151 ceph-mon[36481]: pgmap v2674: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 341 B/s wr, 15 op/s 2024-03-28T19:13:10.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:10 smithi045 ceph-mon[31773]: pgmap v2674: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 341 B/s wr, 15 op/s 2024-03-28T19:13:10.523 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4843 sec 2024-03-28T19:13:11.523 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4844 sec 2024-03-28T19:13:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:12 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4199.308386 secs 2024-03-28T19:13:12.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:12 smithi071 ceph-mon[36597]: pgmap v2675: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 341 B/s wr, 16 op/s 2024-03-28T19:13:12.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:12 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4199.308386 secs 2024-03-28T19:13:12.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:12 smithi151 ceph-mon[36481]: pgmap v2675: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 341 B/s wr, 16 op/s 2024-03-28T19:13:12.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:12 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4199.308386 secs 2024-03-28T19:13:12.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:12 smithi045 ceph-mon[31773]: pgmap v2675: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 341 B/s wr, 16 op/s 2024-03-28T19:13:12.523 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4845 sec 2024-03-28T19:13:13.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:13 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3490.733900 secs 2024-03-28T19:13:13.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:13 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3490.733900 secs 2024-03-28T19:13:13.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:13 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3490.733900 secs 2024-03-28T19:13:13.523 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4846 sec 2024-03-28T19:13:14.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:14 smithi151 ceph-mon[36481]: pgmap v2676: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 341 B/s wr, 15 op/s 2024-03-28T19:13:14.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:14 smithi045 ceph-mon[31773]: pgmap v2676: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 341 B/s wr, 15 op/s 2024-03-28T19:13:14.523 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4847 sec 2024-03-28T19:13:14.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:14 smithi071 ceph-mon[36597]: pgmap v2676: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 341 B/s wr, 15 op/s 2024-03-28T19:13:15.523 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4848 sec 2024-03-28T19:13:16.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:16 smithi151 ceph-mon[36481]: pgmap v2677: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 341 B/s wr, 15 op/s 2024-03-28T19:13:16.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:16 smithi045 ceph-mon[31773]: pgmap v2677: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 341 B/s wr, 15 op/s 2024-03-28T19:13:16.524 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4849 sec 2024-03-28T19:13:16.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:16 smithi071 ceph-mon[36597]: pgmap v2677: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 341 B/s wr, 15 op/s 2024-03-28T19:13:17.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:17 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4204.308445 secs 2024-03-28T19:13:17.460 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:17 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4204.308445 secs 2024-03-28T19:13:17.524 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4850 sec 2024-03-28T19:13:17.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:17 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4204.308445 secs 2024-03-28T19:13:18.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:18 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3495.733987 secs 2024-03-28T19:13:18.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:18 smithi045 ceph-mon[31773]: pgmap v2678: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 341 B/s wr, 16 op/s 2024-03-28T19:13:18.524 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4851 sec 2024-03-28T19:13:18.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:18 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3495.733987 secs 2024-03-28T19:13:18.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:18 smithi071 ceph-mon[36597]: pgmap v2678: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 341 B/s wr, 16 op/s 2024-03-28T19:13:18.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:18 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3495.733987 secs 2024-03-28T19:13:18.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:18 smithi151 ceph-mon[36481]: pgmap v2678: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 341 B/s wr, 16 op/s 2024-03-28T19:13:19.524 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4852 sec 2024-03-28T19:13:20.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:20 smithi045 ceph-mon[31773]: pgmap v2679: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:20.524 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4853 sec 2024-03-28T19:13:20.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:20 smithi071 ceph-mon[36597]: pgmap v2679: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:20.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:20 smithi151 ceph-mon[36481]: pgmap v2679: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:21.524 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4854 sec 2024-03-28T19:13:22.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:22 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4209.308459 secs 2024-03-28T19:13:22.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:22 smithi045 ceph-mon[31773]: pgmap v2680: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:22.524 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4855 sec 2024-03-28T19:13:22.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:22 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4209.308459 secs 2024-03-28T19:13:22.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:22 smithi071 ceph-mon[36597]: pgmap v2680: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:22.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:22 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4209.308459 secs 2024-03-28T19:13:22.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:22 smithi151 ceph-mon[36481]: pgmap v2680: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:23.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:23 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3500.734048 secs 2024-03-28T19:13:23.524 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4856 sec 2024-03-28T19:13:23.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:23 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3500.734048 secs 2024-03-28T19:13:23.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:23 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3500.734048 secs 2024-03-28T19:13:24.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:24 smithi045 ceph-mon[31773]: pgmap v2681: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:24.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:24 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:13:24.524 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4857 sec 2024-03-28T19:13:24.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:24 smithi071 ceph-mon[36597]: pgmap v2681: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:24.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:24 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:13:24.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:24 smithi151 ceph-mon[36481]: pgmap v2681: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:24.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:24 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:13:25.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:25 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:13:25.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:25 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:13:25.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:25 smithi045 ceph-mon[31773]: pgmap v2682: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:25.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:25 smithi045 ceph-mon[31773]: pgmap v2683: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:13:25.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:25 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:13:25.525 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4858 sec 2024-03-28T19:13:25.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:25 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:13:25.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:25 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:13:25.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:25 smithi071 ceph-mon[36597]: pgmap v2682: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:25.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:25 smithi071 ceph-mon[36597]: pgmap v2683: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:13:25.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:25 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:13:25.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:25 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:13:25.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:25 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:13:25.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:25 smithi151 ceph-mon[36481]: pgmap v2682: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:25.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:25 smithi151 ceph-mon[36481]: pgmap v2683: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:13:25.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:25 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:13:26.525 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4859 sec 2024-03-28T19:13:26.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4214.308519 secs 2024-03-28T19:13:26.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4214.308519 secs 2024-03-28T19:13:26.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4214.308519 secs 2024-03-28T19:13:27.525 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4860 sec 2024-03-28T19:13:27.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:27 smithi045 ceph-mon[31773]: pgmap v2684: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:13:27.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3505.734109 secs 2024-03-28T19:13:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:27 smithi071 ceph-mon[36597]: pgmap v2684: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:13:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3505.734109 secs 2024-03-28T19:13:27.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:27 smithi151 ceph-mon[36481]: pgmap v2684: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:13:27.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3505.734109 secs 2024-03-28T19:13:28.525 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4861 sec 2024-03-28T19:13:29.526 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4862 sec 2024-03-28T19:13:29.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:29 smithi045 ceph-mon[31773]: pgmap v2685: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:13:30.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:29 smithi071 ceph-mon[36597]: pgmap v2685: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:13:30.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:29 smithi151 ceph-mon[36481]: pgmap v2685: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:13:30.525 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4863 sec 2024-03-28T19:13:31.526 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4864 sec 2024-03-28T19:13:31.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:31 smithi045 ceph-mon[31773]: pgmap v2686: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:13:31.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4219.308580 secs 2024-03-28T19:13:32.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:31 smithi071 ceph-mon[36597]: pgmap v2686: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:13:32.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4219.308580 secs 2024-03-28T19:13:32.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:31 smithi151 ceph-mon[36481]: pgmap v2686: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:13:32.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4219.308580 secs 2024-03-28T19:13:32.526 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4865 sec 2024-03-28T19:13:32.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3510.734192 secs 2024-03-28T19:13:33.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3510.734192 secs 2024-03-28T19:13:33.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3510.734192 secs 2024-03-28T19:13:33.526 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4866 sec 2024-03-28T19:13:33.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:33 smithi045 ceph-mon[31773]: pgmap v2687: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:13:34.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:33 smithi071 ceph-mon[36597]: pgmap v2687: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:13:34.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:33 smithi151 ceph-mon[36481]: pgmap v2687: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:13:34.526 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4867 sec 2024-03-28T19:13:35.526 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4868 sec 2024-03-28T19:13:35.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:35 smithi045 ceph-mon[31773]: pgmap v2688: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:13:36.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:35 smithi071 ceph-mon[36597]: pgmap v2688: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:13:36.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:35 smithi151 ceph-mon[36481]: pgmap v2688: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:13:36.526 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4869 sec 2024-03-28T19:13:36.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4224.308642 secs 2024-03-28T19:13:37.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4224.308642 secs 2024-03-28T19:13:37.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4224.308642 secs 2024-03-28T19:13:37.526 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4870 sec 2024-03-28T19:13:37.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:37 smithi045 ceph-mon[31773]: pgmap v2689: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:37.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3515.734205 secs 2024-03-28T19:13:38.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:37 smithi071 ceph-mon[36597]: pgmap v2689: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:38.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3515.734205 secs 2024-03-28T19:13:38.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:37 smithi151 ceph-mon[36481]: pgmap v2689: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:38.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3515.734205 secs 2024-03-28T19:13:38.527 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4871 sec 2024-03-28T19:13:39.527 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4872 sec 2024-03-28T19:13:40.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:39 smithi071 ceph-mon[36597]: pgmap v2690: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:40.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:39 smithi151 ceph-mon[36481]: pgmap v2690: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:40.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:39 smithi045 ceph-mon[31773]: pgmap v2690: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:40.527 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4873 sec 2024-03-28T19:13:41.527 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4874 sec 2024-03-28T19:13:42.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:41 smithi071 ceph-mon[36597]: pgmap v2691: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4229.308704 secs 2024-03-28T19:13:42.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:41 smithi151 ceph-mon[36481]: pgmap v2691: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:42.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4229.308704 secs 2024-03-28T19:13:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:41 smithi045 ceph-mon[31773]: pgmap v2691: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4229.308704 secs 2024-03-28T19:13:42.527 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4875 sec 2024-03-28T19:13:42.973 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3520.734264 secs 2024-03-28T19:13:43.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3520.734264 secs 2024-03-28T19:13:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3520.734264 secs 2024-03-28T19:13:43.527 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4876 sec 2024-03-28T19:13:44.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:43 smithi071 ceph-mon[36597]: pgmap v2692: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:44.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:43 smithi151 ceph-mon[36481]: pgmap v2692: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:44.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:43 smithi045 ceph-mon[31773]: pgmap v2692: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:44.527 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4877 sec 2024-03-28T19:13:45.528 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4878 sec 2024-03-28T19:13:46.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:45 smithi071 ceph-mon[36597]: pgmap v2693: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:46.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:45 smithi151 ceph-mon[36481]: pgmap v2693: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:46.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:45 smithi045 ceph-mon[31773]: pgmap v2693: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:46.528 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4879 sec 2024-03-28T19:13:47.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4234.308787 secs 2024-03-28T19:13:47.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4234.308787 secs 2024-03-28T19:13:47.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4234.308787 secs 2024-03-28T19:13:47.528 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4880 sec 2024-03-28T19:13:47.960 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:47 smithi071 ceph-mon[36597]: pgmap v2694: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:47.960 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3525.734322 secs 2024-03-28T19:13:48.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:47 smithi151 ceph-mon[36481]: pgmap v2694: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:48.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3525.734322 secs 2024-03-28T19:13:48.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:47 smithi045 ceph-mon[31773]: pgmap v2694: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3525.734322 secs 2024-03-28T19:13:48.528 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4881 sec 2024-03-28T19:13:49.528 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4882 sec 2024-03-28T19:13:50.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:49 smithi071 ceph-mon[36597]: pgmap v2695: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:50.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:49 smithi151 ceph-mon[36481]: pgmap v2695: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:50.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:49 smithi045 ceph-mon[31773]: pgmap v2695: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:50.528 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4883 sec 2024-03-28T19:13:51.529 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4884 sec 2024-03-28T19:13:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:51 smithi071 ceph-mon[36597]: pgmap v2696: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4239.308874 secs 2024-03-28T19:13:52.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:51 smithi151 ceph-mon[36481]: pgmap v2696: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:52.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4239.308874 secs 2024-03-28T19:13:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:51 smithi045 ceph-mon[31773]: pgmap v2696: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4239.308874 secs 2024-03-28T19:13:52.528 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4885 sec 2024-03-28T19:13:53.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3530.734363 secs 2024-03-28T19:13:53.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3530.734363 secs 2024-03-28T19:13:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3530.734363 secs 2024-03-28T19:13:53.528 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4886 sec 2024-03-28T19:13:54.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:53 smithi071 ceph-mon[36597]: pgmap v2697: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:54.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:53 smithi151 ceph-mon[36481]: pgmap v2697: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:54.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:53 smithi045 ceph-mon[31773]: pgmap v2697: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:54.529 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4887 sec 2024-03-28T19:13:55.529 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4888 sec 2024-03-28T19:13:56.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:55 smithi071 ceph-mon[36597]: pgmap v2698: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:56.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:55 smithi151 ceph-mon[36481]: pgmap v2698: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:56.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:55 smithi045 ceph-mon[31773]: pgmap v2698: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:13:56.529 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4889 sec 2024-03-28T19:13:57.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4244.308963 secs 2024-03-28T19:13:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4244.308963 secs 2024-03-28T19:13:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4244.308963 secs 2024-03-28T19:13:57.529 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4890 sec 2024-03-28T19:13:58.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:57 smithi071 ceph-mon[36597]: pgmap v2699: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3535.734446 secs 2024-03-28T19:13:58.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:57 smithi151 ceph-mon[36481]: pgmap v2699: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:58.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3535.734446 secs 2024-03-28T19:13:58.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:57 smithi045 ceph-mon[31773]: pgmap v2699: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:13:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3535.734446 secs 2024-03-28T19:13:58.529 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4891 sec 2024-03-28T19:13:59.530 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4892 sec 2024-03-28T19:14:00.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:13:59 smithi071 ceph-mon[36597]: pgmap v2700: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:00.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:13:59 smithi151 ceph-mon[36481]: pgmap v2700: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:00.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:13:59 smithi045 ceph-mon[31773]: pgmap v2700: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:00.530 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4893 sec 2024-03-28T19:14:01.530 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4894 sec 2024-03-28T19:14:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:01 smithi071 ceph-mon[36597]: pgmap v2701: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4249.308998 secs 2024-03-28T19:14:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:01 smithi151 ceph-mon[36481]: pgmap v2701: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4249.308998 secs 2024-03-28T19:14:02.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:01 smithi045 ceph-mon[31773]: pgmap v2701: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:02.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4249.308998 secs 2024-03-28T19:14:02.530 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4895 sec 2024-03-28T19:14:03.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3540.734530 secs 2024-03-28T19:14:03.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3540.734530 secs 2024-03-28T19:14:03.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3540.734530 secs 2024-03-28T19:14:03.530 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4896 sec 2024-03-28T19:14:04.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:03 smithi071 ceph-mon[36597]: pgmap v2702: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:04.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:03 smithi151 ceph-mon[36481]: pgmap v2702: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:04.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:03 smithi045 ceph-mon[31773]: pgmap v2702: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:04.530 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4897 sec 2024-03-28T19:14:05.531 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4898 sec 2024-03-28T19:14:06.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:05 smithi071 ceph-mon[36597]: pgmap v2703: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:06.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:05 smithi151 ceph-mon[36481]: pgmap v2703: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:06.127 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:05 smithi045 ceph-mon[31773]: pgmap v2703: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:06.530 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4899 sec 2024-03-28T19:14:07.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4254.309058 secs 2024-03-28T19:14:07.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4254.309058 secs 2024-03-28T19:14:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4254.309058 secs 2024-03-28T19:14:07.531 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4900 sec 2024-03-28T19:14:08.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:07 smithi071 ceph-mon[36597]: pgmap v2704: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T19:14:08.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3545.734586 secs 2024-03-28T19:14:08.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:07 smithi151 ceph-mon[36481]: pgmap v2704: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T19:14:08.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3545.734586 secs 2024-03-28T19:14:08.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:07 smithi045 ceph-mon[31773]: pgmap v2704: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T19:14:08.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3545.734586 secs 2024-03-28T19:14:08.531 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4901 sec 2024-03-28T19:14:09.531 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4902 sec 2024-03-28T19:14:10.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:09 smithi071 ceph-mon[36597]: pgmap v2705: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T19:14:10.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:09 smithi151 ceph-mon[36481]: pgmap v2705: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T19:14:10.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:09 smithi045 ceph-mon[31773]: pgmap v2705: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T19:14:10.531 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4903 sec 2024-03-28T19:14:11.531 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4904 sec 2024-03-28T19:14:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:11 smithi071 ceph-mon[36597]: pgmap v2706: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T19:14:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4259.309147 secs 2024-03-28T19:14:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:11 smithi151 ceph-mon[36481]: pgmap v2706: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T19:14:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4259.309147 secs 2024-03-28T19:14:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:11 smithi045 ceph-mon[31773]: pgmap v2706: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T19:14:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4259.309147 secs 2024-03-28T19:14:12.531 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4905 sec 2024-03-28T19:14:13.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3550.734647 secs 2024-03-28T19:14:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3550.734647 secs 2024-03-28T19:14:13.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3550.734647 secs 2024-03-28T19:14:13.531 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4906 sec 2024-03-28T19:14:14.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:13 smithi151 ceph-mon[36481]: pgmap v2707: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T19:14:14.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:13 smithi045 ceph-mon[31773]: pgmap v2707: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T19:14:14.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:13 smithi071 ceph-mon[36597]: pgmap v2707: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T19:14:14.532 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4907 sec 2024-03-28T19:14:15.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:14 smithi045 ceph-mon[31773]: pgmap v2708: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T19:14:15.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:14 smithi071 ceph-mon[36597]: pgmap v2708: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T19:14:15.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:14 smithi151 ceph-mon[36481]: pgmap v2708: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T19:14:15.531 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4908 sec 2024-03-28T19:14:16.532 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4909 sec 2024-03-28T19:14:16.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4264.309207 secs 2024-03-28T19:14:16.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4264.309207 secs 2024-03-28T19:14:16.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4264.309207 secs 2024-03-28T19:14:17.532 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4910 sec 2024-03-28T19:14:17.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:17 smithi045 ceph-mon[31773]: pgmap v2709: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T19:14:17.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3555.734706 secs 2024-03-28T19:14:17.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:17 smithi071 ceph-mon[36597]: pgmap v2709: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T19:14:17.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3555.734706 secs 2024-03-28T19:14:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:17 smithi151 ceph-mon[36481]: pgmap v2709: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T19:14:17.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3555.734706 secs 2024-03-28T19:14:18.532 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4911 sec 2024-03-28T19:14:19.532 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4912 sec 2024-03-28T19:14:19.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:19 smithi045 ceph-mon[31773]: pgmap v2710: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:20.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:19 smithi071 ceph-mon[36597]: pgmap v2710: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:20.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:19 smithi151 ceph-mon[36481]: pgmap v2710: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:20.532 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4913 sec 2024-03-28T19:14:21.532 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4914 sec 2024-03-28T19:14:21.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:21 smithi045 ceph-mon[31773]: pgmap v2711: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:21.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4269.309306 secs 2024-03-28T19:14:22.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:21 smithi071 ceph-mon[36597]: pgmap v2711: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:22.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4269.309306 secs 2024-03-28T19:14:22.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:21 smithi151 ceph-mon[36481]: pgmap v2711: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:22.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4269.309306 secs 2024-03-28T19:14:22.532 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4915 sec 2024-03-28T19:14:22.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3560.734768 secs 2024-03-28T19:14:23.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3560.734768 secs 2024-03-28T19:14:23.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3560.734768 secs 2024-03-28T19:14:23.533 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4916 sec 2024-03-28T19:14:24.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:23 smithi071 ceph-mon[36597]: pgmap v2712: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:24.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:23 smithi151 ceph-mon[36481]: pgmap v2712: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:24.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:23 smithi045 ceph-mon[31773]: pgmap v2712: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:24.533 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4917 sec 2024-03-28T19:14:25.533 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4918 sec 2024-03-28T19:14:26.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:25 smithi071 ceph-mon[36597]: pgmap v2713: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:26.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:25 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:14:26.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:25 smithi151 ceph-mon[36481]: pgmap v2713: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:26.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:25 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:14:26.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:25 smithi045 ceph-mon[31773]: pgmap v2713: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:26.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:25 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:14:26.533 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4919 sec 2024-03-28T19:14:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4274.309379 secs 2024-03-28T19:14:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:26 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:14:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:26 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:14:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:26 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:14:27.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4274.309379 secs 2024-03-28T19:14:27.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:26 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:14:27.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:26 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:14:27.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:26 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:14:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4274.309379 secs 2024-03-28T19:14:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:26 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:14:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:26 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:14:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:26 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:14:27.533 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4920 sec 2024-03-28T19:14:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:27 smithi071 ceph-mon[36597]: pgmap v2714: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:14:28.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:27 smithi071 ceph-mon[36597]: pgmap v2715: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:28.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3565.734830 secs 2024-03-28T19:14:28.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:27 smithi151 ceph-mon[36481]: pgmap v2714: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:14:28.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:27 smithi151 ceph-mon[36481]: pgmap v2715: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:28.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3565.734830 secs 2024-03-28T19:14:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:27 smithi045 ceph-mon[31773]: pgmap v2714: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:14:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:27 smithi045 ceph-mon[31773]: pgmap v2715: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3565.734830 secs 2024-03-28T19:14:28.533 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4921 sec 2024-03-28T19:14:29.533 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4922 sec 2024-03-28T19:14:30.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:29 smithi071 ceph-mon[36597]: pgmap v2716: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:30.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:29 smithi151 ceph-mon[36481]: pgmap v2716: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:30.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:29 smithi045 ceph-mon[31773]: pgmap v2716: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:30.533 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4923 sec 2024-03-28T19:14:31.534 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4924 sec 2024-03-28T19:14:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:31 smithi071 ceph-mon[36597]: pgmap v2717: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:32.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4279.309466 secs 2024-03-28T19:14:32.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:31 smithi151 ceph-mon[36481]: pgmap v2717: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:32.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4279.309466 secs 2024-03-28T19:14:32.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:31 smithi045 ceph-mon[31773]: pgmap v2717: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:32.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4279.309466 secs 2024-03-28T19:14:32.533 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4925 sec 2024-03-28T19:14:33.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3570.734899 secs 2024-03-28T19:14:33.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3570.734899 secs 2024-03-28T19:14:33.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3570.734899 secs 2024-03-28T19:14:33.533 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4926 sec 2024-03-28T19:14:34.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:33 smithi071 ceph-mon[36597]: pgmap v2718: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:34.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:33 smithi151 ceph-mon[36481]: pgmap v2718: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:34.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:33 smithi045 ceph-mon[31773]: pgmap v2718: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:34.534 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4927 sec 2024-03-28T19:14:35.534 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4928 sec 2024-03-28T19:14:36.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:35 smithi071 ceph-mon[36597]: pgmap v2719: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:36.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:35 smithi151 ceph-mon[36481]: pgmap v2719: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:36.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:35 smithi045 ceph-mon[31773]: pgmap v2719: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:14:36.534 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4929 sec 2024-03-28T19:14:37.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4284.309526 secs 2024-03-28T19:14:37.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4284.309526 secs 2024-03-28T19:14:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4284.309526 secs 2024-03-28T19:14:37.534 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4930 sec 2024-03-28T19:14:38.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:37 smithi071 ceph-mon[36597]: pgmap v2720: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:14:38.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3575.734956 secs 2024-03-28T19:14:38.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:37 smithi151 ceph-mon[36481]: pgmap v2720: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:14:38.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3575.734956 secs 2024-03-28T19:14:38.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:37 smithi045 ceph-mon[31773]: pgmap v2720: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:14:38.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3575.734956 secs 2024-03-28T19:14:38.534 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4931 sec 2024-03-28T19:14:39.534 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4932 sec 2024-03-28T19:14:40.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:39 smithi071 ceph-mon[36597]: pgmap v2721: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:40.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:39 smithi151 ceph-mon[36481]: pgmap v2721: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:40.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:39 smithi045 ceph-mon[31773]: pgmap v2721: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:40.534 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4933 sec 2024-03-28T19:14:41.535 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4934 sec 2024-03-28T19:14:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:41 smithi071 ceph-mon[36597]: pgmap v2722: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4289.309538 secs 2024-03-28T19:14:42.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:41 smithi151 ceph-mon[36481]: pgmap v2722: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:42.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4289.309538 secs 2024-03-28T19:14:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:41 smithi045 ceph-mon[31773]: pgmap v2722: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4289.309538 secs 2024-03-28T19:14:42.535 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4935 sec 2024-03-28T19:14:43.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3580.734998 secs 2024-03-28T19:14:43.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3580.734998 secs 2024-03-28T19:14:43.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3580.734998 secs 2024-03-28T19:14:43.535 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4936 sec 2024-03-28T19:14:44.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:43 smithi071 ceph-mon[36597]: pgmap v2723: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:44.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:43 smithi151 ceph-mon[36481]: pgmap v2723: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:44.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:43 smithi045 ceph-mon[31773]: pgmap v2723: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:44.536 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4937 sec 2024-03-28T19:14:45.536 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4938 sec 2024-03-28T19:14:46.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:45 smithi071 ceph-mon[36597]: pgmap v2724: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:46.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:45 smithi151 ceph-mon[36481]: pgmap v2724: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:46.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:45 smithi045 ceph-mon[31773]: pgmap v2724: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:46.535 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4939 sec 2024-03-28T19:14:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4294.309622 secs 2024-03-28T19:14:47.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4294.309622 secs 2024-03-28T19:14:47.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4294.309622 secs 2024-03-28T19:14:47.536 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4940 sec 2024-03-28T19:14:48.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:47 smithi071 ceph-mon[36597]: pgmap v2725: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:48.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3585.735055 secs 2024-03-28T19:14:48.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:47 smithi151 ceph-mon[36481]: pgmap v2725: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:48.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3585.735055 secs 2024-03-28T19:14:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:47 smithi045 ceph-mon[31773]: pgmap v2725: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3585.735055 secs 2024-03-28T19:14:48.536 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4941 sec 2024-03-28T19:14:49.536 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4942 sec 2024-03-28T19:14:50.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:49 smithi071 ceph-mon[36597]: pgmap v2726: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:50.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:49 smithi151 ceph-mon[36481]: pgmap v2726: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:50.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:49 smithi045 ceph-mon[31773]: pgmap v2726: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:50.536 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4943 sec 2024-03-28T19:14:51.537 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4944 sec 2024-03-28T19:14:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:51 smithi071 ceph-mon[36597]: pgmap v2727: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:52.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4299.309705 secs 2024-03-28T19:14:52.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:51 smithi151 ceph-mon[36481]: pgmap v2727: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:52.075 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4299.309705 secs 2024-03-28T19:14:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:51 smithi045 ceph-mon[31773]: pgmap v2727: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4299.309705 secs 2024-03-28T19:14:52.536 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4945 sec 2024-03-28T19:14:53.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3590.735093 secs 2024-03-28T19:14:53.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3590.735093 secs 2024-03-28T19:14:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3590.735093 secs 2024-03-28T19:14:53.537 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4946 sec 2024-03-28T19:14:54.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:53 smithi071 ceph-mon[36597]: pgmap v2728: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:54.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:53 smithi151 ceph-mon[36481]: pgmap v2728: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:54.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:53 smithi045 ceph-mon[31773]: pgmap v2728: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:54.537 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4947 sec 2024-03-28T19:14:55.537 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4948 sec 2024-03-28T19:14:56.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:55 smithi071 ceph-mon[36597]: pgmap v2729: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:56.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:55 smithi151 ceph-mon[36481]: pgmap v2729: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:56.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:55 smithi045 ceph-mon[31773]: pgmap v2729: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:14:56.537 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4949 sec 2024-03-28T19:14:57.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4304.309765 secs 2024-03-28T19:14:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4304.309765 secs 2024-03-28T19:14:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4304.309765 secs 2024-03-28T19:14:57.537 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4950 sec 2024-03-28T19:14:58.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:57 smithi071 ceph-mon[36597]: pgmap v2730: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:58.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3595.735176 secs 2024-03-28T19:14:58.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:57 smithi151 ceph-mon[36481]: pgmap v2730: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:58.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3595.735176 secs 2024-03-28T19:14:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:57 smithi045 ceph-mon[31773]: pgmap v2730: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:14:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3595.735176 secs 2024-03-28T19:14:58.537 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4951 sec 2024-03-28T19:14:59.538 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4952 sec 2024-03-28T19:15:00.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:14:59 smithi071 ceph-mon[36597]: pgmap v2731: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:00.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:14:59 smithi151 ceph-mon[36481]: pgmap v2731: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:00.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:14:59 smithi045 ceph-mon[31773]: pgmap v2731: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:00.538 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4953 sec 2024-03-28T19:15:01.538 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4954 sec 2024-03-28T19:15:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:01 smithi151 ceph-mon[36481]: pgmap v2732: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4309.309826 secs 2024-03-28T19:15:02.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:01 smithi045 ceph-mon[31773]: pgmap v2732: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:02.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4309.309826 secs 2024-03-28T19:15:02.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:01 smithi071 ceph-mon[36597]: pgmap v2732: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:02.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4309.309826 secs 2024-03-28T19:15:02.538 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4955 sec 2024-03-28T19:15:03.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3600.735192 secs 2024-03-28T19:15:03.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3600.735192 secs 2024-03-28T19:15:03.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3600.735192 secs 2024-03-28T19:15:03.538 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4956 sec 2024-03-28T19:15:04.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:03 smithi045 ceph-mon[31773]: pgmap v2733: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:04.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:03 smithi071 ceph-mon[36597]: pgmap v2733: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:04.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:03 smithi151 ceph-mon[36481]: pgmap v2733: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:04.538 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4957 sec 2024-03-28T19:15:05.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:04 smithi045 ceph-mon[31773]: pgmap v2734: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:05.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:04 smithi071 ceph-mon[36597]: pgmap v2734: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:05.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:04 smithi151 ceph-mon[36481]: pgmap v2734: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:05.538 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4958 sec 2024-03-28T19:15:06.539 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4959 sec 2024-03-28T19:15:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4314.309885 secs 2024-03-28T19:15:06.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4314.309885 secs 2024-03-28T19:15:06.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4314.309885 secs 2024-03-28T19:15:07.539 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4960 sec 2024-03-28T19:15:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:07 smithi045 ceph-mon[31773]: pgmap v2735: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:07.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3605.735253 secs 2024-03-28T19:15:07.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:07 smithi071 ceph-mon[36597]: pgmap v2735: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3605.735253 secs 2024-03-28T19:15:07.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:07 smithi151 ceph-mon[36481]: pgmap v2735: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:07.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3605.735253 secs 2024-03-28T19:15:08.539 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4961 sec 2024-03-28T19:15:09.539 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4962 sec 2024-03-28T19:15:09.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:09 smithi045 ceph-mon[31773]: pgmap v2736: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:09.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:09 smithi071 ceph-mon[36597]: pgmap v2736: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:09.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:09 smithi151 ceph-mon[36481]: pgmap v2736: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:10.539 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4963 sec 2024-03-28T19:15:11.539 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4964 sec 2024-03-28T19:15:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:11 smithi045 ceph-mon[31773]: pgmap v2737: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4319.309971 secs 2024-03-28T19:15:11.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:11 smithi071 ceph-mon[36597]: pgmap v2737: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4319.309971 secs 2024-03-28T19:15:11.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:11 smithi151 ceph-mon[36481]: pgmap v2737: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:11.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4319.309971 secs 2024-03-28T19:15:12.539 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4965 sec 2024-03-28T19:15:12.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3610.735342 secs 2024-03-28T19:15:12.723 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3610.735342 secs 2024-03-28T19:15:12.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3610.735342 secs 2024-03-28T19:15:13.540 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4966 sec 2024-03-28T19:15:13.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:13 smithi045 ceph-mon[31773]: pgmap v2738: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:13.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:13 smithi071 ceph-mon[36597]: pgmap v2738: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:13.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:13 smithi151 ceph-mon[36481]: pgmap v2738: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:14.540 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4967 sec 2024-03-28T19:15:15.540 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4968 sec 2024-03-28T19:15:15.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:15 smithi045 ceph-mon[31773]: pgmap v2739: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:15.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:15 smithi071 ceph-mon[36597]: pgmap v2739: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:15.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:15 smithi151 ceph-mon[36481]: pgmap v2739: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:16.540 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4969 sec 2024-03-28T19:15:16.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4324.310030 secs 2024-03-28T19:15:16.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4324.310030 secs 2024-03-28T19:15:16.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4324.310030 secs 2024-03-28T19:15:17.540 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4970 sec 2024-03-28T19:15:17.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:17 smithi045 ceph-mon[31773]: pgmap v2740: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:17.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3615.735430 secs 2024-03-28T19:15:17.710 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:17 smithi071 ceph-mon[36597]: pgmap v2740: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:17.710 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3615.735430 secs 2024-03-28T19:15:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:17 smithi151 ceph-mon[36481]: pgmap v2740: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3615.735430 secs 2024-03-28T19:15:18.540 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4971 sec 2024-03-28T19:15:19.541 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4972 sec 2024-03-28T19:15:19.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:19 smithi045 ceph-mon[31773]: pgmap v2741: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:19.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:19 smithi071 ceph-mon[36597]: pgmap v2741: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:19.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:19 smithi151 ceph-mon[36481]: pgmap v2741: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:20.541 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4973 sec 2024-03-28T19:15:21.541 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4974 sec 2024-03-28T19:15:21.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:21 smithi045 ceph-mon[31773]: pgmap v2742: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:21.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4329.310089 secs 2024-03-28T19:15:21.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:21 smithi071 ceph-mon[36597]: pgmap v2742: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4329.310089 secs 2024-03-28T19:15:21.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:21 smithi151 ceph-mon[36481]: pgmap v2742: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:21.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4329.310089 secs 2024-03-28T19:15:22.541 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4975 sec 2024-03-28T19:15:22.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3620.735491 secs 2024-03-28T19:15:22.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3620.735491 secs 2024-03-28T19:15:22.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3620.735491 secs 2024-03-28T19:15:23.541 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4976 sec 2024-03-28T19:15:23.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:23 smithi045 ceph-mon[31773]: pgmap v2743: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:23.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:23 smithi071 ceph-mon[36597]: pgmap v2743: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:23.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:23 smithi151 ceph-mon[36481]: pgmap v2743: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:24.541 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4977 sec 2024-03-28T19:15:25.437 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:25 smithi045 ceph-mon[31773]: pgmap v2744: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:25.541 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4978 sec 2024-03-28T19:15:25.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:25 smithi071 ceph-mon[36597]: pgmap v2744: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:25.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:25 smithi151 ceph-mon[36481]: pgmap v2744: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:26.541 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4979 sec 2024-03-28T19:15:26.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4334.310175 secs 2024-03-28T19:15:26.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4334.310175 secs 2024-03-28T19:15:26.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4334.310175 secs 2024-03-28T19:15:27.542 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4980 sec 2024-03-28T19:15:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:27 smithi071 ceph-mon[36597]: pgmap v2745: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:27 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:15:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3625.735507 secs 2024-03-28T19:15:27.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:27 smithi151 ceph-mon[36481]: pgmap v2745: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:27.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:27 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:15:27.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3625.735507 secs 2024-03-28T19:15:27.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:27 smithi045 ceph-mon[31773]: pgmap v2745: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:27.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:27 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:15:27.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3625.735507 secs 2024-03-28T19:15:28.542 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4981 sec 2024-03-28T19:15:29.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:29 smithi071 ceph-mon[36597]: pgmap v2746: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:29.542 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4982 sec 2024-03-28T19:15:29.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:29 smithi045 ceph-mon[31773]: pgmap v2746: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:29.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:29 smithi151 ceph-mon[36481]: pgmap v2746: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:30.542 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4983 sec 2024-03-28T19:15:31.542 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4984 sec 2024-03-28T19:15:31.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:31 smithi071 ceph-mon[36597]: pgmap v2747: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4339.310258 secs 2024-03-28T19:15:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:31 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:15:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:31 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:15:31.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:31 smithi151 ceph-mon[36481]: pgmap v2747: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:31.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4339.310258 secs 2024-03-28T19:15:31.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:31 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:15:31.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:31 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:15:31.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:31 smithi045 ceph-mon[31773]: pgmap v2747: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:31.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4339.310258 secs 2024-03-28T19:15:31.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:31 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:15:31.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:31 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:15:32.542 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4985 sec 2024-03-28T19:15:32.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3630.735567 secs 2024-03-28T19:15:32.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3630.735567 secs 2024-03-28T19:15:32.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3630.735567 secs 2024-03-28T19:15:33.542 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4986 sec 2024-03-28T19:15:33.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:33 smithi045 ceph-mon[31773]: pgmap v2748: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:33.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:33 smithi071 ceph-mon[36597]: pgmap v2748: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:33.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:33 smithi151 ceph-mon[36481]: pgmap v2748: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:34.542 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4987 sec 2024-03-28T19:15:35.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:34 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:15:35.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:34 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:15:35.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:34 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:15:35.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:34 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:15:35.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:34 smithi071 ceph-mon[36597]: pgmap v2749: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:35.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:34 smithi071 ceph-mon[36597]: pgmap v2750: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:35.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:34 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:15:35.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:34 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:15:35.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:34 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:15:35.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:34 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:15:35.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:34 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:15:35.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:34 smithi151 ceph-mon[36481]: pgmap v2749: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:34 smithi151 ceph-mon[36481]: pgmap v2750: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:35.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:34 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:15:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:34 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:15:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:34 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:15:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:34 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:15:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:34 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:15:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:34 smithi045 ceph-mon[31773]: pgmap v2749: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:34 smithi045 ceph-mon[31773]: pgmap v2750: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:35.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:34 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:15:35.542 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4988 sec 2024-03-28T19:15:36.543 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4989 sec 2024-03-28T19:15:37.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:36 smithi151 ceph-mon[36481]: pgmap v2751: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:37.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4344.310348 secs 2024-03-28T19:15:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:36 smithi045 ceph-mon[31773]: pgmap v2751: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4344.310348 secs 2024-03-28T19:15:37.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:36 smithi071 ceph-mon[36597]: pgmap v2751: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4344.310348 secs 2024-03-28T19:15:37.543 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4990 sec 2024-03-28T19:15:38.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3635.735651 secs 2024-03-28T19:15:38.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3635.735651 secs 2024-03-28T19:15:38.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3635.735651 secs 2024-03-28T19:15:38.543 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4991 sec 2024-03-28T19:15:39.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:38 smithi045 ceph-mon[31773]: pgmap v2752: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:39.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:38 smithi071 ceph-mon[36597]: pgmap v2752: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:39.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:38 smithi151 ceph-mon[36481]: pgmap v2752: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:39.543 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4992 sec 2024-03-28T19:15:40.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:39 smithi045 ceph-mon[31773]: pgmap v2753: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:15:40.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:39 smithi071 ceph-mon[36597]: pgmap v2753: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:15:40.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:39 smithi151 ceph-mon[36481]: pgmap v2753: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:15:40.543 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4993 sec 2024-03-28T19:15:41.544 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4994 sec 2024-03-28T19:15:41.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4349.310388 secs 2024-03-28T19:15:41.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4349.310388 secs 2024-03-28T19:15:41.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4349.310388 secs 2024-03-28T19:15:42.544 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4995 sec 2024-03-28T19:15:42.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3640.735739 secs 2024-03-28T19:15:42.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:42 smithi045 ceph-mon[31773]: pgmap v2754: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:42.723 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3640.735739 secs 2024-03-28T19:15:42.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:42 smithi151 ceph-mon[36481]: pgmap v2754: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:42.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3640.735739 secs 2024-03-28T19:15:42.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:42 smithi071 ceph-mon[36597]: pgmap v2754: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:15:43.544 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4996 sec 2024-03-28T19:15:44.544 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4997 sec 2024-03-28T19:15:45.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:44 smithi045 ceph-mon[31773]: pgmap v2755: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:15:45.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:44 smithi071 ceph-mon[36597]: pgmap v2755: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:15:45.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:44 smithi151 ceph-mon[36481]: pgmap v2755: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:15:45.544 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4998 sec 2024-03-28T19:15:46.544 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 4999 sec 2024-03-28T19:15:47.111 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:46 smithi071 ceph-mon[36597]: pgmap v2756: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:47.111 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4354.310477 secs 2024-03-28T19:15:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:46 smithi045 ceph-mon[31773]: pgmap v2756: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4354.310477 secs 2024-03-28T19:15:47.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:46 smithi151 ceph-mon[36481]: pgmap v2756: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:47.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4354.310477 secs 2024-03-28T19:15:47.544 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5000 sec 2024-03-28T19:15:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3645.735802 secs 2024-03-28T19:15:48.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3645.735802 secs 2024-03-28T19:15:48.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3645.735802 secs 2024-03-28T19:15:48.545 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5001 sec 2024-03-28T19:15:49.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:48 smithi045 ceph-mon[31773]: pgmap v2757: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:49.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:48 smithi071 ceph-mon[36597]: pgmap v2757: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:49.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:48 smithi151 ceph-mon[36481]: pgmap v2757: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:49.544 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5002 sec 2024-03-28T19:15:50.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:49 smithi045 ceph-mon[31773]: pgmap v2758: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:50.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:49 smithi071 ceph-mon[36597]: pgmap v2758: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:50.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:49 smithi151 ceph-mon[36481]: pgmap v2758: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:50.545 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5003 sec 2024-03-28T19:15:51.545 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5004 sec 2024-03-28T19:15:51.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4359.310536 secs 2024-03-28T19:15:51.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4359.310536 secs 2024-03-28T19:15:51.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4359.310536 secs 2024-03-28T19:15:52.545 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5005 sec 2024-03-28T19:15:52.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3650.735862 secs 2024-03-28T19:15:52.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:52 smithi045 ceph-mon[31773]: pgmap v2759: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:52.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3650.735862 secs 2024-03-28T19:15:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:52 smithi071 ceph-mon[36597]: pgmap v2759: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:52.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3650.735862 secs 2024-03-28T19:15:52.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:52 smithi151 ceph-mon[36481]: pgmap v2759: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:53.545 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5006 sec 2024-03-28T19:15:54.545 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5007 sec 2024-03-28T19:15:55.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:54 smithi045 ceph-mon[31773]: pgmap v2760: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:55.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:54 smithi071 ceph-mon[36597]: pgmap v2760: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:55.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:54 smithi151 ceph-mon[36481]: pgmap v2760: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:55.545 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5008 sec 2024-03-28T19:15:56.546 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5009 sec 2024-03-28T19:15:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:56 smithi045 ceph-mon[31773]: pgmap v2761: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4364.310599 secs 2024-03-28T19:15:57.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:56 smithi071 ceph-mon[36597]: pgmap v2761: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:57.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4364.310599 secs 2024-03-28T19:15:57.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:56 smithi151 ceph-mon[36481]: pgmap v2761: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:15:57.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4364.310599 secs 2024-03-28T19:15:57.546 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5010 sec 2024-03-28T19:15:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3655.735924 secs 2024-03-28T19:15:58.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3655.735924 secs 2024-03-28T19:15:58.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3655.735924 secs 2024-03-28T19:15:58.546 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5011 sec 2024-03-28T19:15:59.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:15:58 smithi045 ceph-mon[31773]: pgmap v2762: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:59.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:15:58 smithi071 ceph-mon[36597]: pgmap v2762: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:59.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:15:58 smithi151 ceph-mon[36481]: pgmap v2762: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:15:59.546 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5012 sec 2024-03-28T19:16:00.546 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5013 sec 2024-03-28T19:16:00.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:00 smithi071 ceph-mon[36597]: pgmap v2763: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:00.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:00 smithi151 ceph-mon[36481]: pgmap v2763: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:00.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:00 smithi045 ceph-mon[31773]: pgmap v2763: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:01.546 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5014 sec 2024-03-28T19:16:01.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4369.310663 secs 2024-03-28T19:16:01.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4369.310663 secs 2024-03-28T19:16:01.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4369.310663 secs 2024-03-28T19:16:02.546 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5015 sec 2024-03-28T19:16:02.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3660.735984 secs 2024-03-28T19:16:02.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:02 smithi045 ceph-mon[31773]: pgmap v2764: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:02.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3660.735984 secs 2024-03-28T19:16:02.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:02 smithi071 ceph-mon[36597]: pgmap v2764: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:02.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3660.735984 secs 2024-03-28T19:16:02.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:02 smithi151 ceph-mon[36481]: pgmap v2764: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:03.546 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5016 sec 2024-03-28T19:16:04.547 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5017 sec 2024-03-28T19:16:05.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:04 smithi045 ceph-mon[31773]: pgmap v2765: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:05.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:04 smithi071 ceph-mon[36597]: pgmap v2765: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:05.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:04 smithi151 ceph-mon[36481]: pgmap v2765: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:05.547 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5018 sec 2024-03-28T19:16:06.547 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5019 sec 2024-03-28T19:16:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:06 smithi045 ceph-mon[31773]: pgmap v2766: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4374.310724 secs 2024-03-28T19:16:07.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:06 smithi071 ceph-mon[36597]: pgmap v2766: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4374.310724 secs 2024-03-28T19:16:07.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:06 smithi151 ceph-mon[36481]: pgmap v2766: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:07.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4374.310724 secs 2024-03-28T19:16:07.547 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5020 sec 2024-03-28T19:16:08.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3665.736047 secs 2024-03-28T19:16:08.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:07 smithi045 ceph-mon[31773]: pgmap v2767: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:08.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3665.736047 secs 2024-03-28T19:16:08.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:07 smithi071 ceph-mon[36597]: pgmap v2767: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:08.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3665.736047 secs 2024-03-28T19:16:08.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:07 smithi151 ceph-mon[36481]: pgmap v2767: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:08.547 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5021 sec 2024-03-28T19:16:09.547 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5022 sec 2024-03-28T19:16:10.547 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5023 sec 2024-03-28T19:16:11.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:10 smithi045 ceph-mon[31773]: pgmap v2768: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:11.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:10 smithi071 ceph-mon[36597]: pgmap v2768: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:11.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:10 smithi151 ceph-mon[36481]: pgmap v2768: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:11.547 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5024 sec 2024-03-28T19:16:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4379.310782 secs 2024-03-28T19:16:12.223 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4379.310782 secs 2024-03-28T19:16:12.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4379.310782 secs 2024-03-28T19:16:12.547 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5025 sec 2024-03-28T19:16:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3670.736109 secs 2024-03-28T19:16:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:12 smithi045 ceph-mon[31773]: pgmap v2769: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:13.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3670.736109 secs 2024-03-28T19:16:13.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:12 smithi071 ceph-mon[36597]: pgmap v2769: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:13.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3670.736109 secs 2024-03-28T19:16:13.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:12 smithi151 ceph-mon[36481]: pgmap v2769: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:13.548 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5026 sec 2024-03-28T19:16:14.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:13 smithi045 ceph-mon[31773]: pgmap v2770: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:14.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:13 smithi071 ceph-mon[36597]: pgmap v2770: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:14.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:13 smithi151 ceph-mon[36481]: pgmap v2770: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:14.548 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5027 sec 2024-03-28T19:16:15.548 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5028 sec 2024-03-28T19:16:16.548 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5029 sec 2024-03-28T19:16:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:16 smithi045 ceph-mon[31773]: pgmap v2771: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4384.310844 secs 2024-03-28T19:16:17.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:16 smithi071 ceph-mon[36597]: pgmap v2771: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:17.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4384.310844 secs 2024-03-28T19:16:17.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:16 smithi151 ceph-mon[36481]: pgmap v2771: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:17.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4384.310844 secs 2024-03-28T19:16:17.548 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5030 sec 2024-03-28T19:16:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3675.736196 secs 2024-03-28T19:16:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:17 smithi045 ceph-mon[31773]: pgmap v2772: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:18.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3675.736196 secs 2024-03-28T19:16:18.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:17 smithi071 ceph-mon[36597]: pgmap v2772: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:18.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3675.736196 secs 2024-03-28T19:16:18.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:17 smithi151 ceph-mon[36481]: pgmap v2772: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:18.548 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5031 sec 2024-03-28T19:16:19.549 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5032 sec 2024-03-28T19:16:20.549 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5033 sec 2024-03-28T19:16:21.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:20 smithi045 ceph-mon[31773]: pgmap v2773: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:21.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:20 smithi071 ceph-mon[36597]: pgmap v2773: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:21.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:20 smithi151 ceph-mon[36481]: pgmap v2773: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:21.549 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5034 sec 2024-03-28T19:16:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4389.310931 secs 2024-03-28T19:16:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:21 smithi045 ceph-mon[31773]: pgmap v2774: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:22.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4389.310931 secs 2024-03-28T19:16:22.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:21 smithi071 ceph-mon[36597]: pgmap v2774: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:22.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4389.310931 secs 2024-03-28T19:16:22.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:21 smithi151 ceph-mon[36481]: pgmap v2774: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:22.549 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5035 sec 2024-03-28T19:16:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3680.736257 secs 2024-03-28T19:16:23.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3680.736257 secs 2024-03-28T19:16:23.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3680.736257 secs 2024-03-28T19:16:23.549 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5036 sec 2024-03-28T19:16:24.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:23 smithi045 ceph-mon[31773]: pgmap v2775: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:24.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:23 smithi071 ceph-mon[36597]: pgmap v2775: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:24.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:23 smithi151 ceph-mon[36481]: pgmap v2775: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:24.549 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5037 sec 2024-03-28T19:16:25.550 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5038 sec 2024-03-28T19:16:26.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:25 smithi045 ceph-mon[31773]: pgmap v2776: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:26.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:25 smithi071 ceph-mon[36597]: pgmap v2776: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:26.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:25 smithi151 ceph-mon[36481]: pgmap v2776: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:26.550 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5039 sec 2024-03-28T19:16:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4394.310992 secs 2024-03-28T19:16:27.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4394.310992 secs 2024-03-28T19:16:27.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4394.310992 secs 2024-03-28T19:16:27.550 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5040 sec 2024-03-28T19:16:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3685.736320 secs 2024-03-28T19:16:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:27 smithi045 ceph-mon[31773]: pgmap v2777: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:28.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3685.736320 secs 2024-03-28T19:16:28.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:27 smithi071 ceph-mon[36597]: pgmap v2777: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:28.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3685.736320 secs 2024-03-28T19:16:28.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:27 smithi151 ceph-mon[36481]: pgmap v2777: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:28.550 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5041 sec 2024-03-28T19:16:29.550 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5042 sec 2024-03-28T19:16:30.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:29 smithi045 ceph-mon[31773]: pgmap v2778: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:30.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:29 smithi071 ceph-mon[36597]: pgmap v2778: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:30.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:29 smithi151 ceph-mon[36481]: pgmap v2778: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:30.550 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5043 sec 2024-03-28T19:16:31.550 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5044 sec 2024-03-28T19:16:31.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4399.311080 secs 2024-03-28T19:16:31.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4399.311080 secs 2024-03-28T19:16:31.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4399.311080 secs 2024-03-28T19:16:32.551 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5045 sec 2024-03-28T19:16:32.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3690.736408 secs 2024-03-28T19:16:32.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:32 smithi045 ceph-mon[31773]: pgmap v2779: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:32.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3690.736408 secs 2024-03-28T19:16:32.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:32 smithi071 ceph-mon[36597]: pgmap v2779: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:32.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3690.736408 secs 2024-03-28T19:16:32.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:32 smithi151 ceph-mon[36481]: pgmap v2779: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:33.550 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5046 sec 2024-03-28T19:16:34.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:33 smithi045 ceph-mon[31773]: pgmap v2780: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:34.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:33 smithi071 ceph-mon[36597]: pgmap v2780: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:34.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:33 smithi151 ceph-mon[36481]: pgmap v2780: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:34.550 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5047 sec 2024-03-28T19:16:35.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:34 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:16:35.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:34 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:16:35.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:34 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:16:35.550 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5048 sec 2024-03-28T19:16:36.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:35 smithi151 ceph-mon[36481]: pgmap v2781: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:36.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:35 smithi045 ceph-mon[31773]: pgmap v2781: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:36.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:35 smithi071 ceph-mon[36597]: pgmap v2781: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:36.551 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5049 sec 2024-03-28T19:16:37.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4404.311184 secs 2024-03-28T19:16:37.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4404.311184 secs 2024-03-28T19:16:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4404.311184 secs 2024-03-28T19:16:37.551 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5050 sec 2024-03-28T19:16:38.237 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3695.736497 secs 2024-03-28T19:16:38.237 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:37 smithi151 ceph-mon[36481]: pgmap v2782: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:38.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3695.736497 secs 2024-03-28T19:16:38.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:37 smithi071 ceph-mon[36597]: pgmap v2782: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:38.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3695.736497 secs 2024-03-28T19:16:38.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:37 smithi045 ceph-mon[31773]: pgmap v2782: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:38.551 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5051 sec 2024-03-28T19:16:39.551 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5052 sec 2024-03-28T19:16:39.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:39 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:16:39.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:39 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:16:39.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:39 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:16:39.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:39 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:16:39.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:39 smithi071 ceph-mon[36597]: pgmap v2783: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T19:16:39.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:39 smithi071 ceph-mon[36597]: pgmap v2784: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:16:39.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:39 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:16:39.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:39 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:16:39.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:39 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:16:39.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:39 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:16:39.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:39 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:16:39.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:39 smithi151 ceph-mon[36481]: pgmap v2783: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T19:16:39.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:39 smithi151 ceph-mon[36481]: pgmap v2784: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:16:39.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:39 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:16:39.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:39 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:16:39.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:39 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:16:39.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:39 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:16:39.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:39 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:16:39.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:39 smithi045 ceph-mon[31773]: pgmap v2783: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T19:16:39.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:39 smithi045 ceph-mon[31773]: pgmap v2784: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:16:39.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:39 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:16:40.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5053 sec 2024-03-28T19:16:41.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5054 sec 2024-03-28T19:16:41.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:41 smithi045 ceph-mon[31773]: pgmap v2785: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:16:41.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4409.311289 secs 2024-03-28T19:16:41.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:41 smithi071 ceph-mon[36597]: pgmap v2785: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:16:41.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4409.311289 secs 2024-03-28T19:16:41.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:41 smithi151 ceph-mon[36481]: pgmap v2785: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:16:41.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4409.311289 secs 2024-03-28T19:16:42.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5055 sec 2024-03-28T19:16:42.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3700.736585 secs 2024-03-28T19:16:42.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3700.736585 secs 2024-03-28T19:16:42.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3700.736585 secs 2024-03-28T19:16:43.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5056 sec 2024-03-28T19:16:43.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:43 smithi045 ceph-mon[31773]: pgmap v2786: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:16:43.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:43 smithi071 ceph-mon[36597]: pgmap v2786: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:16:43.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:43 smithi151 ceph-mon[36481]: pgmap v2786: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:16:44.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5057 sec 2024-03-28T19:16:45.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5058 sec 2024-03-28T19:16:45.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:45 smithi045 ceph-mon[31773]: pgmap v2787: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:16:45.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:45 smithi071 ceph-mon[36597]: pgmap v2787: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:16:45.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:45 smithi151 ceph-mon[36481]: pgmap v2787: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:16:46.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5059 sec 2024-03-28T19:16:46.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4414.311372 secs 2024-03-28T19:16:46.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4414.311372 secs 2024-03-28T19:16:46.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4414.311372 secs 2024-03-28T19:16:47.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5060 sec 2024-03-28T19:16:47.710 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:47 smithi071 ceph-mon[36597]: pgmap v2788: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:16:47.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3705.736682 secs 2024-03-28T19:16:47.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:47 smithi151 ceph-mon[36481]: pgmap v2788: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:16:47.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3705.736682 secs 2024-03-28T19:16:47.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:47 smithi045 ceph-mon[31773]: pgmap v2788: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:16:47.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3705.736682 secs 2024-03-28T19:16:48.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5061 sec 2024-03-28T19:16:49.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5062 sec 2024-03-28T19:16:49.718 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:49 smithi045 ceph-mon[31773]: pgmap v2789: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:16:49.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:49 smithi071 ceph-mon[36597]: pgmap v2789: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:16:49.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:49 smithi151 ceph-mon[36481]: pgmap v2789: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:16:50.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5063 sec 2024-03-28T19:16:51.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5064 sec 2024-03-28T19:16:51.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:51 smithi045 ceph-mon[31773]: pgmap v2790: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:51.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4419.311439 secs 2024-03-28T19:16:51.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:51 smithi071 ceph-mon[36597]: pgmap v2790: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4419.311439 secs 2024-03-28T19:16:51.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:51 smithi151 ceph-mon[36481]: pgmap v2790: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:51.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4419.311439 secs 2024-03-28T19:16:52.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5065 sec 2024-03-28T19:16:52.775 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3710.736771 secs 2024-03-28T19:16:52.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3710.736771 secs 2024-03-28T19:16:52.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3710.736771 secs 2024-03-28T19:16:53.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5066 sec 2024-03-28T19:16:53.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:53 smithi071 ceph-mon[36597]: pgmap v2791: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:53.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:53 smithi151 ceph-mon[36481]: pgmap v2791: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:53.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:53 smithi045 ceph-mon[31773]: pgmap v2791: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:54.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5067 sec 2024-03-28T19:16:55.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5068 sec 2024-03-28T19:16:55.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:55 smithi071 ceph-mon[36597]: pgmap v2792: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:55.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:55 smithi151 ceph-mon[36481]: pgmap v2792: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:55.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:55 smithi045 ceph-mon[31773]: pgmap v2792: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:56.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5069 sec 2024-03-28T19:16:56.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4424.311521 secs 2024-03-28T19:16:56.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4424.311521 secs 2024-03-28T19:16:56.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4424.311521 secs 2024-03-28T19:16:57.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5070 sec 2024-03-28T19:16:57.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:57 smithi045 ceph-mon[31773]: pgmap v2793: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:57.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3715.736856 secs 2024-03-28T19:16:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:57 smithi071 ceph-mon[36597]: pgmap v2793: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3715.736856 secs 2024-03-28T19:16:57.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:57 smithi151 ceph-mon[36481]: pgmap v2793: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:16:57.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3715.736856 secs 2024-03-28T19:16:58.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5071 sec 2024-03-28T19:16:59.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5072 sec 2024-03-28T19:16:59.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:16:59 smithi045 ceph-mon[31773]: pgmap v2794: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:59.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:16:59 smithi071 ceph-mon[36597]: pgmap v2794: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:16:59.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:16:59 smithi151 ceph-mon[36481]: pgmap v2794: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:00.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5073 sec 2024-03-28T19:17:01.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5074 sec 2024-03-28T19:17:01.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:01 smithi045 ceph-mon[31773]: pgmap v2795: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:01.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4429.311583 secs 2024-03-28T19:17:01.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:01 smithi071 ceph-mon[36597]: pgmap v2795: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:01.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4429.311583 secs 2024-03-28T19:17:01.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:01 smithi151 ceph-mon[36481]: pgmap v2795: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:01.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4429.311583 secs 2024-03-28T19:17:02.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5075 sec 2024-03-28T19:17:02.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3720.736943 secs 2024-03-28T19:17:02.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3720.736943 secs 2024-03-28T19:17:02.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3720.736943 secs 2024-03-28T19:17:03.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5076 sec 2024-03-28T19:17:03.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:03 smithi071 ceph-mon[36597]: pgmap v2796: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:03.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:03 smithi151 ceph-mon[36481]: pgmap v2796: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:03.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:03 smithi045 ceph-mon[31773]: pgmap v2796: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:04.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5077 sec 2024-03-28T19:17:05.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5078 sec 2024-03-28T19:17:05.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:05 smithi071 ceph-mon[36597]: pgmap v2797: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:05.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:05 smithi151 ceph-mon[36481]: pgmap v2797: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:05.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:05 smithi045 ceph-mon[31773]: pgmap v2797: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:06.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5079 sec 2024-03-28T19:17:06.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4434.311666 secs 2024-03-28T19:17:06.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4434.311666 secs 2024-03-28T19:17:06.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4434.311666 secs 2024-03-28T19:17:07.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5080 sec 2024-03-28T19:17:07.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:07 smithi071 ceph-mon[36597]: pgmap v2798: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:07.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3725.737006 secs 2024-03-28T19:17:07.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:07 smithi151 ceph-mon[36481]: pgmap v2798: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:07.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3725.737006 secs 2024-03-28T19:17:07.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:07 smithi045 ceph-mon[31773]: pgmap v2798: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:07.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3725.737006 secs 2024-03-28T19:17:08.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5081 sec 2024-03-28T19:17:09.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5082 sec 2024-03-28T19:17:09.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:09 smithi071 ceph-mon[36597]: pgmap v2799: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:09.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:09 smithi151 ceph-mon[36481]: pgmap v2799: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:09.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:09 smithi045 ceph-mon[31773]: pgmap v2799: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:10.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5083 sec 2024-03-28T19:17:11.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5084 sec 2024-03-28T19:17:11.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:11 smithi071 ceph-mon[36597]: pgmap v2800: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:11.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4439.311728 secs 2024-03-28T19:17:11.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:11 smithi151 ceph-mon[36481]: pgmap v2800: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:11.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4439.311728 secs 2024-03-28T19:17:11.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:11 smithi045 ceph-mon[31773]: pgmap v2800: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4439.311728 secs 2024-03-28T19:17:12.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5085 sec 2024-03-28T19:17:12.723 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3730.737066 secs 2024-03-28T19:17:12.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3730.737066 secs 2024-03-28T19:17:12.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3730.737066 secs 2024-03-28T19:17:13.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5086 sec 2024-03-28T19:17:13.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:13 smithi071 ceph-mon[36597]: pgmap v2801: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:13.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:13 smithi151 ceph-mon[36481]: pgmap v2801: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:13.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:13 smithi045 ceph-mon[31773]: pgmap v2801: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:14.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5087 sec 2024-03-28T19:17:15.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5088 sec 2024-03-28T19:17:15.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:15 smithi071 ceph-mon[36597]: pgmap v2802: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:15.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:15 smithi151 ceph-mon[36481]: pgmap v2802: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:15.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:15 smithi045 ceph-mon[31773]: pgmap v2802: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:16.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5089 sec 2024-03-28T19:17:16.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4444.311801 secs 2024-03-28T19:17:16.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4444.311801 secs 2024-03-28T19:17:16.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4444.311801 secs 2024-03-28T19:17:17.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5090 sec 2024-03-28T19:17:17.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:17 smithi071 ceph-mon[36597]: pgmap v2803: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:17.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3735.737150 secs 2024-03-28T19:17:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:17 smithi151 ceph-mon[36481]: pgmap v2803: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3735.737150 secs 2024-03-28T19:17:17.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:17 smithi045 ceph-mon[31773]: pgmap v2803: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:17.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3735.737150 secs 2024-03-28T19:17:18.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5091 sec 2024-03-28T19:17:19.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5092 sec 2024-03-28T19:17:19.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:19 smithi071 ceph-mon[36597]: pgmap v2804: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:19.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:19 smithi151 ceph-mon[36481]: pgmap v2804: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:19.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:19 smithi045 ceph-mon[31773]: pgmap v2804: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:20.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5093 sec 2024-03-28T19:17:21.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5094 sec 2024-03-28T19:17:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:21 smithi071 ceph-mon[36597]: pgmap v2805: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:21.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4449.311838 secs 2024-03-28T19:17:21.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:21 smithi151 ceph-mon[36481]: pgmap v2805: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:21.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4449.311838 secs 2024-03-28T19:17:21.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:21 smithi045 ceph-mon[31773]: pgmap v2805: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:21.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4449.311838 secs 2024-03-28T19:17:22.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5095 sec 2024-03-28T19:17:22.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3740.737247 secs 2024-03-28T19:17:22.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3740.737247 secs 2024-03-28T19:17:22.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3740.737247 secs 2024-03-28T19:17:23.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5096 sec 2024-03-28T19:17:23.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:23 smithi071 ceph-mon[36597]: pgmap v2806: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:23.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:23 smithi151 ceph-mon[36481]: pgmap v2806: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:23.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:23 smithi045 ceph-mon[31773]: pgmap v2806: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:24.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5097 sec 2024-03-28T19:17:25.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5098 sec 2024-03-28T19:17:25.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:25 smithi071 ceph-mon[36597]: pgmap v2807: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:25.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:25 smithi151 ceph-mon[36481]: pgmap v2807: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:25.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:25 smithi045 ceph-mon[31773]: pgmap v2807: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:26.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5099 sec 2024-03-28T19:17:26.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4454.311933 secs 2024-03-28T19:17:26.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4454.311933 secs 2024-03-28T19:17:26.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4454.311933 secs 2024-03-28T19:17:27.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5100 sec 2024-03-28T19:17:27.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:27 smithi071 ceph-mon[36597]: pgmap v2808: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:27.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3745.737309 secs 2024-03-28T19:17:27.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:27 smithi151 ceph-mon[36481]: pgmap v2808: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:27.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3745.737309 secs 2024-03-28T19:17:27.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:27 smithi045 ceph-mon[31773]: pgmap v2808: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:27.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3745.737309 secs 2024-03-28T19:17:28.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5101 sec 2024-03-28T19:17:29.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5102 sec 2024-03-28T19:17:29.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:29 smithi071 ceph-mon[36597]: pgmap v2809: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:29.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:29 smithi151 ceph-mon[36481]: pgmap v2809: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:29.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:29 smithi045 ceph-mon[31773]: pgmap v2809: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:30.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5103 sec 2024-03-28T19:17:31.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5104 sec 2024-03-28T19:17:31.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:31 smithi071 ceph-mon[36597]: pgmap v2810: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4459.311992 secs 2024-03-28T19:17:31.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:31 smithi151 ceph-mon[36481]: pgmap v2810: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:31.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4459.311992 secs 2024-03-28T19:17:31.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:31 smithi045 ceph-mon[31773]: pgmap v2810: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:31.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4459.311992 secs 2024-03-28T19:17:32.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5105 sec 2024-03-28T19:17:32.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3750.737400 secs 2024-03-28T19:17:32.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3750.737400 secs 2024-03-28T19:17:32.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3750.737400 secs 2024-03-28T19:17:33.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5106 sec 2024-03-28T19:17:33.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:33 smithi071 ceph-mon[36597]: pgmap v2811: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:33.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:33 smithi151 ceph-mon[36481]: pgmap v2811: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:33.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:33 smithi045 ceph-mon[31773]: pgmap v2811: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:34.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5107 sec 2024-03-28T19:17:35.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5108 sec 2024-03-28T19:17:35.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:35 smithi071 ceph-mon[36597]: pgmap v2812: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:35.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:35 smithi151 ceph-mon[36481]: pgmap v2812: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:35.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:35 smithi045 ceph-mon[31773]: pgmap v2812: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:36.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5109 sec 2024-03-28T19:17:36.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4464.312033 secs 2024-03-28T19:17:36.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4464.312033 secs 2024-03-28T19:17:37.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4464.312033 secs 2024-03-28T19:17:37.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5110 sec 2024-03-28T19:17:37.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:37 smithi151 ceph-mon[36481]: pgmap v2813: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:37.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3755.737440 secs 2024-03-28T19:17:37.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:37 smithi045 ceph-mon[31773]: pgmap v2813: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:37.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3755.737440 secs 2024-03-28T19:17:38.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:37 smithi071 ceph-mon[36597]: pgmap v2813: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:38.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3755.737440 secs 2024-03-28T19:17:38.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5111 sec 2024-03-28T19:17:39.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5112 sec 2024-03-28T19:17:39.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:39 smithi151 ceph-mon[36481]: pgmap v2814: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:39.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:39 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:17:39.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:39 smithi045 ceph-mon[31773]: pgmap v2814: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:39.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:39 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:17:40.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:39 smithi071 ceph-mon[36597]: pgmap v2814: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:40.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:39 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:17:40.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5113 sec 2024-03-28T19:17:40.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:40 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:17:40.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:40 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:17:40.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:40 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:17:41.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:40 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:17:41.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:40 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:17:41.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:40 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:17:41.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:40 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:17:41.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:40 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:17:41.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:40 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:17:41.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5114 sec 2024-03-28T19:17:41.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:41 smithi045 ceph-mon[31773]: pgmap v2815: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:41.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:41 smithi045 ceph-mon[31773]: pgmap v2816: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:17:41.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4469.312121 secs 2024-03-28T19:17:42.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:41 smithi071 ceph-mon[36597]: pgmap v2815: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:41 smithi071 ceph-mon[36597]: pgmap v2816: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:17:42.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4469.312121 secs 2024-03-28T19:17:42.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:41 smithi151 ceph-mon[36481]: pgmap v2815: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:42.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:41 smithi151 ceph-mon[36481]: pgmap v2816: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:17:42.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4469.312121 secs 2024-03-28T19:17:42.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5115 sec 2024-03-28T19:17:42.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3760.737527 secs 2024-03-28T19:17:42.973 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3760.737527 secs 2024-03-28T19:17:43.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3760.737527 secs 2024-03-28T19:17:43.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5116 sec 2024-03-28T19:17:43.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:43 smithi045 ceph-mon[31773]: pgmap v2817: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:44.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:43 smithi071 ceph-mon[36597]: pgmap v2817: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:44.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:43 smithi151 ceph-mon[36481]: pgmap v2817: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:44.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5117 sec 2024-03-28T19:17:45.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5118 sec 2024-03-28T19:17:45.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:45 smithi045 ceph-mon[31773]: pgmap v2818: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:46.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:45 smithi071 ceph-mon[36597]: pgmap v2818: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:46.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:45 smithi151 ceph-mon[36481]: pgmap v2818: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:46.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5119 sec 2024-03-28T19:17:46.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4474.312181 secs 2024-03-28T19:17:47.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4474.312181 secs 2024-03-28T19:17:47.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4474.312181 secs 2024-03-28T19:17:47.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5120 sec 2024-03-28T19:17:47.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:47 smithi045 ceph-mon[31773]: pgmap v2819: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:47.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3765.737586 secs 2024-03-28T19:17:47.960 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:47 smithi071 ceph-mon[36597]: pgmap v2819: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:47.960 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3765.737586 secs 2024-03-28T19:17:48.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:47 smithi151 ceph-mon[36481]: pgmap v2819: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:48.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3765.737586 secs 2024-03-28T19:17:48.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5121 sec 2024-03-28T19:17:49.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5122 sec 2024-03-28T19:17:49.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:49 smithi045 ceph-mon[31773]: pgmap v2820: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:50.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:49 smithi071 ceph-mon[36597]: pgmap v2820: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:50.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:49 smithi151 ceph-mon[36481]: pgmap v2820: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:17:50.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5123 sec 2024-03-28T19:17:51.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5124 sec 2024-03-28T19:17:51.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:51 smithi045 ceph-mon[31773]: pgmap v2821: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:17:51.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4479.312219 secs 2024-03-28T19:17:52.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:51 smithi071 ceph-mon[36597]: pgmap v2821: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:17:52.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4479.312219 secs 2024-03-28T19:17:52.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:51 smithi151 ceph-mon[36481]: pgmap v2821: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:17:52.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4479.312219 secs 2024-03-28T19:17:52.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5125 sec 2024-03-28T19:17:52.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3770.737645 secs 2024-03-28T19:17:53.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3770.737645 secs 2024-03-28T19:17:53.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3770.737645 secs 2024-03-28T19:17:53.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5126 sec 2024-03-28T19:17:53.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:53 smithi045 ceph-mon[31773]: pgmap v2822: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:54.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:53 smithi071 ceph-mon[36597]: pgmap v2822: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:54.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:53 smithi151 ceph-mon[36481]: pgmap v2822: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:17:54.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5127 sec 2024-03-28T19:17:55.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5128 sec 2024-03-28T19:17:55.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:55 smithi045 ceph-mon[31773]: pgmap v2823: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:56.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:55 smithi071 ceph-mon[36597]: pgmap v2823: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:56.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:55 smithi151 ceph-mon[36481]: pgmap v2823: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:17:56.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5129 sec 2024-03-28T19:17:56.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4484.312306 secs 2024-03-28T19:17:57.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4484.312306 secs 2024-03-28T19:17:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4484.312306 secs 2024-03-28T19:17:57.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5130 sec 2024-03-28T19:17:57.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:57 smithi045 ceph-mon[31773]: pgmap v2824: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:17:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3775.737704 secs 2024-03-28T19:17:58.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:57 smithi071 ceph-mon[36597]: pgmap v2824: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:17:58.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3775.737704 secs 2024-03-28T19:17:58.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:57 smithi151 ceph-mon[36481]: pgmap v2824: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:17:58.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3775.737704 secs 2024-03-28T19:17:58.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5131 sec 2024-03-28T19:17:59.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5132 sec 2024-03-28T19:17:59.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:17:59 smithi045 ceph-mon[31773]: pgmap v2825: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-03-28T19:18:00.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:17:59 smithi071 ceph-mon[36597]: pgmap v2825: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-03-28T19:18:00.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:17:59 smithi151 ceph-mon[36481]: pgmap v2825: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-03-28T19:18:00.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5133 sec 2024-03-28T19:18:01.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5134 sec 2024-03-28T19:18:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:01 smithi045 ceph-mon[31773]: pgmap v2826: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-03-28T19:18:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4489.312392 secs 2024-03-28T19:18:02.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:01 smithi071 ceph-mon[36597]: pgmap v2826: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-03-28T19:18:02.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4489.312392 secs 2024-03-28T19:18:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:01 smithi151 ceph-mon[36481]: pgmap v2826: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-03-28T19:18:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4489.312392 secs 2024-03-28T19:18:02.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5135 sec 2024-03-28T19:18:03.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3780.737795 secs 2024-03-28T19:18:03.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3780.737795 secs 2024-03-28T19:18:03.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3780.737795 secs 2024-03-28T19:18:03.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5136 sec 2024-03-28T19:18:04.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:03 smithi071 ceph-mon[36597]: pgmap v2827: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:18:04.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:03 smithi151 ceph-mon[36481]: pgmap v2827: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:18:04.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:03 smithi045 ceph-mon[31773]: pgmap v2827: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-28T19:18:04.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5137 sec 2024-03-28T19:18:05.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5138 sec 2024-03-28T19:18:06.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:05 smithi071 ceph-mon[36597]: pgmap v2828: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-03-28T19:18:06.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:05 smithi151 ceph-mon[36481]: pgmap v2828: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-03-28T19:18:06.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:05 smithi045 ceph-mon[31773]: pgmap v2828: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-03-28T19:18:06.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5139 sec 2024-03-28T19:18:07.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4494.312482 secs 2024-03-28T19:18:07.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4494.312482 secs 2024-03-28T19:18:07.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4494.312482 secs 2024-03-28T19:18:07.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5140 sec 2024-03-28T19:18:08.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:07 smithi071 ceph-mon[36597]: pgmap v2829: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T19:18:08.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3785.737855 secs 2024-03-28T19:18:08.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:07 smithi151 ceph-mon[36481]: pgmap v2829: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T19:18:08.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3785.737855 secs 2024-03-28T19:18:08.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:07 smithi045 ceph-mon[31773]: pgmap v2829: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T19:18:08.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3785.737855 secs 2024-03-28T19:18:08.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5141 sec 2024-03-28T19:18:09.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5142 sec 2024-03-28T19:18:10.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:09 smithi071 ceph-mon[36597]: pgmap v2830: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T19:18:10.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:09 smithi151 ceph-mon[36481]: pgmap v2830: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T19:18:10.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:09 smithi045 ceph-mon[31773]: pgmap v2830: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T19:18:10.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5143 sec 2024-03-28T19:18:11.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5144 sec 2024-03-28T19:18:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:11 smithi071 ceph-mon[36597]: pgmap v2831: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T19:18:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4499.312546 secs 2024-03-28T19:18:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:11 smithi151 ceph-mon[36481]: pgmap v2831: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T19:18:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4499.312546 secs 2024-03-28T19:18:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:11 smithi045 ceph-mon[31773]: pgmap v2831: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T19:18:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4499.312546 secs 2024-03-28T19:18:12.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5145 sec 2024-03-28T19:18:12.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3790.737939 secs 2024-03-28T19:18:13.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3790.737939 secs 2024-03-28T19:18:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3790.737939 secs 2024-03-28T19:18:13.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5146 sec 2024-03-28T19:18:14.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:13 smithi071 ceph-mon[36597]: pgmap v2832: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T19:18:14.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:13 smithi151 ceph-mon[36481]: pgmap v2832: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T19:18:14.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:13 smithi045 ceph-mon[31773]: pgmap v2832: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T19:18:14.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5147 sec 2024-03-28T19:18:15.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5148 sec 2024-03-28T19:18:16.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:15 smithi071 ceph-mon[36597]: pgmap v2833: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T19:18:16.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:15 smithi151 ceph-mon[36481]: pgmap v2833: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T19:18:16.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:15 smithi045 ceph-mon[31773]: pgmap v2833: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T19:18:16.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5149 sec 2024-03-28T19:18:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4504.312583 secs 2024-03-28T19:18:17.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4504.312583 secs 2024-03-28T19:18:17.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4504.312583 secs 2024-03-28T19:18:17.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5150 sec 2024-03-28T19:18:18.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:17 smithi071 ceph-mon[36597]: pgmap v2834: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T19:18:18.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3795.738000 secs 2024-03-28T19:18:18.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:17 smithi151 ceph-mon[36481]: pgmap v2834: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T19:18:18.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3795.738000 secs 2024-03-28T19:18:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:17 smithi045 ceph-mon[31773]: pgmap v2834: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T19:18:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3795.738000 secs 2024-03-28T19:18:18.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5151 sec 2024-03-28T19:18:19.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5152 sec 2024-03-28T19:18:20.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:19 smithi071 ceph-mon[36597]: pgmap v2835: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:20.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:19 smithi151 ceph-mon[36481]: pgmap v2835: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:20.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:19 smithi045 ceph-mon[31773]: pgmap v2835: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:20.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5153 sec 2024-03-28T19:18:21.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5154 sec 2024-03-28T19:18:22.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:21 smithi071 ceph-mon[36597]: pgmap v2836: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:22.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4509.312646 secs 2024-03-28T19:18:22.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:21 smithi151 ceph-mon[36481]: pgmap v2836: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:22.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4509.312646 secs 2024-03-28T19:18:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:21 smithi045 ceph-mon[31773]: pgmap v2836: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4509.312646 secs 2024-03-28T19:18:22.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5155 sec 2024-03-28T19:18:23.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3800.738064 secs 2024-03-28T19:18:23.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3800.738064 secs 2024-03-28T19:18:23.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3800.738064 secs 2024-03-28T19:18:23.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5156 sec 2024-03-28T19:18:24.062 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:23 smithi071 ceph-mon[36597]: pgmap v2837: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:24.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:23 smithi151 ceph-mon[36481]: pgmap v2837: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:24.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:23 smithi045 ceph-mon[31773]: pgmap v2837: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:24.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5157 sec 2024-03-28T19:18:25.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5158 sec 2024-03-28T19:18:26.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:25 smithi071 ceph-mon[36597]: pgmap v2838: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:26.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:25 smithi151 ceph-mon[36481]: pgmap v2838: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:26.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:25 smithi045 ceph-mon[31773]: pgmap v2838: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:26.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5159 sec 2024-03-28T19:18:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4514.312707 secs 2024-03-28T19:18:27.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4514.312707 secs 2024-03-28T19:18:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4514.312707 secs 2024-03-28T19:18:27.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5160 sec 2024-03-28T19:18:28.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:27 smithi071 ceph-mon[36597]: pgmap v2839: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3805.738137 secs 2024-03-28T19:18:28.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:27 smithi151 ceph-mon[36481]: pgmap v2839: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:28.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3805.738137 secs 2024-03-28T19:18:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:27 smithi045 ceph-mon[31773]: pgmap v2839: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3805.738137 secs 2024-03-28T19:18:28.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5161 sec 2024-03-28T19:18:29.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5162 sec 2024-03-28T19:18:30.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:29 smithi071 ceph-mon[36597]: pgmap v2840: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:30.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:29 smithi151 ceph-mon[36481]: pgmap v2840: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:30.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:29 smithi045 ceph-mon[31773]: pgmap v2840: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:30.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5163 sec 2024-03-28T19:18:31.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5164 sec 2024-03-28T19:18:32.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:31 smithi071 ceph-mon[36597]: pgmap v2841: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:32.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4519.312797 secs 2024-03-28T19:18:32.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:31 smithi151 ceph-mon[36481]: pgmap v2841: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:32.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4519.312797 secs 2024-03-28T19:18:32.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:31 smithi045 ceph-mon[31773]: pgmap v2841: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:32.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4519.312797 secs 2024-03-28T19:18:32.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5165 sec 2024-03-28T19:18:32.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3810.738220 secs 2024-03-28T19:18:32.915 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:18:32 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:18:32.623+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:18:32.915 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:18:32 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:18:32.623+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:18:32.915 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:18:32 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:18:32.623+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:18:32.915 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:18:32 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:18:32.623+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:18:32.915 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:18:32 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:18:32.623+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:18:33.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3810.738220 secs 2024-03-28T19:18:33.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3810.738220 secs 2024-03-28T19:18:33.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5166 sec 2024-03-28T19:18:34.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:33 smithi071 ceph-mon[36597]: pgmap v2842: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:34.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:33 smithi151 ceph-mon[36481]: pgmap v2842: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:34.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:33 smithi045 ceph-mon[31773]: pgmap v2842: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:34.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5167 sec 2024-03-28T19:18:35.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5168 sec 2024-03-28T19:18:36.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:35 smithi071 ceph-mon[36597]: pgmap v2843: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:36.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:35 smithi151 ceph-mon[36481]: pgmap v2843: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:36.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:35 smithi045 ceph-mon[31773]: pgmap v2843: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:36.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5169 sec 2024-03-28T19:18:37.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:36 smithi071 ceph-mon[36597]: 5 slow requests, 1 included below; oldest blocked for > 4524.312886 secs 2024-03-28T19:18:37.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:36 smithi071 ceph-mon[36597]: slow request 2884.483995 seconds old, received at 2024-03-28T18:30:31.389436+0000: client_request(client.26705:1 getattr pAsLsXsFs #0x1 2024-03-28T18:30:31.385264+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T19:18:37.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:36 smithi151 ceph-mon[36481]: 5 slow requests, 1 included below; oldest blocked for > 4524.312886 secs 2024-03-28T19:18:37.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:36 smithi151 ceph-mon[36481]: slow request 2884.483995 seconds old, received at 2024-03-28T18:30:31.389436+0000: client_request(client.26705:1 getattr pAsLsXsFs #0x1 2024-03-28T18:30:31.385264+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T19:18:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:36 smithi045 ceph-mon[31773]: 5 slow requests, 1 included below; oldest blocked for > 4524.312886 secs 2024-03-28T19:18:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:36 smithi045 ceph-mon[31773]: slow request 2884.483995 seconds old, received at 2024-03-28T18:30:31.389436+0000: client_request(client.26705:1 getattr pAsLsXsFs #0x1 2024-03-28T18:30:31.385264+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T19:18:37.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5170 sec 2024-03-28T19:18:38.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:37 smithi151 ceph-mon[36481]: pgmap v2844: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T19:18:38.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3815.738323 secs 2024-03-28T19:18:38.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:37 smithi045 ceph-mon[31773]: pgmap v2844: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T19:18:38.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3815.738323 secs 2024-03-28T19:18:38.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:37 smithi071 ceph-mon[36597]: pgmap v2844: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T19:18:38.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3815.738323 secs 2024-03-28T19:18:38.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5171 sec 2024-03-28T19:18:39.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5172 sec 2024-03-28T19:18:40.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:39 smithi151 ceph-mon[36481]: pgmap v2845: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T19:18:40.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:39 smithi045 ceph-mon[31773]: pgmap v2845: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T19:18:40.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:39 smithi071 ceph-mon[36597]: pgmap v2845: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T19:18:40.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5173 sec 2024-03-28T19:18:41.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:40 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:18:41.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:40 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:18:41.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:40 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:18:41.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5174 sec 2024-03-28T19:18:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:41 smithi045 ceph-mon[31773]: pgmap v2846: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T19:18:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4529.312948 secs 2024-03-28T19:18:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:41 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:18:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:41 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:18:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:41 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:18:42.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:41 smithi071 ceph-mon[36597]: pgmap v2846: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T19:18:42.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4529.312948 secs 2024-03-28T19:18:42.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:41 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:18:42.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:41 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:18:42.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:41 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:18:42.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:41 smithi151 ceph-mon[36481]: pgmap v2846: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-28T19:18:42.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4529.312948 secs 2024-03-28T19:18:42.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:41 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:18:42.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:41 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:18:42.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:41 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:18:42.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5175 sec 2024-03-28T19:18:43.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:42 smithi045 ceph-mon[31773]: pgmap v2847: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 176 B/s wr, 5 op/s 2024-03-28T19:18:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:42 smithi045 ceph-mon[31773]: pgmap v2848: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 213 B/s wr, 4 op/s 2024-03-28T19:18:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3820.738340 secs 2024-03-28T19:18:43.223 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:42 smithi151 ceph-mon[36481]: pgmap v2847: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 176 B/s wr, 5 op/s 2024-03-28T19:18:43.223 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:42 smithi151 ceph-mon[36481]: pgmap v2848: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 213 B/s wr, 4 op/s 2024-03-28T19:18:43.223 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3820.738340 secs 2024-03-28T19:18:43.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:42 smithi071 ceph-mon[36597]: pgmap v2847: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 176 B/s wr, 5 op/s 2024-03-28T19:18:43.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:42 smithi071 ceph-mon[36597]: pgmap v2848: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 213 B/s wr, 4 op/s 2024-03-28T19:18:43.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3820.738340 secs 2024-03-28T19:18:43.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5176 sec 2024-03-28T19:18:44.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5177 sec 2024-03-28T19:18:45.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:44 smithi045 ceph-mon[31773]: pgmap v2849: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 213 B/s wr, 4 op/s 2024-03-28T19:18:45.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:44 smithi071 ceph-mon[36597]: pgmap v2849: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 213 B/s wr, 4 op/s 2024-03-28T19:18:45.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:44 smithi151 ceph-mon[36481]: pgmap v2849: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 213 B/s wr, 4 op/s 2024-03-28T19:18:45.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5178 sec 2024-03-28T19:18:46.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5179 sec 2024-03-28T19:18:47.111 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:46 smithi071 ceph-mon[36597]: pgmap v2850: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:47.111 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4534.313035 secs 2024-03-28T19:18:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:46 smithi045 ceph-mon[31773]: pgmap v2850: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4534.313035 secs 2024-03-28T19:18:47.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:46 smithi151 ceph-mon[36481]: pgmap v2850: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:47.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4534.313035 secs 2024-03-28T19:18:47.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5180 sec 2024-03-28T19:18:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3825.738426 secs 2024-03-28T19:18:48.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3825.738426 secs 2024-03-28T19:18:48.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3825.738426 secs 2024-03-28T19:18:48.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5181 sec 2024-03-28T19:18:49.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:48 smithi045 ceph-mon[31773]: pgmap v2851: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:49.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:48 smithi071 ceph-mon[36597]: pgmap v2851: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:49.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:48 smithi151 ceph-mon[36481]: pgmap v2851: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:49.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5182 sec 2024-03-28T19:18:50.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5183 sec 2024-03-28T19:18:51.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:50 smithi045 ceph-mon[31773]: pgmap v2852: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:51.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:50 smithi071 ceph-mon[36597]: pgmap v2852: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:51.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:50 smithi151 ceph-mon[36481]: pgmap v2852: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:18:51.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5184 sec 2024-03-28T19:18:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4539.313069 secs 2024-03-28T19:18:52.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4539.313069 secs 2024-03-28T19:18:52.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4539.313069 secs 2024-03-28T19:18:52.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5185 sec 2024-03-28T19:18:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:52 smithi045 ceph-mon[31773]: pgmap v2853: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:18:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3830.738466 secs 2024-03-28T19:18:53.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:52 smithi071 ceph-mon[36597]: pgmap v2853: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:18:53.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3830.738466 secs 2024-03-28T19:18:53.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:52 smithi151 ceph-mon[36481]: pgmap v2853: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:18:53.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3830.738466 secs 2024-03-28T19:18:53.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5186 sec 2024-03-28T19:18:54.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:53 smithi045 ceph-mon[31773]: pgmap v2854: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:54.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:53 smithi071 ceph-mon[36597]: pgmap v2854: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:54.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:53 smithi151 ceph-mon[36481]: pgmap v2854: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:54.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5187 sec 2024-03-28T19:18:55.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5188 sec 2024-03-28T19:18:56.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5189 sec 2024-03-28T19:18:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:56 smithi045 ceph-mon[31773]: pgmap v2855: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4544.313129 secs 2024-03-28T19:18:57.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:56 smithi071 ceph-mon[36597]: pgmap v2855: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:57.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4544.313129 secs 2024-03-28T19:18:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:56 smithi151 ceph-mon[36481]: pgmap v2855: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:18:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4544.313129 secs 2024-03-28T19:18:57.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5190 sec 2024-03-28T19:18:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3835.738506 secs 2024-03-28T19:18:58.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3835.738506 secs 2024-03-28T19:18:58.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3835.738506 secs 2024-03-28T19:18:58.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5191 sec 2024-03-28T19:18:59.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:18:58 smithi071 ceph-mon[36597]: pgmap v2856: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:59.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:18:58 smithi151 ceph-mon[36481]: pgmap v2856: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:59.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:18:58 smithi045 ceph-mon[31773]: pgmap v2856: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:18:59.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5192 sec 2024-03-28T19:19:00.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5193 sec 2024-03-28T19:19:01.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:00 smithi071 ceph-mon[36597]: pgmap v2857: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:01.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:00 smithi151 ceph-mon[36481]: pgmap v2857: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:01.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:00 smithi045 ceph-mon[31773]: pgmap v2857: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:01.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5194 sec 2024-03-28T19:19:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4549.313217 secs 2024-03-28T19:19:02.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4549.313217 secs 2024-03-28T19:19:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4549.313217 secs 2024-03-28T19:19:02.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5195 sec 2024-03-28T19:19:03.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:02 smithi071 ceph-mon[36597]: pgmap v2858: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:03.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3840.738549 secs 2024-03-28T19:19:03.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:02 smithi151 ceph-mon[36481]: pgmap v2858: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:03.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3840.738549 secs 2024-03-28T19:19:03.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:02 smithi045 ceph-mon[31773]: pgmap v2858: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:03.166 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3840.738549 secs 2024-03-28T19:19:03.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5196 sec 2024-03-28T19:19:04.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5197 sec 2024-03-28T19:19:05.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:04 smithi071 ceph-mon[36597]: pgmap v2859: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:05.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:04 smithi151 ceph-mon[36481]: pgmap v2859: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:05.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:04 smithi045 ceph-mon[31773]: pgmap v2859: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:05.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5198 sec 2024-03-28T19:19:06.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5199 sec 2024-03-28T19:19:07.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:06 smithi071 ceph-mon[36597]: pgmap v2860: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:07.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4554.313305 secs 2024-03-28T19:19:07.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:06 smithi151 ceph-mon[36481]: pgmap v2860: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:07.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4554.313305 secs 2024-03-28T19:19:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:06 smithi045 ceph-mon[31773]: pgmap v2860: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4554.313305 secs 2024-03-28T19:19:07.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5200 sec 2024-03-28T19:19:08.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3845.738634 secs 2024-03-28T19:19:08.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3845.738634 secs 2024-03-28T19:19:08.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3845.738634 secs 2024-03-28T19:19:08.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5201 sec 2024-03-28T19:19:09.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:08 smithi071 ceph-mon[36597]: pgmap v2861: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:09.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:08 smithi151 ceph-mon[36481]: pgmap v2861: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:09.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:08 smithi045 ceph-mon[31773]: pgmap v2861: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:09.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5202 sec 2024-03-28T19:19:10.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5203 sec 2024-03-28T19:19:11.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:10 smithi071 ceph-mon[36597]: pgmap v2862: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:11.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:10 smithi151 ceph-mon[36481]: pgmap v2862: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:11.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:10 smithi045 ceph-mon[31773]: pgmap v2862: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:11.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5204 sec 2024-03-28T19:19:12.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4559.313393 secs 2024-03-28T19:19:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4559.313393 secs 2024-03-28T19:19:12.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4559.313393 secs 2024-03-28T19:19:12.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5205 sec 2024-03-28T19:19:13.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:12 smithi071 ceph-mon[36597]: pgmap v2863: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:13.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3850.738723 secs 2024-03-28T19:19:13.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:12 smithi151 ceph-mon[36481]: pgmap v2863: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:13.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3850.738723 secs 2024-03-28T19:19:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:12 smithi045 ceph-mon[31773]: pgmap v2863: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3850.738723 secs 2024-03-28T19:19:13.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5206 sec 2024-03-28T19:19:14.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5207 sec 2024-03-28T19:19:15.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:14 smithi071 ceph-mon[36597]: pgmap v2864: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:15.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:14 smithi151 ceph-mon[36481]: pgmap v2864: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:15.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:14 smithi045 ceph-mon[31773]: pgmap v2864: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:15.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5208 sec 2024-03-28T19:19:16.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5209 sec 2024-03-28T19:19:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:16 smithi071 ceph-mon[36597]: pgmap v2865: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:17.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4564.313455 secs 2024-03-28T19:19:17.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:16 smithi151 ceph-mon[36481]: pgmap v2865: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:17.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4564.313455 secs 2024-03-28T19:19:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:16 smithi045 ceph-mon[31773]: pgmap v2865: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4564.313455 secs 2024-03-28T19:19:17.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5210 sec 2024-03-28T19:19:18.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3855.738812 secs 2024-03-28T19:19:18.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3855.738812 secs 2024-03-28T19:19:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3855.738812 secs 2024-03-28T19:19:18.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5211 sec 2024-03-28T19:19:19.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:18 smithi071 ceph-mon[36597]: pgmap v2866: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:19.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:18 smithi045 ceph-mon[31773]: pgmap v2866: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:19.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:18 smithi151 ceph-mon[36481]: pgmap v2866: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:19.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5212 sec 2024-03-28T19:19:20.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5213 sec 2024-03-28T19:19:21.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:20 smithi071 ceph-mon[36597]: pgmap v2867: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:21.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:20 smithi151 ceph-mon[36481]: pgmap v2867: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:21.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:20 smithi045 ceph-mon[31773]: pgmap v2867: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:21.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5214 sec 2024-03-28T19:19:22.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4569.313490 secs 2024-03-28T19:19:22.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4569.313490 secs 2024-03-28T19:19:22.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4569.313490 secs 2024-03-28T19:19:22.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5215 sec 2024-03-28T19:19:23.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:22 smithi071 ceph-mon[36597]: pgmap v2868: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:23.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3860.738849 secs 2024-03-28T19:19:23.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:22 smithi151 ceph-mon[36481]: pgmap v2868: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:23.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3860.738849 secs 2024-03-28T19:19:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:22 smithi045 ceph-mon[31773]: pgmap v2868: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3860.738849 secs 2024-03-28T19:19:23.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5216 sec 2024-03-28T19:19:24.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5217 sec 2024-03-28T19:19:25.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:24 smithi071 ceph-mon[36597]: pgmap v2869: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:25.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:24 smithi151 ceph-mon[36481]: pgmap v2869: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:25.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:24 smithi045 ceph-mon[31773]: pgmap v2869: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:25.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5218 sec 2024-03-28T19:19:26.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5219 sec 2024-03-28T19:19:27.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:26 smithi071 ceph-mon[36597]: pgmap v2870: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:27.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4574.313551 secs 2024-03-28T19:19:27.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:26 smithi151 ceph-mon[36481]: pgmap v2870: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:27.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4574.313551 secs 2024-03-28T19:19:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:26 smithi045 ceph-mon[31773]: pgmap v2870: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4574.313551 secs 2024-03-28T19:19:27.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5220 sec 2024-03-28T19:19:28.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3865.738912 secs 2024-03-28T19:19:28.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3865.738912 secs 2024-03-28T19:19:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3865.738912 secs 2024-03-28T19:19:28.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5221 sec 2024-03-28T19:19:29.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:28 smithi071 ceph-mon[36597]: pgmap v2871: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:29.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:28 smithi151 ceph-mon[36481]: pgmap v2871: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:29.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:28 smithi045 ceph-mon[31773]: pgmap v2871: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:29.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5222 sec 2024-03-28T19:19:30.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5223 sec 2024-03-28T19:19:31.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:30 smithi071 ceph-mon[36597]: pgmap v2872: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:31.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:30 smithi151 ceph-mon[36481]: pgmap v2872: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:31.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:30 smithi045 ceph-mon[31773]: pgmap v2872: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:31.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5224 sec 2024-03-28T19:19:32.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4579.313588 secs 2024-03-28T19:19:32.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4579.313588 secs 2024-03-28T19:19:32.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4579.313588 secs 2024-03-28T19:19:32.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5225 sec 2024-03-28T19:19:33.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:32 smithi151 ceph-mon[36481]: pgmap v2873: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:33.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3870.738972 secs 2024-03-28T19:19:33.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:32 smithi045 ceph-mon[31773]: pgmap v2873: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:33.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3870.738972 secs 2024-03-28T19:19:33.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:32 smithi071 ceph-mon[36597]: pgmap v2873: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:33.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3870.738972 secs 2024-03-28T19:19:33.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5226 sec 2024-03-28T19:19:34.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5227 sec 2024-03-28T19:19:35.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:34 smithi151 ceph-mon[36481]: pgmap v2874: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:35.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:34 smithi045 ceph-mon[31773]: pgmap v2874: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:35.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:34 smithi071 ceph-mon[36597]: pgmap v2874: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:35.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5228 sec 2024-03-28T19:19:36.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5229 sec 2024-03-28T19:19:37.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:36 smithi151 ceph-mon[36481]: pgmap v2875: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:37.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4584.313675 secs 2024-03-28T19:19:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:36 smithi045 ceph-mon[31773]: pgmap v2875: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4584.313675 secs 2024-03-28T19:19:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:36 smithi071 ceph-mon[36597]: pgmap v2875: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:37.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4584.313675 secs 2024-03-28T19:19:37.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5230 sec 2024-03-28T19:19:38.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3875.738989 secs 2024-03-28T19:19:38.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3875.738989 secs 2024-03-28T19:19:38.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3875.738989 secs 2024-03-28T19:19:38.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5231 sec 2024-03-28T19:19:39.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:38 smithi045 ceph-mon[31773]: pgmap v2876: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:39.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:38 smithi071 ceph-mon[36597]: pgmap v2876: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:39.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:38 smithi151 ceph-mon[36481]: pgmap v2876: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:39.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5232 sec 2024-03-28T19:19:40.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5233 sec 2024-03-28T19:19:41.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:40 smithi045 ceph-mon[31773]: pgmap v2877: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:41.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:40 smithi071 ceph-mon[36597]: pgmap v2877: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:41.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:40 smithi151 ceph-mon[36481]: pgmap v2877: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:41.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5234 sec 2024-03-28T19:19:42.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4589.313764 secs 2024-03-28T19:19:42.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4589.313764 secs 2024-03-28T19:19:42.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4589.313764 secs 2024-03-28T19:19:42.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5235 sec 2024-03-28T19:19:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:42 smithi045 ceph-mon[31773]: pgmap v2878: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3880.739050 secs 2024-03-28T19:19:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:42 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:19:43.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:42 smithi071 ceph-mon[36597]: pgmap v2878: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:43.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3880.739050 secs 2024-03-28T19:19:43.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:42 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:19:43.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:42 smithi151 ceph-mon[36481]: pgmap v2878: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:43.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3880.739050 secs 2024-03-28T19:19:43.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:42 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:19:43.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5236 sec 2024-03-28T19:19:44.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:44 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:19:44.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:44 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:19:44.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:44 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:19:44.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:44 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:19:44.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:44 smithi071 ceph-mon[36597]: pgmap v2879: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:44.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:44 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:19:44.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:44 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:19:44.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:44 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:19:44.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:44 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:19:44.561 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:44 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:19:44.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:44 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:19:44.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:44 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:19:44.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:44 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:19:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:44 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:19:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:44 smithi151 ceph-mon[36481]: pgmap v2879: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:44 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:19:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:44 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:19:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:44 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:19:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:44 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:19:44.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:44 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:19:44.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5237 sec 2024-03-28T19:19:44.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:44 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:19:44.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:44 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:19:44.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:44 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:19:44.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:44 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:19:44.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:44 smithi045 ceph-mon[31773]: pgmap v2879: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:44.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:44 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:19:44.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:44 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:19:44.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:44 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:19:44.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:44 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:19:44.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:44 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:19:45.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:45 smithi071 ceph-mon[36597]: pgmap v2880: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T19:19:45.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:45 smithi071 ceph-mon[36597]: pgmap v2881: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:19:45.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:45 smithi151 ceph-mon[36481]: pgmap v2880: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T19:19:45.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:45 smithi151 ceph-mon[36481]: pgmap v2881: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:19:45.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5238 sec 2024-03-28T19:19:45.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:45 smithi045 ceph-mon[31773]: pgmap v2880: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-28T19:19:45.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:45 smithi045 ceph-mon[31773]: pgmap v2881: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T19:19:46.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5239 sec 2024-03-28T19:19:47.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:47 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4594.313824 secs 2024-03-28T19:19:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:47 smithi151 ceph-mon[36481]: pgmap v2882: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:19:47.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3885.739110 secs 2024-03-28T19:19:47.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5240 sec 2024-03-28T19:19:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:47 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4594.313824 secs 2024-03-28T19:19:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:47 smithi045 ceph-mon[31773]: pgmap v2882: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:19:47.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3885.739110 secs 2024-03-28T19:19:47.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:47 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4594.313824 secs 2024-03-28T19:19:47.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:47 smithi071 ceph-mon[36597]: pgmap v2882: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:19:47.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3885.739110 secs 2024-03-28T19:19:48.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5241 sec 2024-03-28T19:19:49.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:49 smithi151 ceph-mon[36481]: pgmap v2883: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:19:49.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5242 sec 2024-03-28T19:19:49.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:49 smithi045 ceph-mon[31773]: pgmap v2883: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:19:49.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:49 smithi071 ceph-mon[36597]: pgmap v2883: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:19:50.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5243 sec 2024-03-28T19:19:51.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5244 sec 2024-03-28T19:19:51.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:51 smithi045 ceph-mon[31773]: pgmap v2884: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:19:51.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4599.313883 secs 2024-03-28T19:19:51.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:51 smithi071 ceph-mon[36597]: pgmap v2884: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:19:51.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4599.313883 secs 2024-03-28T19:19:51.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:51 smithi151 ceph-mon[36481]: pgmap v2884: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:19:51.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4599.313883 secs 2024-03-28T19:19:52.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5245 sec 2024-03-28T19:19:52.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3890.739194 secs 2024-03-28T19:19:52.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3890.739194 secs 2024-03-28T19:19:52.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3890.739194 secs 2024-03-28T19:19:53.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5246 sec 2024-03-28T19:19:53.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:53 smithi045 ceph-mon[31773]: pgmap v2885: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:19:53.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:53 smithi071 ceph-mon[36597]: pgmap v2885: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:19:53.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:53 smithi151 ceph-mon[36481]: pgmap v2885: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T19:19:54.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5247 sec 2024-03-28T19:19:55.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5248 sec 2024-03-28T19:19:55.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:55 smithi045 ceph-mon[31773]: pgmap v2886: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:19:55.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:55 smithi071 ceph-mon[36597]: pgmap v2886: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:19:55.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:55 smithi151 ceph-mon[36481]: pgmap v2886: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:19:56.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5249 sec 2024-03-28T19:19:56.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4604.313968 secs 2024-03-28T19:19:56.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4604.313968 secs 2024-03-28T19:19:56.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4604.313968 secs 2024-03-28T19:19:57.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5250 sec 2024-03-28T19:19:57.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:57 smithi045 ceph-mon[31773]: pgmap v2887: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:57.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3895.739256 secs 2024-03-28T19:19:57.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:57 smithi071 ceph-mon[36597]: pgmap v2887: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:57.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3895.739256 secs 2024-03-28T19:19:57.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:57 smithi151 ceph-mon[36481]: pgmap v2887: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:19:57.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3895.739256 secs 2024-03-28T19:19:58.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5251 sec 2024-03-28T19:19:59.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5252 sec 2024-03-28T19:19:59.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:19:59 smithi045 ceph-mon[31773]: pgmap v2888: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:59.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:19:59 smithi071 ceph-mon[36597]: pgmap v2888: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:19:59.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:19:59 smithi151 ceph-mon[36481]: pgmap v2888: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:00.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5253 sec 2024-03-28T19:20:00.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:00 smithi045 ceph-mon[31773]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-03-28T19:20:00.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:00 smithi071 ceph-mon[36597]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-03-28T19:20:00.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:00 smithi151 ceph-mon[36481]: overall HEALTH_WARN 8 failed cephadm daemon(s); 2 MDSs report slow requests 2024-03-28T19:20:01.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5254 sec 2024-03-28T19:20:01.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:01 smithi045 ceph-mon[31773]: pgmap v2889: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:01.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4609.314056 secs 2024-03-28T19:20:01.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:01 smithi071 ceph-mon[36597]: pgmap v2889: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:01.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4609.314056 secs 2024-03-28T19:20:01.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:01 smithi151 ceph-mon[36481]: pgmap v2889: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:01.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4609.314056 secs 2024-03-28T19:20:02.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5255 sec 2024-03-28T19:20:02.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3900.739343 secs 2024-03-28T19:20:02.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3900.739343 secs 2024-03-28T19:20:02.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3900.739343 secs 2024-03-28T19:20:03.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5256 sec 2024-03-28T19:20:03.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:03 smithi045 ceph-mon[31773]: pgmap v2890: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:03.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:03 smithi071 ceph-mon[36597]: pgmap v2890: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:03.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:03 smithi151 ceph-mon[36481]: pgmap v2890: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:04.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5257 sec 2024-03-28T19:20:05.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5258 sec 2024-03-28T19:20:05.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:05 smithi045 ceph-mon[31773]: pgmap v2891: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:05.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:05 smithi071 ceph-mon[36597]: pgmap v2891: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:05.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:05 smithi151 ceph-mon[36481]: pgmap v2891: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:06.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5259 sec 2024-03-28T19:20:06.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4614.314070 secs 2024-03-28T19:20:06.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4614.314070 secs 2024-03-28T19:20:06.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4614.314070 secs 2024-03-28T19:20:07.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5260 sec 2024-03-28T19:20:07.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:07 smithi045 ceph-mon[31773]: pgmap v2892: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:07.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3905.739405 secs 2024-03-28T19:20:07.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:07 smithi071 ceph-mon[36597]: pgmap v2892: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:07.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3905.739405 secs 2024-03-28T19:20:07.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:07 smithi151 ceph-mon[36481]: pgmap v2892: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:07.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3905.739405 secs 2024-03-28T19:20:08.580 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5261 sec 2024-03-28T19:20:09.580 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5262 sec 2024-03-28T19:20:09.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:09 smithi045 ceph-mon[31773]: pgmap v2893: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:09.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:09 smithi071 ceph-mon[36597]: pgmap v2893: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:09.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:09 smithi151 ceph-mon[36481]: pgmap v2893: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:10.580 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5263 sec 2024-03-28T19:20:11.580 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5264 sec 2024-03-28T19:20:11.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:11 smithi045 ceph-mon[31773]: pgmap v2894: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:11.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4619.314158 secs 2024-03-28T19:20:11.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:11 smithi071 ceph-mon[36597]: pgmap v2894: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:11.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4619.314158 secs 2024-03-28T19:20:11.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:11 smithi151 ceph-mon[36481]: pgmap v2894: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:11.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4619.314158 secs 2024-03-28T19:20:12.580 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5265 sec 2024-03-28T19:20:12.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3910.739493 secs 2024-03-28T19:20:12.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3910.739493 secs 2024-03-28T19:20:12.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3910.739493 secs 2024-03-28T19:20:13.580 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5266 sec 2024-03-28T19:20:13.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:13 smithi071 ceph-mon[36597]: pgmap v2895: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:13.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:13 smithi151 ceph-mon[36481]: pgmap v2895: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:13.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:13 smithi045 ceph-mon[31773]: pgmap v2895: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:14.580 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5267 sec 2024-03-28T19:20:15.580 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5268 sec 2024-03-28T19:20:15.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:15 smithi071 ceph-mon[36597]: pgmap v2896: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:15.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:15 smithi151 ceph-mon[36481]: pgmap v2896: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:15.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:15 smithi045 ceph-mon[31773]: pgmap v2896: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:16.581 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5269 sec 2024-03-28T19:20:16.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4624.314246 secs 2024-03-28T19:20:16.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4624.314246 secs 2024-03-28T19:20:16.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4624.314246 secs 2024-03-28T19:20:17.581 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5270 sec 2024-03-28T19:20:17.710 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:17 smithi071 ceph-mon[36597]: pgmap v2897: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:17.710 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3915.739581 secs 2024-03-28T19:20:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:17 smithi151 ceph-mon[36481]: pgmap v2897: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3915.739581 secs 2024-03-28T19:20:17.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:17 smithi045 ceph-mon[31773]: pgmap v2897: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:17.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3915.739581 secs 2024-03-28T19:20:18.581 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5271 sec 2024-03-28T19:20:19.581 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5272 sec 2024-03-28T19:20:19.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:19 smithi045 ceph-mon[31773]: pgmap v2898: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:19.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:19 smithi071 ceph-mon[36597]: pgmap v2898: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:19.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:19 smithi151 ceph-mon[36481]: pgmap v2898: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:20.581 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5273 sec 2024-03-28T19:20:21.581 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5274 sec 2024-03-28T19:20:21.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:21 smithi071 ceph-mon[36597]: pgmap v2899: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:21.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4629.314326 secs 2024-03-28T19:20:21.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:21 smithi151 ceph-mon[36481]: pgmap v2899: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:21.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4629.314326 secs 2024-03-28T19:20:21.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:21 smithi045 ceph-mon[31773]: pgmap v2899: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:21.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4629.314326 secs 2024-03-28T19:20:22.581 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5275 sec 2024-03-28T19:20:22.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3920.739669 secs 2024-03-28T19:20:22.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3920.739669 secs 2024-03-28T19:20:22.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3920.739669 secs 2024-03-28T19:20:23.581 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5276 sec 2024-03-28T19:20:23.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:23 smithi071 ceph-mon[36597]: pgmap v2900: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:23.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:23 smithi151 ceph-mon[36481]: pgmap v2900: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:23.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:23 smithi045 ceph-mon[31773]: pgmap v2900: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:24.582 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5277 sec 2024-03-28T19:20:25.582 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5278 sec 2024-03-28T19:20:25.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:25 smithi071 ceph-mon[36597]: pgmap v2901: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:25.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:25 smithi151 ceph-mon[36481]: pgmap v2901: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:25.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:25 smithi045 ceph-mon[31773]: pgmap v2901: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:26.582 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5279 sec 2024-03-28T19:20:26.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4634.314363 secs 2024-03-28T19:20:26.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4634.314363 secs 2024-03-28T19:20:26.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4634.314363 secs 2024-03-28T19:20:27.582 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5280 sec 2024-03-28T19:20:27.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:27 smithi071 ceph-mon[36597]: pgmap v2902: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:27.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3925.739755 secs 2024-03-28T19:20:27.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:27 smithi151 ceph-mon[36481]: pgmap v2902: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:27.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3925.739755 secs 2024-03-28T19:20:27.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:27 smithi045 ceph-mon[31773]: pgmap v2902: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:27.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3925.739755 secs 2024-03-28T19:20:28.582 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5281 sec 2024-03-28T19:20:29.582 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5282 sec 2024-03-28T19:20:29.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:29 smithi071 ceph-mon[36597]: pgmap v2903: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:29.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:29 smithi151 ceph-mon[36481]: pgmap v2903: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:29.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:29 smithi045 ceph-mon[31773]: pgmap v2903: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:30.582 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5283 sec 2024-03-28T19:20:31.583 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5284 sec 2024-03-28T19:20:31.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:31 smithi071 ceph-mon[36597]: pgmap v2904: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:31.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4639.314425 secs 2024-03-28T19:20:31.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:31 smithi151 ceph-mon[36481]: pgmap v2904: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:31.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4639.314425 secs 2024-03-28T19:20:31.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:31 smithi045 ceph-mon[31773]: pgmap v2904: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:31.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4639.314425 secs 2024-03-28T19:20:32.583 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5285 sec 2024-03-28T19:20:32.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3930.739792 secs 2024-03-28T19:20:32.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3930.739792 secs 2024-03-28T19:20:32.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3930.739792 secs 2024-03-28T19:20:33.583 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5286 sec 2024-03-28T19:20:33.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:33 smithi071 ceph-mon[36597]: pgmap v2905: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:33.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:33 smithi151 ceph-mon[36481]: pgmap v2905: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:33.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:33 smithi045 ceph-mon[31773]: pgmap v2905: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:34.583 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5287 sec 2024-03-28T19:20:35.583 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5288 sec 2024-03-28T19:20:35.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:35 smithi071 ceph-mon[36597]: pgmap v2906: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:35.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:35 smithi151 ceph-mon[36481]: pgmap v2906: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:35.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:35 smithi045 ceph-mon[31773]: pgmap v2906: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:36.583 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5289 sec 2024-03-28T19:20:36.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4644.314486 secs 2024-03-28T19:20:36.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4644.314486 secs 2024-03-28T19:20:36.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4644.314486 secs 2024-03-28T19:20:37.583 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5290 sec 2024-03-28T19:20:37.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:37 smithi071 ceph-mon[36597]: pgmap v2907: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:37.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:37 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3935.739884 secs 2024-03-28T19:20:37.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:37 smithi151 ceph-mon[36481]: pgmap v2907: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:37.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:37 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3935.739884 secs 2024-03-28T19:20:37.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:37 smithi045 ceph-mon[31773]: pgmap v2907: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:37.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:37 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3935.739884 secs 2024-03-28T19:20:38.583 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5291 sec 2024-03-28T19:20:39.584 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5292 sec 2024-03-28T19:20:39.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:39 smithi071 ceph-mon[36597]: pgmap v2908: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:39.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:39 smithi151 ceph-mon[36481]: pgmap v2908: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:39.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:39 smithi045 ceph-mon[31773]: pgmap v2908: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:40.584 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5293 sec 2024-03-28T19:20:41.584 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5294 sec 2024-03-28T19:20:41.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:41 smithi071 ceph-mon[36597]: pgmap v2909: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:41.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4649.314498 secs 2024-03-28T19:20:41.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:41 smithi151 ceph-mon[36481]: pgmap v2909: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:41.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4649.314498 secs 2024-03-28T19:20:41.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:41 smithi045 ceph-mon[31773]: pgmap v2909: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:20:41.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4649.314498 secs 2024-03-28T19:20:42.584 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5295 sec 2024-03-28T19:20:42.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:42 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3940.739967 secs 2024-03-28T19:20:42.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:42 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3940.739967 secs 2024-03-28T19:20:42.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:42 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3940.739967 secs 2024-03-28T19:20:43.584 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5296 sec 2024-03-28T19:20:43.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:43 smithi071 ceph-mon[36597]: pgmap v2910: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:43.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:43 smithi151 ceph-mon[36481]: pgmap v2910: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:43.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:43 smithi045 ceph-mon[31773]: pgmap v2910: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:44.585 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5297 sec 2024-03-28T19:20:44.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:44 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:20:44.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:44 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:20:44.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:44 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:20:45.584 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5298 sec 2024-03-28T19:20:45.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:45 smithi151 ceph-mon[36481]: pgmap v2911: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:45.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:45 smithi045 ceph-mon[31773]: pgmap v2911: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:46.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:45 smithi071 ceph-mon[36597]: pgmap v2911: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:46.584 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5299 sec 2024-03-28T19:20:46.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:46 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:20:46.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:46 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:20:46.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:46 smithi151 ceph-mon[36481]: pgmap v2912: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:20:46.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:46 smithi151 ceph-mon[36481]: pgmap v2913: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:46.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:46 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:20:46.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4654.314515 secs 2024-03-28T19:20:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:46 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:20:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:46 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:20:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:46 smithi045 ceph-mon[31773]: pgmap v2912: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:20:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:46 smithi045 ceph-mon[31773]: pgmap v2913: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:46 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:20:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4654.314515 secs 2024-03-28T19:20:47.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:46 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:20:47.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:46 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:20:47.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:46 smithi071 ceph-mon[36597]: pgmap v2912: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:20:47.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:46 smithi071 ceph-mon[36597]: pgmap v2913: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:46 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:20:47.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4654.314515 secs 2024-03-28T19:20:47.585 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5300 sec 2024-03-28T19:20:47.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:47 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3945.740030 secs 2024-03-28T19:20:47.960 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:47 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3945.740030 secs 2024-03-28T19:20:48.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:47 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3945.740030 secs 2024-03-28T19:20:48.585 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5301 sec 2024-03-28T19:20:48.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:48 smithi045 ceph-mon[31773]: pgmap v2914: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:48.973 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:48 smithi071 ceph-mon[36597]: pgmap v2914: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:49.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:48 smithi151 ceph-mon[36481]: pgmap v2914: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:49.585 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5302 sec 2024-03-28T19:20:50.585 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5303 sec 2024-03-28T19:20:51.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:50 smithi071 ceph-mon[36597]: pgmap v2915: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:20:51.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:50 smithi151 ceph-mon[36481]: pgmap v2915: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:20:51.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:50 smithi045 ceph-mon[31773]: pgmap v2915: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:20:51.585 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5304 sec 2024-03-28T19:20:52.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4659.314578 secs 2024-03-28T19:20:52.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4659.314578 secs 2024-03-28T19:20:52.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4659.314578 secs 2024-03-28T19:20:52.585 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5305 sec 2024-03-28T19:20:53.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:52 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3950.740089 secs 2024-03-28T19:20:53.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:52 smithi071 ceph-mon[36597]: pgmap v2916: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:53.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:52 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3950.740089 secs 2024-03-28T19:20:53.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:52 smithi151 ceph-mon[36481]: pgmap v2916: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:52 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3950.740089 secs 2024-03-28T19:20:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:52 smithi045 ceph-mon[31773]: pgmap v2916: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:53.585 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5306 sec 2024-03-28T19:20:54.586 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5307 sec 2024-03-28T19:20:55.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:54 smithi071 ceph-mon[36597]: pgmap v2917: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:55.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:54 smithi151 ceph-mon[36481]: pgmap v2917: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:55.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:54 smithi045 ceph-mon[31773]: pgmap v2917: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:20:55.586 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5308 sec 2024-03-28T19:20:56.586 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5309 sec 2024-03-28T19:20:57.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:56 smithi071 ceph-mon[36597]: pgmap v2918: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:20:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4664.314641 secs 2024-03-28T19:20:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:56 smithi151 ceph-mon[36481]: pgmap v2918: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:20:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4664.314641 secs 2024-03-28T19:20:57.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:56 smithi045 ceph-mon[31773]: pgmap v2918: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:20:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4664.314641 secs 2024-03-28T19:20:57.586 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5310 sec 2024-03-28T19:20:58.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:57 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3955.740150 secs 2024-03-28T19:20:58.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:57 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3955.740150 secs 2024-03-28T19:20:58.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:57 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3955.740150 secs 2024-03-28T19:20:58.586 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5311 sec 2024-03-28T19:20:59.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:20:58 smithi071 ceph-mon[36597]: pgmap v2919: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:59.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:20:58 smithi151 ceph-mon[36481]: pgmap v2919: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:59.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:20:58 smithi045 ceph-mon[31773]: pgmap v2919: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:20:59.586 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5312 sec 2024-03-28T19:21:00.586 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5313 sec 2024-03-28T19:21:00.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:00 smithi071 ceph-mon[36597]: pgmap v2920: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:00.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:00 smithi151 ceph-mon[36481]: pgmap v2920: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:00.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:00 smithi045 ceph-mon[31773]: pgmap v2920: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:01.587 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5314 sec 2024-03-28T19:21:01.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4669.314702 secs 2024-03-28T19:21:01.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4669.314702 secs 2024-03-28T19:21:01.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4669.314702 secs 2024-03-28T19:21:02.587 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5315 sec 2024-03-28T19:21:02.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:02 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3960.740238 secs 2024-03-28T19:21:02.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:02 smithi071 ceph-mon[36597]: pgmap v2921: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:02.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:02 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3960.740238 secs 2024-03-28T19:21:02.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:02 smithi151 ceph-mon[36481]: pgmap v2921: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:02.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:02 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3960.740238 secs 2024-03-28T19:21:02.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:02 smithi045 ceph-mon[31773]: pgmap v2921: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:03.587 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5316 sec 2024-03-28T19:21:04.587 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5317 sec 2024-03-28T19:21:05.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:04 smithi071 ceph-mon[36597]: pgmap v2922: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:05.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:04 smithi151 ceph-mon[36481]: pgmap v2922: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:05.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:04 smithi045 ceph-mon[31773]: pgmap v2922: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:05.587 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5318 sec 2024-03-28T19:21:06.587 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5319 sec 2024-03-28T19:21:07.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:06 smithi071 ceph-mon[36597]: pgmap v2923: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:07.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4674.314765 secs 2024-03-28T19:21:07.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:06 smithi151 ceph-mon[36481]: pgmap v2923: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:07.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4674.314765 secs 2024-03-28T19:21:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:06 smithi045 ceph-mon[31773]: pgmap v2923: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4674.314765 secs 2024-03-28T19:21:07.587 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5320 sec 2024-03-28T19:21:08.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:07 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3965.740299 secs 2024-03-28T19:21:08.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:07 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3965.740299 secs 2024-03-28T19:21:08.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:07 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3965.740299 secs 2024-03-28T19:21:08.587 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5321 sec 2024-03-28T19:21:09.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:08 smithi071 ceph-mon[36597]: pgmap v2924: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:09.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:08 smithi151 ceph-mon[36481]: pgmap v2924: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:09.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:08 smithi045 ceph-mon[31773]: pgmap v2924: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:09.588 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5322 sec 2024-03-28T19:21:10.588 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5323 sec 2024-03-28T19:21:11.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:10 smithi071 ceph-mon[36597]: pgmap v2925: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:11.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:10 smithi151 ceph-mon[36481]: pgmap v2925: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:11.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:10 smithi045 ceph-mon[31773]: pgmap v2925: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:11.588 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5324 sec 2024-03-28T19:21:12.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4679.314853 secs 2024-03-28T19:21:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4679.314853 secs 2024-03-28T19:21:12.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4679.314853 secs 2024-03-28T19:21:12.588 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5325 sec 2024-03-28T19:21:13.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:12 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3970.740313 secs 2024-03-28T19:21:13.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:12 smithi071 ceph-mon[36597]: pgmap v2926: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:13.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:12 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3970.740313 secs 2024-03-28T19:21:13.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:12 smithi151 ceph-mon[36481]: pgmap v2926: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:13.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:12 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3970.740313 secs 2024-03-28T19:21:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:12 smithi045 ceph-mon[31773]: pgmap v2926: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:13.588 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5326 sec 2024-03-28T19:21:14.588 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5327 sec 2024-03-28T19:21:15.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:14 smithi071 ceph-mon[36597]: pgmap v2927: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:15.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:14 smithi151 ceph-mon[36481]: pgmap v2927: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:15.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:14 smithi045 ceph-mon[31773]: pgmap v2927: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:15.588 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5328 sec 2024-03-28T19:21:16.589 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5329 sec 2024-03-28T19:21:17.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:16 smithi071 ceph-mon[36597]: pgmap v2928: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:17.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4684.314942 secs 2024-03-28T19:21:17.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:16 smithi151 ceph-mon[36481]: pgmap v2928: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:17.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4684.314942 secs 2024-03-28T19:21:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:16 smithi045 ceph-mon[31773]: pgmap v2928: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4684.314942 secs 2024-03-28T19:21:17.589 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5330 sec 2024-03-28T19:21:18.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:17 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3975.740400 secs 2024-03-28T19:21:18.074 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:17 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3975.740400 secs 2024-03-28T19:21:18.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:17 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3975.740400 secs 2024-03-28T19:21:18.589 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5331 sec 2024-03-28T19:21:19.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:18 smithi071 ceph-mon[36597]: pgmap v2929: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:19.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:18 smithi151 ceph-mon[36481]: pgmap v2929: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:19.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:18 smithi045 ceph-mon[31773]: pgmap v2929: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:19.589 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5332 sec 2024-03-28T19:21:20.589 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5333 sec 2024-03-28T19:21:21.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:20 smithi151 ceph-mon[36481]: pgmap v2930: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:21.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:20 smithi045 ceph-mon[31773]: pgmap v2930: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:21.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:20 smithi071 ceph-mon[36597]: pgmap v2930: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:21.589 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5334 sec 2024-03-28T19:21:22.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4689.315027 secs 2024-03-28T19:21:22.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4689.315027 secs 2024-03-28T19:21:22.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4689.315027 secs 2024-03-28T19:21:22.590 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5335 sec 2024-03-28T19:21:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:22 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3980.740486 secs 2024-03-28T19:21:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:22 smithi045 ceph-mon[31773]: pgmap v2931: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:23.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:22 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3980.740486 secs 2024-03-28T19:21:23.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:22 smithi071 ceph-mon[36597]: pgmap v2931: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:23.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:22 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3980.740486 secs 2024-03-28T19:21:23.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:22 smithi151 ceph-mon[36481]: pgmap v2931: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:23.591 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5336 sec 2024-03-28T19:21:24.590 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5337 sec 2024-03-28T19:21:25.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:24 smithi045 ceph-mon[31773]: pgmap v2932: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:25.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:24 smithi071 ceph-mon[36597]: pgmap v2932: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:25.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:24 smithi151 ceph-mon[36481]: pgmap v2932: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:25.590 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5338 sec 2024-03-28T19:21:26.590 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5339 sec 2024-03-28T19:21:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:26 smithi045 ceph-mon[31773]: pgmap v2933: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4694.315116 secs 2024-03-28T19:21:27.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:26 smithi071 ceph-mon[36597]: pgmap v2933: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:27.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4694.315116 secs 2024-03-28T19:21:27.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:26 smithi151 ceph-mon[36481]: pgmap v2933: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:27.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4694.315116 secs 2024-03-28T19:21:27.590 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5340 sec 2024-03-28T19:21:28.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:27 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3985.740573 secs 2024-03-28T19:21:28.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:27 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3985.740573 secs 2024-03-28T19:21:28.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:27 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3985.740573 secs 2024-03-28T19:21:28.590 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5341 sec 2024-03-28T19:21:29.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:28 smithi045 ceph-mon[31773]: pgmap v2934: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:29.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:28 smithi071 ceph-mon[36597]: pgmap v2934: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:29.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:28 smithi151 ceph-mon[36481]: pgmap v2934: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:29.590 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5342 sec 2024-03-28T19:21:30.590 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5343 sec 2024-03-28T19:21:31.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:30 smithi045 ceph-mon[31773]: pgmap v2935: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:31.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:30 smithi071 ceph-mon[36597]: pgmap v2935: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:31.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:30 smithi151 ceph-mon[36481]: pgmap v2935: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:31.591 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5344 sec 2024-03-28T19:21:32.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4699.315203 secs 2024-03-28T19:21:32.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4699.315203 secs 2024-03-28T19:21:32.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4699.315203 secs 2024-03-28T19:21:32.591 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5345 sec 2024-03-28T19:21:33.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:32 smithi045 ceph-mon[31773]: 2 slow requests, 0 included below; oldest blocked for > 3990.740660 secs 2024-03-28T19:21:33.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:32 smithi045 ceph-mon[31773]: pgmap v2936: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:33.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:32 smithi071 ceph-mon[36597]: 2 slow requests, 0 included below; oldest blocked for > 3990.740660 secs 2024-03-28T19:21:33.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:32 smithi071 ceph-mon[36597]: pgmap v2936: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:33.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:32 smithi151 ceph-mon[36481]: 2 slow requests, 0 included below; oldest blocked for > 3990.740660 secs 2024-03-28T19:21:33.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:32 smithi151 ceph-mon[36481]: pgmap v2936: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:33.591 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5346 sec 2024-03-28T19:21:34.591 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5347 sec 2024-03-28T19:21:35.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:34 smithi045 ceph-mon[31773]: pgmap v2937: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:35.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:34 smithi071 ceph-mon[36597]: pgmap v2937: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:35.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:34 smithi151 ceph-mon[36481]: pgmap v2937: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:35.591 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5348 sec 2024-03-28T19:21:36.591 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5349 sec 2024-03-28T19:21:37.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:36 smithi045 ceph-mon[31773]: pgmap v2938: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:37.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4704.315293 secs 2024-03-28T19:21:37.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:36 smithi071 ceph-mon[36597]: pgmap v2938: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:37.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4704.315293 secs 2024-03-28T19:21:37.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:36 smithi151 ceph-mon[36481]: pgmap v2938: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:37.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4704.315293 secs 2024-03-28T19:21:37.592 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5350 sec 2024-03-28T19:21:38.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:37 smithi045 ceph-mon[31773]: 3 slow requests, 1 included below; oldest blocked for > 3995.740719 secs 2024-03-28T19:21:38.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:37 smithi045 ceph-mon[31773]: slow request 184.057332 seconds old, received at 2024-03-28T19:18:32.630514+0000: client_request(client.26759:1 getattr pAsLsXsFs #0x1 2024-03-28T19:18:32.626993+0000 caller_uid=0, caller_gid=0{}) currently joining batch getattr 2024-03-28T19:21:38.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:37 smithi071 ceph-mon[36597]: 3 slow requests, 1 included below; oldest blocked for > 3995.740719 secs 2024-03-28T19:21:38.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:37 smithi071 ceph-mon[36597]: slow request 184.057332 seconds old, received at 2024-03-28T19:18:32.630514+0000: client_request(client.26759:1 getattr pAsLsXsFs #0x1 2024-03-28T19:18:32.626993+0000 caller_uid=0, caller_gid=0{}) currently joining batch getattr 2024-03-28T19:21:38.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:37 smithi151 ceph-mon[36481]: 3 slow requests, 1 included below; oldest blocked for > 3995.740719 secs 2024-03-28T19:21:38.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:37 smithi151 ceph-mon[36481]: slow request 184.057332 seconds old, received at 2024-03-28T19:18:32.630514+0000: client_request(client.26759:1 getattr pAsLsXsFs #0x1 2024-03-28T19:18:32.626993+0000 caller_uid=0, caller_gid=0{}) currently joining batch getattr 2024-03-28T19:21:38.592 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5351 sec 2024-03-28T19:21:39.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:38 smithi045 ceph-mon[31773]: pgmap v2939: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:39.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:38 smithi071 ceph-mon[36597]: pgmap v2939: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:39.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:38 smithi151 ceph-mon[36481]: pgmap v2939: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:39.592 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5352 sec 2024-03-28T19:21:40.592 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5353 sec 2024-03-28T19:21:41.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:40 smithi045 ceph-mon[31773]: pgmap v2940: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:41.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:40 smithi071 ceph-mon[36597]: pgmap v2940: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:41.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:40 smithi151 ceph-mon[36481]: pgmap v2940: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:41.592 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5354 sec 2024-03-28T19:21:42.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4709.315381 secs 2024-03-28T19:21:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:41 smithi045 ceph-mon[31773]: mds.? [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] up:active 2024-03-28T19:21:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:41 smithi045 ceph-mon[31773]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T19:21:42.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4709.315381 secs 2024-03-28T19:21:42.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:41 smithi071 ceph-mon[36597]: mds.? [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] up:active 2024-03-28T19:21:42.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:41 smithi071 ceph-mon[36597]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T19:21:42.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4709.315381 secs 2024-03-28T19:21:42.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:41 smithi151 ceph-mon[36481]: mds.? [v2:172.21.15.71:6836/2008752655,v1:172.21.15.71:6837/2008752655] up:active 2024-03-28T19:21:42.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:41 smithi151 ceph-mon[36481]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-03-28T19:21:42.592 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5355 sec 2024-03-28T19:21:43.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:42 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4000.740780 secs 2024-03-28T19:21:43.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:42 smithi045 ceph-mon[31773]: pgmap v2941: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:43.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:42 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4000.740780 secs 2024-03-28T19:21:43.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:42 smithi071 ceph-mon[36597]: pgmap v2941: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:43.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:42 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4000.740780 secs 2024-03-28T19:21:43.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:42 smithi151 ceph-mon[36481]: pgmap v2941: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:43.592 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5356 sec 2024-03-28T19:21:44.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:43 smithi071 ceph-mon[36597]: pgmap v2942: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:44.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:43 smithi151 ceph-mon[36481]: pgmap v2942: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:44.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:43 smithi045 ceph-mon[31773]: pgmap v2942: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:44.595 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5357 sec 2024-03-28T19:21:45.593 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5358 sec 2024-03-28T19:21:46.593 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5359 sec 2024-03-28T19:21:47.025 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:46 smithi151 ceph-mon[36481]: pgmap v2943: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:47.025 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4714.315420 secs 2024-03-28T19:21:47.025 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:46 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:21:47.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:46 smithi071 ceph-mon[36597]: pgmap v2943: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:47.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4714.315420 secs 2024-03-28T19:21:47.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:46 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:21:47.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:46 smithi045 ceph-mon[31773]: pgmap v2943: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:21:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4714.315420 secs 2024-03-28T19:21:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:46 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:21:47.593 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5360 sec 2024-03-28T19:21:48.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:47 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4005.740894 secs 2024-03-28T19:21:48.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:47 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:21:48.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:47 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:21:48.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:47 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:21:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:47 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4005.740894 secs 2024-03-28T19:21:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:47 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:21:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:47 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:21:48.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:47 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:21:48.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:47 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4005.740894 secs 2024-03-28T19:21:48.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:47 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:21:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:47 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:21:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:47 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:21:48.593 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5361 sec 2024-03-28T19:21:49.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:48 smithi071 ceph-mon[36597]: pgmap v2944: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:49.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:48 smithi071 ceph-mon[36597]: pgmap v2945: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:49.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:48 smithi151 ceph-mon[36481]: pgmap v2944: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:49.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:48 smithi151 ceph-mon[36481]: pgmap v2945: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:49.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:48 smithi045 ceph-mon[31773]: pgmap v2944: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:21:49.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:48 smithi045 ceph-mon[31773]: pgmap v2945: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:49.593 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5362 sec 2024-03-28T19:21:50.593 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5363 sec 2024-03-28T19:21:51.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:50 smithi071 ceph-mon[36597]: pgmap v2946: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:51.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:50 smithi151 ceph-mon[36481]: pgmap v2946: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:51.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:50 smithi045 ceph-mon[31773]: pgmap v2946: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:51.593 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5364 sec 2024-03-28T19:21:52.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4719.315461 secs 2024-03-28T19:21:52.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4719.315461 secs 2024-03-28T19:21:52.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4719.315461 secs 2024-03-28T19:21:52.594 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5365 sec 2024-03-28T19:21:53.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:52 smithi071 ceph-mon[36597]: pgmap v2947: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:53.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:52 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4010.740956 secs 2024-03-28T19:21:53.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:52 smithi045 ceph-mon[31773]: pgmap v2947: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:53.071 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:52 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4010.740956 secs 2024-03-28T19:21:53.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:52 smithi151 ceph-mon[36481]: pgmap v2947: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:53.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:52 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4010.740956 secs 2024-03-28T19:21:53.594 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5366 sec 2024-03-28T19:21:54.594 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5367 sec 2024-03-28T19:21:55.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:54 smithi071 ceph-mon[36597]: pgmap v2948: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:55.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:54 smithi151 ceph-mon[36481]: pgmap v2948: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:55.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:54 smithi045 ceph-mon[31773]: pgmap v2948: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:55.594 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5368 sec 2024-03-28T19:21:56.594 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5369 sec 2024-03-28T19:21:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:56 smithi151 ceph-mon[36481]: pgmap v2949: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4724.315491 secs 2024-03-28T19:21:57.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:56 smithi045 ceph-mon[31773]: pgmap v2949: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4724.315491 secs 2024-03-28T19:21:57.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:56 smithi071 ceph-mon[36597]: pgmap v2949: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:21:57.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4724.315491 secs 2024-03-28T19:21:57.594 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5370 sec 2024-03-28T19:21:58.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:57 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4015.741017 secs 2024-03-28T19:21:58.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:57 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4015.741017 secs 2024-03-28T19:21:58.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:57 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4015.741017 secs 2024-03-28T19:21:58.594 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5371 sec 2024-03-28T19:21:59.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:21:58 smithi045 ceph-mon[31773]: pgmap v2950: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:21:59.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:21:58 smithi071 ceph-mon[36597]: pgmap v2950: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:21:59.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:21:58 smithi151 ceph-mon[36481]: pgmap v2950: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:21:59.595 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5372 sec 2024-03-28T19:22:00.595 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5373 sec 2024-03-28T19:22:01.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:00 smithi045 ceph-mon[31773]: pgmap v2951: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:01.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:00 smithi071 ceph-mon[36597]: pgmap v2951: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:01.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:00 smithi151 ceph-mon[36481]: pgmap v2951: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:01.595 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5374 sec 2024-03-28T19:22:02.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4729.315553 secs 2024-03-28T19:22:02.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4729.315553 secs 2024-03-28T19:22:02.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4729.315553 secs 2024-03-28T19:22:02.595 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5375 sec 2024-03-28T19:22:03.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:02 smithi045 ceph-mon[31773]: pgmap v2952: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:03.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:02 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4020.741102 secs 2024-03-28T19:22:03.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:02 smithi071 ceph-mon[36597]: pgmap v2952: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:03.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:02 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4020.741102 secs 2024-03-28T19:22:03.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:02 smithi151 ceph-mon[36481]: pgmap v2952: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:03.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:02 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4020.741102 secs 2024-03-28T19:22:03.596 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5376 sec 2024-03-28T19:22:04.595 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5377 sec 2024-03-28T19:22:05.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:04 smithi045 ceph-mon[31773]: pgmap v2953: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:05.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:04 smithi071 ceph-mon[36597]: pgmap v2953: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:05.321 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:04 smithi151 ceph-mon[36481]: pgmap v2953: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:05.595 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5378 sec 2024-03-28T19:22:06.596 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5379 sec 2024-03-28T19:22:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:06 smithi045 ceph-mon[31773]: pgmap v2954: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4734.315613 secs 2024-03-28T19:22:07.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:06 smithi071 ceph-mon[36597]: pgmap v2954: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:07.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4734.315613 secs 2024-03-28T19:22:07.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:06 smithi151 ceph-mon[36481]: pgmap v2954: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:07.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4734.315613 secs 2024-03-28T19:22:07.596 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5380 sec 2024-03-28T19:22:08.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:07 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4025.741172 secs 2024-03-28T19:22:08.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:07 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4025.741172 secs 2024-03-28T19:22:08.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:07 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4025.741172 secs 2024-03-28T19:22:08.596 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5381 sec 2024-03-28T19:22:09.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:08 smithi045 ceph-mon[31773]: pgmap v2955: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:09.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:08 smithi071 ceph-mon[36597]: pgmap v2955: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:09.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:08 smithi151 ceph-mon[36481]: pgmap v2955: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:09.596 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5382 sec 2024-03-28T19:22:10.596 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5383 sec 2024-03-28T19:22:11.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:10 smithi045 ceph-mon[31773]: pgmap v2956: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:11.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:10 smithi071 ceph-mon[36597]: pgmap v2956: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:11.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:10 smithi151 ceph-mon[36481]: pgmap v2956: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:11.596 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5384 sec 2024-03-28T19:22:12.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4739.315674 secs 2024-03-28T19:22:12.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4739.315674 secs 2024-03-28T19:22:12.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4739.315674 secs 2024-03-28T19:22:12.597 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5385 sec 2024-03-28T19:22:13.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:12 smithi045 ceph-mon[31773]: pgmap v2957: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:12 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4030.741258 secs 2024-03-28T19:22:13.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:12 smithi071 ceph-mon[36597]: pgmap v2957: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:13.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:12 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4030.741258 secs 2024-03-28T19:22:13.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:12 smithi151 ceph-mon[36481]: pgmap v2957: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:13.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:12 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4030.741258 secs 2024-03-28T19:22:13.597 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5386 sec 2024-03-28T19:22:14.597 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5387 sec 2024-03-28T19:22:15.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:14 smithi045 ceph-mon[31773]: pgmap v2958: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:15.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:14 smithi071 ceph-mon[36597]: pgmap v2958: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:15.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:14 smithi151 ceph-mon[36481]: pgmap v2958: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:15.597 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5388 sec 2024-03-28T19:22:16.597 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5389 sec 2024-03-28T19:22:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:16 smithi045 ceph-mon[31773]: pgmap v2959: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4744.315734 secs 2024-03-28T19:22:17.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:16 smithi071 ceph-mon[36597]: pgmap v2959: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:17.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4744.315734 secs 2024-03-28T19:22:17.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:16 smithi151 ceph-mon[36481]: pgmap v2959: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:17.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4744.315734 secs 2024-03-28T19:22:17.597 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5390 sec 2024-03-28T19:22:18.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:17 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4035.741346 secs 2024-03-28T19:22:18.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:17 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4035.741346 secs 2024-03-28T19:22:18.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:17 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4035.741346 secs 2024-03-28T19:22:18.597 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5391 sec 2024-03-28T19:22:19.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:18 smithi045 ceph-mon[31773]: pgmap v2960: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:19.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:18 smithi071 ceph-mon[36597]: pgmap v2960: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:19.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:18 smithi151 ceph-mon[36481]: pgmap v2960: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:19.597 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5392 sec 2024-03-28T19:22:20.598 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5393 sec 2024-03-28T19:22:21.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:20 smithi045 ceph-mon[31773]: pgmap v2961: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:21.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:20 smithi071 ceph-mon[36597]: pgmap v2961: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:21.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:20 smithi151 ceph-mon[36481]: pgmap v2961: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:21.598 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5394 sec 2024-03-28T19:22:22.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4749.315794 secs 2024-03-28T19:22:22.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4749.315794 secs 2024-03-28T19:22:22.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4749.315794 secs 2024-03-28T19:22:22.598 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5395 sec 2024-03-28T19:22:23.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:22 smithi045 ceph-mon[31773]: pgmap v2962: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:23.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:22 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4040.741408 secs 2024-03-28T19:22:23.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:22 smithi071 ceph-mon[36597]: pgmap v2962: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:23.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:22 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4040.741408 secs 2024-03-28T19:22:23.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:22 smithi151 ceph-mon[36481]: pgmap v2962: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:23.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:22 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4040.741408 secs 2024-03-28T19:22:23.598 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5396 sec 2024-03-28T19:22:24.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:23 smithi071 ceph-mon[36597]: pgmap v2963: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:24.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:23 smithi151 ceph-mon[36481]: pgmap v2963: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:24.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:23 smithi045 ceph-mon[31773]: pgmap v2963: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:24.598 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5397 sec 2024-03-28T19:22:25.598 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5398 sec 2024-03-28T19:22:26.598 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5399 sec 2024-03-28T19:22:26.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:26 smithi045 ceph-mon[31773]: pgmap v2964: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:26.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4754.315852 secs 2024-03-28T19:22:26.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:26 smithi071 ceph-mon[36597]: pgmap v2964: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:26.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4754.315852 secs 2024-03-28T19:22:26.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:26 smithi151 ceph-mon[36481]: pgmap v2964: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:26.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4754.315852 secs 2024-03-28T19:22:27.599 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5400 sec 2024-03-28T19:22:27.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:27 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4045.741468 secs 2024-03-28T19:22:27.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:27 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4045.741468 secs 2024-03-28T19:22:27.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:27 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4045.741468 secs 2024-03-28T19:22:28.599 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5401 sec 2024-03-28T19:22:28.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:28 smithi045 ceph-mon[31773]: pgmap v2965: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:28.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:28 smithi071 ceph-mon[36597]: pgmap v2965: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:28.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:28 smithi151 ceph-mon[36481]: pgmap v2965: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:29.599 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5402 sec 2024-03-28T19:22:30.599 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5403 sec 2024-03-28T19:22:30.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:30 smithi045 ceph-mon[31773]: pgmap v2966: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:30.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:30 smithi071 ceph-mon[36597]: pgmap v2966: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:30.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:30 smithi151 ceph-mon[36481]: pgmap v2966: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:31.599 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5404 sec 2024-03-28T19:22:31.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4759.315912 secs 2024-03-28T19:22:31.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4759.315912 secs 2024-03-28T19:22:31.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4759.315912 secs 2024-03-28T19:22:32.600 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5405 sec 2024-03-28T19:22:32.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:32 smithi045 ceph-mon[31773]: pgmap v2967: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:32.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:32 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4050.741527 secs 2024-03-28T19:22:32.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:32 smithi071 ceph-mon[36597]: pgmap v2967: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:32.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:32 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4050.741527 secs 2024-03-28T19:22:32.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:32 smithi151 ceph-mon[36481]: pgmap v2967: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:32.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:32 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4050.741527 secs 2024-03-28T19:22:33.599 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5406 sec 2024-03-28T19:22:34.600 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5407 sec 2024-03-28T19:22:34.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:34 smithi045 ceph-mon[31773]: pgmap v2968: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:34.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:34 smithi071 ceph-mon[36597]: pgmap v2968: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:34.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:34 smithi151 ceph-mon[36481]: pgmap v2968: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:35.600 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5408 sec 2024-03-28T19:22:36.600 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5409 sec 2024-03-28T19:22:36.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:36 smithi071 ceph-mon[36597]: pgmap v2969: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:36.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4764.315997 secs 2024-03-28T19:22:36.821 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:36 smithi151 ceph-mon[36481]: pgmap v2969: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:36.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4764.315997 secs 2024-03-28T19:22:36.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:36 smithi045 ceph-mon[31773]: pgmap v2969: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:36.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4764.315997 secs 2024-03-28T19:22:37.600 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5410 sec 2024-03-28T19:22:37.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:37 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4055.741564 secs 2024-03-28T19:22:37.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:37 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4055.741564 secs 2024-03-28T19:22:37.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:37 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4055.741564 secs 2024-03-28T19:22:38.600 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5411 sec 2024-03-28T19:22:38.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:38 smithi071 ceph-mon[36597]: pgmap v2970: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:38.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:38 smithi151 ceph-mon[36481]: pgmap v2970: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:38.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:38 smithi045 ceph-mon[31773]: pgmap v2970: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:39.600 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5412 sec 2024-03-28T19:22:40.600 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5413 sec 2024-03-28T19:22:40.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:40 smithi071 ceph-mon[36597]: pgmap v2971: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:40.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:40 smithi151 ceph-mon[36481]: pgmap v2971: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:40.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:40 smithi045 ceph-mon[31773]: pgmap v2971: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:41.601 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5414 sec 2024-03-28T19:22:41.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4769.316086 secs 2024-03-28T19:22:41.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4769.316086 secs 2024-03-28T19:22:41.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4769.316086 secs 2024-03-28T19:22:42.601 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5415 sec 2024-03-28T19:22:42.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:42 smithi071 ceph-mon[36597]: pgmap v2972: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:42.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:42 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4060.741650 secs 2024-03-28T19:22:42.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:42 smithi151 ceph-mon[36481]: pgmap v2972: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:42.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:42 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4060.741650 secs 2024-03-28T19:22:42.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:42 smithi045 ceph-mon[31773]: pgmap v2972: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:42.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:42 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4060.741650 secs 2024-03-28T19:22:43.601 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5416 sec 2024-03-28T19:22:44.601 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5417 sec 2024-03-28T19:22:44.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:44 smithi071 ceph-mon[36597]: pgmap v2973: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:44.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:44 smithi151 ceph-mon[36481]: pgmap v2973: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:44.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:44 smithi045 ceph-mon[31773]: pgmap v2973: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:45.601 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5418 sec 2024-03-28T19:22:46.601 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5419 sec 2024-03-28T19:22:46.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:46 smithi071 ceph-mon[36597]: pgmap v2974: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:46.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4774.316172 secs 2024-03-28T19:22:46.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:46 smithi151 ceph-mon[36481]: pgmap v2974: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:46.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4774.316172 secs 2024-03-28T19:22:46.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:46 smithi045 ceph-mon[31773]: pgmap v2974: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4774.316172 secs 2024-03-28T19:22:47.602 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5420 sec 2024-03-28T19:22:47.723 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:47 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4065.741712 secs 2024-03-28T19:22:47.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:47 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4065.741712 secs 2024-03-28T19:22:47.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:47 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4065.741712 secs 2024-03-28T19:22:48.602 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5421 sec 2024-03-28T19:22:48.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:48 smithi071 ceph-mon[36597]: pgmap v2975: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:48.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:48 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:22:48.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:48 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:48.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:48 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:48.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:48 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:48.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:48 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:48.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:48 smithi151 ceph-mon[36481]: pgmap v2975: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:48.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:48 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:22:48.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:48 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:48.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:48 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:48.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:48 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:48.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:48 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:48.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:48 smithi045 ceph-mon[31773]: pgmap v2975: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:48.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:48 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:22:48.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:48 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:48.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:48 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:48.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:48 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:48.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:48 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:49.602 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5422 sec 2024-03-28T19:22:49.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:49 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:49.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:49 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:49.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:49 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:49.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:49 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:49.821 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:49 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:49.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:49 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:50.602 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5423 sec 2024-03-28T19:22:50.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:50 smithi071 ceph-mon[36597]: pgmap v2976: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:50.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:50 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:50.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:50 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:50.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:50 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:50.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:50 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:50.821 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:50 smithi151 ceph-mon[36481]: pgmap v2976: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:50.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:50 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:50.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:50 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:50.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:50 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:50.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:50 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:50.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:50 smithi045 ceph-mon[31773]: pgmap v2976: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:50.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:50 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:50.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:50 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:50.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:50 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:50.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:50 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:51.602 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5424 sec 2024-03-28T19:22:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:51 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:51 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:51 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:51 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:51 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:51 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:51 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:51 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:51 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T19:22:51.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:51 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T19:22:51.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4779.316261 secs 2024-03-28T19:22:51.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:51 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:51 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:51 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:51 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:51 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:51 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:51 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:51 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:51 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:51 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:51 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:51 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:51 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:51 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:51 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:51 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:51 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T19:22:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:51 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T19:22:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4779.316261 secs 2024-03-28T19:22:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:51 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:51 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:51 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:51 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:51 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:51 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:51 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:51 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:51 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:51 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:51 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:51 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:51 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:51 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:51 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:51 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:51 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi071 to 174.1M 2024-03-28T19:22:51.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:51 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi071 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-28T19:22:51.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4779.316261 secs 2024-03-28T19:22:51.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:51 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:51 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:51 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:51 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:51 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:51 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:51 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:51.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:51 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:52.602 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5425 sec 2024-03-28T19:22:52.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:52 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T19:22:52.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:52 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T19:22:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:52 smithi071 ceph-mon[36597]: pgmap v2977: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:52.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:52 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4070.741774 secs 2024-03-28T19:22:52.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:52 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T19:22:52.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:52 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T19:22:52.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:52 smithi045 ceph-mon[31773]: pgmap v2977: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:52.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:52 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4070.741774 secs 2024-03-28T19:22:52.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:52 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi151 to 174.1M 2024-03-28T19:22:52.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:52 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi151 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T19:22:52.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:52 smithi151 ceph-mon[36481]: pgmap v2977: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:22:52.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:52 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4070.741774 secs 2024-03-28T19:22:53.602 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5426 sec 2024-03-28T19:22:54.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:54 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:54.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:54 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:54.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:54 smithi045 ceph-mon[31773]: pgmap v2978: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:54.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:54 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:54.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:54 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:54.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:54 smithi071 ceph-mon[36597]: pgmap v2978: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:54.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:54 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:54.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:54 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:54.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:54 smithi151 ceph-mon[36481]: pgmap v2978: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:22:54.602 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5427 sec 2024-03-28T19:22:55.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:55 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:55 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:55 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:55 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:55 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:55 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:55 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:55 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:55 smithi045 ceph-mon[31773]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T19:22:55.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:55 smithi045 ceph-mon[31773]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T19:22:55.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:55 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:22:55.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:55 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:22:55.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:55 smithi045 ceph-mon[31773]: pgmap v2979: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T19:22:55.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:55 smithi045 ceph-mon[31773]: pgmap v2980: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:22:55.418 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:55 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:55.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:55 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:55 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:55 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:55 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:55 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:55 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:55 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:55 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:55 smithi071 ceph-mon[36597]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T19:22:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:55 smithi071 ceph-mon[36597]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T19:22:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:55 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:22:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:55 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:22:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:55 smithi071 ceph-mon[36597]: pgmap v2979: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T19:22:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:55 smithi071 ceph-mon[36597]: pgmap v2980: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:22:55.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:55 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:55.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:55 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:55 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:55 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:55 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:55 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:55 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:55 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:55 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T19:22:55.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:55 smithi151 ceph-mon[36481]: Adjusting osd_memory_target on smithi045 to 177.8M 2024-03-28T19:22:55.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:55 smithi151 ceph-mon[36481]: Unable to set osd_memory_target on smithi045 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T19:22:55.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:55 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:22:55.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:55 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:22:55.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:55 smithi151 ceph-mon[36481]: pgmap v2979: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-28T19:22:55.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:55 smithi151 ceph-mon[36481]: pgmap v2980: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-28T19:22:55.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:55 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:22:55.602 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5428 sec 2024-03-28T19:22:56.603 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5429 sec 2024-03-28T19:22:56.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4784.316349 secs 2024-03-28T19:22:56.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4784.316349 secs 2024-03-28T19:22:56.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4784.316349 secs 2024-03-28T19:22:57.603 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5430 sec 2024-03-28T19:22:57.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:57 smithi045 ceph-mon[31773]: pgmap v2981: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:22:57.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:57 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4075.741813 secs 2024-03-28T19:22:57.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:57 smithi071 ceph-mon[36597]: pgmap v2981: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:22:57.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:57 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4075.741813 secs 2024-03-28T19:22:57.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:57 smithi151 ceph-mon[36481]: pgmap v2981: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:22:57.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:57 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4075.741813 secs 2024-03-28T19:22:58.603 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5431 sec 2024-03-28T19:22:59.603 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5432 sec 2024-03-28T19:22:59.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:22:59 smithi045 ceph-mon[31773]: pgmap v2982: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:22:59.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:22:59 smithi071 ceph-mon[36597]: pgmap v2982: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:22:59.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:22:59 smithi151 ceph-mon[36481]: pgmap v2982: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:23:00.603 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5433 sec 2024-03-28T19:23:01.603 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5434 sec 2024-03-28T19:23:01.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:01 smithi045 ceph-mon[31773]: pgmap v2983: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:23:01.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4789.316431 secs 2024-03-28T19:23:01.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:01 smithi071 ceph-mon[36597]: pgmap v2983: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:23:01.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4789.316431 secs 2024-03-28T19:23:01.821 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:01 smithi151 ceph-mon[36481]: pgmap v2983: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:23:01.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4789.316431 secs 2024-03-28T19:23:02.604 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5435 sec 2024-03-28T19:23:02.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:02 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4080.741891 secs 2024-03-28T19:23:02.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:02 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4080.741891 secs 2024-03-28T19:23:02.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:02 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4080.741891 secs 2024-03-28T19:23:03.604 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5436 sec 2024-03-28T19:23:03.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:03 smithi071 ceph-mon[36597]: pgmap v2984: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:23:03.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:03 smithi151 ceph-mon[36481]: pgmap v2984: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:23:03.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:03 smithi045 ceph-mon[31773]: pgmap v2984: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-28T19:23:04.604 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5437 sec 2024-03-28T19:23:05.604 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5438 sec 2024-03-28T19:23:05.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:05 smithi071 ceph-mon[36597]: pgmap v2985: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:23:05.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:05 smithi151 ceph-mon[36481]: pgmap v2985: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:23:05.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:05 smithi045 ceph-mon[31773]: pgmap v2985: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:23:06.604 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5439 sec 2024-03-28T19:23:06.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4794.316514 secs 2024-03-28T19:23:06.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4794.316514 secs 2024-03-28T19:23:06.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4794.316514 secs 2024-03-28T19:23:07.604 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5440 sec 2024-03-28T19:23:07.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:07 smithi071 ceph-mon[36597]: pgmap v2986: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:07.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:07 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4085.741954 secs 2024-03-28T19:23:07.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:07 smithi151 ceph-mon[36481]: pgmap v2986: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:07.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:07 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4085.741954 secs 2024-03-28T19:23:07.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:07 smithi045 ceph-mon[31773]: pgmap v2986: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:07.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:07 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4085.741954 secs 2024-03-28T19:23:08.604 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5441 sec 2024-03-28T19:23:09.605 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5442 sec 2024-03-28T19:23:09.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:09 smithi071 ceph-mon[36597]: pgmap v2987: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:09.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:09 smithi151 ceph-mon[36481]: pgmap v2987: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:09.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:09 smithi045 ceph-mon[31773]: pgmap v2987: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:10.605 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5443 sec 2024-03-28T19:23:11.605 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5444 sec 2024-03-28T19:23:11.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:11 smithi045 ceph-mon[31773]: pgmap v2988: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:11.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4799.316575 secs 2024-03-28T19:23:11.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:11 smithi071 ceph-mon[36597]: pgmap v2988: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:11.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4799.316575 secs 2024-03-28T19:23:11.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:11 smithi151 ceph-mon[36481]: pgmap v2988: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:11.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4799.316575 secs 2024-03-28T19:23:12.605 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5445 sec 2024-03-28T19:23:12.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:12 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4090.742016 secs 2024-03-28T19:23:12.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:12 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4090.742016 secs 2024-03-28T19:23:12.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:12 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4090.742016 secs 2024-03-28T19:23:13.605 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5446 sec 2024-03-28T19:23:13.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:13 smithi071 ceph-mon[36597]: pgmap v2989: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:13.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:13 smithi151 ceph-mon[36481]: pgmap v2989: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:13.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:13 smithi045 ceph-mon[31773]: pgmap v2989: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:14.605 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5447 sec 2024-03-28T19:23:15.605 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5448 sec 2024-03-28T19:23:15.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:15 smithi071 ceph-mon[36597]: pgmap v2990: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:15.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:15 smithi151 ceph-mon[36481]: pgmap v2990: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:15.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:15 smithi045 ceph-mon[31773]: pgmap v2990: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:16.605 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5449 sec 2024-03-28T19:23:16.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4804.316636 secs 2024-03-28T19:23:16.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4804.316636 secs 2024-03-28T19:23:16.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4804.316636 secs 2024-03-28T19:23:17.606 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5450 sec 2024-03-28T19:23:17.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:17 smithi071 ceph-mon[36597]: pgmap v2991: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:17.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:17 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4095.742079 secs 2024-03-28T19:23:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:17 smithi151 ceph-mon[36481]: pgmap v2991: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:17 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4095.742079 secs 2024-03-28T19:23:17.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:17 smithi045 ceph-mon[31773]: pgmap v2991: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:17.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:17 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4095.742079 secs 2024-03-28T19:23:18.606 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5451 sec 2024-03-28T19:23:19.606 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5452 sec 2024-03-28T19:23:19.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:19 smithi071 ceph-mon[36597]: pgmap v2992: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:19.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:19 smithi151 ceph-mon[36481]: pgmap v2992: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:19.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:19 smithi045 ceph-mon[31773]: pgmap v2992: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:20.606 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5453 sec 2024-03-28T19:23:21.606 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5454 sec 2024-03-28T19:23:21.768 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:21 smithi045 ceph-mon[31773]: pgmap v2993: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:21.768 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4809.316726 secs 2024-03-28T19:23:21.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:21 smithi071 ceph-mon[36597]: pgmap v2993: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:21.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4809.316726 secs 2024-03-28T19:23:21.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:21 smithi151 ceph-mon[36481]: pgmap v2993: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:21.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4809.316726 secs 2024-03-28T19:23:22.606 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5455 sec 2024-03-28T19:23:22.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:22 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4100.742163 secs 2024-03-28T19:23:22.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:22 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4100.742163 secs 2024-03-28T19:23:22.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:22 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4100.742163 secs 2024-03-28T19:23:23.606 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5456 sec 2024-03-28T19:23:23.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:23 smithi071 ceph-mon[36597]: pgmap v2994: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:23.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:23 smithi151 ceph-mon[36481]: pgmap v2994: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:23.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:23 smithi045 ceph-mon[31773]: pgmap v2994: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:24.607 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5457 sec 2024-03-28T19:23:25.607 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5458 sec 2024-03-28T19:23:25.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:25 smithi071 ceph-mon[36597]: pgmap v2995: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:25.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:25 smithi151 ceph-mon[36481]: pgmap v2995: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:25.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:25 smithi045 ceph-mon[31773]: pgmap v2995: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:26.607 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5459 sec 2024-03-28T19:23:26.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4814.316788 secs 2024-03-28T19:23:26.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4814.316788 secs 2024-03-28T19:23:26.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4814.316788 secs 2024-03-28T19:23:27.607 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5460 sec 2024-03-28T19:23:27.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:27 smithi071 ceph-mon[36597]: pgmap v2996: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:27.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:27 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4105.742253 secs 2024-03-28T19:23:27.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:27 smithi151 ceph-mon[36481]: pgmap v2996: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:27.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:27 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4105.742253 secs 2024-03-28T19:23:27.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:27 smithi045 ceph-mon[31773]: pgmap v2996: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:27.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:27 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4105.742253 secs 2024-03-28T19:23:28.607 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5461 sec 2024-03-28T19:23:29.607 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5462 sec 2024-03-28T19:23:29.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:29 smithi071 ceph-mon[36597]: pgmap v2997: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:29.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:29 smithi151 ceph-mon[36481]: pgmap v2997: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:29.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:29 smithi045 ceph-mon[31773]: pgmap v2997: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:30.608 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5463 sec 2024-03-28T19:23:31.608 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5464 sec 2024-03-28T19:23:31.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:31 smithi071 ceph-mon[36597]: pgmap v2998: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:31.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4819.316876 secs 2024-03-28T19:23:31.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:31 smithi151 ceph-mon[36481]: pgmap v2998: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:31.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4819.316876 secs 2024-03-28T19:23:31.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:31 smithi045 ceph-mon[31773]: pgmap v2998: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:31.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4819.316876 secs 2024-03-28T19:23:32.608 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5465 sec 2024-03-28T19:23:32.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:32 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4110.742293 secs 2024-03-28T19:23:32.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:32 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4110.742293 secs 2024-03-28T19:23:32.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:32 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4110.742293 secs 2024-03-28T19:23:33.165 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:23:32 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:23:32.814+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:23:33.165 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:23:32 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:23:32.814+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:23:33.165 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:23:32 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:23:32.814+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:23:33.165 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:23:32 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:23:32.814+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:23:33.165 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 28 19:23:32 smithi045 ceph-9f4888ec-ed29-11ee-95d0-87774f69a715-mgr-x[32001]: 2024-03-28T19:23:32.814+0000 7f52f6400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T19:23:33.608 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5466 sec 2024-03-28T19:23:33.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:33 smithi071 ceph-mon[36597]: pgmap v2999: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:33.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:33 smithi151 ceph-mon[36481]: pgmap v2999: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:33.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:33 smithi045 ceph-mon[31773]: pgmap v2999: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:34.608 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5467 sec 2024-03-28T19:23:35.608 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5468 sec 2024-03-28T19:23:35.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:35 smithi071 ceph-mon[36597]: pgmap v3000: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:35.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:35 smithi151 ceph-mon[36481]: pgmap v3000: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:35.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:35 smithi045 ceph-mon[31773]: pgmap v3000: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:36.608 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5469 sec 2024-03-28T19:23:36.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:36 smithi071 ceph-mon[36597]: 5 slow requests, 1 included below; oldest blocked for > 4824.316936 secs 2024-03-28T19:23:36.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:36 smithi071 ceph-mon[36597]: slow request 2884.403911 seconds old, received at 2024-03-28T18:35:31.473570+0000: client_request(client.26711:1 getattr pAsLsXsFs #0x1 2024-03-28T18:35:31.469661+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T19:23:36.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:36 smithi151 ceph-mon[36481]: 5 slow requests, 1 included below; oldest blocked for > 4824.316936 secs 2024-03-28T19:23:36.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:36 smithi151 ceph-mon[36481]: slow request 2884.403911 seconds old, received at 2024-03-28T18:35:31.473570+0000: client_request(client.26711:1 getattr pAsLsXsFs #0x1 2024-03-28T18:35:31.469661+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T19:23:36.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:36 smithi045 ceph-mon[31773]: 5 slow requests, 1 included below; oldest blocked for > 4824.316936 secs 2024-03-28T19:23:36.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:36 smithi045 ceph-mon[31773]: slow request 2884.403911 seconds old, received at 2024-03-28T18:35:31.473570+0000: client_request(client.26711:1 getattr pAsLsXsFs #0x1 2024-03-28T18:35:31.469661+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T19:23:37.609 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5470 sec 2024-03-28T19:23:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:37 smithi071 ceph-mon[36597]: pgmap v3001: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 426 B/s wr, 16 op/s 2024-03-28T19:23:37.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:37 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4115.742354 secs 2024-03-28T19:23:37.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:37 smithi151 ceph-mon[36481]: pgmap v3001: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 426 B/s wr, 16 op/s 2024-03-28T19:23:37.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:37 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4115.742354 secs 2024-03-28T19:23:37.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:37 smithi045 ceph-mon[31773]: pgmap v3001: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 426 B/s wr, 16 op/s 2024-03-28T19:23:37.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:37 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4115.742354 secs 2024-03-28T19:23:38.609 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5471 sec 2024-03-28T19:23:39.609 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5472 sec 2024-03-28T19:23:39.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:39 smithi071 ceph-mon[36597]: pgmap v3002: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 426 B/s wr, 15 op/s 2024-03-28T19:23:39.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:39 smithi151 ceph-mon[36481]: pgmap v3002: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 426 B/s wr, 15 op/s 2024-03-28T19:23:39.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:39 smithi045 ceph-mon[31773]: pgmap v3002: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 426 B/s wr, 15 op/s 2024-03-28T19:23:40.609 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5473 sec 2024-03-28T19:23:41.609 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5474 sec 2024-03-28T19:23:41.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:41 smithi151 ceph-mon[36481]: pgmap v3003: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 426 B/s wr, 16 op/s 2024-03-28T19:23:41.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4829.316981 secs 2024-03-28T19:23:41.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:41 smithi045 ceph-mon[31773]: pgmap v3003: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 426 B/s wr, 16 op/s 2024-03-28T19:23:41.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4829.316981 secs 2024-03-28T19:23:42.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:41 smithi071 ceph-mon[36597]: pgmap v3003: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 426 B/s wr, 16 op/s 2024-03-28T19:23:42.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4829.316981 secs 2024-03-28T19:23:42.609 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5475 sec 2024-03-28T19:23:42.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:42 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4120.742415 secs 2024-03-28T19:23:43.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:42 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4120.742415 secs 2024-03-28T19:23:43.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:42 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4120.742415 secs 2024-03-28T19:23:43.610 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5476 sec 2024-03-28T19:23:43.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:43 smithi045 ceph-mon[31773]: pgmap v3004: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 426 B/s wr, 15 op/s 2024-03-28T19:23:44.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:43 smithi071 ceph-mon[36597]: pgmap v3004: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 426 B/s wr, 15 op/s 2024-03-28T19:23:44.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:43 smithi151 ceph-mon[36481]: pgmap v3004: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 426 B/s wr, 15 op/s 2024-03-28T19:23:44.610 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5477 sec 2024-03-28T19:23:45.610 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5478 sec 2024-03-28T19:23:45.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:45 smithi045 ceph-mon[31773]: pgmap v3005: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 426 B/s wr, 15 op/s 2024-03-28T19:23:46.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:45 smithi071 ceph-mon[36597]: pgmap v3005: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 426 B/s wr, 15 op/s 2024-03-28T19:23:46.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:45 smithi151 ceph-mon[36481]: pgmap v3005: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 426 B/s wr, 15 op/s 2024-03-28T19:23:46.610 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5479 sec 2024-03-28T19:23:46.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4834.317020 secs 2024-03-28T19:23:47.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4834.317020 secs 2024-03-28T19:23:47.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4834.317020 secs 2024-03-28T19:23:47.610 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5480 sec 2024-03-28T19:23:47.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:47 smithi045 ceph-mon[31773]: pgmap v3006: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 426 B/s wr, 16 op/s 2024-03-28T19:23:47.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:47 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4125.742456 secs 2024-03-28T19:23:47.960 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:47 smithi071 ceph-mon[36597]: pgmap v3006: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 426 B/s wr, 16 op/s 2024-03-28T19:23:47.960 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:47 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4125.742456 secs 2024-03-28T19:23:47.973 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:47 smithi151 ceph-mon[36481]: pgmap v3006: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 426 B/s wr, 16 op/s 2024-03-28T19:23:47.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:47 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4125.742456 secs 2024-03-28T19:23:48.610 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5481 sec 2024-03-28T19:23:49.610 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5482 sec 2024-03-28T19:23:49.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:49 smithi045 ceph-mon[31773]: pgmap v3007: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:50.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:49 smithi071 ceph-mon[36597]: pgmap v3007: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:50.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:49 smithi151 ceph-mon[36481]: pgmap v3007: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:50.610 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5483 sec 2024-03-28T19:23:51.610 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5484 sec 2024-03-28T19:23:51.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:51 smithi045 ceph-mon[31773]: pgmap v3008: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:51.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4839.317110 secs 2024-03-28T19:23:52.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:51 smithi071 ceph-mon[36597]: pgmap v3008: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:52.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4839.317110 secs 2024-03-28T19:23:52.071 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:51 smithi151 ceph-mon[36481]: pgmap v3008: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:23:52.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4839.317110 secs 2024-03-28T19:23:52.611 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5485 sec 2024-03-28T19:23:52.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:52 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4130.742517 secs 2024-03-28T19:23:53.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:52 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4130.742517 secs 2024-03-28T19:23:53.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:52 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4130.742517 secs 2024-03-28T19:23:53.611 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5486 sec 2024-03-28T19:23:53.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:53 smithi045 ceph-mon[31773]: pgmap v3009: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:54.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:53 smithi071 ceph-mon[36597]: pgmap v3009: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:54.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:53 smithi151 ceph-mon[36481]: pgmap v3009: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:54.611 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5487 sec 2024-03-28T19:23:55.611 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5488 sec 2024-03-28T19:23:55.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:55 smithi045 ceph-mon[31773]: pgmap v3010: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:55.917 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:55 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:23:56.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:55 smithi071 ceph-mon[36597]: pgmap v3010: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:56.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:55 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:23:56.071 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:55 smithi151 ceph-mon[36481]: pgmap v3010: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:23:56.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:55 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:23:56.611 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5489 sec 2024-03-28T19:23:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4844.317201 secs 2024-03-28T19:23:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:56 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:23:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:56 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:23:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:56 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:23:57.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4844.317201 secs 2024-03-28T19:23:57.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:56 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:23:57.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:56 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:23:57.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:56 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:23:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4844.317201 secs 2024-03-28T19:23:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:56 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:23:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:56 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:23:57.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:56 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:23:57.612 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5490 sec 2024-03-28T19:23:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:57 smithi045 ceph-mon[31773]: pgmap v3011: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:23:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:57 smithi045 ceph-mon[31773]: pgmap v3012: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:23:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:57 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4135.742606 secs 2024-03-28T19:23:58.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:57 smithi071 ceph-mon[36597]: pgmap v3011: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:23:58.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:57 smithi071 ceph-mon[36597]: pgmap v3012: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:23:58.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:57 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4135.742606 secs 2024-03-28T19:23:58.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:57 smithi151 ceph-mon[36481]: pgmap v3011: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:23:58.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:57 smithi151 ceph-mon[36481]: pgmap v3012: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:23:58.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:57 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4135.742606 secs 2024-03-28T19:23:58.612 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5491 sec 2024-03-28T19:23:59.612 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5492 sec 2024-03-28T19:23:59.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:23:59 smithi045 ceph-mon[31773]: pgmap v3013: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:24:00.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:23:59 smithi071 ceph-mon[36597]: pgmap v3013: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:24:00.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:23:59 smithi151 ceph-mon[36481]: pgmap v3013: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:24:00.612 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5493 sec 2024-03-28T19:24:01.612 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5494 sec 2024-03-28T19:24:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:01 smithi045 ceph-mon[31773]: pgmap v3014: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:24:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4849.317240 secs 2024-03-28T19:24:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:01 smithi071 ceph-mon[36597]: pgmap v3014: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:24:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4849.317240 secs 2024-03-28T19:24:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:01 smithi151 ceph-mon[36481]: pgmap v3014: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:24:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4849.317240 secs 2024-03-28T19:24:02.612 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5495 sec 2024-03-28T19:24:03.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:02 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4140.742695 secs 2024-03-28T19:24:03.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:02 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4140.742695 secs 2024-03-28T19:24:03.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:02 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4140.742695 secs 2024-03-28T19:24:03.612 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5496 sec 2024-03-28T19:24:04.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:03 smithi071 ceph-mon[36597]: pgmap v3015: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:24:04.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:03 smithi151 ceph-mon[36481]: pgmap v3015: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:24:04.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:03 smithi045 ceph-mon[31773]: pgmap v3015: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:24:04.612 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5497 sec 2024-03-28T19:24:05.613 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5498 sec 2024-03-28T19:24:06.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:05 smithi071 ceph-mon[36597]: pgmap v3016: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:24:06.071 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:05 smithi151 ceph-mon[36481]: pgmap v3016: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:24:06.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:05 smithi045 ceph-mon[31773]: pgmap v3016: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:24:06.613 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5499 sec 2024-03-28T19:24:07.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4854.317326 secs 2024-03-28T19:24:07.071 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4854.317326 secs 2024-03-28T19:24:07.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4854.317326 secs 2024-03-28T19:24:07.613 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5500 sec 2024-03-28T19:24:08.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:07 smithi071 ceph-mon[36597]: pgmap v3017: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-28T19:24:08.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:07 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4145.742757 secs 2024-03-28T19:24:08.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:07 smithi151 ceph-mon[36481]: pgmap v3017: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-28T19:24:08.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:07 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4145.742757 secs 2024-03-28T19:24:08.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:07 smithi045 ceph-mon[31773]: pgmap v3017: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-28T19:24:08.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:07 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4145.742757 secs 2024-03-28T19:24:08.613 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5501 sec 2024-03-28T19:24:09.613 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5502 sec 2024-03-28T19:24:10.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:09 smithi071 ceph-mon[36597]: pgmap v3018: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T19:24:10.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:09 smithi151 ceph-mon[36481]: pgmap v3018: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T19:24:10.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:09 smithi045 ceph-mon[31773]: pgmap v3018: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T19:24:10.613 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5503 sec 2024-03-28T19:24:11.614 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5504 sec 2024-03-28T19:24:12.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:11 smithi071 ceph-mon[36597]: pgmap v3019: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T19:24:12.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4859.317409 secs 2024-03-28T19:24:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:11 smithi151 ceph-mon[36481]: pgmap v3019: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T19:24:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4859.317409 secs 2024-03-28T19:24:12.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:11 smithi045 ceph-mon[31773]: pgmap v3019: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T19:24:12.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4859.317409 secs 2024-03-28T19:24:12.613 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5505 sec 2024-03-28T19:24:13.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:12 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4150.742818 secs 2024-03-28T19:24:13.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:12 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4150.742818 secs 2024-03-28T19:24:13.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:12 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4150.742818 secs 2024-03-28T19:24:13.614 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5506 sec 2024-03-28T19:24:14.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:13 smithi071 ceph-mon[36597]: pgmap v3020: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T19:24:14.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:13 smithi151 ceph-mon[36481]: pgmap v3020: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T19:24:14.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:13 smithi045 ceph-mon[31773]: pgmap v3020: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T19:24:14.614 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5507 sec 2024-03-28T19:24:15.614 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5508 sec 2024-03-28T19:24:16.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:15 smithi071 ceph-mon[36597]: pgmap v3021: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T19:24:16.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:15 smithi151 ceph-mon[36481]: pgmap v3021: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T19:24:16.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:15 smithi045 ceph-mon[31773]: pgmap v3021: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T19:24:16.614 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5509 sec 2024-03-28T19:24:17.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4864.317463 secs 2024-03-28T19:24:17.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4864.317463 secs 2024-03-28T19:24:17.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4864.317463 secs 2024-03-28T19:24:17.614 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5510 sec 2024-03-28T19:24:18.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:17 smithi071 ceph-mon[36597]: pgmap v3022: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T19:24:18.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:17 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4155.742848 secs 2024-03-28T19:24:18.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:17 smithi151 ceph-mon[36481]: pgmap v3022: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T19:24:18.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:17 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4155.742848 secs 2024-03-28T19:24:18.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:17 smithi045 ceph-mon[31773]: pgmap v3022: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-28T19:24:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:17 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4155.742848 secs 2024-03-28T19:24:18.614 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5511 sec 2024-03-28T19:24:19.614 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5512 sec 2024-03-28T19:24:20.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:19 smithi071 ceph-mon[36597]: pgmap v3023: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:20.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:19 smithi151 ceph-mon[36481]: pgmap v3023: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:20.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:19 smithi045 ceph-mon[31773]: pgmap v3023: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:20.614 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5513 sec 2024-03-28T19:24:21.615 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5514 sec 2024-03-28T19:24:22.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:21 smithi071 ceph-mon[36597]: pgmap v3024: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:22.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4869.317549 secs 2024-03-28T19:24:22.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:21 smithi151 ceph-mon[36481]: pgmap v3024: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:22.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4869.317549 secs 2024-03-28T19:24:22.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:21 smithi045 ceph-mon[31773]: pgmap v3024: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:22.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4869.317549 secs 2024-03-28T19:24:22.615 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5515 sec 2024-03-28T19:24:23.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:22 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4160.742938 secs 2024-03-28T19:24:23.066 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:22 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4160.742938 secs 2024-03-28T19:24:23.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:22 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4160.742938 secs 2024-03-28T19:24:23.615 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5516 sec 2024-03-28T19:24:24.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:23 smithi071 ceph-mon[36597]: pgmap v3025: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:24.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:23 smithi151 ceph-mon[36481]: pgmap v3025: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:24.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:23 smithi045 ceph-mon[31773]: pgmap v3025: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:24.615 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5517 sec 2024-03-28T19:24:25.616 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5518 sec 2024-03-28T19:24:26.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:25 smithi071 ceph-mon[36597]: pgmap v3026: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:26.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:25 smithi151 ceph-mon[36481]: pgmap v3026: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:26.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:25 smithi045 ceph-mon[31773]: pgmap v3026: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:26.616 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5519 sec 2024-03-28T19:24:27.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4874.317608 secs 2024-03-28T19:24:27.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4874.317608 secs 2024-03-28T19:24:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4874.317608 secs 2024-03-28T19:24:27.616 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5520 sec 2024-03-28T19:24:28.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:27 smithi071 ceph-mon[36597]: pgmap v3027: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:28.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:27 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4165.742998 secs 2024-03-28T19:24:28.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:27 smithi151 ceph-mon[36481]: pgmap v3027: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:28.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:27 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4165.742998 secs 2024-03-28T19:24:28.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:27 smithi045 ceph-mon[31773]: pgmap v3027: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:27 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4165.742998 secs 2024-03-28T19:24:28.616 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5521 sec 2024-03-28T19:24:29.616 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5522 sec 2024-03-28T19:24:30.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:29 smithi071 ceph-mon[36597]: pgmap v3028: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:30.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:29 smithi151 ceph-mon[36481]: pgmap v3028: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:30.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:29 smithi045 ceph-mon[31773]: pgmap v3028: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:30.616 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5523 sec 2024-03-28T19:24:31.616 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5524 sec 2024-03-28T19:24:32.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:31 smithi071 ceph-mon[36597]: pgmap v3029: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:32.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4879.317671 secs 2024-03-28T19:24:32.071 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:31 smithi151 ceph-mon[36481]: pgmap v3029: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:32.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4879.317671 secs 2024-03-28T19:24:32.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:31 smithi045 ceph-mon[31773]: pgmap v3029: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:32.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4879.317671 secs 2024-03-28T19:24:32.617 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5525 sec 2024-03-28T19:24:33.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:32 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4170.743087 secs 2024-03-28T19:24:33.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:32 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4170.743087 secs 2024-03-28T19:24:33.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:32 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4170.743087 secs 2024-03-28T19:24:33.616 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5526 sec 2024-03-28T19:24:34.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:33 smithi071 ceph-mon[36597]: pgmap v3030: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:34.071 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:33 smithi151 ceph-mon[36481]: pgmap v3030: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:34.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:33 smithi045 ceph-mon[31773]: pgmap v3030: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:34.617 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5527 sec 2024-03-28T19:24:35.617 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5528 sec 2024-03-28T19:24:36.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:35 smithi071 ceph-mon[36597]: pgmap v3031: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:36.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:35 smithi151 ceph-mon[36481]: pgmap v3031: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:36.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:35 smithi045 ceph-mon[31773]: pgmap v3031: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:36.617 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5529 sec 2024-03-28T19:24:37.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4884.317734 secs 2024-03-28T19:24:37.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4884.317734 secs 2024-03-28T19:24:37.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4884.317734 secs 2024-03-28T19:24:37.617 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5530 sec 2024-03-28T19:24:38.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:37 smithi071 ceph-mon[36597]: pgmap v3032: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:38.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:37 smithi071 ceph-mon[36597]: 3 slow requests, 1 included below; oldest blocked for > 4175.743104 secs 2024-03-28T19:24:38.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:37 smithi071 ceph-mon[36597]: slow request 364.059717 seconds old, received at 2024-03-28T19:18:32.630514+0000: client_request(client.26759:1 getattr pAsLsXsFs #0x1 2024-03-28T19:18:32.626993+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T19:24:38.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:37 smithi151 ceph-mon[36481]: pgmap v3032: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:38.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:37 smithi151 ceph-mon[36481]: 3 slow requests, 1 included below; oldest blocked for > 4175.743104 secs 2024-03-28T19:24:38.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:37 smithi151 ceph-mon[36481]: slow request 364.059717 seconds old, received at 2024-03-28T19:18:32.630514+0000: client_request(client.26759:1 getattr pAsLsXsFs #0x1 2024-03-28T19:18:32.626993+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T19:24:38.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:37 smithi045 ceph-mon[31773]: pgmap v3032: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:38.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:37 smithi045 ceph-mon[31773]: 3 slow requests, 1 included below; oldest blocked for > 4175.743104 secs 2024-03-28T19:24:38.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:37 smithi045 ceph-mon[31773]: slow request 364.059717 seconds old, received at 2024-03-28T19:18:32.630514+0000: client_request(client.26759:1 getattr pAsLsXsFs #0x1 2024-03-28T19:18:32.626993+0000 caller_uid=0, caller_gid=0{}) currently cleaned up request 2024-03-28T19:24:38.617 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5531 sec 2024-03-28T19:24:39.618 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5532 sec 2024-03-28T19:24:40.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:39 smithi071 ceph-mon[36597]: pgmap v3033: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:40.073 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:39 smithi151 ceph-mon[36481]: pgmap v3033: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:40.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:39 smithi045 ceph-mon[31773]: pgmap v3033: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:40.618 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5533 sec 2024-03-28T19:24:41.618 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5534 sec 2024-03-28T19:24:42.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:41 smithi151 ceph-mon[36481]: pgmap v3034: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:42.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4889.317797 secs 2024-03-28T19:24:42.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:41 smithi045 ceph-mon[31773]: pgmap v3034: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4889.317797 secs 2024-03-28T19:24:42.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:41 smithi071 ceph-mon[36597]: pgmap v3034: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:42.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4889.317797 secs 2024-03-28T19:24:42.618 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5535 sec 2024-03-28T19:24:43.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:42 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4180.743192 secs 2024-03-28T19:24:43.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:42 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4180.743192 secs 2024-03-28T19:24:43.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:42 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4180.743192 secs 2024-03-28T19:24:43.618 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5536 sec 2024-03-28T19:24:44.078 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:43 smithi045 ceph-mon[31773]: pgmap v3035: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:44.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:43 smithi071 ceph-mon[36597]: pgmap v3035: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:44.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:43 smithi151 ceph-mon[36481]: pgmap v3035: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:44.618 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5537 sec 2024-03-28T19:24:45.618 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5538 sec 2024-03-28T19:24:46.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:45 smithi045 ceph-mon[31773]: pgmap v3036: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:46.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:45 smithi071 ceph-mon[36597]: pgmap v3036: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:46.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:45 smithi151 ceph-mon[36481]: pgmap v3036: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:46.618 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5539 sec 2024-03-28T19:24:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4894.317861 secs 2024-03-28T19:24:47.210 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4894.317861 secs 2024-03-28T19:24:47.223 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4894.317861 secs 2024-03-28T19:24:47.619 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5540 sec 2024-03-28T19:24:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:47 smithi045 ceph-mon[31773]: pgmap v3037: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:47 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4185.743252 secs 2024-03-28T19:24:48.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:47 smithi071 ceph-mon[36597]: pgmap v3037: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:48.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:47 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4185.743252 secs 2024-03-28T19:24:48.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:47 smithi151 ceph-mon[36481]: pgmap v3037: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:48.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:47 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4185.743252 secs 2024-03-28T19:24:48.619 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5541 sec 2024-03-28T19:24:49.619 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5542 sec 2024-03-28T19:24:50.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:49 smithi045 ceph-mon[31773]: pgmap v3038: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:50.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:49 smithi071 ceph-mon[36597]: pgmap v3038: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:50.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:49 smithi151 ceph-mon[36481]: pgmap v3038: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:50.619 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5543 sec 2024-03-28T19:24:51.619 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5544 sec 2024-03-28T19:24:52.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:51 smithi045 ceph-mon[31773]: pgmap v3039: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4899.317925 secs 2024-03-28T19:24:52.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:51 smithi071 ceph-mon[36597]: pgmap v3039: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:52.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4899.317925 secs 2024-03-28T19:24:52.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:51 smithi151 ceph-mon[36481]: pgmap v3039: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:52.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4899.317925 secs 2024-03-28T19:24:52.619 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5545 sec 2024-03-28T19:24:53.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:52 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4190.743312 secs 2024-03-28T19:24:53.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:52 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4190.743312 secs 2024-03-28T19:24:53.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:52 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4190.743312 secs 2024-03-28T19:24:53.619 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5546 sec 2024-03-28T19:24:54.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:53 smithi045 ceph-mon[31773]: pgmap v3040: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:54.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:53 smithi071 ceph-mon[36597]: pgmap v3040: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:54.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:53 smithi151 ceph-mon[36481]: pgmap v3040: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:54.620 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5547 sec 2024-03-28T19:24:55.620 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5548 sec 2024-03-28T19:24:56.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:55 smithi045 ceph-mon[31773]: pgmap v3041: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:56.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:55 smithi071 ceph-mon[36597]: pgmap v3041: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:56.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:55 smithi151 ceph-mon[36481]: pgmap v3041: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:56.620 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5549 sec 2024-03-28T19:24:57.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4904.317985 secs 2024-03-28T19:24:57.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:56 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:24:57.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4904.317985 secs 2024-03-28T19:24:57.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:56 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:24:57.321 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4904.317985 secs 2024-03-28T19:24:57.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:56 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:24:57.620 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5550 sec 2024-03-28T19:24:58.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:57 smithi045 ceph-mon[31773]: pgmap v3042: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:57 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4195.743375 secs 2024-03-28T19:24:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:57 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:24:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:57 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:24:58.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:57 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:24:58.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:57 smithi071 ceph-mon[36597]: pgmap v3042: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:58.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:57 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4195.743375 secs 2024-03-28T19:24:58.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:57 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:24:58.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:57 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:24:58.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:57 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:24:58.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:57 smithi151 ceph-mon[36481]: pgmap v3042: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:24:58.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:57 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4195.743375 secs 2024-03-28T19:24:58.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:57 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:24:58.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:57 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:24:58.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:57 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:24:58.620 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5551 sec 2024-03-28T19:24:59.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:58 smithi045 ceph-mon[31773]: pgmap v3043: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:59.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:24:58 smithi045 ceph-mon[31773]: pgmap v3044: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:24:59.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:58 smithi071 ceph-mon[36597]: pgmap v3043: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:59.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:24:58 smithi071 ceph-mon[36597]: pgmap v3044: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:24:59.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:58 smithi151 ceph-mon[36481]: pgmap v3043: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:24:59.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:24:58 smithi151 ceph-mon[36481]: pgmap v3044: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:24:59.620 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5552 sec 2024-03-28T19:25:00.620 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5553 sec 2024-03-28T19:25:00.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:00 smithi151 ceph-mon[36481]: pgmap v3045: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:25:00.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:00 smithi045 ceph-mon[31773]: pgmap v3045: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:25:01.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:00 smithi071 ceph-mon[36597]: pgmap v3045: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:25:01.621 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5554 sec 2024-03-28T19:25:01.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4909.318077 secs 2024-03-28T19:25:01.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4909.318077 secs 2024-03-28T19:25:02.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4909.318077 secs 2024-03-28T19:25:02.621 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5555 sec 2024-03-28T19:25:02.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:02 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4200.743464 secs 2024-03-28T19:25:02.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:02 smithi045 ceph-mon[31773]: pgmap v3046: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:25:03.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:02 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4200.743464 secs 2024-03-28T19:25:03.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:02 smithi071 ceph-mon[36597]: pgmap v3046: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:25:03.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:02 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4200.743464 secs 2024-03-28T19:25:03.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:02 smithi151 ceph-mon[36481]: pgmap v3046: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:25:03.621 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5556 sec 2024-03-28T19:25:04.621 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5557 sec 2024-03-28T19:25:05.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:04 smithi071 ceph-mon[36597]: pgmap v3047: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:25:05.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:04 smithi151 ceph-mon[36481]: pgmap v3047: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:25:05.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:04 smithi045 ceph-mon[31773]: pgmap v3047: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:25:05.621 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5558 sec 2024-03-28T19:25:06.621 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5559 sec 2024-03-28T19:25:07.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:06 smithi071 ceph-mon[36597]: pgmap v3048: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:25:07.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4914.318136 secs 2024-03-28T19:25:07.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:06 smithi151 ceph-mon[36481]: pgmap v3048: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:25:07.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4914.318136 secs 2024-03-28T19:25:07.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:06 smithi045 ceph-mon[31773]: pgmap v3048: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:25:07.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4914.318136 secs 2024-03-28T19:25:07.621 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5560 sec 2024-03-28T19:25:08.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:07 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4205.743553 secs 2024-03-28T19:25:08.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:07 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4205.743553 secs 2024-03-28T19:25:08.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:07 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4205.743553 secs 2024-03-28T19:25:08.622 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5561 sec 2024-03-28T19:25:09.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:08 smithi151 ceph-mon[36481]: pgmap v3049: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:25:09.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:08 smithi045 ceph-mon[31773]: pgmap v3049: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:25:09.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:08 smithi071 ceph-mon[36597]: pgmap v3049: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:25:09.622 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5562 sec 2024-03-28T19:25:10.622 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5563 sec 2024-03-28T19:25:11.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:10 smithi045 ceph-mon[31773]: pgmap v3050: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:11.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:10 smithi071 ceph-mon[36597]: pgmap v3050: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:11.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:10 smithi151 ceph-mon[36481]: pgmap v3050: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:11.622 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5564 sec 2024-03-28T19:25:12.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4919.318219 secs 2024-03-28T19:25:12.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4919.318219 secs 2024-03-28T19:25:12.321 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4919.318219 secs 2024-03-28T19:25:12.622 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5565 sec 2024-03-28T19:25:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:12 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4210.743592 secs 2024-03-28T19:25:13.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:12 smithi045 ceph-mon[31773]: pgmap v3051: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:13.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:12 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4210.743592 secs 2024-03-28T19:25:13.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:12 smithi071 ceph-mon[36597]: pgmap v3051: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:13.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:12 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4210.743592 secs 2024-03-28T19:25:13.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:12 smithi151 ceph-mon[36481]: pgmap v3051: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:13.622 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5566 sec 2024-03-28T19:25:14.622 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5567 sec 2024-03-28T19:25:15.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:14 smithi045 ceph-mon[31773]: pgmap v3052: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:15.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:14 smithi071 ceph-mon[36597]: pgmap v3052: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:15.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:14 smithi151 ceph-mon[36481]: pgmap v3052: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:15.623 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5568 sec 2024-03-28T19:25:16.623 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5569 sec 2024-03-28T19:25:17.111 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:16 smithi071 ceph-mon[36597]: pgmap v3053: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:17.111 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4924.318256 secs 2024-03-28T19:25:17.124 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:16 smithi151 ceph-mon[36481]: pgmap v3053: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:17.124 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4924.318256 secs 2024-03-28T19:25:17.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:16 smithi045 ceph-mon[31773]: pgmap v3053: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:17.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4924.318256 secs 2024-03-28T19:25:17.623 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5570 sec 2024-03-28T19:25:18.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:17 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4215.743679 secs 2024-03-28T19:25:18.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:17 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4215.743679 secs 2024-03-28T19:25:18.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:17 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4215.743679 secs 2024-03-28T19:25:18.623 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5571 sec 2024-03-28T19:25:19.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:18 smithi045 ceph-mon[31773]: pgmap v3054: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:19.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:18 smithi071 ceph-mon[36597]: pgmap v3054: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:19.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:18 smithi151 ceph-mon[36481]: pgmap v3054: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:19.623 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5572 sec 2024-03-28T19:25:20.624 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5573 sec 2024-03-28T19:25:21.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:20 smithi045 ceph-mon[31773]: pgmap v3055: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:21.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:20 smithi071 ceph-mon[36597]: pgmap v3055: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:21.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:20 smithi151 ceph-mon[36481]: pgmap v3055: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:21.624 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5574 sec 2024-03-28T19:25:22.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4929.318343 secs 2024-03-28T19:25:22.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4929.318343 secs 2024-03-28T19:25:22.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4929.318343 secs 2024-03-28T19:25:22.624 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5575 sec 2024-03-28T19:25:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:22 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4220.743764 secs 2024-03-28T19:25:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:22 smithi045 ceph-mon[31773]: pgmap v3056: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:23.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:22 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4220.743764 secs 2024-03-28T19:25:23.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:22 smithi071 ceph-mon[36597]: pgmap v3056: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:23.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:22 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4220.743764 secs 2024-03-28T19:25:23.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:22 smithi151 ceph-mon[36481]: pgmap v3056: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:23.624 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5576 sec 2024-03-28T19:25:24.624 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5577 sec 2024-03-28T19:25:25.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:24 smithi045 ceph-mon[31773]: pgmap v3057: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:25.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:24 smithi071 ceph-mon[36597]: pgmap v3057: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:25.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:24 smithi151 ceph-mon[36481]: pgmap v3057: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:25.624 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5578 sec 2024-03-28T19:25:26.624 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5579 sec 2024-03-28T19:25:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:26 smithi045 ceph-mon[31773]: pgmap v3058: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4934.318430 secs 2024-03-28T19:25:27.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:26 smithi071 ceph-mon[36597]: pgmap v3058: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:27.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4934.318430 secs 2024-03-28T19:25:27.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:26 smithi151 ceph-mon[36481]: pgmap v3058: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:27.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4934.318430 secs 2024-03-28T19:25:27.625 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5580 sec 2024-03-28T19:25:28.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:27 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4225.743852 secs 2024-03-28T19:25:28.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:27 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4225.743852 secs 2024-03-28T19:25:28.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:27 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4225.743852 secs 2024-03-28T19:25:28.625 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5581 sec 2024-03-28T19:25:29.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:28 smithi045 ceph-mon[31773]: pgmap v3059: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:29.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:28 smithi071 ceph-mon[36597]: pgmap v3059: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:29.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:28 smithi151 ceph-mon[36481]: pgmap v3059: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:29.625 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5582 sec 2024-03-28T19:25:30.625 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5583 sec 2024-03-28T19:25:31.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:30 smithi045 ceph-mon[31773]: pgmap v3060: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:31.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:30 smithi071 ceph-mon[36597]: pgmap v3060: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:31.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:30 smithi151 ceph-mon[36481]: pgmap v3060: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:31.625 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5584 sec 2024-03-28T19:25:32.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4939.318478 secs 2024-03-28T19:25:32.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4939.318478 secs 2024-03-28T19:25:32.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4939.318478 secs 2024-03-28T19:25:32.625 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5585 sec 2024-03-28T19:25:33.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:32 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4230.743903 secs 2024-03-28T19:25:33.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:32 smithi071 ceph-mon[36597]: pgmap v3061: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:33.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:32 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4230.743903 secs 2024-03-28T19:25:33.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:32 smithi151 ceph-mon[36481]: pgmap v3061: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:33.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:32 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4230.743903 secs 2024-03-28T19:25:33.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:32 smithi045 ceph-mon[31773]: pgmap v3061: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:33.625 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5586 sec 2024-03-28T19:25:34.625 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5587 sec 2024-03-28T19:25:35.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:34 smithi071 ceph-mon[36597]: pgmap v3062: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:35.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:34 smithi151 ceph-mon[36481]: pgmap v3062: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:35.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:34 smithi045 ceph-mon[31773]: pgmap v3062: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:35.625 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5588 sec 2024-03-28T19:25:36.626 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5589 sec 2024-03-28T19:25:37.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:36 smithi071 ceph-mon[36597]: pgmap v3063: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:37.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4944.318564 secs 2024-03-28T19:25:37.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:36 smithi151 ceph-mon[36481]: pgmap v3063: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:37.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4944.318564 secs 2024-03-28T19:25:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:36 smithi045 ceph-mon[31773]: pgmap v3063: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:37.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4944.318564 secs 2024-03-28T19:25:37.626 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5590 sec 2024-03-28T19:25:38.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:37 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4235.743964 secs 2024-03-28T19:25:38.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:37 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4235.743964 secs 2024-03-28T19:25:38.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:37 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4235.743964 secs 2024-03-28T19:25:38.626 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5591 sec 2024-03-28T19:25:39.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:38 smithi071 ceph-mon[36597]: pgmap v3064: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:39.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:38 smithi151 ceph-mon[36481]: pgmap v3064: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:39.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:38 smithi045 ceph-mon[31773]: pgmap v3064: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:39.626 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5592 sec 2024-03-28T19:25:40.626 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5593 sec 2024-03-28T19:25:41.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:40 smithi071 ceph-mon[36597]: pgmap v3065: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:41.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:40 smithi151 ceph-mon[36481]: pgmap v3065: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:41.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:40 smithi045 ceph-mon[31773]: pgmap v3065: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:41.626 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5594 sec 2024-03-28T19:25:42.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4949.318651 secs 2024-03-28T19:25:42.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4949.318651 secs 2024-03-28T19:25:42.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4949.318651 secs 2024-03-28T19:25:42.626 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5595 sec 2024-03-28T19:25:43.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:42 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4240.743980 secs 2024-03-28T19:25:43.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:42 smithi071 ceph-mon[36597]: pgmap v3066: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:43.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:42 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4240.743980 secs 2024-03-28T19:25:43.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:42 smithi151 ceph-mon[36481]: pgmap v3066: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:43.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:42 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4240.743980 secs 2024-03-28T19:25:43.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:42 smithi045 ceph-mon[31773]: pgmap v3066: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:43.627 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5596 sec 2024-03-28T19:25:44.627 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5597 sec 2024-03-28T19:25:45.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:44 smithi071 ceph-mon[36597]: pgmap v3067: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:45.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:44 smithi151 ceph-mon[36481]: pgmap v3067: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:45.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:44 smithi045 ceph-mon[31773]: pgmap v3067: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:45.627 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5598 sec 2024-03-28T19:25:46.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:45 smithi071 ceph-mon[36597]: pgmap v3068: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:46.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:45 smithi045 ceph-mon[31773]: pgmap v3068: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:46.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:45 smithi151 ceph-mon[36481]: pgmap v3068: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:46.627 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5599 sec 2024-03-28T19:25:47.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4954.318734 secs 2024-03-28T19:25:47.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4954.318734 secs 2024-03-28T19:25:47.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4954.318734 secs 2024-03-28T19:25:47.627 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5600 sec 2024-03-28T19:25:48.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:47 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4245.744037 secs 2024-03-28T19:25:48.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:47 smithi071 ceph-mon[36597]: pgmap v3069: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:48.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:47 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4245.744037 secs 2024-03-28T19:25:48.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:47 smithi151 ceph-mon[36481]: pgmap v3069: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:48.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:47 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4245.744037 secs 2024-03-28T19:25:48.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:47 smithi045 ceph-mon[31773]: pgmap v3069: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:48.627 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5601 sec 2024-03-28T19:25:49.628 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5602 sec 2024-03-28T19:25:50.628 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5603 sec 2024-03-28T19:25:51.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:50 smithi045 ceph-mon[31773]: pgmap v3070: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:51.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:50 smithi071 ceph-mon[36597]: pgmap v3070: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:51.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:50 smithi151 ceph-mon[36481]: pgmap v3070: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:51.628 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5604 sec 2024-03-28T19:25:52.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4959.318823 secs 2024-03-28T19:25:52.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4959.318823 secs 2024-03-28T19:25:52.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4959.318823 secs 2024-03-28T19:25:52.628 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5605 sec 2024-03-28T19:25:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:52 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4250.744119 secs 2024-03-28T19:25:53.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:52 smithi045 ceph-mon[31773]: pgmap v3071: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:53.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:52 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4250.744119 secs 2024-03-28T19:25:53.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:52 smithi071 ceph-mon[36597]: pgmap v3071: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:53.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:52 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4250.744119 secs 2024-03-28T19:25:53.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:52 smithi151 ceph-mon[36481]: pgmap v3071: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:53.628 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5606 sec 2024-03-28T19:25:54.628 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5607 sec 2024-03-28T19:25:55.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:54 smithi045 ceph-mon[31773]: pgmap v3072: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:55.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:54 smithi071 ceph-mon[36597]: pgmap v3072: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:55.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:54 smithi151 ceph-mon[36481]: pgmap v3072: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:55.628 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5608 sec 2024-03-28T19:25:56.629 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5609 sec 2024-03-28T19:25:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:56 smithi045 ceph-mon[31773]: pgmap v3073: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:57.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4964.318912 secs 2024-03-28T19:25:57.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:56 smithi071 ceph-mon[36597]: pgmap v3073: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:57.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4964.318912 secs 2024-03-28T19:25:57.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:56 smithi151 ceph-mon[36481]: pgmap v3073: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:25:57.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4964.318912 secs 2024-03-28T19:25:57.629 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5610 sec 2024-03-28T19:25:58.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:57 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4255.744207 secs 2024-03-28T19:25:58.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:57 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4255.744207 secs 2024-03-28T19:25:58.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:57 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4255.744207 secs 2024-03-28T19:25:58.629 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5611 sec 2024-03-28T19:25:59.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:58 smithi045 ceph-mon[31773]: pgmap v3074: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:59.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:58 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:25:59.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:58 smithi071 ceph-mon[36597]: pgmap v3074: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:59.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:58 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:25:59.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:58 smithi151 ceph-mon[36481]: pgmap v3074: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:25:59.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:58 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:25:59.629 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5612 sec 2024-03-28T19:26:00.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:26:00.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:59 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:26:00.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:25:59 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:26:00.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:26:00.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:59 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:26:00.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:25:59 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:26:00.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:26:00.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:59 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:26:00.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:25:59 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:26:00.629 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5613 sec 2024-03-28T19:26:01.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:00 smithi045 ceph-mon[31773]: pgmap v3075: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:01.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:00 smithi045 ceph-mon[31773]: pgmap v3076: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:26:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:00 smithi071 ceph-mon[36597]: pgmap v3075: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:01.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:00 smithi071 ceph-mon[36597]: pgmap v3076: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:26:01.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:00 smithi151 ceph-mon[36481]: pgmap v3075: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:01.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:00 smithi151 ceph-mon[36481]: pgmap v3076: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:26:01.630 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5614 sec 2024-03-28T19:26:02.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4969.318971 secs 2024-03-28T19:26:02.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4969.318971 secs 2024-03-28T19:26:02.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4969.318971 secs 2024-03-28T19:26:02.629 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5615 sec 2024-03-28T19:26:03.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:02 smithi045 ceph-mon[31773]: pgmap v3077: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:26:03.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:02 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4260.744294 secs 2024-03-28T19:26:03.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:02 smithi071 ceph-mon[36597]: pgmap v3077: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:26:03.311 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:02 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4260.744294 secs 2024-03-28T19:26:03.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:02 smithi151 ceph-mon[36481]: pgmap v3077: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:26:03.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:02 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4260.744294 secs 2024-03-28T19:26:03.629 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5616 sec 2024-03-28T19:26:04.629 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5617 sec 2024-03-28T19:26:05.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:04 smithi045 ceph-mon[31773]: pgmap v3078: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:26:05.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:04 smithi071 ceph-mon[36597]: pgmap v3078: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:26:05.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:04 smithi151 ceph-mon[36481]: pgmap v3078: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:26:05.630 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5618 sec 2024-03-28T19:26:06.631 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5619 sec 2024-03-28T19:26:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:06 smithi045 ceph-mon[31773]: pgmap v3079: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:26:07.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4974.319058 secs 2024-03-28T19:26:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:06 smithi071 ceph-mon[36597]: pgmap v3079: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:26:07.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4974.319058 secs 2024-03-28T19:26:07.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:06 smithi151 ceph-mon[36481]: pgmap v3079: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:26:07.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4974.319058 secs 2024-03-28T19:26:07.630 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5620 sec 2024-03-28T19:26:08.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:07 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4265.744331 secs 2024-03-28T19:26:08.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:07 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4265.744331 secs 2024-03-28T19:26:08.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:07 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4265.744331 secs 2024-03-28T19:26:08.630 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5621 sec 2024-03-28T19:26:09.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:08 smithi045 ceph-mon[31773]: pgmap v3080: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:26:09.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:08 smithi071 ceph-mon[36597]: pgmap v3080: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:26:09.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:08 smithi151 ceph-mon[36481]: pgmap v3080: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:26:09.633 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5622 sec 2024-03-28T19:26:10.631 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5623 sec 2024-03-28T19:26:11.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:10 smithi045 ceph-mon[31773]: pgmap v3081: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:26:11.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:10 smithi071 ceph-mon[36597]: pgmap v3081: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:26:11.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:10 smithi151 ceph-mon[36481]: pgmap v3081: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:26:11.631 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5624 sec 2024-03-28T19:26:12.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4979.319145 secs 2024-03-28T19:26:12.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4979.319145 secs 2024-03-28T19:26:12.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4979.319145 secs 2024-03-28T19:26:12.631 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5625 sec 2024-03-28T19:26:13.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:12 smithi071 ceph-mon[36597]: pgmap v3082: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:13.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:12 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4270.744391 secs 2024-03-28T19:26:13.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:12 smithi151 ceph-mon[36481]: pgmap v3082: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:13.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:12 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4270.744391 secs 2024-03-28T19:26:13.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:12 smithi045 ceph-mon[31773]: pgmap v3082: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:13.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:12 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4270.744391 secs 2024-03-28T19:26:13.631 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5626 sec 2024-03-28T19:26:14.631 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5627 sec 2024-03-28T19:26:15.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:14 smithi071 ceph-mon[36597]: pgmap v3083: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:15.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:14 smithi151 ceph-mon[36481]: pgmap v3083: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:15.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:14 smithi045 ceph-mon[31773]: pgmap v3083: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:15.631 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5628 sec 2024-03-28T19:26:16.631 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5629 sec 2024-03-28T19:26:17.262 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:16 smithi045 ceph-mon[31773]: pgmap v3084: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:17.262 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4984.319185 secs 2024-03-28T19:26:17.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:16 smithi071 ceph-mon[36597]: pgmap v3084: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:17.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4984.319185 secs 2024-03-28T19:26:17.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:16 smithi151 ceph-mon[36481]: pgmap v3084: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:17.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4984.319185 secs 2024-03-28T19:26:17.631 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5630 sec 2024-03-28T19:26:18.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:17 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4275.744478 secs 2024-03-28T19:26:18.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:17 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4275.744478 secs 2024-03-28T19:26:18.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:17 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4275.744478 secs 2024-03-28T19:26:18.632 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5631 sec 2024-03-28T19:26:19.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:18 smithi071 ceph-mon[36597]: pgmap v3085: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:19.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:18 smithi151 ceph-mon[36481]: pgmap v3085: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:19.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:18 smithi045 ceph-mon[31773]: pgmap v3085: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:19.631 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5632 sec 2024-03-28T19:26:20.632 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5633 sec 2024-03-28T19:26:21.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:20 smithi071 ceph-mon[36597]: pgmap v3086: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:21.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:20 smithi151 ceph-mon[36481]: pgmap v3086: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:21.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:20 smithi045 ceph-mon[31773]: pgmap v3086: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:21.632 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5634 sec 2024-03-28T19:26:22.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4989.319274 secs 2024-03-28T19:26:22.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4989.319274 secs 2024-03-28T19:26:22.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4989.319274 secs 2024-03-28T19:26:22.632 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5635 sec 2024-03-28T19:26:23.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:22 smithi071 ceph-mon[36597]: pgmap v3087: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:23.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:22 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4280.744565 secs 2024-03-28T19:26:23.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:22 smithi045 ceph-mon[31773]: pgmap v3087: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:23.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:22 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4280.744565 secs 2024-03-28T19:26:23.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:22 smithi151 ceph-mon[36481]: pgmap v3087: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:23.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:22 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4280.744565 secs 2024-03-28T19:26:23.632 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5636 sec 2024-03-28T19:26:24.632 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5637 sec 2024-03-28T19:26:25.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:24 smithi071 ceph-mon[36597]: pgmap v3088: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:25.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:24 smithi151 ceph-mon[36481]: pgmap v3088: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:25.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:24 smithi045 ceph-mon[31773]: pgmap v3088: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:25.632 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5638 sec 2024-03-28T19:26:26.632 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5639 sec 2024-03-28T19:26:27.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:26 smithi071 ceph-mon[36597]: pgmap v3089: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:27.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4994.319361 secs 2024-03-28T19:26:27.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:26 smithi151 ceph-mon[36481]: pgmap v3089: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:27.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4994.319361 secs 2024-03-28T19:26:27.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:26 smithi045 ceph-mon[31773]: pgmap v3089: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:27.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4994.319361 secs 2024-03-28T19:26:27.632 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5640 sec 2024-03-28T19:26:28.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:27 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4285.744604 secs 2024-03-28T19:26:28.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:27 smithi071 ceph-mon[36597]: pgmap v3090: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:28.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:27 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4285.744604 secs 2024-03-28T19:26:28.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:27 smithi151 ceph-mon[36481]: pgmap v3090: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:28.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:27 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4285.744604 secs 2024-03-28T19:26:28.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:27 smithi045 ceph-mon[31773]: pgmap v3090: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:28.633 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5641 sec 2024-03-28T19:26:29.634 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5642 sec 2024-03-28T19:26:30.633 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5643 sec 2024-03-28T19:26:30.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:30 smithi071 ceph-mon[36597]: pgmap v3091: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:30.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:30 smithi151 ceph-mon[36481]: pgmap v3091: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:30.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:30 smithi045 ceph-mon[31773]: pgmap v3091: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:31.633 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5644 sec 2024-03-28T19:26:31.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 4999.319445 secs 2024-03-28T19:26:31.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 4999.319445 secs 2024-03-28T19:26:31.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 4999.319445 secs 2024-03-28T19:26:32.633 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5645 sec 2024-03-28T19:26:32.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:32 smithi071 ceph-mon[36597]: pgmap v3092: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:32.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:32 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4290.744691 secs 2024-03-28T19:26:32.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:32 smithi151 ceph-mon[36481]: pgmap v3092: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:32.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:32 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4290.744691 secs 2024-03-28T19:26:32.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:32 smithi045 ceph-mon[31773]: pgmap v3092: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:32.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:32 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4290.744691 secs 2024-03-28T19:26:33.633 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5646 sec 2024-03-28T19:26:34.634 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5647 sec 2024-03-28T19:26:34.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:34 smithi071 ceph-mon[36597]: pgmap v3093: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:34.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:34 smithi151 ceph-mon[36481]: pgmap v3093: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:34.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:34 smithi045 ceph-mon[31773]: pgmap v3093: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:35.633 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5648 sec 2024-03-28T19:26:36.634 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5649 sec 2024-03-28T19:26:36.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:36 smithi071 ceph-mon[36597]: pgmap v3094: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:36.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5004.319506 secs 2024-03-28T19:26:36.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:36 smithi151 ceph-mon[36481]: pgmap v3094: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:36.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5004.319506 secs 2024-03-28T19:26:36.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:36 smithi045 ceph-mon[31773]: pgmap v3094: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:36.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5004.319506 secs 2024-03-28T19:26:37.634 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5650 sec 2024-03-28T19:26:37.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:37 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4295.744754 secs 2024-03-28T19:26:37.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:37 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4295.744754 secs 2024-03-28T19:26:37.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:37 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4295.744754 secs 2024-03-28T19:26:38.634 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5651 sec 2024-03-28T19:26:38.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:38 smithi071 ceph-mon[36597]: pgmap v3095: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:38.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:38 smithi151 ceph-mon[36481]: pgmap v3095: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:38.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:38 smithi045 ceph-mon[31773]: pgmap v3095: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:39.634 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5652 sec 2024-03-28T19:26:40.634 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5653 sec 2024-03-28T19:26:40.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:40 smithi071 ceph-mon[36597]: pgmap v3096: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:40.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:40 smithi151 ceph-mon[36481]: pgmap v3096: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:40.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:40 smithi045 ceph-mon[31773]: pgmap v3096: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:41.634 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5654 sec 2024-03-28T19:26:41.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5009.319566 secs 2024-03-28T19:26:41.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5009.319566 secs 2024-03-28T19:26:41.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5009.319566 secs 2024-03-28T19:26:42.635 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5655 sec 2024-03-28T19:26:42.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:42 smithi071 ceph-mon[36597]: pgmap v3097: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:42.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:42 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4300.744816 secs 2024-03-28T19:26:42.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:42 smithi151 ceph-mon[36481]: pgmap v3097: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:42.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:42 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4300.744816 secs 2024-03-28T19:26:42.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:42 smithi045 ceph-mon[31773]: pgmap v3097: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:42.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:42 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4300.744816 secs 2024-03-28T19:26:43.635 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5656 sec 2024-03-28T19:26:44.635 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5657 sec 2024-03-28T19:26:44.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:44 smithi071 ceph-mon[36597]: pgmap v3098: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:44.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:44 smithi151 ceph-mon[36481]: pgmap v3098: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:44.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:44 smithi045 ceph-mon[31773]: pgmap v3098: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:45.635 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5658 sec 2024-03-28T19:26:46.635 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5659 sec 2024-03-28T19:26:46.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:46 smithi071 ceph-mon[36597]: pgmap v3099: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:46.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5014.319626 secs 2024-03-28T19:26:46.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:46 smithi151 ceph-mon[36481]: pgmap v3099: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:46.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5014.319626 secs 2024-03-28T19:26:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:46 smithi045 ceph-mon[31773]: pgmap v3099: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:46.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5014.319626 secs 2024-03-28T19:26:47.635 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5660 sec 2024-03-28T19:26:47.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:47 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4305.744908 secs 2024-03-28T19:26:47.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:47 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4305.744908 secs 2024-03-28T19:26:47.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:47 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4305.744908 secs 2024-03-28T19:26:48.636 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5661 sec 2024-03-28T19:26:48.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:48 smithi071 ceph-mon[36597]: pgmap v3100: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:48.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:48 smithi151 ceph-mon[36481]: pgmap v3100: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:48.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:48 smithi045 ceph-mon[31773]: pgmap v3100: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:49.636 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5662 sec 2024-03-28T19:26:50.636 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5663 sec 2024-03-28T19:26:50.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:50 smithi071 ceph-mon[36597]: pgmap v3101: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:50.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:50 smithi151 ceph-mon[36481]: pgmap v3101: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:50.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:50 smithi045 ceph-mon[31773]: pgmap v3101: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:51.636 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5664 sec 2024-03-28T19:26:51.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:51 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5019.319686 secs 2024-03-28T19:26:51.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:51 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5019.319686 secs 2024-03-28T19:26:51.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:51 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5019.319686 secs 2024-03-28T19:26:52.636 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5665 sec 2024-03-28T19:26:52.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:52 smithi071 ceph-mon[36597]: pgmap v3102: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:52.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:52 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4310.744969 secs 2024-03-28T19:26:52.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:52 smithi151 ceph-mon[36481]: pgmap v3102: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:52.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:52 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4310.744969 secs 2024-03-28T19:26:52.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:52 smithi045 ceph-mon[31773]: pgmap v3102: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:52.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:52 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4310.744969 secs 2024-03-28T19:26:53.636 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5666 sec 2024-03-28T19:26:54.636 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5667 sec 2024-03-28T19:26:54.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:54 smithi071 ceph-mon[36597]: pgmap v3103: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:54.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:54 smithi151 ceph-mon[36481]: pgmap v3103: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:54.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:54 smithi045 ceph-mon[31773]: pgmap v3103: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:55.636 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5668 sec 2024-03-28T19:26:56.637 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5669 sec 2024-03-28T19:26:56.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:56 smithi071 ceph-mon[36597]: pgmap v3104: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:56.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5024.319758 secs 2024-03-28T19:26:56.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:56 smithi151 ceph-mon[36481]: pgmap v3104: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:56.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5024.319758 secs 2024-03-28T19:26:56.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:56 smithi045 ceph-mon[31773]: pgmap v3104: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:26:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5024.319758 secs 2024-03-28T19:26:57.637 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5670 sec 2024-03-28T19:26:57.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:57 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4315.745029 secs 2024-03-28T19:26:57.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:57 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4315.745029 secs 2024-03-28T19:26:57.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:57 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4315.745029 secs 2024-03-28T19:26:58.637 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5671 sec 2024-03-28T19:26:58.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:26:58 smithi151 ceph-mon[36481]: pgmap v3105: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:58.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:26:58 smithi045 ceph-mon[31773]: pgmap v3105: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:59.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:26:58 smithi071 ceph-mon[36597]: pgmap v3105: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:26:59.637 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5672 sec 2024-03-28T19:27:00.637 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5673 sec 2024-03-28T19:27:00.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:00 smithi045 ceph-mon[31773]: pgmap v3106: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:00.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:00 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:27:01.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:00 smithi071 ceph-mon[36597]: pgmap v3106: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:01.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:00 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:27:01.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:00 smithi151 ceph-mon[36481]: pgmap v3106: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:01.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:00 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:27:01.637 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5674 sec 2024-03-28T19:27:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:01 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5029.319798 secs 2024-03-28T19:27:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:01 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:27:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:01 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:27:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:01 smithi045 ceph-mon[31773]: pgmap v3107: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:27:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:01 smithi045 ceph-mon[31773]: pgmap v3108: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:01.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:01 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:27:02.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:01 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5029.319798 secs 2024-03-28T19:27:02.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:01 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:27:02.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:01 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:27:02.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:01 smithi071 ceph-mon[36597]: pgmap v3107: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:27:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:01 smithi071 ceph-mon[36597]: pgmap v3108: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:02.060 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:01 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:27:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:01 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5029.319798 secs 2024-03-28T19:27:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:01 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:27:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:01 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:27:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:01 smithi151 ceph-mon[36481]: pgmap v3107: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-28T19:27:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:01 smithi151 ceph-mon[36481]: pgmap v3108: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:02.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:01 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:27:02.637 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5675 sec 2024-03-28T19:27:02.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:02 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4320.745092 secs 2024-03-28T19:27:03.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:02 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4320.745092 secs 2024-03-28T19:27:03.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:02 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4320.745092 secs 2024-03-28T19:27:03.637 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5676 sec 2024-03-28T19:27:04.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:03 smithi071 ceph-mon[36597]: pgmap v3109: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:04.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:03 smithi151 ceph-mon[36481]: pgmap v3109: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:04.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:03 smithi045 ceph-mon[31773]: pgmap v3109: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:04.638 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5677 sec 2024-03-28T19:27:05.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:05 smithi071 ceph-mon[36597]: pgmap v3110: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:27:05.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:05 smithi151 ceph-mon[36481]: pgmap v3110: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:27:05.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:05 smithi045 ceph-mon[31773]: pgmap v3110: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:27:05.638 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5678 sec 2024-03-28T19:27:06.638 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5679 sec 2024-03-28T19:27:06.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5034.319856 secs 2024-03-28T19:27:06.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5034.319856 secs 2024-03-28T19:27:06.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5034.319856 secs 2024-03-28T19:27:07.638 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5680 sec 2024-03-28T19:27:07.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:07 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4325.745155 secs 2024-03-28T19:27:07.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:07 smithi071 ceph-mon[36597]: pgmap v3111: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:07.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:07 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4325.745155 secs 2024-03-28T19:27:07.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:07 smithi151 ceph-mon[36481]: pgmap v3111: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:07.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:07 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4325.745155 secs 2024-03-28T19:27:07.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:07 smithi045 ceph-mon[31773]: pgmap v3111: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:08.640 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5681 sec 2024-03-28T19:27:09.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:09 smithi071 ceph-mon[36597]: pgmap v3112: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:09.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:09 smithi151 ceph-mon[36481]: pgmap v3112: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:09.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:09 smithi045 ceph-mon[31773]: pgmap v3112: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:27:09.638 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5682 sec 2024-03-28T19:27:10.638 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5683 sec 2024-03-28T19:27:11.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:11 smithi071 ceph-mon[36597]: pgmap v3113: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:27:11.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:11 smithi151 ceph-mon[36481]: pgmap v3113: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:27:11.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:11 smithi045 ceph-mon[31773]: pgmap v3113: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:27:11.638 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5684 sec 2024-03-28T19:27:12.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:12 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5039.319894 secs 2024-03-28T19:27:12.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:12 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5039.319894 secs 2024-03-28T19:27:12.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:12 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5039.319894 secs 2024-03-28T19:27:12.639 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5685 sec 2024-03-28T19:27:13.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:13 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4330.745247 secs 2024-03-28T19:27:13.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:13 smithi071 ceph-mon[36597]: pgmap v3114: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:13.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:13 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4330.745247 secs 2024-03-28T19:27:13.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:13 smithi151 ceph-mon[36481]: pgmap v3114: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:13.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:13 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4330.745247 secs 2024-03-28T19:27:13.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:13 smithi045 ceph-mon[31773]: pgmap v3114: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:13.639 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5686 sec 2024-03-28T19:27:14.639 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5687 sec 2024-03-28T19:27:15.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:15 smithi071 ceph-mon[36597]: pgmap v3115: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:15.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:15 smithi151 ceph-mon[36481]: pgmap v3115: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:15.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:15 smithi045 ceph-mon[31773]: pgmap v3115: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:15.639 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5688 sec 2024-03-28T19:27:16.639 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5689 sec 2024-03-28T19:27:16.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5044.319971 secs 2024-03-28T19:27:16.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5044.319971 secs 2024-03-28T19:27:16.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5044.319971 secs 2024-03-28T19:27:17.639 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5690 sec 2024-03-28T19:27:17.710 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:17 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4335.745309 secs 2024-03-28T19:27:17.711 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:17 smithi071 ceph-mon[36597]: pgmap v3116: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:17.723 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:17 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4335.745309 secs 2024-03-28T19:27:17.724 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:17 smithi151 ceph-mon[36481]: pgmap v3116: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:17.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:17 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4335.745309 secs 2024-03-28T19:27:17.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:17 smithi045 ceph-mon[31773]: pgmap v3116: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:18.639 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5691 sec 2024-03-28T19:27:19.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:19 smithi071 ceph-mon[36597]: pgmap v3117: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:19.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:19 smithi151 ceph-mon[36481]: pgmap v3117: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:19.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:19 smithi045 ceph-mon[31773]: pgmap v3117: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:19.640 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5692 sec 2024-03-28T19:27:20.640 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5693 sec 2024-03-28T19:27:21.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:21 smithi071 ceph-mon[36597]: pgmap v3118: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:21.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:21 smithi151 ceph-mon[36481]: pgmap v3118: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:21.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:21 smithi045 ceph-mon[31773]: pgmap v3118: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:21.640 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5694 sec 2024-03-28T19:27:22.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:22 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5049.320018 secs 2024-03-28T19:27:22.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:22 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5049.320018 secs 2024-03-28T19:27:22.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:22 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5049.320018 secs 2024-03-28T19:27:22.640 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5695 sec 2024-03-28T19:27:23.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:23 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4340.745400 secs 2024-03-28T19:27:23.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:23 smithi071 ceph-mon[36597]: pgmap v3119: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:23.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:23 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4340.745400 secs 2024-03-28T19:27:23.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:23 smithi151 ceph-mon[36481]: pgmap v3119: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:23.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:23 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4340.745400 secs 2024-03-28T19:27:23.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:23 smithi045 ceph-mon[31773]: pgmap v3119: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:23.640 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5696 sec 2024-03-28T19:27:24.640 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5697 sec 2024-03-28T19:27:25.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:25 smithi071 ceph-mon[36597]: pgmap v3120: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:25.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:25 smithi151 ceph-mon[36481]: pgmap v3120: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:25.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:25 smithi045 ceph-mon[31773]: pgmap v3120: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:25.640 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5698 sec 2024-03-28T19:27:26.641 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5699 sec 2024-03-28T19:27:26.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5054.320102 secs 2024-03-28T19:27:26.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5054.320102 secs 2024-03-28T19:27:26.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5054.320102 secs 2024-03-28T19:27:27.641 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5700 sec 2024-03-28T19:27:27.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:27 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4345.745463 secs 2024-03-28T19:27:27.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:27 smithi071 ceph-mon[36597]: pgmap v3121: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:27.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:27 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4345.745463 secs 2024-03-28T19:27:27.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:27 smithi151 ceph-mon[36481]: pgmap v3121: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:27.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:27 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4345.745463 secs 2024-03-28T19:27:27.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:27 smithi045 ceph-mon[31773]: pgmap v3121: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:28.641 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5701 sec 2024-03-28T19:27:29.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:29 smithi071 ceph-mon[36597]: pgmap v3122: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:29.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:29 smithi151 ceph-mon[36481]: pgmap v3122: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:29.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:29 smithi045 ceph-mon[31773]: pgmap v3122: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:29.641 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5702 sec 2024-03-28T19:27:30.641 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5703 sec 2024-03-28T19:27:31.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:31 smithi071 ceph-mon[36597]: pgmap v3123: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:31.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:31 smithi151 ceph-mon[36481]: pgmap v3123: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:31.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:31 smithi045 ceph-mon[31773]: pgmap v3123: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:31.641 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5704 sec 2024-03-28T19:27:32.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:32 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5059.320192 secs 2024-03-28T19:27:32.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:32 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5059.320192 secs 2024-03-28T19:27:32.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:32 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5059.320192 secs 2024-03-28T19:27:32.641 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5705 sec 2024-03-28T19:27:33.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:33 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4350.745520 secs 2024-03-28T19:27:33.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:33 smithi071 ceph-mon[36597]: pgmap v3124: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:33.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:33 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4350.745520 secs 2024-03-28T19:27:33.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:33 smithi151 ceph-mon[36481]: pgmap v3124: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:33.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:33 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4350.745520 secs 2024-03-28T19:27:33.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:33 smithi045 ceph-mon[31773]: pgmap v3124: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:33.642 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5706 sec 2024-03-28T19:27:34.642 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5707 sec 2024-03-28T19:27:35.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:35 smithi071 ceph-mon[36597]: pgmap v3125: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:35.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:35 smithi151 ceph-mon[36481]: pgmap v3125: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:35.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:35 smithi045 ceph-mon[31773]: pgmap v3125: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:35.642 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5708 sec 2024-03-28T19:27:36.642 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5709 sec 2024-03-28T19:27:36.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5064.320280 secs 2024-03-28T19:27:36.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5064.320280 secs 2024-03-28T19:27:36.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5064.320280 secs 2024-03-28T19:27:37.642 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5710 sec 2024-03-28T19:27:37.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:37 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4355.745603 secs 2024-03-28T19:27:37.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:37 smithi071 ceph-mon[36597]: pgmap v3126: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:37.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:37 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4355.745603 secs 2024-03-28T19:27:37.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:37 smithi151 ceph-mon[36481]: pgmap v3126: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:37.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:37 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4355.745603 secs 2024-03-28T19:27:37.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:37 smithi045 ceph-mon[31773]: pgmap v3126: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:38.642 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5711 sec 2024-03-28T19:27:39.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:39 smithi071 ceph-mon[36597]: pgmap v3127: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:39.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:39 smithi151 ceph-mon[36481]: pgmap v3127: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:39.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:39 smithi045 ceph-mon[31773]: pgmap v3127: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:39.642 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5712 sec 2024-03-28T19:27:40.643 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5713 sec 2024-03-28T19:27:41.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:41 smithi071 ceph-mon[36597]: pgmap v3128: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:41.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:41 smithi151 ceph-mon[36481]: pgmap v3128: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:41.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:41 smithi045 ceph-mon[31773]: pgmap v3128: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:41.643 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5714 sec 2024-03-28T19:27:42.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:42 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5069.320367 secs 2024-03-28T19:27:42.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:42 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5069.320367 secs 2024-03-28T19:27:42.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:42 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5069.320367 secs 2024-03-28T19:27:42.643 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5715 sec 2024-03-28T19:27:43.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:43 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4360.745663 secs 2024-03-28T19:27:43.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:43 smithi071 ceph-mon[36597]: pgmap v3129: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:43.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:43 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4360.745663 secs 2024-03-28T19:27:43.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:43 smithi151 ceph-mon[36481]: pgmap v3129: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:43.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:43 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4360.745663 secs 2024-03-28T19:27:43.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:43 smithi045 ceph-mon[31773]: pgmap v3129: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:43.643 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5716 sec 2024-03-28T19:27:44.643 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5717 sec 2024-03-28T19:27:45.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:45 smithi071 ceph-mon[36597]: pgmap v3130: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:45.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:45 smithi151 ceph-mon[36481]: pgmap v3130: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:45.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:45 smithi045 ceph-mon[31773]: pgmap v3130: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:45.643 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5718 sec 2024-03-28T19:27:46.644 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5719 sec 2024-03-28T19:27:46.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5074.320455 secs 2024-03-28T19:27:46.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5074.320455 secs 2024-03-28T19:27:46.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5074.320455 secs 2024-03-28T19:27:47.645 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5720 sec 2024-03-28T19:27:47.710 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:47 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4365.745751 secs 2024-03-28T19:27:47.710 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:47 smithi071 ceph-mon[36597]: pgmap v3131: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:47.723 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:47 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4365.745751 secs 2024-03-28T19:27:47.723 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:47 smithi151 ceph-mon[36481]: pgmap v3131: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:47.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:47 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4365.745751 secs 2024-03-28T19:27:47.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:47 smithi045 ceph-mon[31773]: pgmap v3131: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:48.644 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5721 sec 2024-03-28T19:27:49.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:49 smithi071 ceph-mon[36597]: pgmap v3132: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:49.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:49 smithi151 ceph-mon[36481]: pgmap v3132: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:49.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:49 smithi045 ceph-mon[31773]: pgmap v3132: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:49.644 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5722 sec 2024-03-28T19:27:50.644 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5723 sec 2024-03-28T19:27:51.218 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:51 smithi045 ceph-mon[31773]: pgmap v3133: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:51.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:51 smithi071 ceph-mon[36597]: pgmap v3133: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:51.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:51 smithi151 ceph-mon[36481]: pgmap v3133: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:51.644 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5724 sec 2024-03-28T19:27:52.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:52 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5079.320487 secs 2024-03-28T19:27:52.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:52 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5079.320487 secs 2024-03-28T19:27:52.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:52 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5079.320487 secs 2024-03-28T19:27:52.644 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5725 sec 2024-03-28T19:27:53.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:53 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4370.745838 secs 2024-03-28T19:27:53.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:53 smithi071 ceph-mon[36597]: pgmap v3134: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:53.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:53 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4370.745838 secs 2024-03-28T19:27:53.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:53 smithi151 ceph-mon[36481]: pgmap v3134: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:53.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:53 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4370.745838 secs 2024-03-28T19:27:53.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:53 smithi045 ceph-mon[31773]: pgmap v3134: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:53.644 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5726 sec 2024-03-28T19:27:54.645 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5727 sec 2024-03-28T19:27:55.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:55 smithi071 ceph-mon[36597]: pgmap v3135: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:55.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:55 smithi151 ceph-mon[36481]: pgmap v3135: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:55.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:55 smithi045 ceph-mon[31773]: pgmap v3135: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:55.645 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5728 sec 2024-03-28T19:27:56.645 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5729 sec 2024-03-28T19:27:56.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:56 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5084.320547 secs 2024-03-28T19:27:56.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:56 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5084.320547 secs 2024-03-28T19:27:56.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:56 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5084.320547 secs 2024-03-28T19:27:57.645 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5730 sec 2024-03-28T19:27:57.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:57 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4375.745900 secs 2024-03-28T19:27:57.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:57 smithi071 ceph-mon[36597]: pgmap v3136: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:57.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:57 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4375.745900 secs 2024-03-28T19:27:57.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:57 smithi151 ceph-mon[36481]: pgmap v3136: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:57 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4375.745900 secs 2024-03-28T19:27:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:57 smithi045 ceph-mon[31773]: pgmap v3136: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:27:58.645 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5731 sec 2024-03-28T19:27:59.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:27:59 smithi071 ceph-mon[36597]: pgmap v3137: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:59.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:27:59 smithi151 ceph-mon[36481]: pgmap v3137: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:59.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:27:59 smithi045 ceph-mon[31773]: pgmap v3137: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:27:59.645 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5732 sec 2024-03-28T19:28:00.646 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5733 sec 2024-03-28T19:28:01.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:01 smithi071 ceph-mon[36597]: pgmap v3138: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:01.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:01 smithi151 ceph-mon[36481]: pgmap v3138: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:01.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:01 smithi045 ceph-mon[31773]: pgmap v3138: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:01.645 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5734 sec 2024-03-28T19:28:02.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:02 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5089.320606 secs 2024-03-28T19:28:02.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:02 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:28:02.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:02 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5089.320606 secs 2024-03-28T19:28:02.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:02 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:28:02.414 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:02 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5089.320606 secs 2024-03-28T19:28:02.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:02 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T19:28:02.646 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5735 sec 2024-03-28T19:28:03.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:03 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4380.745964 secs 2024-03-28T19:28:03.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:03 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:28:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:03 smithi071 ceph-mon[36597]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:28:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:03 smithi071 ceph-mon[36597]: pgmap v3139: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:03 smithi071 ceph-mon[36597]: pgmap v3140: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:03.310 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:03 smithi071 ceph-mon[36597]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:28:03.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:03 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4380.745964 secs 2024-03-28T19:28:03.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:03 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:28:03.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:03 smithi151 ceph-mon[36481]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:28:03.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:03 smithi151 ceph-mon[36481]: pgmap v3139: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:03.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:03 smithi151 ceph-mon[36481]: pgmap v3140: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:03.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:03 smithi151 ceph-mon[36481]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:28:03.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:03 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4380.745964 secs 2024-03-28T19:28:03.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:03 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T19:28:03.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:03 smithi045 ceph-mon[31773]: from='mgr.25354 172.21.15.45:0/574487584' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T19:28:03.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:03 smithi045 ceph-mon[31773]: pgmap v3139: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:03.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:03 smithi045 ceph-mon[31773]: pgmap v3140: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:03.415 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:03 smithi045 ceph-mon[31773]: from='mgr.25354 ' entity='mgr.x' 2024-03-28T19:28:03.646 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5736 sec 2024-03-28T19:28:04.646 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5737 sec 2024-03-28T19:28:05.646 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5738 sec 2024-03-28T19:28:05.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:05 smithi045 ceph-mon[31773]: pgmap v3141: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:28:06.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:05 smithi071 ceph-mon[36597]: pgmap v3141: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:28:06.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:05 smithi151 ceph-mon[36481]: pgmap v3141: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-28T19:28:06.646 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5739 sec 2024-03-28T19:28:06.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:06 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5094.320621 secs 2024-03-28T19:28:07.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:06 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5094.320621 secs 2024-03-28T19:28:07.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:06 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5094.320621 secs 2024-03-28T19:28:07.647 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5740 sec 2024-03-28T19:28:07.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:07 smithi045 ceph-mon[31773]: pgmap v3142: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:07.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:07 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4385.746027 secs 2024-03-28T19:28:08.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:07 smithi071 ceph-mon[36597]: pgmap v3142: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:08.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:07 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4385.746027 secs 2024-03-28T19:28:08.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:07 smithi151 ceph-mon[36481]: pgmap v3142: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:08.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:07 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4385.746027 secs 2024-03-28T19:28:08.646 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5741 sec 2024-03-28T19:28:09.647 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5742 sec 2024-03-28T19:28:09.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:09 smithi045 ceph-mon[31773]: pgmap v3143: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:10.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:09 smithi071 ceph-mon[36597]: pgmap v3143: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:10.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:09 smithi151 ceph-mon[36481]: pgmap v3143: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:10.647 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5743 sec 2024-03-28T19:28:11.647 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5744 sec 2024-03-28T19:28:11.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:11 smithi045 ceph-mon[31773]: pgmap v3144: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:11.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:11 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5099.320685 secs 2024-03-28T19:28:12.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:11 smithi071 ceph-mon[36597]: pgmap v3144: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:12.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:11 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5099.320685 secs 2024-03-28T19:28:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:11 smithi151 ceph-mon[36481]: pgmap v3144: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-28T19:28:12.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:11 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5099.320685 secs 2024-03-28T19:28:12.647 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5745 sec 2024-03-28T19:28:12.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:12 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4390.746090 secs 2024-03-28T19:28:13.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:12 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4390.746090 secs 2024-03-28T19:28:13.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:12 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4390.746090 secs 2024-03-28T19:28:13.647 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5746 sec 2024-03-28T19:28:13.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:13 smithi045 ceph-mon[31773]: pgmap v3145: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:28:14.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:13 smithi071 ceph-mon[36597]: pgmap v3145: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:28:14.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:13 smithi151 ceph-mon[36481]: pgmap v3145: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-28T19:28:14.647 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5747 sec 2024-03-28T19:28:15.647 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5748 sec 2024-03-28T19:28:15.914 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:15 smithi045 ceph-mon[31773]: pgmap v3146: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:16.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:15 smithi071 ceph-mon[36597]: pgmap v3146: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:16.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:15 smithi151 ceph-mon[36481]: pgmap v3146: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:16.647 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5749 sec 2024-03-28T19:28:17.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:16 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5104.320796 secs 2024-03-28T19:28:17.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:16 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5104.320796 secs 2024-03-28T19:28:17.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:16 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5104.320796 secs 2024-03-28T19:28:17.648 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5750 sec 2024-03-28T19:28:17.960 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:17 smithi071 ceph-mon[36597]: pgmap v3147: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:17.960 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:17 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4395.746167 secs 2024-03-28T19:28:17.973 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:17 smithi151 ceph-mon[36481]: pgmap v3147: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:17.973 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:17 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4395.746167 secs 2024-03-28T19:28:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:17 smithi045 ceph-mon[31773]: pgmap v3147: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:18.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:17 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4395.746167 secs 2024-03-28T19:28:18.648 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5751 sec 2024-03-28T19:28:19.648 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5752 sec 2024-03-28T19:28:20.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:19 smithi071 ceph-mon[36597]: pgmap v3148: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:20.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:19 smithi151 ceph-mon[36481]: pgmap v3148: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:20.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:19 smithi045 ceph-mon[31773]: pgmap v3148: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:20.648 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5753 sec 2024-03-28T19:28:21.648 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5754 sec 2024-03-28T19:28:22.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:21 smithi071 ceph-mon[36597]: pgmap v3149: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:22.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:21 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5109.320858 secs 2024-03-28T19:28:22.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:21 smithi151 ceph-mon[36481]: pgmap v3149: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:22.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:21 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5109.320858 secs 2024-03-28T19:28:22.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:21 smithi045 ceph-mon[31773]: pgmap v3149: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:22.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:21 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5109.320858 secs 2024-03-28T19:28:22.648 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5755 sec 2024-03-28T19:28:23.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:22 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4400.746207 secs 2024-03-28T19:28:23.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:22 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4400.746207 secs 2024-03-28T19:28:23.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:22 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4400.746207 secs 2024-03-28T19:28:23.649 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5756 sec 2024-03-28T19:28:24.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:23 smithi071 ceph-mon[36597]: pgmap v3150: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:24.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:23 smithi151 ceph-mon[36481]: pgmap v3150: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:24.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:23 smithi045 ceph-mon[31773]: pgmap v3150: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:24.649 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5757 sec 2024-03-28T19:28:25.649 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5758 sec 2024-03-28T19:28:25.968 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:25 smithi045 ceph-mon[31773]: pgmap v3151: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:26.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:25 smithi071 ceph-mon[36597]: pgmap v3151: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:26.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:25 smithi151 ceph-mon[36481]: pgmap v3151: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:26.649 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5759 sec 2024-03-28T19:28:27.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:26 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5114.320918 secs 2024-03-28T19:28:27.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:26 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5114.320918 secs 2024-03-28T19:28:27.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:26 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5114.320918 secs 2024-03-28T19:28:27.649 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5760 sec 2024-03-28T19:28:28.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:27 smithi071 ceph-mon[36597]: pgmap v3152: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:28.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:27 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4405.746265 secs 2024-03-28T19:28:28.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:27 smithi151 ceph-mon[36481]: pgmap v3152: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:28.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:27 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4405.746265 secs 2024-03-28T19:28:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:27 smithi045 ceph-mon[31773]: pgmap v3152: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:28.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:27 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4405.746265 secs 2024-03-28T19:28:28.649 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5761 sec 2024-03-28T19:28:29.649 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5762 sec 2024-03-28T19:28:30.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:29 smithi071 ceph-mon[36597]: pgmap v3153: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:30.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:29 smithi151 ceph-mon[36481]: pgmap v3153: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:30.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:29 smithi045 ceph-mon[31773]: pgmap v3153: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:30.649 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5763 sec 2024-03-28T19:28:31.650 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5764 sec 2024-03-28T19:28:32.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:31 smithi071 ceph-mon[36597]: pgmap v3154: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:32.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:31 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5119.320978 secs 2024-03-28T19:28:32.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:31 smithi151 ceph-mon[36481]: pgmap v3154: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:32.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:31 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5119.320978 secs 2024-03-28T19:28:32.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:31 smithi045 ceph-mon[31773]: pgmap v3154: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T19:28:32.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:31 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5119.320978 secs 2024-03-28T19:28:32.650 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5765 sec 2024-03-28T19:28:33.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:32 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4410.746353 secs 2024-03-28T19:28:33.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:32 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4410.746353 secs 2024-03-28T19:28:33.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:32 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4410.746353 secs 2024-03-28T19:28:33.650 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5766 sec 2024-03-28T19:28:34.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:33 smithi071 ceph-mon[36597]: pgmap v3155: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:34.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:33 smithi151 ceph-mon[36481]: pgmap v3155: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:34.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:33 smithi045 ceph-mon[31773]: pgmap v3155: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:34.650 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5767 sec 2024-03-28T19:28:35.650 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5768 sec 2024-03-28T19:28:36.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:35 smithi071 ceph-mon[36597]: pgmap v3156: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:36.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:35 smithi151 ceph-mon[36481]: pgmap v3156: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:36.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:35 smithi045 ceph-mon[31773]: pgmap v3156: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:36.650 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5769 sec 2024-03-28T19:28:37.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:36 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5124.321056 secs 2024-03-28T19:28:37.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:36 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5124.321056 secs 2024-03-28T19:28:37.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:36 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5124.321056 secs 2024-03-28T19:28:37.650 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5770 sec 2024-03-28T19:28:38.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:37 smithi071 ceph-mon[36597]: pgmap v3157: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T19:28:38.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:37 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4415.746442 secs 2024-03-28T19:28:38.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:37 smithi045 ceph-mon[31773]: pgmap v3157: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T19:28:38.062 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:37 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4415.746442 secs 2024-03-28T19:28:38.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:37 smithi151 ceph-mon[36481]: pgmap v3157: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T19:28:38.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:37 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4415.746442 secs 2024-03-28T19:28:38.650 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5771 sec 2024-03-28T19:28:39.651 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5772 sec 2024-03-28T19:28:40.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:39 smithi071 ceph-mon[36597]: pgmap v3158: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T19:28:40.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:39 smithi151 ceph-mon[36481]: pgmap v3158: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T19:28:40.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:39 smithi045 ceph-mon[31773]: pgmap v3158: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T19:28:40.651 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5773 sec 2024-03-28T19:28:41.651 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5774 sec 2024-03-28T19:28:42.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:41 smithi071 ceph-mon[36597]: pgmap v3159: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T19:28:42.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:41 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5129.321148 secs 2024-03-28T19:28:42.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:41 smithi151 ceph-mon[36481]: pgmap v3159: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T19:28:42.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:41 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5129.321148 secs 2024-03-28T19:28:42.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:41 smithi045 ceph-mon[31773]: pgmap v3159: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T19:28:42.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:41 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5129.321148 secs 2024-03-28T19:28:42.651 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5775 sec 2024-03-28T19:28:43.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:42 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4420.746530 secs 2024-03-28T19:28:43.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:42 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4420.746530 secs 2024-03-28T19:28:43.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:42 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4420.746530 secs 2024-03-28T19:28:43.651 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5776 sec 2024-03-28T19:28:44.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:43 smithi071 ceph-mon[36597]: pgmap v3160: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T19:28:44.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:43 smithi151 ceph-mon[36481]: pgmap v3160: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T19:28:44.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:43 smithi045 ceph-mon[31773]: pgmap v3160: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T19:28:44.652 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5777 sec 2024-03-28T19:28:45.652 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5778 sec 2024-03-28T19:28:46.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:45 smithi071 ceph-mon[36597]: pgmap v3161: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T19:28:46.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:45 smithi151 ceph-mon[36481]: pgmap v3161: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T19:28:46.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:45 smithi045 ceph-mon[31773]: pgmap v3161: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-28T19:28:46.652 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5779 sec 2024-03-28T19:28:47.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:46 smithi071 ceph-mon[36597]: 5 slow requests, 0 included below; oldest blocked for > 5134.321234 secs 2024-03-28T19:28:47.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:46 smithi151 ceph-mon[36481]: 5 slow requests, 0 included below; oldest blocked for > 5134.321234 secs 2024-03-28T19:28:47.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:46 smithi045 ceph-mon[31773]: 5 slow requests, 0 included below; oldest blocked for > 5134.321234 secs 2024-03-28T19:28:47.652 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5780 sec 2024-03-28T19:28:48.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:47 smithi071 ceph-mon[36597]: pgmap v3162: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T19:28:48.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:47 smithi071 ceph-mon[36597]: 3 slow requests, 0 included below; oldest blocked for > 4425.746618 secs 2024-03-28T19:28:48.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:47 smithi151 ceph-mon[36481]: pgmap v3162: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T19:28:48.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:47 smithi151 ceph-mon[36481]: 3 slow requests, 0 included below; oldest blocked for > 4425.746618 secs 2024-03-28T19:28:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:47 smithi045 ceph-mon[31773]: pgmap v3162: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T19:28:48.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:47 smithi045 ceph-mon[31773]: 3 slow requests, 0 included below; oldest blocked for > 4425.746618 secs 2024-03-28T19:28:48.652 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5781 sec 2024-03-28T19:28:49.653 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5782 sec 2024-03-28T19:28:50.059 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 28 19:28:49 smithi071 ceph-mon[36597]: pgmap v3163: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:50.072 INFO:journalctl@ceph.mon.c.smithi151.stdout:Mar 28 19:28:49 smithi151 ceph-mon[36481]: pgmap v3163: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:50.165 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 28 19:28:49 smithi045 ceph-mon[31773]: pgmap v3163: 97 pgs: 97 active+clean; 27 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-28T19:28:50.652 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5783 sec 2024-03-28T19:28:51.653 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 5784 sec 2024-03-28T19:28:51.829 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-03-28T19:28:51.829 DEBUG:teuthology.task.console_log:Killing console logger for smithi045 2024-03-28T19:28:51.830 DEBUG:teuthology.task.console_log:Killing console logger for smithi071 2024-03-28T19:28:51.830 DEBUG:teuthology.task.console_log:Killing console logger for smithi151 2024-03-28T19:28:51.830 DEBUG:teuthology.exit:Finished running handlers